Interventions that measure: XR, games and smart toys
What if every moment of play could also be a measurement? A game in augmented reality, a virtual scene for practising social cues, a toy that responds to how a child plays: each can train attention, social skills or emotion recognition, and each can record gaze, movement, choices and physiology along the way, so clinicians see more than a final score and can tailor the next session. Headsets with built-in eye tracking, language-model characters and sensor-equipped toys are within reach for a two-day build, together with a plan to test the best idea with patients. Worth a first look: augmented-reality cues for attention after stroke.
Resources
- Wearables and phone sensing
- Brain signals and imaging
- Sensitive data handling
- GazeBaseVR — eye tracking recorded in VR (figshare, CC BY 4.0) ↗
- Data platform and governance
- Video / vision models
- Lab equipment
- Speech models
- Language-model endpoints
- OpenXR — cross-headset XR standard ↗
- Godot — open game engine with XR support (MIT) ↗
- A-Frame — VR/AR in the browser via WebXR (MIT) ↗
- Unity XR Interaction Toolkit — examples ↗
- jsPsych — browser-based tasks for gamified assessment (MIT) ↗
- PsychoPy — experiment and task builder (GPL-3.0) ↗
- Pupil Labs — open eye tracking (LGPL-3.0) ↗
- Tobii Pro Glasses 3 — wearable eye tracking (commercial) ↗
- Lab Streaming Layer — sync gaze, physiology and in-game events ↗
- Arduino — open hardware for sensor toys ↗
- ESP32 Arduino core — wireless microcontroller (LGPL-2.1) ↗
- llama.cpp — on-device language model (MIT) ↗
- whisper.cpp — on-device speech recognition (MIT) ↗
- Piper — on-device speech synthesis (MIT) ↗
