Beyond the human eye: behaviour from video and audio
Some of the most telling signs in mental health last less than a second: a micro-movement, a glance held too long, the moment words and face tell different stories. Trained clinicians catch some of them; coding them by hand is slow and varies between coders. Here, video and audio of play, conversation and therapy become behavioural signatures precise enough to follow one person across sessions, a step towards precision and personalised medicine. Over two days, vision-language models, audio-visual models and pose tracking meet open footage and human coders, and clinicians question and correct the labels in plain language. Leads: change across sessions, movement patterns shared by overlapping conditions.
Resources
- Video / vision models
- Speech models
- Language-model endpoints
- Anonymization tools
- LISBET — social behaviour transformer, Bellone lab (academic non-commercial) ↗
- OpenFace — facial action units, gaze, head pose ↗
- DeepFace — facial affect analysis ↗
- Wav2Vec2 / HuBERT — speech representations ↗
- EmoBox — speech emotion recognition toolkit ↗
- Label Studio — annotation platform ↗
- Kineo — calibration-free 3D motion capture from ordinary cameras (research use only) ↗
