This project aligns facial expression, EEG, EMG, ECG, skin conductance, speech and language in time, and builds an emotion recognition engine that reports state, intensity and trajectory in context, as a basis for mental-health monitoring and emotion-aware human–AI interaction.
Data in
- Facial expression, gesture and gaze
- EEG, EMG, ECG and skin conductance
- Speech signal and prosody
- Written language, activity and situational context
What it is meant to deliver
- Emotion classification with intensity
- Temporal dynamics and early risk detection
- Personalized insights and recommendations
- Support for mental-health monitoring and clinicians
Overview
The full picture.
From face, body signals, voice and words to an emotional state a clinician or a system can act on. Planned framework; no results are reported here.
01Modalities
- Visualfacial expression, head pose, gesture, gaze
- PhysiologicalEEG, EMG, ECG, skin conductance, respiration
- Audiospeech signal, prosody, voice quality, speaking rate
- Textwritten language, sentiment, notes and diaries
- Contextactivity, social interaction, location, time
02Harmonize & align
- Preprocessing
- Temporal synchronisation
- Feature fusion
- Missing-data handling
- Emotion knowledge graphbrain–body signals, behavior, context, interventions
03Recognition engine
Emotion engineState · intensity · trajectory · context
Models
- Deep multimodal learning
- CNN & transformer
- Graph learning
- Neuro-symbolic & causal AI
Analysis
- Emotion classification (happiness, sadness, anger, fear, neutral)
- Temporal & context analysis (dynamics, early risk detection)
- Personalized insight & recommendation
04Applications
- Mental health monitoringdepression, anxiety, stress; relapse warning
- Personalized interventionadaptive digital therapeutics, behavior change
- Clinical decision supportemotion profiling, treatment response monitoring
- Human–AI interactionemotion-aware conversational AI, assistive robots
✓Trustworthy, responsible & ethical AI
- Privacy & security
- Fairness & bias mitigation
- Explainability
- Robustness & reliability
- Human in the loop
- Regulatory compliance & ethics
- Early detection
- Personalized care
- Improved outcomes
- Scalable solutions
- Healthier individuals & communities
Illustrations generated with ChatGPT for the AI4Health Lab. They depict research themes, not study data.