Real-time emotion detection by quantitative facial motion analysis

by Jordan R. Saadon, Fan Yang, Ryan Burgert, Selma Mohammad, Theresa Gammel, Michael Sepe, Miriam Rafailovich, Charles B. Mikell, Pawel Polak, Sima Mofakham

Background

Research into mood and emotion has often depended on slow and subjective self-report, highlighting a need for rapid, accurate, and objective assessment tools.

Methods

To address this gap, we developed a method using digital image speckle correlation (DISC), which tracks subtle changes in facial expressions invisible to the naked eye, to assess emotions in real-time. We presented ten participants with visual stimuli triggering neutral, happy, and sad emotions and quantified their associated facial responses via detailed DISC analysis.

Results

We identified key alterations in facial expression (facial maps) that reliably signal changes in mood state across all individuals based on these data. Furthermore, principal component analysis of these facial maps identified regions associated with happy and sad emotions. Compared with commercial deep learning solutions that use individual images to detect facial expressions and classify emotions, such as Amazon Rekognition, our DISC-based classifiers utilize frame-to-frame changes. Our data show that DISC-based classifiers deliver substantially better predictions, and they are inherently free of racial or gender bias.

Limitations

Our sample size was limited, and participants were aware their faces were recorded on video. Despite this, our results remained consistent across individuals.

Conclusions

We demonstrate that DISC-based facial analysis can be used to reliably identify an individual’s emotion and may provide a robust and economic modality for real-time, noninvasive clinical monitoring in the future.