Repository logo
 

Alone versus in-a-group: A multi-modal framework for automatic affect recognition

Accepted version
Peer-reviewed

Type

Article

Change log

Authors

Mou, W 
Patras, I 

Abstract

jats:pRecognition and analysis of human affect has been researched extensively within the field of computer science in the past two decades. However, most of the past research in automatic analysis of human affect has focused on the recognition of affect displayed by people in individual settings and little attention has been paid to the analysis of the affect expressed in group settings. In this article, we first analyze the affect expressed by each individual in terms of arousal and valence dimensions in both individual and group videos and then propose methods to recognize the contextual information, i.e., whether a person is alone or in-a-group by analyzing their face and body behavioral cues. For affect analysis, we first devise affect recognition models separately in individual and group videos and then introduce a cross-condition affect recognition model that is trained by combining the two different types of data. We conduct a set of experiments on two datasets that contain both individual and group videos. Our experiments show that (1) the proposed Volume Quantized Local Zernike Moments Fisher Vector outperforms other unimodal features in affect analysis; (2) the temporal learning model, Long-Short Term Memory Networks, works better than the static learning model, Support Vector Machine; (3) decision fusion helps to improve affect recognition, indicating that body behaviors carry emotional information that is complementary rather than redundant to the emotion content in facial behaviors; and (4) it is possible to predict the context, i.e., whether a person is alone or in-a-group, using their non-verbal behavioral cues.</jats:p>

Description

Keywords

Affect analysis, multimodal interaction, group settings, non-verbal behaviours, context analysis

Journal Title

ACM Transactions on Multimedia Computing, Communications and Applications

Conference Name

Journal ISSN

1551-6857
1551-6865

Volume Title

15

Publisher

Association for Computing Machinery (ACM)
Sponsorship
Engineering and Physical Sciences Research Council (EP/L00416X/1)