Repository logo
 

AULA-Caps: Lifecycle-Aware Capsule Networks for Spatio-Temporal Analysis of Facial Actions

Accepted version
Peer-reviewed

Type

Conference Object

Change log

Authors

Kalkan, Sinan 

Abstract

Most state-of-the-art approaches for Facial Action Unit (AU) detection rely on evaluating static frames, encoding a snapshot of heightened facial activity. In real-world interactions, however, facial expressions are more subtle and evolve over time requiring AU detection models to learn spatial as well as temporal information. In this work, we focus on both spatial and spatio-temporal features encoding the temporal evolution of facial AU activation. We propose the Action Unit Lifecycle- Aware Capsule Network (AULA-Caps) for AU detection using both frame and sequence-level features. While, at the frame- level, the capsule layers of AULA-Caps learn spatial feature primitives to determine AU activations, at the sequence-level, it learns temporal dependencies between contiguous frames by focusing on relevant spatio-temporal segments in the sequence. The learnt feature capsules are routed together such that the model learns to selectively focus on spatial or spatio-temporal information depending upon the AU lifecycle. The proposed model is evaluated on popular benchmarks, namely BP4D and GFT datasets, obtaining state-of-the-art results for both.

Description

Keywords

Affective Computing, Facial Action Units, Capsule Networks, Computer Vision, Machine Learning, Neural Networks

Journal Title

2021 16TH IEEE INTERNATIONAL CONFERENCE ON AUTOMATIC FACE AND GESTURE RECOGNITION (FG 2021)

Conference Name

IEEE International Conference on Automatic Face and Gesture Recognition (FG) 2021

Journal ISSN

2326-5396

Volume Title

Publisher

IEEE
Sponsorship
EPSRC (2107412)
Engineering and Physical Sciences Research Council (EP/R030782/1)
European Commission Horizon 2020 (H2020) Societal Challenges (826232)
EPSRC grant EP/R513180/1 (ref. 2107412). EPSRC project ARoEQ under grant ref. EP/R030782/1. European Union’s Horizon 2020 research and innovation programme WorkingAge project under grant agreement No. 826232.