Repository logo
 

XFlow: Cross-Modal Deep Neural Networks for Audiovisual Classification.

Accepted version
Peer-reviewed

Type

Article

Change log

Authors

Cangea, Catalina 
Velickovic, Petar 

Abstract

In recent years, there have been numerous developments toward solving multimodal tasks, aiming to learn a stronger representation than through a single modality. Certain aspects of the data can be particularly useful in this case--for example, correlations in the space or time domain across modalities--but should be wisely exploited in order to benefit from their full predictive potential. We propose two deep learning architectures with multimodal cross connections that allow for dataflow between several feature extractors (XFlow). Our models derive more interpretable features and achieve better performances than models that do not exchange representations, usefully exploiting correlations between audio and visual data, which have a different dimensionality and are nontrivially exchangeable. This article improves on the existing multimodal deep learning algorithms in two essential ways: 1) it presents a novel method for performing cross modality (before features are learned from individual modalities) and 2) extends the previously proposed cross connections that only transfer information between the streams that process compatible data. Illustrating some of the representations learned by the connections, we analyze their contribution to the increase in discrimination ability and reveal their compatibility with a lip-reading network intermediate representation. We provide the research community with Digits, a new data set consisting of three data types extracted from videos of people saying the digits 0-9. Results show that both cross-modal architectures outperform their baselines (by up to 11.5%) when evaluated on the AVletters, CUAVE, and Digits data sets, achieving the state-of-the-art results.

Description

Keywords

Feature extraction, Task analysis, Videos, Correlation, Visualization, Neural networks, Deep learning, Audiovisual, cross modality, deep learning, integration, machine learning, multimodal

Journal Title

IEEE Trans Neural Netw Learn Syst

Conference Name

Journal ISSN

2162-237X
2162-2388

Volume Title

Publisher

IEEE
Sponsorship
NERC (2221169)
European Commission Horizon 2020 (H2020) Industrial Leadership (IL) (634821)