Scaling Up Sign Spotting Through Sign Language Dictionaries
Publication Date
2022-06Journal Title
International Journal of Computer Vision
ISSN
0920-5691
Publisher
Springer Science and Business Media LLC
Volume
130
Issue
6
Pages
1416-1439
Language
en
Type
Article
This Version
VoR
Metadata
Show full item recordCitation
Varol, G., Momeni, L., Albanie, S., Afouras, T., & Zisserman, A. (2022). Scaling Up Sign Spotting Through Sign Language Dictionaries. International Journal of Computer Vision, 130 (6), 1416-1439. https://doi.org/10.1007/s11263-022-01589-6
Abstract
<jats:title>Abstract</jats:title><jats:p>The focus of this work is <jats:italic>sign spotting</jats:italic>–given a video of an isolated sign, our task is to identify <jats:italic>whether</jats:italic> and <jats:italic>where</jats:italic> it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) <jats:italic>watching</jats:italic> existing footage which is sparsely labelled using mouthing cues; (2) <jats:italic>reading</jats:italic> associated subtitles (readily available translations of the signed content) which provide additional <jats:italic>weak-supervision</jats:italic>; (3) <jats:italic>looking up</jats:italic> words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, <jats:sc>BslDict</jats:sc>, to facilitate study of this task. The dataset, models and code are available at our project page.
</jats:p>
Keywords
Article, Special issue on Advances in Computer Vision and Applications, Sign language recognition, Sign spotting, Few-shot learning
Identifiers
s11263-022-01589-6, 1589
External DOI: https://doi.org/10.1007/s11263-022-01589-6
This record's URL: https://www.repository.cam.ac.uk/handle/1810/337554
Rights
Licence:
http://creativecommons.org/licenses/by/4.0/
Statistics
Total file downloads (since January 2020). For more information on metrics see the
IRUS guide.