Scaling Up Sign Spotting Through Sign Language Dictionaries
Publication Date
2022Journal Title
International Journal of Computer Vision
ISSN
0920-5691
Publisher
Springer Science and Business Media LLC
Volume
130
Issue
6
Pages
1416-1439
Language
en
Type
Article
This Version
VoR
Metadata
Show full item recordCitation
Varol, G., Momeni, L., Albanie, S., Afouras, T., & Zisserman, A. (2022). Scaling Up Sign Spotting Through Sign Language Dictionaries. International Journal of Computer Vision, 130 (6), 1416-1439. https://doi.org/10.1007/s11263-022-01589-6
Abstract
<jats:title>Abstract</jats:title><jats:p>The focus of this work is<jats:italic>sign spotting</jats:italic>–given a video of an isolated sign, our task is to identify<jats:italic>whether</jats:italic>and<jats:italic>where</jats:italic>it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1)<jats:italic>watching</jats:italic>existing footage which is sparsely labelled using mouthing cues; (2)<jats:italic>reading</jats:italic>associated subtitles (readily available translations of the signed content) which provide additional<jats:italic>weak-supervision</jats:italic>; (3)<jats:italic>looking up</jats:italic>words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs,<jats:sc>BslDict</jats:sc>, to facilitate study of this task. The dataset, models and code are available at our project page.</jats:p>
Keywords
Sign language recognition, Sign spotting, Few-shot learning
Identifiers
s11263-022-01589-6, 1589
External DOI: https://doi.org/10.1007/s11263-022-01589-6
This record's URL: https://www.repository.cam.ac.uk/handle/1810/337554
Rights
Licence:
http://creativecommons.org/licenses/by/4.0/
Statistics
Total file downloads (since January 2020). For more information on metrics see the
IRUS guide.
Recommended or similar items
The current recommendation prototype on the Apollo Repository will be turned off on 03 February 2023. Although the pilot has been fruitful for both parties, the service provider IKVA is focusing on horizon scanning products and so the recommender service can no longer be supported. We recognise the importance of recommender services in supporting research discovery and are evaluating offerings from other service providers. If you would like to offer feedback on this decision please contact us on: support@repository.cam.ac.uk