Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction
View / Open Files
Publication Date
2018-12Journal Title
Transactions of the Association for Computational Linguistics
ISSN
2307-387X
Publisher
MIT Press - Journals
Volume
6
Pages
451-465
Language
en
Type
Article
Metadata
Show full item recordCitation
Gerz, D., Vulić, I., Ponti, E., Naradowsky, J., Reichart, R., & Korhonen, A. (2018). Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction. Transactions of the Association for Computational Linguistics, 6 451-465. https://doi.org/10.1162/tacl_a_00032
Abstract
<jats:p> Neural architectures are prominent in the construction of language models (LMs). However, word-level prediction is typically agnostic of subword-level information (characters and character sequences) and operates over a closed vocabulary, consisting of a limited word set. Indeed, while subword-aware models boost performance across a variety of NLP tasks, previous work did not evaluate the ability of these models to assist next-word prediction in language modeling tasks. Such subword-level informed models should be particularly effective for morphologically-rich languages (MRLs) that exhibit high type-to-token ratios. In this work, we present a large-scale LM study on 50 typologically diverse languages covering a wide variety of morphological systems, and offer new LM benchmarks to the community, while considering subword-level information. The main technical contribution of our work is a novel method for injecting subword-level information into semantic word vectors, integrated into the neural language modeling training, to facilitate word-level prediction. We conduct experiments in the LM setting where the number of infrequent words is large, and demonstrate strong perplexity gains across our 50 languages, especially for morphologically-rich languages. Our code and data sets are publicly available. </jats:p>
Sponsorship
This work is supported by the ERC Consolidator Grant LEXICAL (648909)
Funder references
European Research Council (648909)
Identifiers
External DOI: https://doi.org/10.1162/tacl_a_00032
This record's URL: https://www.repository.cam.ac.uk/handle/1810/279936
Rights
Attribution 4.0 International (CC BY 4.0)
Licence URL: https://creativecommons.org/licenses/by/4.0/
Statistics
Total file downloads (since January 2020). For more information on metrics see the
IRUS guide.