Repository logo
 

The cross-linguistic performance of word segmentation models over time.

Accepted version
Peer-reviewed

Loading...
Thumbnail Image

Type

Article

Change log

Authors

Altmann-Richer, Emma 
Buttery, Paula 

Abstract

We select three word segmentation models with psycholinguistic foundations - transitional probabilities, the diphone-based segmenter, and PUDDLE - which track phoneme co-occurrence and positional frequencies in input strings, and in the case of PUDDLE build lexical and diphone inventories. The models are evaluated on caregiver utterances in 132 CHILDES corpora representing 28 languages and 11.9 m words. PUDDLE shows the best performance overall, albeit with wide cross-linguistic variation. We explore the reasons for this variation, fitting regression models to performance scores with linguistic properties which capture lexico-phonological characteristics of the input: word length, utterance length, diversity in the lexicon, the frequency of one-word utterances, the regularity of phoneme patterns at word boundaries, and the distribution of diphones in each language. These properties together explain four-tenths of the observed variation in segmentation performance, a strong outcome and a solid foundation for studying further variables which make the segmentation task difficult.

Description

Keywords

CHILDES, statistical learning, word segmentation, Humans, Language, Models, Theoretical, Psycholinguistics, Speech

Journal Title

J Child Lang

Conference Name

Journal ISSN

0305-0009
1469-7602

Volume Title

46

Publisher

Cambridge University Press (CUP)

Rights

All rights reserved
Sponsorship
Cambridge Assessment (unknown)