Repository logo
 

Evaluation by association: A systematic study of quantitative word association evaluation

Accepted version
Peer-reviewed

Type

Conference Object

Change log

Authors

Vulić, I 
Kiela, D 
Korhonen, A 

Abstract

Recent work on evaluating representation learning architectures in NLP has established a need for evaluation protocols based on subconscious cognitive measures rather than manually tailored intrinsic similarity and relatedness tasks. In this work, we propose a novel evaluation framework that enables large-scale evaluation of such architectures in the free word association (WA) task, which is firmly grounded in cognitive theories of human semantic representation. This evaluation is facilitated by the existence of large manually constructed repositories of word association data. In this paper, we (1) present a detailed analysis of the new quantitative WA evaluation protocol, (2) suggest new evaluation metrics for the WA task inspired by its direct analogy with information retrieval problems, (3) evaluate various state-of-the-art representation models on this task, and (4) discuss the relationship between WA and prior evaluations of semantic representation with well-known similarity and relatedness evaluation sets. We have made the WA evaluation toolkit publicly available.

Description

Keywords

Evaluation of semantic representation, Word association, Word embeddings, Evaluation protocols

Journal Title

15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017 - Proceedings of Conference

Conference Name

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers

Journal ISSN

Volume Title

1

Publisher

Association for Computational Linguistics
Sponsorship
European Research Council (648909)