RELPRON: A Relative Clause Evaluation Data Set for Compositional Distributional Semantics
View / Open Files
Publication Date
2016-12-01Journal Title
Computational Linguistics
ISSN
0891-2017
Publisher
MIT Press
Volume
42
Issue
4
Pages
661-701
Language
English
Type
Article
This Version
VoR
Metadata
Show full item recordCitation
Rimell, L., Maillard, J., Polajnar, T., & Clark, S. (2016). RELPRON: A Relative Clause Evaluation Data Set for Compositional Distributional Semantics. Computational Linguistics, 42 (4), 661-701. https://doi.org/10.1162/COLI_a_00263
Abstract
This article introduces RELPRON, a large data set of subject and object relative clauses, for the evaluation of methods in compositional distributional semantics. RELPRON targets an intermediate level of grammatical complexity between content-word pairs and full sentences. The task involves matching terms, such as “wisdom,” with representative properties, such as “quality that experience teaches.” A unique feature of RELPRON is that it is built from attested properties, but without the need for them to appear in relative clause format in the source corpus. The article also presents some initial experiments on RELPRON, using a variety of composition methods including simple baselines, arithmetic operators on vectors, and finally, more complex methods in which argument-taking words are represented as tensors. The latter methods are based on the Categorial framework, which is described in detail. The results show that vector addition is difficult to beat—in line with the existing literature—but that an implementation of the Categorial framework based on the Practical Lexical Function model is able to match the performance of vector addition. The article finishes with an in-depth analysis of RELPRON, showing how results vary across subject and object relative clauses, across different head nouns, and how the methods perform on the subtasks necessary for capturing relative clause semantics, as well as providing a qualitative analysis highlighting some of the more common errors. Our hope is that the competitive results presented here, in which the best systems are on average ranking one out of every two properties correctly for a given term, will inspire new approaches to the RELPRON ranking task and other tasks based on linguistically interesting constructions.
Relationships
Is supplemented by: https://doi.org/10.17863/CAM.298
Sponsorship
Laura Rimell and Stephen Clark were supported by EPSRC grant EP/I037512/1. Jean Maillard is supported by an EPSRC Doctoral Training Grant and a St. John’s Scholarship. Laura Rimell, Tamara Polajnar, and Stephen Clark are supported by ERC Starting Grant DisCoTex (306920).
Funder references
EPSRC (1344767)
Identifiers
External DOI: https://doi.org/10.1162/COLI_a_00263
This record's URL: https://www.repository.cam.ac.uk/handle/1810/263723
Rights
Licence:
http://www.rioxx.net/licenses/all-rights-reserved