Repository logo
 

RELPRON: A Relative Clause Evaluation Dataset for Compositional Distributional Semantics

Accepted version
Peer-reviewed

Type

Article

Change log

Authors

Rimell, Laura 
Maillard, Jean 
Polajnar, Tamara 
Clark, Stephen 

Abstract

This article introduces RELPRON, a large dataset of subject and object relative clauses, for the evaluation of methods in compositional distributional semantics. RELPRON targets an intermediate level of grammatical complexity between content-word pairs and full sentences. The task involves matching terms, such as ‘wisdom’, with representative properties, such as ‘quality that experience teaches’. A unique feature of RELPRON is that it is built from attested properties, but without the need for them to appear in relative clause format in the source corpus. The article also presents some initial experiments on RELPRON, using a variety of composition methods including simple baselines, arithmetic operators on vectors, and finally more complex methods in which argument-taking words are represented as tensors. The latter methods are based on the Categorial framework, which is described in detail. The results show that vector addition is difficult to beat — in line with the existing literature — but that an implementation of the Categorial framework based on the Practical Lexical Function model is able to match the performance of vector addition. The article finishes with an in-depth analysis of RELPRON, showing how results vary across subject and object relative clauses, across different head nouns, and how the methods perform on the subtasks necessary for capturing relative clause semantics, as well as providing a qualitative analysis highlighting some of the more common errors. Our hope is that the competitive results presented here, in which the best systems are on average ranking one out of every two properties correctly for a given term, will inspire new approaches to the RELPRON ranking task and other tasks based on linguistically interesting constructions.

Description

This is the author accepted manuscript. The final version is available from MIT Press via http://dx.doi.org/10.1162/COLI_a_00263

Keywords

Journal Title

Computational Linguistics

Conference Name

Journal ISSN

Volume Title

Publisher

MIT Press
Sponsorship
Laura Rimell and Stephen Clark were supported by EPSRC grant EP/I037512/1. Jean Maillard is supported by an EPSRC Doctoral Training Grant and a St John’s Scholarship. Laura Rimell, Tamara Polajnar and Stephen Clark are supported by ERC Starting Grant DisCoTex (306920).
Relationships
Is supplemented by: