Repository logo
 

A Pragmatic Guide to Geoparsing Evaluation

Accepted version
Peer-reviewed

Type

Article

Change log

Authors

Collier, nigel 
Pilehvar, mohammad 

Abstract

Empirical methods in geoparsing have thus far lacked a standard evaluation framework describing the task, metrics and data used to compare state-of-the-art systems. Evaluation is further made inconsistent, even unrepresentative of real-world usage by the lack of distinction between the different types of toponyms, which necessitates new guidelines, a consolidation of metrics and a detailed toponym taxonomy with implications for Named Entity Recognition (NER) and beyond. To address these deficiencies, our manuscript introduces a new framework in three parts. Part 1) Task Definition: clarified via corpus linguistic analysis proposing a fine-grained Pragmatic Taxonomy of Toponyms. Part 2) Metrics: discussed and reviewed for a rigorous evaluation including recommendations for NER/Geoparsing practitioners. Part 3) Evaluation Data: shared via a new dataset called GeoWebNews to provide test/train examples and enable immediate use of our contributions. In addition to fine-grained Geotagging and Toponym Resolution (Geocoding), this dataset is also suitable for prototyping and evaluating machine learning NLP models.

Description

Keywords

Journal Title

Language Resources and Evaluation

Conference Name

Journal ISSN

0010-4817

Volume Title

Publisher

Paradigm Press

Publisher DOI

Publisher URL

Rights

All rights reserved
Sponsorship
Natural Environment Research Council (1649558)
Engineering and Physical Sciences Research Council (EP/M005089/1)