A pragmatic guide to geoparsing evaluation

Change log
Pilehvar, Mohammad Taher 
Collier, Nigel 

Abstract: Empirical methods in geoparsing have thus far lacked a standard evaluation framework describing the task, metrics and data used to compare state-of-the-art systems. Evaluation is further made inconsistent, even unrepresentative of real world usage by the lack of distinction between the different types of toponyms, which necessitates new guidelines, a consolidation of metrics and a detailed toponym taxonomy with implications for Named Entity Recognition (NER) and beyond. To address these deficiencies, our manuscript introduces a new framework in three parts. (Part 1) Task Definition: clarified via corpus linguistic analysis proposing a fine-grained Pragmatic Taxonomy of Toponyms. (Part 2) Metrics: discussed and reviewed for a rigorous evaluation including recommendations for NER/Geoparsing practitioners. (Part 3) Evaluation data: shared via a new dataset called GeoWebNews to provide test/train examples and enable immediate use of our contributions. In addition to fine-grained Geotagging and Toponym Resolution (Geocoding), this dataset is also suitable for prototyping and evaluating machine learning NLP models.

Original Paper, Geoparsing, Toponym resolution, Geotagging, Geocoding, Named Entity Recognition, Machine learning, Evaluation framework, Geonames, Toponyms, Natural language understanding, Pragmatics
Journal Title
Language Resources and Evaluation
Conference Name
Journal ISSN
Volume Title
Springer Netherlands
Natural Environment Research Council (NE/M009009/1)
Medical Research Council (MR/M025160/1)
Engineering and Physical Sciences Research Council (EP/M005089/1)