Highres: Highlight-based reference-less evaluation of summarization
Publication Date
2019Journal Title
ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference
Conference Name
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
ISBN
9781950737482
Publisher
Association for Computational Linguistics
Pages
3381-3392
Type
Conference Object
This Version
VoR
Metadata
Show full item recordCitation
Hardy, Narayan, S., & Vlachos, A. (2019). Highres: Highlight-based reference-less evaluation of summarization. ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, 3381-3392. https://doi.org/10.18653/v1/p19-1330
Abstract
There has been substantial progress in summarization research enabled by the availability of novel, often large-scale, datasets and recent advances on neural network-based approaches. However, manual evaluation of the system generated summaries is inconsistent due to the difficulty the task poses to human non-expert readers. To address this is- sue, we propose a novel approach for manual evaluation, HIGHlight-based Reference-less Evaluation of Summarization (HIGHRES), in which summaries are assessed by multiple an- notators against the source document via manually highlighted salient content in the latter. Thus summary assessment on the source document by human judges is facilitated, while the highlights can be used for evaluating multiple systems. To validate our approach we employ crowd-workers to augment with high- lights a recently proposed dataset and compare two state-of-the-art systems. We demonstrate that HIGHRES improves inter-annotator agreement in comparison to using the source document directly, while they help emphasize differences among systems that would be ignored under other evaluation approaches.
Sponsorship
Engineering and Physical Sciences Research Council (EP/R021643/2)
Embargo Lift Date
2100-01-01
Identifiers
External DOI: https://doi.org/10.18653/v1/p19-1330
This record's URL: https://www.repository.cam.ac.uk/handle/1810/293477
Statistics
Total file downloads (since January 2020). For more information on metrics see the
IRUS guide.