Show simple item record

dc.contributor.authorHardy,en
dc.contributor.authorNarayan, Sen
dc.contributor.authorVlachos, Andreasen
dc.date.accessioned2019-06-07T23:30:08Z
dc.date.available2019-06-07T23:30:08Z
dc.date.issued2020-01-01en
dc.identifier.isbn9781950737482en
dc.identifier.urihttps://www.repository.cam.ac.uk/handle/1810/293477
dc.description.abstractThere has been substantial progress in summarization research enabled by the availability of novel, often large-scale, datasets and recent advances on neural network-based approaches. However, manual evaluation of the system generated summaries is inconsistent due to the difficulty the task poses to human non-expert readers. To address this is- sue, we propose a novel approach for manual evaluation, HIGHlight-based Reference-less Evaluation of Summarization (HIGHRES), in which summaries are assessed by multiple an- notators against the source document via manually highlighted salient content in the latter. Thus summary assessment on the source document by human judges is facilitated, while the highlights can be used for evaluating multiple systems. To validate our approach we employ crowd-workers to augment with high- lights a recently proposed dataset and compare two state-of-the-art systems. We demonstrate that HIGHRES improves inter-annotator agreement in comparison to using the source document directly, while they help emphasize differences among systems that would be ignored under other evaluation approaches.
dc.publisherAssociation for Computational Linguistics
dc.rightsAttribution 4.0 International
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.titleHighres: Highlight-based reference-less evaluation of summarizationen
dc.typeConference Object
prism.endingPage3392
prism.publicationDate2020en
prism.publicationNameACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conferenceen
prism.startingPage3381
dc.identifier.doi10.17863/CAM.40621
dcterms.dateAccepted2019-05-14en
rioxxterms.versionVoR
rioxxterms.licenseref.urihttp://www.rioxx.net/licenses/all-rights-reserveden
rioxxterms.licenseref.startdate2020-01-01en
dc.contributor.orcidVlachos, Andreas [0000-0003-2123-5071]
rioxxterms.typeConference Paper/Proceeding/Abstracten
pubs.funder-project-idEPSRC (EP/R021643/2)
cam.orpheus.successThu Nov 05 11:54:03 GMT 2020 - The item has an open VoR version.*
rioxxterms.freetoread.startdate2100-01-01


Files in this item

Thumbnail
Thumbnail

This item appears in the following Collection(s)

Show simple item record

Attribution 4.0 International
Except where otherwise noted, this item's licence is described as Attribution 4.0 International