Repository logo
 

Evaluation of a Concept Mapping Task Using Named Entity Recognition and Normalization in Unstructured Clinical Text.

Published version
Peer-reviewed

Change log

Authors

Gildersleeve, Roger  ORCID logo  https://orcid.org/0000-0002-5374-8421
Kanter, Andrew S 
Chaudhry, Afzal 

Abstract

In this pilot study, we explore the feasibility and accuracy of using a query in a commercial natural language processing engine in a named entity recognition and normalization task to extract a wide spectrum of clinical concepts from free text clinical letters. Editorial guidance developed by two independent clinicians was used to annotate sixty anonymized clinic letters to create the gold standard. Concepts were categorized by semantic type, and labels were applied to indicate contextual attributes such as negation. The natural language processing (NLP) engine was Linguamatics I2E version 5.3.1, equipped with an algorithm for contextualizing words and phrases and an ontology of terms from Intelligent Medical Objects to which those tokens were mapped. Performance of the engine was assessed on a training set of the documents using precision, recall, and the F1 score, with subset analysis for semantic type, accurate negation, exact versus partial conceptual matching, and discontinuous text. The engine underwent tuning, and the final performance was determined for a test set. The test set showed an F1 score of 0.81 and 0.84 using strict and relaxed criteria respectively when appropriate negation was not required and 0.75 and 0.77 when it was. F1 scores were higher when concepts were derived from continuous text only. This pilot study showed that a commercially available NLP engine delivered good overall results for identifying a wide spectrum of structured clinical concepts. Such a system holds promise for extracting concepts from free text to populate problem lists or for data mining projects.

Description

Keywords

Annotation, Clinical letters, Gold standard, Named entity recognition, Natural language processing, Text mining

Journal Title

J Healthc Inform Res

Conference Name

Journal ISSN

2509-4971
2509-498X

Volume Title

4

Publisher

Springer Science and Business Media LLC
Sponsorship
Cambridge NIHR Biomedical Research Centre (RCAG/929 and RCAG/030)