Repository logo
 

Two is better than one: using a single emotion lexicon can lead to unreliable conclusions

Published version
Peer-reviewed

Change log

Authors

Stillwell, D 

Abstract

Emotion lexicons became a popular method for quantifying affect in large amounts of textual data (e.g., social media posts). There are multiple independently developed emotion lexicons which tend to correlate positively with one another but not entirely. Such differences between lexicons may not matter if they are just unsystematic noise, but if there are systematic differences this could affect conclusions of a study. The goal of this paper is to examine whether two extensively used, apparently domain-independent lexicons for emotion analysis would give the same answer to a theory-driven research question. Specifically, we use the Linguistic Inquiry and Word Count (LIWC) and NRC Word-Emotion Association Lexicon (NRC). As an example, we investigate whether older people have more positive expression through their language use. We examined nearly 5 million tweets created by 3,573 people between 18 to 78 years old and found that both methods show an increase in positive affect until age 50. After that age, however, according to LIWC, positive affect drops sharply, whereas according to NRC, the growth of positive affect increases steadily until age 65 and then levels off. Thus, using one or the other method would lead researchers to drastically different theoretical conclusions regarding affect in older age. We unpack why the two methods give inconsistent conclusions and show this was mostly due to a particular class of words: those related to politics. We conclude that using a single lexicon might lead to unreliable conclusions, so we suggest that researchers should routinely use at least two lexicons. If both lexicons come to the same conclusion then the research evidence is reliable, but if not then researchers should further examine the lexicons to find out what difference might be causing inconclusive result.

Description

Acknowledgements: We would like to thank Sylwia Adamus, Rachela Antosz-Rekucka, Monika Biały, Jakub Cacek, Zuzanna Garncarek, Katarzyna Hryniewiecka, and Ewa Ilczuk from Jagiellonian University who contributed to data collection.

Keywords

Research Article, Social sciences, Biology and life sciences, Computer and information sciences, People and places

Journal Title

PLoS One

Conference Name

Journal ISSN

1932-6203
1932-6203

Volume Title

Publisher

Public Library of Science (PLoS)
Sponsorship
the Ministry of Science and Higher Education in Poland (MOBILITY PLUS (1614/MOB/V/2017/0))