Through Thick and Thin: Comparing Traditional Qualitative Analysis and Natural Language Processing Techniques Using Narrative Data from Police Officers

Abstract

Traditional qualitative analysis can unearth nuanced insights into social problems through the systematic examination of textual or other non-numerical data. However, qualitative approaches are sometimes overly time-intensive or infeasible, especially when working with the large textual datasets that are increasingly available in today’s big data landscape. This study compares traditional qualitative analysis and natural language processing (NLP) techniques using independent teams that analyzed the same narrative survey responses from police officers at a large metropolitan agency (N = 715). We found that while both methods extracted broadly similar high-level themes, the traditional qualitative analysis identified additional high-level themes and elicited more depth and contextual richness from the data. The results of our comparison suggest that certain NLP tools may be beneficial when rapid high-level thematic extraction is the intended goal of a study. Nonetheless, these tools cannot currently replace human-driven qualitative analysis, which is essential for providing detail and interpretive richness.