Reconsidering a prominent finding on the spillover effects of police killings of unarmed Black Americans

Abstract

This analysis reproduces Bor, et al.’s (2018) seminal study of the effect of police killings of unarmed Black Americans on the mental health of Black communities. Using BRFSS data, a national repeated cross-sectional survey, Bor, et al. found these police killings had significant negative effects on Black Americans’ mental health at the state-month level. Re-analysis found that its unadjusted results, which were not reported in the study, were null. Second, the study’s reported significant population average exposure effect was a mechanical artifact of a weighting process that confounded exposure and outcome. Third, this significance was hypersensitive: it arose from the inclusion of 22 highly weighted respondents, comprising 0.02% of the sample (N=103,710). Fourth, the direction of this finding was also hypersensitive; removing 340 heavily-weighted observations (0.3% of the sample) reversed the outcome direction, producing an (insignificant) estimate that police killings of unarmed Black people improved Black Americans’ mental health. Fifth, correcting a week-day fixed effect that the original model specified but mistakenly calculated as a day-of-the-month fixed effect exacerbated these hypersensitivities (insignificance: n=10, 0.01%; direction: n= 315, 0.3%). Sixth, the study’s causal model prioritized a media market mechanism that created exposure spillover (i.e., Stable Unit Treatment Value Assumption) violations. The study’s null unadjusted results, its significance arising only from the confounding effects of weighting, the hypersensitivity of significance to a small number of extreme weights, and a causal model that violates necessary assumptions cast doubt on the robustness of the original findings.