Mischievous responders taint LGBQ health estimates in national survey

The new research was led by Joseph Cimpian, an associate professor of economics and education policy at New York University's Steinhardt School, and published in the American Journal of Public Health. The study found that "mischievous responders," research survey respondents who mislead researchers by providing extreme and untruthful responses to multiple items, have significantly affected overall estimates of health disparities between LGBQ and heterosexual youth.

"Not all adolescents take surveys as seriously as researchers would like," said Cimpian. "Some youth may find it funny to give exaggerated responses, reporting they are impossibly tall, have never been to the dentist, and eat carrots with extreme frequency. Problematically, these mischievous responders also find it funny to falsely claim they are 'LGBQ' and to exaggerate risky outcomes such as drug use, thereby inflating estimates of LGBQ youth drug use."

Cimpian utilized data from the Youth Risk Behavior Survey (YRBS), a large Centers for Disease Control and Prevention public health dataset on more than 140,000 high school students, to identify potentially mischievous responders and then examined how the inclusion of their responses may have an impact on estimates of health disparities in 20 different outcomes, from hopelessness to heroin use, between LGBQ and heterosexual youth.

Likely mischievous responders were identified using machine-learning techniques that seek out unusual patters of responses to items that should be theoretically unrelated to sexual identity (e.g., how often someone eats carrots, or their height) on self-administered questionnaires. After identifying the most likely mischievous youths, Cimpian and his research team were able to assess the data for changes in the estimates of the youth health disparities.

After removing the most likely mischievous respondents, Cimpian and his colleagues found that youth health disparities between LGBQ and heterosexual males dropped by an average of 46 percent, while dropping by 23 percent on average among females. Cimpian pointed out, however, that not all outcomes were affected equally.

"We found that drug and alcohol related disparities were most affected by removing potentially mischievous responders. With the suspect cases, researchers would conclude that LGBQ youth were drinking alcohol and using drugs like heroin and ecstasy at substantially higher rates than their heterosexual peers. But when the suspect cases were removed, those disparities diminished considerably and vanished in some instances," reported Cimpian.

"Yet, outcomes like being the victim of bullying and considering suicide were virtually unaffected," Cimpian continued. "This suggests that there are important health disparities observed in the literature, largely related to the victimization LGBQ youth experience, that are not caused by aberrant data, and that policymakers and health professionals need to pay careful attention to."

The researchers then repeated their analyses on a separate nationally representative dataset from the Centers for Disease Control and Prevention of over 14,000 students and replicated their initial findings, this time finding average reductions of 64 percent among males and 12 percent among females, further suggesting that mischievous responders affect male disparities more than female disparities. They also performed several additional statistical tests to ensure the patterns they observed were robust to research modifications. Overall, the research provides strong evidence that mischievous responders contribute to inflated estimates of the risk and deviance of LGBQ youth. The changes in disparity estimates are most pronounced for drug- and alcohol-use and also for suicide attempts, and also among males more than among females. Disparity estimates for being bullied, feeling sad or hopeless, and thinking about suicide were not noticeably affected.

"This research demonstrates why it is essential for researchers to perform a wide range of tests to ensure their survey data lead to robust findings," Cimpian added. "The research we put out can have a substantial effect on public policy and on what gets communicated to the public. It is critical that we take the necessary steps to make sure that what we present is accurate and to be forthcoming about the assumptions and limitations of research."