Editor’s note: The study summarized in this article was retracted in February 2019, at the request of the authors and the editors in chief of the journal that published it, Social Science & Medicine. In a statement accompanying the now-retracted paper, the journal wrote, “The reason for the retraction is that the authors discovered an error in the study, which, once corrected, rendered the association between structural stigma and mortality risk no longer statistically significant in the sample of 914 sexual minorities.”
ThinkProgress has also updated some of the language in this article to account for this development.
University of Texas at Austin professor Mark Regnerus, notorious for his failed efforts to prove that same-sex couples make inferior parents, is challenging a 2014 study that found negative health outcomes for gay, lesbian, and bi (LGB) people who live in communities with significant anti-gay prejudice.
Regnerus claims to have run his own analysis of the data using 10 different approaches, including a “more refined” strategy than was described in the original study, and could not replicate the results. Whereas the first study claimed anti-gay attitudes significantly hampered life expectancy and had several physical and mental healthy consequences, Regnerus found no such correlations.
Anti-LGBT conservatives were quick to champion his findings. Naomi Schaefer Riley of the Independent Women’s Forum suggested the original study was the latest example of the “crisis of replication,” studies whose results cannot be recreated in subsequent experiments. At National Review, Maggie Gallagher compared the situation to a 2014 study that claimed profound results for changing voters’ minds on marriage equality with canvassing but was found to have been largely fabricated. And Focus on the Family’s Glenn T. Stanton tried to drive home the idea that LGBT people are inherently broken and their mental health challenges cannot be blamed on stigma.
The conflicting studies
Mark Hatzenbuehler, associate professor of sociomedical science at Columbia University’s Mailman School of Public Health and lead researcher on the original study, told ThinkProgress that he was not convinced by Regnerus’ findings. “My colleagues and I stand by our results,” he said, adding that they were working on a “detailed response” to Regnerus’ article that they would submit for peer review.
The original study that Hatzenbuehler and his colleagues conducted didn’t involve collecting any new data, but using information available from massive yearly national studies. The General Social Survey provided data about prejudicial attitudes and the National Death Index provided information about mortality rates. These were then juxtaposed, and the researchers claimed to have found that LGB people living in high-prejudice communities had shorter life spans and experienced substantially elevated levels of suicide, homicide/violence, and cardiovascular diseases than those living in more welcoming communities.
Regnerus countered these results, saying he tried 10 different methods of imputation of that data and could not produce the same results. Imputation is a statistical process for filling in missing data with estimates that help flesh out the usable results, and there are countless ways to do it.
Hatzenbuehler explained to ThinkProgress that his team used an approach called “multiple imputation,” but he also noted, “We confirmed our analysis without imputing any missing data and our results were consistent.”
According to Hatzenbuehler, Regnerus should have replicated the results using the same approach. But he told ThinkProgress that’s not what happened. “Regnerus claims to have failed to replicate our results. We have had limited time to evaluate his paper, but an initial review of his code revealed several differences between his approach and ours that we suspect led to the discrepancies in the findings,” he claimed.
He added, “Setting aside the question of whether his study is therefore truly a replication, we welcome attempts to replicate our work.”
He noted the data are all publicly available, so anyone can attempt a replication at any time. Furthermore, his team’s response to Regnerus’ article will include all of the code for the original analyses that they ran, as well as for the re-analyses that they are currently conducting.
There is no trend of pro-gay junk science
Conservatives have tried to point to Hatzenbuehler’s study to conjure a trend of pro-gay junk science — attempting to compare it to a fraudulent canvassing study that was retracted last year.
Not only are there significant distinctions between the two studies, but the conclusions of the fraudulent study also appear to have been demonstrated by a valid one.
In Michael LaCour’s now-retracted 2014 study, he claimed that if canvassers went out door-to-door and had thorough personal conversations with voters about marriage equality, they could dramatically improve voters’ support for same-sex marriage. Unfortunately, it turns out he fabricated the data that informed his impressive results. Even though canvassers had actually gone out to conduct the experiment, LaCour seemed to have completely invented the survey results that assessed voters’ reactions, essentially making all the canvassing work a waste of time, money, and manpower — at least in terms of the study.
When researchers David Broockman (Stanford University) and Joshua Kalla (UC Berkeley) sought to replicate LaCour’s study, they found discrepancies in what he had published. They reached out to LaCour for the original raw data, but he could not produce them, and by pulling on that thread, they found that he had actually fabricated many aspects of the study.
But Broockman and Kalla were still interested in researching canvassing. So they started LaCour’s research over — this time studying public attitudes on transgender issues with a whole new army of canvassers. To compensate for LaCour’s missteps, they were incredibly transparent about their data collection.
When they published their results, it turned out LaCour probably would have been right if he’d actually completed his study authentically. Canvassing conversations actually can change attitudes on LGBT issues in a significant way, they claimed.
As Hatzenbuehler noted, Regnerus didn’t have to collect new data in the way that Broockman and Kalla did; he just had to reproduce the same analyses Hatzenbuehler and his colleagues had conducted for the original study. Hatzenbuehler claimed that Regnerus didn’t do that, and allegedly did not reach out to Hatzenbuehler for a clear understanding of what statistical processes his team used so that he could replicate them.
Broockman told ThinkProgress that these “replication controversies usually take time and effort to understand,” cautioning against a rush to judgment about the merits of either side. Still, he emphasized the importance of researchers publicly sharing the code they use in their studies, noting, “It’s pretty difficult to evaluate their claims without it.”
Referring to Regnerus’ attempt to undermine Hatzenbuehler’s research, Broockman added, “It is not like this is the only study indicating that experiencing prejudice has negative consequences.”
The future of LGBT research
Conservatives trying to resist LGBT equality have been waging war in academia as much as in politics. This new challenge to research about the impacts of stigma joins debates over issues like same-sex parenting, transgender children, and ex-gay therapy still being fought by a few holdouts who have not supported scientific consensuses.
On all of these issues, more information will provide more clarity, and Hatzenbuehler hopes that the impact of stigma is no exception. “One important aspect of that work is that it does not rely on any single study,” he said. “Separate research groups have used diverse methods to document the negative health effects of structural stigma among LGB individuals. Continuing to explore the link between structural stigma and health outcomes remains a vital topic for public health research.”