A study appearing in the “online first” section of JAMA Pediatrics began making the media rounds earlier this week. Its authors analyzed population-based data that were collected over sixteen years by the Centers for Disease Control and Prevention, with a sample consisting of more than 750,000 American high-school students. The authors conclude that the arrival of state-level same-sex marriage policies was associated with a dip in the overall teen suicide rate of those states, especially among students identifying as sexual minorities.

If the authors are right, and these data really mean that 134,000 fewer adolescents are attempting suicide each year, as the authors extrapolate, that is indeed welcome news. However, I’m not yet convinced that that is the case.

The skeptic in me balks at the idea that a public policy shift on (adult) marriage access would have anything to do with how the average teenager grapples with feelings of depression and self-worth. Still, I’m willing to believe that it could have a positive impact on sexual minorities—at least for a short while—given the symbolic status of the 2015 Obergefell decision. Unlike attitude shifts in public opinion polls, such a legal victory is a seismic and tangible event—a mountaintop experience for their supporters. And yet for teenagers these remain largely symbolic victories, since we live in a country where marriage in general is retreating rapidly and where those who eventually marry do so at older ages than ever before.

Social Stigma, Suicide, and Social Science

Start your day with Public Discourse

Sign up and get our daily essays sent straight to your inbox.

So how do the authors think legal or legislative decisions concerning adults reduce suicide attempts in teenagers? While the study’s lead author, Julia Raifman, said she couldn’t know for sure—a view I hold as well—she went on to offer three possible explanations, each of which involved a pathway to reduced social stigma among or toward LGBT adolescents. Columbia University professor and stigma guru Mark Hatzenbuehler concurred with Raifman in his opinion piece appearing alongside the new study.

It’s possible that this phenomenon is real and that one of Raifman and Hatzenbuehler’s proposed explanations for it is correct. But consider this: the key finding in Hatzenbuehler’s popular study about the deleterious effects of stigma on the premature mortality of sexual minorities is one that cannot be replicated. I should know—I tried to do it. And that’s not the only snag Hatzenbuehler has hit of late. My research team recently informed him that his 2014 American Journal of Public Health study isn’t entirely replicable, either.

Social stigma is real; of that I have no doubt. But I suspect it’s being overused as a convenient explanation for the gaps—the differences—between the experiences of heterosexual Americans and those of sexual minorities. Indeed, in a recent study of Swedish married couples, those in same-sex marriages continue to display elevated risk of suicide, when contrasted with persons in opposite-sex marriages. It’s hard to be more tolerant than Sweden. Something more than stigma may be at work here.

Back to the study at hand. First, let’s get our bearings straight. Despite popular impressions to the contrary, youth suicide rates are the lowest among age groups in America, but differ profoundly between males and females. Boys ages 10 to 14 take their own lives at a rate under 3 per 100,000 persons per year, while the rate among those ages 15 to 24 is more than five times as high, at 17 (per 100,000). By contrast, the suicide rate for men over age 75—a group we hear little about and who seem to merit less public sympathy given their age—is a whopping 39. No rate among women ever approaches anything like that. In fact, the highest rate appears among those 45 to 64 years old (9.8 per 100,000), and is half that among women ages 15 to 24. It’s certainly accurate to say that suicide is on the rise, including among the youngest Americans. In the Swedish study just mentioned, the suicide rate in same-sex marriages is 33 per 100,000 persons, three times that (11) in opposite-sex unions. But these are actual suicides. Women far outpace men in attempted suicides, which is the subject matter of the JAMA study.

Unfortunately, four concerns prevent me from swallowing this study’s unusual conclusion:

1. Such a massive sample size can cause problems in understanding effects.

2. The authors inexplicably ignored several similar (available) outcomes.

3. All sexual minorities are lumped together unnecessarily.

4. Stigma dominates their interpretation, bypassing other competing explanations.

Too Big to Fail?

I realize it may sound strange to complain about massive sample sizes after I’ve spent so much time criticizing the tiny samples I’m used to seeing in studies of sexual minorities, but it’s true: big samples have the opposite problem of small ones. Too few cases and you’re apt to perceive “no differences” when they genuinely exist. Too many cases and you’re apt to see statistically significant differences everywhere, even if they are substantively miniscule. In other words, massive samples can make the insignificant significant. For example, the JAMA study authors report (in Table 2) a net decline in teen suicide attempts of 0.6 percentage points after same-sex marriage policies were implemented. That is, the detectable rate dropped from 8.6 percent to 8.0 percent.

The more cautious thing would have been to compare attempted suicides of sexual minority students not with “all students” but with all other students—i.e., those who didn’t self-identify as sexual minorities. There certainly is no sample-size concern impeding this approach. Had the authors done this, they might have revealed no overall effect of same-sex marriage legislation on the suicide risk of non-minority students. In other words, the overall effect—that 0.6 percentage point drop—may be driven by the larger drop noted in sexual minority suicide attempts. If that’s the case, the focus of the study should be on them, not all adolescents.

To their credit, the authors report the confidence intervals, revealing that the actual decline may be as little as one-tenth of one percentage point (from 8.6 to 8.5 percent) in suicide attempts among the population of adolescents, or as large as 1.2 percentage points. What concerns me more, however, is their use of the 95th percentile of confidence interval here, when such a large sample enables (and prudence counsels) one to employ a much higher threshold of confidence (say, 99.9 or 99.99 percentile). This is what is called the “p-value problem,” and more and more scientists are getting serious about it. Even the 4-percentage-point drop in sexual minority suicide—a more believable effect, in my opinion—only uses a p-value of 0.01 (that is, the 99th percentile confidence interval). In other words, I worry that the findings in this study are too fragile to employ a more confident (and appropriate) p-value hypothesis test. But I won’t know for sure until I replicate it.

Second, we are treated to only one outcome—attempted suicide—when at least three others are available in the CDC data: (1) thinking about committing suicide, (2) planning how to commit suicide, and (3) attempting suicide that resulted in an injury that had to be treated medically. This last one in particular is a glaring absence. Is it because the outcome they used was the only one that displayed a statistically significant effect of same-sex marriage legislation? I don’t know, but it is an answerable empirical question.

Third, I see no reason in a massive dataset to lump gay and lesbian teens (2 percent of the sample) together with bisexuals (6 percent), and those who said they were “not sure” about their sexuality (3 percent). Large samples such as this one offer analysts the opportunity to evaluate smaller groups by themselves. That the authors did not do this makes me wonder why they didn’t, especially given that bisexual teens outnumber gay and lesbian teens in these data by a ratio of 3-to-1. Could it be that the findings on suicide attempts are driven by bisexual teens, especially since their pattern of suicide ideation and planning rates outpaced the same among gay and lesbian teens in earlier iterations of this data? In the Relationships in America data, bisexual adults exhibit far greater suicide ideation than any other group (analyses not shown).

Fourth and finally, the authors’ models overlook numerous factors that are known to prompt both depression and suicide ideation in sexual minorities. This smacks of omitted variable bias, a common problem but one they could have mitigated because they have the measures to do so. Instead, they float explanations about unmeasured stigma, even while the 2015 data enable them to include a pair of bullying measures that clearly matter. Again, it’s possible that the effect of same-sex marriage shifts on attempted suicide was too fragile to accommodate paying attention to the far more proximate effect of bullying.

Another missing factor—the elephant in the room, perhaps—is the specter of sexual assault and abuse. Forced sex was experienced by 17 percent of youth who reported only same-sex sexual contact in the survey administrations from 2001 to 2009, a figure 63 percent higher than that cited by youth who reported only opposite-sex sexual contact. “Soul murder” was how the mother of a graduate school friend of mine put it after her son hanged himself, more than a decade after enduring the sexual abuse of a scoutmaster. It was my first up-close encounter with the human toll of sexual abuse. No amount of stigma suppression could ever salve that pain. Among bisexual teen girls who attempted suicide—as reported in the 2015 wave of data—37 percent said they had been sexually assaulted. To be sure, these are not the accounts of every troubled LGBT youth. But where it is, it is a significant wound, one ignored in the JAMA study.

It is not easy being a teenager in 2017 in America. And it’s certainly more complicated for those who identify as gay, lesbian, bisexual, or transgender, despite the fact that tolerant attitudes are more common in the general population than in decades past. Same-sex marriage signifies many things to its fans and its foes. And now it appears associated with a short-term emotional bump for sexual minority youth in America. But is it a robust, long-term panacea for the emotional struggles of teenagers? My concerns about the fragility of this new study, together with the evidence from tolerant Sweden, suggest the answer is “unlikely.”