Imagine if evangelical sociologists set out to document how the children of evangelical Christian parents fare in life. Imagine that they begin their effort by recruiting parents of children who attend Sunday School classes at places like Wheaton Bible Church outside Chicago and Saddleback Community Church in Orange County, California—both located in prosperous communities with above-average social capital and support for families, children, and faith. They choose this approach because churchgoing, self-identified evangelicals with children under age eighteen comprise less than 3 percent of the population of American adults (this is true), and the researchers figure it will be easier to recruit participants than to evaluate those who might show up randomly in a population-based sample. They know a random sample is best, but they cite “cost constraints” and “difficult research constraints” in justifying their decision to use a convenience sample.

Then the scholars survey the parents, asking them questions about how their kids are faring. They compile the results and call it the American Christian Family Study. The study includes a comparison sample of other parents and children, pulled from a fine population-based survey so as to display what average children from average families look like. The evangelical kids compare well; they do better, actually, than the children from average families across the country. Their parents are more likely to report being married, educated, stable, and employed. The parents tell the researchers that the kids are faring well, too—they don’t have many emotional challenges, are doing well in school, and are generally getting along well in life. The study’s initial findings are published in a peer-reviewed social science journal, and they help to improve public perception of evangelical parents.

Would the social scientific community consider this study a solid one, employing high-quality sample selection methods and useful both for understanding the experience of Christian households in America and for comparing this group of children with other children? To put it mildly, it’s unlikely. And I would agree with them.

The Australian Study of Child Health in Same-Sex Families

Start your day with Public Discourse

Sign up and get our daily essays sent straight to your inbox.

You will not, however, witness very many scholarly misgivings about a new published study analyzing data from the Australian Study of Child Health in Same-Sex Families (ACHESS), even though I’ve just offered a close analogy of its sampling and comparative strategy. I do not bear ill will toward the research team; data collection is no simple task. I won’t impugn the motivations of the author and his collaborators. Those vary widely, and everyone has his own. I don’t care about the source of the funding. But the study deserves some critical commentary.

The authors declare that the “study aims to describe the physical, mental and social wellbeing of Australian children with same-sex attracted parents, and the impact that stigma has on them.” They conclude that “children with same-sex attracted parents score higher than population samples on a number of parent-reported measures of child health.” The study has generated headlines such as this one from the Washington Post: “Children of same-sex couples happier and healthier than peers, research shows.”

But we cannot learn this from the ACHESS study, because of these two sentences in the study’s methodology section:

The convenience sample was recruited using online and traditional recruitment techniques, accessing same-sex attracted parents through news media, community events and community groups. Three hundred and ninety eligible parents contacted the researchers…

The ACHESS’s interim report, issued just under two years ago, foreshadowed the positive conclusions of the recently-published article—in the same journal, no less—and had more to say about its sampling approach:

Initial recruitment will . . . include advertisements and media releases in gay and lesbian press, flyers at gay and lesbian social and support groups, and investigator attendance at gay and lesbian community events . . . Primarily recruitment will be through emails posted on gay and lesbian community email lists aimed at same-sex parenting. This will include, but not be limited to, Gay Dads Australia and the Rainbow Families Council of Victoria.

I don’t know if there’s any other way to say this than to suggest that—like my opening scenario—this is not the way to build a sense of average same-sex households with children. To compare the results from such an unusual sample with that of a population-based sample of everyone else is just suspect science. And I may be putting that too mildly.

Non-Random Samples and Social Desirability Bias

It’s not the first time this approach has met with considerable publication and media success. The ACHESS study is a lot like the National Longitudinal Lesbian Family Study (NLLFS), except that it’s larger and newer. I realize that 500 cases is not a number to scoff at, and that such populations are a small minority to begin with. But until social scientists decide to do the difficult, expensive work of locating same-sex attracted parents (however defined) through random, population-based sampling strategies—preferably ones that do not “give away” the primary research question(s) up front, as ACHESS did—we simply cannot know whether claims like “no differences” or “happier and healthier than” are true, valid, and on target. Why? Because this non-random sample reflects those who actively pursued participating in the study, personal and political motivations included. In such a charged environment, the public—including judges and media—would do well to demand better-quality research designs, not just results they approve of.

Snowball sampling doesn’t cut it. When I want to know who’s most apt to win the next election, I don’t ask my friends whom they support. Nor do I field a survey asking interested people to participate. No, I want a random sample of the sort often conducted by Gallup, NORC, or Knowledge Networks.

Another reason for healthy skepticism is that the ACHESS participants—parents reporting about their children’s lives—are all well aware of the political import of the study topic, and an unknown number of them certainly signed up for that very reason. As a result, it seems unwise to trust their self-reports, given the high risk of “social desirability bias,” or the tendency to portray oneself (or here, one’s children) as better than they actually are. Again, it is impossible to know exactly how much of a problem such bias presents in this situation. But I think the temptation to report positive assessments could be elevated in this self-selected sample and on this sensitive topic. (In the end, the differences between the ACHESS parent reports and the population-based comparisons were more modest—about 3 to 6 percent—than I’d expect.)

Skepticism about the ACHESS sample is all the more reason to do a random study that doesn’t advertise its intentions beforehand. That’s exactly why the survey I oversaw, the New Family Structures Study (NFSS), elected to talk to the children after they had grown up, to skip the parents entirely to ensure a more independent assessment, not to broadcast our key research questions in the title or initial screener questionnaire, and to locate participants randomly in a large population-based sample. If you’ve been paying attention, however, you’ll know that my NFSS studies—which mapped 248 respondents who told us their mother or father had been in a same-sex relationship—came to rather different conclusions than the ACHESS study has.

New Reproductive Technologies

To be sure, the ACHESS study includes many children born in comparatively new ways—80 percent of those with a female parent(s) were born via home insemination or by assisted reproductive technology (ART), and 82 percent of those with a male parent(s) were born via surrogacy. The NFSS mapped an earlier generation in which ART and surrogacy were uncommon. But just how common ART and surrogacy are today in the average same-sex household remains unknown in most Western countries, including the US and Australia.

Indeed, most children born via ART and surrogacy are, from the start, set apart from the 99 percent of children who are not—even if the data were collected randomly—by the comparative expense involved in their conception and acquisition. This is not consonant with the average couple’s experience, whether that couple is an opposite-sex or same-sex pair. In other words, there were few unplanned pregnancies among the ACHESS parents. In this study, as in much of the same-sex marriage movement, the public is treated only to the lives and experiences of the LGBT elite. Those with more modest means are missing in action.

What Do We Really Know about Same-Sex Parenting?

It may appear to readers that most, if not all, studies in this domain are hopelessly flawed. It’s not true. Scholars can and do agree on a variety of conclusions when it comes to same-sex households and child outcomes. So what do we know with confidence?

– Same-sex parenting is rare. Less than 2 percent of Americans fit the description. Among same-sex households that want children, the population most apt to use assisted reproduction—white, educated women—is actually the demographic group least interested in having children.

– Children fare better in an environment of household stability. In the NFSS, stability was largely absent when an adult child reported a parental same-sex relationship. Hence, their life experiences were (on average) notably more challenging than those of their peers with married mothers and fathers. Some critics felt this was an “unfair” comparison. But if social reality is unfair, there’s not much that any sociologist can do about that.

It warms my heart that a byproduct of the war over the meaning of the NFSS data has been a unified admission that divorce and other household disruptions—like new partners—takes a toll on kids well into their adult years. That was not always a consensus among observers of marriage and family.

But will same-sex parents’ relationships be more or less stable in the future? On the one hand, we know that same-sex relationships in general—across multiple datasets—remain more fragile than opposite-sex ones (and to be fair, no group is performing all that well). We can argue about why this is so, but it is. Nevertheless, it’s too early to tell if this remains the case with same-sex marriages, given their comparatively small number and the pent-up demand (reflecting greater longevity) characteristic of the earliest marriages.

On the other hand, the “planned” nature of new forms of same-sex parenting (e.g., ART) no doubt reflects more deliberation (and more money) than most unplanned pregnancies, even though all such ART and surrogacy births reflect diminished kinship. That is, somebody’s not a biological parent of the child. In parenting studies, wealth and planning are beneficial resources, while diminished kinship is a risk.

Whether such “planned” parenthood is the new normal—the average—in same-sex relationships is unknown. I have my doubts. But there’s no doubt that this is the face of same-sex parenting: the well-adjusted, ART-generated child of a 30-something, upper-middle-class lesbian mother and her partner. It’s what scholars, judges, and the media demand as a comparison category today. The ACHESS certainly delivered on that. And yet the reality of same-sex relationships in parents’ lives—the average experience—has been something quite different, as the NFSS revealed.

This was brought home to me during a recent conversation with a University of Colorado professor. He shared with me that he and his family live next door to a woman—a mother—who’s been through three same-sex relationships in the past several years that they’ve been neighbors. The professor relayed that on numerous occasions he’d find himself in his back yard playing with his kids, only to notice the neighbor’s son peering over the fence, watching. The boy’s mother, aware of this, confessed to the professor, “I’m doing the best I can.”

If the NFSS’s accounts of household instability do not portend the future of gay parenting in America—and they may not—then you can expect stories like this to become rarer in reality as same-sex parenting arrangements indeed outperform their peers. But I know better than to expect social realities to change as rapidly as have attitudes about same-sex marriage.

I sometimes wonder why I even bother voicing such concerns. In my short legal career, I witnessed US District Court Judge Bernard Friedman dismiss my NFSS-based study and the analyses of US and Canadian census data by my colleagues, preferring to appear hip at his dinner parties by throwing our evidence under the bus rather than responding to it. In a nation that seems to be rapidly devolving into one big junior high school, evidence no longer appears to matter. Only allegiances do.