An important new collection of peer-reviewed scholarly papers entitled No Differences? How Children in Same-Sex Households Fare has just been released by the Witherspoon Institute. The papers included and summarized in the book all study the nexus between children’s well-being and the structure of the families in which they are raised. In particular, the authors focus on the efficacy of families in which the adults are involved in a physically intimate same-sex relationship.

This is an increasingly important topic, as many countries have extended their definition of marriage to encompass same-sex couples, while others are considering doing so. Because of their policy relevance, the papers in this volume will be read by many for their importance to the approval of particular referenda, or to the outcome of a specific Supreme Court test case. That is unfortunate.

A reading of the sometimes apoplectic reactions in the press, and even among some academics, to the papers by professors Marks and Regnerus reminds one of Samuel Johnson’s aphorism that “prejudice, not being founded on reason, cannot be removed by argument.” It is clear that for some people, scientific research on this subject is irrelevant. The new volume is meant for those who still approach the topic of parenting and sexuality with open minds.

The Problem with Convenience Samples

Start your day with Public Discourse

Sign up and get our daily essays sent straight to your inbox.

The first paper included in the volume, by Loren Marks, examines the foundations of the position taken by the American Psychological Association (APA) on what it calls “lesbian and gay parenting.” The 2005 APA monograph setting forth that organization’s position asserts that the question of whether the childrearing efficacy of parents in same-sex relationships is at least the equal of that of heterosexual couples is settled, and that the serious academic literature speaks with a single voice on the matter.

Marks reviews an extensive literature on the topic and finds that most of the studies on the subject rely on “convenience samples”: groups of respondents that cannot be considered cross-sections of the population at large. Convenience samples are a staple of the literature because same-sex parenting is rare, and so recruiting same-sex parents for a study generally involves placing ads at day-care centers and in publications aimed at the LGBT population, or contacting people by way of their network of friends. While they can provide a useful window on the experience of parents in same-sex relationships, Marks notes that convenience samples suffer from two generic problems. First, the sample sizes are very small; one of the better studies might include a dozen or two lesbian families and a comparable number of heterosexual families. In such a small sample, only enormous differences in children’s outcomes will rise to the level of statistical significance. Technically speaking, estimates of the difference between outcomes for same-sex parents and those for heterosexual couples suffer from low “power.” Moreover, because convenience samples do not constitute a random cross-section of the population, they are not representative, and so estimates based on them suffer from a problem known to statisticians as “bias.”

Marks also notes that many of the small studies either fail to identify a comparison group of heterosexual parents, or they compare educated and affluent lesbian couples to single heterosexual parents. He suggests that better comparison groups might consist of married heterosexual parents or of all heterosexual parents. Certainly that would be the case if one wanted to maintain that there was no difference between the status quo outcomes for children of parents in same-sex relationships and those of heterosexual married parents, as some have seemed to want to do.

Marks highlights three studies that avoid small convenience samples and work with much larger random samples, two of which can be found in the new volume, in the chapter by Mark Regnerus and the chapter by Douglas Allen, Catherine Pakaluk, and Joseph Price.

The Regnerus Study

Mark Regnerus’s much-bruited study takes an innovative approach to solving the difficulties of generating a large sample of parents in same-sex relationships. Working with the internet-polling firm Knowledge Networks, now acquired by GfK, Regnerus contacted over 15,000 young and early-middle-aged adults and asked them about their childhoods, including whether at least one of their parents had been involved in a same-sex relationship while they were growing up.

Regnerus notes that this retrospective definition of having had a parent in a same-sex relationship overlaps with, but is not identical with, having had a parent who self-identified as gay or lesbian. Some critics have objected to his use of “gay and lesbian parents” as shorthand for “the parents of the adults in his sample who had been involved in a same-sex romantic relationship,” but he is very clear about his definition.

Even after having cast such a wide net, Regnerus finds but a few hundred adults who had grown up in households with a parent in a same-sex relationship (for 163 cases, it was their mother; in 73 cases, it was their father). Professor Regnerus finds that there are numerous dimensions on which adults whose parents had had same-sex relationships encountered worse life outcomes than those raised by both biological parents who were married to each other. The outcomes for the children of parents who had had same-sex relationships more closely resembled those for children of single-parent heterosexual households than the outcomes for those who grew up under the protection of an intact heterosexual marriage. Regnerus suggests that much of the reason for this is the instability of the partnerships of the parents who had had same-sex relationships.

On the one hand, this does not tell us much about what happens in lesbian households in which both adults have advanced degrees, earn high incomes, and manage to stay together for decades. On the other hand, it may reflect reality for the majority of children who grow up with parents who have had same-sex relationships.

Going Straight to the Source: The Advantages and Limitations of Census Data

Of course, even with the reach of a professional internet-polling firm and a survey of over 15,000 individuals, sampling bias can intrude itself, as people cannot be compelled to respond. Moreover, the relative rarity of parents in same-sex relationships means that even a sample size in the tens of thousands of individuals will yield a relatively small set of parents in same-sex relationships, attenuating statistical power. The only way you could avoid these problems would be to get the Census Bureau to conduct your survey as part of the decennial census, to which individuals are required by law to respond.

This is exactly what the other two studies mentioned in Loren Marks’s chapter do. Michael Rosenfeld accessed the 5-percent random sample from the 2000 decennial census, and he compared the outcomes for children of same-sex couples with those of married heterosexual couples. His performance variable was grade retention—whether a child was reported to be at least a grade behind. The cost of using the census data comes in the form of being able to access relatively few variables as compared with the Regnerus survey.

In their chapter, Douglas Allen, Catherine Pakaluk, and Joseph Price look at the same dataset considered by Michael Rosenfeld, who failed to reject the null hypothesis that the children in the sample with same-sex parents (there were 8,632 in the 5-percent sample) are more likely to be retained (i.e., held back a grade) than are the sampled children who were being raised by married heterosexual parents, of whom there were 1,189,893 in the 5-percent sample. First, the authors show that Rosenfeld’s data-analysis strategy had low power. In particular, while one cannot reject that the children of same-sex parents are not statistically significantly more likely to be retained than are the children of married heterosexual parents, neither do they differ in a statistically significant way from the sampled children of never-married women (there were 77,879 in the entire sample). Indeed, the parameter estimates for the children with same-sex parents are very similar to those of the children of the never-married women, which in turn are statistically significantly worse than the outcomes for the children with married heterosexual parents.

Does Family Instability Cause Children of Same-Sex Couples to Fare Worse?

Allen, Pakaluk, and Price then go on to show that Rosenfeld’s finding of no difference hinges on two critical decisions: first, he eliminated geographically mobile children who had changed domicile during the preceding five years, and second, he excluded children who had been adopted. Including either group, even with a control variable for the group mean, expands the sample size and increases the power of the tests.

In either case, the result is that the sharper estimates reveal that the children of same-sex parents more closely resemble the children of single heterosexual parents than they do the children of married parents. In a companion piece, which also appears in the new volume, the same authors further hone their analysis by considering only the children (in fourth and eighth grade) whose grade level is expressly asked about in the census. Again, the sharper data lead to a more statistically significant difference between the children of parents in same-sex relationships and those of heterosexual married parents.

Rosenfeld argues strenuously for the need to exclude the data for mobile and adopted children, on the grounds that mobility is a proxy for family instability, while he contends that same-sex couples are more likely to adopt troubled children. Yet even with controls for those variables added to the model, when the data are included the same-sex parenting variable again becomes significant. The picture that emerges from this exercise is that familial instability may be an important mechanism by which the children of parents in same-sex relationships fare worse than those with married heterosexual parents.

Differences between Gay and Lesbian-Headed Households

In yet another chapter in the new volume, Douglas Allen continues the quest for a large probability sample by working with a 20-percent sample from the 2006 Canadian Census. By 2006, gay and lesbian marriage was legal throughout Canada, and the census question only identified a child as having gay or lesbian parents if she (or he) responded affirmatively to the question, “Are you the child of a (male/female) same-sex married or common-law couple?” This survey provides a glimpse of what society might look like immediately after the legalization of same-sex marriage.

For children of both gay and of lesbian parents, Allen finds significant deficits in high-school completion. Moreover, while he generally finds that late-adolescent children of lesbian parents have high-school completion rates that resemble those of the children of single women, and overall the late-adolescent children of gay parents resemble children raised by single men, there is an additional feature to the data—the daughters of gay male parents fare worse than the sons, whereas no similar gender disparity emerges in any of the other family structures he studies. Allen ends by calling for the research community to distinguish between gay and lesbian couples raising children, rather than aggregating them into the same category.

Finally, there are two chapters by Walter Schumm. In the first, he notes that while a different researcher might have made different research-design choices than did Mark Regnerus, the choices Regnerus opted for are all within the range of standard practice for the sort of research he is doing. Indeed, the sample design shortcomings of the Knowledge Networks survey are less severe than for any of the widely cited studies based on convenience samples.

Family Instability Harms Kids—But are Same-Sex Families Necessarily Less Stable?

The papers in this volume follow a trajectory, from the concerns raised by Loren Marks about small convenience samples, to the large survey conducted by Mark Regnerus, to the gigantic census samples analyzed by Douglas Allen, Catherine Pakaluk, and Joseph Price. A picture emerges: in a cross-section of children raised by parents in same-sex relationships, life outcomes tend to resemble those of children raised by single and divorced parents. So perhaps the mechanism for this process is the instability of the families headed by parents in same-sex relationships?

The second Schumm chapter follows up on this question, combing the literature for references to the instability of lesbian households (gay parents are less studied, and the census samples suggest that lesbian couples raising children are less rare than gay couples doing the same). The point of departure is the claim by Biblarz and Stacey that lesbian parents are more likely to separate than are married heterosexual parents. While the Biblarz and Stacey conjecture is based on the same sort of convenience samples that much of this volume succeeds in transcending, Schumm compares study after study of lesbian parents to population data on heterosexual marriages. He concludes that lesbian couples are about twice as likely to split as are heterosexual married couples.

The research papers each contribute to our understanding of the relationship between family structure and the welfare of children. They collectively show us that family structure does matter for children’s outcomes and that we are not justified in maintaining an a priori assumption that parents in same-sex relationships do as well at raising children as do married heterosexual couples.

While all of this research is interesting, and while it contributes to our store of knowledge, many questions remain. These papers do not settle the question of parents in same-sex relationships, but they, along with several recent research papers by Paul Sullins, do warn us that the matter is still in doubt. The peremptory endorsement of same-sex parenting by the American Psychological Association has created a misleading impression on the part of policy makers about the strength of the existing evidence for the “no difference” hypothesis, while it has undoubtedly discouraged needed research on the still unsettled matter of how and why the outcomes of same-sex parenting differ from parenting by married heterosexual parents. When science and politics intersect, the consequences can be toxic for the former. Some have suggested that the instability of lesbian couples is due to the absence of a marriage option, though the Canadian study casts doubt on this assertion, as does work by Andersson and colleagues on lesbian registered partnerships in Sweden, which indicates that there, too, the risk of divorce is over twice that for heterosexual marriages.

How public policy proceeds from here is, of course, a political matter. Some see same-sex marriage as the centerpiece of individual freedom and want to push it into law by any means necessary, while the deeply held religious beliefs of others lead them to oppose it as profoundly sinful. Pragmatists might note that several countries have recently volunteered their next generation as guinea pigs in the huge social experiment of same-sex marriage. The papers in this volume suggest that we really don’t yet know how that experiment is going to turn out.

Appendix: Understanding Social Science Research: Bias and Power

The first question someone not immersed in the empirical social-science literature might ask is, “Why is this so complicated? Aren’t there enough studies already for us to have settled these questions?” Two of the main reasons we still have a lot to learn on this subject have to do with the technical issues of bias and power. These are statistical terms of art with somewhat misleading names.

A biased statistical estimator for a population characteristic will tend to take on a value that is unrepresentative of the characteristic it is meant to estimate. To give an example, in 1936, Literary Digest magazine conducted a massive opinion poll about the November presidential election. They collected responses from readers and supplemented their data with a survey of telephone subscribers and automobile owners. Their prediction? A landslide win for Alf Landon. They were half right; there was a landslide, but it was for Roosevelt, not for Landon, a man of whom you may have never even heard. Technically speaking, their polling was biased. Why? Their readers tended to be Republicans, as did, in 1936, telephone subscribers and the owners of automobiles—so they oversampled Republicans and undersampled Democrats.

What about power? When we conduct a hypothesis test, we assemble the statistical evidence relevant to the hypothesis and its alternative and then come to a verdict, very much as a court might do. So let’s think about “trying” the case of our hypothesis against the alternative. To make matters concrete, suppose we test the null hypothesis that women and men are equally likely to identify as Democrats against the alternative hypothesis that women are more likely to be Democrats than are men. In this case, we can potentially commit two sorts of errors. We commit a so-called type I error if we falsely reject our hypothesis, whereas we commit a type II error if we falsely accept it.

Now for the complicated part. The rules of hypothesis testing give the benefit of the doubt to the null hypothesis, just as criminal law embodies a presumption of innocence. So, analysts will proceed by choosing a probability of a type I error (corresponding to false rejection).Typical choices for the probability of a type I error include 10 percent, 5 percent, and 1 percent probabilities of falsely rejecting the null hypothesis.​ The probability of a type I error is called the size of the hypothesis test. However, while this approach controls the probability of false rejection (a type I error) it can leave one with a very high probability of falsely accepting an incorrect null hypothesis (a type II error). The power of a test is the probability that it does not make a type II error by falsely accepting the null hypothesis.

Power calculations are made more complicated by the fact that most of the alternative hypotheses we are interested in are in fact composite hypotheses. So if we test the null hypothesis that women and men are equally likely to identify as Democrats against the alternative that women are more likely to be Democrats, we are really testing against a whole series of alternatives: that women are twice as likely to be Democrats, that they are but 1 percent more likely, and so forth. The smaller the difference between the null hypothesis and the alternative, the lower the power—that is, the more likely we are to falsely accept the null hypothesis when in fact the alternative is true. The power of a test is also affected by how large a sample we have; all else equal, large samples will give us more power.

Of course, all of these probability calculations are made with respect to a standard set of assumptions that are at best only approximately correct, but the same tool kit is used by pretty much everyone doing quantitative sociological studies of the family.