1985 marked the peak of survey research’s golden age in the United States. Almost all households had a phone, and people picked up the line when a call came in. Answering machines were still novel, found in fewer than ten percent of households. When pollsters called a potential respondent and asked them to take a survey, more often than not they complied. Telephones freed interviewers to work across time zones from a central location. Unlike the days of face-to-face interviewing in the 1950s and 1960s, researchers did not need to physically go to the respondent; the respondent would come to them.
Over time, this golden age faded away. Caller ID made it possible for potential respondents to avoid unknown numbers, decreasing survey contact rates. Those people that could be reached became increasingly obstinate, refusing to participate in surveys altogether. By the late 1990s, it appeared that polls were in crisis, with response rates dropping precipitously with each passing year. The future became even more bleak with the rise of cell phones, which increased both the costs and the logistical complexity of conducting surveys. With the introduction of internet polling, the business of survey research became more complicated still. These changes have altogether upended the world of polling. With the apparent failures of recent elections in mind, some question whether it will be at all possible to accurately measure public opinion. Perhaps survey research is dead.
Or maybe not. In his thoughtful and timely new book, Polling at a Crossroads, Michael Bailey offers a systematic framework for approaching the polling enterprise as it stands today. Bailey advances our understanding of modern polling’s challenges while providing practical solutions for moving forward, in what is a must-read for anyone interested in survey research in today’s world.
Bailey’s book examines the performance and practice of surveys through the lens of non-response—understanding polls by paying close attention to both those people who comply with our requests to be interviewed and those we cannot reach. Whether the distinctions between the groups are “ignorable” or “non-ignorable” becomes a critical factor, as this identification proves essential for understanding the challenges facing modern polling.
When pollsters draw samples that are not representative of the population on measured quantities—such as age or education—they can seek to adjust their samples to match known population distributions (drawn, for example, from the Census). As Bailey discusses, the most common tool to make such adjustments is weighting, that is, increasing the weight given to underrepresented groups and decreasing the weight of overrepresented groups. Weighting works well as a solution if the people we do not reach are just like the people we do reach. However, taking a survey is a choice, so this assumption rarely holds. People who agree, or indeed volunteer, to be interviewed are often systematically different from the people who opt out of a survey. These individuals may be more compliant, more engaged with the subject matter, or they may differ in a host of other ways that are difficult to quantify. This leads to a serious and less easily addressable problem—non-ignorable nonresponse. In practice, it is difficult to know if we have adequately captured the differences between our sample and the full population. These unknown unknowns can trip us up and skew our results. Put simply, under conditions of non-ignorable non-response, polls can fail spectacularly.
The bulk of Bailey’s book takes head-on the practical problems arising from non-ignorable nonresponse. The first step is diagnosis. Bailey discusses different ways to examine polling data in a thoughtful manner to find evidence of non-ignorable nonresponse. While non-ignorable nonresponse depends on the correlation of unobserved variables, Bailey shows it can leave traces in the data, namely because survey averages will change as response rates vary. If, for example, respondents who are easier to reach are more supportive of a given candidate or policy, we can tell that we have a problem. With care, we can diagnose problems through careful analysis.
But Polling at a Crossroads does not stop at diagnosing non-ignorable nonresponse. The second half of the book tackles the more challenging question of how to fix these problems. How can we leverage information to account for—and adjust—differences between the people who happily take our surveys and the people who refuse?
Bailey begins by introducing bounding methods, which define the range of possible population values consistent with observed data. These methods enable us to characterize the uncertainty we have about our polling results. He acknowledges, however, that for most practical cases, these bounds are so wide as to be uninformative of the underlying picture of public opinion. To learn anything concrete from our polls, we need to turn to model-based approaches that articulate the assumptions that allow us to characterize public opinion.
The book addresses a variety of models, beginning with the Heckman selection model. Though this workhorse has somewhat fallen out of favor because of its restrictive distributional assumptions, Bailey makes an important point about its use: selection models may come with assumptions, but the consequences of those assumptions pale against the existing (and largely unspoken) assumption that there is no nonresponse problem with surveys. From this relatively restricted model, Bailey moves to considering more flexible solutions. A chapter on “next generation” selection models introduces techniques such as the Copula method, permitting researchers to employ distributions beyond the bivariate normal assumed by the Heckman model.
Bailey then explores the possibility of collecting additional data to further improve the realism and performance of techniques used to account for non-ignorable nonresponse. For example, he discusses focusing on a subset of respondents through a randomized response instrument to learn more about the larger selection mechanism that drives the data generating process. Throughout, the author does an excellent job of both giving the intuition behind the different techniques he discusses and also assessing the trade-offs required to further specify what the public thinks at a given moment in time. He provides helpful empirical examples to illustrate the methods—from assessments of HIV prevalence across different countries in Africa to estimating racial attitudes in the United States.
One aspect of the book I particularly appreciated was Bailey’s willingness to address the heated discussions in polling about the pros and cons of probability versus non-probability polls. Proponents and fierce critics exist on both sides. Bailey points out that this larger argument has obscured fundamental problems with polls in general. Whether probabilistic or not, all polls share the (often unstated) assumption that we can just ignore non-ignorable nonresponse. Bailey argues convincingly that we need to reassess that assumption for all polls. We need to take polls as they are and figure out design and model-based solutions to account for problems.
I also found the book’s practical resources especially valuable. Chapter 2 provides an excellent overview of the history of polling, while Chapter 3 offers intuitive explanations of weighting that would be appropriate for undergraduates or, for that matter, anyone interested in the subject. While the diagnostics and solutions become more technical in later chapters, the intuition is still well presented and would work effectively in a graduate class. Finally, Chapters 12 and 13 walk through several examples applying the framework in realms of politics and health. These are very useful guides for those seeking to apply the methods and logic of the book to their own work.
Yet what is most valuable about this book is that it offers an accessible conceptual framework to build on for the future. Bailey advances a paradigm that accounts for both ignorable nonresponse and non-ignorable nonresponse, providing a framework for collecting new data and using existing data in new ways. The problems might change at the margins, and the fixes might evolve, but with a fully formed understanding of the issue at hand, we can implement new solutions.
Polling at a Crossroads arrives at a critical moment for survey research. As response rates continue to decline and new technologies transform how we communicate, Bailey’s approach provides both a sobering assessment of current challenges and a constructive path forward. In addition, Bailey’s clear exposition and practical examples make even complex concepts accessible. This book should be required reading for survey methodologists, pollsters, and anyone who relies on survey data to understand public opinion. Bailey has diagnosed what ails modern polling but also has provided a comprehensive treatment plan. Whether survey research is truly dead or merely transformed, Bailey’s work ensures that we can approach its future with both clear eyes and suitable tools.
