This chapter turns to the second of the three dimensions of the evaluative framework discussed in the Introduction: the credibility of the causal and statistical models that analysts employ. To make causal inferences, analysts must maintain some hypotheses about data-generating processes. These hypotheses are “maintained” because evidence does not permit researchers to verify all such a priori assumptions, at least not completely. Indeed, inferring causation requires a theory of how observed data are generated (that is, a response schedule; Freedman 2009: 85–95; Heckman 2000). This theory is a hypothetical account of how one variable would respond if the scholar intervened and manipulated other variables. In observational studies—including natural experiments—the researcher never actually intervenes to change any variables, so this theory remains, at least to some extent, hypothetical.

Yet, data produced by social and political processes can be used to estimate the expected magnitude of a change in one variable that would arise if one were to manipulate other variables—assuming that the researcher has a correct theory of the data-generating process. In quantitative analysis, this theory is usually expressed in the form of a formal statistical model; underlying causal assumptions may or may not be stated explicitly. The key question is whether the maintained hypotheses implied by a given model are plausible depictions of the true data-generating process—and, especially, how that plausibility can be probed and, at least to some extent, validated.

Qualitative methods play a crucial role in natural experiments. Indeed, I argue that the knowledge of context and detailed information on process that qualitative methods often facilitate is crucial for the method’s persuasive use. The goal of this chapter is to develop this idea systematically, by providing a framework for thinking about the general utility of qualitative evidence in natural experiments while also providing several examples from recent social-scientific research. This in turn may help provide the foundation for better use of qualitative methods in future natural-experimental research.

The topics discussed in this chapter relate not just to analysis—the focus of this Part II of the book—but also to the discovery and evaluation of natural experiments (the focus of Parts I and III, respectively). However, the central interplay between qualitative and quantitative analysis in many successful natural experiments suggests that this material is best presented in conjunction with the previous two chapters on quantitative analysis.

Suppose that a researcher has discovered a credible natural experiment and designed a study akin to the best examples I have surveyed in previous chapters. This researcher then faces a central question: how should the data be analyzed? This second part of the book takes up this central question.

For reasons I will discuss extensively, the analysis of natural experiments should ideally leverage both qualitative and quantitative methods. There are important choices to be made with respect to both modes of analysis. This chapter and the next discuss quantitative analysis, leaving qualitative tools to Chapter 7. In this chapter, I describe the definition and estimation of causal effects. The role of chance variability—including assumptions about sampling processes and procedures for calculating standard errors—is left for Chapter 6.

A central theme of this chapter and the next is that the quantitative analysis of natural experiments can and should be very simple. Often, a minimum of mathematical manipulation is involved. For example, straightforward contrasts between the treatment and control groups—such as the difference in average outcomes in these two groups—often suffices to provide evidence of causal effects. Such difference-of-means (or difference-of-percentages) analysis plays a central role in each kind of natural experiment, from standard natural experiments and regression discontinuities to instrumental-variables designs.

The potential simplicity of quantitative data analysis provides an important advantage of natural experiments, for a number of reasons. First, it implies that statistical results are often easy to convey and interpret. Generally, just one or a few main quantitative tests are most relevant to testing key hypotheses. Rather than presenting the estimated coefficients from multivariate models in long tables of regression results, analysts may have more space in articles to discuss the research design and substantive import of the results.

In this chapter and the next, I consider two specific kinds of natural experiments: regression-discontinuity designs in this chapter, and instrumental-variables designs in the next. When appropriately invoked, both kinds of designs meet the key definitional criterion of natural experiments—namely, random or as-if random assignment. Yet they differ in other ways, and their use tends to raise characteristic issues of discovery, analysis, and interpretation that are specific to each kind of design. This makes it valuable to discuss these two kinds of natural experiments separately in various chapters, where they can each be given more detailed attention.

The regression-discontinuity design was proposed by Thistlethwaite and Campbell (1960) and discussed extensively by Trochim (1984). Yet, for somewhat unclear reasons—and despite the extensive efforts that Donald Campbell’s research team at Northwestern University put into studying it—this research design was relatively underutilized for several decades after Campbell and his team originally proposed it. However, empirical applications have exploded in the social sciences over the past decade or so. In this chapter, I survey many recent applications, as a way to shed some light on the art of discovering productive opportunities for these modes of research. Issues of analysis and interpretation are mostly postponed for Parts II and III of this book, though some mention of them is also made here.

An instrumental-variables design relies on the idea of as-if random in yet another way. Consider the challenge of inferring the impact of a given independent variable on a particular dependent variable—where this inference is made more difficult, given the strong possibility that reciprocal causation or confounding may pose a problem for causal inference. The solution offered by the instrumental-variables design is to find an additional variable—an instrument—that is correlated with the independent variable but could not be influenced by the dependent variable or correlated with its other causes. Thus, units are assigned at random or as-if at random, not to the key independent variable of interest, but rather to this instrumental variable.

Recall, for instance, Angrist’s (1990 a) study of military conscription discussed in the Introduction. Eligibility for the Vietnam draft was randomly assigned to young men, via numbers from 1 to 366 that were matched to each potential draftee’s birth date; men with lottery numbers above a particular cutoff value were not subject to the draft. Comparing men with lottery numbers above and below the cutoff estimates the effect of draft eligibility. This is “intention-to-treat” analysis, as described in Box 4.1: males are compared according their draft eligibility status, regardless of whether they actually served in the military. Intention to treat is a key principle of natural-experimental analysis, and intention-to-treat analysis should usually be reported in write-ups of research results.

The previous chapter introduced the Neyman causal model and described that model as a sensible starting point for many natural experiments. Parameters such as the average causal effect are defined in terms of this causal model. Yet, estimators of those parameters, such as the observed difference of means in treatment and control groups, also depend on the model of the chance—i.e., stochastic—process that gives rise to observed data. The Neyman urn model is one such model: it says that units are sampled at random from the study group and placed into treatment and control groups.

Because this sampling process is stochastic, estimators such as the difference of means will vary across different realizations of the data-generating process. Suppose Nature could run a given natural experiment over and over again, drawing units at random from an urn and assigning them to the treatment and control group. The data would likely turn out somewhat differently each time, due to chance error. Some units that went into the treatment group in one realization of the natural experiment would go the second time into control, and vice versa. Thus, the observed difference of means would differ across each hypothetical replication of the natural experiment. The spread or distribution of all the hypothetical differences of means is called the sampling distribution of the difference of means. A natural measure of the size of this spread is the standard deviation of this sampling distribution—the standard error. The standard error tells us how much the observed difference of means is likely to vary from the mean of the sampling distribution of the estimator, in any given replication.

Defining and dealing with contradictory truth table rows

The notion of a contradictory truth table row is easier to understand with crisp sets. It describes a situation in which those cases that are members in a truth table row do not share the same membership in the outcome. Put differently, the same row leads to both the occurrence and the non-occurrence of the outcome. Since truth table rows are, in essence, statements of sufficiency, such an empirical situation suggests a logical contradiction, for it would mean that the very same combination of conditions (aka truth table row) produces both Y and ͠Y. The analytic problem is that, based on the empirical evidence, it is not straightforward to decide whether this row is sufficient for Y, ͠Y, or neither and, consequently, whether it should be included in the logical minimization for outcome Y, outcome ͠Y, or neither. It cannot, however, be included in both minimization procedures.

There are several, mutually non-exclusive strategies for dissolving logically contradictory truth table rows in either csQCA or fsQCA prior to the logical minimization, and there is another set of strategies for handling such contradictory rows during the minimization procedure (Ragin 1987: 113–18; Rihoux and De Meur 2009). Let us first turn to the strategies for dissolving the contradiction.

Set-theoretic approaches in the social sciences

Arguments about set relations are pervasive in the social sciences, but this is not always obvious. Take, for example, Brady’s (2010) intriguing deconstruction of the widely debated claim that, in the 2000 US Presidential Election, George W. Bush lost about 10,000 votes because Al Gore had been declared the winner before the closure of the polling stations in those western counties of Florida that are on Central Standard Time (i.e., the Panhandle). This claim is made by Lott (2000), who arrived at this inference by estimating a “‘difference-in-differences’ form of regression analysis, based on data-set observations” (Brady 2010: 238). Using causal-process observations, Brady cogently shows that this inference is “highly implausible” (241) and that, instead of 10,000 lost voters, a more adequate estimate would be a maximum of 224 or, even more realistically, 28 to 56 voters (NB: total voters, not percentage!). Brady successfully frames his debate of Lott as an argument in favor of causal-process observations – “diagnostic ‘nuggets’ of data that make a strong contribution to causal inference” (Brady 2010: 237).

Brady ’s argument is set-theoretic in nature (Goertz and Mahoney 2012 ). In essence, he claims that the set of voters not voting for Bush due to the premature announcement of Gore as the winner (Y) can only be very small because membership in this set requires simultaneous membership in several other sets. Such allegedly lost Bush voters must, of course, also be members of the set of registered voters in the Panhandle counties (P), who are also members of the set of voters who had not yet voted (V), and the set of voters who had received the news through the media (M).

Beyond the Standard Analysis: expanding the types of counterfactuals

In section 6.4.3, we introduced the distinction between different types of assumptions, or counterfactual claims: simplifying assumptions, and easy and difficult counterfactuals. In the following, we further differentiate between types of counterfactuals. We will label them good counterfactuals and untenable counterfactuals, the latter consisting of implausible and incoherent counterfactuals.

Incoherent counterfactuals are defined as assumptions that contradict claims made about the same remainder at a different moment of the analytical process. This fallacy can happen in two ways. First, the researcher performs separate analyses for both Y and ~Y and includes the same remainder into both minimization procedures. By doing so, the researcher is effectively saying that the same logical remainder is sufficient for both the occurrence and the non-occurrence of an outcome. This type of assumption is already discussed in the literature under the label contradictory assumption (Yamasaki and Rihoux 2009 ). The second form of incoherent counterfactuals can occur when researchers make a claim of necessity but then also allow a logical remainder to be part of a sufficiency solution that contradicts that claim of necessity. To see how this happens, recall that if X is necessary for Y, then Y cannot occur in the presence of ~X. Formally: if X ←Y, then ~X → ~Y ( section 3.3.3 ). Hence, if a researcher claims that X is necessary for Y, then any logical remainder row that implies condition ~X cannot be used for a counterfactual claim following which it would produce outcome Y. This is, so far, a widely overlooked pitfall.

The two-step approach

The two-step approach (Schneider and Wagemann 2006; Schneider 2008) is based on the idea that often in comparative social science research endeavors, “conditions … can be divided into two groups, which can be labeled ‘remote’ and ‘proximate’ factors” (Schneider and Wagemann 2006: 759; see also Kitschelt 2003). Remoteness and proximity can be defined along various dimensions. For instance, remote factors are relatively stable over time. Also, their origin is often remote from the outcome to be explained on the space and time dimensions. As a consequence, remote factors cannot usually be easily altered by actors and are treated as exogenously given. In many research settings, remote factors are adequately labeled structural factors, contexts, historical legacies, etc. Proximate factors, by contrast, vary over time and originate not so far back in the past. They can be relatively easily modified by actors; often, they even describe human action itself. Remoteness and proximity can thus be interpreted not only in terms of space and time, but also with regard to the causal closeness which they are assumed to have.

Two-step thinking is of course not exclusive to QCA. Many sub-disciplines of the social sciences refer to causal processes (proximate factors) which unfold within certain contexts (re mote factors). Kitschelt ( 1999 ), for instance, argues that explanations that rely exclusively on remote (structural) factors provide for causal depth, but fall short of demonstrating the causal mechanisms that link deep, distant causes with an outcome. By contrast, explanations based on proximate factors display causal mechanisms (often, but not necessarily, at the micro-level). Consequently, a good causal statement consists of finding the right balance between the two core features: causal depth and causal mechanisms (Schneider and Wagemann 2006 : 761f.).

Pitfalls in inferring necessity from suffi ciency solution terms

h ere are two pitfalls related to the analysis of necessity: the disappearance oftrue necessary conditions and the appearance of false necessary conditions .We illustrate both fallacies using data from published QCA. As before in thebook, we do not aim at reanalyzing the original studies but rather alter themin order to better demonstrate our methodological arguments.

Hidden necessary conditions

Hidden necessary conditions can occur due to two, mutually non-exclusivefeatures of the data at hand. One reason consists in the kind of assumptionsmade on logical remainders. The other reason rests in the treatment of less-than-perfect set relations. In the following, we provide one example for eachsource. From these examples, we then derive the general conditions underwhich this phenomenon occurs.

Hidden necessary conditions due to incoherent counterfactuals

Stokke (2004) aims at identifying the conditions under which the strategy ofshaming makes hitherto non-compliant countries observe international fishingrules (SUCCESS). He identifies five conditions: advice (A); commitment (C); shadow of the future (S); inconvenience (I); and reverberation (R). h eten countries evenly split into five successful and five unsuccessful incidencesof shaming. The cases fall into eight different truth table rows (four connectedto success and four to the lack of success). Given a truth table with 25 conditions,there are 32 − 8 = 24 logical remainders (Table 9.1).