The Challenge of Scaling Deliberation

The theory and practice of what has come to be called “deliberative democracy” have been revived for the modern era with a focus on deliberative microcosms selected through random sampling or “sortition.” One merit of organized deliberation with a good sample is that it should, in theory, permit inferences about what the entire population would think about an issue if it could deliberate about it under similarly good conditions. However, advocates freely admit that the public will usually not be effectively motivated to consider issues in depth under anything like the “good conditions” of an organized deliberation. What Jane Mansbridge has called “everyday talk” (roughly equivalent to discussion in ordinary life outside of organized deliberative settings) is thought to be deliberative, sometimes, and to some degree, but less so than what could be achieved with an organized design (Mansbridge Reference Mansbridge and Macedo1999).

There are many challenges to the deliberative quality of mass discussion in ordinary life. The public is routinely subjected to one-sided advocacy and misinformation, facilitated by social media sharing amongst the like-minded, while, at the same time, other portions of the public often remain inattentive or disengaged from major policy questions. Low information levels and inattention are often explained as an understandable consequence of what Anthony Downs called “rational ignorance,” and its companion “rational apathy” (a term commonly applied to shareholder voting in public companies but equally applicable to citizens for many issues). Why should I vote or become involved in thinking about public issues if my individual views are likely at best to have only a miniscule impact on policy (Downs Reference Downs1957; Hardin Reference Hardin2002)? At the same time, other portions of the public may be intensely motivated, but highly unrepresentative. The voices that are motivated to speak up via self-selected processes are likely to be unrepresentative, and non-deliberative while conventional polling, even when conducted with good samples, will offer results that are some distance from the overall public’s considered judgments.

The rationale for convening deliberative microcosms such as Deliberative Polling is that they attempt to offer a picture of what the entire public would think under good conditions. The aspiration is to convene a process that gives voice to “the will of the people” (at least on selected issues) through a combination of inclusion and deliberation. In some ways, random sampling offers an admirable form of inclusion compared to most democratic practices. For example, deliberation with a good stratified random sample is likely to be more representative than a low turnout election or referendum from the same population. But even if a deliberating microcosm is highly representative of the views of the entire population (in its pre-deliberation views), it will inevitably be the case that most people whose “will” is being represented (through the inclusion of random sampling) are not actively involved, and probably not even aware of the process. As Cristina Lafont has pointedly asked, what about those who are left out of the deliberations? If the deliberations produce significant opinion changes then many of them will disagree and not feel represented by the results (Lafont (Reference Lafont2020).

One answer to this challenge is the hypothetical claim already mentioned—the results offer a representation of what the public as a whole would think about an issue under stipulated good conditions. And those who are not deliberating are included in the public as a whole from whom the sample is selected. Of course, the claim about “good conditions” must be continually interrogated, both conceptually and empirically. But that is not our focus here (see Fishkin Reference Fishkin2018 for a general account and Farrar et al. Reference Farrar, Fishkin, Green, List, Luskin and Paluck2010 and Sandefur et al. Reference Sandefur, Birdsall, Fishkin and Moyo2022 for experiments disaggregating aspects of the Deliberative Polling treatment).

A second response to the problem of those left out might be some robust attempt to bring the rest of the public into the dialogue, at least in terms of exposure and awareness. This can be done with media coverage and by advocates of the results invoking the reasons for the post deliberation opinions of the mini-public, or the fact that the results were complied with by decision makers. If the sample is representative, those arguments and the conclusions they support should have some purchase on public opinion of the broader public. Hence broadcasting and print media may amplify the process of deliberation and its conclusions, perhaps with some constructive effect on the viewers (see Rasinski, Bradburn, and Lauen ( Reference Rasinski, Bradburn, Lauen, McCombs and Reynolds1999) for an important early study of television viewing of a mini-public deliberation; also see the literature on survey experiments informing people of the results of mini-public deliberations such as Muradova, Culloty, and Suiter Reference Muradova, Culloty and Suiter2023, Muradova and Suiter Reference Muradova and Suiter2022; Germann, Manen, and Murdova Reference Germann, Marien and Muradova2022.)

Even more effective might be a third response—an attempt to actually engage the broader public in organized deliberations. The idea is not merely to expose members of the broader public to deliberations by others in a random sample, but rather, to engage them in a deliberative process organized on the same design. The experience of actually deliberating is likely to be more consequential than mere exposure to some aspect of deliberation by others. Especially on highly polarized issues, mere exposure may have little effect (see the well-established literature on CIE, the “continued influence effect” of misinformation despite exposure to correction, [Ecker et al. Reference Ecker, Lewandowsky, Cook, Schmid, Fazio, Brashier, Kendeou, Vraga and Amazeen2022, 15]) and it will sometimes backfire (Chong and Druckman Reference Chong and Druckman2007; Nyhan and Reifler Reference Nyhan and Reifler2010).Footnote ¹

However, participation in moderated discussions with diverse others in an appropriately organized design, appears to depolarize opinion and move participants to their considered judgments even on highly contested and polarized issues (Fishkin et al. Reference Fishkin, Siu, Diamond and Bradburn2021). But these conclusions are supported by results from face-to-face deliberations. Because technology offers the promise of more cost-effective and practical scaling of participation in the deliberative process itself (rather than merely the scaling of awareness of deliberations by others) we are interested in whether or not such results can be replicated with new technology. Here we will investigate whether an online, AI Assisted moderator could produce some of the same effects that have been found for those who participate in Deliberative Polls in the face-to-face mode.Footnote ² This is not a merely academic question. If the automated AI-assisted moderator facilitates an effective form of deliberation (as we will inquire later with the hypotheses provided) then scaling participation in the deliberative process becomes a more realistic and attractive possibility.

The idea of organized mass deliberation is proposed in the scheme for “Deliberation Day” (Ackerman and Fishkin Reference Ackerman and Fishkin2004). There the idea was to have the whole country deliberate face-to-face in innumerable small groups a few weeks before a national election. A plan for these discussions in every locality of the United States was described and advocated. However, organized face-to-face deliberation is a massive undertaking. It would probably require legislation and an organized institutional support structure.

The AI-Assisted Online Deliberation Platform

The online version of Deliberative Polling, and indeed, of other deliberative processes, suggests strategies for more cost-effective and realistic implementations for scaling the deliberative process to entire communities, states, or even the nation as a whole. The online deliberations were implemented in this project via the AI-Assisted Stanford Online Deliberation Platform.Footnote ³ In theory it can handle any number of small groups (of approximately ten people in each group) in synchronous video-based discussions. The process moderates itself, with a queue for talking time (currently set at 45 seconds per intervention), nudges speakers who are not participating, intervenes for uncivil language, produces nearly instant transcripts of all the discussions to document the arguments for research purposes, orchestrates the movement of the discussions through an agenda of proposals and initial pros and cons of each proposal, etc. and coordinates the group in identifying its two most important questions for the plenary sessions. Evaluations of the platform by participants show ratings as high or higher than projects with human moderators (refer to online appendix table 1 for the evaluations). We call the platform “AI-Assisted” rather than AI facilitated because at crucial points, it draws on the collective intelligence of each small group, rather than having the AI make the decision itself for the group. Has the group sufficiently discussed the first proposal? Has it covered the pros and cons of that proposal? The group is polled on these and similar decisions as it moves through the process. In effect, the platform assists the group in moderating itself.

The prospect of mass automated deliberation is at hand. What are some key issues to engage the deliberative theory and empirical communities about the potentially realistic possibility of mass deliberation via this kind of technology? What do we need to know to evaluate whether mass applications on this model would usefully contribute to the democratic process? Our initial answers will come from a larger than usual application of Deliberative Polling, but the data also offers a picture that we can use for evaluation of some key elements of what might become organized mass deliberation beyond the confines of random samples.Footnote ⁴ We are proposing Deliberative Scaling on the same model of discussion as Deliberative Polling, with the same (automated) moderator and the same kinds of briefing materials used in the national experiment discussed here.

Given the scale and ambition of mass organized deliberation, why should we take the idea seriously? Because there is evidence that deliberation in organized settings can address three notable problems with the operation of mass democracy. Some of this evidence comes, suggestively, from face-to-face deliberations with random samples. Do we have any reason to believe that similar results might follow if technology were used to scale deliberations on the same model of discussion with larger populations?

Here we will investigate the application of a specific technology which closely models the Deliberative Polling design used in face-to-face deliberations. We will draw inferences from its use in the largest Deliberative Poll conducted up to that time, nearly 1,000 participants in 104 small groups; plus, nearly 700 in a pre-post control group that only answered the same questionnaire at the time of recruitment and at the end of the process).Footnote ⁵

Deliberations by random samples, mostly in face-to-face discussions, seem to produce at least three notable effects, which if scaled to very large numbers in the broader society, would likely help with three major challenges facing democracy. First, competitive democracies often seem crippled by extreme partisan polarization. The divisions between parties at the mass level are not just extreme, they are alleged to be “calcified” or intractable (see Sides, Tausanovitch, and Vavreck Reference Sides, Tausanovitch and Vavreck2022), fueling not only division but deadlock (Hetherington and Rudolph Reference Hetherington and Rudolph2015). Yet there is evidence that face-to-face deliberation can produce significant partisan depolarization. Can the same be true of deliberations with the AI-assisted online moderator? Second, democracies are supposed to make a connection between the will of the people and what is actually done. If deliberative democracy applications are intended to clarify the “public will” they must identify the considerations that support the public’s considered judgments. Face-to-face deliberations have a record of doing so. Will the automated online process be equally successful? Third, the predominant accounts of voting behavior in our competitive democracies are that the tribalism of party loyalties mostly determines who turns out and who gets elected (Green, Palmquist, and Schickler Reference Green, Palmquist and Schickler2004; Achen and Bartels Reference Achen and Bartels2016). The idea that voters evaluate policies on the merits and cast their votes accordingly is commonly treated as a quaint leftover from the now widely abandoned “classical theory” of democracy (Schumpeter Reference Schumpeter1942; Achen and Bartels Reference Achen and Bartels2016; Posner Reference Posner2003; Shapiro Reference Shapiro2003). But if voters do not really consider policy positions in casting their votes, then how are they to exercise popular control? On the other hand, there is evidence that face-to-face deliberation, whether in juries or deliberative mini-publics such as Deliberative Polling creates a latent variable of civic engagement that has a significant effect on voting (Gastil, Deess, and Weiner Reference Gastil, Deess and Weiser2002, Fishkin et al. Reference Fishkin, Bolotnyy, Lerner, Siu and Bradburn2024). Do these effects occur with the automated online process?

We propose three hypotheses corresponding to these three notable effects. The hypotheses are based on results from face-to-face deliberations as well as related literature.

One of the possible benefits of scaling mass deliberation might be the diminution of extreme partisan polarization. A national face-to-face Deliberative Poll produced some striking results along these lines (Fishkin et al. Reference Fishkin, Siu, Diamond and Bradburn2021). Do we see similar evidence for the online process with the automated platform? Following our previous work (Fishkin et al. Reference Fishkin, Siu, Diamond and Bradburn2021) we define depolarization in terms of whether or not the mean positions of the two parties move closer together. This can result from one-sided or two-sided movement (Fishkin et al. Reference Fishkin, Siu, Diamond and Bradburn2021, 4).Footnote ⁶ Why might this happen? As we argued in Fishkin et al. Reference Fishkin, Siu, Diamond and Bradburn2021, “Kunda hypothesized … that there are two kinds of “motivated reasoning”—“directional” and “accuracy based” (Kunda Reference Kunda1990). There is an obvious argument that discussion will increase partisan polarization on issues that are already polarized because of directional motivated reasoning. But if an experimental treatment could encourage the second kind of motivated reasoning—accuracy-based—then reasoning on the merits would likely result. Further, if on issues of partisan polarization, people have arrived at positions without seriously considering (or even encountering) arguments on the other side, then we might well expect accuracy-based motivated reasoning to reduce partisan polarization by overcoming the legacy of previously one-sided reasoning. The expectation here is not that deliberation will always depolarize but that it will likely depolarize on issues where participants begin the exercise with high levels of partisan polarization. Generally, Kunda concluded that subjects will be motivated to be accurate when they think their views will matter, and when they have to share the reasons supporting them” (Kunda Reference Kunda1990, 481).Footnote ⁷ This is the framework we applied in Fishkin et al. Reference Fishkin, Siu, Diamond and Bradburn2021, 1468, and we employ it again here.

Deliberative mini-publics are designed to foster the weighing of competing reasons by the participants in an evidence-based environment. But it is an empirical question whether they succeed in fostering opinions and opinion movements that show the effect of identifiable reasons. For evidence of the role of identifiable reasons in face-to-face Deliberative Polls, see Fishkin Reference Fishkin2018, especially Part III. Given that the topic in these discussions is climate change, we also rely on Krosnick et al. Reference Krosnick, Holbrook, Lowe and Visser2006, who identified questions that affect the importance of climate in U.S. national public opinion.

Lastly, does deliberation with the online automated moderator platform help facilitate deliberative voting in actual elections?

The very idea of a deliberative voter, one who thoughtfully considers the arguments for and against candidates (or parties or policies) and then votes to a considerable degree in accord with those considered judgments has been largely dismissed in democratic theory since Schumpeter relegated it to the “classical theory” of democracy, to be superseded by the modern “competitive theory” (which makes no mention of the public will, just the “competitive struggle for the people’s vote”). The most systematic, empirically based reprisal of Schumpeter’s account, the “realist theory” of Achen and Bartels (Reference Achen and Bartels2016) dismisses anything like the deliberative voter as a “folk theory” never to be realized. Yet despite Schumpeter and his followers, the aspiration has a continuing hold on normative theories of democracy even beyond the literature on deliberative democracy. Robert Dahl, for example, included the requirement of “enlightened understanding” on “the issues to be decided”, as one of the essential criteria for a system to be considered democratic. However, he offered little basis for concluding it might be practical (apart from his endorsement of what we would now call a deliberative minipublic, the “mini-populous” in his concluding reflections on an advanced democratic society (Dahl Reference Dahl1989, 126-128 and 340-341).

Can deliberation in an organized setting bring the deliberative voter to life? In principle, might we use technology to produce this effect? If so, this might make scaling to large numbers a more practical aspiration, particularly if the effect had staying power.

Gastil et al. (Reference Gastil, Deess and Weiser2002) found indirect effects on voting from serving on a jury that reached a verdict. They found that depth of deliberation (which they measured by the number of counts considered by a jury) was one of the mediators for the likelihood of voting (see also Gastil et al. Reference Gastil, Dees, Weiser and Simmons2010, where they extend the analysis of jury participation’s effect on turnout with additional data). Fishkin et al. (Reference Fishkin, Bolotnyy, Lerner, Siu and Bradburn2024) found that indirect effects from face-to-face deliberation affected voting a year later in the presidential election. In that case the dependent variable was not just turnout but voting intention about whom to vote for. Once again the analysis focused on a latent variable of civic engagement analyzed via causal mediation analysis.

America in One Room: Climate and Energy

What would the American public really think about our climate and energy challenges if it had the chance to deliberate about them in depth, with good and balanced information? If the American people—or in this case, a representative sample of them—could consider the pros and cons of our different energy options, which would they support? Which would they cut back on? What possible paths to Net Zero (the point at which we are not adding to the total of greenhouse gases in the atmosphere) would seem plausible to them? Which proposals would they resist? Can the public’s conclusions chart a way forward on our climate and energy dilemmas—a way forward that transcends our great divisions, especially our deep partisan differences? The aim of the project was not to persuade the participants to move in any particular direction. There were balanced briefing materials with suggested pros and cons for each proposal, balanced plenary panels, and efforts to equalize the participation of all the participants.

NORC at the University of Chicago selected a nationally representative sample of 962 respondents to deliberate on 72 substantive questions, and a second representative sample (a pre- and post-control group of 661) that did not deliberate but took essentially the same questionnaire in the same time period. Each group completed the survey upon being recruited (in early August) and again at the end of the experiment (in late September 2021) as well as a year later (close to the mid-term Congressional elections). The treatment and control groups were two independent samples both drawn from the same probability-based panel, selected so that there is no overlap between the two samples and weighted so that they are comparable. See the online appendix section on “Sampling and Weighting” for a detailed description as well as the Balance Table (A2) in the online appendix.

On 66 of the 72 issue propositions in the survey, the participants changed significantly over the course of the deliberation. Almost all the changes were toward doing more to combat climate change, and these changes were generally in the same direction across party and demographic divides. Democrats were initially more supportive of ambitious policies to address climate change, and Republicans were initially more skeptical (with Independents falling in between the two). However, by the end of the deliberations, majorities of Republicans had come to support the general principle of “serious action to reduce greenhouse gases,” along with a number of specific proposals to “dramatically accelerate” adoption of renewable sources of energy and to slow deforestation. At the same time, Democrats became more supportive of including a new generation of nuclear power plants in the future energy mix, and a majority of Republicans remained wary of a hard deadline of 2050 for phasing out oil and natural gas.

Following our work in Fishkin et al. (Reference Fishkin, Siu, Diamond and Bradburn2021) and Fishkin et al. (Reference Fishkin, Bolotnyy, Lerner, Siu and Bradburn2024) we focus our attention on the most extremely polarized issues and then use those items to construct a policy-based score (PBS).Footnote ⁸ We focus on these particular issues out of the 72 total policy questions posed to study participants, because they enable us to clearly classify an individual on a left-right policy dimension. Only for the polarized issues are Democrat respondents concentrated on the left and Republican respondents concentrated on the right, allowing for a unidimensional scale. These are also the issues where beliefs are strongest and most passionate, making any sort of change especially difficult and unlikely. We thus view any changes we measure to an individual’s PBS as a lower-bound on the capacity of deliberation to change minds and attitudes on policy issues. Following the methodology in Fishkin et al. (Reference Fishkin, Bolotnyy, Lerner, Siu and Bradburn2024), which was an experiment involving in-person deliberation, has the additional benefit of allowing us to test whether deliberation on the automated platform can generate similar depolarization as measured for the most extremely polarized issues from the face-to-face project.

While Fishkin et al. (Reference Fishkin, Bolotnyy, Lerner, Siu and Bradburn2024) from the face-to-face study, used 26 questions across five different issue areas to construct its PBS, our questions here only relate to climate. When we follow the same criteria for question selection, we identify only 9 questions that can be classified as extreme partisan polarization. These criteria require 1) 50% or more of Democrats and Republicans to be at opposite sides of an issue and for 2) 15% or more of Democrats and Republicans to be at opposite extremes of an issue.Footnote ⁹ While our results are robust to using a PBS constructed in this way out of nine questions, they are noisier than if we use a measure of extreme polarization that is constructed out of a larger set of questions. Thus, the results we present throughout the main body of this paper use a twenty-seven question PBS, with these questions identified only using criteria 2) provided earlier (Alpha = 0.98).Footnote ¹⁰

The PBS ranges from 0 to 10, with 0 denoting most favorable to climate action and 10 denoting least favorable. The T1 score (using the pre-period survey) and T2 score (using the post-period survey) for each individual are an average over their responses to each of the twenty-seven questions. Before averaging, we make sure that the response to each question is converted to the PBS rubric (e.g., if the extreme Republican response to the question was 0, we flip everyone’s scores, turning 0 s into 10, 1 s into 9s, etc. before averaging). We do this because of the way some questions were worded.

Did Deliberation Produce More Deliberative Voters?

Hypothesis 3 posits that deliberation will produce a latent variable of civic engagement that will yield more deliberative voters. Recall that the national deliberation in this experiment occurred in late summer /early autumn 2021 (T1 at time of recruitment in early August, T2 after deliberation in late September) and the third questionnaire (T3) was fielded in October 2022 before the mid-term elections. Did the deliberations in 2021 affect vote choice a year later in the midterms in 2022? Because of the time lapse of a year, because the intervention was via an online discussion, and because the election was a mid-term for control of Congress (rather than a presidential election) the hypothesis sets a high bar.

We will look at how much deliberation predicted support for Democratic control of Congress in the mid-term elections through the prism of how much it changed the relative importance of climate. Because the Republicans are generally more skeptical of climate change and the Democrats more in favor of actions to address climate change,Footnote ¹⁵ we hypothesize that support for Democratic control for Congress will be correlated with an increase in concern for climate change resulting from participation in the deliberative groups.Footnote ¹⁶

To test this proposition, we fit a logistic regression model on support for Democratic control of Congress against deliberation. We also interacted deliberation with respondent prioritization of climate as an issue that drove their vote. Admittedly, we are assuming that those who support action on climate change will see that as a reason for supporting democratic control of Congress, given the evident party differences on the issue. Obviously, depending on the issue, the impact on deliberative voting could yield support for either party (depending on the party positions on the issue in question).

Model 1 in table 3 tests this directly near the time of the election by interacting participation with respondent reported climate importance on preference for control of Congress. If participation in the deliberation treatment has the effect we predict, we should see a significant and positive effect in this interaction, meaning that the linking of climate salience with preference for control of Congress is only true for participants in the deliberation treatment. We find exactly that—while also finding no strong direct impact of participation independent of climate importance and no impact of climate importance for the non-participants on preference for control of Congress.

Table 3 DV: Support democratic control of congress (Time 3)

+ p<0.1

* p<0.05

** p<0.01

*** p<0.0.01

Note: The models are logit regressions with random intercepts for state. Models include demographic controls for respondent age, gender, race, income, education, and region, respondent party ID, as well as respondent T1 PBS. We include survey weights to ensure the balance between the treated and control groups. The results in table 3 hold when we run OLS regressions instead of logistic regressions (refer to online appendix table A6), to address concerns raised by Berry, DeMerrit, and Escarey (Reference Berry, Jacqueline and Esarey2010) about logistic regression and interactions. Because we are estimating an average treatment effect, using a linear probability model should produce consistent results (Angrist and Pischke Reference Angrist and Pischke2009). We also show the marginal effects plot to demonstrate how different the two sets of equations are.

We use Models 2 and 3 to test placebo importance measures to see if the effect exists for issues that were not discussed during the deliberation. Instead of climate importance, Models 2 and 3 use crime and democracy importance as plausible substitutes that should be independent of deliberation, since neither issue was discussed. And for those two issues, we find the opposite relationship than we did with climate: the overall issue importance for crime or democracy strongly predicts respondent preference for control of Congress and there is no moderating effect by participation in the treatment at all. That is to say, it appears unlikely that respondent participation had any discernible impact on the overall importance respondents cited for the issues of crime and democracy, and to that effect, it is only through affecting overall climate importance that the treatment drove changes in respondent preferences. We see this illustrated more clearly in figures 6 and 7, which are marginal effects plots derived from Models 1 and 2 from table 3.

Figure 6 Predicted support of democratic congress: Climate importance

Notes: Marginal Effects plot based on Model 1 from table 3. All control variables are held at either their median for continuous variables or their mode for categorical variables.

Figure 7 Predicted support of democratic congress: Crime importance

Notes: Marginal Effects plot based on Model 2 from table 3. All control variables are held at either their median for continuous variables or their mode for categorical variables.

We see in figure 6 that, as climate increases in overall issue importance, the probability that a respondent will support Democratic control of Congress increases hardly at all for members of the control group, but quickly for members of the treatment group. And, among those who listed climate as the most important issue, there was a 10% difference in their likelihood of supporting Democratic control of Congress by treatment status. Indeed, the opposite relationship also holds true: if, after the deliberation and discussion over climate issues you still did not think that climate was an important issue weighing on your vote, you were 23% less likely to support Democratic control of Congress than if you had not participated at all in the first place.

Figure 7 provides a revealing counterfactual to figure 6 by looking at crime as opposed to climate, an issue that should not have been affected by participation in the deliberation. And, indeed, we see that borne out when comparing the treatment and control groups on the figure: there is virtually no difference in the relationship between the importance of crime on vote and on preference for Democratic control of Congress. There is a consistently strong connection between crime importance and lessened support for Democratic control of Congress – indeed stronger than climate issues—but one that is unaffected by the treatment.Footnote ¹⁷

Causal Mediation Analysis and Deliberative Voting

It may seem surprising that a weekend of deliberation could have an effect on voting preferences a year later. Most proposals to insert organized deliberations into an election context look only at short-term effects. What are the opinions (and reasons) of the sample soon before primary elections (Fishkin Reference Fishkin1991; McCombs and Reynolds Reference McCombs and Reynolds1999), before ballot propositions in the United States (as with the Citizens Initiative Review, see Gastil Reference Gastil2000 and Gastil and Noblock Reference Gastil and Noblock2020) or before national referendums (Andersen and Hansen Reference Andersen and Hansen2007). Few have contemplated long-term effects of deliberation (or indeed any other intervention) on voting behavior for periods as long as a year preceding an election.

Unlike the two cases cited earlier, Gastil et al. (Reference Gastil, Deess and Weiser2002) and our own work in Fishkin et al Reference Fishkin, Bolotnyy, Lerner, Siu and Bradburn2024, the intervention here was online, not face to face. The online intervention is of special interest because of its potential for scaling. Furthermore, because the deliberations were about a specific substantive topic, we can compare the salience of that topic at the time of the election with the salience of other topics as a voting issue. In this case, our results show that deliberation in an organized process, even through an automated online discussion, can create more deliberative voters in a national election a year later.

But what accounts for the persistence of climate salience when there are so many other issues in a highly contentious national election? We turn to causal mediation analysis.

Causal Mediation Analysis: Estimating Direct and Indirect Effects of Deliberation

This section reprises the framework we used in Fishkin et al. (Reference Fishkin, Bolotnyy, Lerner, Siu and Bradburn2024) because we are exploring whether the effects from the face-to-face case can be replicated with the online AI-assisted deliberations. The traditional method of exploring relationships between a treatment and outcomes is by using a regression model. However, this method fails to disentangle underlying causes and effects that are indirect rather than direct. In our case, we know that there is an effect of participating in the deliberations on an individual’s climate preferences and it seems to have an impact on electoral preferences over a year out. It is, however, unsatisfactory to state that participating in the deliberations is the direct cause of these electoral preferences: surely there were intermediate steps caused by the deliberations that, when taken together, affect these outcomes.

When faced with the possibility of indirect effects, investigators may have prior knowledge that an explanatory variable plausibly exerts its effect on an outcome via direct and indirect pathways. In the indirect pathway, there exists a mediator that transmits the causal effect. As noted, we outline mediation analysis later following along closely with the discussion of mediation analysis in our previous work (Fishkin et al. Reference Fishkin, Bolotnyy, Lerner, Siu and Bradburn2024, 13-16). We follow Imai, Keele, and Tingley (Reference Imai, Keele and Tingley2010) and Imai et al. (Reference Imai, Keele, Tingley and Yamamoto2011) (see also Zhang et al. Reference Zhang, Zheng, Kim, Van Poucke, Lin and Lan2016 building on the same framework).

Suppose we have variables T and Y indicating the treatment variable and outcome variable, respectively. Mediation in its simplest form involves adding a mediator M between T and Y. The sequential ignorability assumption, critical to causal mediation analysis, states that the treatment (explanatory variable T) is first assumed to be ignorable given the pre-treatment covariates, and then the mediator variable (M) is assumed to be ignorable given the observed value of the treatment as well as the pre-treatment covariates (Imai, Keele, and Tingley Reference Imai, Keele and Tingley2010; Imai et al. Reference Imai, Keele, Tingley and Yamamoto2011).

The first part is often satisfied by randomization, while the second part implies that there are no unmeasured confounding variables between mediator and outcome. The standard mediation analysis starts with three equations, usually modeled with continuous outcomes (though advances in methods now allow for most parametric modeling approaches for stage of the mediation):

([1]) $$ \mathrm{Y}={\mathrm{i}}_1+\mathrm{cT}+{\mathrm{e}}_1 $$

([2]) $$ \mathrm{Y}={\mathrm{i}}_2+\mathrm{c}\hbox{'}\mathrm{T}+\mathrm{bM}+{\mathrm{e}}_2 $$

([3]) $$ \mathrm{M}={\mathrm{i}}_3+\mathrm{aT}+{\mathrm{e}}_3 $$

As Zang et al. explicate this standard approach:

([4]) $$ \mathrm{Y}={\mathrm{i}}_2+{\mathrm{bi}}_3+\left(\mathrm{c}\hbox{'}+\mathrm{ab}\right)\mathrm{T}+{\mathrm{e}}_2+{\mathrm{be}}_3 $$

This standard setting for mediation analysis was refined and brought into the potential outcomes framework in Imai et al. (Reference Imai, Keele, Tingley and Yamamoto2011). The authors propose a set of methods that unifies the approach to identifying direct and indirect effects, relying on a set of assumptions that are more readily testable than classical mediation analysis provides.

Causal Mediation Analysis: Climate Change and Voting

We follow the procedures for using causal mediation analysis to study indirect effects of deliberation laid out in Fishkin et al. (Reference Fishkin, Bolotnyy, Lerner, Siu and Bradburn2024), where we looked at the effects of deliberation on voting behavior through an activation of a latent civic awakening. We believe our research design lends itself to a similar analytic framework for this online case. We ran causal mediation analysis to determine the indirect effect of deliberation (from T1 to T2) on respondents prioritizing climate as a consideration in voting behavior and on respondent preferences for which party controls Congress at T3. We believe the mediating effects of deliberation on these two respective outcomes are going to be changes in how much the respondent worried about climate (as represented in the “Worry Climate Scale”), how many true things they believe about the climate (as represented in the “Climate Belief Scale”), and in how knowledgeable about the climate they are (as represented in the “Climate Knowledge Scale”). We view these three mediators as an overall latent climate engagement dimension, induced by deliberation at the event a year earlier. For more details on the model, refer to the “Addendum on Causal Mediation Analysis” in the online appendix.

In this analysis we adapt our approach to the indirect effects of deliberation on voting in Fishkin et al. Reference Fishkin, Bolotnyy, Lerner, Siu and Bradburn2024, 17. Participation in the deliberations significantly increased worry about climate, the number of correctly held climate beliefs, and the overall knowledge about climate, as demonstrated in pervious sections. Because of this, we know that it is possible that these three mediators will have significant indirect effects on outcomes, even if there is weaker evidence for a direct effect of these mediators. Table 4 shows the indirect effects of deliberation on climate worry, beliefs, and knowledge on how important climate is to vote choice, and respondent preference that Democrats control Congress. The values are the average causal mediated effect, with bootstrapped 95% confidence intervals which follow.

Table 4 Average causal mediated effect (ACME) of deliberation (95% CI)

* p<.1

** p<.05

*** p<.01

Note: Each model is fit using a generalized linear mixed effects model for both the mediators and the dependent variables—linear models for each of the mediators and for the dependent variables. Random intercepts were fit at the state level. Models include demographic controls for age, gender, race, income, education, and region, as well as respondent T1 PBS. We include survey weights to ensure the balance between the treated and control groups. Observations include participants and control groups members. Climate importance on vote choice is a 0–10 scaled variable, with 0 being of no importance at all, and with 10 being extremely important for respondent vote choice. Congress Control Vote Preference for Democrats is a binary variable (1 if support Democratic control of both chambers, 0 otherwise). Models are fit using the “mediation” package in R with 95% CI included in the parenthesis. This approach is generally parallel to our work in Fishkin et al. Reference Fishkin, Bolotnyy, Lerner, Siu and Bradburn2024.

We find consistent effects of deliberation on the aggregate latent climate dimension—as measured through changes in climate worry, belief, and knowledge—on respondent valuation of the importance of climate on their vote choice. Note that we do not get to see their specific vote choice, so this is only a self-reported measure of what was important to their vote. The fact that the effect is consistent across five of the six indicators of latent climate engagement is indicative of the robustness of the effect. We do not find an indirect effect on respondent preferences for who controls Congress—with the exception of the Climate Belief Scale in both formulations (time 1 to time 2 and time 1 to time 3) and the Climate Knowledge Scale from time 1 to time 3. The presence of these indirect effects, as measured by the causal mediation analysis, suggests that the effects of participating persisted in ways that raised climate salience even a year after the event. And as we have shown, climate salience at the time of the election has an effect on deliberative voting. Voters connect the salience they attribute to climate with voting for a Democratic congress. By contrast the control group did not prioritize climate but instead prioritized crime or other issues in their voting preferences for who should control Congress.