1.1 An Expert-Sourced Measure of U.S. Federal Judges

The JuDJIS method introduced here incorporates a technique that, to my knowledge, has not been used to measure ideology systematically in any field: computational text analysis of expert evaluations. The method involves a novel text-analysis technique, quantifying to date over three decades of evaluations by tens of thousands of legal experts, eventually covering over 2,200 judges. The evaluations are compiled by academic publisher Wolters Kluwer’s Almanac of the Federal Judiciary and are provided by lawyers and ex-law clerks—in their own words in response to prompts—with professional familiarity with each of the judges, including the judges’ written opinions, judging styles, and courtroom/chambers demeanors (Rosen Reference Rosen2023). In what follows, I set forth the assumptions underlying JuDJIS’s behavioral model of measuring judicial behavior.

1.2 Behavioral Model and Theoretical Assumptions

The choice to measure ideology from written expert evaluations makes two primary assumptions. First, ideology is a latent trait and therefore not directly observable (Fischman and Law Reference Fischman and Law2009). But I believe that a judge’s “normative beliefs about the appropriate functions of law and courts” (Cope Reference Cope, Epstein, Grendstad, Šadl and Weinshall2024) directly cause him or her to apply the law in certain ways in written opinions and oral decisions. This judicial behavior—e.g., written orders and opinions, private settlement discussions in chambers, and comments made from the bench—is therefore a direct manifestation of the judge’s ideology, and it is the closest thing to ideology itself that can be observed. The JuDJIS methodology is therefore unique among comprehensive lower-court measures, in that it captures expressed ideology, rather than others’ expectations or predictions of ideology.

Second, judicial “votes” are undeniably a key type of judicial behavior and highly probative of judicial ideology. But a judge’s judicial reasoning in her written opinions, orders, and analysis from the bench—which the Almanac evaluations capture indirectly via the lawyers’ observations of those behaviors—provides clearer insight into her judicial ideology than her decision to simply affirm or reverse.

For these reasons, a well-designed expert-sourced method based on observed judicial behavior should be able to come reasonably close to observing latent judging traits, including judicial ideology. At a minimum, expert-sourced evaluations are likely to relate more closely to ideology than the observations underlying most or all current judicial ideology measurement approaches, for instance, pre-confirmation speculation about future behavior (Segal and Cover Reference Segal and Cover1989), the campaign contributions of law clerks (Bonica et al. Reference Bonica, Chilton, Goldin, Rozema and Sen2017), judges’ pre-confirmation political campaign contributions (Bonica and Sen Reference Bonica and Sen2017), the appointing president’s party (Nagel Reference Nagel1961; Schubert Reference Schubert1960), the congressional voting records of pre-nomination supporting senators (Epstein et al. Reference Epstein, Martin, Segal and Westerland2007), and perhaps even case votes (Martin and Quinn Reference Martin and Quinn2002; Spaeth et al. Reference Spaeth, Epstein, Ruger, Whittington, Segal and Martin2014).

2.1 The Almanac

The Almanac has been published by Wolters Kluwer Publishing since 1985, with robust lawyers’ evaluations collected by professional surveyors since 1990. It contains detailed biographical data and subjective evaluations entries on all judges in the federal judiciary (including senior judges, as well as bankruptcy, magistrate, and other Article I judges). The judges’ entries comprise biographical information, key cases, and, most important for these purposes, the exclusive-to-the-Almanac lawyers’ evaluations. In response to interviewer prompts, a set of lawyers evaluates different aspects of each judge using the lawyers’ own words. The Almanac routinely updates the judges’ entries, with all judges within a given district or circuit updated in a single batch. The responses form the corpus for the data set.

Wolters Kluwer contracts with professional survey firms to conduct the lawyers’ evaluations. For each judge, they seek a “cross-section of the legal community” with substantial and recent familiarity with that judge (Rosen Reference Rosen2023, Introduction 1). All evaluators are guaranteed anonymity to promote candor. The surveyors attempt to represent lawyers in different sectors, that is, “criminal defense lawyers, plaintiffs’ tort lawyers, defendants’ tort lawyers, commercial lawyers, civil rights lawyers, etc.,” as well as former law clerks (Rosen Reference Rosen2023, Introduction 1).

The strategy to identify the appropriate sample of lawyers is tailored to the particular jurisdiction in question, as different types of districts (e.g., rural/urban, Northeast/South) have different dynamics between and within the federal courts and bar, but the overarching goal for every court is to achieve a representative sample of those familiar with the judges of the court (see Rosen Reference Rosen2023). Indeed, the business model of the for-profit publication depends on its reputation for accuracy, incentivizing it to present valid and unbiased evaluations.

The surveyors identify lawyers through a variety of means, including official court records of appearances and third-party publications listing prominent lawyers in the district or circuit. In general, the lawyers interviewed have personally appeared multiple times over the previous few years before the judge in question. For judges who have served for several years, the surveyors interview eight-to-ten lawyers per survey. In general, all lawyer comments are published, often abridged for the most relevant and substantive language. Over a typical 10-year period, approximately 16 to 30 different lawyers give comments on any established judge. The Supplementary Material gives examples of typical evaluations.

2.2 Text-Analysis: Hierarchical Ngram Analysis

To analyze the corpus, I use a novel text-analysis method that I introduce here: hierarchical ngrams. The method has several advantages—particularly for this type of corpus and research objective—over large-language model machine-learning approaches, most notably, its transparency, explainability, and replicability (see, e.g., Albaugh et al. Reference Albaugh, Sevenans, Soroka and Loewen2013; Albaugh et al. Reference Albaugh, Soroka, Joly, Loewen, Sevenans and Walgrave2014; Grimmer and Stewart Reference Grimmer and Stewart2013). It also overcomes some shortcomings of conventional lexicon-based methods like unigrams or bigrams, such as their inability to draw meaning from syntactic context (cf. Farah and Kakisim Reference Farah and Kakisim2023). The hierarchical ngram method also performs comparably to large-language models in predicting case outcomes (but with the critical benefits of full transparency and replicability). See Section A2 of the Supplementary Material for analyses.

The hierarchical ngram method involves the following steps. First, every ngram from length 2 (bigram) to length 9 (novagram) in the corpus was identified. For the Circuit Ideology dataset, this initial process generated a set of 4,791 unique ngrams. For each of the eight ngram lengths, a threshold frequency was determined for inclusion in the coded dictionary, without regard to content or meaning. The threshold level was determined after examining the relative amount of information contained in each set of ngrams, balancing the conflicting objectives of the greatest possible context and maintaining a dictionary of manageable size.Footnote ⁴ For the Circuit Ideology dataset, the resulting dictionary comprises 2,176 unique ngrams.

Second, a set of trained coders—each upper-level J.D. students with backgrounds in federal courts—assigned a value, ranging from $-$ 3 (extremely liberal) to 3 (extremely conservative), or 99 (no ideological salience) to each ngram. For other datasets, appropriate alternative values are used. All ngrams were coded by three coders using the codebook contained in the Supplementary Material. Intercoder agreement was high;Footnote ⁵ I resolved discrepancies. A majority of ngrams were coded as non-salient, with 45.4% of ngrams assigned a value of $-$ 3 to 3. The [ $-$ 3,3] scale was then converted to a [ $-$ 1,1] scale, as with some existing ideology measures, such as Martin and Quinn (Reference Martin and Quinn2002) and JCS.

Next, the dictionary values were assigned to the judges in the corpus. As Figure 1 shows, a judge evaluation (j) comprises one or more comments (each by a single expert reviewer, on a given topic of a given judge in a given year). Each such comment comprises one or more ngrams (g) of length 2–9.

Figure 1 Structure of document components (evaluation-comment-ngram).

To assign the dictionary values to the corpus, I use the hierarchical ngram algorithm described below and in the Supplementary Material. Intuitively, for each comment, the algorithm identifies all ngrams in the comment that have been assigned a score in the dictionary. There are often multiple non-overlapping ngrams in a given comment that each receive a score. For example, in the following comment, <In employment cases, she’s not really pro-employee, but not really pro-business either; she’s usually right down the middle >, each of the three underlined ngrams would be scored.

Not all ngrams are counted, however, as many overlap with, or are nested within, other ngrams. For instance, the comment <she’s not a particularly harsh sentencer in drug cases> contains the ngrams <harsh sentencer>, <particularly harsh sentencer>, and <not a particularly harsh sentencer>. Each of these ngrams obviously has different meanings. In order to avoid counting and tallying redundant—or, worse, conflicting—scores of overlapping or nested ngrams, the algorithm recognizes a hierarchy of ngrams based on length: only the senior ngram, i.e., the longest in any set of nested ngrams is considered and scored.

(1) $$ \begin{align}i_c = \frac{ \sum_{g=1}^{G_c} i_g}{G_c} \end{align} $$

(2) $$ \begin{align} i_j = \frac{\sum_{c=1}^{C_j} i_c}{C_j} \end{align} $$

After the hierarchial process identifies the ngrams to be scored, a judge evaluation is calculated in the following way. First, the ideology of a given comment, $i_c$ , is defined by Equation (1), where a comment, c, comprises $G_c$ ngrams, and each ngram is assigned an ideology, $i_g$ . Thus, $i_c$ is the mean ideology score of the $G_c$ ngrams that make up comment c. In turn, each judge, j, for a given year or set of years has an observed ideology, $i_j$ , defined as in Equation (2), where the judge’s evaluation(s), j, comprises $C_j$ comments. Thus, $i_j$ is the mean ideology score of the $C_j$ comments that make up the judge evaluation(s).Footnote ⁶

To illustrate how this process can produce scores for particular judges, Figure 2 shows the distribution of comment scores for eight circuit judges: four Democratic-appointed (top row) and four Republican-appointed (bottom row). Each judge differs from the others in both mean ideology score and in distribution and variance. These histograms thus illustrate how the measure can be sensitive to small true variation between individual judges, even those appointed by the same president at about the same time.

Figure 2 Distribution of JuDJIS ideology comment scores for sample of eight circuit judges, aggregated over their tenures. Note: Vertical dashed lines denote the mean score. Gray bands denote 90% confidence intervals around the mean score.

For instance, while comments on Judge Stephen Reinhardt’s ideology were not uniform, the majority of comments placed him in the liberal range ( $-.5$ to $-.8$ ). This relative agreement, combined with the sheer number of comments amassed over his long career on the bench, contribute to fairly high confidence (and therefore, narrow confidence intervals, as indicated by the gray vertical band) about his mean score (indicated by the vertical dashed lines). In contrast, comments on Judge Richard Posner take a bimodal distribution, with most evaluators split between labeling him moderate ( $-.2$ to $.2$ ) and conservative ( $.5$ to $.8$ ). Perhaps this division occurs because Judge Posner is often characterized as more libertarian than conservative (Harcourt Reference Harcourt2007), meaning his somewhat left-leaning views on some social issues made it difficult to reach a consensus on his placement on a single-dimensional, left-right scale. The resulting uncertainty about his mean score is somewhat larger than that for Judge Reinhardt.

2.3 Advantages and Potential Critiques

Before empirically validating the method, I explore several of the method’s a priori advantages and potential critiques. Given the theory and method underlying the JuDJIS measures, they have several attractive properties and advantages over existing methods for measuring judicial ideology in the circuit and district courts.

First, the JuDJIS method can include the entire Article III judiciary on one scale, and it tracks changes over time, running from 1990 to the present. Moreover, by drawing on the experiences of legal experts who have studied their opinions and interacted with them in person over time, the JuDJIS scores capture the more subtle nuances of judging, albeit indirectly, beyond simple votes to affirm or reverse. In this one narrow sense, the technique is similar to Segal and Cover (Reference Segal and Cover1989)’s analysis of media op-eds, which, for nominees with judicial experience, draw on the judge’s previous opinions. But unlike Segal and Cover’s source material, which is locked in before the justice is even confirmed, the JuDJIS evaluations are updated continuously based on the judge’s conduct in that judicial position.

Finally, the JuDJIS scores are sensitive to small true variation between judges and to ideological evolution over time. Indeed, because the evaluators’ scores constitute a sample of a theoretical population of all potential evaluators, it can estimate standard errors and confidence intervals for each point estimate.Footnote ⁷

The JuDJIS method is also potentially subject to some critiques, which I attempt to anticipate and address here. First, the evaluations necessarily come from lawyers most familiar with the judges, as no single set of experts is substantially familiar with all (or even a meaningful fraction of) the federal judiciary. Thus, different mixes of evaluators evaluate the judges. So although I cannot definitively rule them out, there is no particular reason to expect systematic differences in evaluator characteristics, given the Almanac’s objectives and survey methodology. Indeed, it has this trait in common with other leading and established social science indicators.Footnote ⁸ Consider further that federal appellate lawyers are generally cosmopolitan legal actors, with many practicing in several circuits. Appellate lawyers tend to keep abreast of developments in other circuits and the Supreme Court. There is no requirement that the lawyers interviewed for the Almanac are geographically located within the circuit they evaluate—only that they have personal experience and expertise with the judge in question.

A related potential issue is the possibility of systemic conscious or unconscious bias against members of certain demographic groups, based on ethnicity, gender, or age, for example (Sen Reference Sen2014a, Reference Sen2014b). First, I note that this potential issue exists for other existing measures of ideology such as JCS and CBI, albeit in different ways.Footnote ⁹ Moreover, recent empirical evidence suggests that demographic-driven bias in evaluating judges is less of a concern than some may fear. In a series of conjoint experiments, Ono and Zilis (Reference Ono and Zilis2022) find that, when asked to evaluate the degree of bias they expect judges with different profiles to exhibit in immigration and abortion cases, the aggregated subject pool either does not distinguish between either male and female judges, or between Black and non-Black judges, or between Asian and non-Asian judges, or else expects that female judges, Black judges, and Asian judges, respectively, are less likely to be biased.Footnote ¹⁰ And to the extent people nonetheless make snap judgements when they have access only to a judge’s superficial characteristics, studies in social psychology (e.g., Pettigrew and Tropp Reference Pettigrew and Tropp2013) have repeatedly shown that bias is mitigated or eliminated as the evaluator interacts with the subject, as with the Almanac’s expert evaluators.

Finally, it is likely that a form of ideological drift occurs, in which what is considered, say, “moderate,” is different in 1990 and 2010. This challenge exists for essentially all attempts to measure ideology over time.Footnote ¹¹ For instance, the NOMINATE-dervied JCS scores are determined relative to the issues facing Congress, such that the meaning of a moderate judge depends on what constitutes centrist behavior in Congress during the given period. Likewise, the JuDJIS scores must be interpreted as estimating a judge’s position relative to the ideology norms of that period. Thus, in using these scores—as with other ideology measures—researchers should be cautious in interpreting changes over long periods. In constrast, over short periods such as within a given administration, change can be interpreted with a relatively high degree of confidence.

3.1 Data Comparison

First, Figure 3 is a matrix of pairwise scatterplots, comparing the JuDJIS Circuit Ideology scores (over each judge’s full tenure) with the scores for those same judges (where available) based on: the Party-of-the-Appointing-President (PAP); the JCS; and the CBI scores. The matrix also compares each of the scores with every other one. A least-squares regression of the y-axis scores on the x-axis scores is indicated by a gray line, with 90% confidence intervals indicated by the lighter gray band. The r values denote the correlation coefficients. The matrix diagonal (from top-left to bottom-right) of Figure 3 is a set of histograms indicating the respective distributions of each of the dataset values.

Figure 3 Matrix of pairwise scatterplots: JuDJIS vs. PAP vs. JCS vs. CBI measures.

Note: Observations denote judges (averaged over each judge’s Court of Appeals tenure) 1990–2024; red ‘R’ = Republican-appointed; blue ‘D’ = Democratic-appointed. Top-left r values indicate correlation coefficient of two score sets.

Reviewing the matrix of scatterplots and histograms, a few things are apparent. First, the datasets have notably different distributions. JCS is bimodal on a $-$ 1 to 1 scale, with the majority of scores falling either in the center-left range ( $-$ .6 to $-$ .4) or center-right range (0.4 to 0.6). This distribution is not surprising given JCS’s design, which assigns judges’ ideological values based primarily on Senate voting records; for all of the covered period, Congress has been highly polarized by party, albeit to different degrees (Barber et al. Reference Barber and McCarty2016; Poole and Rosenthal Reference Poole and Rosenthal1985, Reference Poole and Rosenthal2000). In contrast, both JuDJIS and CBI are unimodal, with CBI skewed right (many more liberal judges than conservative, and a long right tail). Given that lawyers as a group, and especially recent law grads (who are most like to serve as clerks), are more liberal than society generally (Bonica, Chilton, and Sen Reference Bonica, Chilton and Sen2016), this distribution is also unsurprising given CBI’s methodology. Finally, the distribution of JuDJIS data appears to be quite close to normal.

Turning to the scatterplots, JuDJIS is positively correlated with each of the three other ideology measures at statistically significant levels. The substantive correlations are each moderately high, with Pearson correlation coefficients of $r = 0.683$ (vs. PAP), $r = 0.639$ , (vs. JCS), and $r = 0.603$ (vs. CBI). Again, these moderate levels of correlation are unsurprising, given (as discussed in the Supplementary Material) the measures’ different implicit assumptions about the nature of ideology and the strategies for measuring it. Such moderate correlation levels imply that, while the four measures may all be attempting to measure the same general underlying concept, it is plausible that a study’s choice of ideology measure might sometimes affect the results (cf. Cope, Crabtree, and Fariss Reference Cope, Crabtree and Fariss2020).

Another trait the scatterplots reveal is the degree to which the different measures can distinguish between individual judges. To different degrees, both PAP and JCS place judges in noticeable silos, in which many judges share the same ideology score. That this would occur for PAP is self-evident, as there are only two parties that have nominated judges. JCS’s silos are far more numerous, and they group fewer judges together. But they result from a similar phenomenon: a given political actor’s ideology, sometimes that of the president, is attributed to all judges whose appointment for which the actor is responsible. Both CBI and JuDJIS feature very few judges with the same scores, in part because each is based on at least several-dozen individual campaign contributions (for CBI) or evaluations (for JuDJIS). As a result, these scores are more sensitive to small differences between any two judges’ latent ideologies.

Examining a few of the judges lying outside the diagonal also illustrates some key differences between JuDJIS and other methods. For instance, consider the two labeled judges in the JuDJIS-JCS scatterplot in Figure 3, the Fifth Circuit’s John Minor Wisdom (1957–99) and the Ninth Circuit’s Richard Tallman (2000–present). Judge Wisdom was an Eisenhower appointee; a liberal Southern Republican from New Orleans, Judge Wisdom was one of the “Fifth Circuit Four,” a group of judges who significantly expanded civil rights for African-Americans during the 1950s and 60s in the face of strong, sometimes violent, local White opposition (Grinstein Reference Grinstein2020). Taking senior status in 1977, Wisdom was considered among the most progressive judges in the country until his death in 1999. Drawing on the evaluations of the practicing bar, JuDJIS rates Wisdom among the 1% most liberal judges in the data set. JCS, considering the politics of his appointers, rates his as a moderate conservative. Conversely, 1999 Clinton-appointee Judge Richard Tallman was a Republican lawyer who was personally recommended by a prominent conservative Washington state supreme court justice/former Ninth Circuit nominee. Clinton agreed to nominate Tallman as part of a political deal in which Washington’s Republican senator agreed to unblock three of Clinton’s preferred nominees (Slotnick Reference Slotnick2006). JuDJIS, in part reflecting Tallman’s 69% conservative voting record in en banc cases, rates him a moderate conservative. JCS, based on the congressional voting record of Washington’s Democratic (Clinton’s party) senator, considers him a moderate liberal.

3.2 Predicting Case Outcomes

Although they produce several interesting insights, gauging validity by observing how well measures correlate takes us only so far. A better test of a measure’s value is how well it predicts behavior (Cope Reference Cope, Epstein, Grendstad, Šadl and Weinshall2024), that is, its predictive validity. I therefore proceed to determine how well the JuDJIS Circuit Ideology data predict case outcomes relative to how well the three major existing circuit-judge ideology measures, respectively, predict those outcomes. To do so, I draw on a new dataset of en banc decisions, comprising all such decisions during the relevant period (1990–2023), covering all numbered circuits and the D.C. Circuit. By way of background, a federal circuit court hears a small number of cases en banc, in which the whole court typically reviews a decision of a circuit panel. The cases tend to be contentious and are more likely to feature issues implicating traditional ideological cleavages. I validate the data using these cases because they span all circuits and relevant years, and they contain a greater proportion of “harder” cases, i.e., those in which a judge’s ideology is more likely to be salient. Note, however, that “hard” is not equivalent to “ideological” in the traditional, political sense. Not all of these cases involve legal issues that traditionally divide liberals and conservatives; many of the en banc courts split over issues with less traditional ideological salience, such as technical or procedural legal questions. On average, however, we should expect stronger relationships between ideology scores and case outcomes for these hard cases as a whole.

Each of the 414 decisions, including dissents and any concurrences, was read, and each judge’s vote was classified as either conservative or liberal. Using these data, I conduct three tests: (1) goodness of fit; (2) a logit regression, comparing the respective normalized correlation coefficients; and (3) a receiver operating characteristic (ROC) curve, comparing the respective measures’ areas under the curve.

Predictive Validation Using En Banc Votes: Logit Regression

I next estimate a logit model, regressing votes (whether the judge voted for a conservative outcome) on the predictor (the ideology score of the judge, as respectively estimated by JuDJIS, PAP, JCS, and CBI). Table 1 and Figure 4 present the results. The coefficients indicate the marginal effects of a two-standard-deviation increase in conservativeness in each respective ideology measure on the probability of a conservative vote. (The overall incidence of conservative votes in the full data set is approximately 55.9%) Thus, for example, a judge with a JuDJIS ideology score of 0.41 is about 45.3% percentage points more likely to cast a conservative vote than a judge with a JuDJIS ideology score of $-$ 0.14. Each of the four scores are associated with votes at highly significant levels (with JuDJIS’s being the most statistically significant). But a two standard-deviation change in JuDJIS score is associated with a substantially greater change in probability of a conservative vote than the equivalent change for the other three scores.

Table 1 Logit predictions: Marginal effects of judge score on probability of casting a conservative en banc vote.

Note: Coefficients are normalized to indicate the change in probability associated with a two standard-deviation change in the given score. p scores are in parentheses.

Figure 4 Probability of a conservative vote as a function of judge ideology score, by measure: en banc cases.

Figure 4 graphically illustrates this difference. The four graphs plot, for each vote, the judge’s ideology score on the x-axis. The judge’s votes are plotted on the y-axis, with conservative votes (1) at the top and liberal ones (0) at the bottom. (They are vertically jittered to show density.) For each graph, a logit regression curve shows the relationship between the two variables. Though the correlations are positive and significant in all four cases, the difference in the relationships’ magnitude is evident from the shapes of the respective s-curves.

Predictive Validation Using En Banc Votes: ROC Analysis

To further gauge predictive validity, I next estimate a logistic regression and plot a ROC curve for each of the four measures. Long common in the diagnostic medicine literature and now often used in political research (Imai and Khanna Reference Imai and Khanna2016; Mueller and Rauh Reference Mueller and Rauh2018), an ROC curve is an arguably more-intuitive method for assessing how well a metric accurately classifies observations into binary outcomes. In this case, the predicted variable is whether the judge casts a conservative vote, as described above. The predicting variable is the ideology score of the judge, as respectively estimated by JuDJIS, PAP, JCS, and CBI.Footnote ¹² For every judge ideology threshold, the ROC curve plots the true positive rate against the false positive rate (Fischman Reference Fischman2011; Wang Reference Wang2019). The area under the curve (AUC) therefore represents each measure’s relative success at predicting votes (see Hanley and McNeil Reference Hanley and McNeil1982). Specifically, it denotes the probability that the given measure will rank a randomly chosen conservative vote as conservative instead of liberal.

The top panel of Figure 5 displays the ROC curves. The bottom panel gives the AUC values for each of the four measures. JuDJIS achieves an AUC value of 0.736; PAP’s AUC value is 0.672; JCS’s AUC value is 0.675; and CBI’s AUC value is 0.645. A DeLong test indicates that PAP is statistically indistinguishable from JCS ( $p = 0.793$ ) and marginally significantly higher than CBI ( $p = 0.115$ ). But JuDJIS performs significantly better than all three: since JCS is about 17.5 percentage points above a random classification and JuDJIS is 23.6 percentage points above random, JuDJIS’s performance represents an improvement of 34.9% over JCS’s performance. A DeLong test indicates that the difference between the two is highly significant ( $p = 6.10 \times 10^{-12}$ ).

Figure 5 Top: ROC curves comparing four measures’ success at predicting en banc votes; Bottom: Comparative areas under the ROC curve.