Social scientists have quickly adopted large language models (LLMs) for their ability to annotate documents without supervised training, an ability known as zero-shot classification. However, due to their computational demands, cost, and often proprietary nature, these models are frequently at odds with open science standards. This article introduces the Political Domain Enhanced BERT-based Algorithm for Textual Entailment (DEBATE) language models: Foundation models for zero-shot, few-shot, and supervised classification of political documents. As zero-shot classifiers, the models are designed to be used for common, well-defined tasks, such as topic and opinion classification. When used in this context, the DEBATE models are not only as good as state-of-the-art LLMs at zero-shot classification, but are orders of magnitude more efficient and completely open source. We further demonstrate that the models are effective few-shot learners. With a simple random sample of 10–25 documents, they can outperform supervised classifiers trained on hundreds or thousands of documents and state-of-the-art generative models. Additionally, we release the PolNLI dataset used to train these models—a corpus of over 200,000 political documents with highly accurate labels across over 800 classification tasks.

In multiparty systems, parties signal conflict through communication, yet standard approaches to measuring partisan conflict in communication consider only the verbal dimension. We expand the study of partisan conflict to the nonverbal dimension by developing a measure of conflict signaling based on variation in a speaker’s expressed emotional arousal, as indicated by changes in vocal pitch. We demonstrate our approach using comprehensive audio data from parliamentary debates in Denmark spanning more than two decades. We find that arousal reflects prevailing patterns of partisan polarization and predicts subsequent legislative behavior. Moreover, we show that consistent with a strategic model of behavior, arousal tracks the electoral and policy incentives faced by legislators. All results persist when we account for the verbal content of speech. By documenting a novel dimension of elite communication of partisan conflict and providing evidence for the strategic use of nonverbal signals, our findings deepen our understanding of the nature of elite partisan communication.

Codebooks—documents that operationalize concepts and outline annotation procedures—are used almost universally by social scientists when coding political texts. To code these texts automatically, researchers are increasingly turning to generative large language models (LLMs). However, there is limited empirical evidence on whether “off-the-shelf” LLMs faithfully follow real-world codebook operationalizations and measure complex political constructs with sufficient accuracy. To address this, we gather and curate three real-world political science codebooks—covering protest events, political violence, and manifestos—along with their unstructured texts and human-coded labels. We also propose a five-stage framework for codebook-LLM measurement: Preparing a codebook for both humans and LLMs, testing LLMs’ basic capabilities on a codebook, evaluating zero-shot measurement accuracy (i.e., off-the-shelf performance), analyzing errors, and further (parameter-efficient) supervised training of LLMs. We provide an empirical demonstration of this framework using our three codebook datasets and several pre-trained 7–12 billion open-weight LLMs. We find current open-weight LLMs have limitations in following codebooks zero-shot, but that supervised instruction-tuning can substantially improve performance. Rather than suggesting the “best” LLM, our contribution lies in our codebook datasets, evaluation framework, and guidance for applied researchers who wish to implement their own codebook-LLM measurement projects.

The ability of news media to report on events and opinions that are critical of the executive branch of government is central to media freedom and a marker of meaningful democratization. Existing indices use scoring criteria or expert surveys to develop country year measures of media criticism. In this article, we introduce a computationally inexpensive and fully open-source method for estimating media criticism from news articles using à la carte (ALC) word embeddings. We validate our approach using Arabic-language news media published during the Arab Spring. An applied example demonstrates how our technique generates credible estimates of changes in media criticism after a democratic transition is ended by a military coup. Experiments demonstrate the method works even with sparse data. Analyses of synthetic news media demonstrate that the method extends to multiple languages. Our approach points to new possibilities in the monitoring of media freedom within authoritarian and democratizing settings.

We use instruction-tuned large language models (LLMs) like GPT-4, Llama 3, MiXtral, or Aya to position political texts within policy and ideological spaces. We ask an LLM where a tweet or a sentence of a political text stands on the focal dimension and take the average of the LLM responses to position political actors such as US Senators, or longer texts such as UK party manifestos or EU policy speeches given in 10 different languages. The correlations between the position estimates obtained with the best LLMs and benchmarks based on text coding by experts, crowdworkers, or roll call votes exceed .90. This approach is generally more accurate than the positions obtained with supervised classifiers trained on large amounts of research data. Using instruction-tuned LLMs to position texts in policy and ideological spaces is fast, cost-efficient, reliable, and reproducible (in the case of open LLMs) even if the texts are short and written in different languages. We conclude with cautionary notes about the need for empirical validation.

Past work on closed-ended survey responses demonstrates that inferring stable political attitudes requires separating signal from noise in “top of the head” answers to researchers’ questions. We outline a corresponding theory of the open-ended response, in which respondents make narrow, stand-in statements to convey more abstract, general attitudes. We then present a method designed to infer those attitudes. Our approach leverages co-variation with words used relatively frequently across respondents to infer what else they could have said without substantively changing what they meant—linking narrow themes to each other through associations with contextually prevalent words. This reflects the intuition that a respondent may use different specific statements at different points in time to convey similar meaning. We validate this approach using panel data in which respondents answer the same open-ended questions (concerning healthcare policy, most important problems, and evaluations of political parties) at multiple points in time, showing that our method’s output consistently exhibits higher within-subject correlations than hand-coding of narrow response categories, topic modeling, and large language model output. Finally, we show how large language models can be used to complement—but not, at present, substitute—our “implied word” method.

Word embeddings are now a vital resource for social science research. However, obtaining high-quality training data for non-English languages can be difficult, and fitting embeddings therein may be computationally expensive. In addition, social scientists typically want to make statistical comparisons and do hypothesis tests on embeddings, yet this is nontrivial with current approaches. We provide three new data resources designed to ameliorate the union of these issues: (1) a new version of fastText model embeddings, (2) a multilanguage “a la carte” (ALC) embedding version of the fastText model, and (3) a multilanguage ALC embedding version of the well-known GloVe model. All three are fit to Wikipedia corpora. These materials are aimed at “low-resource” settings where the analysts lack access to large corpora in their language of interest or to the computational resources required to produce high-quality vector representations. We make these resources available for 40 languages, along with a code pipeline for another 117 languages available from Wikipedia corpora. We extensively validate the materials via reconstruction tests and other proofs-of-concept. We also conduct human crowdworker tests for our embeddings for Arabic, French, (traditional Mandarin) Chinese, Japanese, Korean, Russian, and Spanish. Finally, we offer some advice to practitioners using our resources.

Scholars studying organizations often work with multiple datasets lacking shared identifiers or covariates. In such situations, researchers usually use approximate string (“fuzzy”) matching methods to combine datasets. String matching, although useful, faces fundamental challenges. Even where two strings appear similar to humans, fuzzy matching often struggles because it fails to adapt to the informativeness of the character combinations. In response, a number of machine learning methods have been developed to refine string matching. Yet, the effectiveness of these methods is limited by the size and diversity of training data. This paper introduces data from a prominent employment networking site (LinkedIn) as a massive training corpus to address these limitations. By leveraging information from the LinkedIn corpus regarding organizational name-to-name links, we incorporate trillions of name pair examples into various methods to enhance existing matching benchmarks and performance by explicitly maximizing match probabilities. We also show how relationships between organization names can be modeled using a network representation of the LinkedIn data. In illustrative merging tasks involving lobbying firms, we document improvements when using the LinkedIn corpus in matching calibration and make all data and methods open source.

Stance detection is identifying expressed beliefs in a document. While researchers widely use sentiment analysis for this, recent research demonstrates that sentiment and stance are distinct. This paper advances text analysis methods by precisely defining stance detection and outlining three approaches: supervised classification, natural language inference, and in-context learning. I discuss how document context and trade-offs between resources and workload should inform your methods. For all three approaches I provide guidance on application and validation techniques, as well as coding tutorials for implementation. Finally, I demonstrate how newer classification approaches can replicate supervised classifiers.

This article evaluates the claim that industrial policy is seeing a revival in developed economies, using text-as-data evidence from UK government policy papers. Structural topic modeling shows that content which can be related to industrial policy has indeed seen a large increase in prevalence over the past decade compared to the baseline of the post-1980 liberal era. Moreover, such content is shown to be increasingly central to post-2010 economic policy based on its position in the network of topics, on the number of downloads of documents associated with it, and on inclusion in important papers. An automated text summarization algorithm is used to extract the fragments which are most representative for these developments, and these are shown to closely match common definitions of industrial policy. A sentiment analysis algorithm is then used to extract the motivations given for policy proposals in representative documents, and indicates that declining economic competitiveness is a central concern.

How do racial stereotypes affect perceptions in foreign policy? Race and racism as topics have long been marginalized in the study of international relations but are receiving renewed attention. In this article we assess the role of implicit racial bias in internal, originally classified assessments by the US foreign policy bureaucracy during the Cold War. We use a combination of dictionary-based and supervised machine learning techniques to identify the presence of four racial tropes in a unique corpus of intelligence documents: almost 5,000 President's Daily Briefs given to Kennedy, Johnson, Nixon, and Ford. We argue and find that entries about countries that the US deemed “racialized Others”—specifically, countries in the Global South, newly independent states, and some specific regional groupings—feature an especially large number of racial tropes. Entries about foreign developments in these places are more likely to feature interpretations that infantilize, invoke animal-based analogies, or imply irrationality or belligerence. This association holds even when accounting for the presence of conflict, the regime type of the country being analyzed, the invocation of leaders, and the topics being discussed. The article makes two primary contributions. First, it adds to the revival of attention to race but gives special emphasis to implicit racialized thinking and its appearance in bureaucratic settings. Second, we show the promise of new tools for identifying racial and other forms of implicit bias in foreign policy texts.

A recent paper by Häffner et al. (2023, Political Analysis 31, 481–499) introduces an interpretable deep learning approach for domain-specific dictionary creation, where it is claimed that the dictionary-based approach outperforms finetuned language models in predictive accuracy while retaining interpretability. We show that the dictionary-based approach’s reported superiority over large language models, BERT specifically, is due to the fact that most of the parameters in the language models are excluded from finetuning. In this letter, we first discuss the architecture of BERT models, then explain the limitations of finetuning only the top classification layer, and lastly we report results where finetuned language models outperform the newly proposed dictionary-based approach by 27% in terms of $R^2$ and 46% in terms of mean squared error once we allow these parameters to learn during finetuning. Researchers interested in large language models, text classification, and text regression should find our results useful. Our code and data are publicly available.

Supervised machine learning is an increasingly popular tool for analyzing large political text corpora. The main disadvantage of supervised machine learning is the need for thousands of manually annotated training data points. This issue is particularly important in the social sciences where most new research questions require new training data for a new task tailored to the specific research question. This paper analyses how deep transfer learning can help address this challenge by accumulating “prior knowledge” in language models. Models like BERT can learn statistical language patterns through pre-training (“language knowledge”), and reliance on task-specific data can be reduced by training on universal tasks like natural language inference (NLI; “task knowledge”). We demonstrate the benefits of transfer learning on a wide range of eight tasks. Across these eight tasks, our BERT-NLI model fine-tuned on 100 to 2,500 texts performs on average 10.7 to 18.3 percentage points better than classical models without transfer learning. Our study indicates that BERT-NLI fine-tuned on 500 texts achieves similar performance as classical models trained on around 5,000 texts. Moreover, we show that transfer learning works particularly well on imbalanced data. We conclude by discussing limitations of transfer learning and by outlining new opportunities for political science research.

Research on political style suggests that where women make arguments that are more emotional, empathetic and positive, men use language that is more analytical, aggressive and complex. However, existing work does not consider how gendered patterns of style vary over time. Focusing on the UK, we argue that pressures for female politicians to conform to stereotypically ‘feminine’ styles have diminished in recent years. To test this argument, we describe novel quantitative text-analysis approaches for measuring a diverse set of styles at scale in political speech data. Analysing UK parliamentary debates between 1997 and 2019, we show that the debating styles of female MPs have changed substantially over time, as women in Parliament have increasingly adopted stylistic traits that are typically associated with ‘masculine’ stereotypes of communication. Our findings imply that prominent gender-based stereotypes of politicians' behaviour are significantly worse descriptors of empirical reality now than they were in the past.

Most analyses dealing with the interaction of parties in parliament assume their interests to be fixed between elections. However, a rational perspective suggests that parties adapt their behaviour throughout the legislative term. I argue that this change is influenced by incentives and possibilities to shape legislation and the need to distinguish oneself from competitors. While for government parties it matters whether they have to share offices, for opposition parties the influence on policy-making is important. By examining the sentiment of all parliamentary speeches on bill proposals from six established democracies over more than twenty years, I analyse institutional and contextual effects. The results show that single-party governments tend to become more positive towards the end of the legislative cycle compared to coalition governments. On the other hand, opposition parties under minority governments, or with more institutionalised influence on government bills, show a more negative trend in comparison to their counterparts.

Topic models, as developed in computer science, are effective tools for exploring and summarizing large document collections. When applied in social science research, however, they are commonly used for measurement, a task that requires careful validation to ensure that the model outputs actually capture the desired concept of interest. In this paper, we review current practices for topic validation in the field and show that extensive model validation is increasingly rare, or at least not systematically reported in papers and appendices. To supplement current practices, we refine an existing crowd-sourcing method by Chang and coauthors for validating topic quality and go on to create new procedures for validating conceptual labels provided by the researcher. We illustrate our method with an analysis of Facebook posts by U.S. Senators and provide software and guidance for researchers wishing to validate their own topic models. While tailored, case-specific validation exercises will always be best, we aim to improve standard practices by providing a general-purpose tool to validate topics as measures.

New forms of work intermediation – the gig economy – and the growing use of advanced digital technologies – the new knowledge economy – are changing the nature of work. The digitalization of work, however, is shaped by how countries respond to it. But how countries respond to digitalization, we argue, depends on how digitalization is perceived in the first place. Using text-as-data methods on a novel corpus of translated newspaper and policy documents from eight European countries as well as qualitative evidence from interviews and secondary sources, we show that there are clear country effects in how digitalization is framed and fought over. Drawing on discursive–institutionalist and coalitional approaches, we argue that institutional differences explain these discursive differences by structuring interpretative struggles in favor of the social coalitions that support them. Actors, however, can also challenge these institutions by using the discursive agency to change these underlying support coalitions.

How do extremist sympathizers respond to counter-radicalization efforts? Over the past decade, programs to counter violent extremism have mushroomed around the world, but little is known of their effectiveness. This study uses social media data to examine how counter-radicalization efforts shape engagement with extremist groups in the online world. Matching geolocated Twitter data on Islamic State sympathizers with granular information on counter-extremism activities in the United States, I find that, rather than deradicalizing, these efforts led Islamic State sympathizers to act strategically to avoid detection. After counter-extremism activities, the group's supporters on Twitter who were in the vicinity of these events began self-censoring expressions of support for the Islamic State, altered profile images and screen names, and encouraged followers to migrate to Telegram, an encrypted network not viewable by the public. These findings reveal previously unknown patterns in the effects of counter-extremism programs in the digital era.

We consider evidence for the assertion that backbench members of parliament (MPs) in the UK have become less distinctive from one another in terms of their speech. Noting that this claim has considerable normative and substantive implications, we review theory and findings in the area, which are ultimately ambiguous on this question. We then provide a new statistical model of distinctiveness that extends traditional efforts to statistically characterize the “style” of authors and apply it to a corpus of Hansard speeches from 1935 to 2018. In the aggregate, we find no evidence for the claim of more homogeneity. But this hides intriguing covariate effects: at the MP-level, panel regression results demonstrate that on average, more senior backbenchers tend to be less “different” in speech terms. We also show, however, that this pattern is changing: in recent times, it is more experienced MPs who speak most distinctively.