1.1.1 Word Embedding Models and Neural Language Models

Naturally occurring texts are a rich source of data. The task of language models (LMs) to predict continuation of a textual sequence can be also thought of as the task of classifying contexts (sequences of words) according to which word(s) can serve as a likely continuation of the context. In this case, the number of classes is huge as each vocabulary item is its own class.

For example, one can take as input a single word context (e.g., scientific), encode it as a vector, and use the model to predict likelihood scores for different words to appear in the context of scientific. Such a model will assign high scores to words like approach or major, and low scores to words that are not particularly likely to appear near scientific. In practice, a model represents each context (e.g., the word scientific) with a relatively low dimensional vector v from which a vector of scores for all vocabulary items is derived via matrix multiplication $M^{WE}v$, where the word embedding matrix $M^{WE}$ contains compact vectors of words in the vocabulary. There are good reasons to use vectors of relatively low dimensionality. First, they are more practical in computation, including various vector operations used in neural network models. Second, features of lower dimensional vectors may better approximate abstract features of words, including features corresponding to semantic properties. Because of this, inputs with similarities in meaning or syntactic properties end up with substantial overlap in their vector features.

Systems that predict likely words in this way are known as word embedding models. They operate with individual word inputs, as in the example just provided.

Early methods that estimated word vectors from corpus data included the Hyperspace Analog of Language (Burgess & Lund, Reference Burgess and Lund1995) and Latent Semantic Analysis, also known as LSA (Landauer & Dumais, Reference Landauer and Dumais1997). The advent of neural network methods in the 2010s led to the creation of several efficient algorithms for word vector estimation, which were released as word2vec (Mikolov et al., Reference Mikolov, Chen, Corrado and Dean2013) and GloVe (Pennington et al., Reference Pennington, Socher and Manning2014). Similar to word2vec, fastText (Bojanowski et al., Reference Bojanowski, Grave, Joulin and Mikolov2017) extended its coverage to rare and unseen words by exploiting cues from the character sequences within the word. These algorithms proved robust and fared better in empirical evaluations than earlier methods (Baroni, Dinu, & Kruszewski, Reference Baroni, Dinu and Kruszewski2014).

In contrast to (static) word embedding models, neural LMs predict the probability distribution over the next word (or other text elements) given a sequence of other elements in context – for example, the sequence Let’s use the scientific $\dots$ or The cat is sitting $\dots$). For the latter sequence, the next word prediction may look as in Table 1. As seen in this example, the sequence is predicted to be continued with a dot or “on,” with prepositions like “under” or “by” predicted less likely, and many other words having negligible predicted probabilities (rounded to 0 in Table 1).

The Sequence Neural Models: From Recurrent Networks to Transformers

Often, vector operations proceed via multiple computation steps – that is, output vector v is computed from vector u that is itself computed from the input vector x. The intermediate computation steps are called the hidden layers of the model, and a model that includes hidden layers is considered a deep neural network. Machine learning methods that create such models are known as deep learning.

For most purposes, assigning vectors to words is not enough if the goal is to process diverse kinds of structures, such as phrases, sentences, or longer texts. This motivates several types of sequence models, which can adapt to inputs of variable length. In all sequence models, the input is a sequence of vectors representing text units – for example, words, and the output is a sequence of calculated vectors:

$x_1,x_2\dots x_n⟶h_1,h_2\dots h_n.$(1)

The vector representations $h_1,h_2\dots h_n$ that a sequence model derives can then be used for diverse tasks such as sequence classification, tagging (token classification), etcetera.

The oldest type of sequence neural network, inspired by real-time signal processing in humans, is the recurrent neural network (RNN). Recurrent neural networks process the input one element at a time, computing the memory representation $h_k$ from $h_{k-1}$ and the $k$ th input element $x_k$. Simple recurrent networks (SRNs), proposed by Elman (Reference Elman1990), already showed promising results on toy linguistic input, but presented diverse problems at training time. More efficient recurrent architectures were proposed later, with two gaining wide adoption: the Long Short-Term Memory, or LSTM (Hochreiter & Schmidhuber, Reference Hochreiter and Schmidhuber1997), and the Gated Recurrent Unit, or GRU (Cho et al., Reference Cho, Van Merriënboer and Gulcehre2014).

Most current applications, however, rely on the Transformer model (Vaswani et al., Reference Vaswani, Shazeer and Parmar2017); see Section 1.1.2. In addition to other practical benefits, the self-attention mechanism underlying Transformer models makes it easier to learn and execute nonlocal operations on the sequence.

Regardless of the precise underlying architecture, sequence models can be used to produce contextualized token vectors. If $x_k$ is an input word embedding, corresponding $h_k$ in the output represents the k th word in context. One can also select one of the output vectors, often the last one $h_n$, to represent the whole sequence. For example, in a task like natural language inference, the vector resulting from processing the concatenation of the premise and the hypothesis can serve to provide features labeling the example as entailment / contradiction / neutral; see Section 2.

1.1.2 Transformer Architecture

Self-Attention

The attention mechanism originally gained wide acceptance in text processing in the field of machine translation as a useful addition to RNNs, starting from Bahdanau, Cho, and Bengio (Reference Bahdanau, Cho and Bengio2014). Later, Vaswani et al. (Reference Vaswani, Shazeer and Parmar2017) introduced self-attention as the core mechanism for sequence processing that completely replaced RNNs. Self-attention allows for efficient training of ever-larger models on ever-larger data that was not technically possible with RNNs.

We reproduce the equation of self-attention here:

$Attention(Q,K,V)=softmax\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V,$(2)

where Q, K, and V are computed from the underlying sequence embedding M by multiplying it with matrices of numeric parameters: $Q=M{W}^Q,K=M{W}^K,$ $V=M{W}^V$. The softmax function normalizes the scores that reflect the match between “query” vectors in Q and the “key” vectors in K, so that the weights for each position are positive and sum to 1; see the following example. Several self-attention components, called “attention heads,” are computed in parallel and combined into “multihead attention” Multihead, which is then combined with the input embedding matrix M via the residual connection, giving M+Multihead(M) as output.

Informally, self-attention copies to a given token vector $m_k$ information from token vectors at other positions. The tokens to be copied from are determined by the match between the query vector of token $m_k$ from matrix Q, and the key vectors of all tokens from matrix K.

For example, take input M to encode text as a sequence of numeric vectors:

$\begin{array}{cccc}-7& -1& -10& -7\\ -1& -4& 5& 2\\ -6& 6& 2& 7\\ \text{the}& \text{cat}& \text{is}& \text{sitting}\end{array}$

Vectors encoding tokens can be similar to each other across some or all dimensions. The vectors for is and sitting are the most similar. For example, the sign of all their vector components is the same: negative for the first dimension and positive for the others. Numbers in token vectors can encode semantic and syntactic information – for example, the second dimension could partly encode part of speech, with positive values for verbs.

A match between queries and keys is computed as $Q{K}^T$, which after applying vector size and softmax normalization gives an attention matrix – for example:

$\begin{array}{cccc}0& 0& 0& 0\\ 0.99& 0.01& 0.86& 0\\ 0.01& 0.19& 0.14& 1\\ 0& 0.8& 0& 0\end{array}$

The attention matrix specifies how much update each token’s vector receives from different tokens. In our example, the last token sitting gets its entire update from token is, while token the receives 89 percent of the update from cat.

The values of the updates are taken from a separate matrix V – for example:

$\begin{array}{cccc}-8.7& 2.4& 2.3& 5.8\\ -5.5& 1.1& -2.6& 0.01\\ -8& -3.6& -3& -2.5\\ \text{the}& \text{cat}& \text{is}& \text{sitting}\end{array}$

The attention update $\begin{array}{cccc}2.3& 5.1& 2.3& 2.3\\ 1.1& -0.49& 0.59& -2.6\\ -3.6& -2.6& -3.6& -3\\ \text{the}& \text{cat}& \text{is}& \text{sitting}\end{array}$ is added to the input.

After self-attention, the vector representations become more contextualized. In our toy example, both the and is received most of their attention update from cat. As a result, the vectors of the and is now signal aspects of their relation to the word cat. Being updated with similar information, these tokens also become more similar to each other:

$\begin{array}{cccc}-4.7& 4.1& -7.7& -4.7\\ 0.1& -4.5& 5.6& -0.6\\ -9.7& 3.4& -1.6& 4\\ \text{the}& \text{cat}& \text{is}& \text{sitting}\end{array}$

There are several other kinds of computation steps in Transformers, but self-attention is central. Informally, self-attention allows for information flow between positions in the sequence by selecting which positions to copy information from (via K to Q matching) and what form this information takes (via the V matrix). Roughly speaking, self-attention is the operation of selecting positions according to features in K and copying features from V. Components of self-attention specify the source, target, and nature of the copied information. For instance, Transformers could naturally approximate rules like “copy into the vector of a verb (encoded in Q) ontological semantic features (V) from the closest noun to the left (K).”

1.1.3 Neural Model Training

Deep neural network models include a large number of numeric parameters that need to be estimated, or learned, from data. In the case of Transformer LMs, these trainable parameters include numeric values in vectors of all tokens in the vocabulary and in matrices that define the model’s self-attention and feedforward operations.

Ultimately, large language models (LLMs) are evaluated on downstream tasks. For example, the natural language inference task often boils down to classifying sentence pairs as exemplifying an entailment, a contradiction, or neither.

Pretraining and Fine-Tuning

In modern deep learning models, a common approach taken in achieving state-of-the-art results in specific tasks combines self-supervised pretraining with task-specific fine-tuning, illustrated in Figure 1.

Figure 1 Two-step training paradigm: in the pretraining stage, the model is trained on large text corpora on a word prediction task. Task-specific training (fine-tuning) happens separately, with the pretrained model as a starting point.

Typically, an LLM is pretrained on a distributional task, meaning that its output representations are optimized for predicting a match between the context and the textual element that can appear in it. For instance, the vector representation of a sentence can be trained to predict what continuations the sentence is likely to have. In GPT-like models (Radford et al., Reference Radford, Wu and Child2019), the training signal comes from predicting the next token in the context of the preceding sequence of tokens. In other models (like BERT; Devlin et al., Reference Devlin, Chang, Lee, Toutanova, Burstein, Doran and Solorio2019), token prediction happens in a bidirectional context (sentence with gaps), with tokens to be predicted typically replaced by a dedicated [MASK] token. As such, the vector representation that is useful for token prediction cannot be immediately applied to alternative tasks involving reasoning, question answering, or sentiment analysis, inviting additional approaches including fine-tuning; see, however, Radford et al. (Reference Radford, Wu and Child2019) and Brown et al. (Reference Brown, Mann, Ryder, Larochelle, Ranzato and Hadsell2020) for influential views on the transfer of pretrained LMs to new tasks without such computationally expensive steps.

Vector outputs of a pretrained model can serve as input to a simpler neural component such as a feedforward neural network. The latter makes the actual task-specific predictions, such as whether one sentence in a pair entails the other. The whole pipeline can then be trained on the task-specific data (e.g., inference data), updating both the feedforward network’s weights and the weights of the pretrained LM. The resulting fine-tuned LM differs from the original pretrained one, and produces task-specific vector representations of the input. Note that fine-tuning only produces reasonable empirical results when applied to a pretrained model, rather than learning the weights from scratch on task-specific data. One can think of the process of fine-tuning as highlighting the features of compositional representations produced by the pretrained model that are relevant for the specific task at hand, and suppressing irrelevant features. The intuition here is that the distributional pretraining allows the model to extract a wide set of features from the text, different subsets of which are useful for different downstream tasks. Features useful for one task (e.g., inference) may happen to be complementary to the features useful for another task (e.g., sentiment analysis), so instances of the same model fine-tuned on these tasks may prove quite distinct. Note that fine-tuning mainly affects representations at the top layers of deep models while the bulk of processing that happens in a majority of layers remains largely intact (Mickus et al., Reference Mickus, Paperno and Constant2022).

Both pretraining and fine-tuning follow the end-to-end training approach: model parameters (weights) are not estimated for each module separately. Instead, even in the biggest models, all parameters are tuned in parallel, with an eye on how they affect the output in a given task. In pretraining, the output is the likelihood score assigned to the currently predicted token, and in fine-tuning, to the currently predicted other output such as the likelihood score of different classes in a classification task.

Fine-tuning a neural network on a dataset may lead to a loss of its generality. There is a risk that a model adapts to biases of the data on which it was fine-tuned, learning shallow statistical regularities. For example, the presence of the word not may be associated in a fine-tuning dataset with the example being a contradiction. A system fine-tuned on such a dataset may learn the shallow heuristic not$\Rightarrow$ contradiction and fail to apply correctly to data from other sources where the heuristic is not helpful. For more discussion, see Section 2.

Methods for adjusting model parameters in neural networks rely on gradient descent. In simple terms, this means that each numeric parameter of the deep neural network is updated proportionally to the degree to which its change moves the model’s prediction toward the desired output. Measures of discrepancy between the prediction and the desired outputs are known as loss functions.

Beyond Attention: Chain of Thought

Under the chain of thought, the model is prompted to produce the text of intermediate steps through which one can arrive at the output, which can help with recursion.

The following example taken from the Google AI blog illustrates the chain-of-thought prompting at work:Footnote ⁴

Example input

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?

Example output

Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5+6 = 11. The answer is 11.

In the example, only the last sentence of the output constitutes the answer to the question. The rest of the output only helps the model to arrive at the final answer. Indeed, chain-of-thought prompting improves the few-shot learning of Transformer models on tasks that involve multiple reasoning steps.

With this technical background in mind, we can turn to the second main ingredient of our survey: natural language semantics.

Truth Conditions or Inference?

Which of the two notions – truth conditions or semantic inference – is taken as basic with respect to the other one defines the two views on natural language semantics that, in turn, provide two different perspectives on semantics in deep learning models. Let us zoom in on this a bit.

The currently most widely adopted version of compositional formal semantics builds on truth conditions – a view that can be traced back to the philosophical tradition that includes Alfred Tarski, Rudolf Carnap, Donald Davidson, David Lewis, and Richard Montague. As famously formulated by David Lewis (Reference Lewis1970), “Semantics with no treatment of truth conditions is not semantics.” Under this truth-conditional view – we will call it referential to contrast with its alternative – sentence-level semantics amounts to an association between sentences and sets of situations that make them true. Semantic inference relations within sentence pairs would then be mediated by sets of situations they describe: the relation holds by virtue of a set-theoretic relation between sentence meanings.

Objects that sentences are mapped to can have different specifics – they can be situations, worlds, circumstances, models, cases, etcetera, depending on the implementation. There is also variation among systems in whether the mapping between sentences and objects that express truth conditions is direct (Kratzer & Heim, Reference Kratzer and Heim1998; Montague, Reference Montague1970) or indirect via a representation language, typically some logic (Coppock & Champollion, Reference Coppock and Champollion2022; Montague, Reference Montague1973). The core of the referential view on semantics would remain the same: Meaning is defined by reference, understood as a mapping between linguistic objects on something external to language itself.

Alternatively, semantic inference relations (including but not limited to entailment) can be taken as basic – defined directly on sentence representations, without referencing the situations or worlds. We will call the view that builds on semantic relations the inferential view (Fitch, Reference Fitch1973; Lakoff, Reference Lakoff1970; Moss, Reference Moss2010, Reference Moss2015; Murzi & Steinberger, Reference Murzi and Steinberger2017; Schroeder-Heister, Reference Schroeder-Heister2018; Sommers, Reference Sommers1982; Van Benthem, Reference Van Benthem1986, Reference Van Benthem2008). This description groups together theories that have very important differences with each other, but, crucially for our discussion, they all capitalize on semantic relations between linguistic expressions (primarily sentences) as the core semantic notion.

The guiding observation for this view is that, given that people reason using language, the logical structures underlying human reasoning should correspond to the grammatical structure of natural language in a deep way. If these regularities are given central stage in analyzing meaning, reference and truth conditions can be explained as their by-product. This program can be summed up in two theses:

1. 1. Meanings of linguistic expressions are determined by their role in inference.

2. 2. To understand a linguistic expression is to know its role in inference.

The difference between the referential and inferential views is deep, but at the same time it carries mostly metasemantic value: it is a difference in the order of explanation and departing points such as formal and traditional logics. Radical versions of these views can also be seen as endpoints on the scale of importance of corresponding intuitions for semantics – those of truth conditions and those of inference. In practice, the views of most semanticists probably lie somewhere in between: grounding in nonlinguistic information has doubtless potential to enrich linguistic meanings; on the other hand, at least for some semantic phenomena, it is useful to directly examine semantic relations between expressions.

The importance of the referential/inferential distinction in the context of deep learning has to do with the fact that most of the deep learning models we will discuss are trained on exclusively textual data. This means that the representations these models develop are not referentially grounded to anything external to linguistic data itself (see, however, Section 4 on vision-and-language models).

The text-only training setup has stirred a debate around the semantic properties of LM representations. Do models trained on exclusively textual data develop representations that encode the full range of semantic information? Can tasks formulated as text-only be informative and useful for enhancing and/or probing models’ semantic capabilities? We will now give an overview of this debate.

Grounding Argument against Semantics in Text-Only Models

Language is inherently grounded in a variety of extralinguistic experiences (Barsalou, Reference Barsalou2008; Clark, Reference Clark1996; Harnad, Reference Harnad1990; Meteyard et al., Reference Meteyard, Cuadrado, Bahrami and Vigliocco2012; Parikh, Reference Parikh2001). Linguistic communication essentially involves a connection between what we say and what we mean, naturally implemented as a mapping between two separate spaces – the “what we say” and the “what we mean,” respectively. The expression the smell of coffee, for example, describes a corresponding nonlinguistic olfactory experience. Can an agent that has not been exposed to the “what we mean” side of messages develop an understanding of what any message means?

The architecture of a lot of widely used computational models for language does not involve explicit mapping between text and “states of affairs” (although see Radford et al. Reference Radford, Kim and Hallacy2021 and Section 4); they are usually not trained with the objective of mapping between object language and such model-theoretic space. This has led many to conclude that such models do not encode semantics at all – a conclusion that seems practically unavoidable under a referentialist truth-conditional view on semantics.

An influential position piece elaborating on this argument is Bender and Koller (Reference Bender and Koller2020), even though it might be a stretch to classify their position as strictly referentialist (their “what is meant” includes things like communicative intent, which is not really model-theoretic). In their own words, they “argue that the language modeling task, because it only uses form as training data, cannot in principle lead to learning of meaning.” Since the language modeling task is that of string prediction, the “meanings” – whatever they are – are not in the training signal. Bender and Koller conclude that, for this reason, meanings cannot be learned as the result of this process, since LMs are not provided the means of solving the “symbol grounding problem” (Harnad, Reference Harnad1990) – that is, they have no means to connect text representations to the world these texts are used to communicate about.

To illustrate this position, Bender and Koller introduce a thought experiment that they call the octopus test, largely inspired by the Turing test for artificial intelligence (Turing, Reference Turing2009). In the scenario, two people are stranded on two islands and are communicating via telegraph using an underwater cable. Meanwhile, an intelligent octopus underwater is eavesdropping on their conversations and, being extremely good at detecting statistical patterns, learns to predict the two people’s replies to each other. Eventually, the octopus inserts itself into the conversation, successfully pretending to be one of the people. But when facing a situation that requires real-world knowledge of what a coconut is, the octopus fails since it knows nothing about the referent of the word.

Bender and Koller (Reference Bender and Koller2020) conclude that statistical patterns of co-occurrence cannot be enough to develop knowledge of meaning.Footnote ⁵ In the discussion that followed, other researchers cast doubt on this conclusion. Let us now review their arguments in favor of semantics without grounding (Merrill, Warstadt, & Linzen, Reference Potts2020; Piantadosi & Hill; Reference Piantadosi and Hill2022; Potts, Reference Merrill, Warstadt and Linzen2022).

Meaning without Grounding?

Consider a counterargument: It is one thing for a semantic theory to predict that text-based models should be unable to encode semantic information – but it is up to the actual behavior of these models to either support this or suggest otherwise.

Following Potts (Reference Potts2020), let us shift our focus from a priori conclusions to a more practical reformulation: “Is it possible for language models to achieve truly robust and general capabilities to answer questions, reason with language, and translate between languages?” In this way, the extent to which the models can do so defines the extent to which they encode semantics (and therefore, have the capacity to achieve natural language understanding), regardless of the training data and objective.

There are at least two reasons for optimism. First, the general ability of deep learning models to acquire abstract information not explicitly given during training has been shown on, for example, hierarchical syntactic structure; see for instance the survey in Linzen and Baroni (Reference Linzen and Baroni2021). Second, empirically, we do not really know which types of input are necessary for humans to learn meanings and manipulate them. Visual grounding is clearly not necessary as congenitally blind people still acquire language (Landau & Gleitman, Reference Landau and Gleitman1985); the same holds for smell (returning to the example with the smell of coffee), and so on. This does not mean that human semantic knowledge does not have a grounding component to it at all, but the extent to which human semantic representations can be constructed in the absence of different types of grounding suggests that the same can in principle hold of artificial learners.

Piantadosi and Hill (Reference Piantadosi and Hill2022) address the same question from the perspective of conceptual role theory – a view on cognition in many ways close to the inferentialist semantic paradigm (see Margolis and Laurence Reference Margolis and Laurence1999 for an overview of conceptual role theory and its alternatives). While acknowledging that the string-prediction training setup differs in format from human language acquisition, they review arguments suggesting that the meaning of a significant fraction of natural language expressions is primarily determined by the role they play in a larger mental theory rather than their reference.

Studies in language acquisition show some support for this idea: learning the meanings of various classes of words relies heavily on structural linguistic information (Gleitman Reference Gleitman1990; Gleitman et al., Reference Gleitman, Cassidy, Nappa, Papafragou and Trueswell2005; Landau and Gleitman Reference Landau and Gleitman1985). This is particularly true of expressions for concepts without observable correlates such as, for instance, verbs like think or believe (see Hacquard and Lidz Reference Hacquard and Lidz2022 for a review).

Taking this view to its extreme, a system relying on relations within one modality is not necessarily meaningless, with additional modalities providing various enrichments. Reference or grounding then adds to the “conceptual role” the word plays. The signal that the learner gets from text alone is already quite rich in conceptual role information, explicit and implicit. The task of the learner is to invert from observations to mechanisms that generate these data (see Merrill et al. Reference Merrill, Warstadt and Linzen2022 for an estimation of how entailment could be learned by a text-only model under some simplifying assumptions).

This perspective, again, gives a practical turn to the question of semantics in text-only LMs: In order to know whether LMs learn to represent semantics during training and what it looks like, one has to examine the models’ internal representations and how they relate to each other.

This practical angle motivates our Element: We will take a closer look at how these models perform on semantic tasks and examine the semantic properties of their internal representations.

The result can sometimes be disappointing: Despite the often-reported impressive performance of current deep learning models, upon closer investigation, it often turns out to be mere pattern memorization or bias propagation – and the sometimes “superhuman” scores on such tasks go down dramatically when the benchmark datasets are manipulated in a relevant way. At the same time, studies of LM representations reveal rich semantic structures such as color space geometry (Abdou et al., Reference Abdou, Kulmizev and Hershcovich2021) or the relative geographical positions of major cities (Gatti et al., Reference Gatti, Marelli, Vecchi and Rinaldi2022); some work shows indications that contextual representations of the latest text-only LMs implicitly encode models of entities and situations evolving as text progresses (Li, Nye, & Andreas Reference Li, Nye and Andreas2021; but see Kim and Schuster Reference Kim and Schuster2023 for a critique of these results).

In this overview, we would like to give the reader a balanced picture of challenges and successes in this domain and suggest possible future directions.

We are now moving on to the main part of the Element. We went over both technical and theoretical background for the upcoming discussion. We introduced vector representations for words and larger linguistic sequences and discussed how such representations are usually obtained from deep neural network models, often ones based on the Transformer architecture (Devlin et al., Reference Devlin, Chang, Lee, Toutanova, Burstein, Doran and Solorio2019; Radford et al., Reference Radford, Wu and Child2019). We introduced the main notions and intuitions in theoretical semantics: truth conditions and semantic inference. Finally, we highlighted the tension between the text-only setup common in deep learning language modeling and the architecture of most common theoretical semantics frameworks that involve a separate interpretation space. This tension is the driving point for the rest of the discussion.

We will start the main part with an overview of reasoning and inference in deep learning models (Section 2), then we turn to compositionality (Section 3) and language grounding (Section 4).

2.1.1 Collecting Text–Hypothesis Pairs

Many textual inference datasets are created in two major steps: first, collecting Text–Hypothesis pairs, and second, annotating them with inference labels. The methods of collecting Text–Hypothesis pairs can roughly be divided into human-elicited, semiautomated, and fully automated methods.

The human-elicited method involves human annotators in creating an inference problem – for example, creating an entirely new problem, pairing existing sentences, or providing a Hypothesis given a Text or vice versa. Initially, inference problems in the series of RTE challenges were human-elicited by expert annotators and the organizers of the challenges. Due to the involvement of experts, the collection process was expensive and each iteration of the challenge prepared only 1,000 to 1,600 new inference problems. A step forward in the human-elicited collection came from Bowman et al. (Reference Bowman, Angeli, Potts and Manning2015), who created the Stanford NLI (SNLI) dataset, a collection of circa 570,000 sentence pairs. Hypotheses were written by crowd workers given a premise sentence and a target inference label. The size of SNLI has triggered a surge of deep learning models for textual inference. A collection protocol similar to SNLI was used to create another large inference dataset, multi-genre NLI (MNLI et al., Reference Williams, Nangia and Bowman2018).

Semiautomated collection methods partially automatize the generation of sentences or automatically transform existing sentences. Manual work usually involves verification of Text–Hypothesis pairs on fluency or carrying out certain tasks that are difficult to reliably automatize. Marelli et al. (Reference Marelli, Menini and Baroni2014) were the first to semi automatically collect about 10,000 sentence pairs for the Sentences Involving Compositional Knowledge (SICK) dataset.

There are three main groups of approaches when collecting inference pairs with a fully automated method. The first approach takes advantage of already existing textual inference datasets and automatically modifies the problems. For example, Naik et al. (Reference Naik, Ravichander and Sadeh2018) modify MNLI data to create a stress test on spelling errors and various distractions (e.g., a high word overlap and length mismatch between a premise and a hypothesis). The second approach, as demonstrated by White et al. (Reference White, Rastogi, Duh and Van Durme2017), recasts datasets for other NLP tasks as inference datasets. The third approach automatically generates Text–Hypothesis pairs. This is usually done with the help of manually predesigned templates or formal grammar such as regular or context-free grammar. To automatically generate inference problems, Geiger et al. (Reference Geiger, Cases, Karttunen and Potts2018) use the regular grammar in (4) to construct sentences. Optional elements are marked with ?, Q$\in${every, not every, some, no}, and other grammatical category variables range over predefined sets of words.

1. (4) Q     Adj? N     (does not)? Adv? V     Q     Adj? N

All these methods are actively used when collecting Text–Hypothesis pairs for new textual inference datasets. When pairs are human-written, one needs to be aware of potential biases that human annotators might introduce (see Section 2.1.4 for more details). Inference datasets generated fully automatically usually focus on a particular set of semantic phenomena and tend to have sentences with less structural or lexical diversity. Finally, semiautomated methods try to combine the best of both worlds to produce Text–Hypothesis pairs with diversity and at scale.

2.1.2 Annotating Inferences

Annotation of textual inferences means labeling Text–Hypothesis pairs with a ground truth inference label. Methods of annotating inferences can be roughly divided into three categories.

For human annotation, usually, crowd workers rather than experts or trained annotators are employed to produce judgments. The gold label of an inference problem is commonly set to the label that receives a majority of votes from annotators. For example, a Text–Hypothesis pair in the SICK dataset is labeled as entailment if at least three out of five crowdsourced judgments are in agreement. When annotating a pair, sometimes one of the inference judgments comes from the author of the pair. For instance, this is the case for SNLI, MNLI, and the datasets of RTE challenges.

Automatic annotation of inferences is typically used when inference pairs are fully automatically generated (see Section 2.1.1). When modifying or recasting an existing dataset, an automatic annotation method can simply map original labels to inference labels. For example, if an original inference problem is entailment, a new problem that is obtained by adding an informative and consistent conjunct to a Hypothesis will have a neutral inference label.

The third approach is to use human annotations for a task simpler than inference. For example, the Monotonicity Entailment Dataset (MED, Yanaka et al. Reference Yanaka, Mineshima and Bekki2019a) asks crowd workers to make certain phrases in a sentence more specific – for example, make spectator in every spectator bought a ticket more specific with female spectator. With the help of the human-elicited phrasal inference and the monotonicity calculus (see Section 2.2.2), one can automatically detect that the original sentence entails the new sentence obtained with the phrase replacement.

It is important to keep in mind that not all gold labels are gold (see Section 2.1.5). Human annotation might introduce erroneous gold labels due to insufficient annotation guidelines or ambiguity. For example, a substantial number of gold labels in the SICK dataset are inconsistently applied to the inference problems (Kalouli et al., Reference Kalouli, Hu and Webb2023; Kalouli Real, & de Paiva, Reference Kalouli, Real and de Paiva2017; Marelli et al., Reference Marelli, Menini and Baroni2014). The reason behind this is that annotators interpreted indefinite noun phrases in different ways: A boy is running and A boy is not running can be judged as contradiction or neutral depending on coreference or lack thereof (see Section 2.1.3 for more discussion).

2.1.3 Two Interpretations of Contradiction

The contradiction label was introduced at the third RTE challenge (Giampiccolo et al., Reference Giampiccolo, Magnini, Dagan and Dolan2007) as part of a pilot three-way classification of textual inference. Unlike the two-way classification in previous RTE datasets, the three-way classification distinguishes contradiction ($\text{⫫}$) and neutral (##) in non-entailment inferences ($\nRightarrow$). The contradiction label was defined in a similar vague fashion as the entailment label. In particular, according to de Marneffe, Rafferty, and Manning (Reference de Marneffe, Rafferty and Manning2008), contradiction occurs when a Text and a Hypothesis are extremely unlikely to be true simultaneously. For the contradiction label, the annotation guidelines instructed that compatible referring expressions had the same reference in the absence of clear countervailing evidence.Footnote ⁷ This definition of contradiction worked well for the RTE challenge datasets mainly because the datasets kept Text–Hypothesis pairs grounded in natural data, which means that the pairs contained longer Texts and more definite NPs and named entities.

Annotation of the SICK dataset showed that if the co-reference of compatible referring expressions is not explicitly instructed for caption-like sentence pairs, crowd workers provide mixed annotations for the inference problems involving indefinite NPs and negation. For example, the SICK inference problems in (5) and (6) have the exact same structure from an inference perspective, but (5) gets the neutral gold label while (6) gets contradiction:

1. (5) A couple is not looking at a map. ## A couple is looking at a map.

2. (6) A soccer ball is not rolling into a goal net.

   $\text{⫫}$ A soccer ball is rolling into a goal net.

Many Text–Hypothesis pairs are not in a contradiction relation if no co-reference of entities or events is assumed. Recall the example where A boy is running and A boy is not running do not form a contradiction pair unless a boy in both sentences refers to the same entity. If event co-reference is adopted, a pair like A cat is sleeping and A dog is sleeping would become a contradiction: The only participant of the sleeping event cannot be both a cat and a dog. Even worse, event co-reference would make A cat is sleeping and A dog is running a contradiction due to the incompatibility of sleeping and running events. Such a notion of contradiction is highly odd from a purely logical perspective.

To instruct crowd workers about annotating coreference-enforced contradiction, the authors of SNLI grounded sentences in photos without showing actual photos to the crowd workers. In particular, the crowd workers were asked whether a Hypothesis could definitely be a true, might be a true, or definitely be a false description of a photo whose caption was the Text.Footnote ⁸ Such a guideline prevents the co-reference issue the SICK dataset suffers from, but on the other hand, it introduces somewhat odd contradiction problems that involve unrelated sentences as illustrated by an SNLI problem in (Footnote 7). It is important to note that problems like (Footnote 7) are labeled as neutral in SICK. Hence models should not be trained on SICK and evaluated on SNLI/MNLI or vice versa as these datasets use different interpretations of contradiction.Footnote ⁹

1. (7) Dog carry ${}_{[}\text{sic}]$ leash in mouth runs through marsh.

   $\text{⫫}$ A ship hitting an iceberg.

The majority of the existing inference datasets adopt the co-reference-enforced notion of contradiction. Several inference datasets are annotated with binary labels, entailment and non-entailment, and avoid opting for one of the contradiction notions.

2.1.4 Biases in Textual Inference

The main idea behind collecting textual inference datasets is to teach an NLP system regularities governing NLI or to evaluate its semantic capacity. However, high system performance on a particular inference dataset does not necessarily mean that the system has learned the underlying inference regularities. It might easily be the case that the system learned accidentally introduced regularities behind the gold labels in the dataset. For example, a high word overlap between a Text and a Hypothesis is often a good indicator of the entailment relation, but it has little to do with the underlying rationale of inferences. Learning such accidental regularities might be easily overlooked in deep learning as models employ representations and transformations that are opaque for humans. Next we present two biases in textual inference datasets that further encourage models to learn accidental regularities about inferences.

The hypothesis-only bias is a dataset bias that allows models to achieve relatively high accuracy on the dataset while the models take only a Hypothesis as an input, completely ignoring the Text part. The hypothesis-only bias for the SNLI and MNLI datasets was concurrently reported by several works (Gururangan et al., Reference Gururangan, Swayamdipta and Levy2018; Poliak et al., Reference Poliak, Naradowsky and Haldar2018; Tsuchiya, Reference Tsuchiya2018). They showed that some neural models can correctly classify 63–69 percent of SNLI problems by looking only at a Hypothesis. This accuracy is twice as high as the majority baseline (34 percent).Footnote ¹⁰ For MNLI, the hypothesis-only performance range is 52–53 percent compared to 35 percent of the majority baseline. The root of the hypothesis-only bias lies in the data collection method of SNLI and MNLI. For example, in the test part of SNLI, 90 percent of inference problems with a word form of sleep in a Hypothesis is labeled as contradiction. This reflects the tactics crowd workers used to quickly provide a Hypothesis sentence per inference label. Using several neural models as examples, Gururangan et al. (Reference Gururangan, Swayamdipta and Levy2018) showed that after training on the datasets, the hypothesis-only bias gets projected into the predictions of the models.

Another common bias associated with inference datasets and learned by models is a high word overlap between a Text and a Hypothesis for entailment problems. The Heuristic Analysis for NLI Systems (HANS) dataset by McCoy, Pavlick, and Linzen (Reference McCoy, Pavlick and Linzen2019) intends to evaluate a model on the extent it uses a word-overlap heuristic for entailment classification. The dataset covers three types of heuristics depending on whether a Hypothesis is a subset, subsequence, or constituent of a Text. The entailment and non-entailment inference problems for each heuristic are given in (8). Note that every word in the shared Hypothesis sentence occurs in the Text sentences.

(8)	a.	Subset heuristic
		(i) The cat with a collar slept.	$\Rightarrow$
		(ii) The cat saw that the dog slept.
	b.	Subsequence heuristic
		(i) The dog and the cat slept.	$\Rightarrow$	The cat slept.
		(ii) The dog near the cat slept.
	c.	Constituent heuristic
		(i) The dog saw the cat slept.	$\Rightarrow$
		(ii) If the cat slept, the dog was away.

Several works (He, Wang, & Zhang, Reference He, Wang and Zhang2020; McCoy, Pavlick, & Linzen, Reference McCoy, Pavlick and Linzen2019) showed that when neural models fine-tuned on large inference datasets are evaluated on HANS, the accuracy on (i)-style problems is much higher than on (ii). For instance, the accuracy gap is greater than 70 percent for BERT fine-tuned on MNLI. This indicates that the neural models have difficulties to distinguish high lexical overlap from entailment.

Besides the two mentioned biases, there are also other dataset biases. A reversed word overlap bias is a tendency to label a problem with a low word overlap as non-entailment (Rajaee, Yaghoobzadeh, & Pilehvar, Reference Rajaee, Yaghoobzadeh and Pilehvar2022). Yet another bias is a negation bias, which is a preference to classify a problem as contradiction if it contains a negation word. The negation bias exists in SICK, SNLI, and MNLI (Gururangan et al., Reference Gururangan, Swayamdipta and Levy2018; Lai & Hockenmaier, Reference Lai and Hockenmaier2014). There is an entire research line in the textual inference that attempts to debias inference models.

2.1.5 Should the Textual Inference Task Be Categorical?

Textual inference is modeled as a two- or three-way classification task. But taking into account the soft nature of the entailment and contradiction notions, is a categorical classification suitable for textual inference? There have been at least two proposals for an alternative modeling of the textual inference task. One proposal models textual inference as a subjective probability of entailment while another one uses the distribution of human judgments over the inference labels instead of a single inference label.

Chen et al. (Reference Chen, Jiang and Poliak2020) argue for uncertain NLI (UNLI) where a Text–Hypothesis pair is estimated with a probability score rather than a single inference label. The probability score represents an average of subjective probabilities elicited from crowd workers.Footnote ¹¹ An inference problem that gets the neutral gold label in SNLI but 0.84 entailment probability in UNLI is given in (9):

1. (9) A man is singing into a microphone.

   $\left(0.84\right)\Rightarrow$ A man is performing on stage.

Nie, Zhou, and Bansal (Reference Nie, Zhou and Bansal2020) modeled the textual inference task as predicting a probability distribution over the inference labels. Following a proposal by Pavlick and Kwiatkowski (Reference Pavlick and Kwiatkowski2019), they created the ChaosNLI dataset where gold standard distributions per inference problem were derived from 100 crowdsourced judgments. (10) illustrates an SNLI problem that originally had the entailment gold label obtained as a majority label from three entailment and two neutral judgments. However, after re-annotating the problem as a part of ChaosNLI, it gets contradiction as the most probable label in the label distribution.

1. (10) The lady wearing a red coat is giving a speech.

   $\left[\right(0.40)\Rightarrow ,(0.01)\#\#,(0.59\left)\text{⫫}\right]$ Woman is the center of attention.

In total, 25 percent of the SNLI problems that were reused in ChaosNLI received a major inference label different from the original SNLI label. This indicates that the inference gold labels that are defined as a majority among several judgments are difficult to replicate and begs a question about the adequacy of the gold standard inference labels and the categorical nature of the textual inference task.

In the subsection, we covered several crucial characteristics of the textual inference task. We summarized common methods of creating datasets, namely collecting and annotating Text–Hypothesis pairs. During the dataset creation, one can control the interpretation of the contradiction label via annotation guidelines–whether to opt for the co-reference-enforced contradiction or a more logical notion that largely narrows down the contradiction inference problems. However, it is not easy to keep inference datasets free from biases, especially when the sentences are collected via crowdsourcing. Notable biases of inference datasets are the hypothesis-only bias and the high word overlap for entailment problems. Finally, there are inference problems for which a single inference label is not representative. While there have been at least two suggestions for abandoning a single inference label in textual inference, most of the inference datasets are still created as a two- or three-way classification task.

2.2.1 The FraCaS Test Suite

We start with the FraCaS test suite (Cooper et al., Reference Cooper, Crouch and Eijck1996) as it covers several semantic phenomena that have been intensively studied in semantics literature. The FraCaS test suite was originally created as a yes/no/unknown-QA test suite for NLP systems.Footnote ¹³ Only later it was converted and used as a textual inference test suite by MacCartney and Manning (Reference MacCartney and Manning2007). It contains only 334 well-formed inference problems but has nine focused sections covering generalized quantifiers (74), plurals (33), nominal anaphora (28), ellipsis (55), adjectives (22), comparatives (31), temporal reference (70), verbs (8), and attitudes (13). Background knowledge is explicitly encoded in the FraCaS inference problems as premises (e.g., Every Swede is a Scandinavian), and (multistep) logic-based reasoning is the only challenge built in the dataset.

The FraCaS inference dataset has been rarely used for evaluating LMs due to its small size and imbalance of inference labels (e.g., entailment covers 52 percent of the problems while contradiction covers only 9 percent).

As was already mentioned, the size and imbalance of the labels make FraCaS a nonrepresentative evaluation set. Its treatment of semantic phenomena and clear structure motivate new ways of creating inference problems and datasets. FraCaS is mainly used for testing logic-based approaches (Abzianidze, Reference Abzianidze2016; Bernardy & Chatzikyriakidis, Reference Bernardy and Chatzikyriakidis2021; Hu, Chen, & Moss, Reference Hu, Chen and Moss2019).

2.2.2 Monotonicity

Reasoning with monotonicity is the most common phenomenon on which LMS have been evaluated. This is because monotonicity reasoning is well studied from a formal semantics point of view (Icard & Moss, Reference Icard and Moss2014; Van Benthem, Reference Van Benthem1986) and captures inferences that can be characterized by phrase substitutions directly in surface forms, without translations into an intermediate formal meaning representation. This facilitates the automatic generation of inference problems on monotonicity reasoning.

Not all phrase substitutions in a sentence result in a new sentence that is entailed from the original one. With the help of monotonicity reasoning, we can identify certain entailment-preserving substitutions. This is done by modeling the monotonicity properties of lexical units where quantifiers get the spotlight. Let us interpret the quantifier most as a binary function from unary predicates to $\{0,1\}$ (for false and true, respectively), where it is non-monotone in its first argument position and upward monotone in its second argument position. This can be denoted as most$(x^{\circ },y^{\uparrow })$. Since most is upward monotone in y’s position, inserting more general predicates in “most dogs y” should not decrease its truth value: “most dogs are running” $\le$ “most dogs are moving”, where $\le$ can be interpreted as entailment. In the case of the non-monotone position of x, we cannot predict an order between the values of “most x are running” when two comparable arguments (e.g., dog and pet) are inserted in it: “most dogs are running” does not entail “most pets are running” and vice versa.

It gets more complicated when dealing with nested scopes of monotone operators. Let us analyze (11) as (11^ʹ), where each function is marked with monotonicity properties.Footnote ¹⁴ Following (11^ʹ), each word in (11) is colored based on its polarity – that is, the monotonicity property of the position that is a result of interference of monotone functions. Green (red) stands for an upward (downward, respectively) monotone position. When green (red) words are replaced with synonymous or more general (more specific) concepts, the resulting sentence is entailed from the initial one as demonstrated by (11)⇒(12); The results of replacement in (12) are underlined.

1. (11) Every person without a mustache who consumed alcohol tasted most snacks.

2. (11′) Every ${}^{\downarrow \uparrow }($ who ${}^{\uparrow \uparrow }($ without ${}^{\uparrow \downarrow }$(person, a mustache), consumed ${}^{\uparrow }$(alcohol)$)$, tasted ${}^{\uparrow }($ most ${}^{\circ }$(snacks)$))$

3. (12) Every man without facial hair who drank whiskey tried some snacks.

Textual inference datasets on monotonicity reasoning are usually (semi) automatically generated. The generation process goes as follows: (a) Polarity marking automatically detects the polarity of sub-phrases in a sentence by exploiting a syntactic structure and monotone operators in the sentence, (b) Phrase substitution substitutes polarity-marked sub-phrases with more general or specific phrases, and (c) Entailment labeling induces entailment relations based on the polarity of the substituted sub-phrases and the specificity order between substituted and substituting sub-phrases. Vanilla monotonicity reasoning cannot capture contradiction relations, hence most monotonicity-based inference datasets cover only entailment and non-entailment labels. For the extension of monotonicity reasoning with an exclusion relation, see MacCartney and Manning (Reference MacCartney and Manning2009) and Icard (Reference Icard2012).

One of the first monotonicity-based inference datasets, the Monotonicity Entailment Dataset (MED), was semiautomatically created by Yanaka et al. (Reference Yanaka, Mineshima and Bekki2019a). The final dataset contains more than 5,000 problems. While the problems are evenly balanced between entailment and non-entailment classes, underlying monotonicity phenomena are unevenly distributed: upward (34 percent), downward (61 percent), non (5 percent). The MED dataset is only intended for evaluation and comes with no training part.Footnote ¹⁵

Yanaka et al. (Reference Yanaka, Mineshima and Bekki2019a) evaluated top textual inference models at that time, including BERT, and found that the models underperform (below the majority baseline) on downward-monotone problems when trained on standard training sets, SNLI and MNLI. When augmenting a training set with the HELP dataset, the experiments showed that if a portion of the upward (downward) monotonicity problems increases in the training set, it hurts models to learn the downward (upward respectively) monotonicity reasoning. Chen (Reference Chen2021) reports the highest score by an LM on MED: a model with a tree structure encoder (Zhou, Liu, & Pan, Reference Zhou, Liu and Pan2016) and a self-attention (Lin et al., Reference Lin, Feng and dos Santos2017) obtains an accuracy of 75.7 percent. A substantial improvement (93.4 percent) is reported by Chen, Gao, and Moss (Reference Chen, Gao and Moss2021) with a hybrid system that combines a monotonicity reasoning system with lexical databases and LLMs. However, such hybrid systems have an obvious advantage over purely neural models as they can faithfully mimic the algorithm underlying the creation of the evaluation data.

In contrast to the MED dataset, the monotonicity part of Semantic Fragments (hereafter referred to as monFrag) by Richardson et al. (Reference Richardson, Hu, Moss and Sabharwal2020) is fully automatically created: The sentence pairs are generated with a regular grammar using a restricted vocabulary of size 119 and following the polarity markings induced from monotone operators. Such controlled generation of the pairs backed up with polarity computation of Hu et al. (Reference Hu, Chen and Moss2019) guarantees correct assignments of three-way inference labels to the generated problems. monFrag contains 10K problems equally distributed over three labels and divided into simple and hard parts based on the number of relative clauses in sentences and the vocabulary size of quantifiers per part. A sample problem from the dataset is given in (13).

1. (13) All black mammals saw exactly 5 stallions who danced $\text{⫫}$

   Some black rabbits did not see exactly 5 stallions who danced

As a result of their probing experiments, Richardson et al. (Reference Richardson, Hu, Moss and Sabharwal2020) found that LMs poorly generalize on monFrag, namely one of the best results is obtained by BERT: 62.8 accuracy score when trained on SNLI and MNLI. They also show that BERT predicts monFrag with 97.8 percent accuracy when fine-tuned on 2,000 similar monotonicity problems while its score decreases only by 1.3 percent on the MNLI development set.

Monotonicity reasoning represents a substantial challenge for LMs when it comes to distinguishing the reasoning processes driven by downward and upward monotone operators. While within the limited vocabulary (of $\approx$ 100 words) LMs overall learn the monotonicity reasoning in monFrag, generalizing monotonicity reasoning for a larger vocabulary remains a difficult problem.

For more details related to monotonicity and LMs, we refer readers to the following works: Yanaka et al. (Reference Yanaka, Mineshima, Bekki and Inui2020) show LMs having difficulties to systematically generalize on monotonicity reasoning when syntactic structures in the training and test sets differ; Geiger, Richardson, and Potts (Reference Geiger, Richardson and Potts2020) demonstrate that BERT partially mirrors the causal dynamics of the algorithm that models a fragment of monotonicity reasoning restricted to negation and lexical entailment; and Geiger et al. (Reference Geiger, Cases, Karttunen and Potts2018) emphasize the importance of alignment for reasoning with monotone quantifiers.

2.2.3 Negation

Understanding and processing negation is a challenging task for NLP systems, including those based on deep neural networks. The experiments by Kassner and Schütze (Reference Kassner and Schütze2020) and Ettinger (Reference Ettinger2020) showed that when using BERT as an LM to predict a word in a sentence and in its negated version, BERT shows little to no sensitivity to the presence of negation. Additionally, Ribeiro et al. (Reference Ribeiro, Wu, Guestrin and Singh2020) demonstrated how inserting negation can mislead prominent commercial models for sentiment analysis.

Negation is present as a part of the challenge in most of the monotonicity reasoning-based inference datasets since it is one of the main sources of downward monotone operators. However, the complementing nature of negation is not fully captured by vanilla monotonicity reasoning. There are also synthetic challenge test sets (Richardson et al., Reference Richardson, Hu, Moss and Sabharwal2020) and adversarial/stress sets (Naik et al., Reference Naik, Ravichander and Sadeh2018) that focus on negation, but their coverage and the naturalness of sentences are rather low. Instead, we will discuss the textual inference dataset from Hossain et al. (Reference Hossain, Kovatchev and Dutta2020), hereafter referred to as negNLI, which is a manually created and labeled dataset of size 4,500. It builds on the standard inference datasets such as RTE, SNLI, and MNLI.

The motivation behind creating negNLI was to test SOTA transformer models and their training datasets on the proper treatment of negations and their coverage, respectively. To create new inference problems, they extracted 500 Text–Hypothesis pairs per dataset (in total 1,500), added negation manually to the main verb of each sentence, and formed three new negation-involving problems: T_neg-H, T-H_neg, and T_neg-H_neg. Hence, negNLI consists of three subparts, negRTE, negSNLI, and negMNLI, corresponding to RTE, SNLI, and MNLI, respectively.

After experimenting with transformers such as BERT, RoBERTa (Liu et al., Reference Liu, Ott and Goyal2019) and XLNet (Yang et al., Reference Yang, Dai and Yang2019), Hossain et al. (Reference Hossain, Kovatchev and Dutta2020) found that the models underperform on negNLI when trained on the standard inference datasets. The results of these experiments are negative despite the problems in the subparts of negNLI being very similar to the original inference problems, differing only in terms of inserted negation particles.

Another inference dataset on negation worth mentioning is the NaN-NLI test suite (Truong et al., Reference Truong, Otmakhova and Baldwin2022), where NaN stands for Not another Negation. It is a small curated set of 258 inference problems and is intended only for evaluation use. The distinct feature of NaN-NLI is that it covers types of negation that rarely affect the inference labels in the datasets: nonverbal (e.g., not all and not very) and sub-clausal (e.g., negating a prepositional phrase as in not for the first time). The premises in the dataset are drawn from Pullum & Huddleston (Reference Pullum and Huddleston2002). For each premise, the authors handcrafted around five hypotheses to form inference problems driven by a negation item.

In the evaluation experiments, Truong et al. (Reference Truong, Otmakhova and Baldwin2022) use two pretrained LMs: RoBERTa and negRoBERTa, a variant of RoBERTa pretrained with negation data augmentation and a negation cue masking strategy (Truong et al., Reference Truong, Baldwin, Cohn and Verspoor2022). Both models are fine-tuned on MNLI and MNLI augmented with negMNLI of Hossain et al. (Reference Hossain, Kovatchev and Dutta2020). The highest results are obtained when fine-tuning the models on the augmented data. The obtained scores of both models are comparable (ca. 62.7 percent) and represent a moderate improvement over the majority class baseline (45.3 percent).

Negative results on modeling negation are also reported by Hartmann et al. (Reference Hartmann, de Lhoneux and Hershcovich2021) when evaluating the multilingual BERT model on five languages. Unlike previous datasets, Hartmann et al. (Reference Hartmann, de Lhoneux and Hershcovich2021) structured their multilingual inference dataset in minimal pairs of inference problems. In this way, the dataset tests a model on whether it correctly recognizes the effect the presence and absence of negation have on inference labels.

Classifying textual inference problems with negation remains a challenge for LMs, stemming from the scope-taking nature of negation and its ability to flip the meaning of a phrase. The latter behavior contrasts with the general word insertion mechanism, which usually introduces additional information to the meaning (e.g., inserting adjuncts or complements).

2.2.4 Implicatures and Presuppositions

Implicatures and presuppositions are pragmatic inferences that are different from standard logical entailment. While implicatures are defeasible suggestions made by an utterance, presuppositions are assumed true by an utterance as they are essential for interpreting its meaning. Unlike entailments, presuppositions can survive even when they are embedded under questions, conditionals, and negation. For instance, (14) shows examples of a presupposition and an implicature (of the type usually called scalar implicature).

1. (14) Some of John’s kids are playing outside.

   presupposes that      John has kids.

   implicates that      One of John’s kids is not playing outside.

Note that the same presupposition would still be available if we considered the negated version of the sentence Some of John’s kids are not playing outside or the question Are some of John’s kids playing outside?.

The implicature in (14) arises because an alternative to some – namely all – could have been used, but it was not. Pragmatic reasoning about why this alternative was not used can lead to a conclusion from (14) that this alternative is not true (i.e., not all of John’s kids are playing outside). The implicature can be canceled with the follow-up elaborating sentence Actually all of John’s kids are playing outside.

The inference relation built into inference datasets has an imprecise definition that says “T entails (contradicts) H if humans reading T would typically infer that H is most likely true (false)” (see p. 21) and represents a weaker relation than logical entailment. This raises a question: What is the relation between the entailment that textual inference models learn and pragmatic inferences like implicatures and presuppositions? Do textual inference models recognize implicatures as entailment or as neutral? Are they robust enough to consistently accommodate presuppositions?

To answer these questions, Jeretic et al. (Reference Jeretic, Warstadt, Bhooshan and Williams2020) automatically created an inference dataset, called ImpPres, focusing on scalar implicatures and presuppositions. The problems were generated from predefined sentence templates, in total more than 25,000. The scalar implicature part consists of six subparts, each focusing on a particular lexical scale: determiners $⟨$ some,all $⟩$, connectives $⟨$ or,and, modals $⟨$ can,have to $⟩$, numerals $⟨$ 2,3 $⟩$, gradable adjectives $⟨$ good,excellent $⟩$, and gradable verbs $⟨$ run,sprint $⟩$. The presupposition part has eight subparts involving all N, both, change of state, cleft existence, cleft uniqueness, only, possessed definites, and questions. As noted by the dataset authors, ImpPres is solely intended for evaluation purposes since the patterns in the dataset can be easily learned.

Experiments conducted on MNLI-trained BERT showed that with some consistency BERT uses pragmatic inference when some is in a premise – that is, identifies examples like $⟨$some N V$⟩$, all N V as contradiction. However, experiments on other subparts suggested that BERT cannot distinguish the scalar pairs for connectives and gradable concepts – for example, it treats X is good and X is excellent as semantically equivalent, and it inconsistently handles the cases of numerals and modals. Evaluation on the presupposition part reveals that BERT predicts entailment for presuppositions of clefts (e.g., it is X who V ⇒ Someone V), possessed definites, only, and questions (e.g., John knew why Ann left $\Rightarrow$ Ann left) but fails to do so for numerals (e.g., Both N V $\Rightarrow$ Exactly two N V) and change of state (e.g., X was healed $\Rightarrow$ X used to be ill).

Jeretic et al. (Reference Jeretic, Warstadt, Bhooshan and Williams2020) conclude that the pragmatic reasoning capacity of BERT mostly comes from the pretraining stage – that is, masked language modeling, as MNLI contains an insufficient number of pragmatic inferences and almost no samples of those triggered lexically. This leaves the question open whether LMs are able to consistently carry out pragmatic reasoning.

A follow-up study by Parrish et al. (Reference Parrish, Schuster and Warstadt2021) created a test dataset of more than 2,000 inference problems on presuppositions. In the dataset, the Text represents naturally occurring multiple sentences while the Hypothesis is manually constructed for each Text. To model the gradable nature of presupposition projection/cancellation, they also designed variants of Text that contain negated presupposition triggers. The results of their experiments show that models performed comparably to humans on relatively simple cases (e.g., cleft, numeric determiners, and temporal adverbs) but failed to fully capture human-level context sensitivity and gradience.

For related work, we refer the readers to Jiang and de Marneffe (Reference Jiang and de Marneffe2019), Ross and Pavlick (Reference Ross and Pavlick2019), and Schuster, Chen, and Degen (Reference Schuster, Chen and Degen2020). Jiang and de Marneffe (Reference Jiang and de Marneffe2019) recast samples of CommitmentBank (de Marneffe, Simons, & Tonhauser, Reference de Marneffe, Simons and Tonhauser2019) to inference problems, where the Text consists of multiple sentences, and the Hypothesis is a complement of clause-embedding verbs under entailment-canceling environments (conditional, negation, modal, and question). Based on the experiments with BERT models, they concluded that the models still do not capture the full complexity of pragmatic reasoning. Ross and Pavlick (Reference Ross and Pavlick2019) studied whether BERT can make correct inferences about veridicality in verb–complement constructions. While the projectivity behavior of verb–complement verbs is different from presupposition projection, they share similarities when it comes to inferring embedded meaning. Schuster et al. (Reference Schuster, Chen and Degen2020) explored whether an LSTM-based sentence encoder can be used to predict the strength of scalar inferences, namely predicting semantic similarity between some kids play and some, but not all, kids play.

2.2.5 Other Targeted Inference Datasets

In addition to the discussed inference datasets, there are many other datasets that focus on semantic phenomena beyond the scope of the section. Kober et al. (Reference Kober, Bijl de Vroe and Steedman2019) designed and manually annotated a set of sentence pairs that require reasoning with tense and aspect.Footnote ¹⁶ Ravichander et al. (Reference Ravichander, Naik, Rose and Hovy2019) prepared the equate dataset for quantitative reasoning formatted as inference problems. Saha, Nie, and Bansal (Reference Saha, Nie and Bansal2020) constructed the conjnli challenge set to evaluate LMs on understanding connectives (like and, or, but, nor) in conjunction with quantifiers and negation. In addition to the monotonicity fragment, Richardson et al. (Reference Richardson, Hu, Moss and Sabharwal2020) created synthetic data fragments for negation, Boolean connectives, quantifiers, and comparatives. Abzianidze et al. (Reference Abzianidze, Zwarts and Winter2023) curated inference problems on spatial reasoning and showed that LMs are far from mastering it. Liu et al. (Reference Liu, Wu, Michael, Bouamor, Pino and Bali2023) designed an inference dataset, called AmbiEnt, to evaluate models on reasoning with ambiguous sentences involving a variety of lexical, syntactic, and pragmatic ambiguities. The dataset shifts from three-way classification to multi-label classification with three inference labels. Inference problems that are sensitive to the ambiguity of the Text are classified with more than one inference label.

3.1.1 Similarity-Based Tests

One approach to establishing whether compositional vector semantic representations are satisfactory relies on the notion of similarity. Vector spaces have inherent similarity structures that can be measured numerically with metrics like the cosine. The cosine values serve as the models’ similarity or relatedness predictions for pairs of sentences or phrases, and are compared to numeric similarity or relatedness scores produced by human annotators for the same phrase or sentence pairs. Metrics of choice for composition model evaluations are typically correlation coefficients (Pearson’s or Spearman’s).

The first such similarity datasets were rather small – for example, human similarity judgments for adjective-noun, noun-noun, and verb-object combinations for 108 phrase pairs for each type in Mitchell and Lapata (Reference Mitchell and Lapata2010); determiner–noun combinations (Bernardi et al., Reference Bernardi, Dinu, Marelli and Baroni2013) and sentences with transitive verbs (Kartsaklis, Sadrzadeh, & Pulman, Reference Kartsaklis, Sadrzadeh and Pulman2013).

These small controlled datasets featuring dozens of phrases or sentences raise concerns of generality and ecological validity. They might not be representative of semantic composition in general. As a result, a model might work well for this data but fail to extend to other phrases or more complex data.

This motivated more ecologically valid datasets consisting of varied sentences with a range of syntactic structures. The Semantic Textual Similarity (STS) task, introduced by Agirre et al. (Reference Agirre, Cer, Diab and Gonzalez-Agirre2012), presents sentence pairs annotated on a scale from 0 (“on different topics”) to 5 (“completely equivalent”). The original sentences in the pairs were taken from a variety of sources, such as image and video descriptions and outputs of machine translation models. The SICK dataset (Marelli et al., Reference Marelli, Menini and Baroni2014) tries to control for phenomena such as proper nouns that may affect model predictions but are distinct from composition and could confound the evaluation of compositional models.

There are also alternatives to human judgments on similarity or relatedness for evaluation of compositional representations. One proposal is that the similarity of vector representations of phrases should correspond to how often one of the components in the phrase is expressed by the same lexical item across languages (Ryzhova, Kyuseva, & Paperno, Reference Ryzhova, Kyuseva and Paperno2016). For example, the vector of the phrase sharp knife is expected to be more similar to that of sharp saw than sharp needle because across languages the former two consistently use the same translation for sharp (e.g., French tranchant), while the latter often differs (French aigu).

There is also the rank approach to intrinsic similarity-based evaluation. While ingenious, it has limited applicability and can only be used with vector models that can produce vector representations of phrases that are comparable to vectors of words. One can think of such a model as processing a corpus where every occurrence of the phrase red car is represented as a single token red_car. Such a model can then estimate a vector for the phrase red car (the observed phrase vector) just like it creates vectors for words red and car when they occur outside of the phrase. Ideally, an adequate composition model should predict a compositional vector for red car that closely resembles the observed vector of red_car. One metric of success for a compositional model is the rank: If the observed phrase vector is closer to the compositional one than vectors of other words and phrases, the model’s prediction is on the right track and the rank is 1; if the compositional model is further off track, the rank of the “correct” phrase vector is higher.

Rank evaluation of vector composition was first applied by Baroni and Zamparelli (Reference Baroni and Zamparelli2010) and extended more broadly by Dima et al., (Reference Dima, de Kok, Witte and Hinrichs2019). See also Boleda, Baroni, McNally, et al. (Reference Boleda, Baroni and McNally2013) on adjective–noun vector composition for non-intersective adjectives.

3.1.2 Representation Testing on Downstream Tasks

Compositionality of models can also be estimated indirectly via downstream tasks. The assumption is that solving the specific task requires adequate semantic representations, which must be compositional. Such tasks include inference (section 2), sentiment analysis (determining how positively a text, typically customer feedback, describes a certain object), and QA (Rajpurkar et al., Reference Rajpurkar, Zhang, Lopyrev and Liang2016):

1. (15) passage:

   In meteorology, precipitation is any product

   of the condensation of atmospheric water vapor

   that falls under gravity. $<\dots >$

   question:

   What causes precipitation to fall?

Closely related to QA, the LAMBADA task (Paperno et al., Reference Paperno, Kruszewski and Lazaridou2016) is a fill-in-the-blank task where understanding of a whole passage above and beyond the immediate sentential context of the masked word is required to fulfill the task successfully. The LAMBADA task therefore approximately measures the ability of LMs to process compositional meaning of discourse. The LAMBADA task was challenging to all models at the time the dataset was introduced, but large LMs with few-shot learning on the task (Brown et al., Reference Brown, Mann, Ryder, Larochelle, Ranzato and Hadsell2020; Chowdhery et al., Reference Chowdhery, Narang and Devlin2022) showed impressive progress on LAMBADA.

Such tasks have been instrumental in validating models; they all clearly involve compositional meaning. For example, a negation placed in a well-chosen position in the text can completely change the entailment relation between two sentences, the set of correct answers to a question about the text, or the text’s sentiment. In other cases, a negation placed elsewhere might not interfere with the meaning of the text in the same ways, showing that proper treatment of negation requires compositionality: deriving the meaning of a complex text both from the elements in the text and the way they are combined.

3.1.3 Compositional Tasks

3.2.1 Levels of Composition

Compositional models exist for all levels of linguistic structure. For morphology, there were different attempts to use morpheme decomposition in computing vector representations of derived words (Botha & Blunsom, Reference Botha, Blunsom, Xing and Jebara2014; Lazaridou et al., Reference Lazaridou, Marelli, Zamparelli and Baroni2013; Luong, Socher, & Manning, Reference Luong, Socher and Manning2013). Vector-based representations are thereby learned for individual morphemes. Soricut and Och (Reference Soricut and Och2015) combine a simple composition model with morphology induction. Most current NLP models do away with morphemes altogether. The simple and efficient fastText model (Bojanowski et al., Reference Bojanowski, Grave, Joulin and Mikolov2017) approximates a word’s vector as the sum of its character n-gram embeddings: rather than simply using the distribution of, for example, hipster across contexts, the system collects and sums the distributions of n-grams of characters – for example, hips, ipst, pste, ster. The contrasts in distributional informativeness of hips or ster versus ipst or pste might effectively approximate the effect of segmenting a word into morphemes. Another approach, standard in modern LMs, is subword tokenization, which may or may not correspond to morphemes. At the same time, there is evidence suggesting that morphologically informed segmentation might outperform sub-word segmentation (Hofmann, Pierrehumbert, & Schütze, Reference Hofmann, Pierrehumbert and Schütze2021). Among sub-word-based alternatives (including BPE but also others, e.g., Jinman et al., Reference Jinman, Zhong, Zhang and Liang2020; Pinter, Guthrie, and Eisenstein Reference Pinter, Guthrie and Eisenstein2017; fastText remains a robust method for producing rare word vectors (Prokhorov et al., Reference Prokhorov, Pilehvar and Kartsaklis2019; Vulić et al., Reference Vulić, Baker and Ponti2020).

In phrase- and sentence-level composition, many earlier models relied on parse tree representations as input, and therefore featured recursive composition following grammatical structure (Clark, Coecke, & Sadrzadeh, Reference Clark, Coecke and Sadrzadeh2008; Irsoy & Cardie, Reference Irsoy and Cardie2014; Le & Zuidema, Reference Le and Zuidema2015; Paperno, Pham, & Baroni, Reference Paperno, Pham and Baroni2014; Socher et al., Reference Socher, Huval, Manning and Ng2012; Socher et al., Reference Socher, Perelygin and Wu2013). However, state-of-the-art LMs are instead trained end to end on text data without explicit parsing.

The general principle of having the same composition model for all levels of language structure up to the level of discourse has evolved as computational models grew more sophisticated. It was already present in latent semantic analysis (Landauer & Dumais, Reference Landauer and Dumais1997) in the simple form of vector addition. Modern LMs such as BERT and GPT employ a much more flexible mechanism of self-attention that has the same cross-level coverage from tokens up to monological or dialogical texts.

3.2.2 Theoretically Simple Models of Composition

The Additive Model of Composition

Assume that two items such as words that have vector representations are combined. What is the vector representation of their combination? The simplest approach to vector composition consists in adding up vectors of component words together. Repeated addition effectively treats text as a bag of words, meaning that word order and syntactic structure are ignored; texts with the same words in them are processed identically. Despite its simplicity, vector addition is surprisingly effective and robust in practice. For example, the sum of high-quality word vectors outperformed more sophisticated approaches to vector composition in the study of preposition ambiguity (Ritter et al., Reference Ritter, Long and Paperno2015). Vector addition has been used as a method for arriving at meaning representations of phrases, sentences, and even texts at least since Landauer and Dumais (Reference Landauer and Dumais1997). More recently, sentence representations as summed contextualized token vectors from Transformer-based models were suggested (Cer et al., Reference Cer, Yang and Kong2018). Additive composition is efficient for a good reason. Ultimately, dimensions of word vectors are used to predict in which contexts the word is likely to be used; this is the objective of word embeddings and neural LMs. This means that values in word vectors translate into scores of statistical association between words and their contexts, which are usually related to the Pointwise Mutual Information (PMI) score (Levy & Goldberg, Reference Levy and Goldberg2014):

(6)

where $p\left(w\right),p\left(c\right)$ are probabilities of the word and the context and $p(w,c)$ is the probability of their joint occurrence. Under the idealizing assumption that two words’ associations with contexts do not interact non-trivially, it follows that the sum of two words’ PMI values for a given context approximates these two words’ combination’s PMI for the same context. As a result, if vector dimensions of words correspond to PMIs as they do in models like GloVe and skip-gram, then the sum $\overrightarrow{car}+\overrightarrow{red}$ approximates the distributional profile of the phrase red car (Paperno & Baroni, Reference Paperno and Baroni2016). If dimensions of $\overrightarrow{car}$ indicate that car raises the probability of context $c$ by $a$ orders of magnitude, and dimensions of $\overrightarrow{red}$ indicate that red raises the probability of context $c$ by $b$ orders of magnitude, then the phrase red car plausibly raises the probability of context $c$ by $a+b$ orders of magnitude. This suggests additive vector composition as a strong baseline to the extent that words’ PMI scores are linear functions of their vector dimensions. For models that do not include log transformation in the calculation of association scores, as in Mitchell and Lapata (Reference Mitchell and Lapata2010), pointwise multiplication rather than addition is competitive.

Parametric Approaches to Vector Composition

The additive model has clear practical advantages. However, its conceptual issues are equally obvious. For instance, addition is effectively a bag-of-words model, agnostic of word order and syntactic structure. Addition predicts the exact same vectors for sentences Cats chase mice and Mice chase cats.

This observation motivates various parametric approaches to vector composition. This means that the vector of the phrase includes not just vector representations of the words involved, but also additional numeric parameters. Such parameters can be learned from distributional properties of phrases that can themselves be encoded in vectors. One simple parametric approach that proved efficient in different evaluations such as Mitchell and Lapata (Reference Mitchell and Lapata2010) is weighted addition:

$\overrightarrow{AB}=\alpha \text{⃗}A+\beta \text{⃗}B$(7)

where $\alpha$ and $\beta$ are scalar factors. For example, the phrase vector $\overrightarrow{redcar}$ can be computed by combining vectors for words red and car with different weights (e.g., $0.6\overrightarrow{red}+0.4\overrightarrow{car}$)

In Mitchell and Lapata’s experiments, different weight combinations were estimated for different types of phrases. the first component (adjective) received a high weight in adjective–noun phrases while the second component (noun) had a higher weight in noun–noun compounds. One problem of the weighted addition is its monotonicity. If relations between vectors are expected to be useful in predicting entailment relations between words and phrases, the composition system should allow for different monotonic properties of elements in composition. For example, the determiner some maintains the entailment properties between nouns it combines with while no reverts them; however, in the case of (weighted) addition, relations between some dog versus some animal and no dog versus no animal would be characterized by the exact same linear offset. In contrast, richer models of composition allow for both monotonic and non-monotonic computation, and more powerful transformer-based LLMs discussed in what follows are known to exploit context monotonicity (Bylinina & Tikhonov, Reference Bylinina and Tikhonov2022; Jumelet et al., Reference Jumelet, Denic and Szymanik2021).

These issues of additive models of composition are addressed by richer parametric models, which allow compositional combinations to proceed in more differentiated, even idiosyncratic ways. Directly inspired by type-driven semantic theory, the Lexical Function model (Baroni & Zamparelli, Reference Baroni and Zamparelli2010) treats one element in the phrase as a function and the other as its argument. The functions in question are linear, so composition reduces to the multiplication of the argument vector by the function-specific matrix:

$\overrightarrow{AB}=mat\left(A\right)\text{⃗}B,$(8)

An extension of the lexical function model to higher-order functions includes using multidimensional tensors in addition to matrices (Grefenstette et al., Reference Grefenstette, Dinu and Zhang2013). However, the increase in the number of parameters brought about with the introduction of tensors renders such compositional models increasingly impractical, motivating proposals such as the Practical Lexical Function model (Paperno et al., Reference Paperno, Pham and Baroni2014).

In contrast to Lexical Function and versions thereof, other highly parametric approaches apply matrix weights to both elements of the composition and related highly parametric approaches (Dima et al., Reference Dima, de Kok, Witte and Hinrichs2019; Guevara, Reference Guevara2011; Socher et al., Reference Socher, Huval, Manning and Ng2012, Reference Socher, Perelygin and Wu2013):

$\overrightarrow{AB}=ma{t}_1\text{⃗}A+ma{t}_2\text{⃗}B.$(9)

where the matrices $ma{t}_1,ma{t}_2$ can be specific for the lexical items $B,S$ combined, or be shared across lexical items. Some studies, such as Gamallo (Reference Gamallo2021), experimented with a different compositional approach based on syntactic dependencies rather than constituent structure.

The problem with all the parametric approaches to vector composition is in scaling up to diverse use cases on arbitrarily complex examples. End-to-end approaches such as LLMs work better for most tasks. They are not only more robust as they scale up rather easily to bigger input data, but they also do not depend on parsing quality or efficiency, all while sharing parameters of composition across different types of constructions.

3.2.3 Composition in State-of-the-Art Transformer Models