We provide a unified, theoretical basis on which measures of data reliability may be derived or evaluated, for both quantitative and qualitative data. This approach evaluates reliability as the “proportional reduction in loss” (PRL) that is attained in a sample by an optimal estimator. The resulting measure is between 0 and 1, linearly related to expected loss, and provides a direct way of contrasting the measured reliability in the sample with the least reliable and most reliable data-generating cases. The PRL measure is a generalization of many of the commonly-used reliability measures.

We show how the quantitative measures from generalizability theory can be derived as PRL measures (including Cronbach's alpha and measures proposed by Winer). For categorical data, we develop a new measure for the general case in which each of N judges assigns a subject to one of K categories and show that it is equivalent to a measure proposed by Perreault and Leigh for the case where N is 2.

We propose a coefficient of agreement to assess the degree of concordance between two independent groups of raters classifying items on a nominal scale. This coefficient, defined on a population-based model, extends the classical Cohen’s kappa coefficient for quantifying agreement between two raters. Weighted and intraclass versions of the coefficient are also given and their sampling variance is determined by the Jackknife method. The method is illustrated on medical education data which motivated the research.

Several measures of agreement, such as the Perreault–Leigh coefficient, the ${\mathrm{AC}}_1$\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\textsc {AC}_{1}$$\end{document}, and the recent coefficient of van Oest, are based on explicit models of how judges make their ratings. To handle such measures of agreement under a common umbrella, we propose a class of models called guessing models, which contains most models of how judges make their ratings. Every guessing model have an associated measure of agreement we call the knowledge coefficient. Under certain assumptions on the guessing models, the knowledge coefficient will be equal to the multi-rater Cohen’s kappa, Fleiss’ kappa, the Brennan–Prediger coefficient, or other less-established measures of agreement. We provide several sample estimators of the knowledge coefficient, valid under varying assumptions, and their asymptotic distributions. After a sensitivity analysis and a simulation study of confidence intervals, we find that the Brennan–Prediger coefficient typically outperforms the others, with much better coverage under unfavorable circumstances.

Reliability and agreement studies are of paramount importance. They do contribute to the quality of studies by providing information about the amount of error inherent to any diagnosis, score or measurement. Guidelines for reporting reliability and agreement studies were recently provided. While the use of the kappa-like family is advised for categorical and ordinal scales, no further guideline in the choice of a weighting scheme is given. In the present paper, a new simple and practical interpretation of the linear- and quadratic-weighted kappa coefficients is given. This will help researchers in motivating their choice of a weighting scheme.

Experiments are considered where each of a sample of subjects is assigned to one of C categories separately by each of a fixed or varying group of observers. Building on earlier publications, general procedures are proposed to analyze agreements and disagreements among observers. In the case of a varying group of observers, it is shown that it is not necessary to demand a constant number of observers per subject. In the case of a fixed group of observers, the problem of missing data is considered.

The procedures are illustrated within the context of two clinical diagnosis examples. In the first example it is investigated which categories are relatively hard to distinguish from one another; a new theorem is applied that shows a useful property of the statistic kappa. In the second example it is investigated if a subgroup of observers can be found with a significantly higher degree of interobserver agreement.

Most measures of agreement are chance-corrected. They differ in three dimensions: their definition of chance agreement, their choice of disagreement function, and how they handle multiple raters. Chance agreement is usually defined in a pairwise manner, following either Cohen’s kappa or Fleiss’s kappa. The disagreement function is usually a nominal, quadratic, or absolute value function. But how to handle multiple raters is contentious, with the main contenders being Fleiss’s kappa, Conger’s kappa, and Hubert’s kappa, the variant of Fleiss’s kappa where agreement is said to occur only if every rater agrees. More generally, multi-rater agreement coefficients can be defined in a g-wise way, where the disagreement weighting function uses g raters instead of two. This paper contains two main contributions. (a) We propose using Fréchet variances to handle the case of multiple raters. The Fréchet variances are intuitive disagreement measures and turn out to generalize the nominal, quadratic, and absolute value functions to the case of more than two raters. (b) We derive the limit theory of g-wise weighted agreement coefficients, with chance agreement of the Cohen-type or Fleiss-type, for the case where every item is rated by the same number of raters. Trying out three confidence interval constructions, we end up recommending calculating confidence intervals using the arcsine transform or the Fisher transform.

This paper deals with a construction, which we dub Non-Agreeing Degree (NAD) constructions, with the distinguishing property that the agreement pattern between subjects and degree predicates is optionally disrupted, even in languages (like Spanish) where verbs commonly agree with their subjects. We show that the agreeing versus non-agreeing alternation comes with important semantic differences for the interpretation of the degree construction. We provide a first systematic description of this type of constructions and postulate a formal syntactic and semantic analysis. We argue that NAD constructions are characterized by degree predicates that introduce a non-conventional nominal scale and by subjects that are interpreted as equally non-conventional units of measurement. We postulate an intensionalization process on the subject of NAD constructions, which we capture via a general nominalization function that allows a default as well as an ordinary agreement pattern between subject and copula.

English morphosyntactic agreement, such as determiner–noun agreement in These cabs broke down and noun–verb agreement in The cabs break down, has a few interesting properties that enable us to investigate whether agreement has a psycholinguistic function, that is, whether it helps the listener process linguistic information expressed by a speaker. The present project relies on these properties in a perception experiment, examines the two aforementioned types of English agreement, and aims at analyzing whether and how native English listeners benefit from agreement. The two types of agreement were contrasted with cases without any overtly agreeing elements (e.g. The cabs broke down). Native speakers of English with normal hearing heard short English sentences in quiet and in more or less intense white noise and were requested to indicate whether the second word of the sentence (e.g. These cabs broke down) was a singular or plural noun. Accuracy was entered as the response variable in the binomial logistic regression model. Results showed that overt determiner–noun agreement clearly increased response accuracy, while noun–verb agreement had at best marginal effects. The findings are interpreted against the background of functional aspects of linguistic structures in English, in the context of unfavorable listening conditions in particular.

Recent research has highlighted the character and importance of the study of agreement. This paper, paralleling work on the more familiar concept of deep disagreement, will provide a first articulation of the character and implications of deep agreements, that is, agreements so deep that disagreement cannot overcome them. To do so, I start by outlining the main features of deep disagreement. I then provide a brief characterization of agreement in general to ground the discussion of the unique characteristics of deep agreements and compare them to deep disagreements. After distinguishing the concept from other more familiar ones, I then point to a few of the major social and theoretical issues the existence of deep agreements poses before concluding with some remarks pointing to valuable areas for future research.

In the literature on philosophical progress it is often assumed that agreement is a necessary condition for progress. This assumption is sensible only if agreement is a reliable sign of the truth, since agreement on false answers to philosophical questions would not constitute progress. This paper asks whether agreement among philosophers is (or would be) likely to be a reliable sign of truth. Insights from social choice theory are used to identify the conditions under which agreement among philosophers would be a reliable indicator of the truth, and it is argued that we lack good reason to think that philosophical inquiry meets these conditions. The upshot is that philosophical agreement is epistemically uninformative: agreement on the answer to a philosophical question does not supply even a prima facie reason to think that the agreed-upon view is true. However, the epistemic uninformativeness of philosophical agreement is not an indictment of philosophy's progress, because philosophy is valuable independent of its ability to generate agreement on the correct answers to philosophical questions.

This study investigated the morphosyntax of adjectival concord in case and number and subject-verb person agreement by monolingual and bilingual speakers of Russian. The main focus of the study is on the potential factors that may trigger divergence between Heritage Language (HL) speakers and those speakers who are dominant in that language, be they monolingual or bilingual. We considered the effects of cross-linguistic influence; limited input (as indexed by Age of Onset of Bilingualism, AOB), and working-memory limitations. An auditory offline grammaticality judgment task was performed by 119 adult participants split into four groups: (1) Monolingual Russian-speaking controls (MonoControl), (2) Immigrant Controls, that is, Russian-Hebrew bilinguals with AOB after the age of 13 (IMMControl); (3) bilinguals with AOB between 5–13 (BL-Late); and (4) bilinguals with AOB before the age of 5 (BL-Early). The latter group represents HL speakers. We did not find effects of cross-linguistic influence or extra memory load; at the same time, the effects of AOB were robust. Additionally, HL speakers (BL-Early group) differed from the other groups in poor performance on adjectival concord, but patterned with the others on person agreement, which indicates that the feature [person] is more robust than other agreement/concord features in HL grammars.

No studies have investigated the neural correlates of Number and Person agreement processing in bilinguals. Because a previous fMRI study showed difference in L1 and L2 morphosyntactic processing of L1 Turkish–L2 Persian bilinguals, it was of interest whether this difference can be specifically attributed to Number or Person processing. Therefore, we reanalyzed these data at the whole-brain level, revealing a selective response for Number Violations in the pars opercularis (PO), whereas Number and Person Violations activated the posterior superior temporal gyrus (pSTG). These results support the decomposition of agreement projections and their neuroanatomical substrates in bilinguals and confirm the involvement of systematically different feature-checking and feature-mapping mechanisms in Number and Person agreement but shared mechanisms between L1 and L2. Moreover, at variance with previous reports, Number Violations evoked more effects than Person Violations in pSTG, suggesting qualitatively different processing underlying R-expression and pronominal controllers.