This editorial suggests ways in which mental health science reform could yield more robust research and faster clinical progress. These include better animal and other models, a shift to transdiagnostic and clinically pragmatic classification systems, improved measurement, mission mapping and an entrepreneurial mindset aimed at taking advances rapidly to scale.

We examine the generalizability of single-topic studies, focusing on how often their confidence intervals capture the typical treatment effect from a larger population of possible studies. We show that the confidence intervals from these single-topic studies capture the typical effect from a population of topics at well below the nominal rate. For a plausible scenario, the confidence interval from a single-topic study might only be half as wide as an interval that captures the typical effect at the nominal rate. We highlight three important conclusions. First, we emphasize that researchers and readers must take care when generalizing the inferences from single-topic studies to a larger population of possible studies. Second, we demonstrate the critical importance of similarity across topics in drawing inferences and encourage researchers to consider designs that explicitly estimate and leverage similarity. Third, we emphasize that, despite their limitations, single-topic experiments have some important advantages.

Experimental asset markets with a constant fundamental value ($\mathrm{F}\mathrm{V}$) have grown in importance in recent years. A methodological examination of the robustness of experimental results in such a setting which has been shown to produce bubbles, however, is lacking. In a laboratory experiment with 280 subjects, we investigate whether specific design features are sufficient to influence experimental results. In detail, we (1) vary the visual representation of the price chart, and (2) provide subjects with full information about the FV process. We find overvaluation and bubble formation to be reduced when trading prices are displayed at the upper end of the price chart. Surprisingly, we do not find any effects when subjects have full information about the FV process.

Most economic experiments designed to test theories carefully choose specific games. This paper reports on an experimental design to evaluate how well the minimax hypothesis describes behavior across a population of games. Past studies suggest that the hypothesis is more accurate the closer the equilibrium is to equal probability play of all actions, but many differences between the designs makes direct comparison impossible. We examine the minimax hypothesis by randomly sampling constant sum games with two players and two actions with a unique equilibrium in mixed strategies. Only varying the games, we find behavior is more consistent with minimax play the closer the mixed strategy equilibrium is to equal probability play of each action. The results are robust over all iterations as well as early and final play. Experimental designs in which the game is a variable allow some conclusions to be drawn that cannot be drawn from more conventional experimental designs.

Diabetes and depression have a bidirectional relationship, but some antidepressants (such as the tricyclics) may have detrimental effects in diabetes that are exacerbated by behavioural changes associated with depression. This month's Cochrane Review evaluated the efficacy of psychological and pharmacological treatments of comorbid depression in diabetes and found that such interventions have a moderate and clinically significant effect on depression outcomes in people with diabetes. However, conclusions were limited by significant heterogeneity within examined populations and interventions, and significant risk of bias within trials. This commentary critically appraises the review and aims to contextualise its findings.

When researchers design an experiment, they usually hold potentially relevant features of the experiment constant. We call these details the “topic” of the experiment. For example, researchers studying the impact of party cues on attitudes must inform respondents of the parties’ positions on a particular policy. In doing so, researchers implement just one of many possible designs . Clifford, Leeper, and Rainey (2023. “Generalizing Survey Experiments Using Topic Sampling: An Application to Party Cues.” Forthcoming in Political Behavior. https://doi.org/10.1007/s11109-023-09870-1) argue that researchers should implement many of the possible designs in parallel—what they call “topic sampling”—to generalize to a larger population of topics. We describe two estimators for topic-sampling designs: First, we describe a nonparametric estimator of the typical effect that is unbiased under the assumptions of the design; and second, we describe a hierarchical model that researchers can use to describe the heterogeneity. We suggest describing the heterogeneity across topics in three ways: (1) the standard deviation in treatment effects across topics, (2) the treatment effects for particular topics, and (3) how the treatment effects for particular topics vary with topic-level predictors. We evaluate the performance of the hierarchical model using the Strengthening Democracy Challenge megastudy and show that the hierarchical model works well.

Researchers are often interested in whether discrimination on the basis of racial cues persists above and beyond discrimination on the basis of nonracial attributes that decision makers—e.g., employers and legislators—infer from such cues. We show that existing audit experiments may be unable to parse these mechanisms because of an asymmetry in when decision makers are exposed to cues of race and additional signals intended to rule out discrimination due to other attributes. For example, email audit experiments typically cue race via the name in the email address, at which point legislators can choose to open the email, but cue other attributes in the body of the email, which decision makers can be exposed to only after opening the email. We derive the bias resulting from this asymmetry and then propose two distinct solutions for email audit experiments. The first exposes decision makers to all cues before the decision to open. The second crafts the email to ensure no discrimination in opening and then exposes decision makers to all cues in the body of the email after opening. This second solution works without measures of opening, but can be improved when researchers do measure opening, even if with error.

The bumblebee gut parasite, Crithidia bombi, is widespread and prevalent in the field. Its interaction with Bombus spp. is a well-established epidemiological model. It is spread faecal-orally between colonies via the shared use of flowers when foraging. Accurately measuring the level of infection in bumblebees is important for assessing its distribution in the field, and also when conducting epidemiological experiments. Studies generally use 1 of 2 methods for measuring infection. One approach measures infection in faeces whereas the other method measures infection in guts. We tested whether the method of measuring infection affected the estimation of infection. Bumblebees were inoculated with a standardized inoculum and infection was measured 1 week later using either the faecal or gut method. We found that when the gut method was used to measure infection intensity estimates were significantly different to and approximately double those from the faecal method. These results have implications for the interpretation of previous study results and for the planning of future studies. Given the importance of bumblebees as pollinators, the impact of C. bombi on bumblebee health, and its use as an epidemiological model, we call on researchers to move towards consistent quantification of infections to enable future comparisons and meta-analyses of studies.

Even in the best-designed experiment, noncompliance can complicate analysis. While the intent-to-treat effect remains identified, randomization alone no longer identifies the complier average causal effect (CACE). Instrumental variables approaches, which rely on the exclusion restriction, can suffer from high variance, particularly when the experiment has a low compliance rate. We provide a framework which broadens the set of design and analysis techniques political science researchers can use when addressing noncompliance. Building on the growing literature about the advantages of ex-ante design decisions to improve precision, we show blocking on variables related to both compliance and the outcome can greatly improve all the estimators we propose. Drawing on work in statistics, we introduce the principal ignorability assumption and a class of principal score weighting estimators, which can exhibit large gains in precision in low compliance settings. We then combine principal ignorability and blocking with a simple estimation strategy to derive a more efficient estimation strategy for the CACE. In a re-evaluation of a study on the effect of GOTV on turnout, we find that the principal ignorability approaches result in confidence intervals roughly half the size of traditional instrumental variable approaches.

The majority of research papers in computer-assisted language learning (CALL) report on primarily quantitative studies measuring the effectiveness of pedagogical interventions in relation to language learning outcomes. These studies are frequently referred to in the literature as experiments, although this designation is often incorrect because of the approach to sampling that has been used. This methodological discussion paper provides a broad overview of the current CALL literature, examining reported trends in the field that relate to experimental research and the recommendations made for improving practice. It finds that little attention is given to sampling, even in review articles. This indicates that sampling problems are widespread and that there may be limited awareness of the role of formal sampling procedures in experimental reasoning. The paper then reviews the roles of two key aspects of sampling in experiments: random selection of participants and random assignation of participants to control and experimental conditions. The corresponding differences between experimental and quasi-experimental studies are discussed, along with the implications for interpreting a study’s results. Acknowledging that genuine experimental sampling procedures will not be possible for many CALL researchers, the final section of the paper presents practical recommendations for improved design, reporting, review, and interpretation of quasi-experimental studies in the field.