Search filters are single-concept systematic search strategies created by experts. Filters are a valuable resource for systematic searchers. Typically, filters are designed for a single database in a single interface. If researchers do not have access to that specific interface, the existing filter will be unusable without translation. Filter translation is a complex process that requires an understanding of information retrieval concepts, as well as the unique indexing and search functionality of databases and interfaces. The authors undertook a project to translate an APA PsycInfo search filter for Randomized Controlled Trials/Clinical Controlled Trials (RCT/CCT), developed by Canada’s Drug Agency, from the Wolters Kluwer Health Ovid interface to the EBSCO Information Services EBSCOhost interface. We present here a guide for translation, from the first principles of systematic searching to fine details of the relevant database and interfaces, based on our experience and illustrated by a worked example. We discuss each element of a systematic search in a stepwise process, addressing both the underlying information retrieval concepts and the technical strategies for effective translation between the two interfaces. We end with a discussion on translation challenges, with some guidance on how to mitigate potential impacts on sensitivity. While we have endeavored to explain the workings of this process accessibly for researchers who are not experts in systematic searching, anyone undertaking a search translation project should work with a trained information specialist if they lack information retrieval expertise or are unfamiliar with the inner workings of the database, the original interface, and the destination interface.

Random effects meta-analysis model is an important tool for integrating results from multiple independent studies. However, the standard model is based on the assumption of normal distributions for both random effects and within-study errors, making it susceptible to outlying studies. Although robust modeling using the t distribution is an appealing idea, the existing work, that explores the use of the t distribution only for random effects, involves complicated numerical integration and numerical optimization. In this article, a novel robust meta-analysis model using the t distribution is proposed (tMeta). The novelty is that the marginal distribution of the effect size in tMeta follows the t distribution, enabling that tMeta can simultaneously accommodate and detect outlying studies in a simple and adaptive manner. A simple and fast EM-type algorithm is developed for maximum likelihood estimation. Due to the mathematical tractability of the t distribution, tMeta frees from numerical integration and allows for efficient optimization. Experiments on real data demonstrate that tMeta is compared favorably with related competitors in situations involving mild outliers. Moreover, in the presence of gross outliers, while related competitors may fail, tMeta continues to perform consistently and robustly.

Systematic searches of published literature are a vital component of systematic reviews. When search strings are not “sensitive,” they may miss many relevant studies limiting, or even biasing, the range of evidence available for synthesis. Concerningly, conducting and reporting evaluations (validations) of the sensitivity of the used search strings is rare, according to our survey of published systematic reviews and protocols. Potential reasons may involve a lack of familiarity or inaccessibility of complex sensitivity evaluation approaches. We first clarify the main concepts and principles of search string evaluation. We then present a simple procedure for estimating a relative recall of a search string. It is based on a pre-defined set of “benchmark” publications. The relative recall, that is, the sensitivity of the search string, is the retrieval overlap between the evaluated search string and a search string that captures only the benchmark publications. If there is little overlap (i.e., low recall or sensitivity), the evaluated search string should be improved to ensure that most of the relevant literature can be captured. The presented benchmarking approach can be applied to one or more online databases or search platforms. It is illustrated by five accessible, hands-on tutorials for commonly used online literature sources. Overall, our work provides an assessment of the current state of search string evaluations in published systematic reviews and protocols. It also paves the way to improve evaluation and reporting practices to make evidence synthesis more transparent and robust.

Trials registry records represent a challenge in deduplication compared to deduplicating studies reported in journals and exported from bibliographic databases such as MEDLINE. We demonstrate why this is the case and propose a method to deduplicate registry records from the WHO International Clinical Trials Registry Platform (ICTRP) and ClinicalTrials.gov (CTG) specifically in the reference management tool EndNote (desktop version). We believe that our method is not only more efficient but that it will minimise the risk of registry records being incorrectly removed as duplicates in automated deduplication. The method has seven steps and is detailed in this tutorial as a step-by-step guide.

Double-zero-event studies (DZS) pose a challenge for accurately estimating the overall treatment effect in meta-analysis (MA). Current approaches, such as continuity correction or omission of DZS, are commonly employed, yet these ad hoc methods can yield biased conclusions. Although the standard bivariate generalized linear mixed model (BGLMM) can accommodate DZS, it fails to address the potential systemic differences between DZS and other studies. In this article, we propose a zero-inflated bivariate generalized linear mixed model (ZIBGLMM) to tackle this issue. This two-component finite mixture model includes zero inflation for a subpopulation with negligible or extremely low risk. We develop both frequentist and Bayesian versions of ZIBGLMM and examine its performance in estimating risk ratios against the BGLMM and conventional two-stage MA that excludes DZS. Through extensive simulation studies and real-world MA case studies, we demonstrate that ZIBGLMM outperforms the BGLMM and conventional two-stage MA that excludes DZS in estimating the true effect size with substantially less bias and comparable coverage probability.

Data visualization is crucial for effectively communicating knowledge in meta-analysis. However, existing visualization methods in meta-analysis have predominantly focused on quantitative aspects, such as forest plots and funnel plots, thereby neglecting qualitative information that is equally important for end-users in science, policy, and practice. We introduce a framework consisting of a series of visualization toolkits designed to enrich meta-analyses by borrowing approaches from other research synthesis methods, including systematic evidence mapping (scoping reviews), bibliometrics (bibliometric analysis), and alternative impact metric analysis. These “enrichment” toolkits aim to facilitate the synthesis of both quantitative and qualitative evidence, along with the assessment of the academic and nonacademic influences of the meta-analytic evidence base. While the meta-analysis yields quantitative insights, the enrichment analyses, and visualizations provide user-friendly summaries of qualitative information on the evidence base. For example, a systematic evidence map can visualize study characteristics, unraveling knowledge gaps and methodological differences. Bibliometric analysis offers a visual assessment of the nonindependent evidence, such as hyper-dominant authors and countries, and funding sources, potentially informing the risk of bias. Alternative impact metric analysis employs alternative metrics to gauge societal influence and research translation (e.g., policy and patent citations) of studies in the meta-analysis. We provide a dedicated webpage showcasing sample visualizations and providing step-by-step implementation in open-source software R (https://yefeng0920.github.io/MA_Map_Bib/). Additionally, we offer a guide on leveraging three commercially free large language models (LLMs) to help adapt the sample script, enabling users with less R coding experience to visualize their own meta-analytic evidence base.

Several methods have been proposed for the synthesis of continuous outcomes reported on different scales, including the Standardised Mean Difference (SMD) and the Ratio of Means (RoM). SMDs can be formed by dividing the study mean treatment effect either by a study-specific (Study-SMD) or a scale-specific (Scale-SMD) standard deviation (SD). We compared the performance of RoM to the different standardisation methods with and without meta-regression (MR) on baseline severity, in a Bayesian network meta-analysis (NMA) of 14 treatments for depression, reported on five different scales. There was substantial between-study variation in the SDs reported on the same scale. Based on the Deviance Information Criterion, RoM was preferred as having better model fit than the SMD models. Model fit for SMD models was not improved with meta-regression. Percentage shrinkage was used as a scale-independent measure with higher % shrinkage indicating lower heterogeneity. Heterogeneity was lowest for RoM (20.5% shrinkage), then Scale-SMD (18.2% shrinkage), and highest for Study-SMD (16.7% shrinkage). Model choice impacted which treatment was estimated to be most effective. However, all models picked out the same three highest-ranked treatments using the GRADE criteria. Alongside other indicators, higher shrinkage of RoM models suggests that treatments for depression act multiplicatively rather than additively. Further research is needed to determine whether these findings extend to Patient- and Clinician-Reported Outcomes used in other application areas. Where treatment effects are additive, we recommend using Scale-SMD for standardisation to avoid the additional heterogeneity introduced by Study-SMD.

Network meta-analysis (NMA), also known as mixed treatment comparison meta-analysis or multiple treatments meta-analysis, extends conventional pairwise meta-analysis by simultaneously synthesizing multiple interventions in a single integrated analysis. Despite the growing popularity of NMA within comparative effectiveness research, it comes with potential challenges. For example, within-study correlations among treatment comparisons are rarely reported in the published literature. Yet, these correlations are pivotal for valid statistical inference. As demonstrated in earlier studies, ignoring these correlations can inflate mean squared errors of the resulting point estimates and lead to inaccurate standard error estimates. This article introduces a composite likelihood-based approach that ensures accurate statistical inference without requiring knowledge of the within-study correlations. The proposed method is computationally robust and efficient, with substantially reduced computational time compared to the state-of-the-science methods implemented in R packages. The proposed method was evaluated through extensive simulations and applied to two important applications including an NMA comparing interventions for primary open-angle glaucoma, and another comparing treatments for chronic prostatitis and chronic pelvic pain syndrome.

While the Institute of Education Science’s ERIC is often recommended for comprehensive literature searching in the field of education, there are several other specialized education databases to discover education literature. This study investigates journal coverage overlaps between four specialized education databases: Education Source (EBSCO), Education Database (ProQuest), ERIC (Institute of Education Sciences), and Educator’s Reference Complete (Gale). Out of a total of 4,695 unique journals analyzed, there are 2,831 journals uniquely covered by only one database, as well as many journals covered by only two or three databases. Findings show that evidence synthesis projects and literature reviews benefit from the careful selection of multiple specialized education databases and that ERIC is insufficient as the primary education database for comprehensive searching in the field.

With the increasing volume of scientific literature, there is a need to streamline the screening process for titles and abstracts in systematic reviews, reduce the workload for reviewers, and minimize errors. This study validated artificial intelligence (AI) tools, specifically Llama 3 70B via Groq’s application programming interface (API) and ChatGPT-4o mini via OpenAI’s API, for automating this process in biomedical research. It compared these AI tools with human reviewers using 1,081 articles after duplicate removal. Each AI model was tested in three configurations to assess sensitivity, specificity, predictive values, and likelihood ratios. The Llama 3 model’s LLA_2 configuration achieved 77.5% sensitivity and 91.4% specificity, with 90.2% accuracy, a positive predictive value (PPV) of 44.3%, and a negative predictive value (NPV) of 97.9%. The ChatGPT-4o mini model’s CHAT_2 configuration showed 56.2% sensitivity, 95.1% specificity, 92.0% accuracy, a PPV of 50.6%, and an NPV of 96.1%. Both models demonstrated strong specificity, with CHAT_2 having higher overall accuracy. Despite these promising results, manual validation remains necessary to address false positives and negatives, ensuring that no important studies are overlooked. This study suggests that AI can significantly enhance efficiency and accuracy in systematic reviews, potentially revolutionizing not only biomedical research but also other fields requiring extensive literature reviews.

Conventional meta-analyses (both fixed and random effects) of correlations are biased due to the mechanical relationship between the estimated correlation and its standard error. Simulations that are closely calibrated to match actual research conditions widely seen across correlational studies in psychology corroborate these biases and suggest two solutions: UWLS+3 and HS. UWLS+3 is a simple inverse-variance weighted average (the unrestricted weighted least squares) that adjusts the degrees of freedom and thereby reduces small-sample bias to scientific negligibility. UWLS+3 as well as the Hunter and Schmidt approach (HS) are less biased than conventional random-effects estimates of correlations and Fisher’s z, whether or not there is publication selection bias. However, publication selection bias remains a ubiquitous source of bias and false-positive findings. Despite the relationship between the estimated correlation and its standard error in the absence of selective reporting, the precision-effect test/precision-effect estimate with standard error (PET-PEESE) nearly eradicates publication selection bias. Surprisingly, PET-PEESE keeps the rate of false positives (i.e., type I errors) within their nominal levels under the typical conditions widely seen across psychological research whether there is publication selection bias, or not.

Network meta-analysis allows the synthesis of relative effects from several treatments. Two broad approaches are available to synthesize the data: arm-synthesis and contrast-synthesis, with several models that can be fitted within each. Limited evaluations comparing these approaches are available. We re-analyzed 118 networks of interventions with binary outcomes using three contrast-synthesis models (CSM; one fitted in a frequentist framework and two in a Bayesian framework) and two arm-synthesis models (ASM; both fitted in a Bayesian framework). We compared the estimated log odds ratios, their standard errors, ranking measures and the between-trial heterogeneity using the different models and investigated if differences in the results were modified by network characteristics. In general, we observed good agreement with respect to the odds ratios, their standard errors and the ranking metrics between the two Bayesian CSMs. However, differences were observed when comparing the frequentist CSM and the ASMs to each other and to the Bayesian CSMs. The network characteristics that we investigated, which represented the connectedness of the networks and rareness of events, were associated with the differences observed between models, but no single factor was associated with the differences across all of the metrics. In conclusion, we found that different models used to synthesize evidence in a network meta-analysis (NMA) can yield different estimates of odds ratios and standard errors that can impact the final ranking of the treatment options compared.

Systematic reviews are essential for evidence-based health care, but conducting them is time- and resource-consuming. To date, efforts have been made to accelerate and (semi-)automate various steps of systematic reviews through the use of artificial intelligence (AI) and the emergence of large language models (LLMs) promises further opportunities. One crucial but complex task within systematic review conduct is assessing the risk of bias (RoB) of included studies. Therefore, the aim of this study was to test the LLM Claude 2 for RoB assessment of 100 randomized controlled trials, published in English language from 2013 onwards, using the revised Cochrane risk of bias tool (‘RoB 2’; involving judgements for five specific domains and an overall judgement). We assessed the agreement of RoB judgements by Claude with human judgements published in Cochrane reviews. The observed agreement between Claude and Cochrane authors ranged from 41% for the overall judgement to 71% for domain 4 (‘outcome measurement’). Cohen’s κ was lowest for domain 5 (‘selective reporting’; 0.10 (95% confidence interval (CI): −0.10–0.31)) and highest for domain 3 (‘missing data’; 0.31 (95% CI: 0.10–0.52)), indicating slight to fair agreement. Fair agreement was found for the overall judgement (Cohen’s κ: 0.22 (95% CI: 0.06–0.38)). Sensitivity analyses using alternative prompting techniques or the more recent version Claude 3 did not result in substantial changes. Currently, Claude’s RoB 2 judgements cannot replace human RoB assessment. However, the potential of LLMs to support RoB assessment should be further explored.

Limited evidence on relative effectiveness is common in Health Technology Assessment (HTA), often due to sparse evidence on the population of interest or study-design constraints. When evidence directly relating to the policy decision is limited, the evidence base could be extended to incorporate indirectly related evidence. For instance, a sparse evidence base in children could borrow strength from evidence in adults to improve estimation and reduce uncertainty. In HTA, indirect evidence has typically been either disregarded (‘splitting’; no information-sharing) or included without considering any differences (‘lumping’; full information-sharing). However, sophisticated methods that impose moderate degrees of information-sharing have been proposed. We describe and implement multiple information-sharing methods in a case-study evaluating the effectiveness, cost-effectiveness and value of further research of intravenous immunoglobulin for severe sepsis and septic shock. We also provide metrics to determine the degree of information-sharing. Results indicate that method choice can have significant impact. Across information-sharing models, odds ratio estimates ranged between 0.55 and 0.90 and incremental cost-effectiveness ratios between £16,000–52,000 per quality-adjusted life year gained. The need for a future trial also differed by information-sharing model. Heterogeneity in the indirect evidence should also be carefully considered, as it may significantly impact estimates. We conclude that when indirect evidence is relevant to an assessment of effectiveness, the full range of information-sharing methods should be considered. The final selection should be based on a deliberative process that considers not only the plausibility of the methods’ assumptions but also the imposed degree of information-sharing.

The current meta-analytic structural equation modeling (MASEM) techniques cannot properly deal with cases where there are multiple effect sizes available for the same relationship from the same study. Existing applications either treat these effect sizes as independent, randomly select one effect size amongst many, or create an average effect size. None of these approaches deal with the inherent dependency in effect sizes, and either leads to biased estimates or loss of information and power. An alternative technique is to use univariate three-level modeling in the two-stage approach to model these dependencies. These different strategies for dealing with dependent effect sizes in the context of MASEM have not been previously compared in a simulation study. This study aims to compare the performance of these strategies across different conditions; varying the number of studies, the number of dependent effect sizes within studies, the correlation between the dependent effect sizes, the magnitude of the path coefficient, and the between-studies variance. We examine the relative bias in parameter estimates and standard errors, coverage proportions of confidence intervals, as well as mean standard error and power as measures of efficiency. The results suggest that there is not one method that performs well across all these criteria, pointing to the need for better methods.

When conducting a systematic review, screening the vast body of literature to identify the small set of relevant studies is a labour-intensive and error-prone process. Although there is an increasing number of fully automated tools for screening, their performance is suboptimal and varies substantially across review topic areas. Many of these tools are only trained on small datasets, and most are not tested on a wide range of review topic areas. This study presents two systematic review datasets compiled from more than 8600 systematic reviews and more than 540000 abstracts covering 51 research topic areas in health and medical research. These datasets are the largest of their kinds to date. We demonstrate their utility in training and evaluating language models for title and abstract screening. Our dataset includes detailed metadata of each review, including title, background, objectives and selection criteria. We demonstrated that a small language model trained on this dataset with additional metadata has excellent performance with an average recall above 95% and specificity over 70% across a wide range of review topic areas. Future research can build on our dataset to further improve the performance of fully automated tools for systematic review title and abstract screening.

Unanchored population-adjusted indirect comparisons (PAICs) such as matching-adjusted indirect comparison (MAIC) and simulated treatment comparison (STC) attracted a significant attention in the health technology assessment field in recent years. These methods allow for indirect comparisons by balancing different patient characteristics in single-arm studies in the case where individual patient-level data are only available for one study. However, the validity of findings from unanchored MAIC/STC analyses is frequently questioned by decision makers, due to the assumption that all potential prognostic factors and effect modifiers are accounted for. Addressing this critical concern, we introduce a sensitivity analysis algorithm for unanchored PAICs by extending quantitative bias analysis techniques traditionally used in epidemiology. Our proposed sensitivity analysis involves simulating important covariates that were not reported by the comparator study when conducting unanchored STC and enables the formal evaluating of the impact of unmeasured confounding in a quantitative manner without additional assumptions. We demonstrate the practical application of this method through a real-world case study of metastatic colorectal cancer, highlighting its utility in enhancing the robustness and credibility of unanchored PAIC results. Our findings emphasise the necessity of formal quantitative sensitivity analysis in interpreting unanchored PAIC results, as it quantifies the robustness of conclusions regarding potential unmeasured confounders and supports more robust, reliable, and informative decision-making in healthcare.

The importance of network meta-analysis (NMA) methods for time-to-event (TTE) that do not rely on the proportional hazard (PH) assumption is increasingly recognized in oncology, where clinical trials evaluating new interventions versus standard comparators often violate this assumption. However, existing NMA methods that allow for time-varying treatment effects do not directly leverage individual events and censor times that can be reconstructed from Kaplan–Meier curves, which may be more accurate than discrete hazards. They are also challenging to implement given reparameterizations that rely on discrete hazards. Additionally, two-step methods require assumptions regarding within-study normality and variance. We propose a one-step fully Bayesian parametric individual patient data (IPD)-NMA model that fits TTE data with the exact likelihood and allows for time-varying treatment effects. We define fixed or random effects with the following distributions: Weibull, Gompertz, log-normal, log-logistic, gamma, or generalized gamma distributions. We apply the one-step model to a network of randomized controlled trials (RCTs) evaluating multiple interventions for advanced melanoma and compare results with those obtained with the two-step approach. Additionally, a simulation study was performed to compare the proposed one-step method to the two-step method. The one-step method allows for straightforward model selection among the “standard” distributions, now including gamma and generalized gamma, with treatment effects on either the scale alone or with multivariate treatment effects. Generalized gamma offers flexibility to model U-shaped hazards within a network of RCTs, with accessible interpretation of parameters that simplifies to exponential, Weibull, log-normal, or gamma in special cases.