What is already known?

• • Umbrella reviews (URs) aim to integrate and analyze findings from multiple systematic reviews on a given topic.

• • URs are a popular type of evidence synthesis.

• • Limited understanding of the methodologies used to analyze and present findings of URs exist.

What is new?

• • This cross-sectional study of a sample of 100 URs showed a strong emphasis on synthesizing quantitative data with limited examples of URs containing qualitative data located.

• • Software used in analysis and presentation is often not reported.

• • Many URs do not address primary study overlap nor assess certainty of the evidence.

Potential impact for Research Synthesis Methods readers

• • Reviewers conducting URs should assess primary study overlap, assess certainty of the evidence, and report software used for analysis and presentation of data.

3.1 Search strategy

The search for URs was conducted in PubMed (JL) with a time frame between March 5, 2023 and March 5, 2024. PubMed was selected as the sole database to answer the research question because it provides extensive coverage of biomedical literature relevant to this study, including the majority of key healthcare journals. This 1-year time period was chosen to ensure the feasibility and currency of the project. As there are no subject headings, subsets, or search filters for URs currently available in PubMed, the search strategy developed by Pieper and colleaguesReference Pieper, Antoine, Mathes, Neugebauer and Eikermann⁸ was adapted during consultation with a librarian (RJ) (see Appendix A).

3.2 Inclusion and exclusion criteria

Umbrella reviews with a focus on health were included. Reviews were restricted to the English language due to team capacity. For the purpose of this project, the focus was exclusively on URs that contained SRs or meta-analyses as their unit of analysis; reviews including scoping reviews, literature reviews, and primary studies in addition to SRs were excluded. Methodological reviews were also excluded.

3.3 Sample size

This cross-sectional study aimed to include 100 URs which were randomly selected from the search results using a random number generator.Reference Furey¹⁵ The sample size was agreed upon by the project team as it was considered sufficient to address the research question while remaining feasible with the available resources.

3.4 Screening and selection

Search results were imported into Excel and screened at title and abstract by two independent screeners (JL and JD) with any disagreements resolved by a third reviewer (CS). A random number generator was then used to select 100 records which were then imported into Covidence (an online platform used to conduct systematic reviews). It was anticipated that some records may not meet inclusion criteria at full-text review to get to the agreed sample size so an additional 20 records were randomly selected and set aside to use following full-text review if required. Full-text screening was undertaken by five screeners (CS, RF, JL, PB, and HK) and a pilot on five records was conducted to ensure consistency between screeners.

3.5 Data extraction and analysis

A data extraction form (Appendix B) was created and revised following group discussion. A pilot test was undertaken to assess the robustness of the data extraction tool, and relevant changes were subsequently made. Eight URs were included in the pilot phase, two more than originally specified in the review protocol. This modification was introduced to accommodate additional study team members, with each pair piloting two different papers. Data focusing on bibliographical information, descriptive characteristics of the URs (e.g., methodological and reporting guidance followed, critical appraisal tool used), and analysis and presentation methods (e.g., certainty of evidence method used and software used) were extracted. While the protocol specified that all extractions would be undertaken in duplicate, it was decided that extractions would be conducted by one author and a sample of extractions validated by another author (JL or CS) due to the type of data being extracted. Covidence was employed as a tool to manage the extraction process. In total, 20 extractions were verified representing 20% of included reviews. Any disagreements were addressed by discussion.

A descriptive analysis of the results, including tabular presentation and narrative, was conducted. Where appropriate, frequency distributions were presented, with percentages calculated and rounded to the nearest whole number. The mean, standard deviation, and median were calculated for the number or reviews included in the URs.

4.1 Study inclusion

The PubMed search was conducted on April 30, 2024, resulting in 1183 records identified and title and abstracts screened for eligibility. Following screening, 728 records remained and using the random number generator,Reference Furey¹⁵ 100 records were selected and screened at full text. Of those, 12 were excluded (11 did not exclusively include SRs or meta-analyses as their unit of analysis and one was not an UR), so a further 12 records were selected from the additional set of 20 records (described previously). One hundred URs were included in the study (see Figure 1). A table of UR characteristics is presented in Supplementary Material 1, a summary table of the results is presented in Supplementary Material 2, and the citation details of the reviews is included in Supplementary Material 3.

Figure 1 Study identification and inclusion flow chart.

4.2 Review nomenclature

Seventy-one umbrella reviews were published in 2023 and 29 in 2024. Topics were diverse across the health spectrum. Most included reviews referred to themselves as an “umbrella review” (n = 65, 65%); this was followed by an “overview” (n = 30, 30%). Seven reviews (7%) used variations of the term “systematic review” (e.g., systematic review of systematic reviews, systematic review of reviews, systematic review of meta-analyses), and two reviews used the term “rapid review” (e.g., rapid review of reviews, rapid review of systematic reviews). Some reviews used multiple terms within the article.

4.3 Framework for question formulation

The majority of URs (n = 66, 66%) referred to either a protocol being published or publicly available or the UR being registered. Over half of the included URs utilized a PICO format (Population, Intervention, Comparator, Outcome) for their question development and/or eligibility criteria (n = 43 [43%] PICO, n = 14 [14%] PICOS [Study design or Setting], n = 1 [1%] PICOTS [Time]). Eight percent of URs (n = 8) used a version of the Population, Exposure, Comparator, Outcome (PECO) format (PECO [n = 1, 1%], PECOS [Study design] [n = 1, 1%], PEO [n = 6, 6%]), while four URs (4%) used PICo (Population, Interest, Context). Two URs (2%) used PCC (Population, Concept, Context) and one UR (1%) used SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research Type). Population, Outcomes, and Comparator were used once (1%), as were Population and Phenomena of interest (1%). A quarter of URs did not explicitly specify the framework used (n = 25, 25%).

4.4 Review composition

The number of SRs included in 99 URs ranged from 4 to 175 with a mean of 24.3 (standard deviation [SD] = 24.5) and a median of 16. One UR did not report the number of SRs but instead provided the total number of RCTs included in all SRs in the UR.

The majority of the URs included quantitative SRs (n = 98, 98%), either exclusively or alongside qualitative or mixed method SRs. Seven percent of URs (n = 7) also included meta-analyses, while one (1%) also included a network meta-analysis.

Of the URs that contained qualitative evidence, most were from mixed methods SRs (n = 7, 7%) with only 2 URs located that exclusively included qualitative SRs (n = 2, 2%). Additionally, one UR (n = 1) included narrative reviews (n = 1, 1%).

Considering the designs of primary studies included in the SRs, the most frequently mentioned were randomized controlled trials (RCTs) in 68% of URs (n = 68). This was followed by cohort studies and case–control studies, included in 36 (36%) and 26 (26%) of URs, respectively. Additionally, quasi-experimental studies and cross-sectional studies were reported in 24% (n = 24) and 20% (n = 20) of the URs, respectively. Other designs included in the SRs were case series/reports (n = 12, 12%), prevalence studies (n = 3, 3%), and mixed methods studies (n = 4, 4%). Six URs (6%) included qualitative studies. Notably, 12 URs (12%) did not mention what study designs were included, representing a lack of methodological detail.

4.5 Methodological and reporting guidance

In 55 URs (55%), there was no mention of methodological guidance followed. Cochrane methodological guidanceReference Pollock, Fernandes, Becker, Pieper and Hartling⁵ was referred to in 32 URs (32%), while 14 used JBI guidanceReference Aromataris, Fernandez, Godfrey, Holly, Khalil and Tungpunkom⁴ (14%). One UR cited both Cochrane and JBI methodological guidance (1%), while three reviews cited Cochrane guidance and other sources (3%).

In terms of reporting guidance, the majority of reviews stated they followed the Preferred Reporting Items for Systematic Reviews and Meta-AnalysesReference Page, McKenzie, Bossuyt, Boutron, Hoffmann and Mulrow¹⁶ (PRISMA) reporting standards (n = 73, 73%), followed by the Preferred Reporting Items for Overviews of ReviewsReference Gates, Gates and Pieper⁹ (PRIOR) (n = 15, 15%) and Preferred Reporting Items for Overviews of Systematic Reviews including harm checklistReference Bougioukas, Liakos, Tsapas, Ntzani and Haidich¹⁷ (PRIO-harms) (n = 3, 3%). Other guidance mentioned included Enhancing Transparency in Reporting the Synthesis of Qualitative ResearchReference Tong, Flemming, McInnes, Oliver and Craig¹⁸ (ENTREQ), the PRISMA extension for diagnostic test accuracyReference McInnes, Moher and Thombs¹⁹ (PRISMA-DTA), Brief Review Checklist (BRC), and the PRISMA extensions for scoping reviewsReference Tricco, Lillie and Zarin²⁰ (PRISMA-ScR), with one review, respectively (1% each). Sixteen percent of reviews (n = 16) made no mention of reporting guidance.

4.6 Primary study overlap

Primary study overlap refers to two of more systematic reviews or meta-analyses included in a UR that share one or more of the same primary studies. Study overlap is a unique methodological challenge in URs and can impact validity and reliability. Primary study overlap was reported in 48 included URs (48%); 21 URs used corrected covered area calculationsReference Pieper, Antoine, Mathes, Neugebauer and Eikermann⁸ (21%), 16 provided a narrative description (16%), and 15 presented a tabular display (15%). The Graphical Representation of Overlap for OVErviewsReference Pérez-Bracchiglione, Meza and Bangdiwala²¹ (GROOVE) was mentioned in two URs and 14 URs (14%) included multiple formats. Of those URs where the overlap of the primary studies was considered and where multiple outcomes were reported, 13 reviews (13%) reported the overlap for individual outcomes.

4.7 Critical appraisal

Critical appraisal is the process of assessing the methodological quality of included systematic reviews or meta-analyses included in a UR and is considered an essential step in the review process. Almost all URs (n = 97, 97%) undertook critical appraisal of the included SRs. The most common instrument used was AMSTAR-2 (A MeaSurement Tool to Assess systematic Reviews)Reference Shea, Reeves and Wells¹¹ (n = 69, 69%), followed by the JBI toolReference Aromataris, Fernandez, Godfrey, Holly, Khalil and Tungpunkom⁴ (n = 10, 10%), ROBISReference Whiting, Savović and Higgins¹² (n = 10, 10%), and the original AMSTARReference Shea, Grimshaw and Wells²² (n = 8, 8%). Other URs referred to various tools such as but not limited to: SANRA (Scale for the Assessment of Narrative Review Articles)Reference Baethge, Goldbeck-Wood and Mertens²³ criteria for narrative reviews, the Health Evidence checklist for systematic reviews,²⁴ and the Cochrane Risk of Bias toolReference Jørgensen, Paludan-Müller and Laursen²⁵ (n = 1, 1%, respectively).

Only four URs (4%) excluded SRs based on methodology quality. The threshold/criteria used for this decision varied, but related to excluding reviews that were considered low or very low methodological quality.

Of them, three used the JBI tool, while one used AMSTAR-2. For the three URs using the JBI tool, the following details were provided:

• • One UR excluded low-quality reviews with low determined if the score was below 40%Reference Sufrate-Sorzano, Pérez and Juárez-Vela²⁶

• • One UR excluded two papers whose quality level was lower than 40%Reference Sufrate-Sorzano, Pérez and Juárez-Vela²⁶

• • One UR planned to exclude studies with a minimum score of 4Reference Reis, Gaspar, Paiva, Sousa and Machado²⁷

The UR that used AMSTAR-2 stated that “3 were rejected due to very low confidence according to AMSTAR 2.”Reference Babot-Pereña and Blanco-Blanco²⁸

4.8 Data analysis

All URs containing quantitative data analyzed data narratively. Of the 98 URs (98%) that analyzed quantitative data only, 30 (30%) included meta-analysis, 27 (27%) utilized basic descriptive analysis, and 3 (3%) used some form of data categorization.

Of the seven URs (7%) that included qualitative data (either from qualitative reviews or mixed methods reviews), seven URs (7%) analyzed data narratively, three included a method of qualitative synthesis (3%), and one each reported basic descriptive analysis (1%) and a process of categorization (1%).

Of the 100, 27 included URs (27%) referred to using software to analyze or present data. Tools used included Stata (n = 12), GRADEPro GDT (n = 5), MS Excel (n = 5), RevMan (n = 5), R (n = 3), Meta-umbrella (n = 2), and Comprehensive Meta-analysis (n = 2). Other software such as NVIVO, SPSS, Cytoscape, MetaXl, VOS Viewer, and STATISTICA were each mentioned once (n = 1).

4.9 Data presentation

While all URs including quantitative data provided narrative to describe their results, eight (8%) exclusively used text for presentation. To support the presentation of the results, the most popular method used was tabular display (n = 76, 78%), followed by forest plots (n = 29, 30%), and Summary of Findings tables (n = 18, 18%). Other approaches included figures (n = 5, 5%), the JBI stop light table (n = 3, 3%), bar charts (n = 3, 3%), and one each used line graphs, heat maps, evidence maps, funnel plots, and radar plots (1%). Over a third of URs used multiple presentation formats in addition to text (n = 41, 41%).

Of the eight URs containing qualitative data, five (63%) utilized tabular display to present their results, two used Summary of Findings (25%), and two used figures (25%).

The majority of URs presented between one and five tables or figures. A small number of URs (n = 2, 2%) reported no tables or figures, while 7% (n = 7) included only one. A larger portion of URs (n = 14, 14%) included two tables or figures, and 19% (n = 19) included three. Most URs used between four (n = 17) and five (n = 20) tables or figures (37% combined), with nine (9%) reporting six tables or figures. Including more than six tables or figures was rare, with only seven URs (7%) using seven or more, including one (1%) UR with 32 tables or figures.

4.10 Data presented in supplementary materials

Of the 50 (50%) URs that included Supplementary Materials, 23 (46%) provided data directly related to the analysis of results. Among these, 16 (32%) were associated with the main analyses, encompassing meta-analyses of primary data (n = 3, 6%), unspecified main analyses (n = 1, 2%), robust variance estimation (RVE) techniques (n = 1, 2%), statistical test results and reported associations (n = 2, 4%), subgroup analyses (n = 2, 4%), and narrative synthesis (n = 2, 4%). Presentation of results was also common, with additional forest plots (n = 7, 14%), funnel plots (n = 1, 2%), and other visual outputs such as figures, charts, or detailed tables (n = 5, 10%). Additionally, seven (14%) URs included assessments of overlap within their Supplementary Materials, while five (10%) provided data related to supplementary analyses, such as sensitivity analyses (n = 2, 4%) and other forms of additional analysis not featured in the main text (n = 3, 6%).

4.11 Certainty assessment

Certainty assessment (confidence assessment is the term often used for qualitative research) refers to determining how trustworthy and applicable a body of evidence is. Just over half of the URs did not assess the certainty of the evidence (n = 52, 52%). Of those that did assess certainty (48%), most used the Grading of Recommendations Assessment, Development, and EvaluationReference Schünemann, Brożek, Guyatt and Oxman²⁹ (GRADE) approach (n = 39, 39%). Other approaches included Confidence in the Evidence from Reviews of Qualitative researchReference Lewin, Booth and Glenton³⁰ (CerQual) (n = 4, 4%) and NutriGradeReference Schwingshackl, Knüppel and Schwedhelm³¹ (n = 2, 2%). Eight URs (8%) referred to using other individual methods with some referencing other documents/criteria (e.g., the SIGN Scale), while others appeared to create their own criteria for assessment (e.g., The overall confidence [high, moderate, low, and critically low] was graded as follows—high: no or one noncritical weakness; moderate: more than one noncritical weakness; low: one critical flaw with or without noncritical weaknesses; and critically low: more than one critical flaw with or without noncritical weaknesses).