Objective: To explore the impact of methodologic issues onthe results of meta-analyses. The following issues were examined:the type of literature search strategy used; inclusion or exclusionof non–peer-reviewed studies; the inclusion or exclusion of non-English language publications; the effect of trial quality; and theinclusion or exclusion of non–placebo-controlled studies.

Methods: The International Study of Perioperative Transfusion (ISPOT) meta-analyses were used to evaluate each of the methodologic issues. The 10 meta-analyses consisted of technologies to reduce the need for perioperative red blood cell transfusion. The number of trials for each of the meta-analyses varied from 2 to 45. Both EMBASE and MEDLINE searches were conducted, including the use of systematic search strategies.

Results: MEDLINE identified the vast majority of trials. Alone,MEDLINE would have missed 8 studies compared to 10 for EMBASE. Use ofthe systematic search strategies greatly reduced the number ofarticles to be reviewed compared to open searches. Type ofpublication, country of study origin, inclusion of non-Englishpublications, and trial quality had very little impact on theestimates of effect. The use of placebo versus open-label controlaffected the magnitude of the odds ratio for two of the meta-analyses. The results of the two meta-analyses were not statisticallysignificant if only placebo-controlled trials were included.

Conclusions: While methodologic issues had very little impacton the ISPOT meta-analyses, further studies are needed in a varietyof other clinical settings. Because MEDLINE, coupled with a reviewof the references in the identified trials, identified the vastmajority of trials, one needs to consider the costs and benefits ofsearching EMBASE and the pursuance of unpublished and unindexed trials.