Hostname: page-component-745bb68f8f-l4dxg Total loading time: 0 Render date: 2025-01-11T07:45:59.376Z Has data issue: false hasContentIssue false
  • English
  • Français

How much searching is enough? Comprehensive versus optimal retrieval for technology assessments

Published online by Cambridge University Press:  06 October 2010

Andrew Booth
Andrew Booth*
Affiliation:
University of Sheffield
Get access Rights & Permissions [Opens in a new window]

Abstract

Objectives: The aim of this study is to review briefly different methods for determining the optimal retrieval of studies for inclusion in a health technology assessment (HTA) report.

Methods: This study reviews the methodology literature related to specific methods for evaluating yield from literature searching strategies and for deciding whether to continue or desist in the searching process.

Results: Eight different methods were identified. These include using the Capture–recapture technique; obtaining Feedback from the commissioner of the HTA report; seeking the Disconfirming case; undertaking comparison against a known Gold standard; evaluating retrieval of Known items; recognizing the Law of diminishing returns, specifying a priori Stopping rules, and identifying a point of Theoretical saturation.

Conclusions: While this study identified a variety of possible methods, there has been very little formal evaluation of the specific strengths and weaknesses of the different techniques. The author proposes an evaluation agenda drawing on an examination of existing data together with exploration of the specific impact of missing relevant studies.

Keywords

DatabasesBibliographicTechnology assessmentBiomedicalInformation retrievalSelection biasSensitivity and specificity
Type
THEME SECTION: INFORMATION RETRIEVAL FOR HTA
Information
International Journal of Technology Assessment in Health Care , Volume 26 , Issue 4 , October 2010 , pp. 431 - 435
Copyright
Copyright © Cambridge University Press 2010

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

1. Allen, IE, Olkin, I. Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999;282:634–635.Google Scholar
2. Bayliss, SE, Dretzke, J. Health technology assessment in social care: A case study of randomized controlled trial retrieval. Int J Technol Assess Health Care. 2006;22:39–46.CrossRefGoogle ScholarPubMed
3. Bennett, DA, Latham, NK, Stretton, C, et al. Capture-recapture is a potentially useful method for assessing publication bias. J Clin Epidemiol. 2004;57:349–357.CrossRefGoogle ScholarPubMed
4. Booth, A. The number needed to retrieve: A practically useful measure of information retrieval? Health Info Libr J. 2006;23:229–232.Google Scholar
5. Chilcott, J, Brennan, A, Booth, A, et al. The role of modelling in prioritising and planning clinical trials. Health Technol Assess. 2003;7:iii, 1–125.Google Scholar
6. Dickersin, K, Scherer, R, Lefebvre, C. Identifying relevant studies for systematic reviews. BMJ. 1994;309:1286–1291.Google Scholar
7. Egger, M, Juni, P, Bartlett, C, et al. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7:1–76.CrossRefGoogle ScholarPubMed
8. Grant, MJ, Booth, A. A typology of reviews: An analysis of 14 review types and associated methodologies. Health Info Libr J. 2009;26:91–108.Google Scholar
9. Greenhalgh, T, Peacock, R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. BMJ. 2005;331:1064–1065.CrossRefGoogle ScholarPubMed
10. Ioannidis, J. Why most published research findings are false. PLoS Med. 2005;2:e124.CrossRefGoogle ScholarPubMed
11. Kastner, M, Straus, S, Goldsmith, CH. Estimating the Horizon of articles to decide when to stop searching in systematic reviews: An example using a systematic review of RCTs evaluating osteoporosis clinical decision support tools. AMIA Annu Symp Proc. 2007;389–393.Google Scholar
12. Kastner, M, Straus, S, McKibbon, K, et al. The capture-mark-recapture technique can be used as a stopping rule when searching in systematic reviews. J Clin Epidemiol. 2008;62:149–157.Google Scholar
13. Khan, KS, Coomarasamy, A. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ. 2006;6:59.CrossRefGoogle ScholarPubMed
14. Löhönen, J, Isohanni, M, Nieminen, P, et al. Coverage of the bibliographic databases in mental health research. Nord J Psychiatry. 2010;64:181–188.Google Scholar
15. Noyes, J, Popay, J, Pearson, A, et al. Qualitative research and Cochrane reviews. In: Higgins, JPT, S, Green, eds. Cochrane handbook for systematic reviews of interventions. Version 5.0.1 [updated September 2008]. The Cochrane Collaboration, 2008. www.cochrane-handbook.org.CrossRefGoogle Scholar
16. Ogilvie, D, Hamilton, V, Egan, M, et al. Systematic reviews of health effects of social interventions: 1. Finding the evidence: How far should you go? J Epidemiol Community Health. 2005;59:804–808.CrossRefGoogle ScholarPubMed
17. Pirolli, P, Card, S. Information foraging in information access environments. In: Proceedings of the Human Factors in Computing Systems, CHI '95. Association for Computing Machinery, 1995.CrossRefGoogle Scholar
18. Rosenthal, R. The file drawer problem and tolerance for null results. Psychol Bull. 1979;86:638–641.CrossRefGoogle Scholar
19. Rotstein, D, Laupacis, A. Differences between systematic reviews and health technology assessments: A trade-off between the ideals of scientific rigor and the realities of policy making. Int J Technol Assess Health Care. 2004;20:177–183.CrossRefGoogle ScholarPubMed
20. Royle, P, Milne, R. Literature searching for randomized controlled trials used in Cochrane reviews: Rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19:591–603.Google Scholar
21. Sampson, M, Shojania, KG, Garritty, C, et al. Systematic reviews can be produced and published faster. J Clin Epidemiol. 2008;61:531–536.Google Scholar
22. Shojania, KG, Sampson, M, Ansari, MT, et al. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147:224–233.Google Scholar
23. Spoor, P, Airey, M, Bennett, C, et al. Use of the capture-recapture technique to evaluate the completeness of systematic literature searches. BMJ. 1996;313:342–343.CrossRefGoogle ScholarPubMed
24. Stevinson, C, Lawlor, D. Searching multiple databases for systematic reviews: Added value or diminishing returns? Complement Ther Med. 2004;12:228–232.Google Scholar
25. Watt, A, Cameron, A, Sturm, L, et al. Rapid reviews versus full systematic reviews: An inventory of current methods and practice in health technology assessment. Int J Technol Assess Health Care. 2008;24:133–139.CrossRefGoogle ScholarPubMed
26. Watt, A, Cameron, A, Sturm, L, et al. Rapid versus full systematic reviews: Validity in clinical practice? ANZ J Surg. 2008;78:1037–1040.Google Scholar
27. Weale, AR, Lear, PA. Randomised controlled trials and quality of journals. Lancet. 2003;361:1749–1750.Google Scholar