Search Strategies to Identify Randomized Trials in MEDLINE

Search Strategies to Identify Randomized Trials in MEDLINE A Systematic Review Ellen Crumley, MLIS EBL 2003 Conference 6 June 2003, Edmonton Alberta Research Centre for Child Health Evidence Structured Abstract      Background: Search strategies for identifying trials have to balance the need to locate all relevant studies (recall) with the practical consideration of retrieving a manageable number of records (precision). Searching is dependent upon the resources to which a researcher has access and the terms used in the strategies can be influenced by knowledge of content area and the person conducting the search. Objective: To systematically review studies which have compared two or more search strategies to find reports of randomised trials in MEDLINE. Methods: In January 2003, the Cochrane Methodology Register, MEDLINE, EMBASE, CINAHL, ISI’s Web of Science, and Library and Information Science Abstracts were searched to identify relevant studies irrespective of language. Eligible studies compared two different search strategies for identifying randomised trials in MEDLINE. Search strategies were categorized as either simple or complex. The main outcome measure was the comparison of recall and precision for each strategy. Information was also collected on the years covered by the search, and methodological quality of the study. Results: From 54 potential studies, 12 met the inclusion criteria. Recall ranged from 9% to 93% (simple searches) and 20% to 100% (complex searches). Precision ranged from 7% to 88% (simple) and 2% to 60% (complex). The 3-phase Cochrane search strategy had a recall from 18-99.6% and a precision ranging from depending upon the content area. Conclusions: Overall, searches using both keywords and mesh had better precision and recall. There is no guideline for whether to use recall or precision to guide how a search should be conducted and this remains an individual and content specific decision. The most popular terms used to search for RCT/CCTs were: random either truncated or in combination with trial (N=23), random allocation [MeSH] (N=9), RCT as a publication type (N=10), trial as a keyword or truncated (N=14) and double blind or control* used in combination with trial (N=8 for both terms). Using randomized controlled trial as a publication type was not adequate to ensure retrieval of at least the major trials in a specific area. There needs to be further research into searching other resources such as CENTRAL in the Cochrane Library, EMBASE and CINAHL to determine their contribution to the pool of available RCT/CCTs already available in MEDLINE. Systematic Review Process       What is a systematic review / meta-analysis RCT/qRCT/CCT WHERE to search HOW to search WHICH search strategies/terms How extensive? Searching is an Art     Subjective Confounding Selection and Misclassification Bias Example Objective  To complete a systematic overview of studies which compare at least two different search strategies for RCTs in the MEDLINE database. Methods 1. 2. 3. 4. 5. 6. Comprehensive search (215) Screening (54) Inclusion/exclusion (12) Quality assessment Data extraction Write-up (MOOSE) Inclusion Criteria Compared or tested two or more different search strategies for identifying RCTs in MEDLINE AND  Included recall and/or precision or the information to calculate one or both of these  Exclusion Criteria     Did not use MEDLINE (e.g., PubMed) Did not include RCTs No data provided from which the recall and/or precision can be calculated Search strategy compared between different databases but not within MEDLINE Recall  the number of relevant studies retrieved the total number of relevant studies x 100% Recall R e t r i e v e d Relevant Yes No a c b d Yes 20 80 a a+c = 20% = 20 20 + 80 No Precision  the number of relevant studies retrieved the total number of studies retrieved x 100% Precision R e t r i e v e d Relevant Yes No Yes a a+b = 67% = 20 20 + 10 20 a b 10 No c d Results - Study Characteristics       12 studies included (9 articles, 3 abstracts) All English UK & US = 4; Canada, Denmark, Australia & Netherlands = 1 each 1988-2001 Study objective 11 content areas Content Areas       NSAIDs for rheumatoid arthritis Ophthalmology, optometry Liver diseases Newborn diseases Prosthodontics Epilepsy  Aids Cognitive behavior therapy for major depressive disorders     Obstetrical anesthesia Pain relief High grade glioma Reporting Flaws      MEDLINE software Years MEDLINE searched Year search conducted RCT definition MOOSE/CONSORT Simple & Complex Searches     Definitions 18 simple 21 complex (includes Cochrane strategy) Equal distribution of content areas Recall vs. Precision: Simple Searches 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Rheumatoid arthritis General health care Chemotherapy Sensitivity Precision 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Searches Recall vs. Precision: Complex Searches 120.0% 100.0% Liver disease 80.0% Sensitivity Precision 60.0% 40.0% 20.0% 0.0% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Searches Cochrane Search Strategy        5 articles, 3-phase strategy only Epilepsy: recall 86%, precision 35% Pain relief: recall 18%, precision 9% Gen. Med Journals: recall 92.7%, precision 21% Chemotherapy: recall 18%, precision 12% Pain relief: recall 99.6% Depression: recall 97%, precision 35% Search Terms      MeSH: Comparative study, double blind method, random allocation, randomized controlled trials Keywords: double blind, random$, trial$ Publication type: randomized controlled trial Cochrane Search Strategy Subheadings, Tag MeSH Terms     Comparative study (6) Double blind method (5) Random allocation (9) Randomized controlled trials (1) Keywords or Textwords      Random$ (23) Trial$ (14) Double blind$ (8) Control$ or controlled trial$ (8) Compar$ (6) Publication Type   Randomized controlled trial (10) Clinical trial (4) Why Studies Missed in MEDLINE       Inadequate, improper or selective indexing Keywords/methodology not reported by author Journal not indexed in database Article/issue omitted or missing in database Poorly written abstract or wrong abstract Article published before database created Why Searches Missed Studies      Limited use of free text methodologic terms Limited use of truncation Insufficient or restricted search strategy No age category assigned Both subject headings and text words not used Discussion    Is the difference in recall/precision due to the subject or to the search strategy? How can optimal recall and precision be achieved? What are the best terms to use when searching for RCT/CCTs? Study Authors’ Conclusions    The MeSH heading comparative study did better than using double-blind as a textword or subject heading One study’s 100% sensitive strategies (RCT.pt & random$) had “prohibitively low precision” “The search strategies developed by the Cochrane Collaboration failed to find all the RCTs indexed in the database”. Study Authors Conclusions   Using clinical trial and randomized controlled trials as publication types were “the most productive searches” Recall increased from 53% to 65% and precision decreased from 99% to 78% when randomized controlled trials and random allocation[MeSH] vs. keywords randomized and random* were compared. Conclusions     Inconclusive evidence Next steps How would you apply this to next systematic review you search for? FURTHER RESEARCH ABSOLUTELY NECESSARY