Scientific Resource Center for the Effective Health Care Program Evelyn P. Whitlock, M.D., M.P.H. Oregon Evidence-based Practice Center Oregon Health & Science University Kaiser Permanente Center for Health Research Portland, OR 97239 June 26, 2006 Purpose • Overview role of Scientific Resource Center (SRC) in EHC program • Outline a current methodological issue— appropriate use of observational studies--in conducting systematic reviews, including comparative effectiveness reviews (CERs) • Overview current thinking and guidance about observational studies in CERs Effective Health Care Program Review and synthesize knowledge. •The Evidence-based Practice Centers systematically review published and unpublished scientific evidence Promote and generate knowledge. •The DEcIDE Research Network studies new scientific evidence and analytic tools in an accelerated and practical format Compile the findings and translate knowledge. •The Eisenberg Communications and Decision Sciences Center compiles the research results into a variety of useful formats for stakeholders DHHS AHRQ EHP Scientific Resource Center for EHP DEcIDE NETWORK COMPARATIVE EFFECTIVENESS REVIEWS EPC EPC EPC DEcIDE A DEcIDE B A B C EPC EPC EPC DEcIDE C DEcIDE D D E F EPC EPC EPC DEcIDE E DEcIDE F G H I Research Generation Research Synthesis DECISION SCIENCES CENTER Research Translation Scientific Resource Center for the Effective Health Care Program The SRC supports the EHC Program as a whole, with specific responsibilities to: •Communicate with stakeholders •Coordinate public input on Priority Conditions and Research Topics •Assist with the development of research topics and key questions •Coordinate peer review and public input for comparative effectiveness reviews (CERs) •Provide technical assistance to EPCs conducting (CERs) •Support methodological progress in effectiveness reviews and research projects Early CER Report Goals • Reports written with user in mind – SRC responsible for insuring reports have • Clear, concise findings • Consistent layout • Transparent methodology • Methodological validity and consistency • Appropriate application of methodologies – Direct versus Indirect Evidence – Effectiveness versus Efficacy – Observational data 2 extreme views about observational studies • Observational studies aren’t useful. RCTs are the gold standard and the only valid design for ―truth‖ • Observational study evidence trumps RCT evidence. RCTs are not applicable to real-world practice Inconsistencies using observational studies across systematic reviews I. Terminology II. Uses and criteria for inclusion III. Quality assessment Norris S, Atkins D. Annals of Internal Medicine 2005;142:1112-1119. I. Terminology • ―Observational‖ studies: Lack investigator allocation to an intervention • Studies not trials • Rely entirely on studies of association • Include: case series, cross-sectional, case- control & cohort studies, before and after, time-series, database studies, historical controls • ―Non-randomized studies‖ is broader term II. Uses of non-randomized studies in systematic reviews • Design hierarchy in EBM (RCT the highest design) • Some health care effectiveness research demands non-RCT (NRS) approaches – RCTs aren’t ethical, efficient or practical (surgery, adverse effects, public health interventions, organizational change) • Some questions aren’t answerable with RCTs – Etiology – Prognosis – Prevalence – Compliance All NRS designs are not equal • Experimental (two or more groups) – Quasi-randomized trial, non-randomized or controlled clinical trial • Single group designs • Treatment-defined comparison groups – Historical or non-concurrent cohort study, prospective or retrospective cohort studies • Outcome-defined comparison groups – Prospective (nested) or retrospective case-control studies When considering including a NRS design, you must define • Which designs are acceptable given the research question and field of inquiry – Surgery/emerging technology, adverse effects, public health, org change (EPOC) – Type of designs possible • When their use will produce reasonably unbiased results – Critical quality issues – Expected effect size Case example: EPOC reviews • Randomised controlled trials (RCTs) • Controlled clinical trials (CCTs) • Controlled before-and-after studies (CBAs) • Interrupted time series (ITS) Contents of EPOC specialised register by study design Number of studies in register: 1400+ ITS CBA 8% RCT 18% 68% CCT 6% Non-randomised studies in EPOC reviews Number of studies in 15 reviews: 180 CBA ITS 4% 12% CCT 1% RCT 83% III. Quality Assessment: Factors to consider when evaluating a NRS design • Design-specific threats to validity • Review-specific threats to validity Empiric Guidance on NRS vs RCT evidence – Well-designed cohort or case-control & RCTs have similar effect sizes (24 clinical topics). Concato et al. Benson et al NEJM 2000;342:1887-92. – ―Strong evidence that quasi-randomized trials provided biased effect size estimates of about 30%‖--at least for medical Rx. Cochrane NRSG approach (http://www.cochrane.dk/nrsmg) – Results of RCTs and NRS sometimes but not always differ. Deeks. HTA report. 2003;Vol 7:no. 27. NRS vs. RCTs • Observational studies have at times been misleading in formulating RCTs. Kramer B et al. NIH Report: Being smart about clinical trials. 2005. • Highly cited clinical research can be modified or overturned (more likely with NRS than RCTs). Ioannadis JP. JAMA 2005;294:218-228. Main uses of NRS designs in comparative effectiveness reviews (CERs) • Clinical situations not suited to trial investigation • Extending findings of available RCTs • Real-world outcomes(do effect sizes in efficacy trials hold up?) • Persistence and adherence Clinical situations commonly without trials • Off-label use • Contraindications • Proven effect • Cultural barriers Extending findings from available RCTs • Population limitations (homogeneous, limited co-morbidities, unstudied vulnerable groups) • Small sample sizes (adverse events, rare events) • Short follow-up (maintenance of benefits, adverse events) • Important outcomes not available (patient priorities, long-term effects, natural history/background rate) Real-world outcomes • Applicability (how generalizable are trials to real-world practice?) – Incorporates community clinical practice and adherence/persistence issues Comparing control of bias in RCT & NRS Source of bias RCTs Cohort/other NR Comments studies Selection bias Randomi- Control for Many other zation confounders study-specific threats Performance Double- Exposure Misclassifica- bias blinding measurement tion/ non- (Exposure) comparability Attrition bias Complete Complete F/U What amount is F/U critical? Detection bias Masked Masked Misclassifica- (Outcome) assessmt assessmt tion Types of Selection Bias & their control in NR Studies Source of Bias Consideration • How patients are • Sampling/selection identified criteria • How allocation • Confounding by occurred/who chose indication • Attempts to balance • Baseline comparability & allocated groups design features • Other prognostic • ID and control of factors confounders Current methodological issues for NRS in CERs • Locating studies • Appropriate inclusions/exclusions • Reporting (STROBE) • Quality rating • Defining minimum data sets for applicability information • Synthesizing • applicability & other observational data • effectiveness data from both RCTs and NRS • Determining effectiveness from NRS (subgroups) RCT Figure: Considerations in using nonRCT/CCTs in Always consider CCT systematic reviews of healthcare interventions if available Other designs: IF WHEN -Observational -Non-observational Should non-RCT/CCTs be considered for a CER? When should specific studies be used? Consider: Etiologic hypotheses: Yes 1. Study design Adverse events: Yes 2. Quality Efficacy/effectiveness - Consider: 3. Precision/sample size 4. Inception cohort 1. Key question - Consider probability of selection bias or confounding 2. Body of RCT/CCT evidence - If inadequate for population & comparator characteristics HOW of interest → include* - If adequate → consider How should non-RCT/CCTs be used? *Situations where relevant RCTs may be insufficient in number: How should data be synthesized? 1. RCTs unnecessary 4. RCTs difficult to implement - Pooling of non-RCT data - Obvious benefit - Rare outcome - FE vs RE models 2. RCTs inappropriate - Need long-term follow-up - Pooling across designs - Clinical equipoise - Randomization →↓ benefit - Patient preferences - Unit randomization difficult - Policy/political/legal objections - Potential contamination 3. RCTs inadequate of control group How present and use these data? - Applicability: populations of - Complex intervention - Relationship/comparison to interest not in RCT/CCTs - Hazardous intervention - Infeasible intervention RCT/CCTs Categories based on Black 1996 - Conclusions - Potential biases - Generalizability SL Norris 3,13,06 Non-randomized studies (NRS): the current bottom line • For reviews, use the best available evidence • NRS may be the best available • NRS designs are not equal & studies within a particular design are not equal • Internal validity (truth) issues vs. external validity issues (relevance) is a balance • Analyses within and across RCTs are really NRS: – Subgroup analyses – Meta-analyses Effective Health Care Program www.effectivehealthcare.ahrq.gov www.Strobe-statement.org Evelyn.firstname.lastname@example.org Minimum criteria for including CBAs or ITSs in EPOC reviews • CBAs – concurrent data collection both before and after the intervention – comparable second site as a control • ITSs – minimum of three data points before and after the intervention Additional quality criteria for CBAs • Baseline comparability • Protection against contamination Additional criteria for ITS designs • Intervention unlikely to affect data collection • Sufficient data points for statistical inference • Intervention independent of other changes Methods for Comparative Evidence Reviews Marian McDonagh, PharmD Oregon Evidence-based Practice Center Oregon Health & Science University CER Topic Selection • Related to 1 or more of the 10 Priority Conditions – Suggestions for CERs (EPCs) – Suggestions for new research (DEcIDEs) • Topics proposed through public process – Federal register – Web site (www.effectivehealthcare.ahrq.gov) • Initial topic selection – SRC does preliminary search for existing reviews or research – SRC develops draft key questions CER Topic Selection • List of topics brought to Stakeholder group for comment – Prioritize and determine category: CER or DEcIDE • Final Topic Selection – SRC revises draft key questions – Final topics and questions submitted to AHRQ – Topics assigned to EPCs or DEcIDEs Current CER Topics Arthritis and non-traumatic joint disorders 1. Evidence for Benefits and Safety of Analgesics 2. Evidence for Comparative Short- and Long- Term Benefits, Harms, and Outcomes of bis- phosphanates, Combined Exercise, Diet, and High-Dose Calcium, and Selective Estrogen Receptor Modifier for Low Bone Density Current CER Topics Cancer 1. New Diagnostic Technologies for Evaluation of Abnormal Breast Cancer Screening 2. Effectiveness of Epoetin and Darbepoetin for Managing Anemia in Patients Undergoing Cancer Treatment 3. Evidence for Comparative Short- and Long- Term Benefits, Risks, and Outcomes of Therapies for Localized Prostate Cancer Current CER Topics Depression and other mood disorders 1. Evidence for Off-Label Use of Atypical Anti- Psychotic Medications 2. Comparative Effectiveness of Pharmacotherapeutics for Depression Diabetes mellitus 1. Evidence for Comparative Effectiveness of Newer Oral Agents vs. Older Agents (e.g., metformin and sulfonylureas) for Initial Therapy for Diabetes (Oral Hypoglycemics) Current CER Topics Peptic ulcer disease and dyspepsia 1. Comparative Effectiveness of Management Strategies for Gastroesophageal Reflux Disease Stroke and hypertension 1. Evidence for Renal Artery Stenting vs. Aggressive Anti-hypertensive Medical Therapy for Mild Renal Artery Stenosis Priority Conditions without CER Topics Assigned •Chronic obstructive pulmonary disease and asthma •Dementia including Alzheimer's disease •Ischemic heart disease •Pneumonia AHRQ selects topics and posts RFTO EPC responds to RFTO EPC to conduct CER selected by AHRQ Topic completion Conference Call with TOO, EPC and RC EPC assembles expert advisors as needed RC completes FDA and other searches, collects timeline industry submissions EPC conducts sesarches, Reviews literature from searches, retrieves articles Receive dossiers, FDA documents etc from RC RC works with EPC to identify peer reviewers; forward list to TOO TOO approves or modifies peer reviewer list EPC begins report Submit Draft report to RC and send for peer review RC Peer Review Draft Report RC posts for Public Comment EPC receives comments from RC RC summarizes research gaps and forward to next Stakeholder meeting Receive peer review comments - notify RC if serious concerns raised Public comments forwarded to EPC Receive Public Comment. Incorporate and summarize public and peer review comments Submit Final report to RC along with summary of comments and responses Receive and post Final CER week# -3 -1 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 Key Question Development • Questions relate to: – Effectiveness – Harms and Tolerability – Sub-populations • Age, Race/ethnicity, Gender, Co-morbidities • Inclusion Criteria based on key questions • Draft Questions (SRC) – Stakeholder group for comment – Publicly posted for comment • Final Questions – Go to EPC CER Report Processes • Draft reports undergo – Peer Review – Public Comment via web site – Scientific Resource Center review – Stakeholder group for comment • Every report includes – Executive summary – Summary table by Key Question – Every report will have a slide show • Final Reports posted to web site – Reports to be updated as needed – SRC to monitor literature to determine need for update Other challenges with using ―observational‖ studies • Finding them all • Using what you find (reporting) • Kuper H et.al. BMC Medical Research Methodology 2006,6:4. http://www.biomedcentral.com/1471-2288/6/4. • Stroup et.al. JAMA 2000;283:2008-2012.
Pages to are hidden for
"Comparative Effectiveness Research The New Frontier III"Please download to view full document