Comparative Effectiveness Research The New Frontier III by sammyc2007

VIEWS: 103 PAGES: 43

									  Scientific Resource Center
             for the
Effective Health Care Program
       Evelyn P. Whitlock, M.D., M.P.H.
Oregon Evidence-based Practice Center
      Oregon Health & Science University
 Kaiser Permanente Center for Health Research
             Portland, OR 97239

                June 26, 2006

• Overview role of Scientific Resource Center
  (SRC) in EHC program
• Outline a current methodological issue—
  appropriate use of observational studies--in
  conducting systematic reviews, including
  comparative effectiveness reviews (CERs)
• Overview current thinking and guidance
  about observational studies in CERs
     Effective Health Care Program

Review and synthesize knowledge.
•The Evidence-based Practice Centers systematically
review published and unpublished scientific evidence

Promote and generate knowledge.
•The DEcIDE Research Network studies new scientific
evidence and analytic tools in an accelerated and practical

Compile the findings and translate knowledge.
•The Eisenberg Communications and Decision Sciences
Center compiles the research results into a variety of
useful formats for stakeholders

                                 AHRQ EHP

                              Scientific Resource
                                Center for EHP

     DEcIDE NETWORK                                 COMPARATIVE
                                               EFFECTIVENESS REVIEWS

                                                    EPC    EPC    EPC
                                                     A      B      C

                                                    EPC    EPC    EPC
DEcIDE C       DEcIDE D                              D      E      F

                                                    EPC    EPC    EPC
DEcIDE E       DEcIDE F                              G      H      I

Research Generation                                 Research Synthesis

                          DECISION SCIENCES CENTER

                            Research Translation
   Scientific Resource Center
              for the
 Effective Health Care Program
The SRC supports the EHC Program as a whole, with specific
responsibilities to:

 •Communicate with stakeholders
 •Coordinate public input on Priority Conditions and Research Topics
 •Assist with the development of research topics and key questions
 •Coordinate peer review and public input for comparative effectiveness
 reviews (CERs)
 •Provide technical assistance to EPCs conducting (CERs)
 •Support methodological progress in effectiveness reviews and research
    Early CER Report Goals
• Reports written with user in mind
  – SRC responsible for insuring reports have
    •   Clear, concise findings
    •   Consistent layout
    •   Transparent methodology
    •   Methodological validity and consistency
• Appropriate application of methodologies
  – Direct versus Indirect Evidence
  – Effectiveness versus Efficacy
  – Observational data
     2 extreme views about
      observational studies
• Observational studies aren’t useful.
  RCTs are the gold standard and the
  only valid design for ―truth‖

• Observational study evidence trumps
  RCT evidence. RCTs are not
  applicable to real-world practice
   Inconsistencies using
observational studies across
     systematic reviews
      I. Terminology
      II. Uses and criteria for
      III. Quality assessment
      Norris S, Atkins D. Annals of Internal Medicine
            I. Terminology

• ―Observational‖ studies: Lack investigator
  allocation to an intervention
• Studies not trials
• Rely entirely on studies of association
• Include: case series, cross-sectional, case-
  control & cohort studies, before and after,
  time-series, database studies, historical
• ―Non-randomized studies‖ is broader term
  II. Uses of non-randomized
 studies in systematic reviews
• Design hierarchy in EBM (RCT the highest design)
• Some health care effectiveness research demands
  non-RCT (NRS) approaches
   – RCTs aren’t ethical, efficient or practical (surgery, adverse
     effects, public health interventions, organizational change)
• Some questions aren’t answerable with RCTs
   –   Etiology
   –   Prognosis
   –   Prevalence
   –   Compliance
All NRS designs are not equal
• Experimental (two or more groups)
  – Quasi-randomized trial, non-randomized or
    controlled clinical trial
• Single group designs
• Treatment-defined comparison groups
  – Historical or non-concurrent cohort study,
    prospective or retrospective cohort studies
• Outcome-defined comparison groups
  – Prospective (nested) or retrospective case-control
When considering including a
NRS design, you must define
• Which designs are acceptable given the
  research question and field of inquiry
  – Surgery/emerging technology, adverse
    effects, public health, org change (EPOC)
  – Type of designs possible
• When their use will produce reasonably
  unbiased results
  – Critical quality issues
  – Expected effect size
Case example: EPOC reviews

• Randomised controlled trials
• Controlled clinical trials (CCTs)
• Controlled before-and-after studies
• Interrupted time series (ITS)
Contents of EPOC specialised
  register by study design
   Number of studies in register: 1400+

              8%                   RCT

Non-randomised studies in
     EPOC reviews
  Number of studies in 15 reviews: 180

      CBA     ITS
      4%     12%
III. Quality Assessment: Factors
   to consider when evaluating a
            NRS design

• Design-specific threats to validity

• Review-specific threats to validity
Empiric Guidance on NRS vs
       RCT evidence
 – Well-designed cohort or case-control &
   RCTs have similar effect sizes (24 clinical
   topics). Concato et al. Benson et al NEJM 2000;342:1887-92.
 – ―Strong evidence that quasi-randomized
   trials provided biased effect size estimates
   of about 30%‖--at least for medical Rx.
   Cochrane NRSG approach (

 – Results of RCTs and NRS sometimes but
   not always differ. Deeks. HTA report. 2003;Vol 7:no. 27.
                    NRS vs. RCTs

• Observational studies have at times
  been misleading in formulating RCTs.
  Kramer B et al. NIH Report: Being smart about clinical trials. 2005.

• Highly cited clinical research can be
  modified or overturned (more likely with
  NRS than RCTs). Ioannadis JP. JAMA 2005;294:218-228.
 Main uses of NRS designs in
  comparative effectiveness
       reviews (CERs)
• Clinical situations not suited to trial
• Extending findings of available RCTs
• Real-world outcomes(do effect sizes in
  efficacy trials hold up?)
• Persistence and adherence
Clinical situations commonly
         without trials

      •   Off-label use
      •   Contraindications
      •   Proven effect
      •   Cultural barriers
     Extending findings from
         available RCTs
• Population limitations (homogeneous, limited
  co-morbidities, unstudied vulnerable groups)
• Small sample sizes (adverse events, rare
• Short follow-up (maintenance of benefits,
  adverse events)
• Important outcomes not available (patient
  priorities, long-term effects, natural
  history/background rate)
      Real-world outcomes

• Applicability (how generalizable are
  trials to real-world practice?)
  – Incorporates community clinical practice
    and adherence/persistence issues
      Comparing control of bias in RCT & NRS
Source of bias   RCTs       Cohort/other NR Comments
Selection bias   Randomi-   Control for    Many other
                 zation     confounders    study-specific

Performance      Double-    Exposure       Misclassifica-
bias             blinding   measurement    tion/ non-
(Exposure)                                 comparability

Attrition bias   Complete   Complete F/U   What amount is
                 F/U                       critical?

Detection bias   Masked     Masked         Misclassifica-
(Outcome)        assessmt   assessmt       tion
  Types of Selection Bias & their
      control in NR Studies
   Source of Bias          Consideration
• How patients are      • Sampling/selection
  identified              criteria
• How allocation        • Confounding by
  occurred/who chose      indication
• Attempts to balance   • Baseline
                          comparability &
  allocated groups
                          design features
• Other prognostic      • ID and control of
  factors                 confounders
Current methodological issues
      for NRS in CERs
• Locating studies
• Appropriate inclusions/exclusions
• Reporting (STROBE)
• Quality rating
• Defining minimum data sets for applicability
• Synthesizing
    • applicability & other observational data
    • effectiveness data from both RCTs and NRS
• Determining effectiveness from NRS
                                                                                                 Figure: Considerations in using nonRCT/CCTs in
                                        Always consider
  CCT                                                                                            systematic reviews of healthcare interventions
                                        if available

Other designs:                                   IF                                                                 WHEN
-Non-observational                 Should non-RCT/CCTs be considered for a CER?                              When should specific studies be used?

                                  Etiologic hypotheses: Yes
                                                                                                                    1. Study design
                                  Adverse events: Yes
                                                                                                                    2. Quality
                                 Efficacy/effectiveness - Consider:                                                 3. Precision/sample size
                                                                                                                    4. Inception cohort
                                 1. Key question
                                      - Consider probability of
                                      selection bias or confounding
                                 2. Body of RCT/CCT evidence
                                      - If inadequate for population
                                      & comparator characteristics                                                      HOW
                                      of interest → include*
                                      - If adequate → consider                                               How should non-RCT/CCTs be used?

                    *Situations where relevant RCTs may be insufficient in number:                             How should data be synthesized?
                    1. RCTs unnecessary                    4. RCTs difficult to implement                      - Pooling of non-RCT data
                          - Obvious benefit                     - Rare outcome                                 - FE vs RE models
                    2. RCTs inappropriate                       - Need long-term follow-up                     - Pooling across designs
                          - Clinical equipoise                  - Randomization →↓ benefit
                          - Patient preferences                 - Unit randomization difficult
                          - Policy/political/legal objections - Potential contamination
                    3. RCTs inadequate                            of control group                               How present and use these data?
                          - Applicability: populations of       - Complex intervention                           - Relationship/comparison to
                             interest not in RCT/CCTs           - Hazardous intervention
                                                                - Infeasible intervention
                    Categories based on Black 1996                                                               - Conclusions
                                                                                                                 - Potential biases
                                                                                                                 - Generalizability
SL Norris 3,13,06
  Non-randomized studies
(NRS): the current bottom line
• For reviews, use the best available evidence
• NRS may be the best available
• NRS designs are not equal & studies within a
  particular design are not equal
• Internal validity (truth) issues vs. external validity
  issues (relevance) is a balance
• Analyses within and across RCTs are really NRS:
   – Subgroup analyses
   – Meta-analyses
Effective Health Care Program
Minimum criteria for including
CBAs or ITSs in EPOC reviews
• CBAs
  – concurrent data collection both before
    and after the intervention
  – comparable second site as a control
• ITSs
  – minimum of three data points before and
    after the intervention
  Additional quality criteria for
• Baseline comparability
• Protection against contamination
     Additional criteria for ITS
• Intervention unlikely to affect data
• Sufficient data points for statistical
• Intervention independent of other
Methods for Comparative
   Evidence Reviews

     Marian McDonagh, PharmD

Oregon Evidence-based Practice Center
  Oregon Health & Science University
        CER Topic Selection
• Related to 1 or more of the 10 Priority
  – Suggestions for CERs (EPCs)
  – Suggestions for new research (DEcIDEs)
• Topics proposed through public process
  – Federal register
  – Web site (
• Initial topic selection
  – SRC does preliminary search for existing
    reviews or research
  – SRC develops draft key questions
        CER Topic Selection

• List of topics brought to Stakeholder
  group for comment
  – Prioritize and determine category: CER or
• Final Topic Selection
  – SRC revises draft key questions
  – Final topics and questions submitted to AHRQ
  – Topics assigned to EPCs or DEcIDEs
        Current CER Topics

Arthritis and non-traumatic joint disorders
  1. Evidence for Benefits and Safety of Analgesics
  2. Evidence for Comparative Short- and Long-
     Term Benefits, Harms, and Outcomes of bis-
     phosphanates, Combined Exercise, Diet, and
     High-Dose Calcium, and Selective Estrogen
     Receptor Modifier for Low Bone Density
       Current CER Topics
 1. New Diagnostic Technologies for Evaluation of
    Abnormal Breast Cancer Screening
 2. Effectiveness of Epoetin and Darbepoetin for
    Managing Anemia in Patients Undergoing
    Cancer Treatment
 3. Evidence for Comparative Short- and Long-
    Term Benefits, Risks, and Outcomes of
    Therapies for Localized Prostate Cancer
       Current CER Topics
Depression and other mood disorders
  1. Evidence for Off-Label Use of Atypical Anti-
     Psychotic Medications
  2. Comparative Effectiveness of
     Pharmacotherapeutics for Depression
Diabetes mellitus
  1. Evidence for Comparative Effectiveness of
     Newer Oral Agents vs. Older Agents (e.g.,
     metformin and sulfonylureas) for Initial
     Therapy for Diabetes (Oral Hypoglycemics)
        Current CER Topics

Peptic ulcer disease and dyspepsia
  1. Comparative Effectiveness of Management
     Strategies for Gastroesophageal Reflux

Stroke and hypertension
  1. Evidence for Renal Artery Stenting vs.
     Aggressive Anti-hypertensive Medical Therapy
     for Mild Renal Artery Stenosis
Priority Conditions without CER
         Topics Assigned
•Chronic obstructive pulmonary disease and

•Dementia including Alzheimer's disease

•Ischemic heart disease

AHRQ selects topics and posts RFTO
EPC responds to RFTO
EPC to conduct CER selected by AHRQ
                                                                    Topic completion
Conference Call with TOO, EPC and RC
EPC assembles expert advisors as needed
RC completes FDA and other searches, collects
industry submissions
EPC conducts sesarches, Reviews literature from
searches, retrieves articles
Receive dossiers, FDA documents etc from RC
RC works with EPC to identify peer reviewers;
forward list to TOO
TOO approves or modifies peer reviewer list
EPC begins report
Submit Draft report to RC and send for peer
RC Peer Review Draft Report
RC posts for Public Comment
EPC receives comments from RC
RC summarizes research gaps and forward to
next Stakeholder meeting
Receive peer review comments - notify RC if
serious concerns raised
Public comments forwarded to EPC
Receive Public Comment. Incorporate and
summarize public and peer review comments
Submit Final report to RC along with summary of
comments and responses
Receive and post Final CER
                                          week# -3 -1   1   3   5   7   9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41
  Key Question Development

• Questions relate to:
  – Effectiveness
  – Harms and Tolerability
  – Sub-populations
     • Age, Race/ethnicity, Gender, Co-morbidities
• Inclusion Criteria based on key questions
• Draft Questions (SRC)
  – Stakeholder group for comment
  – Publicly posted for comment
• Final Questions
  – Go to EPC
       CER Report Processes
• Draft reports undergo
  –   Peer Review
  –   Public Comment via web site
  –   Scientific Resource Center review
  –   Stakeholder group for comment
• Every report includes
  – Executive summary
  – Summary table by Key Question
  – Every report will have a slide show
• Final Reports posted to web site
  – Reports to be updated as needed
  – SRC to monitor literature to determine need for
    Other challenges with using
      ―observational‖ studies
• Finding them all
• Using what you find (reporting)

•   Kuper H BMC Medical Research Methodology 2006,6:4.
•   Stroup JAMA 2000;283:2008-2012.

To top