Critical Appraisal Critical Appraisal Therapy Critical Appraisal by ert554898

VIEWS: 32 PAGES: 42

									Critical Appraisal - Therapy
        Philipp Dahm, MD MHSc FACS
        Associate Professor of Urology
                  Florida
    University of Florida, College of Medicine
           Session Objectives

                      y
• Apply the hierachy of evidence for studies of
    pp y
  therapy to your practice
• Desribe the three-tiered User’s Guide framework
  for critically appraising a study of therapy
• Be able to appraise an individual study for its
                 pp                        y
  validity, impact and applicability
Evidence-Based Medicine Cycle
                          Patient
                          dilemma
                Ask                    Assess


          Acquire
          A   i




               Appraise

Hierarchy of
        y                    Apply         Patient’s
 Evidence                            Values & Preferences
          Hierarchy of Evidence
                                         Meta
                                         Meta-
                                        analysis
Level
L l1            Randomized Trials          Less Bi
                                           L Bias

Level 2    Prospective Cohort Studies

Level 3       Case Control Studies

Level 4           p
            Retrospective Case Series

Level 5              Opinion               More Bias
         Levels of Evidence in
         Urological Literature
16% and 15% of studies in urological literature in 2000 and
 2005, respectively, provided high (I and II) level evidence

  80%


  60%
                                                         2000
  40%                                                    2005


  20%


   0%
             I          II         III         IV
                     Level of Evidence
                                         Borawski K et al, J Urol (2007)
Critical Appraisal
         Users
       • Users’ Guide to the Medical
         Literature (2. edition)
       • The EBM reference source
       • Compilation of review
            ti l that l t to
         articles th t relate t
         practice of EBM
       • Limitations:
           p       y g
         - primarily targets internists
          - excessive (>700 pages)
                sub specialty
          - not sub-specialty specific
           Critical Appraisal
    How to Assess An Article About Therapy


1. Are the results valid?

2. What are the results?

3. Can I apply the results to patient care?


                 JAMA “User’s Guide to the Medical Literature” (2002)
           Validity

   Validity is the degree that a
study appropriately answers the
    question being asked or
 appropriately measures what it
       intends to measure.
                    Bias
In contrast to random error, bias leads to
                         error
systematic deviation of the underlying truth,
i.e.            direction.
i e bias has a direction


Bias may intrude as a result of differences that
existed between treatment and control groups
at the time patients entered the study
                          and/or
reflect differences that develop after the study
begins.
begins
       Strategies for Reducing Bias
Source of Bias                 Strategy for Reducing Bias
Differences noted at the start of the study
Treatment and controls         Randomization and concealed
differ in prognosis            allocation

Differences that arise as the study progresses
Placebo effects                       g patients
                               Blinding of p

Co-intervention                Blinding of health care providers

Biased outcome assessment      Blinding of outcome assessors

        follow-up
Loss of follow up                              follow-up
                               Ensure complete follow up
       Bias Detective

Quest o e e yt g
Question everything!

Beware of hidden tigers!
Ioannidis, JP PLOS (2005)
    5 Safeguards Against Bias
•   Randomization
•   Concealed allocation
•   Blinding
•                      follow up
    Completeness of follow-up
•   Intention-to-treat analysis
   Empirical Evidence of Bias
        Inadequate RCT Methodology
Associated with Exaggerated Treatment Effects

    Allocation concealment
          - inadequate           +41%
          - unclear               30%
                                 +30%
    Blinding
                double blinding   17%
          - no “double-blinding” +17%


         Inadequate RCT methodology
         associated with bias

                         Schulz, KF et al, JAMA (1995)
             Scenario
         p      g
You are planning to attend an EBM
Workshop in Oxford, England
The plane journey from Tabriz is > 10
hours
  friend f            back from a t i t
A f i d of your came b k f         trip to
Canada complaining of leg pain
She was ultimately diagnosed with a deep
vein thrombosis (DVT) and treated with
prolonged hospitalization
       Your Questions
           y             p g
What is my risk of developing a DVT on
my trip to Oxford?
Is there a way to reduce my chance of
developing a DVT?
       The Best Evidence
                ,            (    )
Ref: Scurr et al, The Lancet (2001)
Study Design: RCT
Population: Passengers from the UK
making flight journeys > 8 hours
Intervention: Below knee compression
       g
stockings
Comparison: no special measures
Outcome: DVT per duplex US
          Randomization
              Definition

  Performed to achieve b l
-P f       dt     hi            for both
                       balance f b th
  known and unknown prognostic variables
  between treatment and control group
- Observed differences not greater than
  might be expected due to chance
     Allocation Concealment
                Definition

- Investigators should not be able to
  determine the allocation of the next patient
  to be entered into a trial
- Decision to accept/reject participant should
  b made i i
  be                        f        i
         d in ignorance of next assignment
- ≠ Blinding
           g
  What Methods are Concealed?

Use Hospital Chart
Numbers to Randomize      NO
Patients
Place Patient Treatment
Allocations into
                          MAYBE
Envelopes
Use Telephone
Randomization System
                          YES
   When Concealment is LOST
• Australian investigators undertook a
                       p
  randomized trial of open versus
  laparoscopic appendectomy. The trial
                           day.
  ran smoothly during the day

• But “not so” smoothly at night.
Was Randomization Concealed?
Was Randomization Concealed?


     Garbage Can
    Randomization




 “If you open enough
   envelopes, you’ll
          ll
  eventually get your
 treatment of choice!”
Randomization/Allocation Concealment

   “Volunteers were randomized by sealed
   envelopes
   envelopes”


    Method of randomization/allocation
         l     tl   than ideal
   concealment less th id l
                      Blinding
                      Definition
- Prevents study participants’ and study personnel’s
  knowledge of study group assignment
  Protects t i l f
- P t t trial from:
      1) treatment differences between the groups other
             than the     d i d treatments
             th th randomized t t         t
      2) biased assessment of outcomes (least important
             for d th     t
             f death as outcome) )
- Need to assure that neither participants nor research
  teams can identify treatment assignment
Can Surgeons be Blinded?
Blinding




           Devereux PJ, EBMH 37 (2002)
               Blinding
Participants:
“the passengers were aware of the treatment”
Health care providers:
“the nurse removed the stockings from those
p
patients who had continued to wear them..”
Data collectors:
 ..duplex                           group
“..duplex technician unaware of the group”

Participant not blinded (also not feasible)
Personnel not blinded
Outcome assessors blinded
                     Follow up
     Completeness of Follow-up
               Definition

                      non random
- Protects trial from non-random loss of
  participants
  Guards       i t l ti bias
- G d against selection bi
                        Follow up
        Completeness of Follow-up
           Worst Case Scenario Method

          Intervention                    Control
            (N=250)                       (N=250)
          SF-36: 75pts
                   p                 SF-36: 60 pts
                                               p

           20% Loss                     20% Loss

                                                  50 pts (mean
50 pts (mean score       Intervention ~
                         I t     ti             score: 80) “Doing
40) “Doing poorly                                 well and don’t
                             Control
     and seek                                            back”
                                                   come back
 alternative care”       68 versus 64
                    Completeness of FU
                      231 patients randomized

             116 no stockings         115 stockings
16 not evaluated                                15 not evaluated
pre-/post (13.8%)                               pre-/post (13.0%)

                    l t data
            100 complete d t                l t data
                                    100 complete d t


            12/116 had DVT            0/115 had DVT

     Worst case scenario:
    15 additional pts in stocking group with DVT: 15/115
    No additional pts in control group with DVT: 12/116
                  y
              Analysis of an RCT
         Per Protocol versus Intention-to-treat

Intention-to-Treat: Every participant analyzed as randomized
  (regardless whether pt actually received treatment)
Advantage: Guards against other bias
Disadvantage: May underestimate the full treatment effect


Per Protocol: Analyses limited to those participants in both
groups who actually received the treatment
Disadvantage: Pts who adhere to study treatment may be
different than the drop-outs
                     Intention to Treat
    Per Protocol vs. Intention-to-Treat
  Example: An RCT of surgery plus immunotherapy
          (Arm A) versus immunotherapy alone (Arm B)
                           10 pts #                 10 pts #
                                 Immunotherapy
                                                                       Survival: 80/100
              n=100
                                  1m                            12 m
Randomization
                                       OR       Immunotherapy
              n=100                                                    Survival: 80/100
                                  1m                            12 m
                           10 pts #                10 pts #

  Intention to treat analysis:
  Intention-to-treat                        1 year survival rate: 80/100 = 80%
  (i.e. analyze as randomized)

  Per protocol analysis:                    1 year survival rate: 80/90 = 88%
  (i.e. analyze as actually treated)
        Intention to Treat
        Intention-to-Treat

“All [….] analyses were done on an
intention-to-treat
intention to treat basis which included all
randomize patients”
Analyzed appropriately
   Beware of Initial
    Appearances
Looking at the surface is not
         enough!
    5 Safeguards Against Bias
•   Randomization
•   Concealed allocation
•   Blinding
•                      follow up
    Completeness of follow-up
•   Intention-to-treat analysis
    Safeguards Against Bias


Randomization                 Not ideal
Concealed allocation          Not ideal
Blinding
  Patients                      No
  Personnel
  P        l                    No
  Outcome assessors             Yes
Completeness of follow-up     > 80%
Intention-to-treat analysis
                       y       Yes
      Q     y          p     g
      Quality of RCT Reporting
                  Consort Criteria
- Full-text publication only evidence that study is valid
- Consolidated Standards of Reporting Trials
- Developed in 1996, revised in 2002
  Checklist f    items th t include:
- Ch kli t of 22 it    that i l d
       - description of randomization (#8)
       - allocation concealment (#9 and #10)
       - masking (#11)
       - patient flow/completeness of follow-up (#13 an #14)
       - intention to treat analysis (#16)


                                           www.consort-statement.org
Endorsing CONSORT as a family!
          Critical Appraisal
    How to Assess An Article About Therapy


1. Are the results valid?

2. What are the results?

3. Can I apply the results to patient care?


                 JAMA “User’s Guide to the Medical Literature” (2008)
Evidence-Based Medicine Cycle
                          Patient
                          dilemma
                Ask                    Assess


          Acquire
          A   i




               Appraise

Hierarchy of
        y                    Apply         Patient’s
 Evidence                            Values & Preferences

								
To top