Docstoc

Absolute Fragility Index

Document Sample
Absolute Fragility Index Powered By Docstoc
					     Goals of presentation
• basic approach to when trial large enough

• how RCTs can mislead

• p-values and confidence in estimates
  – fragility

• fragility study
  When is sample size adequate?
• atrial fib at risk of stroke
• warfarin increases serious gi bleeding
  – 3% per year

• 1,000 patients 1 less stroke
  – 30 more bleeds for each stroke prevented

• 1,000 patients 100 less strokes
  – 3 strokes prevented for each bleed

• where is your threshold?
  – how many strokes in 100 with 3% bleeding?
1.0%   0
1.0%   0
1.0%   0
1.0%   0
    EBM perspective on RCTs
• although RCT at top of Tx hierarchy
  – randomization does not guarantee results
    approximate truth

• two broad reasons RCTs (or any other
  study) mislead
• systematic deviations from truth
  – bias
• non-systematic deviations from the truth
    • random error – chance
        2 hypothetical RCTs
• Evaluating new treatments to prevent MI

• Placebo controlled

• Outcome death

• Identical methodology
  – concealed, blinded, complete f/u, ITT
    • low risk of bias
Trial 1     Tx       Placebo    P value
          (n=100)    (n=100)
 MI          1          9        0.02



Trial 2      Tx       Placebo   P value
          (n=4000)   (n=4000)
 MI         200        250       0.02
                 Question
What best characterizes your belief in whether
  there is a real effect of treatment?
• Substantially more likely for Trial 1
• Modestly more likely for Trial 1
• Similar for both trials
• Modestly more likely for Trial 2
• Substantially more likely for Trial 2
Trial 1     Tx       Placebo        RR
          (n=100)    (n=100)     (95% CI)
  MI         1          9           0.10
                                 (0 – 0.77)

Trial 2      Tx       Placebo       RR
          (n=4000)   (n=4000)     95% CI
 MI         200        250          0.80
                                (0.63 – 0.95)
                 Question
What best characterizes your belief in whether
  there is a real effect of treatment?
• Substantially more likely for Trial 1
• Modestly more likely for Trial 1
• Similar for both trials
• Modestly more likely for Trial 2
• Substantially more likely for Trial 2
 Small variation in hypothetical trials
• Add 2 events to treatment groups
  – what would happen to p values?
    • 1st trial of 200 patients P value 0.13 (i.e., fragile p value)
    • 2nd trial of 8000 patients P value remains 0.02

• Absolute Fragility Index (AFI)
  – minimum number of patients required to switch from
    non-event to event in Tx group to reverse statistical
    significance
    • 1st trial AFI – 1
    • 2nd trial AFI - 9
Trial 1      Tx      Placebo    P value       RRR         AFI
           (n=100)   (n=100)                (95% CI)


  MI          1        9         0.02          0.10        1
                                            (0 - 0.77)



Trial 1      Tx     Placebo    P value        RR          AFI
          (n=4000) (n=4000)                (95% CI)

 MI         200      250        0.02          0.80         9
                                          (0.63- 0..95)
                 Question
What best characterizes your belief in whether
  there is a real effect of treatment?
• Substantially more likely for Trial 1
• Modestly more likely for Trial 1
• Similar for both trials
• Modestly more likely for Trial 2
• Substantially more likely for Trial 2
        Many examples of early
       smaller trials contradicted
•   monoclonal antibody HA-1A in sepsis
•   Vitamin E to prevent MI and death
•   Magnesium for mortality reduction after MI
•   ACE inhibitors for reducing diabetes
•   Nitrates for mortality reduction after MI
•   Aspirin decrease pregnancy-induced hypertension
•   albumen for mortality reduction in critically ill
•   intensive glucose control in critically ill
          Recent experience
• Beta-blocker trial
  – randomized 112 patients
  – 11 deaths
    • 2 beta-blocker group, 9 control group
       – P = 0.02

• POISE Trial
  – randomized 8351 patients
  – 226 deaths
    • 129 beta-blocker group, 97 placebo group
       – P = 0.03
   Fragility study - Objectives
• To characterize sample of contemporary trials
  – using Absolute Fragility Index
                   Methods
• Eligibility
  – Parallel limb RCT with 1:1 randomization
  – Dichotomous outcome favoring experimental
    treatment (p<0.05 or CI excluding null)
• Search
  – starting in 2009
  – consecutive RCTs from high impact journals
• AFI computed for each trial
          Computation of
   Absolute Fragility Index (AFI)
• For each trial 2 x 2 table was constructed
  – Fisher’s exact test computed

• Events were iteratively added to intervention
  group and non-events subtracted
  – keeping total sample size constant

• AFI was computed as
  – number of events required to increase p-value > 0.05
     Baseline characteristics
                                    N (%)
Journal
     NEJM                           35 (41)
     Lancet                         29 (34)
     JAMA                            9 (11)
     Ann Int Med                    12 (14)
Specialty
    Oncology                        13 (15)
    Cardiovascular & Respiratory    12 (14)
    Immunology & Infection          12 (14)
    Endocrinology                    9 (11)
    Gastroenterology                 9 (11)
    Other                           30 (35)
Outcome
    Composite Outcome               26 (31)
    Type of Endpoint “Beneficial”   28 (33)
    Reported HR                     17 (20)
    Adjusted Effect                  4 (5)
                           Results
                                 Median (Minimum – Maximum)

        Sample Size                   649 (15 to 162,243)

          # Events                      137 (6 to 4516)

        Relative Risk                   0.64 (0 to 0.99)

P-value by Fisher’s Exact Test     0.0032 (4.0x10-17 to 0.52)
Distribution of Sample Sizes
    Percentile    Sample Size
       Min            15
       10th          103
       20th          154
       30th          268
       40th          416
       50th          649
       60th          866
       70th          1198
       80th          2287
       90th          3846
      Max           162243
Distribution of number of events
      Percentile     Events
         Min           6
         10th         30
         20th         48
         30th         75
        40th           93
        50th          139
        60th          211
        70th          239
        80th          360
        90th          620
        Max          4516
             Fragility Index
• Absolute Fragility Index
  – median 8
  – 25th percentile 3


• Loss to follow-up exceeded AFI
  – 32%
Absolute Fragility Index




                           26
AFI versus starting p-value




                              AFI=3
Pairwise correlations of AFI
                           AFI
                            R
                        (p-value)
                         0.059
           N
                         (0.59)

                          0.25
       # Events
                         (0.02)

                         -0.072
   Control Event Rate
                         (0.51)

                         -0.009
     Relative Risk
                         (0.93)
Number of patients or events?
   • Trial 1
     – Intervention 1/1,000,000
     – Control 2/1,000,000
   • Confidence in RR 0.5
     – Low, medium, high?

   • Trial 2
     – Intervention 25/100
     – Control 50/100
   • Confidence in RR 0.5
     – Low, medium, or high?
                Limitations
• Thresholds are arbitrary

• Other potential ways to identify “fragility”
  – simplicity is key for uptake

• Not all trial designs may be suitable for
  this metric
  – continuous outcomes, non- 1:1 randomization
                   Conclusion
• RCT is a powerful study design
• “Positive” trial often hinges on few events
• The Absolute Fragility Index may improve trial
  interpretation
• p-value borderline view with skepticism
  – particularly small number events
  – beware misleading CI with small number events

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:1
posted:5/22/2014
language:English
pages:31