futility presentation may 2009 final by c9t0f87l

VIEWS: 4 PAGES: 34

									Stopping Trials for Futility
        Ranjit Lall
        (May 2009)
OVERVIEW


• Summary of a 1 day workshop on “Stopping Trial for Futility”

• Funded by the RSS/NIHR HTA/MRC

• 11th November 2008 in London

• Speakers:
    Jon Nicholl (ScHARR, University of Sheffield, Chair person)
    John Whitehead (Lancaster University)
    Andy Grieve (King’s College London)
    Karl-Fredrik Burman (Director of Statistical Science, Astra Zeneca)
WHAT IS FUTILITY?


• The term ‘futility’ is used to refer to the inability of the trial to
  achieve its objectives

• On-going monitoring of trials for safety, treatment efficacy or futility

• Early interims – review will focus more on safety, quality of conduct
  and trial integrity rather than efficacy.

• Later interims - may include formal efficacy or futility analysis
WHAT IS FUTILITY?


• A futility analysis causes a clinical trial to be stopped as soon as it
  becomes clear that a negative outcome is inevitable and thus it is no
  longer worthwhile continuing the trial to its completion.
PUBLICLY FUNDED TRIALS

• Early phase trials are supported by MRC

• Later phase trials supported by HTA

• HTA budget = £88 million pa
• Nearly all spent on trials

• Typical design of phase III trials

   – parallel group RCT, multi-centre, 3-5 years, £1million - £2.5
   million, N = 400- 4000
FUTILITY DESIGNS


•   Futility analyses are common in pharmaceutical trials

• NO HTA funded trials have planned a futility analysis

•   Questions:

    - Why is this?
    - Should futility analyses be designed into HTA trials?
    - If so, which trials?
    - And when and how should they be done?
Should HTA do futility analyses?

• YES –

(i) As pointed out by Ware, Muller and Braunwald (1985),

“……………..early termination for futility could reduce enormous expenditure of
  resources, human and financial, involved in the conduct of trials that
  ultimately provide a negative answer regarding the value of the medical
  innovation….”.

(ii) It is ethically wrong to continue to recruit patients to trials with little hope of
     achieving helpful results
Should HTA do futility analyses?

• NO –
   (i) All well conducted trials provide valuable evidence

   (ii) The cost of designing trials with a planned futility analysis would
   outweigh any savings
Should HTA do futility analyses?

• IT DEPENDS –
(i) In placebo controlled trials, futility suggests no worthwhile benefit (i.e. little
    chance of finding evidence that the intervention is better than placebo)

(ii) In head-to-head trials, treatment will be far from futile (so is futility analyses
     inappropriate in head-to –head trials)?

(iii) Futility analyses should be conducted in large expensive trials in which
      early stopping could save substantial resources

(iv) Futility analysis most useful where there are several treatments being
     compared, and there is a desire to eliminate one (for better resource use)
Extension grants

•   54% of MRC/HTA trials seek extensions

•   Usually due to long set up/slow recruitment

•   55% of trials with extensions still don’t achieve N

•   Cost of extensions for HTA = £50K - £1 million

•   Current HTA ask for a ‘futility analysis’ in terms of conditional power on
    application.

•   Almost never done as intended or refused
Fixed Interim analyses

• Focus Type I error (i.e. Rejecting the H0 when its true)

• Methods used include
  - Pocock (1977)
  - O’Brien and Fleming (1979)
  - Haybrittle and Peto (1976)
  - Lans and DeMets alpha spending function (1983, 1989)
 Futility Analysis (cont’d)

• Two fundamental approaches:

(i)   Using the above group sequential methods to compute the
      stopping boundaries for futility (in a similar way to computing
      stopping boundaries for efficacy)

•     The boundaries can be defined to allow for early stopping for
      futility only, or to allow for early stopping for either futility or efficacy
Futility Analysis (cont’d)

(ii) Conditional power
•   The power of a trial tells whether a clinical trial is likely to have high
    probability to detect a pre-defined treatment effect of interest.


•   Very low power implies that a trial is unlikely to reach statistical significance
    even if there is a true effect.


•   One should never begin a trial with low power.


•   However, sometimes low power becomes apparent only after a trial is well
    under way.
Futility Analysis (cont’d)


(ii) Conditional power: based on the probability of a getting a
    statistically significant final result, conditional on the data collected
    so far (in the interim)

• Stochastic curtailment – a frequentist method
• Predictive power – partially Bayesian methods
• Predictive probability – fully Bayesian methods
    Futility Analysis (cont’d)

     (a) Stochastic curtailment:

•    At planning stages, sample size based on (a) Difference - d; (b) power
     (80% or 90%); (c) type I error rate

•    At the interim, recalculate the power to detect a difference d.

•    Calculations involve the observed data and an assumption regarding the
     distribution of the future data

•    This method assumes that the observed difference between the treatments
     at the interim stage is the true treatment difference, d.
    Futility Analysis (cont’d)

     (a) Stochastic curtailment:


•    Detect e.g.<20% power no point continuing;
•    Detect e.g.>50% power then continue.

•    This cut-off is pre-specified (choice would depend on the risk/benefit
     considerations)

•    Take other information into account before deciding on stopping or not
     extending.
    Futility Analysis (cont’d)

     (b) Predictive power:

•     Conditional power calculations use the observed data and an assumed
     future treatment effect

•    Not sure what assumptions to make

•    Predictive power approach addresses this by averaging the conditional
     power function over a range of treatment difference, with weights based on
     the observed treatment effect
Futility Analysis (cont’d)

    (b) Predictive probability:

•    Prior probability: based on the interim data and the prior distribution of
    the parameter for the treatment effect;

•   Statistical model: mathematical model which expresses the relationship of
    the parameters and the data

•   Posterior probability: Probability of a clinically important treatment effect
Futility Analysis (cont’d)

• If early results show:

• Intervention is better than expected
       conditional power will be high

• Intervention is worse that expected
         conditional power will be low (unless your sample size
  increases)
Futility Analysis (cont’d)

The two most challenging aspects:

• the selection of optimal stopping thresholds;
• the timing of the analysis

both of which require the balancing of various risks.
     Ask the audience
(questions for the workshop)
5
         Should we plan to do futility analysis in publicly-
         funded trials ?

    1.    Yes – for all trials (built into
          design)
    2.    Yes – but only for extension
          requests
    3.    Never
    4.    Undecided                                                   0%          0%      0%          0%




                                                                                                        ed
                                                                                           r
                                                                                         ve
                                                                               .. .
                                                                 .




                                                                                                    c id
                                                                i..




                                                                                       Ne
                                                                             ns
                                                            ilt




                                                                                                  de
                                                                           te
                                                          bu




                                                                                               Un
                                                                        ex
                                                         s(



                                                                         r
                                                                      fo
                                                         al
                                                    tri



                                                                   ly
                                                                 on
                                                    ll
          0
                                                  ra



                                                                 t
                                                              bu
                                                fo
                                               s–



                                                           s–
                                             Ye



                                                         Ye
Should we plan to do futility analysis in publicly-
funded trials?
                                          53 %

1. Yes – for all trials (built into
   design)
1. Yes – but only for extension
   requests
                                                        21%
1. Never                                                                      17%

2. Undecided                                                        9%




                                      Yes – for all    Yes but     Never   Undecided
                                      Appropriate       only for
                                      Trials (built   extension
                                      into design)    requests
Should we plan to do futility analysis in publicly-
funded trials?

•   May not be appropriate for all trials, but justification for doing/not
    doing should be clearly laid out in the application

•   Should be made as a requirement by funders (and the funders
    should clearly lay out what is required by them)

•   Clear distinction needs to be made between trial futility (e.g. lack
    of recruitment-operational aspect) and treatment futility (lack of
    efficacy - statistical)
5




    In what circumstances are they appropriate?


        1.  Only in placebo-
           controlled trials
        2. In both placebo-
           controlled and head-
           to head trials
                                                              0%               0%         0%
        3. Undecided




                                                                                            d
                                                                                         de
                                                        l..




                                                                            l..




                                                                                         ci
                                                        l




                                                                          ol
                                                     ro




                                                                                       de
                                                                        tr
                                                    t




                                                                      on
                                                 on




                                                                                    Un
                                                -c




                                                                        c
                                                                     o-
                                              bo




                                                                  eb
                                            ce




                                                               ac
                                             a




                                                              pl
                                          pl
    0
                                      in




                                                          th
                                                        bo
                                      y
                                  nl




                                                     In
                                  O
In what circumstances are they appropriate?

1.   Only in placebo-controlled
                                                            44%
     trials
2.   In both placebo-
     controlled and head                 29%
                                                                           27%
     to-head trials
3.   Undecided




                                       Only in         In both placebo-   undecided
                                      appropriate       controlled and
                                  placebo-controlled     head-to-head
                                        trials              trials
In what circumstances are they appropriate?

•   It may be difficult to predict in advance



•   Futility analyses may not be very appropriate for very large
    pragmatic trials, where other information (other than efficacy) is
    important
5   Should funding bodies insist upon futility analyses
    before granting extensions?


    1. Yes
    2. No
    3. Undecided



                                    0%      0%        0%

     0
                                    s




                                                         d
                                           No
                                  Ye




                                                      de
                                                      ci
                                                    de
                                                 Un
Should funding bodies insist upon futility analyses
before granting extensions?

1. Yes
                           47%
2. No
3. Undecided
                                     27%
                                                25%




                           Yes       No      Undecided
Should funding bodies insist upon futility analyses
before granting extensions?

Should not be insisted upon, but if investigator refuses
to do futility then justification should be given
    What is futile?
5
    How low can the conditional power be before stopping
    a trial


          1.   1%
          2.   5%
          3.   10%
          4.   15%
          5.   20%
          6.   30%
          7.   40%
          8.   50%                    0%        0%         0%         0%         0%         0%     0%     0%         0%


    0     9.   It depends
                                 1%


                                           5%


                                                       %


                                                                  %


                                                                             %


                                                                                        %


                                                                                                   %


                                                                                                          %


                                                                                                                     s
                                                                                                                 nd
                                                     10


                                                                15


                                                                           20


                                                                                      30


                                                                                                 40


                                                                                                        50


                                                                                                                 e
                                                                                                             dep
                                                                                                          It
What is futile?
How low can the conditional power be before stopping a trial?

1.   1%
2.   5%
                                                                                   72%
3.   10%
4.   15%
5.   20%
6.   30%
7.    40%
8.   50%
9.   It depends                                5%    5% 5% 5%                5%
                                          0%                           2%
                                     0%

                                     1%   5%   10%   15%   20%   30%   40%   50%     It
                                                                                   depends
What is futile?
How low can the conditional power be before stopping a trial?


•   Futility analysis should be focused on effect size (rather than
    conditional power)



•   A trial may be considered as futile in its ability to recruit patients,
    but would add information to a evidence based data
Thank you !!

								
To top