Docstoc

Aberdeen

Document Sample
Aberdeen Powered By Docstoc
					When is there Sufficient Evidence?


                     Karl Claxton,
    Department of Economics and Related Studies and
             Centre for Health Economics,
                  University of York.
     Sufficient evidence for decision making
• Informing decisions
• Prioritising research
    – Examples from the NHS HTA Pilot
•   Designing research
•   Evaluating regulatory policies
•   Implementation
•   UK policy impact
•   Implications and challenges
                    The decisions

• Should a technology be adopted given existing information?
   – Which clinical strategies are cost-effective?
   – For which patient groups?

• Is additional evidence required?
   – What type of evidence
   – What type of studies
   – For which patient groups
   – How much evidence
                 Informing decisions

• Requirements
  –   Structuring clinical decision problems
  –   Characterisation of decision uncertainty
  –   Value of additional research
  –   Decisions consistent with objectives and constraints
      Structuring clinical decision problems

• Requirements
  –   Compare all alternative interventions/strategies
  –   Explore the full range of clinical policies
  –   For range of patient groups
  –   Over an appropriate time horizon
  –   Combine evidence from variety sources
               Characterising uncertainty

• Evidence for all model parameters
   – Systematic searching for all parameters
   – Methods of synthesis
• Direct evidence
   – Different types of study (potential bias/exchangeability)
• Networks of evidence
   – Indirect and mixed comparisons
   – Evidence on functions of parameters
• Probabilistic analysis of the decision model
           Mixed and indirect comparisons
                                     Alternative interventions

• Compare all                             A     B     C     D
                              RCTs
  interventions                      1      x     x
• No RCT of all
• Pair wise comparisons?             2      x           x
• Use all the information            3      x                x
   – Mixed comparisons
   – Same assumptions as             4            x     x
     random effect meta
     analysis                        5            x          x
   – Estimate posterior LOR
     with correlations               6                  x    x
      Networks of evidence


             A
A+B                    A+C

           A+B+C

 B                      C

            B+C
            Value of additional research
Is further evidence required?
• Expected value of perfect information (EVPI)
   – Maximum return to research (decision problem)
   – Necessary condition (EVPI>costs)


What type of evidence?
• EVPI for parameters
   – Maximum return to different types of research
   – Focusing research design
                      An example
• A pilot study of value of information analysis to
  inform the NHS health technology assessment
  programme
   – Screening for age-related macular degeneration
   – Manual chest physiotherapy techniques for asthma and
     chronic obstructive pulmonary disease
   – *long-term antibiotic treatment for preventing recurrent
     urinary tract infections (UTI) in children*
                 Structuring the decision problem
Frequency of        Number of                   Progressive        End-stage renal disease
recurrent UTIs      pyelonephritic attacks      renal scaring


   No UTI


                                                                                  Transplant
  1 UTI           Pyelonephritic
                     attack



  2 UTIs          Pyelonephritic
                                    Number of     Progressive
                     attack                          renal      Development      Age at
                                    attacks         scaring       of ESRD      ESRD onset


  3 UTIs          Pyelonephritic
                     attack

                                                                                   Dialysis
                  Pyelonephritic
 4 UTIs              attack
                         The evidence
• Effectiveness
   – Existing reviews (variable quality)
   – Meta analysis, Multiple parameter synthesis
• Natural history
   – Epidemiological studies
   – Pooled trial baselines
   – Registry studies
• Quality of life
   – Published studies
   – Survey
• Costs
   – Published studies
   – Published unit costs and dosage (BNF, PSSRU, CIPFA)
                                           Characterising decision uncertainty
                              1


                                                                                               Intermittent
                             0.9
                                                                                               Cotrimoxazole

                             0.8                                                               Nitrofurantoin
                                                                                               Trimethoprim
                             0.7                                                               Frontier
Probability cost-effective




                             0.6


                             0.5


                             0.4


                             0.3


                             0.2


                             0.1


                              0
                                   £0   £10,000   £20,000          £30,000           £40,000   £50,000          £60,000
                                                       Threshold for cost-effectiveness
         The irrelevance of inference?
• The choice between alternative technologies should
  be based on expectation. Inference is irrelevant to
  treatment choice

• The only valid reason to characterise the uncertainty
  surrounding outcomes of interest is to establish the
  value of additional information

• Distinguish the separate steps of deciding which
  technology should be chosen, given existing
  information, from the question of whether more
  evidence is required to support this decision
          Is further evidence required?
• Expected cost of uncertainty
   – Adoption based on existing information is uncertain
   – There is a probability of making the wrong decision
   – There are costs of making the wrong decision


• Expected value of perfect information (EVPI)
   – Maximum return to research (decision problem)
   – Necessary condition (EVPI>costs)
              EVPI for the decision problem
               Treatment A   Treatment B   Treatment C   Optimal treatment   Max NB
Iteration 1        11            12            13               C             13
Iteration 2        12            10             9               A             12
Iteration 3        13            18            15               B             18
Iteration 4        14            16            17               C              17
Iteration 5        15            14            11               A             15
                                                                              15
Expected NB        13            14            13


Current information = 14
Perfect information = 15
EVPI      = 15- 14 = 1


          EVPI = Eθ maxj NB(j, θ) - maxj Eθ NB(j, θ)
                                             Is further evidence required?
                                                     Population EVPI

                 £4,000,000



                 £3,500,000



                 £3,000,000



                 £2,500,000
Populaion EVPI




                 £2,000,000



                 £1,500,000



                 £1,000,000



                  £500,000



                        £0
                              £0   £10,000       £20,000         £30,000          £40,000   £50,000   £60,000
                                                       Cost-effectiveness threshold
            EVPI for parameters
EVPPIφ = Eφ maxj Eψ |φ NB(j, φ,ψ) – maxj Eθ NB(j,θ)

Where: φ = parameter of interest
       ψ = other uncertainties

Some implications:
• Information about an input is only valuable if it
  changes our decision
• Information is only valuable if the parameter does
  not resolve at its expected value

General solution (linear and non linear models)
                                                     Expected Value of Perfect Information



                             Fu




                                                £0
                                                     £500,000
                                                                £1,000,000
                                                                             £1,500,000
                                                                                          £2,000,000
                                                                                                       £2,500,000
                               ll
                                 m
                  Fr                od
   Fr                eq                  el
     eq                  ue
            ue              nc
               nc              y
                  y              of
                                    U
                    of                  TI
                        py
                          el
                             on
                                ep
                                   hr
                                      i ti
    U                                      s
     ti l                 R
          i ty             is
               of             k
                                of
                  U                PR
                    TI
                       /p                S
                         ye
                            lo
                              ne
                                 ph
                                     rit
                                         is
         D
           ev
               el                   ES
                  op                    R
                      m                    D
       C                 en
         on                  to
             se                  fE
Ef              qu                    SR
   fe               en                     D
     ct                 ce
        of                   s
            Lo                 of
               ng                   ES
                                        R
                   -t e
                       rm                  D
        Ef                   an
            fe                   ti b
               ct                    io
                  of                   ti c
                      co                    s
                          tri
        Ef
           fe                 m
              ct                 ox
                  of                 az
                                        ol
                      ni                   e
                         tro
                             fu
          Ef
              fe                ra
                ct                  na
                                       to
                                          in
                                                                                                                    What type of evidence?




                    of
                         tri
                             m
                               et
                                  ho
                Ef                    pr
                    fe                   im
                       ct
                           <
                               6
                                 m
                Ef                   on
                    fe                  th
                       ct                  s
                           >
                               6
                                 m
                                     on
                                        th
                                           s
                Prioritising research

• Other methods
   – Unrelated to the returns from research
   – Research as a means changing clinical practice

• Statistical decision theory
   – Reduction in the costs of decision uncertainty
   – Value consistent with objective and constraints of
     service provision
   – Allocation within and between clinical areas
   – Allocation between research and service provision
          Design of future research
• EVPI exceeds cost of research?
   – Necessary but not sufficient


• Benefit of sample information
   – Predict possible future samples and posteriors
   – Expected net benefits with and with out sample information
   – Expected value of sample information (EVSI)


• Societal payoff from proposed research
   – EVSI - cost of sampling
   – Expected net benefit of sampling (ENBS)
            EVSI: predicting the results of proposed research

      N=1           NBj                 NBj                                         N=20         NBk             NBj


                                                                                                                  Prior Distribution
                                                                                                                  Predicted Posterior Distributions




                                                                              pdf
pdf




                                         Prior Distribution
                                         Predicted Posterior Distr ibutions




      0.0     0.2   0.4   NBj           0.6            0.8             1.0          0.0    0.2     0.4    NBj          0.6         0.8            1.0
                          Probability                                                                    Probability
                                   Expected net benefit of sample information (all parameters)
                      £3,500,000
                                             EVSI
                                             Cost(n)
                      £3,000,000             ENBS


                      £2,500,000
EVSI, cost(n), ENBS




                      £2,000,000




                      £1,500,000




                      £1,000,000




                       £500,000




                             £0
                                   0   100    200      300   400   500   600     700    800      900   1000   1100   1200   1300   1400   1500
                                                                               Sample size (n)
                                                              ENBS and n* for a portfolio of research designs
                                             £1,800,000
                                                                    PIP
                                                                    RSD,PHZ
                                             £1,600,000
                                                                    UPA
Expected net benefit of sample information




                                             £1,400,000



                                             £1,200,000



                                             £1,000,000



                                              £800,000



                                              £600,000



                                              £400,000



                                              £200,000



                                                    £0
                                                          0   100     200   300   400   500   600     700    800      900   1000   1100   1200   1300   1400   1500
                                                                                                    Sample size (n)
           Efficient research designs

•   Sufficient condition (ENBS>0)
•   Optimal sample size
•   Allocation of patients to the arms of a trial
•   Optimal stopping of sequential trial
•   Which endpoints should be included?
•   Optimal follow-up
•   What are relevant alternatives
•   Optimal portfolios of research
•   Optimal development programmes
     Regulation of health technologies
 “...A reasonable basis for a claim [of cost-effectiveness]
depends on a number of factors relevant to the benefits and
costs of substantiating a particular claim. These factors
include: the type of product, the consequences of a false
claim, the benefits of a truthful claim, the costs of
developing substantiation for the claim ...”

(FDA, section 114, modernization act of 1997)
       Regulation of health technologies
• Is it efficient to conduct further research?
   – Define a claim/submission as “substantiated” and evidence
     as “competent and reliable” when it is not efficient to
     conduct further research

• Should demand:
   – More evidence for some technologies as compared to others
   – Different types of evidence for different technologies
   – Different amounts of evidence for the same technology in
     different circumstances

• Role of the regulator?
   – Policing the priors
          Evaluating regulatory policies

• Societal value of information
   – Socially optimal development decisions
   – Socially optimal amount/type of evidence
• Commercial value of information
   –   Commercial payoff function
   –   Payoff conditional on licensing and reimbursement
   –   Commercially optimal development decisions
   –   Commercially optimal amount/type of evidence
• Optimal policies
   – Were commercial development and information decisions
     match societal needs
                        Implementation
• Value of information and the value of implementation
                                  Information
       Implementation       Current         “Perfect”
          Current             A                 B
          “Perfect”           C                 D

• Realising the value of information
   - EVPI = D-C
   - “Realisable” EVPI = B-A
• Value of changing clinical practice
   - Current information = C-A
   - With additional information = D-B
• Evaluate policies to change clinical practice
   - Can invest in information, implementation or both = D-A
                 UK Policy impact
• National Institute for Clinical Excellence
   – Pilot study to inform research recommendations
   – Guidance on methods of technology appraisal
       – Decision analytic modelling required
       – Evidence synthesis and probabilistic analysis required
       – Value of information analysis recommended


• NHS Health Technology Assessment Programme
   – Pilot study of setting research priorities

• Medical Research Council
   – Programme on value of information
   – Programme on evidence synthesis for decision making
   – Cancer Trials Unit Programme
         Implications and challenges
• Rational decision making
   - Adoption decisions
   - Research decisions
   - Additional objectives, constraints and irreversibility

• Characterising uncertainty
   - Indirect comparisons and networks of evidence
   - Bias and exchangeability
   - Model structure

• Computation
   - Correlation and complex data structures
   - Non multi-linear and complex models
   - Dimensions of design space

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:8
posted:11/30/2011
language:English
pages:31