Chances Are

Document Sample
Chances Are Powered By Docstoc
					Chances Are

     A look at probability
     and its application to
     beef production and
     diagnostic testing
Everyday probabilities

            Some every day probabilities
Probability concepts

   Likelihood
   Predictability
   Certainty/Confidence (or the lack of)
Basis of probability

   Counting outcomes
    –   How many cows do I have (100)
    –   How many cows have calves at their side this year
    –   How many cows were exposed to the bull (97)
    –   How many cows were diagnosed pregnant (91)
    –   How many cows had bull calves (37)
    –   How many cows required assistance at calving (7)
Truth of Probability

“Not everything that counts can be counted,
and not everything that can be counted
counts.” Einstein

   What is the likelihood that a cow selected from
    the herd described in the previous slide does
    not have a calf at her side?
    –   Likelihood=Odds count the possibilities
            The number of cows that did not have a calf at their side
             was 18
            The number of cows that did have calves at their side was 82
            So the likelihood/odds would be 18:82

   If you select a cow at random from the herd what are
    the chances that you will select a cow without a calf.
     –   Prediction=Probability= the potential of an event expressed
         as a relative frequency or mathematically as f/n where n is
         the total number of events and f is the number of events of
     –   The sum of all possible events is 1. If I flip a coin the
         probability it is a head is .5 a tail .5 the probability it is a head
         or a tail is 1
     –   In this example n=100 and f=18 the probability of any one
         cow selected meeting the criteria is 18/100 or .18 and if the
         selection was made 100 times 18% of the time you would
         expect to select a cow without a calf.
Predictability               (continued)

   Sometimes we might be interested in how many times
    an event might occur, such as to evaluate a new test I
    test 5 cows from a herd for “Disease X” what is the
    probability that the new test will find 3 test positive and
    2 test negative?
   For simplicity assume.
    –   We already know the status of the animals as being infected.
        The prevalence is 1.00
    –   After testing the animals are returned to the herd (in this
        scenario with known infection it doesn’t make any difference)
    –   The Se of the test is .9 (the probability the test is negative is .1
        and the probability the test is positive is .9) & the Sp is 1
So how do we approach the
Let TP = test positive and TN = test negative
TP=.9 TN=.1 If we perform the test 5 times the outcome
  would look like this:
                          TP5+5TP4TN1+10TP3TN2+10TP2TN3 +5TP1TN4 +TN5
The coefficient1 corresponds to the number of possible
  combinations and the exponent to the number of times
  that event might occur, there are 10 possible
  combinations of getting 3 test positives and 2 test
  negatives. The probability then is 10*.93*.12 or .0729.
  1   coefficien t 
                       k!(n  k )!
  Expanding the problem

What would be the probability of at least 3 test
If you consider that you don’t know they are all infected
    but know the prevalence then values change by
    factoring in prevalence and the initial formula would
    change to look like this; (still Sp of 1)
                    P5(TP+TN)5 + SP5(1-P)5
If you consider a test that is less than 100% specific
    then you ADD in the confusion of the false positive
    and a more complex formula.

   If I select 5 cows at random to check to see if
    the cows have been exposed to “X disease”
    using a newly validated diagnostic test, how
    certain/confident would I be that by testing 5
    cows a statement regarding the presence of “X
    disease” could be made.

Sample size formulas

              1           D  1
   n  1         D
                        * N 
                              2  

         n=required sample size
         α=confidence level
         D=number of diseased animals
         N=total population or herd size
Short comings of the formula

   Only applies to a perfect test
    –   Diagnostic tests have limitations
            Se
            Sp
            A positive test does not indicate disease nor a negative test
             an absence of disease
   Trial and error to determine confidence (or a
    complex math formula)
    What information do you need to
    answer the questions about sample

   Test parameters Se and Sp is the test perfect
    or imperfect?
   Is the population in question large or small?
   Estimated prevalence?
    –   Sources for estimated prevalence
            Literature
            National reports, NAHMS
            Diagnostic Labs (may be a biased answer)
            Clinical experience/local surveys (a topic for later)
            Others?
Methods to estimate confidence

            Sample size spread sheet
Formulas behind the spreadsheet

                Sample size to detect a positive animal with an imperfect test and a large
                  group (Can be used with Goal seek to determine prevalence if , Se, Sp and
                  sample size is known).

                                                              ln( )
                                             S                                 a
                                                  ln[ p (1  Se)  (1  p ) Sp]

                                             Where  = 1 -  (where  is the desired level of confidence)
                                                   S = sample size
                                                   p = prevalence
                                                   Se = test sensitivity
                                                   Sp = test specificity

                   Formula to calculate  (the level of confidence (probability) of detecting an
                    infected animal) for an imperfect test and a small group (Can be used with
                    Goal seek to determine prevalence if , Se, Sp and sample size are set to
                    predetermined values.)

                                                       pM  (1  p ) M 
                                                          
                                                       x  ( s  x)   
                                                                   

                                                                           (1 Se) Sp
                                                                                  x   (sx)
                                                 x 0       M 
                                                             
                                                            s 
                                                             
                                             Where  = 1 -  (where  is the desired level of confidence)
                                                   S = sample size
                                                   p = prevalence
                                                   M = herd size
                                                   Se = test sensitivity
                                                   Sp = test specificity

            Vose, David. Risk Analysis a Quantitative Guide
Kennedy’s oversimplified
   P=the proportion diseased/infected/of interest (D) in a
    population (N)
   (1-P)=the proportion NOT D
   Se=the proportion that are D that a test detects (T+|D)
   (1-Se)=the proportion D the test fails to detect (false
   Sp=the proportion NOT D (ND) that a test correctly
    identifies negative (T-)
   (1-Sp)=the proportion ND a test incorrectly identifies D
    (false positives)
Kennedy’s oversimplified formulas

A.          D                    N=Total Population
     P                          D=Diseased
            N                    ND=Not Diseased
                  ND             P=Prevalence
B.   (1  P )                   Se=Sensitivity
            T 
C.   Se                         T+=Test positive
             D                   T-=Test negative
                   (T  | D )
D.   (1  Se) 
            T 
     Sp 
E.          ND
                   (T  | ND )
     (1  Sp ) 
F.                     ND
     More of Kennedy’s oversimplified

                      ( P  Se)
                                            PPV=Positive Predictive Value

     PPV 
                                            NPV=Negative Predictive Value
            ((1  Se)  (1  P)  P  Se)
                                            EFF=Efficiency proportion of positives
                                            and negatives correctly classified
                                            AP=Apparent Prevalence
B.   NPV 
            (1  Se)  SP
C.   EFF  P  Se  (1  P)  Sp
D.   AP  P  Se  (1  P)  (1  Sp)
Sources of diagnostic test error
   the lab never gets it right
Where testing error happens

   Pre-analytical error sources, wrong sample,
    mishandled sample, improper sample collection, etc.
    Starts from collection and goes until analysis begins.
   Analytical error, analytic variation such as mechanical
    wear and tear or inherent error such as that seen with
    a set of spring type scales.
   Biological variation, an average means some are
    higher and some are lower.
   Post-analytical error, reporting errors misread values
    or misreported values, transposition of figures, etc.
Test expectations

   Repeatability
    –   Consider flipping a fair coin 5 times the chances of all being
        heads is.5x.5x.5x.5.x5 or 3.125%.
    –   Consider flipping a weighted coin 5 times that is expected to
        be heads 90% of the time the chances that 5 heads will be
        returned is .9x.9x.9x.9x.9 or 59%, meaning 41% of the time
        one or more tails would occur.
    –   The same principle applies to a diagnostic test.
    –   So how do you interpret when two labs disagree?
                                                            Did the test Miss

   How many miles would you expect to drive before you
    get a nail in your tire?
   In 2004 how many aviation fatalities occurred per
    100,000 hours flown?
    –   General aviation 2.15
    –   Commercial aviation .08
   In every 1000 diagnostic tests performed how many
    times does the test fail beyond incorrect results due to
    Se and Sp?
   Does repeating a task increase the probability
    something will go wrong? i.e.
    –   You and your neighbor purchase new tires for your cars. Both
        of you drive to the same place to work each day on the same
        road, but you come home for lunch while he doesn’t, who is
        more likely to get a nail in one of their tires?
    –   Aviation gas gets cheap so I fly twice as much will I be more
        likely to become a fatality? Maybe
    –   If I run 1000 individual diagnostic tests am I more likely to
        misclassify an animal than if I ran 10 test each containing
        samples of 100 animals? Controversial
        Each mile you drive, each hour you fly, or each test you run are independent events.
        Over a given period the number of times that an event occurred is a rate
        (rates have units probabilities do not) Risk is the probability of a negative event.
        Think about life insurance. On the other hand dependent events have changing probabilities.
Why pooled testing?

   Pooled testing offers advantages over
    individual testing
    –   Allows the diagnostician to take advantage of highly
        sensitive and specific tests while minimizing cost
    –   Diminishes cumulative testing error over individual
Why not pooled testing?

 –   Potential impact of dilution diminishing Se
 –   Logistical requirements for pooling samples
     (pooling of individual samples can be labor
 –   Loss of samples for follow-up testing on
     positive pools
Assumptions associated with
pooled testing

   Pooled test Se must be approximately the
    same as individual test Se
   Samples must be easily obtainable
   Pools must represent a homogenous mixture
    of samples
   The outcome is binomially distributed, i.e. a discrete
    probability distribution of the number of successes in a sequence of
    independent yes/no events each yielding success with a probability p
Our human counterparts institute
pooled testing strategies

   For generations the military has attempted to screen its
    applicants/inductees to insure they were healthy and
    would not become a liability on the battlefield.
   Early screening involved a physical exam to insure all
    parts were present and properly located.
   Later blood tests for infectious disease became
    available and were included in the screening process
Pooled testing during WWII
                    Syphilis had plagued the
                     military since the first soldier
                     marched off to war.
                    They could mandate controls
                     after recruitment to help slow
                     its spread but that was not
                    To minimize the risk they
                     looked to tests that would
                     detect carriers before they
                     were inducted.
Military Test for Syphilis
                       The test used during WWII to
                        insure inductees were free
                        from infection was a
                        Wasserman type blood test.
                          – A sample of blood is
                            drawn from each inductee.
                          – Then each sample is
                       The procedure was
                        expensive, time consuming
                        and amplified testing error.
  Time and cost of test encouraged a
  change in the process
      The military implemented a procedure where a small
       quantity of blood from multiple inductees was pooled
       and a single test was run on the pool
      Sufficient blood remained that the positive individual
       among the pool could be identified.
      The study of using pooled testing as a screen lead to
       two conclusions/considerations on pooling.
         –    Prevalence must be low enough to make pooling more
         –    It must be easier to obtain an observation on a group than on
              the individuals within the group (minimize the number of tests).
reference Robert Dorffman The Detection of Defective Members of Large Populations, Annals of Mathematical Statistics,
Vol. 14, Dec 1943
  More Recent Use of Pooled Testing
  Strategy in Human Medicine

       ELISA and Western Blot tests were used to screen for
        human immunodeficiency virus (HIV).
       An ELISA test was used initially and then a Western
        Blot was the confirmatory test.
       The ELISA alone was prone to falsely classifying
        samples positive and therefore may result in an
        overestimation of prevalence,
       Western blots were done to confirm HIV presence, but
        are expensive.

Reference; Tu, Litvak, Pagano. Studies of AIDS and HIV surveilance, screening tests: can we get more by doing less?, Xin M. Tu
Eugene Litvak, Marcello Pagano, Statistics in Medicine, Vol 13, 1905-1919 (1994).
Pooling to Screen for HIV

   Blood Mobile—time and money made
    individual testing at the human “herd” level
    unappealing plus creating issues of false
   So what about pooling samples,
    –   Up to 15 samples were pooled without a loss of Se
    –   Pooling diminished false positives
    –   Less cost
    –   Fewer tests were needed
A step further on the pooling

   A JAMA article Jul 2002 described the
    following protocol.
    –   Pool samples of blood in groups of 10 to determine
        the absence of HIV antibodies
    –   From the negative pools, form pools of 90
        individuals and run RT-PCR to detect the presence
        of the HIV virus
    –   Used to find the presence of the virus prior to the
        time antibodies are formed allowing earlier
        treatment and preventing spread
Trial results
   8000 people visiting publicly funded HIV clinics in the
    Southern USA were subjects of the test
   Antibody tests found 39 long term infected individuals
    (those that had formed Ab to HIV)
   RT-PCR testing of pools of 90 serologically negative
    samples found 4 additional positive individuals.
   The cost to find the 4 additional individuals was
    $4109.00 per individual, if individual PCR’s had been
    done the cost would have been ~$360,000.00 per
    positive individual.
Pooled testing/Screening Human

   Screening tests have been used to identify
    infected individuals in large populations, such
    as the military or blood donors.
   Screening tests are used to estimate
Veterinarians and screening tests

   Limited applications of screening test
    –   Salmonella contamination of eggs
    –   Johne’s fecal pools
    –   BVD
    –   T. foetus

   Human medicine has implemented the concept
    of pooling when human life is at risk, should
    veterinarians be open to the concept to
    address herd health issues in livestock?
   Possible veterinary applications
    –   Screening to evaluate treatment success
    –   Determine prevalence prior to instituting control
    –   Screening to evaluate vaccine success
Estimation of prevalence using
pooled PCR

                    x           k

                Se     
      AP  1       m 
                SeSp1 
                        
               =        
      Se  sensitivity Sp  specificity
      k  pool size m  pools tested
      x  positive pools
      formula from prev med 39 (1999)
      Cowling, Gardner, Johnson

Shared By: