Reliability of Screening Tests

Document Sample
Reliability of Screening Tests Powered By Docstoc
					      Reliability of Screening Tests
RELIABILITY:

The extent to which the screening test will
produce the same or very similar results each
time it is administered.

--- A test must be reliable before it can be valid.

--- However, an invalid test can demonstrate
    high reliability.
      Reliability of Screening Tests
Sources of variability that can affect the
  reproducibility of results of a screening test:

1. Biological variation (e.g. blood pressure)
2. Reliability of the instrument itself
3. Intra-observer variability (differences in
       repeated measurement by the same
       screener)
4. Inter-observer variability (inconsistency in
      the way different screeners apply or
      interpret test results)
    Use of Multiple Screening Tests
Sequential (two-stage) testing:

A less expensive, less invasive, or less
uncomfortable test is performed first…

those who screen positive are referred for
further testing using a test which may have
greater sensitivity and specificity…

reduces false positives, hence an increase in net
specificity.
  Use of Multiple Screening Tests

Simultaneous testing:

Multiple tests are used simultaneously…

Person tests “positive” if there is a positive
result on any of the tests employed…

reduces false negatives, hence an increase in
sensitivity … but at the expense of decreased
specificity.
Measuring the performance
 (yield) of a screening test
          Performance Yield

   People with positive screening test
    results will also test positive on the
    diagnostic test:
         Predictive Value Positive (PV+)
   People with negative screening test
    results are actually free of disease
         Predictive Value Negative (PV-)
            Performance Yield
                   True Disease Status
                       +        -

 Results of +           a          b
 Screening
    Test
            -           c          d

Predictive value positive (PV+): The probability that a
person actually has the disease given that he or she
tests positive.

      PV+     =    a / (a + b)
            Performance Yield
                   True Disease Status
                       +        -

 Results of +           a           b
 Screening
    Test
            -           c           d

Predictive value negative (PV-): The probability that a
person is truly disease free given that he or she
tests negative.

      PV-     =    d / (c + d)
              Performance Yield
                     True Disease Status
                         +        -

  Results of +           400         995
  Screening
     Test
             -           100        98905

Sensitivity: a / (a + c) = 400 / (400 + 100) =       80%
Specificity: d / (b + d) = 98905 / (995 + 98905) =   99%
PV+:         a / (a + b) = 400 / (400 + 995) =       29%
PV-:         d / (c + d) = 98905 / (100 + 98905) =   99%
            Performance Yield
                  True Disease Status
                      +        -

  Results of +        400         995
  Screening
     Test
             -        100        98905

PV+:       a / (a + b) = 400 / (400 + 995) = 29%

Among persons who screen positive, 29% are found
to have the disease.
            Performance Yield
                  True Disease Status
                      +        -

  Results of +        400         995
  Screening
     Test
             -        100        98905

PV-:       d / (c + d) = 98905 / (100 + 98905) = 99.9%

Among persons who screen negative, 99.9% are found
to be disease free.
            Performance Yield
Factors that influence PV+ and PV-

1.   The more specific the test, the higher
     the PV+

2.   The higher the prevalence of preclinical
     disease in the screened population, the
     higher the PV+

3.   The more sensitive the test, the higher
     the PV-
             Performance Yield

Prevalence (%)   Sensitivity   Specificity    PV+

     0.1           90%            95%        1.8%

     1.0           90%            95%        15.4%

     5.0           90%            95%        48.6%

     50.0          90%            95%        94.7%
            Performance Yield

Thus, the PV+ is maximized when used in “high
risk” populations since the prevalence of pre-
clinical disease is higher than in the general
population….

screening a total population for a relatively
infrequent disease can be very wasteful of
resources and may yield few previously
undetected cases.
       Effectiveness of Screening

Evaluating if the screening program reduced
morbidity and mortality from the disease:

1.   Overall shift in severity of disease at the
     time of diagnosis.

2.   Compare cause-specific mortality from
     among those whose disease was picked up
     by screening versus those with a diagnosis
     related to symptoms.
      Effectiveness of Screening

Evaluating if the screening program reduced
morbidity and mortality from the disease:

3.   Reduction in disease-related
     complications.

4.   Improvement of quality of life in screened
      individuals.
         Effectiveness of Screening
In reality, establishing the sensitivity and
   specificity of screening tests may be difficult…

often times, data are only available on persons
   who screen positive and are referred for
   further testing.

     a    b      Data are available for cells “a” and
                 “b” only.
     c    d
                 Permits calculation of PV+ only
     Effectiveness of Screening
Sources of bias in evaluating screening
  programs:

1. Self-selection bias (volunteer bias)

2. Lead time bias

3. Length bias

4. Over-diagnosis bias
        Effectiveness of Screening

1.    Self-selection bias (volunteer bias):

---   Volunteers for screening programs may be
      healthier, on average, than persons who do
      not participate in screening programs.

On the other hand….
---  The “worried well” may be more likely to
     participate and may be at overall higher risk
     due to family history or lifestyle
     characteristics.
       Effectiveness of Screening
2.    Lead time bias:

Lead time: The interval between “diagnosis” of
disease at screening and when it would have
been detected from clinical symptoms.

---   Survival may appear to be increased
      among screen-detected cases simply
      because diagnosis was made earlier in the
      course of the disease.
        Effectiveness of Screening
3.     Length bias (prognostic selection):

----   The overrepresentation among screen-
       detected cases of those with a long pre-
       clinical phase, and thus a more favorable
       prognosis.

---    Those with a long pre-clinical phase are
       more readily detectable by screening than
       more rapidly progressing cases with a
       short pre-clinical phase.
        Effectiveness of Screening
4.     Over-diagnosis bias:

----   Persons who screen positive and are truly
       disease free (false positives), yet are
       erroneously diagnosed as having the
       disease.

---    Since these persons are truly disease
       free, we expect a more favorable long-
       term outcome – giving the appearance of
       an effective screening program.
        Effectiveness of Screening
Possible interpretations for null results in a
screening program evaluation include:

----   The disease has an extremely short
       detectable pre-clinical phase.

---    Current therapeutic intervention is no
       more effective when provided earlier than
       at usual time of diagnosis.

---    Inadequacies of care provided to those
       who screen positive.
     Effectiveness of Screening

Study Designs Used to Evaluate Screening
Programs:

1.   Ecological Studies

2.   Observational Analytic Studies

3.   Randomized Trials (infrequent and
     difficult to carry out)