Between Blind Faith and Skepticism Advice for coping with by Honey Claws

VIEWS: 4 PAGES: 13

									BETWEEN BLIND FAITH AND
SKEPTICISM:
ADVICE FOR COPING WITH
STATE OF THE ART
STATISTICAL METHODS

A largely road safety perspective by
Bhagwant Persaud
Ryerson University
(Paper review Chair, Committee ANB20)
BLIND FAITH
              Trust in “experts” –
              jump if they say so!

              Trust that research
              must be good because
              the methodology is
              exotic (sounding)

              Trust in meta-
              analyses of research
              results
EXOTIC METHODS IN TRB 09
PAPERS

Latent Class Model         Reliability Process
Poisson-Based Wavelet      Kuznets Relationship
Shrinkage Site Selection   Two-fluid Model
(WASSS) Method             Parameters
Bayesian Cohort Model      Extension Evaluation
Generalized Additive       Method
Models                     Ordered Response
Synergetics Theory         Models
Conditional Inference      Shannon Entropy
Forests                    Approach
SKEPTICISM
             Sophistication in
             research is
             unnecessary
             Research addresses
             problem that does not
             exist
             Simpler methods can
             be used effectively, so
             why bother with more
             complex ones
WHY BLIND FAITH MAY BE
MISGUIDED
 Incorrect application of more sophisticated
 methodology

 Application may be in the wrong context –
 solution looking for a problem

 Wheel has already been discovered
     Do we need new incident detection algorithms?
     Do we need new methods for network screening?
     Do we need new methods for doing regression analysis?
     Do we need new traffic assignment models?
WHY BLIND FAITH MAY BE
MISGUIDED
 Simpler methods may in fact be better – or at
 least as good
   Litmus test: Is a conclusion/decision different with a
   more complex method
     More complex methods are often “validated” with results
     from simpler methods!!
     More complex methods often “validated” by replicating
     reality but simpler methods not validated against the same
     reality


 The UPSHOT:
   SKEPTICISM CAN BE UNDERSTANDABLE
WHY SKEPTICISM MAY BE
MISGUIDED
 More complicated methods may in fact be
 necessary and address a real problem
   Skeptics may be in denial because the solution is too
   complicated to understand or apply
     E.g., Using Bayesian methods to account for regression to
     the mean in before-after studies


 The UPSHOT:
   BLIND FAITH CAN BE UNDERSTANDABLE
ILLUSTRATIONS FROM
BEFORE-AFTER ROAD
SAFETY STUDIES

Contexts have been slightly altered to protect the
innocent
  MISGUIDED SKEPTICISM: “REGRESSION
  TO THE MEAN DOES EXIST OR IS
  INSIGNIFICANT”
   Overwhelming evidence --untreated sites
      High accident frequency sites will have fewer
      accidents in a subsequent period and vice versa


   Overwhelming evidence -- treated sites e.g., the
   compelling case for improving protection at rail –
   highway crossings
Accidents before              = 286
Accidents after               = 114
Apparent savings              = 172 (60% reduction)
Accidents expected (EB Method)= 208
Actual savings     208 - 114  = 94 (45% reduction)
MISGUIDED BLIND FAITH EXAMPLE
1
An empirical Bayes treatment evaluation study is
based on 3 sites
EB estimates that 4.1 accidents would have been
recorded “after” had treatment not been implemented
4 accidents were actually recorded.
The “accident modification factor” is approximately
(4/4.1) = 0.976 with a standard error of 0.40.

It is concluded that the result is statistically
significant since 0 is not included in the 95%
confidence interval of 0.976 +/- two standard errors
MISGUIDED BLIND FAITH EXAMPLE
2
Empirical Bayes treatment evaluation studies require
safety performance functions (SPFs)

SPFs are calibrated from untreated sites similar to the
treated ones before treatment.

These functions correct for regression to the mean.

One study calibrated the following intersection SPF
from untreated sites PLUS treated ones before treatment

Accidents/year = α (AADT)-0.15 (# of Left lanes)-0.005
MISGUIDED BLIND FAITH EXAMPLE
3
 Meta analysis combines and weighs results for
 numerous studies of the same treatment/factor to
 arrive at a “best” estimate
 There is a tendency to place blind faith in the
 results of the meta analyses.

 A recent meta analysis concluded that a treatment
 can be harmful. Results were combined from:
   Two flawed (unpublished) EB study with large
   sample size the showed a safety disbenefit
   A solid, published, EB study with relatively small
   sample size that showed a safety benefit
MISGUIDED BLIND FAITH EXAMPLE
4
When entire populations are treated there is no bias
by selection and no regression to the mean
Thus the EB methodology is not needed to evaluate
an overall treatment effect.
A comparison group is used to “control” for changes
between the periods before and after treatment
A study of truck driver legislation compared “fatigue”
accidents (F) before and after; “non-fatigue” (NF)
accidents was used as a comparison group

NF increased; F decreased – even more when NF
was used as a control

								
To top