Docstoc

LECTURE AND DISCUSSION -- APPRAISING RISKS

Document Sample
LECTURE AND DISCUSSION -- APPRAISING RISKS Powered By Docstoc
					APPRAISING KNOWLEDGE UNDER UNCERTAINTY

0. OVERVIEW OF UNIT
PERSPECTIVES ON KNOWLEDGE APPRAISAL -- REVIEW VISCUSI MORGAN; ADD JASANOFF
METHODS OF APPRAISAL:-- FULL SPECTRUM FROM IN HOUSE TO ADVERSARIAL TO THIRD PARTY
HEALTH & ENVIRONMENT CASES
    PASSIVE SMOKING - FUNDING AND RESEARCH=> PEER REVIEW AND PUBLICATION
    TRANSFATS - FUNDING AND RESEARCH => DISCOUNTING INFO ON BASIS OF FUNDING     LEC
    IMPLANTS - JUDICIAL APPRAISAL OF RISKS => DAUBERT ON SCIENCE
    PARTICULATES AND HEALTH EFFECTS UNCERTAINTY => THIRD PARTY ASSESS HEI
    METHYL MERCURY AND HEALTH EFFECTS UNCERTAINTY => THIRD PARTY ASSESS NRC LEC
SECURITY CASES
    ASSESSING PAST PERFORMANCE OF PATRIOT => THIRD PARTY AMERICAN PHYSICAL SOCIETY
    PROJECTING CAPABILITIES OF NATIONAL MISSILE DEFENSE => CLASSIFICATION AND GAG
    ASSESSING THREAT BOMBER&MISSILE GAP,COUNTERFORCE=>BURPOL&COMPET ASSESS
    ASSESSING UNCONVENTIONAL THREATS: IRAQ AND WMD =>ALL OF ABOVE . . .

1. BASIC PERSPECTIVES ON ASSESSMENT
POSITIVIST ECONOMIC
POSITIVIST DECISION ANALYTIC
CONSTRUCTIVIST
=> WHAT DOES EACH OF PERSPECTIVES TREAT AS OBJECTIVE AND AS SUBJECTIVE?
=> WHAT ARE MOST SIGNIFICANT AREAS OF DISAGREEMENT?
=> DO YOU ENTER THIS UNIT WITH A PREFERENCE FOR ANY OF THE THREE?
=> CAN THESE VIEWS SIMPLY BE COMBINED, OR DO YOU HAVE TO MAKE A CHOICE?
A. POSITIVIST ECONOMIC – REVIEW OF VISCUSI ET AL
SECURE OBJECTIVE KNOWLEDGE ON . . .
     RISKS AND UNCERTAINTY == SEE TABLES 19.3
     VALUE OF LIFE == WILLINGNESS TO PAY SURVEY AND LABOR DATA
     COSTS OF MITIGATION OF RISKS == TABLE 20.4
APPLY UTILITARIAN EXPECTED VALUE DECISION CRITERION TO EVALUATE ALTERNATIVES
WHERE ACTUAL OUTCOMES DIFFER FROM IDEALIZED MODEL, MOVE REALITY TOWARD MODEL
     CORRECT FOR ERRORS IN RISK PERCEPTION? MASS OR EXPERT
     TAKE ACCOUNT OF COSTS
     USE MARGINAL RATHER THAN AVERAGE COSTS
     SET STANDARDS TO UTILIZE FULL INFORMATION ON VARIATION IN COSTS AND RISKS

B. POSITIVIST ASSESSMENT AND DECISION SCIENCE – REVIEW OF GRANGER MORGAN
SECURE OBJECTIVE KNOWLEDGE ON RISK ASSESSMENT AND UNCERTAINTY . . .
     EXPOSURES: MODELS AND FAILURE MODES
     EFFECTS: DOSE RESPONSE
WHERE CONFRONT UNCERTAINTY....
     USE BEST AVAILABLE INFORMATION
     USE SUBJECTIVE ASSESSMENTS WITH SENSITIVITY ANALYSIS
WHAT ARE SUBJECTIVE POLICY ISSUES?
     CHOICE OF UTILITY VS RIGHTS VS TECHNOLOGY BASED APPROACHES TO VALUATION
     EXPLICIT JUDGEMENT ON CONSERVATIVE OR LIBERAL STANDARDS FOR RISK APPRAISAL
CHOOSE STANDARDS FOR VALUING OUTCOMES -- BUT BE CONSISTENT
WHERE ACTUAL OUTCOMES DIFFER FROM OBJECTIVE ASSESSMENTS . . .
     DO NOT THROW OUT MASS VIEWS AUTOMATICALLY FOLK WISDOM POSSIBLE:
     DEVIATIONS FROM SIMPLE EXPECTED VALUE DREAD, LEVEL OF UNDERSTANDING, EQUITY....
     USE RISK COMMUNICATION TO PROVIDE FOUNDATION FOR INFORMED CHOICE
C. CONSTRUCTIVIST - JASANOFF


Knowledge claims are deconstructed during the rulemaking process,
exposing areas of weakness or uncertainty and threatening the cognitive
authority of science. At the same time, the legitimacy of the final regulatory
decision depends on the regulator's ability to reconstruct a plausible scientific
rationale for the proposed action. The processes of deconstructing and
reconstructing knowledge claims give rise to competition among scientists,
public officials, and political interest groups all of whom have a stake in
determining how policy-relevant science should be interpreted and by whom.
All of the actors use boundary defining language in order to distinguish
between science and policy, and to allocate the right to interpret science in
ways that further their own interests.
                                                    Sheila Jasanoff
PHASES AND PROCESSES
* Knowledge claims are deconstructed during the rulemaking process,
exposing areas of weakness or uncertainty and threatening the cognitive
authority of science.
* At the same time, the legitimacy of the final regulatory decision
depends on the regulator's ability to reconstruct a plausible scientific
rationale for the proposed action.

STAKES AND METHODS
* The processes of deconstructing and reconstructing knowledge claims give
rise to competition among scientists, public officials, and political interest
groups all of whom have a stake in determining how policy-relevant science
should be interpreted and by whom.
* All of the actors use boundary defining language in order to distinguish
between science and policy, and to allocate the right to interpret science in
ways that further their own interests.
(1) MODUS OPERANDI
OBSERVE OUTCOMES AND DEBATES OVER RISK ASSESSMENT AND RISK MANAGEMENT
SEEK TO UNDERSTAND CHANGES IN DISCOURSE IN TERMS OF . . .
*    MATERIAL INTERESTS OF ACTORS: RESOURCES AND STRUGGLE TO ACQUIRE THEM
*    IDENTITY AND SELF DEFINITION BY ACTORS: HOW SCIENTISTS SHOULD BEHAVE
*    PRIOR USES OF CONCEPTS: CONSTRAINTS OF PATH DEPENDENCY

(2) EXAMPLES OF CONTESTED BOUNDARIES BETWEEN SCIENCE AND POLICY
a. SCIENCE AS DISTINCT FROM TRANSCIENCE:
      EXAMPLE: GORSUCH/TODHUNTER/ASHFORD
      SCIENTISTS TRY TO DRAW LINE TO PRESERVE LEGITIMACY BY CUTTING THEMSELVES OFF
      FROM AREAS OF UNCERTAINTY AND CONFLICT....
      BUT CAN ONLY DO SO BY GIVING UP CONTROL OVER BLURRY AREAS
b. RISK ASSESSMENT AS DISTINCT FROM RISK MANAGEMENT
      EXAMPLES: BENZENE AND OSHA, MCCRAY AND NRC VS GRAMM AND OMB VS RUCKELSHAUS
      CANNOT DIFFERENTIATE CLEANLY BETWEEN RISK ASSESSMENT AND MANAGEMENT . . .
      BUT NEED TO PRETEND TO DIFFERENTIATE CLEANLY
c. SCIENTIFIC PEER REVIEW AS DISTINCT FROM POLICYMAKING
      EXAMPLES: EPA AND SCIENCE ADVISORY BOARD, FDA PUBLIC BOARD OF INQUIRY ASPERTINE
      POLICY MAKERS SEEK PEER REVIEW TO EVADE RESPONSIBILITY BUT....
      DOES NOT WORK WELL IN AREAS CHARACTERIZED BY UNCERTAINTY AND CONTROVERSY.

(3) BOTTOM LINE -- "science is subjected to extreme deconstruction in the US regulatory process ... the
legitimacy of regulatory decisions ultimately rests on persuasive reconstruction of justifications in scientific,
economic and legal terms. "
      NOTE: SHOULD VISCUSI AND MORGAN BE SEEN AS JASANOFF'S DATA?
      NOTE: IS CONSTRUCTIVIST USE OF LANGUAGE IMPRECISE OR SOPHISTICATED OR BOHT?
D. CREDIBLE ASSESSMENT OF SCIENTIFIC AND TECHNICAL INFORMATION

   IN HOUSE EVALUATION AS BASELINE - DoD, EPA, OSTP, OTA

   NOMINALLY NONADVERSARIAL METHODS – “HONEST” BROKERS (MCCRAY)
      EXTERNAL CONSULTANTS -- BELTWAY BANDITS, CONSULTING FIRMS
      FFRG - RAND AND MITRE and LINCOLN LABS
      EXTERNAL ADVISORY GROUPS - EPA SCIENCE ADVISORY GROUP AND CASAC
      AD HOC CONFERENCE - NIEHS-OSTP METHYL MERCURY FAROE - SEYCHELLES
      THIRD PARTY ASSESSMENT SPECIALISTS - HEALTH EFFECTS INSTITUTE
      PROFESSIONAL ASSOCIATIONS - NAS/NRC, APS
      UNIVERSITIES DoD=>TRW       GAO=>LINCOLNLABS=>MIT
      PEER REVIEW / PUBLICATION (SHORT TERM) -- REVIEWERS AND EDITORS
      LAW – UNBIASED JUDGE SCREENS EVIDENCE AND DECIDES

   NOMINALLY ADVERSARIAL METHODS
      LAW -- ATTORNEYS PRESENT SIDES                  => JURY
      TEAM A TEAM B AS CHECK ON MOTIVATED BIASES      => DECISIONMAKER
      PEER REVIEW / PUBLICATION (LONG TERM)           => SCIENTISTS
WHERE DO THESE KNOWLEDGE APPRAISAL OPTIONS FALL IN DIAGRAMS BELOW?

CREDIBILITY     |
                |
                |
                |
                |
                |
                |
                |_______________________________________________
                                 CONTROLLABILITY
SECRECY         |
                |
                |
                |
                |
                |
                |
                |_______________________________________________
                                 ACCURACY
SOME OTHER DIMENSIONS
RESPONSIBILITY -- DUCKING RESPONSIBILITY FOR CONTROVERSY BY HANDING OFF
TIMELINESS -- DELAYING DECISIONS UNTIL THE NEXT WATCH
FROM PERSPECTIVE OF DEMANDER OF EVALUATION
CORPORATION/REGULATORY AGENCY/CONGRESS
TO WHOM WILL YOU TURN FOR ASSESSMENT ON WHAT ISSUES?
* WHITE HOUSE AND DoD ON LIKELY EFFICACY OF NMD?
* AG-INDUSTRY ON SAFETY OF GMO FOODS?
* DoD ON EFFICACY OF V22?
* WHITE HOUSE AND EPA ON CLIMATE CHANGE?

FROM PERSPECTIVE OF SUPPLIER OF EVALUATION
SUPPLY: GAIN CREDIBILITY, WEALTH, POWER . . . & AVOID EXTINCTION
TOPIC SELECTION AND DEFINITION - WHICH ISSUES WOULD YOU TAKE?
FUNDING SOURCES - HOW DIVERSE?
PANEL COMPOSITION - HOW WOULD YOU PICK?
PROCESS OF APPRAISAL - HOW WOULD YOU PROCEED?
   .... CAN POTENTIAL EVALUATORS FINESSE THE THIRD RAIL PROBLEM?

HOW WIDELY CAN INTERMEDIARIES BE USED TO ASSESS KNOWLEDGE?
ORG       HISTORY      SETTING        LEADERSHIP     MODUS             KEY DECISIONS   INDEPENDENCE
                                                     OPERANDI
CASAC     1977 CAA     EPA            7 MEMBERS      6 TIMES A YEAR,   EPA             HOUSED IN
                       FEDERAL        5 ACADEMIC     DEVOLVE WORK      ADMINISTRATOR   SCIENCE
                       ADVISORY       1 STATE        ONTO PANELS       CONTROLS        ADVISORY
                       COMMITTEE      AGENCY                           MEMBERSHIP      BOARD
                       ACT                                             AND PROJECTS
                                                                                       OPEN WITH
                                                                                       VOTING
HEI       1980         EPA AND        BOARD COX,     EXTRAMURAL        HEALTH          PRESTIGE AND
          NONPROFIT    MOTOR          EPA AND        TEAM TO           RESEARCH        FIREWALLS, DE
                       VEHICLE        KENNEDY        CONDUCT           COMMITTEE       NOVO REVIEW
                                                     RESEARCH AND      SETS PROJECTS
                                                     REAPPARAISAL      AND
                                                     S                 PERSONNEL
NIH CDP   1977         WI OFFICE OF   DIRECTOR NIH   ROUTINIZED --     NIH DIRECTOR    ABSENCE OF
          120          MEDICAL        AND OMAR       NONFED NON        AND OMAR        ADVOCATES
          CONSENSUS    APPLICATION    STAFF          ADVOC             STAFF           REDUCES
          STATEMENTS   S RESEARCH                    1.5 DAYS          APPROVE         PRESSURE
          TO DATE      OMAR IN NIH                   TESTIMONY         TOPICS          (EXCEPT NIH?)
                                                     COMPOSE AND       PROJECTS
                                                     RELEASE           PERSONNEL
                                                     STATEMENTS        * PUBLIC
                                                                       HEALTH IMPORT
                                                                       * SCIENTIFIC
                                                                       INFORMATION
                                                                       * NEED
NRC   1916 WWI    WORKING        ELECTED        12-15 EXPERTS   EXCOM OF       FUNDERS AND
                  ARM OF NAS     MEMBERSHIP     4-6 MEETINGS    GOVERNING      SPONSORS DO
                  AND NAE AND    NRC BORAD      18-27 MONTHS    NRC BOARD      NOT CONTROL
                  INST MEDIC     OF FIVE FROM   COSTS 250k+     APPROVES
                  STAFF 11OO     EACH ORG       STAFF           PROJECDTS
                  1000 COMM                     INFLUENCE       COMMITTEE
                  10000          STANDING       VARIABLE        MEMBERS
                  EXPERTS        COMMISSIONS    COMMITTEE       APPROVED BY
                  105TH CONG                    BALANCE         SUBORGS
                  INITIATED 45                  CRITICAL        RELEASE
                  STUDIES                                       REQUIRES
                  300-350                                       SIGNOFF OF
                  REPORTS/YR                                    COMMITTEE
                  2/3                                           MEMBES PLUS
                  KNOWELDGE                                     REPORT
                  ASSESSMENT                                    REVIEW
                  1/3 RESEARCH                                  COMMITTEE
                  PRIORITIES
                  FOR FEDS


OTA   1972-1995   TECH           LEGISLATIVE    18-30 MONTHS    TAB APPROVAL   NOT
                  ASSESSMENT     OVERSIGHT      ADVISORYY       FOR NEW        MUCH....BUT
                  BOARD OF 12    AND FUNDING    COMMITTEE       PROJECTDS      STILL PULLED
                  LEGISLATORS    EXTERNAL       PLUS STRONG     APPOINTMENTS   OFF REPORTS
                                 ADVISORY       STAFF 3-5       OF PERSONNEL
                                 BOARD          2 TO 3          BY DIRECTOR
                                                MEETINGS        RELEASE TAB
                                                LONG REPORTS    APPROVAL
CASES OF KNOWLEDGE ASSESSMENT: WHERE DO THESE MODELS WORK BEST? WORST?

ACADEMIC PEER REVIEW -- PASSIVE SMOKING; TRANSFATS; CHEMOTHERAPY
=> REVIEWERS AND EDITORS ACCEPT CONVENTIONS, RESIST UNUSUAL
=> BUT BROAD ARRAY OF MATERIAL IS EVENTUALLY PUBLISHED

JURY WITH ADVERSARIAL LEGAL SYSTEM – IMPLANT EFFECTS ON IMMUNE SYSTEM
=> PLAINTIFFS WIN; SAFETY STUDIES INITIATED; ANGEL WRITES; FDA RECONSIDERS
=> DAUBERT DECISION – JUDGES NOW SCREEN EVIDENCE

CASAC EPA ON ENVIRONMENTAL ISSUES
=> CREDIBLE ASSESSMENT OF SCIENTIFIC EVIDENCE WITH EFFECTS ON POLICY
=> IS BUSH ADMINISTRATION NOW PACKING ADVISORY COMMITTEE?

HEALTH EFFECTS INSTITUTE REANALYSIS OF HSPH SIX CITIES
=> STABILIZE EPA PARTICULATES STANDARDS
=> INDIRECT EFFECT ON AMERICAN TRUCKERS VS EPA

NIEHS-OSTP CONFERENCE AND NAS PANEL ON METHYL MERCURY
FAROE ISLANDS AND SEYCHELLES STUDIES
=> REVISION OF EPA MERCURY STANDARDS

NAS ON ASBESTOS IN WATER
=> REREVISION OF EPA ASBESTOS STANDARD

AMERICAN PHYSICAL SOCIETY PANEL ON EFFECTIVENESS OF PATRIOT MISSILE DURING GULF WAR
=> DELAYED FOR YEARS AND NO EFFECT

OFFICE OF TECHNOLOGY ASSESSMENT AND EFFICACY OF STRATEGIC DEFENSE INITIATIVE
=> NO EFFECT ON POLICY AND DEMISE OF OTA
LOOK AHEAD TO CONCLUDING SESSIONS -- OPEN QUESTIONS ON CREDIBLE ASSESSMENT
* ON WHAT ISSUES MIGHT THIRD PARTY ASSESSMENT HELP BRING BETTER SCIENCE INTO DECISIONMAKING?
* WHAT EXISTING ORGANIZATIONS OR COMBINATIONS OF ORGANIZATIONS MIGHT SERVE AS CREDIBLE
   EVALUATORS OF EVIDENCE IN NATIONAL AND INTERNATIONAL CONTROVERSIES?
     DISCUSS PROBLEMS ASSOCIATED WITH TRW SENSOR TEST CASE
     DISCUSS APS CHOICES RE PATRIOT
     DISCUSS OTA AND SDI
     DISCUSS HEI AND PARTICULATES
     DISCUSS NIEHS NAS AND METHYL MERCURY


3. RESEARCH AFFILIATIONS AND RESEARCH RESULTS: PASSIVE SMOKING
* WHAT ARE THE KEY FINDINGS OF BARNES AND BERO
     PRESENT TABLES 1-4 OHP
* ASSESSMENT OF METHODOLOGY IN LIGHT OF LETTERS TO EDITOR
* INTERPRETATIONS OF CORRELATION BETWEEN AFFILIATION AND RESULTS
     BRIBERY: BUY OR LEASE RESEARCHERS TO GET THE RESULTS YOU WANT
     PREDICTABILITY: FUND RESEACHERS THAT ARE LIKELY TO YIELD RESULTS YOU WANT
* INTERPRETATION OF PEER REVIEW AND QUALITY INDEX NONCORREL
     PEER REVIEW INEFFECTIVE
     QUALITY INDEX MEANINGLESS
     IMPUTATION OF RESULTS WRONG
* WHAT ARE THE PRESCRIPTIVE IMPLICATIONS OF FINDINGS FOR . . .
     RESEARCH FUNDING STRATEGIES
     EDITORS OF SCIENTIFIC JOURNALS
     JOURNALISTS WRITING STORIES
     POLICYMAKERS SEEKING INFORMATION ON RISKS
* ARE THERE SIDE COSTS ASSOCIATED WITH YOUR PRESCRIPTIONS?
COUNTER CASE - FRED KUMMEROW UNIVERSITY OF ILLINOIS
    DIET AND HEART DISEASE – TRANSFATS AND NATURAL SOURCES OF CHLOLESTEROL
    CURRENT DEVELOPMENT – TRANSFAT WARNING -- LABELS BEING PHASED IN
    PAST STANDARD – SUBSTITUTE FOODS WITH TRANSFATS TO CUT BUTTER AND EGGS
RESEARCH 1959 TO 1970
    AUTOPSY – HEART DISEASE AND TRANSFATS IN ARTERIES – SUGGESTIVE
    SMALL EXPERIMENTAL STUDIES ON RATS – DIET AND MATERNAL MILK
    LARGE EXPERIMENTAL STUDIES ON PIGS
FUNDING SOURCES
    Burnsides oil revenue to UI earmarked for Fred => 12 million
    National Institutes of Health => 400 K
    Wallace Foundation [Henry Wallace => Jean Douglas] => 100K
    Gifts from egg hatchery in Gibson City => 25 K and from dairy industry => 25 K

REAL WORLD DEVELOPMENTS 1968-1973
American Heart Association Diet Advisory Committee
 150000 copies warning on transfats…Shortening Institute objected with drawn and replaced
FDA Hearing
 Cooper -- Head of NIH testified that eating eggs caused heart disease
 Kummerow -- testimony contradicted Cooper, emphasized transfats
 No more NIH funding…. Five year NIH renewal gone.
Drew Pearson attack -- 25K contribution from egg industry

POST SCRIPT research in US, Canada, UK, Finland link transfats, arterial plaques, heart disease
POST SCRIPT CBC documentary and Kraft
4. IMPLANT CONTROVERSY - MARCIA ANGELL

BASIC CHRONOLOGY
* Manufacturers introduce with assured safety despite absence of studies establishing safety
      No warnings of health effects or uncertainty over health effects
* Case reports on connective tissue disorder
* A few spectacular tort cases
* FDA Kessler ban 12 years after introduction
      No testing, no evidence of safety, noisy public hearings and NEJM piece.
      Angell: No systematic studies. local complications only.
* Explosion of tort cases
* Cancer scares not borne out by studies.
* Autoimmune problems new focus
      belief in connective tissue diseases including lupus, arthritis, scleroderma, polymyalgia
      harder to study connective tissue disease - symptoms less clear, diagnosis murky
      Clinical case reports but no systematic studies wi incidence with and without implants
* Implant manufacturers lose suits, agree to 4.25B settlement 440,000 register 15,000 opt out
ANGELL'S SIX THEMES
1 place of regulation in American life: political perception and FDA -- why call for evidence so late?
* once cases in and fear rising, demand evidence so as not to appear to cave in to manufacturers
* letter of the law -- no evidence of safety
QUERY: was there regulatory failure in earlier granting of permission to market in the first place?
2 impact of tort law on American life -- why litigation succeeded despite absence of science on harm
* cases beget cases
* US system -- no loser pay costs
QUERY: what tort reforms would make sense?
3 marginal role of scientific evidence: public does not understand how science works
* large scale epidemiological studies => good
* animal studies, laboratory studies, case reports => suspect
* small studies -- size of study and threshold of detection
QUERY: after-the-fact epi studies deemed more conclusive by Angell. How approve?
4 science in courtroom -- pre Daubert
* expert witnesses chosen by sides to give opinions that become legal evidence
* opinions may be based studies or unsystematic experience
* need not provide basis for conclusions except under rare informed cross exam
       => sci -- slow accumulation of information
       => law -- adversarial - controlled argument
"why did you do a study with existing studies already complete?"
"all studies tentative, looking for consistency across different studies"
"so your study and its conclusions are tentative?'
5 profit motive by manufacturers and lawyers -- harass Dr. Gabriel re documents, data, deterring research
6 media coverage - hyped dangers with emphasis on coverup by manufacturers and FDA
GENERAL QUESTIONS AND OBSERVATIONS FOR CLASS DISCUSSION
=> Did the system fail? Manufacturers did not test for safety when reason to believe that may be dangers.
FDA did not regulate. Tort system generated incentives for study. Until systematic studies done, FDA should
have banned.
=> What if studies had come in with dangers? How would story read? Are there examples of inadequate
testing flagged by torts that resemble this case? Asbestos? Tobacco?
=> Is the incentive for manufacturers to test for safety stronger or weaker now than at the time of the
breast implant tort actions? Set up for Daubert.
=> Harassment of Gabriel => set up Harvard study and data controversy

JUDICIAL SYSTEM AND KNOWLEDGE ASSESSMENT
”Science is simply common sense at its best; that is, rigidly accurate in observation and merciless to a fallacy
in logic"
      Thomas Henry Huxley as cited in Reference Manual on Scientific Evidence.

TWO BASIC VIEWS
Adversarial: struggle between defendents and prosecutors/plaintiffs exposes weaknesses and strengths.
     judge responsible for facilitating fair conflict between sides
Technocratic: judge and advisors assess quality of science and appraise credibility of knowledge
DAUBERT v MERRELL DOW PHARM 1993
Supreme Court obligated judges to perform gatekeeping. Must make a preliminary assessment of whether the
reasoning or methodology underlying the testimony is scientifically valid and of whether that reasoning or
methodology properly can be applied to the facts at issue. Replaced general acceptance (Frye) with
"evidentiary reliability"
* Toxic tort - benedictin and birth defects
* Daubert factors
      1. theory or technique can be (and has been) tested -- empirics
      2. subjected to peer review or publication
      3. existence of known or potential error rates and existence and maintenance of standards
      4. widespread acceptance within a relevant scientific community
=> judges warned not to engage in wholesale exclusion -- cross examination, presentation of contrary
evidence, careful instruction are appropriate means by which evidence based on valid principles may be
challenged.
=> limited screening by trial judge will on occasion prevent jury from hearing of authentic breakthroughs.
Rules not designed for cosmic understanding, only to resolve legal disputes

GENERAL ELECTRIC COMPANY V JOINER 1997
Supreme Court ruled appellate courts should let trial court evidentiary rulings stand unless clear abuse of
discretion.
* toxic tort - smoker, family history, PCB exposure
* plaintiff witnesses excluded by trial judge
* infant mice dosed with massive PCB develop different cancer
* plaintiff claimed that pooled reanalysis data -results would have been significant even if individual studies not
* Breyer suggests use of court appointed experts to determine admissibility of complex scientific or technical
evidence
 => Merits and demerits of letting jury handle this?
KUMHO TIRE CO V CARMICHAEL - 1999
Supreme Court extended gatekeeping to "technical" and "other specialized" knowledge
* product liability, tire blew out causing accident
* expert witness visual inspection of tire, defect in manufacture and design
* defendent moved to exclude, trial court concurred - Daubert four tests applied to tech
circuit court of appeals -- "skill or experience based observations" OK -- differ from science.
supreme court - reiterated GE and offered criticisms of expert witness Carlson
=> Why not take it to jury? Would exactly the sort of attack offered be comprehensible to jury?
Note: Kumho Tire allows courts to make judgemnt to accept experience, even though Carlson found
implausible and unscientific in this case.... Daubert four factors not rigid requirement
Note: Kumho Tire asks expert to apply same level of intellectual rigor that characterizes the practice of an
expert in the relevant field.

Seventh Circuit Decisions by Judge Posner
ROSEN V CIBA GEIGY -- barred testimony by Dr. Fozzard on nicotine patch / heart attack
BRAUN V LORILLARD -- barred testimony by Schwartz on asbestos fibers and cancer
=> if witness departs from accepted methodology, must justify wi intellectual rigor using approach consistent
with how colleagues in relevant field or discipline would proceed to establish a proposition
EXAMPLES OF ISSUES THAT MAY REQUIRE JUDGEMENT

BREYER EXAMPLES
Judges asked to understand how statistical analyses work
       Hunt v. Cromartie - plausible alternative interpretation of redistricting as discrimination
       Department of Commerce v US House -- would standards of counting affect plaintiffs?
Right to assisted suicide? can medical technology reduce or eliminate risk of dying in pain?
Is a person so dangerous and mentally ill that threat to public safety requires indefinite noncriminal
confinement?
In criminal cases, scientific validity of DNA sampling, voiceprints, predictions of future dangerousness.
In administrative review safety of drug, risks attending nuclear waste disposal, leakage potential of toxic waste,
risks to wildlife from building a dam
In patent law, understanding and defining originality, utility, nonobvious.
In tort law -- assign responsibility for injury or death to exposures, blame in accidents.
In antitrust -- effects of technology bundling and investments on competition?
BERGER EXAMPLES OF CHALLENGES AND STATUS
Challenges to astrology and necromancy -- no problem
Challenging nonorthodox branch of traditional discipline
      courts reject clinical ecology -- multiple chemical sensitivity (ashford) not validated by testing.
Upshot: No basis for appellate courts overturning district court decision rejecting such witnesses
Challenging reliability of traditional field of expertise -- forensic science
      finger printing
      handwriting analysis
      ballistics
      bitemark analysis
      DNA (OK)
      Upshot: usually accepted because prosecutors rely on this accepted witchcraft
Challenging expert testimony that does not prove causation
      toxic torts and product liability rest explicitly on proof of causation
      some unresolved links often exist
      reliance on animal studies often challenged -- animal evidence rejected in GE
      - Black v Food Lion - fall and fibromyalgia - MD testimony on disease excluded without proof of causation
      - Moore vs Ashland Chemical - reactive airways disorder after cleaning up chemical spill. excluded
      expert medical testimony - no proof -- fumes at higher levels in study, temporal inference no good,
      surmise without proof....not thoroughly tested
-Heller v Shaw Industries -- respiratory problems VOC carpet. excluded testimony of industrial hygenist on rug
as source of VOCs. excluded medical expert. OK to bar hygenist, not OK to kill medical expert. accepted
clinical judgement including temporal evidence.
PROBLEMS
* judges lack scientific training
* many cases fall within area of scientific uncertainty
* must respect juries constitutionally specified role (even if results incorrect)

BRYER SUGGESTIONS
External expert advice
      President - Science Advisor - OSTP
      Congress - NAS
      Judiciary - NAS Program in Science, Technology and Law; advisors
Pretrial conferences to narrow scientific issues in dispute
Pretrial hearings where potential experts subject to examination by court
Appointment of specially trained clerks or scientific special masters
Appointment of independent experts (going beyond expertise offered by parties to dispute)
      Judge Stearns - Harvard Medical School Professor -- genetic engineering patent case
      Judge Pointer - neutral science panel of four scientists from different disciplines
            -- to prepare testimony on scientific basis of claims in implant case
            -- video taped ("consistent" "savings of time and expense")
AAAS-ABA National Conference of Lawyers and Scientists -- slate of candidates from science and
professional organizations -- where court has determined that traditional means of clarifying issues under the
adversarial system are unlikely to yield the information that is necessary for a principled resolution of disputed
issue

=> Are these procedures "neutral"
=> What are effects of Daubert and successor cases likely to be?
=> Link back to Angell and implants controversy
=> Link back to EU triazolam and GMO administrative rulings
5. HSPH AND ACS PARTICULATES STUDIES: UNCERTAINTY AND THIRD PARTY KNOWLEDGE APPRAISAL

CASE REPORTS
* CAUSALITY DIFFICULT OR IMPOSSIBLE TO INFER

EPIDEMIOLOGICAL STUDIES
* FALSE POSITIVES FROM
     INADEQUATE CONTROLS
     CONFOUNDERS
     SELECTION EFFECTS
* FALSE NEGATIVE OR INSIGNIFICANT RESULTS FROM
     IMPOSITION OF CONTROLS ON PATHS OF CAUSATION
     USE OF CORRELATED NOMINALLY INDEPENDENT VARIABLES IN MODEL
* TYRANNY OF STATISTICAL SIGNIFICANCE

ANALYSIS OF SECULAR TRENDS IN DEFECTS BEFORE AND AFTER INTRODUCTION/WITHDRAWAL
* WHAT ELSE IS GOING ON APART FROM CHANGE IN TREATMENT?

ANIMAL STUDIES
* GENERALIZABILITY -- SPECIES SPECIFICITY OF EFFECTS

IN VITRO STUDIES
* GENERALIZABILITY -- ARTIFICIALITY OF LABORATORY IN VITRO EXPOSURES

CHEMICAL STRUCTURE COMPARISONS OF SUBSTANCES WITH KNOWN AND UNKNOWN RISKS
* HOW DEFINE DEGREE OF ISOMORPHISM
* CONTROVERSY OVER PROJECTED SIMILARITY IN EFFECTS

CONTROLLED HUMAN STUDIES
* HARD TO DO ONCE HARM IS SUSPECTED
FIGHTS OVER EPIDEMIOLOGICAL EVIDENCE
I=> INSERT SLIDES FROM DOCKERY
HARVARD SCHOOL OF PUBLIC HEALTH SIX CITIES STUDY
      DOCKERY ET AL NEJM 1993
      8111 ADULTS IN SIX CITYES 14-16 YEARS
      LEAST-MOST POLLUTED FINE PARTICLES AND SULFATES => 26% INCREASE IN MORTALITY
      FINE PARTICLES => CARDIO PULMONARY DISEASE
AMERICAN CANCER SOCIETY STUDY
      POPE ET AL AM
      REV RESPIRATORY AND CRITICAL CARE MEDICINE 1995
      552138 ADULTS IN 154 CITES 1982-89; 50 CITIES WI FINE PARTICLE DATA AVAILABLE
      LEAST-MOST POLLUTED FINE PARTICLES => 17% SULFATES => 15%
EPA NATIONAL AMBIENT AIR QUALITY STANDARDS 1997

INDUSTRY CHARGES AGAINST HSPH AND ACS
     BIAS BY RESEARCHERS
          => "LEAD ZEALOTS"
          => MOTIVE TO SECURE FUNDING FOR RESEARCH
     FLAWS IN RESEARCH DESIGN
          => CODING ERRORS
          => HANDLING OF DROP OUTS FROM COHERT
          => INADEQUATE CONTROLS FOR SEDENTARY LIFESTYLE AND SMOKE INHALATION
          => MEASURES FOR SHORT PERIOD - INVALIDATE LONG TERM CONCLUSIONS
          => ADJUSTED MORTALITY RATE RATIOS
     FEDERAL FUNDING
          => "SECRECY" BY NOT RELEASING DATA
          => "COVERUP" OF FLAWED DESIGN AND CODING ERRORS
HEALTH EFFECTS INSTITUTE: JULY 2000 REANALYSIS OF HSPH AND ACS STUDIES
* WHO FUNDED THE REAPPRAISAL? WHO PARTICIPATED?
* WHAT METHODS WERE USED?
          => REPLICATION AND VALIDATION - BOTH FINE
          => SENSITIVITY ANALYSIS
               PHYSICAL ACTIVITY, LUNG FUNCTION MARITAL STATUS - NO EFFECT
               SENSITIVITY OF SUBGROUPS - LOWER EDUCATION MORE SENSITIVE
               LINEARITY AND NONLINEARITY OF RESPONSES ACS - MIXED RESULTS
               RUN WITH TIME DEPENDENT VARIATION IN POLLUTANTS - ASSOCIATION DROPS
               ECOLOGIC COVARIANTS - IE PHYSICIANS PER CAPITA
* WHAT ARE THE EFFECTS OF THIS EXERCISE? IN RETROSPECT, WHAT WERE PROS AND CONS OF APPROACH?
* HOW WIDELY COULD INTERMEDIARIES BE USED TO ASSESS KNOWLEDGE?
     => CASAC, HEI, HEINZ CENTER, NIH CONSENSUS DEVELOPMENT PROGRAM, NRC . . .
          AND THE LATE DEARLY DEPARTED OTA

SIDEBAR AMERICAN TRUCKING ASSOCIATION VS EPA
* FIGHT OVER EPA PARTICULATE AND OZONE STANDARD
* KEY ISSUES RAISED IN CASE
      DELEGATION OF POWERS FROM CONGRESS TO EPA UNCONSTITUTIONAL
      STANDARDS SET ARBITRARY AND LACKING IN DUE PROCESS
           INADEQUACY OF EVIDENCE ON HEALTH EFFECTS ON PARTICULATES AND OZONE
           NO CLEAR THRESHOLD RE PARTICULATES OR OZONE, HENCE CUTOFFS ARBITRARY
* CIRCUIT COURT OF APPEALS DECISION THREATENED MASSIVE CHANGES IN REGULATION
* SUPREME COURT REVERSED CIRCUIT COURT OF APPEALS DECISION
6. FAROE AND SEYCHELLES STUDIES ON METHYL MERCURY: AD HOC CONFERENCE=>NRC PANEL

POLICY ISSUE: EPA REFERENCE DOSE FOR METHYL MERCURY
    POWER PLANTS 45%
    MUNICIPAL INCINERATORS 25%
    OTHER SOURCES 30%

LARGE N PEER REVIEWED STUDIES
MATERNAL LOW DOSE EXPOSURE => CHILDREN NEUROBEHAVIORAL EFFECT
FAROE ISLANDS STUDY          IQ LOSS AND DEVELOPMENTAL DELAYS
SEYCHELLE ISLANDS            IQ GAINS AND DEVELOPMENTAL BENEFITS

METHODS OF KNOWLEDGE ASSESSMENT IN CONTROVERSY:
AD HOC CONFERENCE NIEHS OSTP EPA => SEEK ADJACENT EXPERTISE => NOT SURE, FUND MORE
NRC STUDY REQUESTED BY CONGRESS => PANELS => ACCEPT FAROES AND EPA REF DOSE
                                         WORKSHOP
                                     Organized by
               Committee on Environmental and Natural Resources (CENR)
                   Office of Science and Technology Policy (OSTP)
Scientific Issues Relevant to Assessment of Health Effects from Exposure to Methylmercury
                                         November 18-20, 1998




Report
Individual Panel Reports                                            Appendix   I
     -Exposure Panel                                                Appendix   I-A
     -Neurobehavioral Endpoints Panel                               Appendix   I-B
     -Confounders & Variables Panel                                 Appendix   I-C
     -Design/Statistics Panel                                       Appendix   I-D
     -Experimental Panel                                            Appendix   I-E
Reports from Study Teams                                            Appendix   II
     -Iraq & Seychelles Studies                                     Appendix   II-A
     -Faroe Islands Studies                                         Appendix   II-B
     -Amazon Studies                                                Appendix   II-C
Comments from Study Groups on Workshop Report and Panel             Appendix III
Reports
     -Iraq & Seychelles                                             Appendix   III-A
     -Faroe Islands                                                 Appendix   III-B
References                                                          Appendix   IV
Workshop Program                                                    Appendix   V
Attendees List                                                      Appendix   VI
PANEL FINDINGS AND RECOMMENDATIONS

1 Methylmercury is a developmental neurotoxin, but effects at low doses encountered by eating fish
are difficult to evaluate.
2 All the studies reviewed were considered of high scientific quality and the panel recognized that
each of the investigations had overcome significant obstacles to produce important scientific
information. The panel also stated that continued funding of the studies in the Seychelles, Faroes,
and Amazon is necessary for the full potential of those studies to be realized. This is particularly
the case for the Faroes and Seychelles studies, which have assessed and are currently assessing the
potential developmental neurotoxic effects of methylmercury in fish-eating populations. The
developmental studies would benefit by evaluation of common endpoints using similar analytical
methods. It is important to note that the Amazon study did not assess developmental endpoints but
assessed effects in adults.

3 Results from the Faroes and Seychelles studies are credible and
provide valuable insights into the potential health effects of
methylmercury.
4 Some differences are clearly present in results from the Faroes,
Seychelles, and Amazon, but the panel was not able to clearly
identify the sources of these differences. Among possible sources
are the different effects of episodic versus continuous exposure,
ethnic differences in methylmercury responses, lack of common
endpoints in the Faroes and Seychelles studies, and several other
confounders or modifying factors such as those found in diet and
lifestyle, as well as in chemicals present in seafood, which is
the source of methylmercury to these populations. The other
chemical constituents of seafood that may be explanatory include
those that may be beneficial to fetal neurodevelopment (i.e.,
omega-3 fatty acids) and those that may be harmful to fetal
neurodevelopment (e.g., PCBs).
5 These studies have provided valuable new information on the potential health effects of
methylmercury but significant uncertainties remain because of issues related to exposure,
neurobehavioral endpoints, confounders and statistics, and design.
EPA's Methylmercury Guideline Is Scientifically Justifiable For Protecting Most Americans,
But Some May Be at Risk                                     NRC Press Release July 11, 200
….The overall weight of the evidence from this comprehensive review led the committee to conclude that
EPA's reference dose is scientifically justifiable for protecting the health of the vast majority of Americans.
….When the agency first developed its guideline five years ago, EPA judged data from a 1971 Iraqi poisoning
incident to be the most relevant. To provide EPA with more appropriate data in formulating its reference dose,
the committee analyzed population studies in the Faroe Islands, Seychelles Islands, and New Zealand. It
concluded that the Faroe Islands analysis should be used by EPA as the critical study for deriving the
reference dose, the report says.
NRC Commission on Life Sciences Board on Environmental Studies and Toxicology
Committee on the Toxicological Effects of Methylmercury
Robert A. Goyer, M.D. (chair) Pathology, University of Western Ontario
H. Vasken Aposhian, Ph.D. Molecular and Cell Biology and Pharmacology University of Arizona
Lenore Arab, Ph.D. Epidemiology and Nutrition, University of North Carolina
David C. Bellinger, Ph.D. Neurology Harvard Medical School
Thomas M. Burbacher, Ph.D. Environmental Health, University of Washington
Thomas A. Burke, Ph.D. Health Policy and Management and Risk Science and Public Policy Johns Hopkins
Joseph L. Jacobson, Ph.D., J.D. Obstetrics/Gynecology and Psychology Wayne State
Lynda P. Knobeloch, Ph.D. Toxicologist Wisconsin Bureau of Environmental Health Madison
Louise M. Ryan, Ph.D. Biostatistics Dana-Farber Cancer Institute and Harvard School of Public Health
Alan H. Stern, Dr.P.H. Risk Analysis New Jersey Department of Environmental Protection
Carol A. Maczka, Ph.D. NRC Study Director
SOME DIFFERENCES IN MAJOR EPI STUDIES

                                 FAROE                  SEYCHELLES              NEW ZEALAND
MERCURY EXPOSURE MEASURE         UMBILICAL CORD BLOOD   MATERNAL HAIR           MATERNAL HAIR
NEUROBEHAVIORAL ENDPOINTS        BROADER                NARROWER                BROADER
CONDITIONS OF TESTING            IN CAPITAL             IN FIELD                IN FIELD
PRIMARY SOURCES                  FISH AND WHALE         FISH                    FISH
DIETARY INTERACTION              PCBS, OMEGA            OMEGA                   OMEGA
FAILURES TO CONTROL              MATERNAL ALCOHOL       ---                     ---
SELECTION BIASES                 ---                    ELIMIN8 BIRTH DEFECTS   ---
DIVERSITY OF POPULATION          VERY LOW               LOW                     HIGH

FINDING - EFFECTS OF LOW DOSE    BAD                    GOOD                    BAD

CENROSTP CONFERENCE => SUSTAINED ANALYSIS UNDERSCORED UNCERTAINTY
NRC                 => TWO STUDIES AGAINST ONE PLUS ANIMAL STUDIES – TILT TO FAROE
                    => USE LOWEST DOSE RESPONSE FOR REFERENCE DOSE – BOSTON NAMING TEST

QUESTIONS ON PROCESS AND RESULTS –
WAS STATEMENT ON UNCERTAINTY A PREDICTABLE PRODUCT OF CONFERENCE?
WAS ACCEPTANCE OF EPA REFERENCE DOSE WITH CAUTION A PREDICTABLE PRODUCT OF NRC PANEL?
WITH BENEFIT OF HINDSIGHT, HOW STRUCTURE KNOWLEDGE ASSESSMENT TO YIELD DIFFERENT RESULTS?
     AT WHAT LOSSES IN TERMS OF CREDIBILITY? WITH WHAT DELAYS?
WITH BENEFIT OF HINDSIGHT, HOW “IMPROVE” KNOWLEDGE ASSESSMENT PROCESS?
6. TRIAZOLAM - ABRAHAM AND SHEPPARD
UK + US DIFFER ON CERTIFICATION TRIAZOLAM "HALCION"™ - SEDATIVE WITH SHORT HALF LIFE
70'S 80'S UK AND US CERTIFIED EFFICACY AND SAFETY AT .5 MG DOSE
90'S US CERTIFIED --   SAFE AND EFFICACIOUS AT .25 MG DOSE
90'S UK WITHDREW -- NOT SAFE .5 MG           NOT EFFICIOUS AT .25 MG

EVIDENCE ON PSYCHOTIC SIDE-EFFECTS
* MURDER CASE WITH HALCION DEFENSE
* SPONTANEOUS REPORTS / MILLION PRESCRIPTIONS TABLE 1
TABLE 1 FROM ABRAHAM AND SHEPPARD
UK Spontaneous Reports of Psychiatric Reactions by Year for Specified
Benzodiazepines per Million Prescriptions
Year          1979 1980 1981 1982 1983 1984 1985 1986
Triazolam     114.0 44.0     14.0    43.0  20.0    5.5    1.8     2.5
Flurazepam      0.9   0.9     0.4      1.3  0.5    0.0    2.2     0.0
Nitrazepam      0.1   0.9     0.5      0.0  0.3    0.0    0.8     0.4
Temazepam       4.1   4.5     3.7      1.9  2.2    1.7    1.3     0.4
Lorazepam       2.4   3.3     1.4      2.0  4.3    1.9    6.7     1.9
Diazepam        0.4   0.7     0.1      0.1  0.2    0.4    0.4     1.0
* INITIAL UPJOHN TRIALS - ERRORS
      PLACEBO 1 EPISODE REPORTED AS 2 EPISODES
      DRUG    7 EPISODES REPORTED AS 1 EPISODE
* NEW TRIALS PLACEBO .5% OTHER DRUG 1.9 % TRIAZOLAM 9.9 %
UK EU COMM ON PROPRIETARY MEDICAL PROCEDURE BANNED
US FDA ADVISORY COMMITTEE 7/8 VOTED SAFE & EFFECTIVE AT .25 MG
Differences                 UK                                                 US
Prior trust                 Trusted Sources                                    never trusted sources
                            =>disillusioned                                    => not disillusioned
                            no benefit of doubt in assessment of data          business as usual in assessment of data and
                            and evaluation of risks                            evaluation of risks
Integration of data         Integration of initial studies, clinical trials,   Consideration of each source of information in
                            postmarket data                                    isolation
Disciplinary                MD/epidemiologist/pub health                       Clinician v Epidemiologist
background of
managers of process
Advisory Committee          Generalists                                 Specialists in pharmacology
Org structure               licensing & postlicensing by different orgs licensing & postlicensing by same org and people
Conflicts of interest for   None                                        9/11 grants from Upjohn
advisory committee                                                      1/11 stockholder
                                                                        7/8 voters - FDA waivers
Ideology                    traditional role of state as protector      Neoliberalism and deregulation
                            -- but deregulation within UK as well       standards dictated by parties
                                                                        effect of AIDS patient demand for easier approval

=> ABRAHAM AND SHEPPARD DO NOT EVALUATE OUTCOMES.
    BOTH US/UK DRIVEN BY ENDURING MATERIAL INTERESTS AND DISCIPLINARY OUTLOOKS
=> WOULD YOU EVALUATE OUTCOMES? WHICH PREFER AND WHY?
=> WHAT FACTORS IMPORTANT? WHAT REFORMS WOULD YOU RECOMMEND?
KNOWLEDGE ASSESSMENT IN SECURITY AFFAIRS -- APPRAISING CAPABILITIES AND THREATS

TECHNICAL ASSESSMENTS OF EFFICACY OF OWN WEAPONS SYSTEMS AND STRATEGIES
    Performance of new weapons systems under ideal conditions
    Performance of new weapons systems under operational conditions with a strategizing adversary
Historical Examples: Battleship vs Aircraft Carriers
Recent Examples: C3I and counterforce, Patriot I, BMD
Error Underestimate => prolonged undue reliance on old system
Error Overestimate => premature undue reliance on new system

ASSESSING THREATS POSED BY OTHERS
    Technical assessment of existence and efficacy of adversarial weapons systems
    Political assessment of adversarial capabilities, intentions and strategy
Historical Examples: Soviet A&H Bomb, Missile & Bomber Gap, Counterforce, NATO Central Front
Recent Examples: Al Quada; Iraq, Iran, North Korea WMD
Error Underestimate => greater vulnerability to surprise attack and deterrence failure
Error Overestimate => unnecessary spiraling via arms expenditures, preemptive war, and preventive war

QUESTIONS ON KNOWLEDGE ASSESSMENT IN SECURITY AFFAIRS
   What are some key sources of uncertainty associated with these tasks?
   What can be known in times of peace? What can be known only in time of war?
   In absence of war, what methods can one use to assess knowledge on efficacy and threats?
   How vulnerable are such methods to institutional and ideological distortion?
   What are some approaches to handling intrinsic uncertainty PLUS need for secrecy?
   What are the costs of false negatives and false positives?
   How can we improve the terms of tradeoff across false negatives and false positives?
ASSESSING THE EFFICACY OF WEAPONS SYSTEMS AND STRATEGIES
* Counterforce Escalation Dominance Strategy with Counterforce and C3I
      Blair OTA C3I technical critique vs USAF 1980s => not tested
* Effects of North Atlantic Naval Strategy in Conventional War
      Posen vs Lehman technical and strategic controversy 1980s => not tested
* US-Iraq Distribution of Casualties and Duration of Campaign
      Posen vs Left re campaign analysis 1991 => resolved by gulf war
* Ballistic Missile Defense
      80s        Strategic Defense Initiative => Office of Technology Assessment
      91-96      Patriot - Bush/Raytheon/Army vs Postol/Israelis => partially resolved by APS panel
      98-00      BMDO/Clinton/Bush/ vs Postol et al
      98-03      DoD => TRW test=>GAO=> Lincoln Laboratories=> MIT
C3I AND COUNTERFORCE ESCALATION DOMINANCE
BRUCE BLAIR, STRATEGIC COMMAND AND CONTROL AND NUCLEAR STRATEGY
     BLAIR: MINUTEMAN LAUNCH CONTROL => YALE SOM, OTA, BROOKINGS
     CLASSIFIED STUDY FOR OFFICE OF TECHNOLOGY ASSESSMENT
     UNCLASSIFIED STUDY FOR BROOKINGS INSTITUTION

QUESTION: IS US CAPABLE OF IMPLEMENTING COUNTERFORCE ESCALATION DOMINANCE STRATEGY?
TASKS: MUST BE ABLE TO PERFORM AND COORDINATE FOLLOWING --
   INTELLIGENCE GATHERING ON DAMAGE TO OWN FORCES AND TO SOVIET FORCES
   RETARGET OWN FORCES AGAINST UNDAMAGED SOVIET FORCES
   WITHHOLD RESERVE TO DETER COUNTERVALUE ATTACK
   COMMUNICATE WITH ADVERSARY DURING WAR TO DISCOURAGE COUNTERVALUE ATTACK
TECHNICAL PROBLEMS: VULNERABILITY OF C3I TO BLAST AND EMP
ORGANIZATIONAL PROBLEMS: SOPS WITHOUT REHEARSAL => FUNCTIONALITY DUBIOUS

INTERESTS POTENTIALLY AFFECTED BY STUDY
   REDUCE US CAPACITY TO COERCE – EXISTENTIAL MUTUAL ASSURED DESTRUCTION
   REDUCE RELIANCE ON US CAPACITY FOR COERCION THAT DOES NOT EXIST
   WOULD ELIMINATE DRIVER FOR PROCURING COUNTERFORCE – ACCURATE LAND BASED ICBMS
   WOULD STRENGTHEN HAND OF ARMY AND WEAKEN HAND OF AIRFORCE FOR FUNDING

KNOWLEDGE APPRAISAL FIGHT
OTA STUDY CLASSIFIED AT EXTREMELY HIGH LEVEL – SEVERELY LIMIT DISCUSSION
UNDERCUT CLAIMS BY REFERENCE TO UNDISCLOSED CLASSIFIED INFORMATION
CONGRESSIONAL COMMITTEE – CLOSED DOOR HEARING, REVOCATION OF CLEARANCE
WALL STREET JOURNAL ARTICLE – RESTORATION OF CLEARANCES AND BLAIR TESTIFIES
CODA1 – NO CHANGE IN US STRATEGY; ONLY EFFECT HEAVIER INVESTMENTS IN C3I
CODA2 – US AIR FORCE ACADEMY CURRICULUM
1991 GULF WAR - PATRIOT I VS SCUDS FIRED AGAINST ISRAEL AND SAUDI ARABIA
SECURITY INTERESTS DURING WAR
      PROTECTING POINT AND AREA TARGETS FROM SCUDS
      KEEPING ISRAEL OUT OF CONFLICT AND REASSURING SAUDI ARABIA
      DISCOURAGING SCUD ATTACKS BY IRAQ
PATRIOT I – A MODIFIED ANTIAIRCRAFT MISSILE PLACED IN SERVICE ON ACCELERATED BASIS
INITIAL CLAIMS – 99 PERCENT EFFECTIVE -- ARMY AND PRESIDENT BUSH
INTERESTS AFTER THE WAR
      REASSURING JAPAN AND SOUTH KOREA RE NORTH KOREA BALLISTIC MISSILE THREAT
      SECURING POLITICAL BASE FOR STRATEGIC BALLISTIC MISSILE DEFENSE
      SALES OF PATRIOT AND REDUCING US PROCUREMENT COSTS
COUNTER CLAIMS – INEFFECTIVE OR WORSE – ISRAELI ANALYSTS AND POSTOL
      GROUND DAMAGE INFLICTED BY INTERCEPTED WARHEADS (ISRAEL)
      PUBLIC VIDEO FOOTAGE (POSTOL)
      RADAR MISSED WARHEAD (TUMBLING BOOSTER AND DEBRIS)
      PROXIMITY FUSE MISTIMED DETONATION
KNOWLEDGE APPRAISAL CONTROVERSY
      APPEAL TO CLASSIFICATION – RELEVANT SECRET INFORMATION
      PEER REVIEW -- INTERNATIONAL SECURITY PUBLISHED SERIES OF ESSAYS
      IMPUGN MOTIVES (ARROW) AND PERSONALITY (POSTOL) THREATEN SECURITY CLEARANCES
AMERICAN PHYSICAL SOCIETY PANEL PUBLIC AFFAIRS SECTION
      APS -- PROS AND CONS ON TAKING ON ASSIGNMENT
      LONG DELAYS ON ISSUANCE OF APS REPORT
CODA: CLINTON BALLISTIC MISSILE DEFENSE – PANETTA FIGHT, AGENTS VISIT POSTOL SUMMER 2000
CODA: TRW TEST, GAO CRITIQUE, LINCOLN LABS BACK TRW, MIT BACKS LINCOLN, MIT BACKS INQUIRY
CODA: 2003 NEW PATRIOT EFFECTIVE – IMPROVED DESIGN EASIER TARGET, BUT FRIENDLY FIRE
=> GOOD EXAMPLES OF DECONSTRUCTION AND RECONSTRUCTION -- . WHAT SHOULD BE DONE?
ASSESSING THREATS – ADVERSARIAL CAPABILITIES AND INTENTIONS
* Soviet Nuclear Programs 1940s early 50s-- US underestimate Soviet time to develop A and H bombs
      Estimate of time to develop based on estimates of diffusion of information and priority assigned
      Reality check -- Testing of devices
* Soviet bombers 1950s -- US overestimated Soviet capability
      Technical intelligence contradicted earlier estimates
* Soviet ICBMs late 1950s and early 1960s -- US overestimated Soviet capability
       Interpretation of less-than-expected investment in bombers for missile capability
      Comparison of US planned forces with Soviet maximum potential forces
      Technical intelligence contradicted earlier estimates
* Soviet-US ICBM mid 1970s-1980s -- estimates of Soviet capacity to "fight and win a nuclear war"
      Modernization and counterforce capability
      Civil defense efforts
      Deployments of ballistic missile submarines
      1976 Competitive Assessment: Team A Howard Stoertz and Team B Richard Pipes
* Soviet-US NATO conventional forces early 1980s -- US overestimated Soviet capability?
      Unresolved -- Mearsheimer -- campaign analysis of NATO central front
* India and Pakistan nuclear programs 2000 -- US underestimated Indian and Pakistani programs
      Estimate of time to develop based on estimates of diffusion of information and priority assigned
      Reality check -- Testing of devices
* Transnational Conventional Terror 2001 -- US underestimated capabilities of Bin Laden organization?
      Reality check -- September 11 WTC and Pentagon
* North Korean and Iraq missile and nuclear programs -- over or underestimate?
      Team A Clinton estimates Team B Donald Rumsfeld
* Iraq and Weapons of Mass Destruction today -- capabilities and conditions for use?
      CIA - Tenet letter to intelligence committee
      DIA and DoD and NSC – Stovepipes vs organizational processes and bureaucratic politics
WHAT ARE SOURCES OF FALSE NEGATIVES, FALSE POSITIVES, AND ERRORS?

Organizational and political problems in assessing threats
Friedman -- "cynical view of scholars" -- intelligence as justification for policies that serve parochial interests
    what biases and heuristics associated with organizational and economic interests? (crass)
    what biases and heuristics are associated with organizational beliefs? (genuine)
    can secrecy AND democratic decisionmaking AND good knowledge appraisal be reconciled?

Service Branches - Air Force, Army, Navy Intelligence
     => common sense / world view
     => parochial organizational interests with chain of command
DIA
     => nominally above services but officers seconded from branches and leadership favors uncontroversial
     => new conventional wisdom, Rumsfeld tried to use DIA to offset CIA, then turned to stovepiping
CIA
     => "professionalism"
     => errors . . . but how much distortion of analysis to satisfy preconceived notions of clientele?

Methods of Structuring Analysis
   Formal NIE under DCI
   Formal Competitive Assessment - Team A and Team B
   Informal Bureacratic Conflict - Kissinger and Wolfowitz extraction and interpretation of raw intelligence

PROBLEMS: FOOLING SELF? FOOLING OTHERS? BLOWBACk?
PRECAUTIONARY PRINCIPLE REDUX – UNCERTAINTY AND KNOWLEDGE ASSESSMENT

“Absolute proof cannot be a precondition for action."
"The message is that there are no knowns. There are things
we know that we know. There are known unknowns, that is to
say there are things we now know we don't know. But there
are also unknown unknowns - things we do not know we don't
know.
"So when we do the best we can and we pull all this
information together, and we then say 'well, that's
basically what we see as the situation', that is really only
the known knowns and the known unknowns. And each year we
discover a few more of those unknown unknowns.”
“There is another way to phrase that, and that is that the
absence of evidence is not evidence of absence.”

Define priors on expected utility of allowing and not allowing.
Define relevant sources of uncertainty.
Define, acquire, and assess relevant information.
Define posteriors on expected utility of allowing and not allowing.
                                  Precautionary Principle               Proof before Action
Security Threats                  US Bush and Rumsfeld                  France Chirac
WMD example                       Preventive war                        Inspect and Embargo
                                  Prior – rogue regimes pose            Prior – regime type not good
                                  major security threats                predictor of security threat
Environmental, and                Denmark, CU, Ashford                  US Bush Sapolsky HIV activists
product/drug safety               Restrict substances and               Allow substances and products
                                  products unless prove no harm unless clear evidence of harm
                                  Prior – strong expectations of        Prior – normal expectation of
                                  risks in selected areas               safety and usefulness
Export Controls                   US OFAC                               EU Allies and Japan
                                  Restrict trade using lower            Allow trade unless clear
                                  threshold of risk                     evidence of risk
                                  Prior – potential risks are high      Prior – potential risks are low
Limits on Research                Rees “Our Final Hour”                 MIT View
                                  Limit research in risky areas         Allow research in all areas
                                  Prior – can identify some             Prior – knowledge is basically
                                  research of benefit to terrorists good, cannot sort in advance
Can we do better than shifting evidentiary thresholds to tilt tradeoffs across Type I or Type II error?
Are priors and presumptions subject to modification? What evidence is deemed to be relevant?
How should relevance of evidence be defined and how should relevant evidence be assessed?
Can we improve the “power curve” through better knowledge assessment methods?

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:13
posted:8/7/2011
language:English
pages:39