APPRAISING KNOWLEDGE UNDER UNCERTAINTY
0. OVERVIEW OF UNIT
PERSPECTIVES ON KNOWLEDGE APPRAISAL -- REVIEW VISCUSI MORGAN; ADD JASANOFF
METHODS OF APPRAISAL:-- FULL SPECTRUM FROM IN HOUSE TO ADVERSARIAL TO THIRD PARTY
HEALTH & ENVIRONMENT CASES
PASSIVE SMOKING - FUNDING AND RESEARCH=> PEER REVIEW AND PUBLICATION
TRANSFATS - FUNDING AND RESEARCH => DISCOUNTING INFO ON BASIS OF FUNDING LEC
IMPLANTS - JUDICIAL APPRAISAL OF RISKS => DAUBERT ON SCIENCE
PARTICULATES AND HEALTH EFFECTS UNCERTAINTY => THIRD PARTY ASSESS HEI
METHYL MERCURY AND HEALTH EFFECTS UNCERTAINTY => THIRD PARTY ASSESS NRC LEC
ASSESSING PAST PERFORMANCE OF PATRIOT => THIRD PARTY AMERICAN PHYSICAL SOCIETY
PROJECTING CAPABILITIES OF NATIONAL MISSILE DEFENSE => CLASSIFICATION AND GAG
ASSESSING THREAT BOMBER&MISSILE GAP,COUNTERFORCE=>BURPOL&COMPET ASSESS
ASSESSING UNCONVENTIONAL THREATS: IRAQ AND WMD =>ALL OF ABOVE . . .
1. BASIC PERSPECTIVES ON ASSESSMENT
POSITIVIST DECISION ANALYTIC
=> WHAT DOES EACH OF PERSPECTIVES TREAT AS OBJECTIVE AND AS SUBJECTIVE?
=> WHAT ARE MOST SIGNIFICANT AREAS OF DISAGREEMENT?
=> DO YOU ENTER THIS UNIT WITH A PREFERENCE FOR ANY OF THE THREE?
=> CAN THESE VIEWS SIMPLY BE COMBINED, OR DO YOU HAVE TO MAKE A CHOICE?
A. POSITIVIST ECONOMIC – REVIEW OF VISCUSI ET AL
SECURE OBJECTIVE KNOWLEDGE ON . . .
RISKS AND UNCERTAINTY == SEE TABLES 19.3
VALUE OF LIFE == WILLINGNESS TO PAY SURVEY AND LABOR DATA
COSTS OF MITIGATION OF RISKS == TABLE 20.4
APPLY UTILITARIAN EXPECTED VALUE DECISION CRITERION TO EVALUATE ALTERNATIVES
WHERE ACTUAL OUTCOMES DIFFER FROM IDEALIZED MODEL, MOVE REALITY TOWARD MODEL
CORRECT FOR ERRORS IN RISK PERCEPTION? MASS OR EXPERT
TAKE ACCOUNT OF COSTS
USE MARGINAL RATHER THAN AVERAGE COSTS
SET STANDARDS TO UTILIZE FULL INFORMATION ON VARIATION IN COSTS AND RISKS
B. POSITIVIST ASSESSMENT AND DECISION SCIENCE – REVIEW OF GRANGER MORGAN
SECURE OBJECTIVE KNOWLEDGE ON RISK ASSESSMENT AND UNCERTAINTY . . .
EXPOSURES: MODELS AND FAILURE MODES
EFFECTS: DOSE RESPONSE
WHERE CONFRONT UNCERTAINTY....
USE BEST AVAILABLE INFORMATION
USE SUBJECTIVE ASSESSMENTS WITH SENSITIVITY ANALYSIS
WHAT ARE SUBJECTIVE POLICY ISSUES?
CHOICE OF UTILITY VS RIGHTS VS TECHNOLOGY BASED APPROACHES TO VALUATION
EXPLICIT JUDGEMENT ON CONSERVATIVE OR LIBERAL STANDARDS FOR RISK APPRAISAL
CHOOSE STANDARDS FOR VALUING OUTCOMES -- BUT BE CONSISTENT
WHERE ACTUAL OUTCOMES DIFFER FROM OBJECTIVE ASSESSMENTS . . .
DO NOT THROW OUT MASS VIEWS AUTOMATICALLY FOLK WISDOM POSSIBLE:
DEVIATIONS FROM SIMPLE EXPECTED VALUE DREAD, LEVEL OF UNDERSTANDING, EQUITY....
USE RISK COMMUNICATION TO PROVIDE FOUNDATION FOR INFORMED CHOICE
C. CONSTRUCTIVIST - JASANOFF
Knowledge claims are deconstructed during the rulemaking process,
exposing areas of weakness or uncertainty and threatening the cognitive
authority of science. At the same time, the legitimacy of the final regulatory
decision depends on the regulator's ability to reconstruct a plausible scientific
rationale for the proposed action. The processes of deconstructing and
reconstructing knowledge claims give rise to competition among scientists,
public officials, and political interest groups all of whom have a stake in
determining how policy-relevant science should be interpreted and by whom.
All of the actors use boundary defining language in order to distinguish
between science and policy, and to allocate the right to interpret science in
ways that further their own interests.
PHASES AND PROCESSES
* Knowledge claims are deconstructed during the rulemaking process,
exposing areas of weakness or uncertainty and threatening the cognitive
authority of science.
* At the same time, the legitimacy of the final regulatory decision
depends on the regulator's ability to reconstruct a plausible scientific
rationale for the proposed action.
STAKES AND METHODS
* The processes of deconstructing and reconstructing knowledge claims give
rise to competition among scientists, public officials, and political interest
groups all of whom have a stake in determining how policy-relevant science
should be interpreted and by whom.
* All of the actors use boundary defining language in order to distinguish
between science and policy, and to allocate the right to interpret science in
ways that further their own interests.
(1) MODUS OPERANDI
OBSERVE OUTCOMES AND DEBATES OVER RISK ASSESSMENT AND RISK MANAGEMENT
SEEK TO UNDERSTAND CHANGES IN DISCOURSE IN TERMS OF . . .
* MATERIAL INTERESTS OF ACTORS: RESOURCES AND STRUGGLE TO ACQUIRE THEM
* IDENTITY AND SELF DEFINITION BY ACTORS: HOW SCIENTISTS SHOULD BEHAVE
* PRIOR USES OF CONCEPTS: CONSTRAINTS OF PATH DEPENDENCY
(2) EXAMPLES OF CONTESTED BOUNDARIES BETWEEN SCIENCE AND POLICY
a. SCIENCE AS DISTINCT FROM TRANSCIENCE:
SCIENTISTS TRY TO DRAW LINE TO PRESERVE LEGITIMACY BY CUTTING THEMSELVES OFF
FROM AREAS OF UNCERTAINTY AND CONFLICT....
BUT CAN ONLY DO SO BY GIVING UP CONTROL OVER BLURRY AREAS
b. RISK ASSESSMENT AS DISTINCT FROM RISK MANAGEMENT
EXAMPLES: BENZENE AND OSHA, MCCRAY AND NRC VS GRAMM AND OMB VS RUCKELSHAUS
CANNOT DIFFERENTIATE CLEANLY BETWEEN RISK ASSESSMENT AND MANAGEMENT . . .
BUT NEED TO PRETEND TO DIFFERENTIATE CLEANLY
c. SCIENTIFIC PEER REVIEW AS DISTINCT FROM POLICYMAKING
EXAMPLES: EPA AND SCIENCE ADVISORY BOARD, FDA PUBLIC BOARD OF INQUIRY ASPERTINE
POLICY MAKERS SEEK PEER REVIEW TO EVADE RESPONSIBILITY BUT....
DOES NOT WORK WELL IN AREAS CHARACTERIZED BY UNCERTAINTY AND CONTROVERSY.
(3) BOTTOM LINE -- "science is subjected to extreme deconstruction in the US regulatory process ... the
legitimacy of regulatory decisions ultimately rests on persuasive reconstruction of justifications in scientific,
economic and legal terms. "
NOTE: SHOULD VISCUSI AND MORGAN BE SEEN AS JASANOFF'S DATA?
NOTE: IS CONSTRUCTIVIST USE OF LANGUAGE IMPRECISE OR SOPHISTICATED OR BOHT?
D. CREDIBLE ASSESSMENT OF SCIENTIFIC AND TECHNICAL INFORMATION
IN HOUSE EVALUATION AS BASELINE - DoD, EPA, OSTP, OTA
NOMINALLY NONADVERSARIAL METHODS – “HONEST” BROKERS (MCCRAY)
EXTERNAL CONSULTANTS -- BELTWAY BANDITS, CONSULTING FIRMS
FFRG - RAND AND MITRE and LINCOLN LABS
EXTERNAL ADVISORY GROUPS - EPA SCIENCE ADVISORY GROUP AND CASAC
AD HOC CONFERENCE - NIEHS-OSTP METHYL MERCURY FAROE - SEYCHELLES
THIRD PARTY ASSESSMENT SPECIALISTS - HEALTH EFFECTS INSTITUTE
PROFESSIONAL ASSOCIATIONS - NAS/NRC, APS
UNIVERSITIES DoD=>TRW GAO=>LINCOLNLABS=>MIT
PEER REVIEW / PUBLICATION (SHORT TERM) -- REVIEWERS AND EDITORS
LAW – UNBIASED JUDGE SCREENS EVIDENCE AND DECIDES
NOMINALLY ADVERSARIAL METHODS
LAW -- ATTORNEYS PRESENT SIDES => JURY
TEAM A TEAM B AS CHECK ON MOTIVATED BIASES => DECISIONMAKER
PEER REVIEW / PUBLICATION (LONG TERM) => SCIENTISTS
WHERE DO THESE KNOWLEDGE APPRAISAL OPTIONS FALL IN DIAGRAMS BELOW?
SOME OTHER DIMENSIONS
RESPONSIBILITY -- DUCKING RESPONSIBILITY FOR CONTROVERSY BY HANDING OFF
TIMELINESS -- DELAYING DECISIONS UNTIL THE NEXT WATCH
FROM PERSPECTIVE OF DEMANDER OF EVALUATION
TO WHOM WILL YOU TURN FOR ASSESSMENT ON WHAT ISSUES?
* WHITE HOUSE AND DoD ON LIKELY EFFICACY OF NMD?
* AG-INDUSTRY ON SAFETY OF GMO FOODS?
* DoD ON EFFICACY OF V22?
* WHITE HOUSE AND EPA ON CLIMATE CHANGE?
FROM PERSPECTIVE OF SUPPLIER OF EVALUATION
SUPPLY: GAIN CREDIBILITY, WEALTH, POWER . . . & AVOID EXTINCTION
TOPIC SELECTION AND DEFINITION - WHICH ISSUES WOULD YOU TAKE?
FUNDING SOURCES - HOW DIVERSE?
PANEL COMPOSITION - HOW WOULD YOU PICK?
PROCESS OF APPRAISAL - HOW WOULD YOU PROCEED?
.... CAN POTENTIAL EVALUATORS FINESSE THE THIRD RAIL PROBLEM?
HOW WIDELY CAN INTERMEDIARIES BE USED TO ASSESS KNOWLEDGE?
ORG HISTORY SETTING LEADERSHIP MODUS KEY DECISIONS INDEPENDENCE
CASAC 1977 CAA EPA 7 MEMBERS 6 TIMES A YEAR, EPA HOUSED IN
FEDERAL 5 ACADEMIC DEVOLVE WORK ADMINISTRATOR SCIENCE
ADVISORY 1 STATE ONTO PANELS CONTROLS ADVISORY
COMMITTEE AGENCY MEMBERSHIP BOARD
ACT AND PROJECTS
HEI 1980 EPA AND BOARD COX, EXTRAMURAL HEALTH PRESTIGE AND
NONPROFIT MOTOR EPA AND TEAM TO RESEARCH FIREWALLS, DE
VEHICLE KENNEDY CONDUCT COMMITTEE NOVO REVIEW
RESEARCH AND SETS PROJECTS
NIH CDP 1977 WI OFFICE OF DIRECTOR NIH ROUTINIZED -- NIH DIRECTOR ABSENCE OF
120 MEDICAL AND OMAR NONFED NON AND OMAR ADVOCATES
CONSENSUS APPLICATION STAFF ADVOC STAFF REDUCES
STATEMENTS S RESEARCH 1.5 DAYS APPROVE PRESSURE
TO DATE OMAR IN NIH TESTIMONY TOPICS (EXCEPT NIH?)
COMPOSE AND PROJECTS
STATEMENTS * PUBLIC
NRC 1916 WWI WORKING ELECTED 12-15 EXPERTS EXCOM OF FUNDERS AND
ARM OF NAS MEMBERSHIP 4-6 MEETINGS GOVERNING SPONSORS DO
AND NAE AND NRC BORAD 18-27 MONTHS NRC BOARD NOT CONTROL
INST MEDIC OF FIVE FROM COSTS 250k+ APPROVES
STAFF 11OO EACH ORG STAFF PROJECDTS
1000 COMM INFLUENCE COMMITTEE
10000 STANDING VARIABLE MEMBERS
EXPERTS COMMISSIONS COMMITTEE APPROVED BY
105TH CONG BALANCE SUBORGS
INITIATED 45 CRITICAL RELEASE
300-350 SIGNOFF OF
2/3 MEMBES PLUS
1/3 RESEARCH COMMITTEE
OTA 1972-1995 TECH LEGISLATIVE 18-30 MONTHS TAB APPROVAL NOT
ASSESSMENT OVERSIGHT ADVISORYY FOR NEW MUCH....BUT
BOARD OF 12 AND FUNDING COMMITTEE PROJECTDS STILL PULLED
LEGISLATORS EXTERNAL PLUS STRONG APPOINTMENTS OFF REPORTS
ADVISORY STAFF 3-5 OF PERSONNEL
BOARD 2 TO 3 BY DIRECTOR
MEETINGS RELEASE TAB
LONG REPORTS APPROVAL
CASES OF KNOWLEDGE ASSESSMENT: WHERE DO THESE MODELS WORK BEST? WORST?
ACADEMIC PEER REVIEW -- PASSIVE SMOKING; TRANSFATS; CHEMOTHERAPY
=> REVIEWERS AND EDITORS ACCEPT CONVENTIONS, RESIST UNUSUAL
=> BUT BROAD ARRAY OF MATERIAL IS EVENTUALLY PUBLISHED
JURY WITH ADVERSARIAL LEGAL SYSTEM – IMPLANT EFFECTS ON IMMUNE SYSTEM
=> PLAINTIFFS WIN; SAFETY STUDIES INITIATED; ANGEL WRITES; FDA RECONSIDERS
=> DAUBERT DECISION – JUDGES NOW SCREEN EVIDENCE
CASAC EPA ON ENVIRONMENTAL ISSUES
=> CREDIBLE ASSESSMENT OF SCIENTIFIC EVIDENCE WITH EFFECTS ON POLICY
=> IS BUSH ADMINISTRATION NOW PACKING ADVISORY COMMITTEE?
HEALTH EFFECTS INSTITUTE REANALYSIS OF HSPH SIX CITIES
=> STABILIZE EPA PARTICULATES STANDARDS
=> INDIRECT EFFECT ON AMERICAN TRUCKERS VS EPA
NIEHS-OSTP CONFERENCE AND NAS PANEL ON METHYL MERCURY
FAROE ISLANDS AND SEYCHELLES STUDIES
=> REVISION OF EPA MERCURY STANDARDS
NAS ON ASBESTOS IN WATER
=> REREVISION OF EPA ASBESTOS STANDARD
AMERICAN PHYSICAL SOCIETY PANEL ON EFFECTIVENESS OF PATRIOT MISSILE DURING GULF WAR
=> DELAYED FOR YEARS AND NO EFFECT
OFFICE OF TECHNOLOGY ASSESSMENT AND EFFICACY OF STRATEGIC DEFENSE INITIATIVE
=> NO EFFECT ON POLICY AND DEMISE OF OTA
LOOK AHEAD TO CONCLUDING SESSIONS -- OPEN QUESTIONS ON CREDIBLE ASSESSMENT
* ON WHAT ISSUES MIGHT THIRD PARTY ASSESSMENT HELP BRING BETTER SCIENCE INTO DECISIONMAKING?
* WHAT EXISTING ORGANIZATIONS OR COMBINATIONS OF ORGANIZATIONS MIGHT SERVE AS CREDIBLE
EVALUATORS OF EVIDENCE IN NATIONAL AND INTERNATIONAL CONTROVERSIES?
DISCUSS PROBLEMS ASSOCIATED WITH TRW SENSOR TEST CASE
DISCUSS APS CHOICES RE PATRIOT
DISCUSS OTA AND SDI
DISCUSS HEI AND PARTICULATES
DISCUSS NIEHS NAS AND METHYL MERCURY
3. RESEARCH AFFILIATIONS AND RESEARCH RESULTS: PASSIVE SMOKING
* WHAT ARE THE KEY FINDINGS OF BARNES AND BERO
PRESENT TABLES 1-4 OHP
* ASSESSMENT OF METHODOLOGY IN LIGHT OF LETTERS TO EDITOR
* INTERPRETATIONS OF CORRELATION BETWEEN AFFILIATION AND RESULTS
BRIBERY: BUY OR LEASE RESEARCHERS TO GET THE RESULTS YOU WANT
PREDICTABILITY: FUND RESEACHERS THAT ARE LIKELY TO YIELD RESULTS YOU WANT
* INTERPRETATION OF PEER REVIEW AND QUALITY INDEX NONCORREL
PEER REVIEW INEFFECTIVE
QUALITY INDEX MEANINGLESS
IMPUTATION OF RESULTS WRONG
* WHAT ARE THE PRESCRIPTIVE IMPLICATIONS OF FINDINGS FOR . . .
RESEARCH FUNDING STRATEGIES
EDITORS OF SCIENTIFIC JOURNALS
JOURNALISTS WRITING STORIES
POLICYMAKERS SEEKING INFORMATION ON RISKS
* ARE THERE SIDE COSTS ASSOCIATED WITH YOUR PRESCRIPTIONS?
COUNTER CASE - FRED KUMMEROW UNIVERSITY OF ILLINOIS
DIET AND HEART DISEASE – TRANSFATS AND NATURAL SOURCES OF CHLOLESTEROL
CURRENT DEVELOPMENT – TRANSFAT WARNING -- LABELS BEING PHASED IN
PAST STANDARD – SUBSTITUTE FOODS WITH TRANSFATS TO CUT BUTTER AND EGGS
RESEARCH 1959 TO 1970
AUTOPSY – HEART DISEASE AND TRANSFATS IN ARTERIES – SUGGESTIVE
SMALL EXPERIMENTAL STUDIES ON RATS – DIET AND MATERNAL MILK
LARGE EXPERIMENTAL STUDIES ON PIGS
Burnsides oil revenue to UI earmarked for Fred => 12 million
National Institutes of Health => 400 K
Wallace Foundation [Henry Wallace => Jean Douglas] => 100K
Gifts from egg hatchery in Gibson City => 25 K and from dairy industry => 25 K
REAL WORLD DEVELOPMENTS 1968-1973
American Heart Association Diet Advisory Committee
150000 copies warning on transfats…Shortening Institute objected with drawn and replaced
Cooper -- Head of NIH testified that eating eggs caused heart disease
Kummerow -- testimony contradicted Cooper, emphasized transfats
No more NIH funding…. Five year NIH renewal gone.
Drew Pearson attack -- 25K contribution from egg industry
POST SCRIPT research in US, Canada, UK, Finland link transfats, arterial plaques, heart disease
POST SCRIPT CBC documentary and Kraft
4. IMPLANT CONTROVERSY - MARCIA ANGELL
* Manufacturers introduce with assured safety despite absence of studies establishing safety
No warnings of health effects or uncertainty over health effects
* Case reports on connective tissue disorder
* A few spectacular tort cases
* FDA Kessler ban 12 years after introduction
No testing, no evidence of safety, noisy public hearings and NEJM piece.
Angell: No systematic studies. local complications only.
* Explosion of tort cases
* Cancer scares not borne out by studies.
* Autoimmune problems new focus
belief in connective tissue diseases including lupus, arthritis, scleroderma, polymyalgia
harder to study connective tissue disease - symptoms less clear, diagnosis murky
Clinical case reports but no systematic studies wi incidence with and without implants
* Implant manufacturers lose suits, agree to 4.25B settlement 440,000 register 15,000 opt out
ANGELL'S SIX THEMES
1 place of regulation in American life: political perception and FDA -- why call for evidence so late?
* once cases in and fear rising, demand evidence so as not to appear to cave in to manufacturers
* letter of the law -- no evidence of safety
QUERY: was there regulatory failure in earlier granting of permission to market in the first place?
2 impact of tort law on American life -- why litigation succeeded despite absence of science on harm
* cases beget cases
* US system -- no loser pay costs
QUERY: what tort reforms would make sense?
3 marginal role of scientific evidence: public does not understand how science works
* large scale epidemiological studies => good
* animal studies, laboratory studies, case reports => suspect
* small studies -- size of study and threshold of detection
QUERY: after-the-fact epi studies deemed more conclusive by Angell. How approve?
4 science in courtroom -- pre Daubert
* expert witnesses chosen by sides to give opinions that become legal evidence
* opinions may be based studies or unsystematic experience
* need not provide basis for conclusions except under rare informed cross exam
=> sci -- slow accumulation of information
=> law -- adversarial - controlled argument
"why did you do a study with existing studies already complete?"
"all studies tentative, looking for consistency across different studies"
"so your study and its conclusions are tentative?'
5 profit motive by manufacturers and lawyers -- harass Dr. Gabriel re documents, data, deterring research
6 media coverage - hyped dangers with emphasis on coverup by manufacturers and FDA
GENERAL QUESTIONS AND OBSERVATIONS FOR CLASS DISCUSSION
=> Did the system fail? Manufacturers did not test for safety when reason to believe that may be dangers.
FDA did not regulate. Tort system generated incentives for study. Until systematic studies done, FDA should
=> What if studies had come in with dangers? How would story read? Are there examples of inadequate
testing flagged by torts that resemble this case? Asbestos? Tobacco?
=> Is the incentive for manufacturers to test for safety stronger or weaker now than at the time of the
breast implant tort actions? Set up for Daubert.
=> Harassment of Gabriel => set up Harvard study and data controversy
JUDICIAL SYSTEM AND KNOWLEDGE ASSESSMENT
”Science is simply common sense at its best; that is, rigidly accurate in observation and merciless to a fallacy
Thomas Henry Huxley as cited in Reference Manual on Scientific Evidence.
TWO BASIC VIEWS
Adversarial: struggle between defendents and prosecutors/plaintiffs exposes weaknesses and strengths.
judge responsible for facilitating fair conflict between sides
Technocratic: judge and advisors assess quality of science and appraise credibility of knowledge
DAUBERT v MERRELL DOW PHARM 1993
Supreme Court obligated judges to perform gatekeeping. Must make a preliminary assessment of whether the
reasoning or methodology underlying the testimony is scientifically valid and of whether that reasoning or
methodology properly can be applied to the facts at issue. Replaced general acceptance (Frye) with
* Toxic tort - benedictin and birth defects
* Daubert factors
1. theory or technique can be (and has been) tested -- empirics
2. subjected to peer review or publication
3. existence of known or potential error rates and existence and maintenance of standards
4. widespread acceptance within a relevant scientific community
=> judges warned not to engage in wholesale exclusion -- cross examination, presentation of contrary
evidence, careful instruction are appropriate means by which evidence based on valid principles may be
=> limited screening by trial judge will on occasion prevent jury from hearing of authentic breakthroughs.
Rules not designed for cosmic understanding, only to resolve legal disputes
GENERAL ELECTRIC COMPANY V JOINER 1997
Supreme Court ruled appellate courts should let trial court evidentiary rulings stand unless clear abuse of
* toxic tort - smoker, family history, PCB exposure
* plaintiff witnesses excluded by trial judge
* infant mice dosed with massive PCB develop different cancer
* plaintiff claimed that pooled reanalysis data -results would have been significant even if individual studies not
* Breyer suggests use of court appointed experts to determine admissibility of complex scientific or technical
=> Merits and demerits of letting jury handle this?
KUMHO TIRE CO V CARMICHAEL - 1999
Supreme Court extended gatekeeping to "technical" and "other specialized" knowledge
* product liability, tire blew out causing accident
* expert witness visual inspection of tire, defect in manufacture and design
* defendent moved to exclude, trial court concurred - Daubert four tests applied to tech
circuit court of appeals -- "skill or experience based observations" OK -- differ from science.
supreme court - reiterated GE and offered criticisms of expert witness Carlson
=> Why not take it to jury? Would exactly the sort of attack offered be comprehensible to jury?
Note: Kumho Tire allows courts to make judgemnt to accept experience, even though Carlson found
implausible and unscientific in this case.... Daubert four factors not rigid requirement
Note: Kumho Tire asks expert to apply same level of intellectual rigor that characterizes the practice of an
expert in the relevant field.
Seventh Circuit Decisions by Judge Posner
ROSEN V CIBA GEIGY -- barred testimony by Dr. Fozzard on nicotine patch / heart attack
BRAUN V LORILLARD -- barred testimony by Schwartz on asbestos fibers and cancer
=> if witness departs from accepted methodology, must justify wi intellectual rigor using approach consistent
with how colleagues in relevant field or discipline would proceed to establish a proposition
EXAMPLES OF ISSUES THAT MAY REQUIRE JUDGEMENT
Judges asked to understand how statistical analyses work
Hunt v. Cromartie - plausible alternative interpretation of redistricting as discrimination
Department of Commerce v US House -- would standards of counting affect plaintiffs?
Right to assisted suicide? can medical technology reduce or eliminate risk of dying in pain?
Is a person so dangerous and mentally ill that threat to public safety requires indefinite noncriminal
In criminal cases, scientific validity of DNA sampling, voiceprints, predictions of future dangerousness.
In administrative review safety of drug, risks attending nuclear waste disposal, leakage potential of toxic waste,
risks to wildlife from building a dam
In patent law, understanding and defining originality, utility, nonobvious.
In tort law -- assign responsibility for injury or death to exposures, blame in accidents.
In antitrust -- effects of technology bundling and investments on competition?
BERGER EXAMPLES OF CHALLENGES AND STATUS
Challenges to astrology and necromancy -- no problem
Challenging nonorthodox branch of traditional discipline
courts reject clinical ecology -- multiple chemical sensitivity (ashford) not validated by testing.
Upshot: No basis for appellate courts overturning district court decision rejecting such witnesses
Challenging reliability of traditional field of expertise -- forensic science
Upshot: usually accepted because prosecutors rely on this accepted witchcraft
Challenging expert testimony that does not prove causation
toxic torts and product liability rest explicitly on proof of causation
some unresolved links often exist
reliance on animal studies often challenged -- animal evidence rejected in GE
- Black v Food Lion - fall and fibromyalgia - MD testimony on disease excluded without proof of causation
- Moore vs Ashland Chemical - reactive airways disorder after cleaning up chemical spill. excluded
expert medical testimony - no proof -- fumes at higher levels in study, temporal inference no good,
surmise without proof....not thoroughly tested
-Heller v Shaw Industries -- respiratory problems VOC carpet. excluded testimony of industrial hygenist on rug
as source of VOCs. excluded medical expert. OK to bar hygenist, not OK to kill medical expert. accepted
clinical judgement including temporal evidence.
* judges lack scientific training
* many cases fall within area of scientific uncertainty
* must respect juries constitutionally specified role (even if results incorrect)
External expert advice
President - Science Advisor - OSTP
Congress - NAS
Judiciary - NAS Program in Science, Technology and Law; advisors
Pretrial conferences to narrow scientific issues in dispute
Pretrial hearings where potential experts subject to examination by court
Appointment of specially trained clerks or scientific special masters
Appointment of independent experts (going beyond expertise offered by parties to dispute)
Judge Stearns - Harvard Medical School Professor -- genetic engineering patent case
Judge Pointer - neutral science panel of four scientists from different disciplines
-- to prepare testimony on scientific basis of claims in implant case
-- video taped ("consistent" "savings of time and expense")
AAAS-ABA National Conference of Lawyers and Scientists -- slate of candidates from science and
professional organizations -- where court has determined that traditional means of clarifying issues under the
adversarial system are unlikely to yield the information that is necessary for a principled resolution of disputed
=> Are these procedures "neutral"
=> What are effects of Daubert and successor cases likely to be?
=> Link back to Angell and implants controversy
=> Link back to EU triazolam and GMO administrative rulings
5. HSPH AND ACS PARTICULATES STUDIES: UNCERTAINTY AND THIRD PARTY KNOWLEDGE APPRAISAL
* CAUSALITY DIFFICULT OR IMPOSSIBLE TO INFER
* FALSE POSITIVES FROM
* FALSE NEGATIVE OR INSIGNIFICANT RESULTS FROM
IMPOSITION OF CONTROLS ON PATHS OF CAUSATION
USE OF CORRELATED NOMINALLY INDEPENDENT VARIABLES IN MODEL
* TYRANNY OF STATISTICAL SIGNIFICANCE
ANALYSIS OF SECULAR TRENDS IN DEFECTS BEFORE AND AFTER INTRODUCTION/WITHDRAWAL
* WHAT ELSE IS GOING ON APART FROM CHANGE IN TREATMENT?
* GENERALIZABILITY -- SPECIES SPECIFICITY OF EFFECTS
IN VITRO STUDIES
* GENERALIZABILITY -- ARTIFICIALITY OF LABORATORY IN VITRO EXPOSURES
CHEMICAL STRUCTURE COMPARISONS OF SUBSTANCES WITH KNOWN AND UNKNOWN RISKS
* HOW DEFINE DEGREE OF ISOMORPHISM
* CONTROVERSY OVER PROJECTED SIMILARITY IN EFFECTS
CONTROLLED HUMAN STUDIES
* HARD TO DO ONCE HARM IS SUSPECTED
FIGHTS OVER EPIDEMIOLOGICAL EVIDENCE
I=> INSERT SLIDES FROM DOCKERY
HARVARD SCHOOL OF PUBLIC HEALTH SIX CITIES STUDY
DOCKERY ET AL NEJM 1993
8111 ADULTS IN SIX CITYES 14-16 YEARS
LEAST-MOST POLLUTED FINE PARTICLES AND SULFATES => 26% INCREASE IN MORTALITY
FINE PARTICLES => CARDIO PULMONARY DISEASE
AMERICAN CANCER SOCIETY STUDY
POPE ET AL AM
REV RESPIRATORY AND CRITICAL CARE MEDICINE 1995
552138 ADULTS IN 154 CITES 1982-89; 50 CITIES WI FINE PARTICLE DATA AVAILABLE
LEAST-MOST POLLUTED FINE PARTICLES => 17% SULFATES => 15%
EPA NATIONAL AMBIENT AIR QUALITY STANDARDS 1997
INDUSTRY CHARGES AGAINST HSPH AND ACS
BIAS BY RESEARCHERS
=> "LEAD ZEALOTS"
=> MOTIVE TO SECURE FUNDING FOR RESEARCH
FLAWS IN RESEARCH DESIGN
=> CODING ERRORS
=> HANDLING OF DROP OUTS FROM COHERT
=> INADEQUATE CONTROLS FOR SEDENTARY LIFESTYLE AND SMOKE INHALATION
=> MEASURES FOR SHORT PERIOD - INVALIDATE LONG TERM CONCLUSIONS
=> ADJUSTED MORTALITY RATE RATIOS
=> "SECRECY" BY NOT RELEASING DATA
=> "COVERUP" OF FLAWED DESIGN AND CODING ERRORS
HEALTH EFFECTS INSTITUTE: JULY 2000 REANALYSIS OF HSPH AND ACS STUDIES
* WHO FUNDED THE REAPPRAISAL? WHO PARTICIPATED?
* WHAT METHODS WERE USED?
=> REPLICATION AND VALIDATION - BOTH FINE
=> SENSITIVITY ANALYSIS
PHYSICAL ACTIVITY, LUNG FUNCTION MARITAL STATUS - NO EFFECT
SENSITIVITY OF SUBGROUPS - LOWER EDUCATION MORE SENSITIVE
LINEARITY AND NONLINEARITY OF RESPONSES ACS - MIXED RESULTS
RUN WITH TIME DEPENDENT VARIATION IN POLLUTANTS - ASSOCIATION DROPS
ECOLOGIC COVARIANTS - IE PHYSICIANS PER CAPITA
* WHAT ARE THE EFFECTS OF THIS EXERCISE? IN RETROSPECT, WHAT WERE PROS AND CONS OF APPROACH?
* HOW WIDELY COULD INTERMEDIARIES BE USED TO ASSESS KNOWLEDGE?
=> CASAC, HEI, HEINZ CENTER, NIH CONSENSUS DEVELOPMENT PROGRAM, NRC . . .
AND THE LATE DEARLY DEPARTED OTA
SIDEBAR AMERICAN TRUCKING ASSOCIATION VS EPA
* FIGHT OVER EPA PARTICULATE AND OZONE STANDARD
* KEY ISSUES RAISED IN CASE
DELEGATION OF POWERS FROM CONGRESS TO EPA UNCONSTITUTIONAL
STANDARDS SET ARBITRARY AND LACKING IN DUE PROCESS
INADEQUACY OF EVIDENCE ON HEALTH EFFECTS ON PARTICULATES AND OZONE
NO CLEAR THRESHOLD RE PARTICULATES OR OZONE, HENCE CUTOFFS ARBITRARY
* CIRCUIT COURT OF APPEALS DECISION THREATENED MASSIVE CHANGES IN REGULATION
* SUPREME COURT REVERSED CIRCUIT COURT OF APPEALS DECISION
6. FAROE AND SEYCHELLES STUDIES ON METHYL MERCURY: AD HOC CONFERENCE=>NRC PANEL
POLICY ISSUE: EPA REFERENCE DOSE FOR METHYL MERCURY
POWER PLANTS 45%
MUNICIPAL INCINERATORS 25%
OTHER SOURCES 30%
LARGE N PEER REVIEWED STUDIES
MATERNAL LOW DOSE EXPOSURE => CHILDREN NEUROBEHAVIORAL EFFECT
FAROE ISLANDS STUDY IQ LOSS AND DEVELOPMENTAL DELAYS
SEYCHELLE ISLANDS IQ GAINS AND DEVELOPMENTAL BENEFITS
METHODS OF KNOWLEDGE ASSESSMENT IN CONTROVERSY:
AD HOC CONFERENCE NIEHS OSTP EPA => SEEK ADJACENT EXPERTISE => NOT SURE, FUND MORE
NRC STUDY REQUESTED BY CONGRESS => PANELS => ACCEPT FAROES AND EPA REF DOSE
Committee on Environmental and Natural Resources (CENR)
Office of Science and Technology Policy (OSTP)
Scientific Issues Relevant to Assessment of Health Effects from Exposure to Methylmercury
November 18-20, 1998
Individual Panel Reports Appendix I
-Exposure Panel Appendix I-A
-Neurobehavioral Endpoints Panel Appendix I-B
-Confounders & Variables Panel Appendix I-C
-Design/Statistics Panel Appendix I-D
-Experimental Panel Appendix I-E
Reports from Study Teams Appendix II
-Iraq & Seychelles Studies Appendix II-A
-Faroe Islands Studies Appendix II-B
-Amazon Studies Appendix II-C
Comments from Study Groups on Workshop Report and Panel Appendix III
-Iraq & Seychelles Appendix III-A
-Faroe Islands Appendix III-B
References Appendix IV
Workshop Program Appendix V
Attendees List Appendix VI
PANEL FINDINGS AND RECOMMENDATIONS
1 Methylmercury is a developmental neurotoxin, but effects at low doses encountered by eating fish
are difficult to evaluate.
2 All the studies reviewed were considered of high scientific quality and the panel recognized that
each of the investigations had overcome significant obstacles to produce important scientific
information. The panel also stated that continued funding of the studies in the Seychelles, Faroes,
and Amazon is necessary for the full potential of those studies to be realized. This is particularly
the case for the Faroes and Seychelles studies, which have assessed and are currently assessing the
potential developmental neurotoxic effects of methylmercury in fish-eating populations. The
developmental studies would benefit by evaluation of common endpoints using similar analytical
methods. It is important to note that the Amazon study did not assess developmental endpoints but
assessed effects in adults.
3 Results from the Faroes and Seychelles studies are credible and
provide valuable insights into the potential health effects of
4 Some differences are clearly present in results from the Faroes,
Seychelles, and Amazon, but the panel was not able to clearly
identify the sources of these differences. Among possible sources
are the different effects of episodic versus continuous exposure,
ethnic differences in methylmercury responses, lack of common
endpoints in the Faroes and Seychelles studies, and several other
confounders or modifying factors such as those found in diet and
lifestyle, as well as in chemicals present in seafood, which is
the source of methylmercury to these populations. The other
chemical constituents of seafood that may be explanatory include
those that may be beneficial to fetal neurodevelopment (i.e.,
omega-3 fatty acids) and those that may be harmful to fetal
neurodevelopment (e.g., PCBs).
5 These studies have provided valuable new information on the potential health effects of
methylmercury but significant uncertainties remain because of issues related to exposure,
neurobehavioral endpoints, confounders and statistics, and design.
EPA's Methylmercury Guideline Is Scientifically Justifiable For Protecting Most Americans,
But Some May Be at Risk NRC Press Release July 11, 200
….The overall weight of the evidence from this comprehensive review led the committee to conclude that
EPA's reference dose is scientifically justifiable for protecting the health of the vast majority of Americans.
….When the agency first developed its guideline five years ago, EPA judged data from a 1971 Iraqi poisoning
incident to be the most relevant. To provide EPA with more appropriate data in formulating its reference dose,
the committee analyzed population studies in the Faroe Islands, Seychelles Islands, and New Zealand. It
concluded that the Faroe Islands analysis should be used by EPA as the critical study for deriving the
reference dose, the report says.
NRC Commission on Life Sciences Board on Environmental Studies and Toxicology
Committee on the Toxicological Effects of Methylmercury
Robert A. Goyer, M.D. (chair) Pathology, University of Western Ontario
H. Vasken Aposhian, Ph.D. Molecular and Cell Biology and Pharmacology University of Arizona
Lenore Arab, Ph.D. Epidemiology and Nutrition, University of North Carolina
David C. Bellinger, Ph.D. Neurology Harvard Medical School
Thomas M. Burbacher, Ph.D. Environmental Health, University of Washington
Thomas A. Burke, Ph.D. Health Policy and Management and Risk Science and Public Policy Johns Hopkins
Joseph L. Jacobson, Ph.D., J.D. Obstetrics/Gynecology and Psychology Wayne State
Lynda P. Knobeloch, Ph.D. Toxicologist Wisconsin Bureau of Environmental Health Madison
Louise M. Ryan, Ph.D. Biostatistics Dana-Farber Cancer Institute and Harvard School of Public Health
Alan H. Stern, Dr.P.H. Risk Analysis New Jersey Department of Environmental Protection
Carol A. Maczka, Ph.D. NRC Study Director
SOME DIFFERENCES IN MAJOR EPI STUDIES
FAROE SEYCHELLES NEW ZEALAND
MERCURY EXPOSURE MEASURE UMBILICAL CORD BLOOD MATERNAL HAIR MATERNAL HAIR
NEUROBEHAVIORAL ENDPOINTS BROADER NARROWER BROADER
CONDITIONS OF TESTING IN CAPITAL IN FIELD IN FIELD
PRIMARY SOURCES FISH AND WHALE FISH FISH
DIETARY INTERACTION PCBS, OMEGA OMEGA OMEGA
FAILURES TO CONTROL MATERNAL ALCOHOL --- ---
SELECTION BIASES --- ELIMIN8 BIRTH DEFECTS ---
DIVERSITY OF POPULATION VERY LOW LOW HIGH
FINDING - EFFECTS OF LOW DOSE BAD GOOD BAD
CENROSTP CONFERENCE => SUSTAINED ANALYSIS UNDERSCORED UNCERTAINTY
NRC => TWO STUDIES AGAINST ONE PLUS ANIMAL STUDIES – TILT TO FAROE
=> USE LOWEST DOSE RESPONSE FOR REFERENCE DOSE – BOSTON NAMING TEST
QUESTIONS ON PROCESS AND RESULTS –
WAS STATEMENT ON UNCERTAINTY A PREDICTABLE PRODUCT OF CONFERENCE?
WAS ACCEPTANCE OF EPA REFERENCE DOSE WITH CAUTION A PREDICTABLE PRODUCT OF NRC PANEL?
WITH BENEFIT OF HINDSIGHT, HOW STRUCTURE KNOWLEDGE ASSESSMENT TO YIELD DIFFERENT RESULTS?
AT WHAT LOSSES IN TERMS OF CREDIBILITY? WITH WHAT DELAYS?
WITH BENEFIT OF HINDSIGHT, HOW “IMPROVE” KNOWLEDGE ASSESSMENT PROCESS?
6. TRIAZOLAM - ABRAHAM AND SHEPPARD
UK + US DIFFER ON CERTIFICATION TRIAZOLAM "HALCION"™ - SEDATIVE WITH SHORT HALF LIFE
70'S 80'S UK AND US CERTIFIED EFFICACY AND SAFETY AT .5 MG DOSE
90'S US CERTIFIED -- SAFE AND EFFICACIOUS AT .25 MG DOSE
90'S UK WITHDREW -- NOT SAFE .5 MG NOT EFFICIOUS AT .25 MG
EVIDENCE ON PSYCHOTIC SIDE-EFFECTS
* MURDER CASE WITH HALCION DEFENSE
* SPONTANEOUS REPORTS / MILLION PRESCRIPTIONS TABLE 1
TABLE 1 FROM ABRAHAM AND SHEPPARD
UK Spontaneous Reports of Psychiatric Reactions by Year for Specified
Benzodiazepines per Million Prescriptions
Year 1979 1980 1981 1982 1983 1984 1985 1986
Triazolam 114.0 44.0 14.0 43.0 20.0 5.5 1.8 2.5
Flurazepam 0.9 0.9 0.4 1.3 0.5 0.0 2.2 0.0
Nitrazepam 0.1 0.9 0.5 0.0 0.3 0.0 0.8 0.4
Temazepam 4.1 4.5 3.7 1.9 2.2 1.7 1.3 0.4
Lorazepam 2.4 3.3 1.4 2.0 4.3 1.9 6.7 1.9
Diazepam 0.4 0.7 0.1 0.1 0.2 0.4 0.4 1.0
* INITIAL UPJOHN TRIALS - ERRORS
PLACEBO 1 EPISODE REPORTED AS 2 EPISODES
DRUG 7 EPISODES REPORTED AS 1 EPISODE
* NEW TRIALS PLACEBO .5% OTHER DRUG 1.9 % TRIAZOLAM 9.9 %
UK EU COMM ON PROPRIETARY MEDICAL PROCEDURE BANNED
US FDA ADVISORY COMMITTEE 7/8 VOTED SAFE & EFFECTIVE AT .25 MG
Differences UK US
Prior trust Trusted Sources never trusted sources
=>disillusioned => not disillusioned
no benefit of doubt in assessment of data business as usual in assessment of data and
and evaluation of risks evaluation of risks
Integration of data Integration of initial studies, clinical trials, Consideration of each source of information in
postmarket data isolation
Disciplinary MD/epidemiologist/pub health Clinician v Epidemiologist
managers of process
Advisory Committee Generalists Specialists in pharmacology
Org structure licensing & postlicensing by different orgs licensing & postlicensing by same org and people
Conflicts of interest for None 9/11 grants from Upjohn
advisory committee 1/11 stockholder
7/8 voters - FDA waivers
Ideology traditional role of state as protector Neoliberalism and deregulation
-- but deregulation within UK as well standards dictated by parties
effect of AIDS patient demand for easier approval
=> ABRAHAM AND SHEPPARD DO NOT EVALUATE OUTCOMES.
BOTH US/UK DRIVEN BY ENDURING MATERIAL INTERESTS AND DISCIPLINARY OUTLOOKS
=> WOULD YOU EVALUATE OUTCOMES? WHICH PREFER AND WHY?
=> WHAT FACTORS IMPORTANT? WHAT REFORMS WOULD YOU RECOMMEND?
KNOWLEDGE ASSESSMENT IN SECURITY AFFAIRS -- APPRAISING CAPABILITIES AND THREATS
TECHNICAL ASSESSMENTS OF EFFICACY OF OWN WEAPONS SYSTEMS AND STRATEGIES
Performance of new weapons systems under ideal conditions
Performance of new weapons systems under operational conditions with a strategizing adversary
Historical Examples: Battleship vs Aircraft Carriers
Recent Examples: C3I and counterforce, Patriot I, BMD
Error Underestimate => prolonged undue reliance on old system
Error Overestimate => premature undue reliance on new system
ASSESSING THREATS POSED BY OTHERS
Technical assessment of existence and efficacy of adversarial weapons systems
Political assessment of adversarial capabilities, intentions and strategy
Historical Examples: Soviet A&H Bomb, Missile & Bomber Gap, Counterforce, NATO Central Front
Recent Examples: Al Quada; Iraq, Iran, North Korea WMD
Error Underestimate => greater vulnerability to surprise attack and deterrence failure
Error Overestimate => unnecessary spiraling via arms expenditures, preemptive war, and preventive war
QUESTIONS ON KNOWLEDGE ASSESSMENT IN SECURITY AFFAIRS
What are some key sources of uncertainty associated with these tasks?
What can be known in times of peace? What can be known only in time of war?
In absence of war, what methods can one use to assess knowledge on efficacy and threats?
How vulnerable are such methods to institutional and ideological distortion?
What are some approaches to handling intrinsic uncertainty PLUS need for secrecy?
What are the costs of false negatives and false positives?
How can we improve the terms of tradeoff across false negatives and false positives?
ASSESSING THE EFFICACY OF WEAPONS SYSTEMS AND STRATEGIES
* Counterforce Escalation Dominance Strategy with Counterforce and C3I
Blair OTA C3I technical critique vs USAF 1980s => not tested
* Effects of North Atlantic Naval Strategy in Conventional War
Posen vs Lehman technical and strategic controversy 1980s => not tested
* US-Iraq Distribution of Casualties and Duration of Campaign
Posen vs Left re campaign analysis 1991 => resolved by gulf war
* Ballistic Missile Defense
80s Strategic Defense Initiative => Office of Technology Assessment
91-96 Patriot - Bush/Raytheon/Army vs Postol/Israelis => partially resolved by APS panel
98-00 BMDO/Clinton/Bush/ vs Postol et al
98-03 DoD => TRW test=>GAO=> Lincoln Laboratories=> MIT
C3I AND COUNTERFORCE ESCALATION DOMINANCE
BRUCE BLAIR, STRATEGIC COMMAND AND CONTROL AND NUCLEAR STRATEGY
BLAIR: MINUTEMAN LAUNCH CONTROL => YALE SOM, OTA, BROOKINGS
CLASSIFIED STUDY FOR OFFICE OF TECHNOLOGY ASSESSMENT
UNCLASSIFIED STUDY FOR BROOKINGS INSTITUTION
QUESTION: IS US CAPABLE OF IMPLEMENTING COUNTERFORCE ESCALATION DOMINANCE STRATEGY?
TASKS: MUST BE ABLE TO PERFORM AND COORDINATE FOLLOWING --
INTELLIGENCE GATHERING ON DAMAGE TO OWN FORCES AND TO SOVIET FORCES
RETARGET OWN FORCES AGAINST UNDAMAGED SOVIET FORCES
WITHHOLD RESERVE TO DETER COUNTERVALUE ATTACK
COMMUNICATE WITH ADVERSARY DURING WAR TO DISCOURAGE COUNTERVALUE ATTACK
TECHNICAL PROBLEMS: VULNERABILITY OF C3I TO BLAST AND EMP
ORGANIZATIONAL PROBLEMS: SOPS WITHOUT REHEARSAL => FUNCTIONALITY DUBIOUS
INTERESTS POTENTIALLY AFFECTED BY STUDY
REDUCE US CAPACITY TO COERCE – EXISTENTIAL MUTUAL ASSURED DESTRUCTION
REDUCE RELIANCE ON US CAPACITY FOR COERCION THAT DOES NOT EXIST
WOULD ELIMINATE DRIVER FOR PROCURING COUNTERFORCE – ACCURATE LAND BASED ICBMS
WOULD STRENGTHEN HAND OF ARMY AND WEAKEN HAND OF AIRFORCE FOR FUNDING
KNOWLEDGE APPRAISAL FIGHT
OTA STUDY CLASSIFIED AT EXTREMELY HIGH LEVEL – SEVERELY LIMIT DISCUSSION
UNDERCUT CLAIMS BY REFERENCE TO UNDISCLOSED CLASSIFIED INFORMATION
CONGRESSIONAL COMMITTEE – CLOSED DOOR HEARING, REVOCATION OF CLEARANCE
WALL STREET JOURNAL ARTICLE – RESTORATION OF CLEARANCES AND BLAIR TESTIFIES
CODA1 – NO CHANGE IN US STRATEGY; ONLY EFFECT HEAVIER INVESTMENTS IN C3I
CODA2 – US AIR FORCE ACADEMY CURRICULUM
1991 GULF WAR - PATRIOT I VS SCUDS FIRED AGAINST ISRAEL AND SAUDI ARABIA
SECURITY INTERESTS DURING WAR
PROTECTING POINT AND AREA TARGETS FROM SCUDS
KEEPING ISRAEL OUT OF CONFLICT AND REASSURING SAUDI ARABIA
DISCOURAGING SCUD ATTACKS BY IRAQ
PATRIOT I – A MODIFIED ANTIAIRCRAFT MISSILE PLACED IN SERVICE ON ACCELERATED BASIS
INITIAL CLAIMS – 99 PERCENT EFFECTIVE -- ARMY AND PRESIDENT BUSH
INTERESTS AFTER THE WAR
REASSURING JAPAN AND SOUTH KOREA RE NORTH KOREA BALLISTIC MISSILE THREAT
SECURING POLITICAL BASE FOR STRATEGIC BALLISTIC MISSILE DEFENSE
SALES OF PATRIOT AND REDUCING US PROCUREMENT COSTS
COUNTER CLAIMS – INEFFECTIVE OR WORSE – ISRAELI ANALYSTS AND POSTOL
GROUND DAMAGE INFLICTED BY INTERCEPTED WARHEADS (ISRAEL)
PUBLIC VIDEO FOOTAGE (POSTOL)
RADAR MISSED WARHEAD (TUMBLING BOOSTER AND DEBRIS)
PROXIMITY FUSE MISTIMED DETONATION
KNOWLEDGE APPRAISAL CONTROVERSY
APPEAL TO CLASSIFICATION – RELEVANT SECRET INFORMATION
PEER REVIEW -- INTERNATIONAL SECURITY PUBLISHED SERIES OF ESSAYS
IMPUGN MOTIVES (ARROW) AND PERSONALITY (POSTOL) THREATEN SECURITY CLEARANCES
AMERICAN PHYSICAL SOCIETY PANEL PUBLIC AFFAIRS SECTION
APS -- PROS AND CONS ON TAKING ON ASSIGNMENT
LONG DELAYS ON ISSUANCE OF APS REPORT
CODA: CLINTON BALLISTIC MISSILE DEFENSE – PANETTA FIGHT, AGENTS VISIT POSTOL SUMMER 2000
CODA: TRW TEST, GAO CRITIQUE, LINCOLN LABS BACK TRW, MIT BACKS LINCOLN, MIT BACKS INQUIRY
CODA: 2003 NEW PATRIOT EFFECTIVE – IMPROVED DESIGN EASIER TARGET, BUT FRIENDLY FIRE
=> GOOD EXAMPLES OF DECONSTRUCTION AND RECONSTRUCTION -- . WHAT SHOULD BE DONE?
ASSESSING THREATS – ADVERSARIAL CAPABILITIES AND INTENTIONS
* Soviet Nuclear Programs 1940s early 50s-- US underestimate Soviet time to develop A and H bombs
Estimate of time to develop based on estimates of diffusion of information and priority assigned
Reality check -- Testing of devices
* Soviet bombers 1950s -- US overestimated Soviet capability
Technical intelligence contradicted earlier estimates
* Soviet ICBMs late 1950s and early 1960s -- US overestimated Soviet capability
Interpretation of less-than-expected investment in bombers for missile capability
Comparison of US planned forces with Soviet maximum potential forces
Technical intelligence contradicted earlier estimates
* Soviet-US ICBM mid 1970s-1980s -- estimates of Soviet capacity to "fight and win a nuclear war"
Modernization and counterforce capability
Civil defense efforts
Deployments of ballistic missile submarines
1976 Competitive Assessment: Team A Howard Stoertz and Team B Richard Pipes
* Soviet-US NATO conventional forces early 1980s -- US overestimated Soviet capability?
Unresolved -- Mearsheimer -- campaign analysis of NATO central front
* India and Pakistan nuclear programs 2000 -- US underestimated Indian and Pakistani programs
Estimate of time to develop based on estimates of diffusion of information and priority assigned
Reality check -- Testing of devices
* Transnational Conventional Terror 2001 -- US underestimated capabilities of Bin Laden organization?
Reality check -- September 11 WTC and Pentagon
* North Korean and Iraq missile and nuclear programs -- over or underestimate?
Team A Clinton estimates Team B Donald Rumsfeld
* Iraq and Weapons of Mass Destruction today -- capabilities and conditions for use?
CIA - Tenet letter to intelligence committee
DIA and DoD and NSC – Stovepipes vs organizational processes and bureaucratic politics
WHAT ARE SOURCES OF FALSE NEGATIVES, FALSE POSITIVES, AND ERRORS?
Organizational and political problems in assessing threats
Friedman -- "cynical view of scholars" -- intelligence as justification for policies that serve parochial interests
what biases and heuristics associated with organizational and economic interests? (crass)
what biases and heuristics are associated with organizational beliefs? (genuine)
can secrecy AND democratic decisionmaking AND good knowledge appraisal be reconciled?
Service Branches - Air Force, Army, Navy Intelligence
=> common sense / world view
=> parochial organizational interests with chain of command
=> nominally above services but officers seconded from branches and leadership favors uncontroversial
=> new conventional wisdom, Rumsfeld tried to use DIA to offset CIA, then turned to stovepiping
=> errors . . . but how much distortion of analysis to satisfy preconceived notions of clientele?
Methods of Structuring Analysis
Formal NIE under DCI
Formal Competitive Assessment - Team A and Team B
Informal Bureacratic Conflict - Kissinger and Wolfowitz extraction and interpretation of raw intelligence
PROBLEMS: FOOLING SELF? FOOLING OTHERS? BLOWBACk?
PRECAUTIONARY PRINCIPLE REDUX – UNCERTAINTY AND KNOWLEDGE ASSESSMENT
“Absolute proof cannot be a precondition for action."
"The message is that there are no knowns. There are things
we know that we know. There are known unknowns, that is to
say there are things we now know we don't know. But there
are also unknown unknowns - things we do not know we don't
"So when we do the best we can and we pull all this
information together, and we then say 'well, that's
basically what we see as the situation', that is really only
the known knowns and the known unknowns. And each year we
discover a few more of those unknown unknowns.”
“There is another way to phrase that, and that is that the
absence of evidence is not evidence of absence.”
Define priors on expected utility of allowing and not allowing.
Define relevant sources of uncertainty.
Define, acquire, and assess relevant information.
Define posteriors on expected utility of allowing and not allowing.
Precautionary Principle Proof before Action
Security Threats US Bush and Rumsfeld France Chirac
WMD example Preventive war Inspect and Embargo
Prior – rogue regimes pose Prior – regime type not good
major security threats predictor of security threat
Environmental, and Denmark, CU, Ashford US Bush Sapolsky HIV activists
product/drug safety Restrict substances and Allow substances and products
products unless prove no harm unless clear evidence of harm
Prior – strong expectations of Prior – normal expectation of
risks in selected areas safety and usefulness
Export Controls US OFAC EU Allies and Japan
Restrict trade using lower Allow trade unless clear
threshold of risk evidence of risk
Prior – potential risks are high Prior – potential risks are low
Limits on Research Rees “Our Final Hour” MIT View
Limit research in risky areas Allow research in all areas
Prior – can identify some Prior – knowledge is basically
research of benefit to terrorists good, cannot sort in advance
Can we do better than shifting evidentiary thresholds to tilt tradeoffs across Type I or Type II error?
Are priors and presumptions subject to modification? What evidence is deemed to be relevant?
How should relevance of evidence be defined and how should relevant evidence be assessed?
Can we improve the “power curve” through better knowledge assessment methods?