Docstoc

Improving Communications Between Risk Assessors, Risk Managers and Research Scientists

Document Sample
Improving Communications Between Risk Assessors, Risk Managers and Research Scientists Powered By Docstoc
					Improving Communications between Risk Assessors, Risk Managers, and Research Scientists

Focusing on the Risk Decision:

D. Warner North
President and Principal Scientist, NorthWorks, Inc. Consulting Professor, Department of Management Science and Engineering, Stanford University E-mail: northworks@mindspring.com Web: www.northworks.net ORACBA September Risk Forum Monday, September 19, 2005

Warner North

1

9-19-2005

Overview: Two Themes
• Risk analysis is as much an art as a science (W. North,
Tuskegee University Int’l S/PS Symposium August 12, 2005: web: http://compepid.tuskegee.edu/RiskConference/Warner%20No rth.ppt#2)

• Risk assessment and risk management should be conceptually
distinct. These two processes should NOT be separated organizationally and carried out sequentially in time. Rather, there should be ongoing good communication between risk assessors and risk managers. It may be advisable to view the process as an iterative, back-and-forth alternation between analysis and deliberation. (W. North, “Reflections on the Red/Mis-Read Book, 20 Years After,” HERA 9:1145-54, 2003; National Research Council, Understanding Risk, 1996)

Warner North

2

9-19-2005

Quote: Theme 1
Risk analysis provides a useful tool kit for dealing with uncertainty and complexity, but it does not make difficult problems simple, especially where societal values are involved. For the foreseeable future, risk analysis is going to be as much of an art as a science. Skilled practitioners will not be plentiful, especially in accomplishing effective two-way risk communication. -- D. Warner North, from First International Symposium, Risk Assessment for
Veterinary Biologicals, Washinton D.C., December 5, 1994. (Page 921 of Limitations, definitions, principles, and methods of risk analysis,” Rev. sci. tech. Off. int. Epiz. (Scientific and Technical Review of the Office International des Epizooties), 14(4):913-923, 1995

Warner North

3

9-19-2005

Music School: Moscow

Warner North

4

9-19-2005

How Do You Get to … ?

Warner North

5

9-19-2005

The Answer:

Practice! Practice!

Warner North

6

9-19-2005

Vision and Mission Statements: S/PS Risk “Art School”
• Vision:
successful, effective decision support in highly uncertain and complex SPS regulatory decisions
Learn technique, master the media (materials), evaluate your audience, learn from your colleagues, recognize that the profession is highly competitive, maintain your integrity.

• Mission (Tools and Training):

Warner North

7

9-19-2005

Curriculum for SPS Risk “Art School”

• Risk Analysis 101:

Risk 101: Use of probabilities and consequences to describe risk, in context of a regulatory decision. How to achieve good documentation and effective communication.

• Risk Analysis 201: Ongoing and adaptive risk

management through evolving regulation, with a proactive scientific research program in support of decision improvements. Use of value-of-information calculations to guide research, allocate research funds. How to achieve good documentation and effective communication.

Warner North

8

9-19-2005

Risk 101
• Use of probabilities and consequences to describe risk, in
context of a regulatory decision. How to achieve good documentation and effective communication

• Main Tools:
– Systems analysis/modeling: fate and transport – Sensitivity analysis – Assessment of probabilities for most important uncertainties (based on data where available – and/or expert judgment) – More sensitivity analysis – Careful documentation: an easy-to-read short summary, comprehensive and transparent backup, sources of data and judgment specified.
9-19-2005

Warner North

9

Risk 101 Example: Microbes on Mars

Do we infect Mars with terrestrial microbes by landing a spacecraft on its surface?

Warner North

10

9-19-2005

New Methodology for Assessing the Probability of Contaminating Mars
D. W. North, B.R. Judd, and J.P. Pezier, Life Sciences and Space Research XIII, P. A. Sneath, ed., AkademieVerlag, Berlin, pp. 103-109, 1975. “Limitations, Definitions, Principles, and Methods of Risk Analysis,” Risk Assessment for Veterinary Biologicals, special issue, Office International des Epizooties, Scientific and Technical Review, Vol. 14, pp. 913-923, 1995. (on website, http://www.northworks.net/limitations.pdf ) 1974 Final Report to NASA: much longer, available on request

Warner North

11

9-19-2005

Mission Contamination Model Results

Warner North

12

9-19-2005

Mission Contamination Model Marginal Sensitivity Analysis
Probability of Contamination Contamination Model Variables Extreme Intermed. Values Intermed. Extreme

Units: = 10**-6

Nominal: 5.9
Low High

Low Bio-Burden Variables
1. bio External 2. bio Covered 3. bio Encapsulated 2.2 3.2 4,000

Low

NOMINAL

High

High

5.5 8 10,000

11 16 20,000

22 32 40,000

55 80 100,000

5 3.1 5

10.7 20.2 10.4

Release Variables 1. rel Hard Landing Probability 3. rel Newly Exposed/Hard, Encaps 4. rel Implanted, Soft 6. rel VTO/Vibration 9. rel VTO/Erosion, Encaps 0.0004 0.0001 0.0001 0.001 0.00001 0.001 0.0002 0.0002 0.002 0.00002 0.002 0.001 0.001 0.01 0.0001 0.004 0.005 0.005 0.05 0.0005 0.01 0.01 0.01 0.01 0.001 5.2 5.4 5.7 5.4 5.4 9.6 10.9 8.7 11.1 10.9

Warner North

13

9-19-2005

Mission Contamination Model Marginal Sensitivity Analysis -2
Probability of Contamination Contamination Model Variables Extreme Low Transport Variables 1 tra Survive Transit 2 tra Find Water 4 tra Water Deposition 5 tra Stay Lodged 0.001 0.0005 0.00005 0.1 0.002 0.001 0.0001 0.2 0.01 0.005 0.0005 0.5 0.05 0.025 0.0025 0.8 0.1 0.05 0.005 0.9 2.2 1.5 5 5.5 45.2 49.9 15.2 10 Intermed. Low NOMINAL Values Intermed. High Extreme High

Units: = 10**-6

Nominal: 5.9
Low High

Reproduction Variables 1 rep Psychrophilic, Anaerobic 2 rep Availability of Nutrients 0.005 0.01 0.01 0.02 0.05 0.1 0.1 0.2 0.25 0.5 0.6 0.6 29.6 29.6

Warner North

14

9-19-2005

National Academy of Sciences, Viewpoint -1992
“… it is the unanimous opinion of the task group that terrestrial organisms have almost no chance of multiplying on the surface of Mars and in fact have little chance of surviving for long periods of time, especially if they are exposed to wind and to UV radiation.”

---- Space Studies Board, National Research Council, Biological Contamination of Mars, 1992, page 49.

Warner North

15

9-19-2005

When the Uncertainties Matter, and Data Are Not Available

• Elicitation of Expert Judgment:
– History: Bayesian school of probability – Extensive research in social sciences: people are not good at processing information about uncertainty. – Need careful approach: trained elicitors, careful framing of questions, good technique

Warner North

16

9-19-2005

References on Probability: Human Fallibility, Good Assessment Technique
•
• • •
D. Kahneman, Paul Slovic, and A. Tversky, Judgment under Uncertainty: Heuristics and Biases, Cambridge Univ. Press, 1982. M. G. Morgan and M. Henrion, Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Public Policy Analysis, Cambridge Univ. Press, 1990 C. S. Spetzler and C.A. Staël von Holstein, “Probability Encoding in Decision Analysis,” Management Science 22: 340-352, 1975 T. S. Wallsten and D. V. Budescu, “Encoding Subjective Probabilities: A Psychological and Psychometric Review,” Management Science 29:151-173, 1983

•

M. G. Morgan and D. W. Keith, “Subjective Judgments by Climate Experts, Environmental Science and Technology 29(10):468A-476A, 1995
9-19-2005

Warner North

17

Range of Expert Opinion: Climate Change

Source: Morgan and Keith, ES&T, 1995

Warner North

18

9-19-2005

Risk 101, Advancing toward Risk 201: Value of Information
• Even if probability assessments are debatable, so that best
• Two examples:
– Air Quality: Sulfur Oxides and Fine Particulate Matter (Omitted: ask me via e-mail if interested) – Weather Modification on Hurricanes
immediate regulatory decision is not clearly evident, there may be an opportunity for robust conclusions on the value of better information via research.

• Reference on Value of Information:

F. Yukota and K. Thompson, “The Value of Information in Environmental Health Risk Management Decisions: Past, Present, Future,” Risk Analysis 24(3):635-650, 2004
9-19-2005

Warner North

19

Expected Value of Further Information
Concept:

Expected value of further information is the
expected value taken over possible outcomes from research of the further information – of best decision taken based on this new information

less
expected value with best decision taken on the basis of current information (can generalize to expected utility). Standard concept or “tool” in decision analysis textbooks. Example: R.T. Clemen, Making Hard Decisions: An Introduction to Decision Analysis, Duxbury Press, 1991.
Warner North
9-19-2005

20

VOI Example: The Decision to Seed Hurricanes
R.A. Howard, J.E. Matheson, D.W. North

Science, 176, 1191-1202, 1972
http://northworks.net/hurricanes.pdf

New York Times, Thursday, October 3, 2002

• Thousands Seek Safety as Hurricane
Nears Gulf Coast
By JEFFREY GETTLEMAN (NYT) 1308 words Late Edition - Final , Section A , Page 24 , Column 3

•

ABSTRACT - About half million people flee southern Louisiana and Texas with approach of Hurricane Lili, which is heading toward Gulf Coast with winds of more than 140 miles per hour, making it daunting Category 4 storm; map; chart of five most intense hurricanes to hit US since 1928; photos (M) Highways across southern Louisiana and Texas were solid columns of steel today as more than half a million people grabbed their valuables and fled their homes, looking for higher, safer ground before Hurricane Lili hit. The exodus of cars and trucks, some with furniture lashed down on their roofs, began in lowlying areas but quickly spread inland as the storm intensified and threatened to become the worst natural disaster here in decades.
22
9-19-2005

•

Warner North

Value of A Seeding Experiment (Government Responsibility Cost Included)
Choice of Whether to Perform Experiment Results of Experiment Operational Seeding Decision Outcomes Total Cost (millions of dollars)

0.038 0.143

$116.00

$116.00 Perform Experiment
$107.96 0.172 $103.44 $87.83 Do not Perform Experiment $110.92 $110.92 Seed Do Not Seed $116.00 0.392 0.255 $116.00 $103.44 Seed Do Not Seed $116.00

+32 % +16 0 -16 -34 +32% +16 0 -16 -34 +32% +16 0 -16 -34 +32% +16 0 -16 -34

$503.7 248.4 105.0 46.7 16.3 503.7 248.4 105.0 46.7 16.3 503.7 248.4 105.0 46.7 16.3 503.7 248.4 105.0 46.7 16.3

$110.92 Expected Value (M$)

Value = $2.96 million Actual cost = $0.25 million Warner North
23

9-19-2005

Summary of the Value of Additional Information on the Effect of Seeding (Values in Millions of Dollars)
“Standard” Hurricane Used in Analysis of Chapters III and IV Expected Property Damage Without Seeding Espected Value of Perfect Information Expected Value of A Field Experimant Consisting of Two Experimental Seedings Espected Value of Field Experiments, Assuming that Prior Operational Seeding is not Permitted: With Government Responsibilty costs With Government Responsibility Costs Equal to Zero $116 (100%) Single Hurricane Season* $220 All Future Hurricane Seasons (discounted at 7%) $3,142

13.6 (11.8%)

26

370

5.4 (4.7%)

10.2

146

8.8 (7.6%)

16.6

238

12.4 (10.7%)

23.5

335

Considering only the 50% of hurricanes that are assumed to be possible candidates for seeding because of tactical consideration. If all hurricanes are assumed to be candidates for operational seeding, the figures of the last two columns should be doubled.

Warner North

24

9-19-2005

Findings and Recommendations
Findings
1. Current meteorological and economic information indicates that the seeding alternative stochastically dominated the nonseeding alternative

Recommendations
1. The present policy prohibiting seeding any hurricane threatening the U.S. should be rescinded 2. A hurricane modification agency with authority to seed operationally should be established 3. Decision procedures supported by further analysis should be developed 4. Modification experiments should be conducted on an expanded scale to provide a more refined basis for making each operational seeding decision
9-19-2005

2. No firm legal basis for operational seeding appears to exist
3. The decision to seed a particular hurricane should take into account its specific characteristics

4. Resolving meteorological uncertainty on the effect of hurricane modification is worth over $20 millions/year
Warner North

25

Risk 201: Adaptive Management under Uncertainty, with Directed Research
• Most SPS decisions are not isolated decisions at one point in
• Value of information from research:
time, but ongoing issues of managing safety in international trade in plant and animal products What is the expected value from research that might show that a pest in crop C from country A will not be present in export shipments? Or cannot survive and replicate in the climate of a country B that imports crop C from country A? together to assure that the appropriate research is done and properly used?

• How can regulatory authorities in countries A and B work

Warner North

26

9-19-2005

A Success Story: Hass Avocados
Anastrepha ludens fruit fly:
“Not all alien invaders are from outer space” - APHIS

Warner North

27

9-19-2005

Hass Avocado Chronology
•
• • • • • • •
1914: US quarantine of Mexican avocados into the lower 48, under Food Protection Act of 1912. 1994: Mexico proposal to APHIS to amend regulation to allow import of Hass avocados from approved orchards in Michoacán Province. January 1997: Workshop by Harvard Center for Risk Analysis, funded by grant from California Avocado Commission (“CAC”). February 5, 1997: USDA APHIS Final Rule allowing Hass avocados into 19 northern states + DC during November through February. October 1999: Mexico asks for modification: longer time, more states. November 1, 2001: APHIS Final “Expansion” Rule: Adds 12 states, time of Oct 15 – April 15. January 31, 2005: APHIS new Final Rule: all months, all states but California, Florida, Hawaii; add these 3 in two years. May 18, 2005: Litigation resolved. Federal Judge Robert Coyle finds legal challenges by CAC to USDA moot. Decision not appealed, suit not re-filed by CAC.

Warner North

28

9-19-2005

My Viewpoint as an “Expert Witness”
• Testimony in June 2002, August 2002 in support of adequacy of
APHIS pest risk analysis, on behalf of Mexican growers. 31 2005 APHIS Rule: all states, all times of year.

• More testimony, March 2004 supporting what became the Jan • Comments to APHIS on draft 2003 draft pest risk assessment.
– Key points: (1) strength of Mexican research by Dr. Martin Aluja: Hass avocado not a host to Anastrepha fruit flies. (2) Inspection of over 10 million fruit: no pests found. (3) Pest risk draft good -- but could be improved, especially communication aspects and some technical details.

Warner North

29

9-19-2005

Pest Risk Assessment Guidelines
• Reference:
– George M. Gray, Jon C. Allen, David E. Burmaster, Stuart H. Gage, James K. Hammitt, Stanley Kaplan, Ralph L. Keeney, Joseph G. Morse, D. Warner North, Jan P. Nyrop, Alina Stahevitch, and Richard Williams, Principles for Conduct of Pest Risk Analysis: Report of an Expert Workshop, Risk Analysis 18(6):773-780, 1998.

Warner North

30

9-19-2005

Risk 201: Adaptive Management under Uncertainty, with Directed Research
• Most SPS decisions are not isolated decisions at one point in
• Value of information from research:
time, but ongoing issues of managing safety in international trade in plant and animal products What is the expected value from research that might show that a pest in crop C from country A will not be present in export shipments? Or cannot survive and replicate in the climate of a country B that imports crop C from country A? together to assure that the appropriate research is done and properly used?

• How can regulatory authorities in countries A and B work

Warner North

31

9-19-2005

National Academy Press, 1996; Catalog 5138

Warner North

32

9-19-2005

Guidance from Understanding Risk: Analytical-Deliberative Process • Inform the interested and affected parties among the public • Respect differences in values and cultures • Seek flexibility and maintain choices • Seek progress in stages, rather than seeking overall solutions.
– Source: National Research Council, Disposition of HighLevel Waste and Spent Nuclear Fuel: The Continuing Societal and Technical Challenges, National Academy Press, 2001, page 130. http://books.nap.edu/catalog/10119.html

Warner North

33

9-19-2005

Coming Next in the National Academies Series …
• Panel on Public Participation in Environmental Assessment and
Decision Making. Last Panel Meeting, Sept 16-17, 2005. Expected date for report: mid-2006. Web: http://www7.nationalacademies.org/hdgc/Public_Participatio n.html http://qp.nas.edu/QuickPlace/publicparticipation/Main.nsf/h_ Toc/4df38292d748069d0525670800167212/?OpenDocument You may need to register first – see instructions. Then click on “MATERIALS” before “Page.” Then go to TAB 12 for:
D. Warner North and Ortwin Renn, “Decision Analytic Tools and Participatory Decision Processes,” Background “State of the Science” Paper - Tab 12.

• Website for workshop papers from Feb 2005 Workshop:

Warner North

34

9-19-2005

Theme #2: Back to 1983, The Red Book (or “Mis-Read” Book)
National Research Council, Risk Assessment in the Federal Government: Managing the Process, 1983 ( available at www.nas.edu ) 20th Anniversary Special Issue: Human and Ecological Risk Assessment (HERA), Vol. 9, No. 5, August 2003. See W. North paper, other papers, editors’ comments (journal not available on line; copies of my paper available today, or by request)

Warner North

35

9-19-2005

“Red Book” Recommendations
• “Conceptual distinction” ( not separation) between risk
assessment and risk management – and ongoing communication between those responsible for risk assessment and those responsible for risk management.

• Risk assessments should be published as publicly available
documents

• These documents should be reviewed by a group of scientists
from outside the agency preparing the document

• Results of this peer review should be published and available

Warner North

36

9-19-2005

Risk Assessment: “Figure 1”

Source: National Research Council, Risk Assessment in the Federal Government: Managing the Process (“Red Book”), 1983, page 21 Warner North
37
9-19-2005

1983 Red/Mis-Read Book: What Went Right – and Wrong?
• Right:
– Formalization of Risk Assessment – Figure 1/Chapter 1 endorsed by National Academies … – Guidelines for risk assessment: OSTP for carcinogens, EPA for carcinogens, …
– EPA Administrator William Rickelshaus’ speeches/papers: “Academy told us to separate risk assessment from risk management …” Committee did NOT say this. – More general problem: agency leaders want appearance of objective science, carried out via standardized process

• Wrong:

• PLEASE READ THE 1983 BOOK!
Warner North

Also 1996 book, and maybe also the 1994 book, Science and Judgment in Risk Assessment.
9-19-2005

38

Optimize the Tool Kit? Optimize the Risk Team!
Risk analysis provides a useful tool kit for dealing with uncertainty and complexity, but it does not make difficult problems simple, especially where societal values are involved. For the foreseeable future, risk analysis is going to be as much of an art as a science. Skilled practitioners will not be plentiful, especially in accomplishing effective two-way risk communication. -- D. Warner North, from First International Symposium, Risk Assessment for Veterinary
Biologicals, Washinton D.C., December 5, 1994. (Page 921 of Limitations, definitions, principles, and methods of risk analysis,” Rev. sci. tech. Off. int. Epiz. (Scientific and Technical Review of the Office International des Epizooties), 14(4):913-923, 1995

Prepare! Practice! Improve with practice! Be as good as you can be in carrying out the art and the science of risk analysis – especially interactive communication with management (and stakeholders, research scientists).
-- D. Warner North, International Conference on Sanitary and Phytosanitary Risk Assessment Methodology August 9, 2005, Expanded for September 19 talk.

Warner North

39

9-19-2005

Concluding Quote
If a man will begin with certainties he shall end in doubts, but if he will be content to begin with doubts, he shall end in certainties.
-- Francis Bacon The Advancement of Learning, 1605.

Warner North

40

9-19-2005


				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:62
posted:5/23/2008
language:English
pages:40
january january
About