Uncertainty
Introduction to Structured Decision Making
Module 14 — Uncertainty
Outline
What is uncertainty? Why is it important?
Classifying uncertainty
Estimating uncertainty
Evaluating uncertainty
• Sensitivity analysis
What is Uncertainty?
Our inability to precisely express some elements of the decision analysis
This could be uncertainty in the objectives
• An inability to value the outcomes
Or uncertainty in the predictions
• Because something is not known perfectly (how many Mus musculus are
currently in this building?)
• Or is unknown (will a Category 5 hurricane hit the US this year?)
Reducible vs. irreducible uncertainty
• Some uncertainties can be resolved if we collect enough data
• Others cannot be resolved until after an event occurs
Why does Uncertainty Matter?
Uncertainty might affect the decision we make
• That is, we might choose a different alternative depending on the
predictions or the valuation of the outcomes
A good decision does not guarantee a favorable outcome, but it accounts for
uncertainty so as to provide the best chance of a favorable outcome
2011 CSP3171, Module 14 – 1 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
A Good Decision?
Favorable Outcome
Chance
Events
Unfavorable Outcome
Good Decision
Decision
Poor Decision
Favorable Outcome
Chance
Events
Unfavorable Outcome
Types of Uncertainty
Linguistic
Epistemic
Aleatory
Linguistic Uncertainty
Vagueness
• Language permits borderline cases
Context dependence
• Meaning depends on context
Ambiguity
• Words can have more than one meaning
Underspecificity
• Unwanted generality in statements
Indeterminacy of terms
• Understanding of terms changes over time
(From: Regan et al. 2002. A taxonomy and treatment of uncertainty
for ecology and conservation biology. Ecol Appl 12:618-628.)
2011 CSP3171, Module 14 – 2 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
Epistemic Uncertainty
Reducible
Sources
• Measurement error
• Systematic error (bias)
• Model uncertainty
• Parametric uncertainty
• Structural uncertainty
• Subjective judgment
Aleatory Uncertainty
Practically irreducible
Environmental stochasticity
• Natural variation in systems over space and time that is difficult to predict
Demographic stochasticity
• The chance events that happen to individuals
Exercise: Classify the Uncertainty
As specifically as possible, identify the type of uncertainty represented by the following
examples:
Counts of bats exiting caves, used as population estimates, don’t include bats
that depart from smaller, alternate exits
____________________________________________________
An entomologist uses the phrase “highly fecund” when talking to a mammalogist
____________________________________________________
The size of tree rings varies from year to year
____________________________________________________
Botanist A and Botanist B disagree on the likelihood of extinction of Blowout
Penstemon
____________________________________________________
A study of fox survival reports the standard error of estimates
____________________________________________________
A paper reports reproductive success in bobolinks but does not report separately
on egg, nestling, and fledgling stages
____________________________________________________
Two models of population dynamics in introduced whooping cranes differ on
whether an Allee effect exists
____________________________________________________
2011 CSP3171, Module 14 – 3 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
Epistemic Uncertainty
Uncertainty that arises from our incomplete knowledge about our system
• This is, theoretically, reducible
Focus on 2 forms (both contained within model uncertainty)
• Parametric uncertainty
• Structural uncertainty
Parametric Uncertainty
Genesis
• Sampling
• Subjective judgment (if using experts)
Expressed as
• Probability distributions for parameters
Special issues
• Bias
Parametric Uncertainty
3
Model R2 K (est.)
Linear (MLE) 51% 11.3
Recruitment
95% CE 34% 9.6
2
95% CE 34% 17.2
1
0
0 5 10 15
Density
2011 CSP3171, Module 14 – 4 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
BLVU rmax (and its uncertainty)
600
Convolution (iid) of:
~ unid (4,6)
median 10.6%
~ unid (20,30)
500 p ~ uni (.84, .99)
b ~ uni (.645, .79)
s0 ~ uni (.7, .9)
s1 ~ logit-norm (.76, .92)
400
Frequency
60% CI
(6.9, 14.3)
300
200
100 95% CI
(2.1, 19.2)
0
–5 % 0 5% 10 % 15 % 20 %
rmax
[Optional instructor-designed example]
2011 CSP3171, Module 14 – 5 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
Structural Uncertainty
Genesis
• Alternative ecological hypotheses
• Different stakeholder preferences
• Different intuitive experience
Subjective judgment may be involved here as well)
Expressed as
• Alternative models
Special issues
• Weights associated with alternative models
Florida Whooping Cranes
Model CR WH CR = Captive-Reared
Surv. Prod. Surv. Prod. WH = Wild-Hatched
1 (Worst Case) CR CR CR CR AWB = Aransas-Wood Buffalo
2 (Best Case) CR CR AWB AWB
3 (Allee) BT CR CR AWB AWB
2011 CSP3171, Module 14 – 6 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
[Optional instructor-designed example]
2011 CSP3171, Module 14 – 7 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
Estimating Uncertainty
Estimating Probabilities
Empirical methods
• Repeated direct observation of the event in question
• Decomposition of the event into component events (e.g., event trees), with
data to inform the component events
• Modeling, with propagation of uncertainty (e.g., PVA)
Event Tree: Successful Burn
Yes Yes Yes
Fire Fire
takes hot
Crew
avail- Burn!
Yes No
able No No
Yes
Rx met
in burn No Yes Yes
window Crew Fire Fire
Yes avail- takes hot
able
No
Rx met No No
No
No again?
No
No
Estimating Probabilities
Subjective methods (see Module 13)
• Direct elicitation from experts
• Methods for discrete and continuous distributions
Estimating Parametric Uncertainty
Likelihood profile
Bayesian posterior distribution
Subjective elicitation
2011 CSP3171, Module 14 – 8 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
Incubator Performance
After 2008 data
Probability Density
After 2007 data
-1.0 -0.5 0.0 0.5 1.0
Difference (Petersime - Kuhl)
[Optional instructor-designed example]
2011 CSP3171, Module 14 – 9 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
Estimating Structural Uncertainty
Ecological theory
• Is the genesis for the different structures
Statistical techniques
• Can be used to estimate parameters of different models
Model selection techniques
• Can then be used to provide initial weights for the models
Pintail Winter Survival
1.0
Additive model
Winter Survival Rate
0.9 Average point
(P-bar,s-bar)
0.8
Compensatory model
0.7
0 5 10 15
Post-harvest Population Size (millions)
[Optional instructor-designed example]
2011 CSP3171, Module 14 – 10 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
Evaluating Uncertainty
Sensitivity Analysis
Examine how the optimal decision and the expected performance are affected by
• Assumptions
• Structural & parametric uncertainty
• Probabilities
• Magnitude of uncertainty
• Weights on objectives
• Utility
• The problem framing itself
Ask whether the decision is robust to uncertainty
• If not, consider revising aspects of the problem
Emphasis
Examine how
• the optimal decision and
• the expected performance
are affected by…
That is, would your decision change?
Sensitivity Analysis Methods
Bracketing over range of uncertainty
• e.g., tornado diagrams
Gradient methods (derivatives)
• e.g., sensitivity & elasticity in matrix models
Simulation
Scenario exploration
2011 CSP3171, Module 14 – 11 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
Do the Differences Matter?
Tornado Diagram for BLVU
1st-yr survival
Adult survival
Age at first breeding
Juvenile survival
Fecundity
Breeding propensity
Age of senescence
4% 6% 8% 10% 12% 14% 16%
Growth Rate
2011 CSP3171, Module 14 – 12 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
[Optional instructor-designed example]
Decision Tree
70,000
Fry
Yes p = 0.8
Does
it
work? No p = 0.2
EV = 58K 10,000
Fry
Yes
Add new technology
to a hatchery? Spreadsheet
No
EV = 40K 40,000
Fry
2011 CSP3171, Module 14 – 13 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
Spreadsheet
Values Expected Values Decision
Yes &
p(tech. Yes & doesn't Don't
Scenario works) works work No Adopt Adopt
Baseline 0.8 70000 10000 40000 58000 40000 Adopt
Don't
Prob 1 0.2 70000 10000 40000 22000 40000 Adopt
Don't
Prob 2 0.4 70000 10000 40000 34000 40000 Adopt
Prob 3 0.6 70000 10000 40000 46000 40000 Adopt
Prob 4 0.8 70000 10000 40000 58000 40000 Adopt
Prob 5 1 70000 10000 40000 70000 40000 Adopt
Don't
Prob 6 70000 10000 40000 10000 40000 Adopt
Don't
High Value 1 0.8 40000 10000 40000 34000 40000 Adopt
Don't
High Value 2 0.8 45000 10000 40000 38000 40000 Adopt
High Value 3 0.8 50000 10000 40000 42000 40000 Adopt
High Value 4 0.8 55000 10000 40000 46000 40000 Adopt
High Value 5 0.8 60000 10000 40000 50000 40000 Adopt
Low Value 1 0.8 70000 0 40000 56000 40000 Adopt
Low Value 2 0.8 70000 5000 40000 57000 40000 Adopt
Low Value 3 0.8 70000 10000 40000 58000 40000 Adopt
Low Value 4 0.8 70000 15000 40000 59000 40000 Adopt
Low Value 5 0.8 70000 20000 40000 60000 40000 Adopt
Current Value 1 0.8 70000 10000 45000 58000 45000 Adopt
Current Value 2 0.8 70000 10000 50000 58000 50000 Adopt
Current Value 3 0.8 70000 10000 55000 58000 55000 Adopt
Don't
Current Value 4 0.8 70000 10000 60000 58000 60000 Adopt
Don't
Current Value 5 0.8 70000 10000 70000 58000 70000 Adopt
2011 CSP3171, Module 14 – 14 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
SMART Spreadsheet
Alternatives
Normalized & Weighted
Consequence Matrix
Scores
Status Minor Major Status Minor Major
Objectives Goal Units
Quo Rep Rep
Wt
Quo Rep Rep
Cost Min $M 2 2 13 .25 .25 .25 0
Environ Benefits Max 0 - 10 1 3 10 .50 0 .11 .5
Disturbance Min 0 - 10 0 1 7 .25 .25 .21 0
Final
0.5 0.58 0.5
Score:
Normalized score = (sij – min(sj)) / (max(sj) – min(sj))
Spreadsheet
Spreadsheet
Class Exercise
Using the rabbit example (Rabbit v2011.xls) with the Allee effect model, explore the
sensitivity of the long-term population size, N(21), to uncertainty in the initial population
size (cell C2, range 800-3000), adult survival (cell C3, range 0.65-0.72), and juvenile
survival (cell C4, range 0.1-0.2). Note that each change of parameters may require
recalculation of the optimal strategy. Where would you invest monitoring effort?
2011 CSP3171, Module 14 – 15 USGS & USFWS-NCTC
Uncertainty
Introduction to Structured Decision Making
Summary
Uncertainty is one of the most challenging impediments to decision-making
An informed approach requires
• Identification of uncertainty
• Estimation of uncertainty
• Evaluation of uncertainty
• Strategies for decisions in the face of uncertainty…
Module developed by:
Sarah J. Converse, USGS Patuxent Wildlife Research Center
Michael C. Runge, USGS Patuxent Wildlife Research Center
James E. Lyons, USFWS Division of Migratory Bird Management
2011 CSP3171, Module 14 – 16 USGS & USFWS-NCTC