Taking Uncertainty Into Account:
Bias Issues Arising from Uncertainty in Risk Models
John A. Major, ASA
Guy Carpenter & Company, Inc.
Example: Exponential Distribution
N=20 observations
T = sample mean; l=1 true mean
MLE EP curve:
ˆ
Q( x) exp x
T
q-exceedance point (PML, VaR)
ˆ
X T ln( q )
q
X.01 = 4.605 actual
Sampling Distribution of T
Estimated PDFs
Client Questions
What is the 1 in 100-yr PML (1% VaR)?
What is probability of exceeding 4.605?
Can you give me an EP curve to answer
these and similar questions?
Does sampling error affect the answer?
Can I get unbiased answers?
3 Kinds of Bias
ˆ
“dollar” or X-bias: E X q vs X q
the average of PML dollar estimates
ˆ
“probabilistic” or P-bias: E Q X q vs q
the average true exceedance probability of
estimated PML points
ˆ
“exceedance” or Q-bias: E Q X q vs q
the average estimated exceedance
probability
Exponential MLE is X-unbiased
ET l
E X q E T ln( q) l ln( q) X q
ˆ
Exponential MLE is X-unbiased
Exponential MLE is P-biased
ˆ
E Q X q q for small q
Expected actual risk is greater than
nominal
Uncertainty increases risk!
Exponential MLE is P-biased
Correcting for P-bias
Predictive distribution
“Prediction interval” in regression
Mix randomness and uncertainty
integrate model pdf over parameter
distribution
Exponential model: Q( x) exp x
T
Predictive result: n
x
Q ( x ) 1
T n
Predictive vs. Model Density
Which to use?
MLE curve is X-unbiased
no uncertainty adjustment, but...
on average, gets right $ answer
Predictive curve is P-unbiased
“takes uncertainty into account” and...
on average, reflects true exceedance pr
But they disagree...
and it gets worse...
Exponential MLE is Q-biased
E QX q q for small q
ˆ
Expected estimated risk is greater than
the true risk (at the specified threshold)
Uncertainty now causes risk to be
overstated!
Exponential MLE is Q-biased
Correcting for Q-bias
Minimum Variance Unbiased Estimator
standard procedure in classical statistics
Rao-Blackwell Theorem
Expectation of unbiased estimator,
conditional on sufficient statistic
Exponential model: Q( x) exp x T
MVUE result: n 1
x
Q ( x ) 1
T n
MVUE vs. Model Density
Paradox
Say we get an estimated T=1 (correct)
MLE says X.01=4.605, Pr{X>4.605}=1%
Predictive: X.01=5.179 is p-unbiased
risk is greater than MLE answer because
impact of uncertainty
MVUE: Pr{X>4.605}=.69% is q-unbiased
risk is less because MLE tends to overstate
exceedance probability
How the Paradox Arises
Conclusions
Uncertainty induces bias in estimators
Biases operate in different directions
depends on the question being asked
There is no monolithic “fix” for taking
uncertainty into account
Predictive distribution fixes p-bias,
while making q-bias worse
Recommendations
First: Show modal estimates (MLE etc.)
Second: Show effect of uncertainty
Keep uncertainty distinct from randomness
Sensitivity testing w.r.t. parameters
Confidence intervals on estimators
Third: Adjust for bias only as necessary
Carefully attend to the question asked
Advise that bias adjustment is equivocal