# Ensemble Forecasting, Forecast Calibration, and Evaluation - PowerPoint

Document Sample

```					Ensemble Forecasting, Forecast
Calibration, and Evaluation

Tom Hamill
tom.hamill@noaa.gov
www.cdc.noaa.gov/people/tom.hamill
1
Topics
• Current state-of-the-art in ensemble
forecasting (EF), and where are we headed?
• Brief review of EF evaluation techniques
• Deficiencies in current approaches for
estimation/calibration/evaluation of EFs
• New EF calibration techniques.
• Methods for summarizing probabilistic
information for end users and forecasters
(time permitting).

2
What does it take to get a ―perfect‖
probabilistic forecast from an
ensemble?

• IF you start with an ensemble of initial
conditions that samples the distribution of
analysis uncertainty, and
• IF your forecast model is perfect (error growth
only due to chaos), and
• IF your ensemble is infinite in size,
• THEN probabilistic forecast ―perfect.‖

… Big IFs                        3
Examining the IFs:
(1) Do we sample the distribution of
analysis uncertainty?
NCEP SREF rank
histogram, 39-h forecast

Not enough spread in the forecasts. Consequently, focus in 1st-generation
EF systems is to design initial perturbations that will grow different from
each other quickly, not on sampling analysis-uncertainty distribution. 4
The Breeding Technique (NCEP)

Breeding takes a pair of forecast
perturbations and periodically
rescales them down and adds
Wang and Bishop showed that
them to the new analysis state.
a larger ensemble formed from
Perturbations only grossly reflect
independent pairs of bred members
analysis errors.
will be comprised of pairs that have 5
almost identical perturbation structures
Singular Vectors (ECMWF)
Singular vectors (SVs)
here indicate the field of
perturbations that
are expected to grow the
most rapidly in a short-
range forecast.
The SV structure depends
upon a choice of norm.

Here, these leading three
SVs have magnitude only
in one area over the globe.

6
Singular vectors, continued
Case 1 (9 Jan 1993)             Case 2 (8 Feb 1993)

final

initial*20

SVs tend to have their initial perturbations in the mid-troposphere, little
amplitude near surface or tropopause. If they’re meant to sample analysis
errors, are analysis errors really near-perfect at these levels?

As SVs evolve, they grow to have amplitudes aloft and near the surface.

Expect SV ensemble forecast spread to be unrepresentative in the early
hours of the forecast (e.g., spread of EFs too small near the surface). 7
A better way to construct initial conditions?
Ensemble-based data assimilation

Observations
and error stats

First Guess 1                     Analysis 1
Ensemble
First Guess 2                     Analysis 2
Data
Assimilation

First Guess N                     Analysis N

An ensemble of forecasts is used to define the error statistics
of the first-guess forecast. An ensemble of analyses are
produced. If designed correctly, they’re sampling the
analysis uncertainty and can be used for initializing EFs and
are lower in error.                                        8
Why might initial conditions from
ensemble data assimilation be
more accurate?

Flow-dependent error statistics for the first guess,
improve the blending of observations and forecasts.

9
Example: 500 hPa height analyses assimilating only SfcP obs

Full NCEP-NCAR
Reanalysis (3DVar)
(120,000+ obs)

Black dots show
Ensemble Filter                            pressure ob
―EnSRF‖                                    locations
(214 surface
pressure obs)                              RMS = 39.8 m

Optimal
Interpolation                              RMS = 82.4 m
(214 surface                               (3D-Var worse)
pressure obs)
10
Perturb the land surface in EFs?

11
Examining the IFs:
(2) Is the forecast model anywhere
close to perfect?
• Model errors
–   due to limited resolution (truncation)
–   due to parameterizations
–   due to numerical methods choices
–   etc.
• Manifestations
– biases, especially near the surface, and in
precipitation
– slow growth of forecast differences among
ensemble members due to coarse grid spacing,
less scale-interaction.
12
Dealing with model errors
(1) Better models (4-km,     (2) Introduce stochastic element into model
60-h WRF for Katrina)

(3) Multi-model ensembles
(4) Calibration

13
Examining the IFs:
(3) Can we run a nearly
infinite ensemble?

Clearly not; CPU availability finite.
-- large ensemble «—» low resolution
(small sampling error, larger model error).
-- small ensemble «—» higher resolution
(large sampling error, smaller model error).
-- optimal size/resolution tradeoff may be different
for different variables (large ensemble for 500 hPa
anomaly correlation, smaller ensemble for fine-
scale precipitation events).
14
Probabilistic Forecast Verification

Relative Economic Value
Many well-established
methods now; won’t
review here. Often
problem-specific
diagnostics are needed
to understand specific
errors in ensemble
forecasts.

15
A tool for exploring calibration issues:
CDC’s ―reforecast‖ data set
• Definition: a data set of retrospective numerical forecasts using
the same model to generate real-time forecasts
• Model: T62L28 NCEP global forecast model, circa 1998
(http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/refcst for details).
• Initial States: NCEP-NCAR reanalysis plus 7 +/- bred modes
(Toth and Kalnay 1993).
• Duration: 15 days runs every day at 00Z from 1978/11/01 to now.
(http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/refcst/week2).
• Data: Selected fields (winds, hgt, temp on 5 press levels, precip,
t2m, u10m, v10m, pwat, prmsl, rh700, heating). NCEP/NCAR
reanalysis verifying fields included (Web form to download at
http://www.cdc.noaa.gov/reforecast).

16
Why reforecast? Bias structure can be difficult
to evaluate with small forecast samples

17
New calibration technique:
Reforecasting with analogs

18
Analog calibration results

resolution - reliability
BSS =
uncertaint y

19
What are appropriate
methods for summarizing
probabilistic
information for end
users? Forecasters?

   Box shows 25-75% range
   Whiskers show full range (or
95% after calibration)
   Central bar shows median

Confident cold spell

20
Methods for summarizing
probabilistic information, continued

Probability Maps

21
What isn’t very helpful to the
end forecaster…
Spaghetti Diagram   Does widely spread
spaghetti indicate
unpredictability or
slack gradient?

552
558
6 dm spread
564
570

552

558

6 dm spread
564

570                  22
Where EF is headed
• Use of ensembles in data assimilation / better
methods for initializing EFs
• Sharing of data across countries (TIGGE) to
do multi-model ensembling
• Higher-resolution ensembles with model-
error parameterizations
• Improved calibration using reforecasts
• Increased use by sophisticated users (e.g.,
coupling into hydrologic models)

23
References and notes
•   Page 3: Getting a “perfect” probabilistic forecast from an ensemble

Ehrendorfer, M., 1994: The Liouville equation and its potential usefulness for the prediction of forecast skill. Part 1:
Theory. Mon. Wea. Rev., 122, 703-713.

•   Page 4: Do we sample the distribution of analysis uncertainty?

Rank histograms from NCEP’s SREF web page, http://wwwt.emc.ncep.noaa.gov/mmb/SREF/SREF.html

•   Page 5: The breeding technique

Toth, Z. and E. Kalnay, 1997: Ensemble forecasting at NCEP and the breeding method. Mon. Wea. Rev., 125, 3297-
3319.

•   Pages 6-7: Singular vectors.

Buizza, R., and T. N. Palmer, 1995: The singular-vector structure of the atmospheric global circulation. J. Atmos.
Sci., 52, 1434-1456.

Barkmeijer, J. M. Van Gijzen, and F. Bouttier, 1998: Singular vectors and estimates of the analysis-error covariance
metric. Quart. J. Royal Meteor. Soc., 124, 1697-1713.

24
•   Pages 8-10: Ensemble-based data assimilation

Hamill, T. M., 2005: Ensemble-based atmospheric data assimilation. To appear in Predictability of Weather and Climate,
Cambridge Press, T. N. Palmer and R. Hagedorn, eds. Available at
http://www.cdc.noaa.gov/people/tom.hamill/efda_review5.pdf .

Whitaker, J. S., G. P. Compo, X. Wei, and T. M. Hamill, 2004: Reanalysis without radiosondes using ensemble data
assimilation. Mon. Wea. Rev., 132, 1190-1200.

•   Page 11: Perturbing the land surface

Sutton, C., T. M. Hamill, and T. T. Warner, 2005: Will perturbing soil moisture improve warm-season ensemble forecasts? A
proof of concept. Mon. Wea. Rev., in review. Available from http://www.cdc.noaa.gov/people/tom.hamill/land_sfc_perts.pdf .

•   Pages 12-13 Model errors

WRF forecast from http://wrf-model.org/plots/realtime_hurricane.php (try 2005-08-27, and rain mixing ratio, 60-h forecast)

Stochastic element picture from Judith Berner’s presentation at the ECMWF workshop on the representation of sub-grid
processes using stochastic-dynamic models. http://www.ecmwf.int/newsevents/meetings/workshops/2005/Sub-
grid_Processes/Presentations.html .

Hagedorn, R., F. J. Doblas-Reyes, and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in
seasonal forecasting – I. Basic concept. Tellus, 57A, 219-233. (multi-model ensemble picture)

Hamill, T. M., J. S. Whitaker, and S. L. Mullen, 2005: Reforecasts, an important data set for improving weather predictions,
Bull. Amer. Meteor. Soc., in press. Available at http://www.cdc.noaa.gov/people/tom.hamill/reforecast_bams4.pdf

•   Page 14: Can we run a nearly infinite ensemble?

Mullen, S. L., and R. Buizza, 2002: The impact of horizontal resolution and ensemble size on probabilistic forecasts of
precipitation by the ECMWF ensemble prediction system. Wea. Forecasting, 17, 173-191.

25
•   Page 15, probabilistic forecast verification

Hamill, T. M., 2001: Interpretation of rank histograms for verifying ensemble forecasts. Mon. Wea. Rev., 129, 550-
560.

Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences. Cambridge Press, 467 pp (section 7.4).

Mason, I. B., 1982: A model for the assessment of weather forecasts. Aust. Meteor. Mag., 30, 291-303. (for the
relative operating characteristic).

Richardson, D. S. , 2000: Skill and relative economic value of the ECMWF ensemble prediction system. Quart. J.
Royal Meteor. Soc., 126, 649-667.

•   Pages 16-19: Calibration using reforecasts.

Hamill, T. M., J. S. Whitaker, and S. L. Mullen, 2005: Reforecasts, an important data set for improving weather
predictions, Bull. Amer. Meteor. Soc., in press. Available at
http://www.cdc.noaa.gov/people/tom.hamill/reforecast_bams4.pdf

•   Page 20: Summarizing probabilistic information for end users:

From Ken Mylne, UK Met Office; also ECMWF newsletter 92, available from www.ecmwf.int .

•   Page 21: Summarizing continued, probability maps

From www.cdc.noaa.gov/reforecast/narr

•   Page 22: What isn’t very helpful (spaghetti diagrams)

From http://www.cdc.noaa.gov/map/images/ens/ens.html .
26

```
DOCUMENT INFO
Shared By:
Categories:
Stats:
 views: 17 posted: 2/17/2010 language: English pages: 26