Get documents about "Annex -
Shared by: wuyunqing
-
Stats
- views:
- 1
- posted:
- 9/4/2011
- language:
- English
- pages:
- 11
Document Sample
WORLD METEOROLOGICAL ORGANIZATION CBS-DPFS/CG-FV/Doc. 4(3)
COMMISSION FOR BASIC SYSTEMS (14.I.2011)
OPAG on DPFS _______
COORDINATION GROUP ON FORECAST
Agenda item : 4
VERIFICATION
Montreal, Canada, 24 – 27 January 2011 ENGLISH ONLY
Use and presentation of CBS scores at ECMWF
(Submitted by David Richardson)
Summary and purpose of document
This document summarises the ways that ECMWF uses the
verification scores that are exchanged between the participating
global NWP centres.
Action Proposed
The meeting is invited to note the use of CBS verification scores at ECMWF and to consider this
during its discussion of the requirements for graphical displays on the LC-DNV web site.
CBS-DPFS/CG-FV/Doc. 4(3), p. 2
Use and presentation of CBS scores at ECMWF
1. Introduction and summary
ECMWF maintains a comprehensive set of verification diagnostics for its deterministic forecasting
system. A range of measures is used to assess different aspects of forecast performance and
different products. Performance is monitored continually and results are reviewed monthly and 3-
monthly at internal ECMWF meetings. Verification results are reported annually to ECMWF’s
external Technical Advisory Committee and Scientific Advisory Committee. The complete set of
annual results is available in ECMWF Technical Memoranda on “Verification statistics and
evaluations of ECMWF forecasts”, downloadable from
http://www.ecmwf.int/publications/library/do/references/list/14
It is very helpful for reporting purposes to select a limited number of headline scores to give an
overall view of progress (it should be emphasised that these headline scores are intended to be an
indicative summary of long-term trends in performance). The main headline score for the ECMWF
deterministic forecast is the anomaly correlation (ACC) for 500 hPa height, evaluated over the
extra-tropics (northern and southern hemispheres and Europe). Progress in synoptic-scale skill for
the deterministic forecast is monitored by the day at which the anomaly correlation (ACC) for 500
hPa height drops below 0.6.
Comparison of ECMWF verification results with those from other centres is a regular part of the
ECMWF verification procedure. These comparisons are helpful in assessing trends, putting
changes in ECMWF scores in the context of the changes at other centres. They are also valuable
in assessing the impact of differences in atmospheric conditions when comparing scores between
different years. Some examples of the use of CBS scores at ECMWF are shown below.
ECMWF computes scores following the current CBS procedures and also in parallel using the new
procedures developed by the CG-FV and agreed at CBS in November 2010. Some comments on
the comparison of the new and old procedures are given as well as on the expected benefits from
the proposed additional scores.
2. Examples of presentation of CBS score at ECMWF
Figure 1 shows time series of the CBS exchanged scores over the northern extratropics for both
500 hPa geopotential height and mean sea level pressure (MSLP). For both parameters, medium-
range forecast errors for all models were lower in winter 2009-10 than in winter 2008-09. ECMWF
maintains a lead over the other centres. Overall, however, the difference in performance between
centres is decreasing. This can be seen particularly clearly in Figure 2 which shows a running 12-
month mean of the monthly scores for 500 hPa geopotential height.
CBS scores include verification against analyses and against radiosonde observations. Both sets
of scores are regularly plotted at ECMWF. Figure 3, shows both 500 hPa geopotential height and
850 hPa wind errors averaged over 12 months.
The comparison for the tropics is summarised in Figure 4 (verification against analyses) and Figure
5 (verification against observations). When verified against the centres’ own analyses, the UK Met
Office has had the lowest short-range errors since mid-2005, while at day 5 ECMWF and the Met
Office performances are similar. The errors of the JMA forecast system have steadily decreased
over several years and are now comparable with those of the Met Office model at both short and
medium ranges. In the tropics, verification against analyses (Figure 4) is very sensitive to the
analysis, in particular its ability to extrapolate information away from observation locations. When
CBS-DPFS/CG-FV/Doc. 4(3), p. 3
verified against observations, the ECMWF, Met Office and JMA models have very similar short-
range errors. In the context of this meeting, it is worth noting the large increase in 850 hPa wind
error against analysis (at day 1) in the Canadian forecasts from 2006 and sudden drop in early
2009 (Figure 4). These are related to the verification procedure and do not reflect differences in
model performance. This does, however, demonstrate the importance of consistent verification
methodology, when comparing forecasts from different centres. This specific issue was discussed
during the first meeting of the CG-FV in November 2009.
ECMWF’s main headline score, ACC for 500 hPa height, was exceptionally good for both Europe
and the northern hemisphere in February 2009, being substantially higher than in any previous
month. This was the first month after a substantial increase in resolution of the ECMWF forecasting
system (from 25km to 16km horizontal grid). Figure 6 compares ACC scores for February 2009
with those for February 2008 for ECMWF and for other centres. This shows that the large increase
in skill for February 2009 was also obtained by the other centres and suggests that the main
reason for the exceptional scores is more likely to be the unusually anomalous atmospheric
conditions in February 2009 than the upgrade to the ECMWF system (a fact confirmed by further
investigation). Note that in this case, the ACC scores for NCEP and the Met Office (UKMO) were
computed at ECMWF so that the same climatology was used for all centres (verification was
against each centres own analysis).
3. Implementation of revised CBS verification procedures
ECMWF has begun to compute scores using the revised CBS verification procedures. These are
produced in parallel with the current operational scores. The new scores are used internally but not
yet disseminated externally. Figure 7 shows an example of the effect of the change in procedure
on the main ECMWF headline score, ACC for 500 hPa height over the northern hemisphere
extratropics. The changes in the new verification procedure are
verification on a 1.5° grid instead of the current 2.5°
truncation of the spectral fields to the appropriate spectral resolution (T120); (previously
there was no truncation of the ECMWF fields)
use of up-to-date climatology, i.e. ERA Interim for a fixed 20 year period (1989-2008), for
ACC
For the verification of 500 hPa height, the only substantial impact is the change of climatology.
Until now ECMWF has used an old climatology prepared in the 1970s. The change to the up-to-
date ERA-Interim climatology gives overall lower ACC as shown in Figure 7. Since the reduction is
consistent over recent years it does not have a significant impact on the perceived long-term rate
of progress of the deterministic forecast.
The new procedures also recommend the exchange of additional verification measures, including
rms forecast and analysis anomalies and standard deviation of forecast and analysis fields. These
are very valuable complements to the ACC and rms error. An example is shown in Figure 8 for 500
hPa height over the extra-tropical northern hemisphere for ECMWF and Met Office (scores for both
centres have been computed at ECMWF). The standard deviations of the analysis anomalies are
overall very similar; there is more difference in the forecast anomalies which for recent years are
lower for the Met Office system than for ECMWF, which stays closer to the analysis values. The
additional information from these measures can help in the interpretation of the basic rms scores
(Figure 1).
CBS-DPFS/CG-FV/Doc. 4(3), p. 4
JMA 12utc T+144
CMC 00utc T+144
Verification to WMO standards UKMO 12utc T+144
NCEP 00utc T+144
Mean sea level pressure
ECMWF 12utc T+144
Root mean square error of forecast M-F 00utc T+48
N Hemisphere Lat 20.0 to 90.0 Lon -180.0 to 180.0 JMA 12utc T+48
CMC 00utc T+48
UKMO 12utc T+48
NCEP 00utc T+48
ECMWF 12utc T+48
10
9
8
7
6
5
4
3
2
1
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
JMA 12utc T+144
CMC 00utc T+144
Verification to WMO standards UKMO 12utc T+144
NCEP 00utc T+144
Mean sea level pressure
ECMWF 12utc T+144
Root mean square error of forecast M-F 00utc T+48
N Hemisphere Lat 20.0 to 90.0 Lon -180.0 to 180.0 JMA 12utc T+48
CMC 00utc T+48
UKMO 12utc T+48
NCEP 00utc T+48
ECMWF 12utc T+48
10
9
8
7
6
5
4
3
2
1
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Figure 1: WMO/CBS exchanged scores from global forecast centres. RMS error over northern
extratropics for 500 hPa geopotential height (top) and MSLP (bottom). In each panel the upper
curves show the 6-day forecast error and the lower curves show the 2-day forecast error. Each
model is verified against its own analysis. JMA = Japan Meteorological Agency, CMC = Canadian
Meteorological Centre, UKMO = the UK Meteorological Office, NCEP = U.S. National Centers for
Environmental Prediction, M-F = Météo-France.
CBS-DPFS/CG-FV/Doc. 4(3), p. 5
ECMWF 00utc T+48
ECMWF 00utc T+144
Time series curves JMA 12utc T+48
500hPa geopotential JMA 12utc T+144
Root mean square error of forecast CMC 00utc T+48
CMC 00utc T+144
NH Extratropics Lat 20.0 to 90.0 Lon -180.0 to 180.0
UKMO 00utc T+48
Moving average (12) wmo_an od
UKMO 00utc T+144
00UTC,verifying NCEP 00utc T+48
NCEP 00utc T+144
m
80
70
60
50
40
30
20
10
2003 2004 2005 2006 2007 2008 2009 2010
Figure 2: WMO/CBS exchanged scores from global forecast centres. RMS error over northern
extratropics for 500 hPa geopotential height. 12-month running mean of the monthly scores from 5
of the global NWP centres. The upper curves show the 6-day forecast error and the lower curves
show the 2-day forecast error. Each model is verified against its own analysis.
CBS-DPFS/CG-FV/Doc. 4(3), p. 6
ECMWF 00utc
Verification to WMO standards M-F 00utc
verification against radiosondes
UKMO 00utc
geopotential 500hPa
NCEP 00utc
Root mean square error of forecast
Europe Lat 35.0 to 75.0 Lon -12.5 to 42.5 CMC 00utc
JMA 00utc
m
100
90
80
70
60
50
40
30
20
10
1 2 3 4 5 6 7 8
Forecast Day
ECMWF 00utc
Verification to WMO standards M-F 00utc
verification against radiosondes
UKMO 00utc
wind 850hPa
NCEP 00utc
Root mean square error of forecast
Europe Lat 35.0 to 75.0 Lon -12.5 to 42.5 CMC 00utc
JMA 00utc
m/s
12
11
10
9
8
7
6
5
4
3
1 2 3 4 5 6 7 8
Forecast Day
Figure 3. WMO/CBS exchanged scores using radiosondes: 500 hPa height (top) and 850 hPa
wind(bottom) RMS error over Europe (annual mean August 2009 – July 2010).
CBS-DPFS/CG-FV/Doc. 4(3), p. 7
JMA 12utc T+120
Verification to WMO standards CMC 00utc T+120
UKMO 12utc T+120
wind 250hPa
NCEP 00utc T+120
Root mean square error of forecast ECMWF 12utc T+120
M-F 00utc T+24
Tropics Lat -20.0 to 20.0 Lon -180.0 to 180.0
JMA 12utc T+24
CMC 00utc T+24
UKMO 12utc T+24
NCEP 00utc T+24
ECMWF 12utc T+24
m/s
13
12
11
10
9
8
7
6
5
4
3
2
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
JMA 12utc T+120
Verification to WMO standards CMC 00utc T+120
UKMO 12utc T+120
wind 850hPa
NCEP 00utc T+120
Root mean square error of forecast ECMWF 12utc T+120
M-F 00utc T+24
Tropics Lat -20.0 to 20.0 Lon -180.0 to 180.0
JMA 12utc T+24
CMC 00utc T+24
UKMO 12utc T+24
NCEP 00utc T+24
ECMWF 12utc T+24
m/s
6.5
6
5.5
5
4.5
4
3.5
3
2.5
2
1.5
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Figure 4: WMO/CBS exchanged scores from global forecast centres. RMS vector wind error over
tropics at 250 hPa (top) and 850 hPa (bottom). In each panel the upper curves show the 5-day
forecast error and the lower curves show the 1-day forecast error. Each model is verified against
its own analysis.
CBS-DPFS/CG-FV/Doc. 4(3), p. 8
JMA 12utc T+120
Verification to WMO standards CMC 00utc T+120
UKMO 12utc T+120
wind 250hPa
NCEP 00utc T+120
Root mean square error of forecast ECMWF 12utc T+120
M-F 00utc T+24
Tropics Lat -20.0 to 20.0 Lon -180.0 to 180.0
JMA 12utc T+24
CMC 00utc T+24
UKMO 12utc T+24
NCEP 00utc T+24
ECMWF 12utc T+48
m/s
14
13
12
11
10
9
8
7
6
5
4
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
JMA 12utc T+120
Verification to WMO standards CMC 00utc T+120
UKMO 12utc T+120
wind 850hPa
NCEP 00utc T+120
Root mean square error of forecast ECMWF 12utc T+120
M-F 00utc T+24
Tropics Lat -20.0 to 20.0 Lon -180.0 to 180.0
JMA 12utc T+24
CMC 00utc T+24
UKMO 12utc T+24
NCEP 00utc T+24
ECMWF 12utc T+48
m/s
8
7.5
7
6.5
6
5.5
5
4.5
4
3.5
3
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Figure 5: As Figure 4 for scores computed against radiosondes observations.
.
CBS-DPFS/CG-FV/Doc. 4(3), p. 9
Mean curves Feb 10
500hPa geopotential
Feb 09
Correlation coefficent of forecast anomaly
n.hem Feb 08
Date: 20060201 00UTC to 20100228 12UTC
Feb 07
oper_an od oper 120 erai
Mean calculation method: standard Feb 06
%
100
90
80
70
60
50
40
1 2 3 4 5 6 7 8 9 10
Forecast Day
Mean curves Feb 10ECMWF
500hPa geopotential Feb 10UKMO
Correlation coefficent of forecast anomaly Feb 10NCEP
n.hem
Feb 09ECMWF
Date: 20090201 00UTC to 20100228 12UTC
oper_an od erai Feb 09UKMO
Mean calculation method: standard Feb 09NCEP
%
100
98
96
94
92
90
88
86
84
82
80
1 2 3 4 5 6
Forecast Day
Figure 6: Anomaly correlation (ACC) error over northern extratropics for 500 hPa geopotential
height for February. Top panel shows the ACC for the ECMWF forecasts for February 2006-2010.
The bottom panel compares the ACC for February 2009 and 2010 for ECMWF, NCEP and the Met
Office (UKMO).
Time series curves
500hPa geopotential Score reaches 60
Correlation coefficent of forecast anomaly Score reaches 60
n.hem
od oper CBS-DPFS/CG-FV/Doc. 4(3), p. 10
MA 12 months
12UTC,beginning 120 MA 12 months
day
10
9.5
9
8.5
8
7.5
7
6.5
6
5.5
5
1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Figure 7: Comparison of current and revised computation of anomaly correlation scores. Monthly
mean (cyan - current; magenta - revised) and 12-month running mean (blue - current; red - revised)
of the forecast range at which the anomaly correlation for 500 hPa height operational forecasts
falls below 60% for the extra-tropical northern hemisphere.
CBS-DPFS/CG-FV/Doc. 4(3), p. 11
Time series curves
500hPa geopotential
ECMWF
Standard deviation of analysis anomaly
NH Extratropics Lat 20.0 to 90.0 Lon -180.0 to 180.0 Met Office
Moving average (12) oper_an od
T+144 00UTC,12UTC,beginning
m
96
95
94
93
92
91
90
89
88
87
86
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Time series curves
500hPa geopotential
ECMWF
Standard deviation of forecast anomaly
NH Extratropics Lat 20.0 to 90.0 Lon -180.0 to 180.0 Met Office
Moving average (12) oper_an od
T+144 00UTC,12UTC,beginning
m
96
95
94
93
92
91
90
89
88
87
86
85
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Figure 8: Example of recommended additional scores: timeseries of monthly mean standard
deviation of analysis (top) and 6-day forecast anomalies over northern extratropics for 500 hPa
geopotential. Each model is verified against its own analysis (computation of scores made at
ECMWF).
