Docstoc

Toward an Effective Short Range Ensemble Forecast System

Document Sample
Toward an Effective Short Range Ensemble Forecast System Powered By Docstoc
					 Toward an Effective Short-Range Ensemble Forecast System


                                          September 18, 2003


                                     Tony Eckel and Cliff Mass
              University of Washington Atmospheric Sciences Department




This research was supported by the DoD Multidisciplinary University Research Initiative (MURI) program
administered by the Office of Naval Research under Grant N00014-01-10745.
                              UW’s Ensemble of Ensembles

                         # of      EF       Initial       Forecast Forecast
            Name        Members   Type     Conditions     Model(s)  Cycle           Domain

            ACME         17       SMMA    8 Ind. Analyses, “Standard”    00Z        36km, 12km
                                          1 Centroid,         MM5
Homegrown




                                          8 Mirrors

            ACMEcore      8       SMMA    Independent     “Standard”     00Z        36km, 12km
                                            Analyses        MM5

            ACMEcore+     8       PMMA    “        “       8 MM5         00Z        36km, 12km
                                                          variations
Imported




            PME           8       MMMA    “        “      8 “native”    00Z, 12Z    36km
                                                          large-scale

                                  SMMA: Single Model Multi-Analysis
                                  PMMA: Perturbed-model Multi-Analysis
                                  MMMA: Multi-model Multi-Analysis
            ACME: Analysis-Centroid Mirroring Ensemble
            PME: Poor Man’s Ensemble
                                                  MM5: PSU/NCAR Mesoscale Modeling System Version 5
                  “Native” Models/Analyses of the PME
                                                                Resolution (~ @ 45 °N )      Objective
Abbreviation/Model/Source                            Type     Computational    Distributed   Analysis
avn, Global Forecast System (GFS),                   Spectral T254 / L64      1.0° / L14     SSI
National Centers for Environmental Prediction                 ~55 km          ~80 km         3D Var

cmcg, Global Environmental Multi-scale (GEM),        Finite   0.9°×0.9°/L28 1.25° / L11      3D Var
Canadian Meteorological Centre                       Diff      ~70 km        ~100 km

eta, limited-area mesoscale model,                   Finite   32 km / L45     90 km / L37    SSI
National Centers for Environmental Prediction        Diff.                                   3D Var

gasp, Global AnalysiS and Prediction model,          Spectral T239 / L29      1.0° / L11     3D Var
Australian Bureau of Meteorology                              ~60 km          ~80 km

jma, Global Spectral Model (GSM),                    Spectral T106 / L21      1.25° / L13    OI
Japan Meteorological Agency                                   ~135 km         ~100 km

ngps, Navy Operational Global Atmos. Pred. System,   Spectral T239 / L30      1.0° / L14     OI
Fleet Numerical Meteorological & Oceanographic Cntr.          ~60 km          ~80 km

tcwb, Global Forecast System,                        Spectral T79 / L18       1.0° / L11     OI
Taiwan Central Weather Bureau                                 ~180 km         ~80 km

ukmo, Unified Model,                                 Finite   5/6°×5/9°/L30 same / L12       3D Var
United Kingdom Meteorological Office                 Diff.    ~60 km
                                                           Design of ACMEcore+

                            PBL                                                            Cumulus
                                      vertical                                       36km      12km                shallow               SST
  IC      ID#                 Soil    diffusion    Cloud Microphysics               Domain    Domain               cumulus Radiation Perturbation         Land Use Table

       ACME     MRF         5-Layer      Y        Simple Ice                     Kain-Fritsch       Kain-Fritsch     N       cloud       standard      standard

  ACMEcore+
avn      plus01 MRF         LSM          Y        Simple Ice                     Kain-Fritsch       Kain-Fritsch     Y       RRTM        SST_pert01    LANDUSE.TBL.plus1

cmcg     plus02 MRF         5-Layer      Y        Reisner II (grpl), Skip4       Grell              Grell            N       cloud       SST_pert02    LANDUSE.TBL.plus2

eta      plus03 Eta         5-Layer      N        Goddard                        Betts-Miller       Grell            Y       RRTM        SST_pert03    LANDUSE.TBL.plus3

gasp     plus04 MRF         LSM          Y        Shultz                         Betts-Miller       Kain-Fritsch     N       RRTM        SST_pert04    LANDUSE.TBL.plus4

jma      plus05 Eta         LSM          N        Reisner II (grpl), Skip4       Kain-Fritsch       Kain-Fritsch     Y       cloud       SST_pert05    LANDUSE.TBL.plus5

ngps     plus06 Blackadar   5-Layer      Y        Shultz                         Grell              Grell            N       RRTM        SST_pert06    LANDUSE.TBL.plus6

tcwb     plus07 Blackadar   5-Layer      Y        Goddard                        Betts-Miller       Grell            Y       cloud       SST_pert07    LANDUSE.TBL.plus7

ukmo     plus08 Eta         LSM          N        Reisner I (mx-phs)             Kain-Fritsch       Kain-Fritsch     N       cloud       SST_pert08    LANDUSE.TBL.plus8
                                                                                                                                                           Perturbations to:
                                                                                                                                                       1) Moisture Availability
                                                                                                                                                       2) Albedo
                                                                                                                                                       3) Roughness Length




Total possible combinations:
    8    × 5 × 2 × 2                                 ×         5             ×           3      ×       2 ×        2 × 2             ×      8         × 8
                                                                                                                                                      = 1,228,800
                                  Effects on EF Forecast Probability

• In a large, well-tuned EF, Forecast Probability (FP) = Observed Relative Frequency (ORF )
                  Initial State              24hr Forecast State         F       48hr Forecast State
                                                                       PD
      Frequency



                                                                 u   e
                                                              Tr
                                                                                                FP = ORF = 72%



                                                                         Parameter Threshold
                                                                        (EX: precip > 10 mm)

• In practice, things go awry from…
      - Undersampling of the PDF (too few ensemble members)
      - Poor representation of IC uncertainty
      - Model deficiencies
           1) Model bias causes a shift in the estimated mean
           2) Sharing of model errors between EF members leads to reduced variance
• EF’s estimated PDF does not match truth’s PDF—degrades resolution and reliability
      Frequency




                                                                                                   FP = 93%
  Multimodel        PME
      Vs.            Vs.
Perturbed-Model   ACMEcore+
                    36-km
                   Domain
                                                                                         Ensemble Dispersion
                                                                                         *Bias-corrected MSLP @ 36 h
                                                                          0.2




                                                            Probability
                                                                          0.1


                                                                                                                                                   *PME
                                                                                                                                                   36h PME
                                                                          0.0                                                                       *ACMEcore+
                                                                                                                                                 36h36h ACMEcore+
                                                                                                                                                    PME
                                                                                1    2         3   4         5    6     7           8   9

                                                                                                   Verification Rank




                          8.0                                                                      8.0
or MSE of EF Mean (mb2)




                                            *ACMEcore+ Spread                                                    *PME Spread
                          7.0                                                                      7.0
                                            MSE of *ACMEcore Mean                                                MSE of *PME Mean

                          6.0                                                                      6.0
                                    σc2   ≈ 107   mb2
                          5.0                                                                      5.0


                          4.0                                                                      4.0


                          3.0                                                                      3.0
    EF Spread




                          2.0                                                                      2.0


                          1.0                                                                      1.0


                          0.0                                                                      0.0                                                       EF Mean’s MSE adjusted
                                0            12            24                   36        48             0       12            24           36        48     by n / n+1 to account for
                                                        Lead Time (h)                                                   Lead Time (h)                        small sample size
                                                                                               Reliability Diagram Comparison
                                                                                                             P(MSLP < 1001mb) @ 36 h



                              1.0
                                                                                 PME                                                                                                                                   ACMEcore+
                                                                                                                                                                 1.0

                                                                 freq. of use
                                      freq. of use




                                                                                                                                                                         freq. of use




                                                                                                                                                                                                        freq. of use
                              0.9                                                                                                                                0.9

                              0.8                                                                                                                                0.8
Observed Relative Frequency




                                                                                                                                   Observed Relative Frequency
                              0.7                                                                                                                                0.7

                              0.6                                                                                                                                0.6

                              0.5                                                                                                                                0.5

                              0.4                                                                                                                                0.4

                              0.3                                                                                                                                0.3
                                                                                                     Sample Climatology
                              0.2                                                                                                                                0.2

                              0.1                                                                                                                                0.1

                              0.0                                                                                                                                0.0
                                    0.0              0.1   0.2   0.3            0.4      0.5   0.6     0.7   0.8    0.9   1.0                                          0.0              0.1    0.2   0.3               0.4     0.5     0.6     0.7   0.8   0.9   1.0
                                                                 Forecast Probability                                                                                                                Forecast Probability



                                                           Resolution                 Reliability     Uncertainty     BSS
                                                                                                                     Skill Score                                                              Resolution                 Reliability         Uncertainty    BSS
                                                                                                                                                                                                                                                           Skill Score

                                      Biased                0.1526                     0.0008          0.1756         0.8641                                            Biased                 0.1443                        0.0013           0.1756        0.8138
                               Corrected                    0.1522                     0.0002          0.1756         0.8655                                      Corrected                    0.1465                        0.0002           0.1756        0.8330
Skill for                            0.95


P(MSLP < 1001 mb)
                                     0.90



        res − rel




                         BSS
BSS =                                0.85
          unc
                                                *PME
                                     0.80       PME
                                                *ACMEcore+
                                                ACMEcore+
                                     0.75
                                            0         12         24          36   48
                                  0.000
                    Reliability




                                  0.001


                                  0.002


                                  0.003
                                     0.17




                                     0.16
                        Resolution




                                     0.15




                                     0.14
                                            0         12         24          36   48
                                                             Lead Time (h)
                           How Well Does an EF Capture Truth?

• Missing Rate (MR) – The percentage of verifications not “encompassed” by the ensemble members

        N + N n +1        N1 : Number of verifications in rank #1
  MR =  1          100
          M                   of a VRH                                          or Missing Rate Error (MRE):
                            Nn+1 : Number of verifications in rank #n+1
                                   where n is the number of members                             2 
                                                                                    MRE = MR −      100
                            M : Number of verifications                                         n +1



• Standardized Verification (VZ ) – The deviation of the verification with respect to the standard deviation
                                    of the ensemble members
         V −e        e : Ensemble mean
  VZ =               V : Verification value
           s
                     s : Ensemble standard deviation

                                                              VZ = −3              VZ = 1       VZ = 3     VZ = 5



• Verification Outlier Percentage (VOP ) – The percentage of verifications not “portrayed” by the
                                          ensemble members (i.e., how often verification is an outlier)

        100 M       0 : 3 s m ≥ Vm − em 
                                                em : Ensemble mean at point m
  VOP =
         M m =1
                ∑   
                    1: 3 s m < Vm − em 
                                                 Vm : Verification value at point m                     E{VOP} ≅ 0.3%
                                                sm : Ensemble standard deviation at point m            for large n and
                                                                                                         normal PDF
                                                     Sample Applications

              0.2           *ACMEcore 36 h MSLP
Probability




                                                                       MR = 36.6%
              0.1                                                      MRE = 14.4%

                                                                       VOP = 9.0%
              0.0
                    1   2   3    4     5       6       7   8     9

                                Verification Rank




Z500 EF Mean               rank 1                                                               Vz < −3
& Verification Extreme Ranks      rank 9                             Z500 EF Mean & Vz                    Vz > 3




                                                                                                 -6




                                                    MR = 27.9%
   *ACMEcore+ 12 h          Forecast                MRE = 5.7%       *ACMEcore+ 12 h Forecast         VOP = 3.3%
                                    Case Study Initialized at 00Z, 20 Dec 2002

                                                    12 h Forecast
                               Z500 EF Mean and Standardized Verification, Vz
             -6


                                 Vz < −3                                       V −e
                                                                        VZ =
                                                                                 s
                                  Vz > 3




*ACMEcore+        VOP = 3.3%




*PME              VOP = 1.2%      Z500 Cent Analysis 12Z, 20 Dec 2002
                                     Case Study Initialized at 00Z, 20 Dec 2002

                                                     24 h Forecast
             -6

                                Z500 EF Mean and Standardized Verification, Vz

                                  Vz < −3                                       V −e
                                                                         VZ =
                                                                                  s
                                   Vz > 3




*ACMEcore+        VOP = 10.5%




*PME              VOP = 7.4%       Z500 Cent Analysis 00Z, 21 Dec 2002
                                Case Study Initialized at 00Z, 20 Dec 2002

                                                36 h Forecast
                           Z500 EF Mean and Standardized Verification, Vz

                             Vz < −3                                       V −e
                                                                    VZ =
                                                                             s
                              Vz > 3




*ACMEcore+   VOP = 13.8%




*PME         VOP = 8.0%       Z500 Cent Analysis 12Z, 21 Dec 2002
                               Case Study Initialized at 00Z, 20 Dec 2002

                                               48 h Forecast
                          Z500 EF Mean and Standardized Verification, Vz

                            Vz < −3                                       V −e
                                                                   VZ =
                                                                            s
                             Vz > 3




*ACMEcore+   VOP = 8.2%




*PME         VOP = 2.9%      Z500 Cent Analysis 00Z, 22 Dec 2002
       Value of        ACMEcore
   Model Diversity       Vs.
For a Mesoscale SREF   ACMEcore+

                         12-km
                        Domain
                                                   0.2
                                                                          (a) Z500                                         VOP
                                                   0.1                                                36h *PME             1.84%
                                                                                                      36h *ACMEcore        5.03%
                                                   0.0                                                 36h
                                                                                                     36h *PM*ACMEcore+     4.16%
                                                         1   2   3    4      5       6   7   8   9

                                                   0.2
                                                                      (b) MSLP
                                                   0.1
                                                                                                      36h *PME             1.55%
   Synoptic                                                                                           36h   *ACMEcore      9.02%
  Parameter                                        0.0                                                 36h *ACME
                                                                                                     36h *PM
                                                                                                                   core+   6.67%
(Errors Depend on                                        1   2   3    4      5       6   7   8   9
 IC Uncertainty)
                                                   0.3
                                                                       (c) WS10
                                                   0.2
             Surface/Mesoscale
                Parameter                          0.1
               (Errors Depend on                                                                      36h *ACMEcore        25.6%
                Model Uncertainty)
                                                   0.0                                                36h
                                                                                                     36h *AC*ACMEcore+     13.3%
                                                         1   2   3    4      5       6   7   8   9




                                                   0.3
                                                                          (d) T2
                                     Probability




                                                   0.2



                                                   0.1

                                                                                                      36h *ACMEcore        43.7%
                                                   0.0                                                36h *ACME
                                                                                                     36h *AC
                                                                                                                  core+    21.0%
                                                         1   2   3    4      5       6   7   8   9

                                                                     Verification Rank
                                                                            Ensemble Dispersion
                                                                               *Bias Corrected

                                                             WS10                                                                                   T2
or MSE of EF Mean (m2/s2)




                            4.5                                                                                       5.0
                                      σc2 ≈ 11.4 m2 / s2                                                                        σc2 ≈ 13.2 ºC2




                                                                                            or MSE of EF Mean (°C2)
                                                                                                                      4.5




                                                                                                               °
                            4.0

                            3.5                                                                                       4.0

                                                                                                                      3.5
                            3.0
                                                                                                                      3.0
                            2.5
                                                                                                                      2.5
                            2.0
                                                                                                                      2.0
                            1.5




                                                                                                   EF Spread
  EF Spread




                                                                                                                      1.5
                            1.0
                                                                                                                      1.0

                            0.5                                                                                       0.5

                            0.0                                                                                       0.0
                                  0            12             24       36        48                                         0           12          24       36       48
                                                           Lead Time (h)                                                                         Lead Time (h)

                                                                   *ACMEcore Spread                                                  ACMEcore Spread (Uncorrected)
                                                                   MSE of *ACMEcore Mean

                                                                   *ACMEcore+ Spread                                                 ACMEcore+ Spread (Uncorrected)
                                                                   MSE of *ACMEcore+ Mean
                             0.6
Skill for
                             0.5
P(T2 < 0°C)
                             0.4

                             0.3




                    BSS
                             0.2                                                                                        *ACMEcore
                                                                                                                        ACMEcore
                             0.1
                                                                                                                        *ACMEcore+
                             0.0                                                                                        ACMEcore+
                            -0.1                                                                                        Uncertainty
                                   00   03   06   09   12   15   18   21   24   27   30   33   36   39   42   45   48
                            0.00


                            0.02
              Reliability




                            0.04


                            0.06



                            0.12
               Resolution




                            0.10


                            0.08


                            0.06


                            0.04
                                   00   03   06   09   12   15   18   21   24   27   30   33   36   39   42   45   48
                                                                      Lead Time (h)
                                   Findings (Old & New)
1) Bias correction improves EF skill (Richardson, 2001; Atger, 2003)

    a) Particularly important for mesoscale SREF in which model biases are often large

    b) Reliability is improved by correctly shifting the forecast PDF location

    c) Resolution is improved through increased sharpness of the forecast PDF

    d) Allows for fair and accurate analysis (i.e., consider statistical consistency)
                                    Findings (Old & New)
2) Inclusion of model diversity in an EF system is critical for complete representation of
   uncertainty (Houtekamer et al., 1996; Stensrud et al., 2000; Mylne et al., 2002 )

     a) Increases the spread of an underdispersive EF to improve statistical consistency

     b) Degree of importance depends on parameter, scale, event of interest, and weather
        regime

     c) Unequal skill among ensemble members is not problematic (Mylne, 2002)

     d) Both reliability and resolution are improved (even with increased spread) through
        better estimation of the forecast PDF

     e) A PME or multimodel system is good at capturing model uncertainty (Evans et al., 2000;
        Ziehmann, 2000; Ebert, 2001; Richardson, 2001; Wandishin et al., 2001; Mylne, 2002)
       -- Does a more thorough job than a perturbed-model system
       -- Slight overdispersion in a small EF may be advantageous
                                               References


Atger, F, 2003: Spatial and interannual variability of the reliability of ensemble-based probabilistic
  forecasts: consequences for calibration. Mon. Wea. Rev., 121, 1509–1523.

Ebert, E. E., 2001: Ability of a poor man’s ensemble to predict the probability and distribution of
  precipitation. Mon. Wea. Rev., 129, 2461–2480.

Evans, R. E., M. S. J. Harrison, R. J. Graham, and K. R. Mylne, 2000: Joint medium-range ensembles
  from the Met. Office and ECMWF systems. Mon. Wea. Rev., 128, 3104–3127.

Houtekamer, P.L., and J. Derome, 1995: Methods for ensemble prediction. Mon. Wea. Rev., 123, 2181–
  2196.

Mylne, K. R., R. E. Evans, and R. T. Clark, 2002: Multimodel multianalysis ensembles in quasi-
  operational medium-range forecasting. Quart. J. Roy. Meteor. Soc., 128, 361–384.

Richardson, D. S., 2001a: Ensembles using multiple models and analyses. Quart. J. Roy. Meteor. Soc.,
   127, 1847–1864.

Wandishin, M. S., S L. Mullen, D. J. Stensrud, and H. E. Brooks, 2001: Evaluation of a short-range
  multimodel ensemble system. Mon. Wea. Rev., 129, 729–747.

Ziehmann, C., 2000: Comparison of a single-model EPS with a multimodel ensemble consisting of a few
   operational models. Tellus, 52A, 280–299
Backup Slides
Success and Failure   ACMEcore
         of             Vs.
      ACME             ACME
                        5.0                                                   (a) MSLP                                                               16

                        4.5                                                                                                                          14
 Average RMSE (mb)




                        4.0                                                                                                                          12




                                                                                                                                                          Average Ranking
                        3.5                                                                                                                          10

                        3.0                                                                                                                          8

                        2.5                                                                                                                          6

12h                     2.0                                                                                                                          4            12h
24h                                                                                                                                                               24h
                        1.5                                                                                                                          2
36h                                                                                                                                                               36h
48h                                                                                                                                                               48h
                        1.0                                                                                                                          0
                              avn   cmcg   eta   gasp   jma   ngps   tcwb   ukmo   cent   avn′   cmcg′   eta′   gasp′   jma′   ngps′   tcwb′ ukmo′


                        3.6                                                    (b) WS10                                                              16

                        3.4                                                                                                                          14
 Average RMSE (m / s)




                        3.2                                                                                                                          12




                                                                                                                                                          Average Ranking
                        3.0                                                                                                                          10

                        2.8                                                                                                                          8

                        2.6                                                                                                                          6

12h                     2.4                                                                                                                          4            12h
24h                                                                                                                                                               24h
                        2.2                                                                                                                          2
36h                                                                                                                                                               36h
48h                                                                                                                                                               48h
                        2.0                                                                                                                          0
                              avn   cmcg   eta   gasp   jma   ngps   tcwb   ukmo   cent   avn′   cmcg′   eta′   gasp′   jma′   ngps′   tcwb′ ukmo′
      1.00
                                                 P(MSLP < 1001 mb)
      0.95


      0.90
BSS
      0.85


      0.80


      0.75


      0.70
              0                   12                  24                  36                   48   *ACMEcore
                                                 Lead Time (h)                                       ACMEcore
                                                                                                    *ACME
                                                                                                     ACME
      0.25
                                             P(WS10 > 18 kt)
      0.20


      0.15
BSS




      0.10


      0.05


      0.00


      -0.05
              00   03   06   09   12   15   18   21   24   27   30   33   36   39   42   45   48
                                                 Lead Time (h)
               0.3
                                                 (a) 36-h MSLP
                                      *ACMEcore




Probability
                                      MR = 36.5%                                     *ACME
               0.2
                                     MRE = 14.2%                                 MR = 26.1%
                                                                                MRE = 15.0%
                                     VOP =   9.1%
               0.1                                                              VOP =       8.7%

               0.0
                     1   2   3   4   5   6   7     8   9   10    11   12   13   14    15   16   17   18




               0.3
                                                 (b) 36-h WS10
                                     *ACMEcore
 Probability




                                     MR = 52.5%                                      *ACME
               0.2
                                     MRE = 30.3%                                 MR = 44.0%
                                     VOP = 25.9%                                MRE = 32.9%
               0.1
                                                                                VOP = 24.7%

               0.0
                     1   2   3   4   5   6   7     8   9   10    11   12   13   14    15   16   17   18


               0.4                                 (c) 36-h T2
                                     *ACMEcore
                                     MR = 68.1%                                      *ACME
               0.3
                                     MRE = 45.9%                                 MR = 62.3%
                                                                                MRE = 51.2%
 Probability




                                     VOP = 44.4%
               0.2
                                                                                VOP = 44.2%

               0.1



               0.0
                     1   2   3   4   5   6   7     8   9   10    11   12   13   14    15   16   17   18
                                                  Verification Rank
                    Case Study Initialized at 00Z, 20 Dec 2002
                                   36 h Forecast



                                   *ACMEcore


     Z500 EF Mean and
Verification Extreme Ranks                                  Z500 EF Mean and Vz

    rank 1      rank 9                                       Vz > 3     Vz < −3




                          MR = 47.9%
                         MRE = 25.7%                                          VOP = 18.5%
                    Case Study Initialized at 00Z, 20 Dec 2002
                                  36 h Forecast



                                    *ACME


     Z500 EF Mean and
Verification Extreme Ranks                                  Z500 EF Mean and Vz

    rank 1      rank 18                                      Vz > 3     Vz < −3




                       MR = 40.7%
                      MRE = 29.6%                                             VOP = 19.5%
                    Case Study Initialized at 00Z, 20 Dec 2002
                                   36 h Forecast



                                       *PME


     Z500 EF Mean and
Verification Extreme Ranks                                  Z500 EF Mean and Vz

    rank 1      rank 9                                       Vz > 3     Vz < −3




                          MR = 37.3%
                         MRE = 15.1%                                          VOP = 8.0%
                                              Ensemble Dispersion
                                              *Bias-corrected MSLP @ 36 h
              0.2




Probability
              0.1


                                                                                                 *PME
                                                                                                 36h PME
              0.0                                                                                 *ACMEcore+
                                                                                               36h36h ACMEcore+
                                                                                                  PME
                         1                2     3        4       5      6         7   8    9

                                                         Verification Rank




                                8.0


                                7.0


                                6.0


                                5.0
                    EF Spread




                                4.0


                                3.0


                                2.0


                                1.0


                                0.0
                                      0             12          24           36       48
                                                             Lead Time (h)
                              0.6
Skill for
                              0.5
P(WS10 > 18 kt)
                              0.4

                              0.3




                      BSS
                              0.2                                                                                        *ACMEcore
                                                                                                                          ACMEcore
                              0.1                                                                                        *ACMEcore+
                              0.0
                                                                                                                          ACMEcore+
                                                                                                                         Uncertainty
                             -0.1
                                    00   03   06   09   12   15   18   21   24   27   30   33   36   39   42   45   48
                            0.000


                            0.001
              Reliability




                            0.002


                            0.003



                            0.006
              Resolution




                            0.005


                            0.004


                            0.003


                            0.002
                                    00   03   06   09   12   15   18   21   24   27   30   33   36   39   42   45   48
                                                                       Lead Time (h)
                         BSS Improvement of ACMEcore+ over ACMEcore



                         0.14                                                                                                 70%
Raw Improvement in BSS                                                  T2 < 0°C
                         0.12                                                                                                 60%




                                                                                                                                    % Improvement in BSS
                         0.10                                                                                                 50%

                         0.08                                                                                                 40%
                                         aw
                         0.06           R                                                                                     30%

                         0.04                                                                                                 20%
                                       %
                         0.02                                                                                                 10%
                                    Ave. SC = 22.8%        Ave. Uncertainty = 0.176
                         0.00                                                                                                 0%
                                0      3      6   9   12     15    18    21   24      27   30   33   36   39   42   45   48
                                                                          Lead Time (h)

                         0.14                                                                                                 70%
                                                      T2 < 0°C (Ocean-masked)
Raw Improvement in BSS




                         0.12                                                                                                 60%




                                                                                                                                    % Improvement in BSS
                         0.10                                                                                                 50%
                                         w
                                       Ra




                         0.08                                                                                                 40%

                         0.06                                                                                                 30%

                         0.04          %                                                                                      20%

                         0.02                                                                                                 10%

                                    Ave. SC = 34.2%        Ave. Uncertainty = 0.225
                         0.00                                                                                                 0%
                                0      3      6   9   12     15    18    21   24      27   30   33   36   39   42   45   48
                                                                          Lead Time (h)
                          BSS Improvement of ACMEcore+ over ACMEcore



                         0.14                                                                                                  70%
Raw Improvement in BSS                                              WS10 > 18 kt
                         0.12                                                                                                  60%




                                                                                                                                     % Improvement in BSS
                         0.10                                                                                                  50%

                         0.08       Ave. SC = 16.2%        Ave. Uncertainty = 0.136                                            40%

                         0.06                                                                                                  30%

                         0.04                                                                                                  20%
                                       Raw
                         0.02                                                                                                  10%
                                      %
                         0.00                                                                                                  0%
                                0      3    6    9    12      15    18   21    24      27   30   33   36   39   42   45   48
                                                                          Lead Time (h)

                         0.14                                                                                                  70%
                                                           WS10 > 18 kt (Ocean-masked)
Raw Improvement in BSS




                         0.12                                                                                                  60%




                                                                                                                                     % Improvement in BSS
                         0.10                                                                                                  50%

                         0.08       Ave. SC = 2.34%        Ave. Uncertainty = 0.0229                                           40%

                         0.06                                                                                                  30%
                                                        %
                         0.04                                                                                                  20%
                                                       Raw
                         0.02                                                                                                  10%

                         0.00                                                                                                  0%
                                0      3    6    9    12      15    18   21    24      27   30   33   36   39   42   45   48
                                                                          Lead Time (h)
                               Probabilistic Forecast Verification
          Continuous                                                   by Discrete Bins

Brier Score                                 Decomposed Brier Score
    1 n
                                                            (              )             (            )
                     2
BS = ∑ (FPi − OBS i )
                                                1 M                             1 M
                                            BS = ∑ N i FPi* − ORFi*            − ∑ N i ORFi* − SC          + SC (1 − SC )
                                                                           2                           2

    n i =1                                      n i =1                          n i =1
                                                       (reliability)                (resolution)            (uncertainty)
n: number of data pairs                      M : number of probability bins (normally 11)
FPi: forecast probability {0.0…1.0}          N : number of data pairs in the bin
ORFi: observation {0.0 = yes, 1.0 = no}      FP*i : binned forecast probability {0.0, 0.1,…1.0}
                                             ORF*i : observation for the bin {0.0 = yes, 1.0 = no}
BS = 1 for perfect forecasts                 SC : sample climatology (total occurrences / total forecasts)
BS = 0 for worst case forecasts




Brier Skill Score                           Skill Score
          BS − BS clim             BS                  reliability − resolution + uncertainy   resolution − reliabilty
BSS =                        = 1−           SS = 1 −                                         =
        BS perfect − BS clim      BS clim                  0      −     0      + uncertainy         uncertainty


BSS = 1 for perfect forecasts                 ADVANTAGES:
BSS < 0 for forecasts worse than climo        1) No need to know long-term climatology of the parameter
                                              2) Can compute SS and visualize BS in a reliability diagram
                             Ideal Calculation of Forecast Probability (FP)
Given a very large ensemble, a PDF could be found a grid point for any parameter (e.g., wind speed, Ws).
For a certain threshold, say Ws ≥ 20kt, the FP is then simply the area under the PDF to the right (1− p value)
                             0.05




                             0.04
                 Frequency



                             0.03




                             0.02




                             0.01                 FP = 77.1%

                               0
                                    0   10   20       30       40    50   60       70

                                                   Wind Speed (kt)


Unfortunately, we work with very small ensembles so we can’t make a good estimate of the PDF.
Plus, we often do not even know what PDF shape to fit.

So we are forced to estimate FP by other means, for a set of Ws forecasts at a point such as:
                                                                          Note: These are random draws from
        Ws = {16.5 21.1 27.3 29.3 33.4 37.4 40.2 47.8}                    the PDF above
Democratic Voting FP
“pushes” FP towards the extreme values, so high FP is normally
                                                                                               FP = 7/8 = 87.5%
over-forecast and low FP is normally under-forecast.
   8/8          7/8            6/8            5/8            4/8          3/8          2/8          1/8          0/8




              16.5        21.1        27.3           29.3          33.4         37.4    40.2        47.8
                      20.0




  9/9      8/9               7/9        6/9            5/9          4/9         3/9      2/9         1/9          0/9
Uniform Ranks FP
                                                    FP = 7/9 + [ (21.1 – 20.0) / (21.1 – 16.5) ] * 1/9 = 80.4%
a continuous, more appropriate approximation.
                            FP When Threshold Falls in an Extreme Rank
Use the tail of a Gumbel PDF to approximate the fraction for the last rank.
 0.05                                                                                           1
                                                                                                                     a b
                                                                                               0.9
                                                                             ξ− x
 0.04
                                                                               β 
                                                        GCDF ( x) = exp − exp    
                                                                                               0.8

                                                                                           0.7

 0.03                                                                                          0.6
                                                                                                                                      fraction
                                                                                                                                       = a/b
                                                               ˆ s 6
                                                                                               0.5
                                                               β=
 0.02                                                             π                            0.4

                                                                                               0.3

 0.01
                                                               ξ = x − γβ
                                                               ˆ        ˆ                      0.2

                                                                                               0.1

   0                                                                                            0
        0   10   20   30   40   50     60   70    80                                                 0   10     20   30    40    50    60    70   80


 0.0,
 -∞              16.5           21.1             27.3       29.3         33.4           37.4             40.2             47.8              ∞
                                                                                                                              50.0




  9/9      8/9                   7/9              6/9         5/9           4/9         3/9              2/9               1/9              0/9
Uniform Ranks FP
                                                          FP = [ (1 – GCDF(50.0)) / (1 – GCDF(47.8)) ] * 1/9 = 8.5%
                               Calibration by Weighted Ranks
Use the verification rank histogram from past cases to define non-uniform, “weighted ranks”.

The ranks to sum up and fraction of the rank where the threshold falls are found the same way as with
uniform ranks, but now the probability within each rank is the chance that truth will occur there.




 0.0,
 -∞        16.5        21.1       27.3       29.3       33.4       37.4       40.2       47.8           ∞
                                                                                             50.0




 1.0      0.83          0.72       0.62       0.54       0.45       0.36       0.27       0.17          0.0
Weighted Ranks FP
                                           FP = [ (1 – CDF(50.0)) / (1 – CDF(47.8)) ] * 0.17 = 13.0%
                    Uniform Ranks vs. Democratic Voting



                                                                                                                              UR
                                                                  1.0

                                                                  0.9
           Data Info
                                                                  0.8
                                                                                                                                             DV
! P(MSLP < 1002mb)




                                    Observed Relative Frequency
                                                                  0.7
! Verification: centroid analysis                                                                                               Zo
                                                                                                                                  ne
                                                                  0.6                                                     ill
                                                                                                                        Sk
! 70 forecasts (Nov 25, 2002 –
                                                                  0.5
  Feb 7, 2003)
                                                                  0.4
! Applied 2-week, running bias
  correction                                                      0.3
                                                                                                              Sample Climatology
! 36km, Outer Domain                                              0.2

                                                                  0.1
! Lead time = 48h
                                                                  0.0
                                                                        0.0   0.1   0.2   0.3   0.4   0.5   0.6   0.7   0.8     0.9    1.0
                                                                                          Forecast Probability