Backtesting Stochastic Mortality Models An Ex-Post Evaluation of by sgdy7777

VIEWS: 11 PAGES: 40

									Backtesting Stochastic Mortality Models:
 An Ex-Post Evaluation of Multi-Period-
       Ahead Density Forecasts

                 Kevin Dowd (CRIS, NUBS)
             Andrew J. G. Cairns (Heriot-Watt)
   David Blake (Pensions Institute, Cass Business School)
                Guy D. Coughlan (JPMorgan)
                 David Epstein (JPMorgan)
              Marwa Khalaf-Allah (JPMorgan)

 4th International Longevity Risk and Capital Market Solutions
                          Conference

                 Amsterdam September 2008
             Purposes of Paper

• To set out a framework to backtest the
  forecast performance of mortality models
  – Backtesting = evaluation of forecasts against
    subsequently realised outcomes
• To apply this backtesting framework to a
  set of mortality models
  – How well do they actually perform?



                                                    2
             Background
– This study is the fourth in a series involving
  a collaboration between Blake, Cairns and
  Dowd and the LifeMetrics team at
  JPMorgan
– Involves actuaries, economists and
  investment bankers
– Of course, it is very easy (and fun!) to
  attack the forecasting „abilities‟ of actuaries
  (remember Equitable?) and investment
  bankers (remember subprime? etc), but we
  should remember…
                                                    3
Its not just actuaries and investment
     bankers who can‟t forecast




                                        4
              Background

– Cairns et alia (2007) examines the empirical
  fits of 8 different mortality models applied to
  E&W and US male mortality data
– Compares model performance
   • Uses a range of qualitative criteria (e.g.,
     biological reasonableness, etc)
   • Uses a range of quantitative criteria (e.g., Bayes
     information criterion)




                                                          5
        Models considered
– Model M1 = Lee-Carter, no cohort effect
– Model M2 = Renshaw-Haberman‟s 2006 cohort
  effect generalisation of M1
– Model M3 = Currie‟s age-period-cohort model
– Model M4 = P-splines model, Currie 2004
– Model M5 = CBD two-factor model, Cairns et al
  (2006), no cohort effect
– Models M6, M7 and M8: alternative cohort-effect
  generalisations of CBD



                                                    6
Second study, Cairns et al (2008)
  – Examines ex ante plausibility of models‟
    density forecasts
  – M4 (P-Splines not considered)
  – Amongst other conclusions, finds that M8
    (which did very well in first study) gives very
    implausible forecasts for US data
  – Hence, decided to drop M8 as well
  – Thus, a model might fit past data well but
    still give unreliable forecasts
     •  Not enough just to look at past fits

                                                      7
Third study, Dowd et al (2008a)
 – Examines the Goodness of Fits of models M1,
   M2B, M3B, M5, M6 and M7 more systematically
    • M2B is a special case of M2, which uses an ARIMA(1,1,0)
      for cohort effect
    • M3B is a special case of M3, which the same
      ARIMA(1,1,0) for cohort effect
 – Basic idea to unravel the models‟ testable
   implications and test them systematically
 – Finds some problems with all models but M2B
   unstable



                                                                8
 Motivation for present study
– A model might
   • Give a good fit to past data and
   • Generate density forecasts that appear plausible ex ante
– And still produce poor forecasts
– Hence, it is essential to test performance of models
  against subsequently realised outcomes
   • This is what backtesting is about
– In the end, it is the forecast performance that really
  matters
– Would you want to drive a car that hadn‟t been
  field-tested?


                                                                9
    Backtesting framework
– Choose metric of interest
  • Could choose mortality rates, survival rates, life
    expectancy, annuity prices etc.
– Select historical lookback window used to
  estimate model params
– Select forecast horizon or lookforward
  window for forecasts
– Implement tests of how well forecasts
  subsequently performed


                                                         10
     Backtesting framework
– We choose focus mainly on mortality rate as metric
– We choose a fixed 10-year lookback window
   • This seems to be emerging as the standard amongst
     practitioners
– We examine a range of backtests:
   • Over contracting horizons
   • Over expanding horizons
   • Over rolling fixed-length horizons
   • Future mortality density tests




                                                         11
     Backtesting framework
– We consider forecasts both with and without
  parameter uncertainty
– Parameter certain case: treat estimates of
  parameters as if known values
– Parameter uncertain case: forecast using a
  Bayesian approach that allows for uncertainty in
  parameter estimates
   • Allows for uncertainty in parameters governing period and
     cohort effects
– Results indicate it is very important to allow for
  parameter uncertainty



                                                                 12
Contracting horizon BT: age 65
                                Males aged 65: Model M1                                            Males aged 65: Model M2B
                   0.04                                                                0.04
  Mortality rate




                                                                      Mortality rate
                   0.03                                                                0.03


                   0.02                                                                0.02


                   0.01                                                                0.01
                      1980   1985   1990     1995       2000   2005                       1980   1985   1990     1995       2000   2005

                               Males aged 65: Model M3B                                             Males aged 65: Model M5
                   0.04                                                                0.04
  Mortality rate




                                                                      Mortality rate
                   0.03                                                                0.03


                   0.02                                                                0.02


                   0.01                                                                0.01
                      1980   1985   1990     1995       2000   2005                       1980   1985   1990     1995       2000   2005

                                Males aged 65: Model M6                                             Males aged 65: Model M7
                   0.04                                                                0.04
  Mortality rate




                                                                      Mortality rate


                   0.03                                                                0.03


                   0.02                                                                0.02


                   0.01                                                                0.01
                      1980   1985   1990     1995       2000   2005                       1980   1985   1990     1995       2000   2005
                                    Stepping off year                                                   Stepping off year




                                                                                                                                          13
Contracting horizon BT: age 75
                                Males aged 75: Model M1                                            Males aged 75: Model M2B

                   0.08                                                                0.08
  Mortality rate




                                                                      Mortality rate
                   0.06                                                                0.06


                   0.04                                                                0.04


                   0.02                                                                0.02
                      1980   1985   1990     1995       2000   2005                       1980   1985   1990     1995       2000   2005

                               Males aged 75: Model M3B                                             Males aged 75: Model M5

                   0.08                                                                0.08
  Mortality rate




                                                                      Mortality rate
                   0.06                                                                0.06


                   0.04                                                                0.04


                   0.02                                                                0.02
                      1980   1985   1990     1995       2000   2005                       1980   1985   1990     1995       2000   2005

                                Males aged 75: Model M6                                             Males aged 75: Model M7

                   0.08                                                                0.08
  Mortality rate




                                                                      Mortality rate
                   0.06                                                                0.06


                   0.04                                                                0.04


                   0.02                                                                0.02
                      1980   1985   1990     1995       2000   2005                       1980   1985   1990     1995       2000   2005
                                    Stepping off year                                                   Stepping off year




                                                                                                                                          14
Contracting horizon BT: age 85
                                Males aged 85: Model M1                                            Males aged 85: Model M2B
                   0.25                                                                0.25

                    0.2                                                                 0.2
  Mortality rate




                                                                      Mortality rate
                   0.15                                                                0.15

                    0.1                                                                 0.1

                   0.05                                                                0.05
                      1980   1985   1990     1995       2000   2005                       1980   1985   1990     1995       2000   2005

                               Males aged 85: Model M3B                                             Males aged 85: Model M5
                   0.25                                                                0.25

                    0.2                                                                 0.2
  Mortality rate




                                                                      Mortality rate
                   0.15                                                                0.15

                    0.1                                                                 0.1

                   0.05                                                                0.05
                      1980   1985   1990     1995       2000   2005                       1980   1985   1990     1995       2000   2005

                                Males aged 85: Model M6                                             Males aged 85: Model M7
                   0.25                                                                0.25

                    0.2                                                                 0.2
  Mortality rate




                                                                      Mortality rate




                   0.15                                                                0.15

                    0.1                                                                 0.1

                   0.05                                                                0.05
                      1980   1985   1990     1995       2000   2005                       1980   1985   1990     1995       2000   2005
                                    Stepping off year                                                   Stepping off year




                                                                                                                                          15
          Conclusions so far

• Big difference between PC and PU
  forecasts
• PU prediction intervals usually considerably
  wider than PC ones
• M2B sometimes unstable
• Now consider expanding horizon
  predictions …

                                                 16
   Prediction-Intervals from 1980: age 65
                              Males aged 65: Model M1                                                   Males aged 65: Model M2B
                            PU: [xL, xM, xU, n] = [0, 25, 1, 27]                                       PU: [xL, xM, xU, n] = [8, 27, 0, 27]
                 0.05       PC: [xL, xM, xU, n] = [7, 25, 1, 27]                            0.05       PC: [xL, xM, xU, n] = [16, 27, 0, 27]
Mortality rate




                                                                           Mortality rate
                 0.04                                                                       0.04

                 0.03                                                                       0.03

                 0.02                                                                       0.02

                 0.01                                                                       0.01
                    1980   1985      1990       1995       2000     2005                       1980   1985      1990       1995       2000     2005

                             Males aged 65: Model M3B                                                    Males aged 65: Model M5
                            PU: [xL, xM, xU, n] = [0, 26, 1, 27]                                       PU: [xL, xM, xU, n] = [1, 27, 0, 27]
                 0.05       PC: [xL, xM, xU, n] = [12, 26, 1, 27]                           0.05       PC: [xL, xM, xU, n] = [18, 27, 0, 27]
Mortality rate




                                                                           Mortality rate
                 0.04                                                                       0.04

                 0.03                                                                       0.03

                 0.02                                                                       0.02

                 0.01                                                                       0.01
                    1980   1985      1990       1995       2000     2005                       1980   1985      1990       1995       2000     2005

                              Males aged 65: Model M6                                                    Males aged 65: Model M7
                 0.06                                                                       0.06
                            PU: [xL, xM, xU, n] = [0, 25, 1, 27]                                       PU: [xL, xM, xU, n] = [0, 19, 1, 27]
                 0.05       PC: [xL, xM, xU, n] = [14, 25, 1, 27]                           0.05       PC: [xL, xM, xU, n] = [7, 19, 1, 27]
Mortality rate




                                                                           Mortality rate




                 0.04                                                                       0.04

                 0.03                                                                       0.03

                 0.02                                                                       0.02

                 0.01                                                                       0.01
                    1980   1985      1990      1995        2000     2005                       1980   1985      1990      1995        2000     2005
                                            Year                                                                       Year
                                                                                                                                                      17
                 Prediction-Intervals from 1980: age 75
                             Males aged 75: Model M1                                                  Males aged 75: Model M2B
                  0.1                                                                       0.1
                           PU: [xL, xM, xU, n] = [1, 27, 0, 27]                                      PU: [xL, xM, xU, n] = [1, 27, 0, 27]
Mortality rate




                                                                          Mortality rate
                 0.08      PC: [xL, xM, xU, n] = [12, 27, 0, 27]                           0.08      PC: [xL, xM, xU, n] = [13, 27, 0, 27]


                 0.06                                                                      0.06


                 0.04                                                                      0.04

                   1980   1985      1990       1995       2000     2005                      1980   1985      1990       1995       2000     2005

                            Males aged 75: Model M3B                                                   Males aged 75: Model M5
                  0.1                                                                       0.1
                           PU: [xL, xM, xU, n] = [1, 27, 0, 27]                                      PU: [xL, xM, xU, n] = [0, 25, 1, 27]
Mortality rate




                                                                          Mortality rate
                 0.08      PC: [xL, xM, xU, n] = [8, 27, 0, 27]                            0.08      PC: [xL, xM, xU, n] = [7, 25, 1, 27]


                 0.06                                                                      0.06


                 0.04                                                                      0.04

                   1980   1985      1990       1995       2000     2005                      1980   1985      1990       1995       2000     2005

                             Males aged 75: Model M6                                                   Males aged 75: Model M7
                  0.1                                                                       0.1
                           PU: [xL, xM, xU, n] = [1, 27, 0, 27]                                      PU: [xL, xM, xU, n] = [1, 27, 0, 27]
Mortality rate




                                                                          Mortality rate


                 0.08      PC: [xL, xM, xU, n] = [8, 27, 0, 27]                            0.08      PC: [xL, xM, xU, n] = [9, 27, 0, 27]


                 0.06                                                                      0.06


                 0.04                                                                      0.04

                   1980   1985      1990      1995        2000     2005                      1980   1985      1990      1995        2000     2005
                                           Year                                                                      Year
                                                                                                                                                    18
                 Prediction-Intervals from 1980: age 85
                              Males aged 85: Model M1                                                  Males aged 85: Model M2B
                 0.25                                                                      0.25
                            PU: [xL, xM, xU, n] = [1, 22, 0, 27]                                       PU: [xL, xM, xU, n] = [0, 7, 1, 27]

                  0.2       PC: [xL, xM, xU, n] = [4, 22, 0, 27]                            0.2        PC: [xL, xM, xU, n] = [0, 5, 1, 27]
Mortality rate




                                                                          Mortality rate
                 0.15                                                                      0.15

                  0.1                                                                       0.1

                 0.05                                                                      0.05
                    1980   1985     1990        1995       2000    2005                       1980   1985      1990       1995        2000   2005

                             Males aged 85: Model M3B                                                   Males aged 85: Model M5
                 0.25                                                                      0.25
                            PU: [xL, xM, xU, n] = [1, 21, 0, 27]                                      PU: [xL, xM, xU, n] = [1, 24, 0, 27]

                  0.2       PC: [xL, xM, xU, n] = [2, 21, 0, 27]                            0.2       PC: [xL, xM, xU, n] = [2, 24, 0, 27]
Mortality rate




                                                                          Mortality rate
                 0.15                                                                      0.15

                  0.1                                                                       0.1

                 0.05                                                                      0.05
                    1980   1985     1990        1995       2000    2005                       1980   1985      1990       1995        2000   2005

                              Males aged 85: Model M6                                                   Males aged 85: Model M7
                 0.25                                                                      0.25
                            PU: [xL, xM, xU, n] = [1, 18, 0, 27]                                      PU: [xL, xM, xU, n] = [1, 26, 0, 27]

                  0.2       PC: [xL, xM, xU, n] = [1, 18, 0, 27]                            0.2       PC: [xL, xM, xU, n] = [5, 26, 0, 27]
Mortality rate




                                                                          Mortality rate




                 0.15                                                                      0.15

                  0.1                                                                       0.1

                 0.05                                                                      0.05
                    1980   1985     1990       1995        2000    2005                       1980   1985      1990      1995         2000   2005
                                            Year                                                                      Year
                                                                                                                                                    19
     Expanding PI conclusions
• PC models have far too many lower
  exceedances
• PU models have exceedances that are much
  closer to expectations
  – Especially for M1, M7 and M3B
  – Suggests that PU forecasts are more plausible
    than PC ones
• Negligible differences between PC and PU
  median predictions
• Very few upper exceedances

                                                    20
     Expanding PI conclusions
• Too few upper exceedances, and two many
  median and lower exceedances
•  some upward bias, especially for PC
  forecasts
• This upward bias is especially pronounced for
  PC forecasts
• Evidence of upward bias less clearcut for PU
  forecasts



                                                  21
Rolling Fixed Horizon Forecasts
• From now on, work with PU forecasts only
• Assume illustrative horizon = 15 years
• Now examine performance of each model in
  turn …




                                             22
                                               Model M1
                                                Age 85: [xL, xM, xU, n] = [1, 10, 0, 12]
                                                Age 75: [xL, xM, xU, n] = [0, 11, 0, 12]

                                                Age 65: [xL, xM, xU, n] = [1, 12, 0, 12]

                                                                                                                       Age 85
                   -1
                 10
Mortality rate




                                                                                                                       Age 75




                                                                                                                       Age 65




                   -2
                 10
                   1995   1996   1997   1998   1999        2000          2001       2002   2003   2004   2005   2006
                                                                  Year




                                                                                                                                23
                                               Model M2B
                                                Age 85: [xL, xM, xU, n] = [1, 5, 0, 12]
                                                Age 75: [xL, xM, xU, n] = [0, 12, 0, 12]

                                                Age 65: [xL, xM, xU, n] = [8, 12, 0, 12]

                                                                                                                        Age 85
                   -1
                 10
Mortality rate




                                                                                                                       Age 75




                                                                                                                       Age 65




                   -2
                 10
                   1995   1996   1997   1998   1999        2000          2001       2002   2003   2004   2005   2006
                                                                  Year




                                                                                                                                 24
                                               Model M3B
                                                 Age 85: [xL, xM, xU, n] = [0, 8, 0, 12]
                                                 Age 75: [xL, xM, xU, n] = [0, 12, 0, 12]

                                                 Age 65: [xL, xM, xU, n] = [2, 12, 0, 12]

                                                                                                                        Age 85
                   -1
                 10
Mortality rate




                                                                                                                        Age 75




                                                                                                                        Age 65




                   -2
                 10
                   1995   1996   1997   1998    1999        2000          2001       2002   2003   2004   2005   2006
                                                                   Year




                                                                                                                                 25
                                               Model M5
                                                Age 85: [xL, xM, xU, n] = [0, 8, 0, 12]
                                                Age 75: [xL, xM, xU, n] = [0, 12, 0, 12]

                                                Age 65: [xL, xM, xU, n] = [9, 12, 0, 12]

                                                                                                                       Age 85
                   -1
                 10
Mortality rate




                                                                                                                       Age 75




                                                                                                                       Age 65




                   -2
                 10
                   1995   1996   1997   1998   1999        2000          2001       2002   2003   2004   2005   2006
                                                                  Year




                                                                                                                                26
                                               Model M6
                                                Age 85: [xL, xM, xU, n] = [0, 4, 0, 12]
                                                Age 75: [xL, xM, xU, n] = [0, 12, 0, 12]

                                               Age 65: [xL, xM, xU, n] = [10, 12, 0, 12]


                                                                                                                       Age 85
                   -1
                 10
Mortality rate




                                                                                                                       Age 75




                                                                                                                       Age 65




                   -2
                 10
                   1995   1996   1997   1998   1999        2000          2001       2002   2003   2004   2005   2006
                                                                  Year




                                                                                                                                27
                                               Model M7
                                                Age 85: [xL, xM, xU, n] = [0, 8, 0, 12]
                                                Age 75: [xL, xM, xU, n] = [0, 12, 0, 12]

                                                Age 65: [xL, xM, xU, n] = [4, 12, 0, 12]

                                                                                                                        Age 85
                   -1
                 10
Mortality rate




                                                                                                                       Age 75




                                                                                                                       Age 65




                   -2
                 10
                   1995   1996   1997   1998   1999        2000          2001       2002   2003   2004   2005   2006
                                                                  Year



                                                                                                                                 28
   Tentative conclusions so far

• Rolling PI charts broadly consistent with
  earlier results
• Some evidence of upward bias but not
  consistent across models or always especially
  compelling
• M2B again shows instability




                                                  29
        Mortality density tests

• Choose age (e.g., 65) and horizon (e.g., 15
  years ahead)
• Use model to project pdf (or cdf) of mortality
  rate 15 years ahead
• Plot realised q on to pdf/cdf
• Obtain associated p-value (or PIT value)
• Reject if p is too far out in either tail




                                                   30
Example: P-Values of Realised Mortality: Males 65, 1980
           Start, Horizon = 26 Years Ahead
                                     Males aged 65: Model M1                                                                 Males aged 65: Model M2B
                  1                                                                                         1
CDF under null




                                                                                          CDF under null
                                   Realised q = 0.0149 : p-value = 0.159                                                 Realised q = 0.0149 : p-value = 0.021
                 0.5                                                                                       0.5




                  0                                                                                         0
                       0   0.005     0.01   0.015   0.02   0.025   0.03    0.035   0.04                          0   0.005   0.01   0.015   0.02   0.025   0.03   0.035   0.04

                                    Males aged 65: Model M3B                                                                 Males aged 65: Model M5
                  1                                                                                         1
CDF under null




                                                                                          CDF under null
                               Realised q = 0.0149 : p-value = 0.074                                                     Realised q = 0.0149 : p-value = 0.049
                 0.5                                                                                       0.5




                  0                                                                                         0
                       0   0.005     0.01   0.015   0.02   0.025   0.03    0.035   0.04                          0   0.005   0.01   0.015   0.02   0.025   0.03   0.035   0.04

                                     Males aged 65: Model M6                                                                 Males aged 65: Model M7
                  1                                                                                         1
CDF under null




                                                                                          CDF under null




                               Realised q = 0.0149 : p-value = 0.052                                                     Realised q = 0.0149 : p-value = 0.165
                 0.5                                                                                       0.5




                  0                                                                                         0
                       0   0.005     0.01   0.015 0.02 0.025       0.03    0.035   0.04                          0   0.005   0.01   0.015 0.02 0.025       0.03   0.035   0.04
                                               Mortality rate                                                                          Mortality rate
                                                                                                                                                                                 31
        Many ways to do this

• For h=25 years ahead: 1 way
  – 1980-2005 only
• For h=24 years ahead, 2 ways
  – 1980-2004, 1981-2005
• For h=23 years ahead, 3 ways
• ….
• For h=1 year ahead, 26 ways
  – 1980-1981, 1981-1982, …, 2004-2005



                                         32
     Lots of cases to consider

• The are 25+24+23+…+1=325 separate cases
  to consider, each equally „legitimate‟
• Need some way to make use of all
  possibilities but consolidate results
• We do so by computing p-values for each
  case and then work with mean p-values from
  each test
• These are reported below for each age, for
  h=5, 10 and 15 years ahead:


                                               33
                                                           Age 65
                        Males aged 65: Model M1                                                Males aged 65: Model M2B
           1                                                                       1
                  Average = 0.290 for forecasts 5 years ahead                             Average = 0.178 for forecasts 5 years ahead
                  Average = 0.188 for forecasts 10 years ahead                            Average = 0.086 for forecasts 10 years ahead
                  Average = 0.143 for forecasts 15 years ahead                            Average = 0.041 for forecasts 15 years ahead
P-value




                                                                        P-value
          0.5                                                                     0.5




           0                                                                       0
           1985       1990          1995          2000           2005              1985       1990          1995          2000           2005

                       Males aged 65: Model M3B                                                 Males aged 65: Model M5
           1                                                                       1
                  Average = 0.259 for forecasts 5 years ahead                             Average = 0.107 for forecasts 5 years ahead
                  Average = 0.164 for forecasts 10 years ahead                            Average = 0.063 for forecasts 10 years ahead
                  Average = 0.109 for forecasts 15 years ahead                            Average = 0.042 for forecasts 15 years ahead
P-value




                                                                        P-value
          0.5                                                                     0.5




           0                                                                       0
           1985       1990          1995          2000           2005              1985       1990          1995          2000           2005

                        Males aged 65: Model M6                                                 Males aged 65: Model M7
           1                                                                       1
                  Average = 0.193 for forecasts 5 years ahead                             Average = 0.270 for forecasts 5 years ahead
                  Average = 0.082 for forecasts 10 years ahead                            Average = 0.178 for forecasts 10 years ahead
                  Average = 0.039for forecasts 15 years ahead                             Average = 0.132 for forecasts 15 years ahead
P-value




                                                                        P-value




          0.5                                                                     0.5




           0                                                                       0
           1985       1990         1995           2000           2005              1985       1990         1995           2000           2005
                                Starting year                                                           Starting year
                                                                                                                                                34
                                                           Age 75
                        Males aged 75: Model M1                                                Males aged 75: Model M2B
           1                                                                       1
                  Average = 0.297 for forecasts 5 years ahead                             Average = 0.330 for forecasts 5 years ahead
                  Average = 0.314 for forecasts 10 years ahead                            Average = 0.326 for forecasts 10 years ahead
                  Average = 0.267 for forecasts 15 years ahead                            Average = 0.321 for forecasts 15 years ahead
P-value




                                                                        P-value
          0.5                                                                     0.5




           0                                                                       0
           1985       1990          1995          2000           2005              1985       1990          1995          2000           2005

                       Males aged 75: Model M3B                                                 Males aged 75: Model M5
           1                                                                       1
                  Average = 0.314 for forecasts 5 years ahead                             Average = 0.308 for forecasts 5 years ahead
                  Average = 0.282 for forecasts 10 years ahead                            Average = 0.291 for forecasts 10 years ahead
                  Average = 0.228 for forecasts 15 years ahead                            Average = 0.228 for forecasts 15 years ahead
P-value




                                                                        P-value
          0.5                                                                     0.5




           0                                                                       0
           1985       1990          1995          2000           2005              1985       1990          1995          2000           2005

                        Males aged 75: Model M6                                                 Males aged 75: Model M7
           1                                                                       1
                  Average = 0.310 for forecasts 5 years ahead                             Average = 0.312 for forecasts 5 years ahead
                  Average = 0.284 for forecasts 10 years ahead                            Average = 0.258 for forecasts 10 years ahead
                  Average = 0.226 for forecasts 15 years ahead                            Average = 0.228 for forecasts 15 years ahead
P-value




                                                                        P-value




          0.5                                                                     0.5




           0                                                                       0
           1985       1990          1995          2000           2005              1985       1990         1995           2000           2005
                                 Starting year                                                          Starting year
                                                                                                                                                35
                                                           Age 85
                        Males aged 85: Model M1                                                Males aged 85: Model M2B
           1                                                                       1
                  Average = 0.240 for forecasts 5 years ahead                              Average = 0.335 for forecasts 5 years ahead
                  Average = 0.326 for forecasts 10 years ahead                            Average = 0.368 for forecasts 10 years ahead
                  Average = 0.282 for forecasts 15 years ahead                            Average = 0.331 for forecasts 15 years ahead
P-value




                                                                        P-value
          0.5                                                                     0.5




           0                                                                       0
           1985       1990          1995          2000           2005              1985       1990          1995          2000           2005

                       Males aged 85: Model M3B                                                 Males aged 85: Model M5
           1                                                                       1
                  Average = 0.318 for forecasts 5 years ahead                              Average = 0.327 for forecasts 5 years ahead
                  Average = 0.386 for forecasts 10 years ahead                            Average = 0.377 for forecasts 10 years ahead
                  Average = 0.367 for forecasts 15 years ahead                            Average = 0.380 for forecasts 15 years ahead
P-value




                                                                        P-value
          0.5                                                                     0.5




           0                                                                       0
           1985       1990          1995          2000           2005              1985       1990          1995          2000           2005

                        Males aged 85: Model M6                                                 Males aged 85: Model M7
           1                                                                       1
                  Average = 0.327 for forecasts 5 years ahead                             Average = 0.330 for forecasts 5 years ahead
                  Average = 0.378 for forecasts 10 years ahead                            Average = 0.370 for forecasts 10 years ahead
                  Average = 0.386 for forecasts 15 years ahead
P-value




                                                                        P-value




                                                                                          Average = 0.371 for forecasts 15 years ahead
          0.5                                                                     0.5




           0                                                                       0
           1985       1990          1995          2000           2005              1985       1990          1995          2000           2005
                                 Starting year                                                           Starting year
                                                                                                                                                36
 Conclusions from these tests

• All models perform well
• No rejections at 1% SL
• Only 3 at 5% SL




                                37
        Overall conclusions

• Study outlines a framework for backtesting
  forecasts of mortality models
• As regards individual models and this dataset:
  – M1, M3B, M5 and M7 perform well most of the time
    and there is little between them
  – M2B unstable
  – Of the Lee-Carter family of models, hard to choose
    between M1 and M3B
  – Of the CBD family, M7 seems to perform best; little
    to choose between M5 and M7


                                                          38
   Two other points stand out

• In many but not all cases, and depending also
  on the model, there is evidence of an upward
  bias in forecasts
  – This is very pronounced for PC forecasts
  – This bias is less pronounced for PU forecasts
• Except maybe for M2B, PU forecasts are more
  plausible than the PC forecasts
•  Very important to take account of param
  uncertainty more or less regardless of the
  model one uses

                                                    39
                     References
• Cairns et al. (2007) “A quantitative comparison of stochastic
  mortality models using data from England & Wales and the
  United States.” Pensions Institute Discussion Paper PI-0701,
  March
• Cairns et al. (2008) “The plausibility of mortality density
  forecasts: An analysis of six stochastic mortality models.”
  Pensions Institute Discussion Paper PI-0801, April.
• Dowd et al. (2008a) “Evaluating the goodness of fit of
  stochastic mortality models.” Pensions Institute Discussion
  Paper PI-0802, September.
• Dowd et al. (2008b) “Backtesting stochastic mortality models:
  An ex-post evaluation of multi-year-ahead density forecasts.”
  Pensions Institute Discussion Paper PI-0803, September.
• These papers are also available at www.lifemetrics.com




                                                                  40

								
To top