Script
FORECASTING – Performance Measures
Slide 1 Welcome back. In this module we give a series of measures that can be used to evaluate which forecasting technique appears to give the best results for a given set of data.
Slide 2 Performance measures are used to compare how closely a given forecasting technique performs given a particular set of data. There is more than one approach one can use for time series forecasting. o For the stationary model we used to forecast demand for yoyos we used A last period technique A 4-period moving average technique And a weighted four period moving average techniques with weights of .4, .3, .2, .1. And there are many other techniques we could have used. We could have done 5 period moving averages; we could have used weights of .5, .3, .15, and .05 in a 4-period weighted approach; we could use exponential smoothing which will be discussed in another module. The question is, for the 52 weeks worth of time series data that we have, which of the techniques that we used appears to be the most effective at forecasting future values for demand. o The answer is -- the one that overall comes the closest. But how we measure “closest” leads to several different approaches.
Slide 3 All performance measures involve calculating forecast errors for the data. How those forecast errors are treated defines the different performance measures. A forecast error o for time period t, o denoted delta t, is simply o The difference between the actual observation at time period t y sub t o And the forecasted value for the same time period, F sub t.
Slide 4 As we have said, there are numerous ways of looking at the forecast errors. One thing that is NOT done is to take a simple average of the forecast errors, because the underestimates or positive error values and the overestimates or negative error values tend to cancel themselves out, and in the long run should be 0. To avoid simply averaging values that can both be positive and negative, one approach that is used is the mean square error (or MSE) approach. As its name implies the forecasted errors are squared, giving all positive numbers, before they are averaged by dividing by the number of periods for which there is both a forecast and an observation. A second “averaging method” that averages only positive values is called the MAD performance measure. MAD stands for mean absolute deviation. The deviation is the forecast error. We simply take the absolute values of these errors and average them. A third “averaging method” takes into account not the distance, in absolute value terms, a forecast is away from the actual value, but its percentage as expressed as a percentage of the actual observation. This averaging method is called the MAPE which stands for the mean absolute percent error. Here the absolute value of each forecast error is divided by the actual value and the results are averaged. A fourth performance measure has nothing to do with averaging. It simply finds the largest absolute error or deviation in the forecasts. This performance measure is called the LAD for “largest absolute deviation.”
Slide 5 Obviously we should compare the forecasts using only one of these performance measures. The question you might ask is, “Which performance measure should I use?” The answer is that this is up to the modeler or the decision maker. Each has its advantages o The MSE measure will give great weight to potential outliers. If these are not eliminated from the data, they could seriously affect which method gets selected. This may or may not be what you want to do. o Since it does not deal with square values, the MAD performance measure gives less weight proportionately to these same outliers. Again, this may or may not be what you wish to do. o The MAPE performance measure gives less overall weight to large deviations if the time series values themselves are large. Again this may or may not be what you wish to do. o And the LAD will tell us if all absolute deviations fall below some threshold level. Although we offered four different performance measures, in general, we will use only one of two – the MSE or the MAD.
Slide 6 Slide 7 This procedure is repeated for the 4-period moving average method Except that the first place where there is both an actual value and a forecasted value is for week 5 in row 6. So the value of delta, the forecast error for that week is B6 minus C6 The square error is D6 circumflex 2 The absolute error is ABS of D6. And the absolute percent error is the absolute value of the forecast error in cell D6 divided by the actual observation for that week in cell B6. As before, these formulas with relative addresses are then dragged to row 53. And the MSE is the average of the entries in column E from E6 to E53. The MAD is the average of the entries in column F from F6 to F53. The MAPE is the average of the entries in column G from G6 to G53. And the LAD is the MAX of the entries in F6 to F53. We now show how to calculate the performance measure for each of our three forecasting techniques for our yoyo example. We begin with the last period technique. Since only weeks 2 through 52 have both an actual observation and a forecasted value, these are the periods for which we get the forecast errors, which we put in column D. Forecast error is defined as actual value minus forecasted value. So for week 2, this error is calculated in cell D3 by cell B3 minus cell C3. The square error for this week then is this error in cell D3 circumflex 2 or squared. This is recorded in cell E3. In cell F3 we record the absolute error for week 2 which is gotten by the absolute value or ABS of cell D3. And the absolute percent error for the forecast for week 2 is gotten by the absolute value of the error in cell D3 divided by the actual value for the week in cell B3. All four of these cells have relative addresses that will appropriately change as we drag the cells D3, E3, F3 and G3 down to the last period in row 53. Then the MSE is the average of the entries from E3 to E53 in column E. The MAD is the average of the entries from F3 to F53 in column F. The MAPE is the average of the absolute percent errors in cells G3 to G53. And the LAD is the maximum or MAX of the entries in column F from F3 to F53.
Slide 8 Slide 9 Let’s summarize these results for the yoyo example. If we were using the MSE method, the last period technique has a mean square error of 19,631, the 4-period moving average method has a means square error of 11,037, and the weighted moving average has a means square error of 11,992. So if we were using the MSE performance measure to select forecasting techniques, of the three techniques evaluated, the simple moving average method has the lowest MSE and hence would be recommended. The 4-period moving average technique also has the lowest MAD, and thus would be selected using that performance measure as well. The same is true for the MAPE and for the LAD Hence the 4-period moving average method performed the best of the three techniques tried under any of the four performance measures. We caution that this will not always be the case. The 4-period weighted moving average has exactly the same calculations as the 4-period simple moving average. The forecast error for week 5 is B6 minus C6 The squared error is D6 circumflex 2 The absolute error is ABS of D6 The absolute percent error is ABS of D6 divided by B6. These four cells are dragged down to row 53. Where the MSE is the average of cells E6 to E53. The MAD is the average of cells F6 to F53. The MAPE is the average of cells G6 to G53. And the LAD is the MAX of F6 to F53.
Slide 10 Let’s review what we’ve discussed in this module. We showed how use performance measures to suggest which of a set of forecasting techniques appears to be the best for a given set of time series data. We noted that all performance measures involve in some way a set of forecast error, delta which is the actual value y minus the forecasted value, F. o The performance measures we illustrated included the Mean Square Error or MSE approach o The Mean Absolute Deviation or MAD approach o The Mean Absolute Percent Error or MAPE approach o And the largest absolute deviation or the LAD approach And we showed how to easily calculate these quantities using Excel. That’s it for this module. Do any assigned homework and I’ll be back to talk to you again next time.