The Evolution of Forecasts (for a sports drink) David Heath Department of Mathematical Sciences Carnegie Mellon University Note: some details have been altered! My charge (as I interpreted it): Discuss the use and usefulness of mathematics to solve a real problem Present a specific, real example in which relatively simple mathematics helped make something better Convey a bit of the “flavor” of doing mathematics motivated by applications The story: Client contacted Peter Jackson and me Client made and distributed a “sports drink” Several years before they had run out, resulting in: Lost sales Angry customers (potentially lost!) They were in a bind An earlier solution They had studied the variability of consumption (related to weather, etc.) They built a simple model and adopted a very conservative (i.e., lots of inventory) inventory policy. Since then, consumption had grown They couldn’t find enough warehouse space to hold all of the inventory Why the problem was difficult Most sales are in June, July and August Production facility had maximum production rate which could produce (each month) perhaps 12 percent of annual sales The policy was to be sure that at the beginning of each month there would be enough product in inventory to cover the next 1.6 months’ forecast demand. This meant that on July 1 they needed to have more than 50% of a year’s sales in inventory This meant that on June 1 they needed lots of inventory (because of their limited production capabilities) What they wanted from us Find a way to make things better: increase “product availability” and/or decrease cost What we needed to do Understand their current forecasting methodology Determine whether current forecasts were “best possible”, or, if could be improved, improve them Build a model to allow computation of system performance. This model should be compatible with their current production planning techniques Complexity Lots of warehouses, months, product sizes, product flavors Their production planning Each month: start with current inventory Of each type In each region In each month Obtain forecasts for demands Use linear programming to solve for optimal production levels for this month for each plant. Solution is for this period and “all” future periods. Carry out the current month of the plan; move to next month (i.e., go back to the top of this page) We first examined their forecasts How can you tell if forecasts are bad? Good forecasts should be a martingale What’s a martingale? Example: fortune of a gambler playing a fair game. Example: Conditional expected values of a future (unknown) quantity given more and more information. One simple property: Martingales have “uncorrelated increments” You can perform simple tests (regression, for example) to determine whether observed increments (i.e., changes in forecasts for a given product, region, and date) are uncorrelated. If they’re correlated, you can use the correlations to improve the forecasts We checked our data and found Forecast changes had fairly high negative correlations. That meant that if the forecasts were raised, it was likely that later they would be reduced. This meant that the forecasts were far from optimal. The forecasts were constructed by: Initially, forecasts (for the sales in some future month) were simply set equal to a multiple (like 1.06) of last year’s demand. As a given month grew closer, the sales estimates for that month were updated Here’s an example of how the forecasts varied over time (and how forecasts eventually became “past data”) Actual Forecast Trivial forecast Delivery March April May June July August Sept Month Observation date March 1 20 25 40 60 100 90 40 April 1 22 A 26 42 63 100 90 40 May 1 22A 24 A 41 62 102 90 40 June 1 22 A 24 A 45 A 66 110 92 40 July 1 22 A 24 A 45 A 65 A 108 91 41 Aug 1 22 A 24 A 45 A 65 A 110 A 92 42 Sept 1 22 A 24 A 45 A 65 A 110 A 88 A 40 Since the forecasts looked bad We decided to find out how they were done I asked to meet the person who made the forecasts It turned out that forecasts came from the sales department. The job of forecasting was given to the newest employee in the sales department! (We asked why. They said that this gave new employees a good introduction to “operations”.) We went to the client Our advice: You have to fix the forecasting problem! We found that others in the firm already had a better forecasting technique. Remember that there were several plants (and warehouses). If one plant were short inventory, product could be shipped to it by truck. The firm had tried to understand how many trucks they needed to handle these shipments. For this they needed forecasts of demand. The produced forecasts as follows: For each “customer” (a “customer” might be a chain of supermarkets, for example) They knew when the customer had last received a shipment. Based on season, temperatures, previous sales etc., they could estimate when this customer would reorder (and reorder, …) Adding these expected orders provided forecasts of future demand. We checked to see whether the successive forecasts looked like martingales (regression analysis again) and found that they did (i.e., we couldn’t improve the forecasts). But we weren’t done! What should we recommend to the client? Client’s questions: How much money were we talking about (saving)? If the benefits were small, then the risks (of having a new model) and the costs of implementation might outweigh the benefits. How should inventory levels be set? To answer these questions We needed to have a simulation. (The production process was far too complex for us to build analytic models.) To run the simulation we needed to be able to generate forecasts which behaved “like” the observed data (from the “store by store customer” analysis). We wanted to be sure that stock-outs would be infrequent, and costs would be low. A simulation process We need to produce “simulated forecasts” to drive the simulation These simulated forecasts should be the results of a good estimation procedure The forecasts should resemble, as closely as possible, the observed forecast behavior. So we analyzed the “good” forecasts to build a model for generating random variables to use in simulations Recall the forecast and actual values Delivery March April May June July August Sept Month Observation date March 1 20 25 40 60 100 90 40 April 1 22 A 26 42 63 100 90 40 May 1 22A 24 A 41 62 102 90 40 June 1 22 A 24 A 45 A 66 110 92 40 July 1 22 A 24 A 45 A 65 A 108 91 41 Aug 1 22 A 24 A 45 A 65 A 110 A 92 42 Sept 1 22 A 24 A 45 A 65 A 110 A 88 A 40 How can we construct these? How do the forecasts evolve? To understand this, take differences Want to compare two forecasts for several months of demand at two successive forecast dates. Example: Months out: 1 2 3 4 5 March 20 25 40 60 100 April 22 26 42 63 100 Difference: 2 1 2 3 0 For each month We look at changes (from each month to the next) of the forecasts for each future month. This gives us a sequence of vectors. The number of components that change from month to month is, in our example, 4. The first such vector is (2,1,2,3), giving the changes, from March 1 to April 1, of the forecast values (and one actual value) for each month. This gets us a set of vectors (or curves). On average (over all curves) the changes should be 0. (This is the martingale property.) We’d like to generate new curves which have essentially the same variance- covariance structure as the data. So we estimate this VCV structure and generate samples for use in the simulation. The first 4 differences: Mar – April 2, 1, 2, 3 Apr – May -2, -1, -1, 2 May – June 4, 4, 8, 2 June – July -1, -2, -1, -1 I’ll call these vectors “wiggles” (They’re the “wiggles” we’ve observed in past forecast changes.) In practice we’d get many such vectors, compute their covariance, and then choose our random variables for a simulation to have a Normal distribution with mean 0 and VCV equal to that of the vectors above. We use “Principal Components” Principal components analysis finds “fundamental wiggles”, based on the data. These “fundamental wiggles”, expressed as vectors, are orthogonal. The fundamental wiggles vary in size; we arrange them in decreasing order of size (so “biggest first”) We sometimes, to simplify the model, decide how many of these wiggles to keep (say M) and keep only wiggles 1,…M How to get the simulated demands and demand forecasts We assume we have a current forecast (vector) for the demand in each future period. To get the next forecast vector we Report the first entry in the forecast vector as the current realized demand. We then “slide the remaining forecasts to the left “one place”. For each “fundamental wiggle” we add (the wiggle), multiplied by an independent standard normal random variable, to our forecast vector The result is our new forecast vector. We feed these forecasts to the simulation routine and get The final presentation Peter Jackson and I presented our work Audience: client’s project team members The head of that group The head of that division I presented the theory and model, and Peter presented the simulation results and our recommendations The room was quiet Everybody seemed quite attentive They were participants in the project The “higher-ups” were present At the end of Peter’s presentation he asserted, “Based on this analysis, we believe you can achieve better product availability and, at the same time, save about $11,000,000 a year.” There was a long moment of silence Nobody spoke Finally, the most senior manager present said, “That’s almost a million a month. We have to do it!” Then there was applause … (Reports of how things went in later years were positive. But, as usual, we didn’t get details.) Leftovers: You can find the slides of this talk, as well as the published article, at www.math.cmu.edu/~heath By the way, we didn’t actually use the data in the say I described. We found that modelling changes in the logarithm of the data worked better. But it required some additional mathematics to make the resulting forecasts martingales.