Document Sample

Alternative Forecasting Methods: Bootstrapping Bryce Bucknell Jim Burke Ken Flores Tim Metts Agenda Scenario Obstacles Regression Model Bootstrapping Applications and Uses Results Scenario You have been recently hired as the statistician for the University of Notre Dame football team. You are tasked with performing a statistical analysis for the first year of the Charlie Weis era. Specifically, you have been asked to develop a regression model that explains the relationship between key statistical categories and the number of points scored by the offense. You have a limited number of data points, so you must also find a way to ensure that the regression results generated by the model are reliable and significant. Problems/Obstacles: Central Limit Theorem Replication of data Sampling Variance of error terms Constrained by the Central Limit Theorem In selecting simple random samples of size n from a population, the _ sampling distribution of the sample mean x can be approximated by a normal probability distribution as the sample size becomes large. It is generally accepted that the sample size must be 30 or greater to satisfy the large-sample condition of the theorem. Sample N = 1 Sample N = 2 Sample N = 3 Sample N = 4 1. http://www.statisticalengineering.com/central_limit_theorem_(summary).htm Central Limit Theorem Central Limit theorem is the foundation for many statistical procedures, because the distribution of the phenomenon under study does NOT have to be Normal because its average WILL tend to be normal. Why is the assumption of a normal distribution important? A normal distribution allows for the application of the empirical rule – 68%, 95% and 99.7% Chebyshev’s Theorem no more than 1/4 of the values are more than 2 standard deviations away from the mean, no more than 1/9 are more than 3 standard deviations away, no more than 1/25 are more than 5 standard deviations away, and so on. The assumption of a normally distributed data allows descriptive statistics to be used to explain the nature of the population Not enough data available? Monte Carlo simulation, a type of spreadsheet simulation, is used to randomly generate values for uncertain variables over and over to simulate a model. Monte Carlo methods randomly select values to create scenarios The random selection process is repeated many times to create multiple scenarios Through the random selection process, the scenarios give a range of possible solutions, some of which are more probable and some less probable As the process is repeated multiple times, 10,000 or more, the average solution will give an approximate answer to the problem The accuracy can be improved by increasing the number of scenarios selected Sampling without Replacement Simple Random Sampling A simple random sample from a population is a sample chosen randomly, so that each possible sample has the same probability of being chosen. In small populations such sampling is typically done "without replacement“ Sampling without replacement results in deliberate avoidance of choosing any member of the population more than once This process should be used when outcomes are mutually exclusive, i.e. poker hands Sampling with Replacement Initial data set is not sufficiently large enough to use simple random sampling without replacement Through Monte Carlo simulation we have been able to replicate the original population Units are sampled from the population one at a time, with each unit being replaced before the next is sampled. One outcome does not affect the other outcomes Allows a greater number of potential outcomes than sampling without replacement If observations were not replaced there would not be enough independent observations to create a sample size of n ≥ 30 Hetroscedasticity vs. Homoscedasticity Residuals Residuals X X Homoscedasticity – constant variance Hetroscedasticity – nonconstant variance All random variables have the same Random variables may have different finite variance variances Simplifies mathematical and Standard errors of regression coefficients may be understated computational treatment T-ratios may be larger than actual Leads to good estimation results in data mining and regression More common with cross sectional data Regression Model For ND Points Scored ND Points = 38.54 + 0.079*b1 - 0.170*b2 - 0.662*b3 - 3.16*b4 b1 = Total Yards Gained b3 = Total Plays b2 = Penalty Yards b4 = Turnovers Audit Trail -- Coefficient Table (Multiple Regression Selected) Series Included Standard Overall Description in Model Coefficient Error T-test F-test F-test ND Points Dependent 38.54 14.26 2.70 7.31 8.92 Total YDS Yes 0.08 0.02 5.29 27.97 Penalty YDS Yes -0.17 0.06 -2.64 6.99 Total Plays Yes -0.66 0.23 -2.84 8.05 Turnovers Yes -3.16 2.50 -1.26 1.59 4 Checks of a Regression Model 1. Do the coefficients have the correct sign? 2. Are the slope terms statistically significant? 3. How well does the model fit the data? 4. Is there any serial correlation? 4 Checks of a Regression Model 1. Do the coefficients have the correct sign? Audit Trail -- Coefficient Table Series Included Description in Model Coefficient ND Points Dependent 38.54 Total YDS Yes 0.08 Penalty YDS Yes -0.17 Total Plays Yes -0.66 Turnovers Yes -3.16 Could this represent a big play factor? 4 Checks of a Regression Model 2. Are the slope terms statistically significant? Audit Trail -- Coefficient Table (Multiple Regression Selected) 3. How well does the model fit the Series Included Standard data? Overall Description in Model Coefficient Error T-test F-test F-test ND Points Dependent 38.54 14.26 2.70 7.31 8.92 Is there any serial correlation? 4. Total YDS Yes 0.08 0.02 5.29 27.97 Penalty YDS Yes -0.17 0.06 -2.64 6.99 Total Plays Yes -0.66 0.23 -2.84 8.05 Turnovers Yes -3.16 2.50 -1.26 1.59 M 0 10 20 30 40 50 60 ay -0 Ju 5 n- 05 Ju l- Au 05 g- 0 Se 5 p- 0 O 5 ct -0 N 5 ov -0 D 5 ec -0 Ja 5 n- 0 ND Points Fe 6 b- 0 M 6 ar -0 Ap 6 r- 0 M 6 ay -0 Ju 6 n- ND Points 06 Ju l- Forecast of ND Points Au 06 g- 0 Se 6 p- 0 O 6 ct -0 Adjusted R2 = 74.22% N 6 ov -0 D 6 Fitted Values ec -0 Ja 6 n- 0 Fe 7 b- 0 M 7 ar -0 Ap 7 4 Checks of a Regression Model r- 07 3. How well does the model fit the data? 4 Checks of a Regression Model 4. Is there any serial correlation? Data is cross sectional With limited data points, how useful is this regression in describing how well the model fits the actual data? Is there a way to tests its reliability? How to test the significance of the analysis What happens when the sample size is not large enough (n ≥ 30)? Bootstrapping is a method for estimating the sampling distribution of an estimator by resampling with replacement from the original sample. Commonly used statistical significance tests are used to determine the likelihood of a result given a random sample and a sample size of n. If the population is not random and does not allow a large enough sample to be drawn, the central limit theorem would not hold true Thus, the statistical significance of the data would not hold Bootstrapping uses replication of the original data to simulate a larger population, thus allowing many samples to be drawn and statistical tests to be calculated How It Works Bootstrapping is a method for estimating the sampling distribution of an estimator by resampling with replacement from the original sample. The bootstrap procedure is a means of estimating the statistical accuracy . . . from the data in a single sample. Bootstrapping is used to mimic the process of selecting many samples when the population is too small to do otherwise The samples are generated from the data in the original sample by copying it many number of times (Monte Carlo Simulation) Samples can then selected at random and descriptive statistics calculated or regressions run for each sample The results generated from the bootstrap samples can be treated as if it they were the result of actual sampling from the original population Characteristics of Bootstrapping Sampling with Full Sample Replacement Random sampling with Bootstrapping Example replacement can be employed to create multiple independent Limited samples for analysis number of Original Data Set observations 1st Random Sample Pittsburgh Navy Michigan Ohio State Michigan State USC Washington Washington Purdue Ohio State USC 109 Copies USC of each BYU observation BYU Tennessee Stanford Navy Pittsburgh Syracuse Creating a Ohio State much larger Stanford sample with Stanford which to work Ohio State Michigan When it should be used Bootstrapping is especially useful in situations when no analytic formula for the sampling distribution is available. Traditional forecasting methods, like exponential smoothing, work well when demand is constant – patterns easily recognized by software In contrast, when demand is irregular, patterns may be difficult to recognize. Therefore, when faced with irregular demand, bootstrapping may be used to provide more accurate forecasts, making some important assumptions… Assumptions and Methodology Bootstrapping makes no assumption regarding the population No normality of error terms No equal variance Allows for accurate forecasts of intermittent demand If the sample is a good approximation of the population, the sampling distribution may be estimated by generating a large number of new samples For small data sets, taking a small representative sample of the data and replicating it will yield superior results Applications and Uses Criminology Statistical significance testing is important in criminology and criminal justice Six of the most popular journals in criminology and criminal justice are dominated by quantitative methods that rely on statistical significance testing However, it poses two potential problems: tautology and violations of assumptions Applications and Uses Criminology Tautology: the null hypothesis is always false because virtually all null hypothesis may be rejected at some sample size Violation of assumptions of regression: errors are homogeneous and errors of independent variables are normally distributed Bootstrapping provides a user-friendly alternative to cross-validation and jackknife to augment statistical significance testing Applications and Uses Actuarial Practice Process of developing an actuarial model begins with the creation of probability distributions of input variables Input variables are generally asset-side generated cash flows (financial) or cash flows generated from the liabilities side (underwriting) Traditional actuarial methodologies are rooted in parametric approaches, which fit prescribed distribution of losses to the data Applications and Uses Actuarial Practice However, experience from the last two decades has shown greater interdependence of loss variables with asset variables Increased complexity has been accompanied by increased competitive pressures and more frequent insolvencies There is a need to use nonparametric methods in modeling loss distributions Bootstrap standard errors and confidence intervals are used to derive the distribution Applications and Uses Classifications Used by Ecologists Ecologists often use cluster analysis as a tool in the classification and mapping of entities such as communities or landscapes However, the researcher has to choose an adequate group partition level and in addition, cluster analysis techniques will always reveal groups Use bootstrap to test statistically for fuzziness of the partitions in cluster analysis Partitions found in bootstrap samples are compared to the observed partition by the similarity of the sampling units that form the groups. Applications and Uses Human Nutrition Inverse regression used to estimate vitamin B-6 requirement of young women Standard statistical methods were used to estimate the mean vitamin B-6 requirement Used bootstrap procedure as a further check for the mean vitamin B-6 requirement by looking at the standard error estimates and confidence intervals Application and Uses Outsourcing Agilent Technologies determined it was time to transfer manufacturing of its 3070 in-circuit test systems from Colorado to Singapore Major concern was the change in environmental test conditions (dry vs humid) Because Agilent tests to tighter factory limits (“guard banding”), they needed to adjust the guard band for Singapore Bootstrap was used to determine the appropriate guard band for Singapore facility An Alternative to the bootstrap Jackknife A statistical method for estimating and removing bias* and for deriving robust estimates of standard errors and confidence intervals Created by systematically dropping out subsets of data one at a time and assessing the resulting variation Bias: A statistical sampling or testing error caused by systematically favoring some outcomes over others A comparison of the Bootstrap & Jackknife Bootstrap Jackknife Yields slightly different Less general technique results when repeated Explores sample on the same data variation differently (when estimating the Yields the same result standard error) each time Not bound to Similar data theoretical requirements distributions Another alternative method Cross-Validation The practice of partitioning data into a sample of data into sub-samples such that the initial analysis is conducted on a single sub-sample (training data), while further sub-samples (test or validation data) are retained “blind” in order for subsequent use in confirming and validating the initial analysis Bootstrap vs. Cross-Validation Bootstrap Cross-Validation Requires a small of Not a resampling data technique More complex Requires large technique – time amounts of data consuming Extremely useful in data mining and artificial intelligence Methodology for ND Points Model Use bootstrapping on ND points scored regression model Goal: determine the reliability of the model Replication, random sampling, and numerous independent regression Calculation of a confidence interval for adjusted R2 Bootstrapping Results R2 Data Sample # Adjusted R^2 Sample # Adjusted R^2 The Mean, Standard 1 0.7351 13 0.7482 Dev., 95% and 99% 2 0.7545 14 0.8719 confidence intervals 3 0.7438 are then calculated in 15 0.7391 excel from the 24 4 0.7968 16 0.9025 observations 5 0.5164 17 0.8634 6 0.6449 18 0.7927 7 0.9951 19 0.6797 8 0.9253 20 0.6765 9 0.8144 21 0.8226 10 0.7631 22 0.9902 11 0.8257 23 0.8812 12 0.9099 24 0.9169 Bootstrapping Results R2 Data Mean: 0.8046 STDEV: 0.1131 Conf 95% 0.0453 or 75.93 - 84.98% Conf 99% 0.0595 or 74.51 - 86.41% So what does this mean for the results of the regression? Can we rely on this model to help predict the number of points per game that will be scored by the 2006 team? Questions?

DOCUMENT INFO

Shared By:

Categories:

Tags:
forecasting methods, time series, econometric methods, forecast accuracy, quantitative methods in marketing, expert systems, bootstrap method, ar model, monte carlo simulation, forecast errors, prediction intervals, role playing, domain knowledge, j. scott armstrong, lead time

Stats:

views: | 41 |

posted: | 9/27/2010 |

language: | English |

pages: | 36 |

OTHER DOCS BY lqz95924

How are you planning on using Docstoc?
BUSINESS
PERSONAL

By registering with docstoc.com you agree to our
privacy policy and
terms of service, and to receive content and offer notifications.

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.