Alternative Forecasting Methods Bootstrapping

Document Sample
Alternative Forecasting Methods Bootstrapping Powered By Docstoc
					Alternative Forecasting
Methods: Bootstrapping
       Bryce Bucknell
         Jim Burke
         Ken Flores
         Tim Metts
Agenda
               Scenario

              Obstacles

          Regression Model

            Bootstrapping

         Applications and Uses

               Results
Scenario
You have been recently hired as the statistician for the University of
Notre Dame football team. You are tasked with performing a statistical
analysis for the first year of the Charlie Weis era. Specifically, you have
been asked to develop a regression model that explains the relationship
between key statistical categories and the number of points scored by the
offense. You have a limited number of data points, so you must also find
a way to ensure that the regression results generated by the model are
reliable and significant.

Problems/Obstacles:
   Central Limit Theorem
   Replication of data
   Sampling
   Variance of error terms
Constrained by the Central Limit Theorem
     In selecting simple random samples of size n from a population, the
                                                _
     sampling distribution of the sample mean x can be approximated by a
     normal probability distribution as the sample size becomes large. It
     is generally accepted that the sample size must be 30 or greater to
     satisfy the large-sample condition of the theorem.




                                  Sample N = 1                                 Sample N = 2




                                  Sample N = 3                                 Sample N = 4

1. http://www.statisticalengineering.com/central_limit_theorem_(summary).htm
Central Limit Theorem
 Central Limit theorem is the foundation for many statistical
 procedures, because the distribution of the phenomenon under
 study does NOT have to be Normal because its average WILL tend to
 be normal.


 Why is the assumption of a normal distribution important?
     A normal distribution allows for the application of the empirical rule – 68%,
      95% and 99.7%

     Chebyshev’s Theorem no more than 1/4 of the values are more than 2
      standard deviations away from the mean, no more than 1/9 are more than
      3 standard deviations away, no more than 1/25 are more than 5 standard
      deviations away, and so on.

     The assumption of a normally distributed data allows descriptive statistics
      to be used to explain the nature of the population
Not enough data available?
Monte Carlo simulation, a type of spreadsheet simulation, is used to
randomly generate values for uncertain variables over and over to
simulate a model.


   Monte Carlo methods randomly select values to create scenarios
   The random selection process is repeated many times to create multiple
    scenarios
   Through the random selection process, the scenarios give a range of
    possible solutions, some of which are more probable and some less
    probable
   As the process is repeated multiple times, 10,000 or more, the average
    solution will give an approximate answer to the problem
   The accuracy can be improved by increasing the number of scenarios
    selected
Sampling without Replacement
 Simple Random Sampling


   A simple random sample from a population is a sample chosen
    randomly, so that each possible sample has the same probability
    of being chosen.

    In small populations such sampling is typically done "without
    replacement“

   Sampling without replacement results in deliberate avoidance of
    choosing any member of the population more than once

   This process should be used when outcomes are mutually
    exclusive, i.e. poker hands
Sampling with Replacement

   Initial data set is not sufficiently large enough to use simple
    random sampling without replacement

   Through Monte Carlo simulation we have been able to replicate
    the original population

   Units are sampled from the population one at a time, with each
    unit being replaced before the next is sampled.

   One outcome does not affect the other outcomes

   Allows a greater number of potential outcomes than sampling
    without replacement

   If observations were not replaced there would not be enough
    independent observations to create a sample size of n ≥ 30
Hetroscedasticity vs. Homoscedasticity




                                                           Residuals
    Residuals




                                               X                                                X




Homoscedasticity – constant variance                  Hetroscedasticity – nonconstant variance

               All random variables have the same           Random variables may have different
                finite variance                               variances

               Simplifies mathematical and                  Standard errors of regression
                                                              coefficients may be understated
                computational treatment
                                                             T-ratios may be larger than actual
               Leads to good estimation results in
                data mining and regression                   More common with cross sectional
                                                              data
Regression Model For ND Points Scored
  ND Points = 38.54 + 0.079*b1 - 0.170*b2 - 0.662*b3 - 3.16*b4



            b1 = Total Yards Gained                                  b3 = Total Plays


            b2 = Penalty Yards                                       b4 = Turnovers



 Audit Trail -- Coefficient Table (Multiple Regression Selected)
       Series                 Included                           Standard                        Overall
   Description                in Model         Coefficient           Error    T-test    F-test    F-test
    ND Points                Dependent              38.54           14.26       2.70     7.31      8.92
    Total YDS                       Yes               0.08            0.02      5.29    27.97
   Penalty YDS                      Yes              -0.17            0.06     -2.64     6.99
    Total Plays                     Yes              -0.66            0.23     -2.84     8.05
    Turnovers                       Yes              -3.16            2.50     -1.26     1.59
4 Checks of a Regression Model

1. Do the coefficients have the correct sign?

2. Are the slope terms statistically
significant?
3. How well does the model fit the data?

4. Is there any serial correlation?
4 Checks of a Regression Model

1. Do the coefficients have the correct sign?

       Audit Trail -- Coefficient Table
             Series                 Included
         Description                in Model   Coefficient
          ND Points                Dependent        38.54
          Total YDS                      Yes          0.08
         Penalty YDS                     Yes         -0.17
          Total Plays                    Yes         -0.66
          Turnovers                      Yes         -3.16


     Could this represent a big play factor?
4 Checks of a Regression Model

2. Are the slope terms statistically significant?



 Audit Trail -- Coefficient Table (Multiple Regression Selected)
3. How well does the model fit the
       Series                 Included                           Standard           data?     Overall
   Description                in Model         Coefficient           Error T-test    F-test    F-test
    ND Points                Dependent              38.54           14.26    2.70     7.31      8.92
     Is there any serial correlation?
4. Total YDS                        Yes               0.08            0.02   5.29    27.97
   Penalty YDS                      Yes              -0.17            0.06  -2.64     6.99
    Total Plays                     Yes              -0.66            0.23  -2.84     8.05
    Turnovers                       Yes              -3.16            2.50  -1.26     1.59
                                               M




                                                          0
                                                              10
                                                                   20
                                                                        30
                                                                             40
                                                                                  50
                                                                                       60
                                                ay
                                                    -0
                                               Ju 5
                                                  n-
                                                     05
                                                Ju
                                                   l-
                                               Au 05
                                                  g-
                                                     0
                                               Se 5
                                                  p-
                                                     0
                                               O 5
                                                 ct
                                                   -0
                                               N 5
                                                 ov
                                                    -0
                                               D 5
                                                 ec
                                                    -0
                                               Ja 5
                                                  n-
                                                     0




                       ND Points
                                               Fe 6
                                                  b-
                                                     0
                                               M 6
                                                 ar
                                                    -0
                                               Ap 6
                                                  r-
                                                     0
                                               M 6
                                                ay
                                                    -0
                                               Ju 6
                                                  n-
                                                                                            ND Points




                                                     06
                                                Ju
                                                   l-
                       Forecast of ND Points   Au 06
                                                  g-
                                                     0
                                               Se 6
                                                  p-
                                                     0
                                               O 6
                                                 ct
                                                   -0
Adjusted R2 = 74.22%                           N 6
                                                 ov
                                                    -0
                                               D 6
                       Fitted Values




                                                 ec
                                                    -0
                                               Ja 6
                                                  n-
                                                     0
                                               Fe 7
                                                  b-
                                                     0
                                               M 7
                                                 ar
                                                    -0
                                               Ap 7
                                                                                                                                                   4 Checks of a Regression Model




                                                  r-
                                                     07
                                                                                                        3. How well does the model fit the data?
4 Checks of a Regression Model
4. Is there any serial correlation?
 Data is cross sectional



 With limited data points, how useful is this
 regression in describing how well the model fits the
 actual data? Is there a way to tests its reliability?
How to test the significance of the analysis
    What happens when the sample size is not large enough (n ≥ 30)?


    Bootstrapping is a method for estimating the sampling distribution of an
    estimator by resampling with replacement from the original sample.


    Commonly used statistical significance tests are used to determine
     the likelihood of a result given a random sample and a sample size
     of n.
    If the population is not random and does not allow a large enough
     sample to be drawn, the central limit theorem would not hold true
    Thus, the statistical significance of the data would not hold
    Bootstrapping uses replication of the original data to simulate a
     larger population, thus allowing many samples to be drawn and
     statistical tests to be calculated
How It Works
Bootstrapping is a method for estimating the sampling distribution of an
estimator by resampling with replacement from the original sample.

    The bootstrap procedure is a means of estimating the statistical accuracy . .
    . from the data in a single sample.

    Bootstrapping is used to mimic the process of selecting many samples when
    the population is too small to do otherwise

    The samples are generated from the data in the original sample by copying
    it many number of times (Monte Carlo Simulation)

    Samples can then selected at random and descriptive statistics calculated or
    regressions run for each sample

   The results generated from the bootstrap samples can be treated as if it they
    were the result of actual sampling from the original population
Characteristics of Bootstrapping


        Sampling with   Full Sample
         Replacement
                                                    Random sampling with
 Bootstrapping Example                              replacement can be
                                                    employed to create
                                                    multiple independent
                      Limited                       samples for analysis
                     number of
Original Data Set   observations                   1st Random Sample
    Pittsburgh                                             Navy
    Michigan                                            Ohio State
  Michigan State                                            USC
   Washington                                           Washington
     Purdue                                             Ohio State
       USC
                               109 Copies                   USC
                                 of each
       BYU                     observation                  BYU
    Tennessee                                            Stanford
      Navy                                               Pittsburgh
    Syracuse                         Creating a         Ohio State
                                    much larger
    Stanford                        sample with          Stanford
                                   which to work
    Ohio State                                           Michigan
When it should be used
Bootstrapping is especially useful in situations when no analytic formula
for the sampling distribution is available.


     Traditional forecasting methods, like exponential
      smoothing, work well when demand is constant
      – patterns easily recognized by software
     In contrast, when demand is irregular, patterns
      may be difficult to recognize.
     Therefore, when faced with irregular demand,
      bootstrapping may be used to provide more
      accurate forecasts, making some important
      assumptions…
Assumptions and Methodology

    Bootstrapping makes no assumption regarding the population

    No normality of error terms

    No equal variance

    Allows for accurate forecasts of intermittent demand

    If the sample is a good approximation of the population, the
     sampling distribution may be estimated by generating a large
     number of new samples

    For small data sets, taking a small representative sample of the data
     and replicating it will yield superior results
Applications and Uses
  Criminology
     Statistical significance testing is important in
      criminology and criminal justice

     Six of the most popular journals in
      criminology and criminal justice are
      dominated by quantitative methods that rely
      on statistical significance testing

     However, it poses two potential problems:
      tautology and violations of assumptions
Applications and Uses
  Criminology
     Tautology: the null hypothesis is always false
      because virtually all null hypothesis may be
      rejected at some sample size

     Violation of assumptions of regression: errors
      are homogeneous and errors of independent
      variables are normally distributed

     Bootstrapping provides a user-friendly
      alternative to cross-validation and jackknife to
      augment statistical significance testing
Applications and Uses
  Actuarial Practice
     Process of developing an actuarial model
      begins with the creation of probability
      distributions of input variables
     Input variables are generally asset-side
      generated cash flows (financial) or cash flows
      generated from the liabilities side
      (underwriting)
     Traditional actuarial methodologies are rooted
      in parametric approaches, which fit prescribed
      distribution of losses to the data
Applications and Uses
  Actuarial Practice
      However, experience from the last two decades has
       shown greater interdependence of loss variables with
       asset variables
      Increased complexity has been accompanied by
       increased competitive pressures and more frequent
       insolvencies
      There is a need to use nonparametric methods in
       modeling loss distributions
      Bootstrap standard errors and confidence intervals
       are used to derive the distribution
Applications and Uses
  Classifications Used by Ecologists
      Ecologists often use cluster analysis as a tool in the
       classification and mapping of entities such as
       communities or landscapes
      However, the researcher has to choose an adequate
       group partition level and in addition, cluster analysis
       techniques will always reveal groups
      Use bootstrap to test statistically for fuzziness of the
       partitions in cluster analysis
      Partitions found in bootstrap samples are compared
       to the observed partition by the similarity of the
       sampling units that form the groups.
Applications and Uses
   Human Nutrition
      Inverse regression used to estimate vitamin
       B-6 requirement of young women
      Standard statistical methods were used to
       estimate the mean vitamin B-6 requirement
      Used bootstrap procedure as a further check
       for the mean vitamin B-6 requirement by
       looking at the standard error estimates and
       confidence intervals
Application and Uses
  Outsourcing
     Agilent Technologies determined it was time to
      transfer manufacturing of its 3070 in-circuit test
      systems from Colorado to Singapore
     Major concern was the change in environmental test
      conditions (dry vs humid)
     Because Agilent tests to tighter factory limits (“guard
      banding”), they needed to adjust the guard band for
      Singapore
     Bootstrap was used to determine the appropriate
      guard band for Singapore facility
An Alternative to the bootstrap

 Jackknife
    A statistical method for estimating and
     removing bias* and for deriving robust
     estimates of standard errors and
     confidence intervals
    Created by systematically dropping out
     subsets of data one at a time and
     assessing the resulting variation

Bias: A statistical sampling or testing error caused by systematically favoring some outcomes
over others
A comparison of the Bootstrap & Jackknife


    Bootstrap                          Jackknife
        Yields slightly different          Less general technique
         results when repeated              Explores sample
         on the same data                    variation differently
         (when estimating the               Yields the same result
         standard error)                     each time
        Not bound to                       Similar data
         theoretical                         requirements
         distributions
Another alternative method

 Cross-Validation
    The practice of partitioning data into a
     sample of data into sub-samples such that
     the initial analysis is conducted on a single
     sub-sample (training data), while further
     sub-samples (test or validation data) are
     retained “blind” in order for subsequent
     use in confirming and validating the initial
     analysis
Bootstrap vs. Cross-Validation

    Bootstrap                    Cross-Validation
        Requires a small of          Not a resampling
         data                          technique
        More complex                 Requires large
         technique – time              amounts of data
         consuming                    Extremely useful in
                                       data mining and
                                       artificial intelligence
Methodology for ND Points Model

    Use bootstrapping on ND points scored
     regression model

    Goal: determine the reliability of the model

    Replication, random sampling, and numerous
     independent regression

    Calculation of a confidence interval for adjusted
     R2
Bootstrapping Results
 R2 Data
 Sample #   Adjusted R^2   Sample #   Adjusted R^2   The Mean, Standard
    1         0.7351         13         0.7482       Dev., 95% and 99%
    2         0.7545         14         0.8719       confidence intervals
    3         0.7438
                                                     are then calculated in
                             15         0.7391
                                                     excel from the 24
    4         0.7968         16         0.9025       observations
    5         0.5164         17         0.8634
    6         0.6449         18         0.7927
    7         0.9951         19         0.6797
    8         0.9253         20         0.6765
    9         0.8144         21         0.8226
   10         0.7631         22         0.9902
    11        0.8257         23         0.8812
   12         0.9099         24         0.9169
Bootstrapping Results

 R2 Data
 Mean:       0.8046
 STDEV:      0.1131

 Conf 95%    0.0453 or 75.93 - 84.98%
 Conf 99%    0.0595 or 74.51 - 86.41%




  So what does this mean for the results of the
  regression?
 Can we rely on this model to help predict the
 number of points per game that will be scored by
 the 2006 team?
Questions?