Testing Bayesian Updating with the AP Top 25 by akgame

VIEWS: 11 PAGES: 23

									Testing Bayesian Updating with the AP Top 25




              Testing Bayesian Updating with the AP Top 25

                                                Daniel F. Stone

                                               Oregon State University


                                               IAREP/SABE 2009
Testing Bayesian Updating with the AP Top 25
   Introduction




“Are people Bayesian?” is still an open question




                  While there’s an extensive literature on the subject, it is mostly
                  experimental

                  Also, the experimental evidence is conflicting; some studies find
                  people are mostly Bayesian, some find they aren’t

                  The main types of non-Bayesian behavior, overreaction and
                  underreaction to new information, are at odds
Testing Bayesian Updating with the AP Top 25
   Introduction




The reason there are not many (any?) field studies that
directly test Bayesian updating is not mysterious



        Dominitz (“Earnings expectations, revisions, and realizations,” ReStat,
        1998):

        “This component of the analysis calls attention to the breadth of data
        required to assess the responsiveness of expectations to new
        information.”
Testing Bayesian Updating with the AP Top 25
   Introduction




I address these issues using the AP Top 25 college football
poll



                  This is a weekly subjective ranking of the top 25 (out of >100)
                  teams conducted by 64 experts every season

                  It is a unique non-experimental data source in that it allows direct
                  observation of the sequentially updated beliefs of experts, with
                  observable signals

                  The data set is also very rich, which gives insight into the causes of
                  the different types of belief updating behavior
Testing Bayesian Updating with the AP Top 25
   Introduction




I find the voters sometimes Bayesian update, overreact and
underreact



                  The voters are on average close to Bayesian, but they overreact to
                  losses by top 1-5 teams and underreact to wins by top 21-25 teams

                  The voters under-respond to other aspects of the signal (besides
                  win/lose), but less so for losses and wins against ranked opponents
                  (more salient signal types)

                  I interpret this to imply voters are more Bayesian than experimental
                  subjects, but face information processing costs
Testing Bayesian Updating with the AP Top 25
   Introduction




The results support and clarify previous literature



                  Barberis/Thaler (Handbook of Economics of Finance, 2003):
                        Salience reconciles representativeness/conservatism, but no direct
                        support (?)


                  Strength vs weight
                        Tversky/Griffin (1992): signals with high strength (extremity) and
                        low weight (credibility) lead to overconfidence (overreaction)

                        Kraemer/Webber (JRU, 2004): information with low strength and
                        high weight is overreacted to if the weight is presented clearly
Testing Bayesian Updating with the AP Top 25
   The Data




The data include almost all of the voter ballots from the
2006-08 seasons
Testing Bayesian Updating with the AP Top 25
   The Data




The data set has some other nice features



                Teams play exactly 1 or 0 games per week, thus, at most one major
                signal per team between rankings

                The historical score data is freely available and observed by voters
                (distributions common knowledge)

                So, voters should Bayesian update rankings; econometrician can
                estimate whether they do
Testing Bayesian Updating with the AP Top 25
   Methodology




I estimate the Bayesian posterior ranks, and compare them
to the observed posterior ranks


                 For each week and voter, I estimate what his/her rankings should be
                 the following week, conditional on an information set

                 For example, if voter Smith ranked Oregon State number 5 in week
                 2 of the 2006 season:
                        5 is the prior rank for that team-voter-week-season
                        If Oregon State beat Oregon by 5 at home, that is the signal
                        If Smith ranks Oregon State number 4 in week 3, 4 is the observed
                        posterior rank (for the team-voter-week-season)
                 I estimate the Bayesian posterior rank (for Oregon
                 State-Smith-3-2006), and compare the Bayesian rank change to the
                 observed change (5 - 4 =1)
Testing Bayesian Updating with the AP Top 25
   Methodology




This method requires that I define the true rankings



                 The AP has not clearly defined the rankings’ criteria; I define the
                 true rankings as the final rankings (by voter-season)

                 This is problematic if voters rank teams on quality and this changes
                 throughout the season, but I show this not to be the case Table 1

                 The truth assumption enables me to estimate the prior and signal
                 distributions using historical data (empirical distributions)

                 The Bayesian posterior rankings can then be computed with a fairly
                 straightforward application of Bayes’ rule
Testing Bayesian Updating with the AP Top 25
   Methodology




The precision of priors is greatest for the highest ranked
teams


                       75%                                        75%
                                                            Final Rank: Top 10
                                                            Final Rank: Top 11−25
                                                            Final Rank: Unranked



                       50%                                       50%




                       25%                                       25%




                        0%                                        0%
                                 1−5   6−10 11−15 16−20 21−25              1−5      6−10 11−15 16−20 21−25
                                       Rank Prior to Wins                           Rank Prior to Losses

           Final (week 16/17) rank frequencies, by prior rank and win/loss signal; weeks 1-7,
                                          2006-08 seasons
Testing Bayesian Updating with the AP Top 25
   Results




Before testing Bayesian updating, I first test the validity of
the estimated Bayesian posteriors

                                 5

                                               Bayesian
                                               Observed
                               4.5



                                 4



                               3.5



                                 3



                               2.5
                                                          Voters


             Mean Bayesian posterior (“Bayesian”), observed posterior (“Observed”) absolute
             deviation from final rank, by voter, in order of lowest to highest mean observed
                                           absolute deviation
Testing Bayesian Updating with the AP Top 25
   Results




Summary stats preview some of the main results


                            Mean Rank Improvement: Prior - Posterior Rank
                           Rank          Wins                 Losses
                                  Actuals Estimates Actuals Estimates
                           1-5     0.06       0.00     -7.65        -5.68
                                 ( 1.29 )   ( 1.41 )  ( 3.60 )     ( 5.67 )
                           6-10    0.74       0.34      -8.37        -8.84
                                 ( 2.05 )   ( 2.46 )  ( 4.22 )     ( 4.64 )
                           11-15   1.43       0.96      -6.48        -8.09
                                 ( 2.49 )   ( 3.47 )  ( 3.17 )     ( 2.74 )
                           16-20   2.29       2.55      -3.92        -4.21
                                 ( 2.88 )   ( 3.53 )  ( 2.13 )     ( 2.03 )
                           21-25   2.65       4.90      -0.42        -0.36
                                 ( 2.54 )   ( 3.63 )  ( 1.02 )     ( 1.06 )
Testing Bayesian Updating with the AP Top 25
   Results




To test Bayesian updating, I construct a variable
representing overreaction
                                  A       E
                                ∆ri,t − ∆ri,t   if i wins in t,
        OVERi,t =                 E       A
                                ∆ri,t − ∆ri,t   otherwise,
                     j
        in which ∆ri,t = rank improvement for i = team-voter-season, t = week
        (j ∈ {A, E }, A=actual (observed), E =estimate)



                OVER represents overreaction to new information
                This is excess rank improvement following wins, and excess rank
                decline following losses
                A negative value of OVER indicates underreaction
                I regress OVER on variables that should not affect it under null
                (Bayesian updating) separately for wins and losses
Testing Bayesian Updating with the AP Top 25
   Results




Voters are insensitive to less salient characteristics of wins1

                                                                           OVER (wins only)
               Home game dummy                                                1.727***
                                                                                (0.228)
               Own score minus opponent score                                 -0.079***
                                                                                (0.008)
               Opponent ranked dummy                                          -1.523***
                                                                                (0.357)
               Opponent ranked × home minus away score                          0.039**
                                                                                (0.019)
               Top 11-15                                                       1.178***
                                                                                (0.366)
               Top 21-25                                                      -1.722***
                                                                                (0.374)
            1 (N= 21,813) OLS regressions with bootstrap standard errors clustered by game. Other
        regressors include dummies for top 1-5 rank, top 6-10 rank (constant dropped), voter and year
        fixed effects (FE), voter FE interacted with week of season, dummy variables for team in same
        state, region as voter, voter experience, difference between aggregate and individual rank and
        previous year aggregate rank.
Testing Bayesian Updating with the AP Top 25
   Results




Voters are more responsive to, but still not fully
appreciative of, less salient characteristics of losses2
                                                                          OVER (losses only)
               Home game dummy                                               -1.330***
                                                                              (0.477)
               Own minus opp. score                                            0.001
                                                                              (0.032)
               Opponent ranked dummy                                           -0.611
                                                                              (0.673)
               Opponent ranked × own minus opp. score                          -0.011
                                                                              (0.034)
               Top 1-5                                                         3.059*
                                                                              (1.564)
               Top 11-15                                                       -0.346
                                                                              (0.838)
               Top 21-25                                                        1.279
                                                                              (0.809)
             2 (N= 6,678) OLS regressions with bootstrap standard errors clustered by game.
Testing Bayesian Updating with the AP Top 25
   Results




The general trends differ by rank group, indicating
unawareness of variation in prior strength3
                                                         Top 1-5        Top 11-15        Top 21-25
             Wins         Home                            0.308          2.419***         3.126***
                                                         ( 0.21 )        ( 0.391 )         (0.378)
                          Own minus opp. score             -0.01        -0.079***        -0.095***
                                                        ( 0.008 )        ( 0.012 )         (0.011)
                          Constant                        0.188           0.786**        -2.037***
                                                        ( 0.221 )        ( 0.393 )         (0.323)
             Losses       Home                            -1.917        -1.410***        -0.200***
                                                        ( 2.095 )        ( 0.499 )         (0.063)
                          Own minus opp. score          1.068***         0.178***          0.013**
                                                        ( 0.384 )        ( 0.066 )         (0.005)
                          Constant                          3.15         -1.479**           0.122*
                                                        ( 2.087 )        ( 0.618 )         (0.069)
            3 Dep. var = OVER for all regressions, estimated separately by prior rank group. OLS
        regressions with bootstrap standard errors clustered by game. Only other regressor is difference
        between aggregate and individual rank and previous year aggregate rank.
Testing Bayesian Updating with the AP Top 25
   Results




Evidence suggests the same voters both under/overreact

                         opponent not ranked, score margin > 20)     0

                                                                   −0.5
                            E(OVER|voter,win, away game,




                                                                    −1

                                                                   −1.5

                                                                    −2

                                                                   −2.5

                                                                    −3

                                                                   −3.5

                                                                    −4
                                                                     −3   −2     −1       0       1       2        3         4        5    6
                                                                          E(OVER|win, home game, opponent not ranked, score margin < 15)

             Mean voter-level OVER following observation of “bad” wins (non-salient negative
              information; x-axis) vs. “good” wins (non-salient positive information; y-axis)
Testing Bayesian Updating with the AP Top 25
   Robustness




The patterns are confirmed with an alternative
methodology


        Estimated priors, observed game results for top 1-5 teams in   2006
         Year Week               Estimated:           Estimated:       Observed:
                          E(rank|ranked in top 25) Pr(unranked)         Pr(Lose)
         2006     1                 6.29                 0.01%           1.56%
                  2                 6.37                 0.01%          18.89%
                  3                 8.68                 2.37%          23.75%
                  4                 5.73                 0.02%           0.00%
                  5                 6.60                 0.00%           0.38%
                  6                 8.15                 3.85%          20.32%
                  7                 7.88                 1.62%          20.13%
Testing Bayesian Updating with the AP Top 25
   Robustness




The patterns are broadly similar in other seasons4



                                                                      Observed        Observed
                                                                   Posterior Rank    Final Rank
          Losing Top 1-5 Teams                                     10.52 ( 0.34 )   13.35 ( 1.71 )
          Winning Top 6-10 Teams                                    7.01 ( 0.11 )   13.67 ( 0.74 )
          p-value (H0 : Losing = Winning)                               0.00             0.86
          Home Top 16-20 Teams, Close Wins                         15.67 ( 0.47 )   27.55 ( 2.49 )
          Away Top 21-25 Teams, Big Wins                           18.98 ( 0.48 )   21.58 ( 1.91 )
          p-value (H0 : Home Close = Away Big)                          0.00             0.06




             4 Weeks 1-6 aggregate polls from 1991-2005 seasons.
Testing Bayesian Updating with the AP Top 25
   Concluding Remarks




Voters are sometimes Bayesian, but also sometimes rely on
heuristics that cause systematic errors



                Although voters are experts, have time and incentives to do a good
                job on their rankings, they seem to be overwhelmed by the task

                They thus seem to process only the most salient information–margin
                of loss–in a Bayesian way

                They are also overconfident in the precision of rankings of low-ranked
                teams, and under-confident in the precision of top-ranked teams
Testing Bayesian Updating with the AP Top 25
   Concluding Remarks




Directions for future work




                Heterogeneity

                Structural belief updating

                Social learning vs conformity
Testing Bayesian Updating with the AP Top 25
   Appendix




        Table: Tests of H0 : mean score differences conditional on final rank groups
        constant throughout season


            Home                     Away         Period        s
                                                                ¯            p-value for
          Final Rank              Final Rank                               s           s
                                                                      H0 : ¯Aug −Oct = ¯Oct−Dec
              1-12                   13-25      Aug-Oct 15     13.6              0.14
              1-12                   13-25     Oct 16-Dec 15   16.9
              1-12                Unranked      Aug-Oct 15     22.6              0.35
              1-12                Unranked     Oct 16-Dec 15   21.0
             13-25                    1-12      Aug-Oct 15     -7.0              0.35
             13-25                    1-12     Oct 16-Dec 15   -4.7
             13-25                Unranked      Aug-Oct 15     15.5              0.09
             13-25                Unranked     Oct 16-Dec 15   12.9
                                                            s
         Notes: “Final Rank” = final AP aggregate rank; ¯ = mean home score - away score;
         ˆs                          s
         σ¯ = estimated std error of ¯. Sample includes games played 1989-2008 with at least
         one Division I-A team on non-neutral field. “Unranked” restricted to teams receiving
                  votes in final aggregate poll in at least one of previous two seasons.

         Back

								
To top