Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

Chen

VIEWS: 3 PAGES: 49

									Information Aggregation:
Experiments and Industrial
Applications




 Kay-Yut Chen
 HP Labs
Agenda

•   Lessons from HP Information Markets
    (Chen and Plott 2002)




•   Scoring Rules and Identification of Experts
    (Chen, Fine and Huberman 2004)
    (Chen and Hogg 2004)



•   Public Information
    (Chen, Fine and Huberman 2004)


                                Experimental Economics Program
HP Information Markets (Chen and Plott)
•   Summary of Events

    –   12 events, from 1996 to 1999

    –   11 events sales related

    –   8 events had official forecasts

•   Methodology & Procedures

    –   Contingent state asset (i.e. winning ticket pays $1, others $0)

    –   Sales amount (unit/revenue) divided into (8-10) finite intervals

    –   Web-based real time double-auction

    –   15-20 min phone training for EVERY subject

    –   Market open for one week at restricted time of the day (typically lunch and after hours)

    –   Market size: 10-25 people

                                             Experimental Economics Program
                             Event 2

     0.3
    0.25                                                        IAM Distribution
     0.2
    0.15                                                        Actual Outcome
P




     0.1
                                                                HP Official
    0.05                                                        Forecast
      0                                                         IAM Prediction
           0   100              200                       300
                     $



                         Experimental Economics Program
Results
                                                                Abs % Errors of IAM Predictions

                                             Last Interval Ignored                     Last Interval Mass at
                                                                                           Lower Bound
     Event       Absolute %     Average      Average            Average     Average    Average Average
                 errors of HP   last 60%     last 50%           last 40%    last 60%   last 50% last 40%
                  forecasts       trade         trade             trade       trade      trade       trade
       2           13.18%         4.61%        4.57%              4.68%       5.63%      5.68%       5.80%
       3           59.55%        57.48%       55.72%             54.60%      59.25%     57.46%      56.32%
       4            8.64%         7.84%        8.15%              8.52%       6.45%      6.77%       7.13%
       5           32.08%        30.93%       31.57%             31.83%      29.74%     30.33%      30.48%
       6           29.69%        24.23%       24.54%             25.30%      22.94%     23.22%      23.93%
       7            4.10%         7.33%        7.02%              6.71%       5.35%      4.91%       4.55%
       8            0.11%         2.00%        2.35%              1.83%       1.53%      1.39%       1.00%
       9           28.31%        23.85%       24.85%             24.39%      17.55%     17.32%      16.54%


T-test P-value                   0.079         0.084              0.071      0.034      0.026       0.022



Random variable x=official error – market error
H0: mean of x=0 Alternate:
mean of x>0
                                           Experimental Economics Program
Business Constraints and Research Issues

•   Not allowed to “bet” players’ own money -> stakes limited to an average of $50 per person


•   Time horizon constraints -> 3 months to be useful


•   Recruit the “right” people


•   Asset design affects the results (How to set the intervals?)


•   Thin markets (sum of price ~ $1.11 to $1.31 over the dollar)


    –   Few players


    –   Not enough participation




                                         Experimental Economics Program
Reporting with Scoring Rule



                                                          Outcome

              A                  B                          C

              Reports of Probability Distribution


              p1                  p2                        p3


                   Pays C1+C2*Log(p3)


                         Experimental Economics Program
Information Aggregation Function




       If reports are independent, Bayes Law applies …



                                   p s1 p s2 ...p s N
              Ps | I  
                              p
                               s
                                             s1   p s2 ...p s N




                             Experimental Economics Program
         Two Complications




•   Non-Risk Neutral Behavior



•   Public Information




                  Experimental Economics Program
                 Dealing with Risks Attitudes:
                   Two-Stage Mechanism




       Event 1
                                                  Stage 1: Information Market
       Event 2

       Event 3
                                            Call Market to Solicit Risk Attitudes
Time




       Event 4

       Event 5
                                    Stage 2: Probability Reporting & Aggregation
       Event 6


       Event 7
                                     Individual Report of Probability Distribution
                                            Nonlinear Aggregated Function
       Event 8




                             Experimental Economics Program
Second Stage: Aggregation Function




      Bayes Law with Behavioral Correction

                                                1        2               N
                  Ps | I  
                               p p ...p         1s        2s               Ns

                               p1s p2s ...p
                                 1  2

                                          s
                                                                             N
                                                                             Ns

                                                                                         Normalizing constant
                                                                                          for individual risks




                                     i=r(V i / i)c
                                                                              Holding value/Risk
          “market” risk
                                                                              - measure relative risk of individuals
          ~sum of prices/winning payoff



                                          Experimental Economics Program
                  Experiments:
           Inducing Diverse Information


                                                                  Outcome

                 A                       B                            C
Box of Balls




                A                                 C                       C
                                                                              Random Draws
                                                                               Provide Info
                               B                                  C

                     * In actual experiments, there are TEN states



                                 Experimental Economics Program
              Comparison To All Information Probability
                          Kullback-Leibler = 1.453

0.900
   0.9
                                                                      Omniscient
   0.8
0.800
                                                                      No Info
   0.7
0.700
   0.6
0.600
Probability




   0.5
0.500                                                                      Series1
   0.4
0.400                                                                      Series2
   0.3
0.300
   0.2
0.200
   0.1
0.100
   0.0
0.000         A       B       C       D    E         F       G   H     I      J
              1   2       3       4   5   6 States
                                               7         8   9   10




                      Experiment 4, Period 17
                         No Information
       Kullback-Leibler Measure



•   Relative entropy

•   Always >=0

•   =0 if two distributions are identical

•   Addictive for independent events




                       Experimental Economics Program
               Comparison To All Information Probability
                          Kullback-Leibler = 1.337

0.900
    0.9
                                                                Omniscient
    0.8
0.800
                                                                IA Mechanism
    0.7
0.700
    0.6
0.600
 Probability




    0.5
0.500                                                                 Series1
    0.4
0.400                                                                 Series2
    0.3
0.300
    0.2
0.200
    0.1
0.100
    0.0
0.000           A     B     C    D    E      F        G    H      I     J
               11   2 2 3   34   4
                                 5   65 States6
                                           7      8   79    8
                                                           10     9     10




                      Experiment 4, Period 17
                             1 Player
               Comparison To All Information Probability
                               Kullback-Leibler = 1.448

0.900
    0.9
                                                                         Omniscient
    0.8
0.800
                                                                         IA Mechanism
    0.7
0.700
    0.6
0.600
 Probability




    0.5
0.500                                                                          Series1
    0.4
0.400                                                                          Series2
    0.3
0.300
    0.2
0.200
    0.1
0.100
    0.0
0.000              A       B       C   D       E        F       G   H      I     J
               1       2       3   4   5   6       States
                                                      7     8   9   10




                           Experiment 4, Period 17
                            2 Players Aggregated
               Comparison To All Information Probability
                               Kullback-Leibler = 1.606

0.900
    0.9
                                                                             Omniscient
    0.8
0.800
                                                                             IA Mechanism
    0.7
0.700
    0.6
0.600
 Probability




    0.5
0.500                                                                              Series1
    0.4
0.400                                                                              Series2
    0.3
0.300
    0.2
0.200
    0.1
0.100
    0.0
0.000              A       B       C   D       E        F       G       H      I     J
               1       2       3   4   5   6       States
                                                      7     8       9   10




                           Experiment 4, Period 17
                            3 Players Aggregated
               Comparison To All Information Probability
                               Kullback-Leibler = 1.362

0.900
    0.9
                                                                         Omniscient
    0.8
0.800
                                                                         IA Mechanism
    0.7
0.700
    0.6
0.600
 Probability




    0.5
0.500                                                                          Series1
    0.4
0.400                                                                          Series2
    0.3
0.300
    0.2
0.200
    0.1
0.100
    0.0
0.000              A       B       C   D       E        F       G   H      I     J
               1       2       3   4   5   6       States
                                                      7     8   9   10




                           Experiment 4, Period 17
                            4 Players Aggregated
               Comparison To All Information Probability
                               Kullback-Leibler = 0.905

0.900
    0.9
                                                                             Omniscient
    0.8
0.800
                                                                             IA Mechanism
    0.7
0.700
    0.6
0.600
 Probability




    0.5
0.500                                                                              Series1
    0.4
0.400                                                                              Series2
    0.3
0.300
    0.2
0.200
    0.1
0.100
    0.0
0.000              A       B       C   D       E        F       G       H      I     J
               1       2       3   4   5   6       States
                                                      7     8       9   10




                           Experiment 4, Period 17
                            5 Players Aggregated
               Comparison To All Information Probability
                               Kullback-Leibler = 1.042

0.900
    0.9
                                                                         Omniscient
    0.8
0.800
                                                                         IA Mechanism
    0.7
0.700
    0.6
0.600
 Probability




    0.5
0.500                                                                          Series1
    0.4
0.400                                                                          Series2
    0.3
0.300
    0.2
0.200
    0.1
0.100
    0.0
0.000              A       B       C   D       E        F       G   H      I     J
               1       2       3   4   5   6       States
                                                      7     8   9   10




                           Experiment 4, Period 17
                            6 Players Aggregated
               Comparison To All Information Probability
                               Kullback-Leibler = 0.550

0.900
    0.9
                                                                             Omniscient
    0.8
0.800
                                                                             IA Mechanism
    0.7
0.700
    0.6
0.600
 Probability




    0.5
0.500                                                                              Series1
    0.4
0.400                                                                              Series2
    0.3
0.300
    0.2
0.200
    0.1
0.100
    0.0
0.000              A       B       C   D       E        F       G       H      I     J
               1       2       3   4   5   6       States
                                                      7     8       9   10




                           Experiment 4, Period 17
                            7 Players Aggregated
               Comparison To All Information Probability
                               Kullback-Leibler = 0.120

0.900
    0.9
                                                                             Omniscient
    0.8
0.800
                                                                             IA Mechanism
    0.7
0.700
    0.6
0.600
 Probability




    0.5
0.500                                                                              Series1
    0.4
0.400                                                                              Series2
    0.3
0.300
    0.2
0.200
    0.1
0.100
    0.0
0.000              A       B       C   D       E        F       G       H      I     J
               1       2       3   4   5   6       States
                                                      7     8       9   10




                           Experiment 4, Period 17
                            8 Players Aggregated
              Comparison To All Information Probability
                           Kullback-Leibler = 0.133

0.900
   0.9
                                                                  Omniscient
   0.8
0.800
                                                                  IA Mechanism
   0.7
0.700
   0.6
0.600
Probability




   0.5
0.500                                                                   Series1
   0.4                                                                  Series2
0.400
   0.3
0.300
   0.2
0.200
   0.1
0.100
   0.0
0.000          A       B       C       D     E      F  G     H      I     J
              1    2       3       4    5         7
                                            6 States 8   9   10




                       Experiment 4, Period 17
                        9 Players Aggregated
                    Comparison To All Information Probability



              0.9
              0.8
                                                         Omniscient
              0.7                                        IA Mechanism
              0.6                                        market
Probability




              0.5                                        Best Individual

              0.4
              0.3
              0.2
              0.1
              0.0
                    A     B    C     D    E     F    G       H        I    J
                                           States



                         Experiment 4, Period 17
KL Measures for Private Info Experiments

            No           Market                                  Nonlinear Aggregation
                                            Best Player
       Information      Prediction                                     Function


       1.977 (0.312)   1.222 (0.650)       0.844 (0.599)             0.553 (1.057)


       1.501 (0.618)   1.112 (0.594)       1.128 (0.389)             0.214 (0.195)


       1.689 (0.576)   1.053 (1.083)       0.876 (0.646)             0.414 (0.404)


       1.635 (0.570)   1.136 (0.193)       1.074 (0.462)             0.413 (0.260)


       1.640 (0.598)   1.371 (0.661)       1.164 (0.944)             0.395 (0.407)




                                Experimental Economics Program
Group Size Performance




                  Experimental Economics Program
Did the Markets Pick out Experts?

    Group    Exp 1      Exp 2              Exp 3           Exp 4   Exp 5

   Random     1.36      0.93                 1.18          1.12    1.15

    Payoff    1.45      1.09                 1.24          1.13    1.39

    Value     0.72      0.91                 0.94          1.13    1.22

   Optimal    0.53      0.72                 0.75          0.83    0.77


    •KL measure of all query data
    •Pick groups of 3



                          Experimental Economics Program
Did Previous Queries Pick out Experts?


       Group    Exp 1    Exp 2             Exp 3          Exp 4   Exp 5

      Random    1.15     0.92               1.18          1.07    1.21

       Query    0.78     0.89               0.71          0.92    0.81

      Optimal   0.60     0.59               0.69          0.72    0.72



     •KL measure of second half of query data
     •Pick groups of 3


                         Experimental Economics Program
Public Information




       •   Information observed by more than one


       •   Double counting problem




                       Experimental Economics Program
                 Information Aggregation with Public Information
                            Kullback-Leibler = 2.591


              1.2
               1.2
              120.00%
                         Omnicient
                         Omnicient
                            Omnicient
               11
              100.00%    Public
                         Public
                              Public
              0.8        IAM No Info
                          IAM
Probability



               0.8
                80.00%
Probability
Probability




              0.6
               0.6
                60.00%

               0.4
              0.4
                40.00%

               0.2
              0.2
                20.00%

               0
               00.00%
                    AA    B
                         AB    C
                              BC       C D D E E F F
                                         D    E      F   GG
                                                         G    HH
                                                              H    II
                                                                    I   JJ
                                                                        J
                                            States
                                               States
                                            States


                          Public Info Experiment 3, Period 9
                                11 Players Aggregated
Dealing with Public Information:
Add a Game to the Second Stage




       Event 1
                                                   Stage 1: Information Market
       Event 2

       Event 3
                                             Call Market to Solicit Risk Attitudes
Time




       Event 4

       Event 5
                                     Stage 2: Probability Reporting & Aggregation
       Event 6
                                   Individual Report of Probability Distribution
       Event 7                    Matching Game to Recover Public Information
                                     Modified Nonlinear Aggregated Function
       Event 8




                           Experimental Economics Program
Assumptions




    •   Individuals know their public information


    •   Private & Public Info Independent


    •   Structure of Public Info Arbitrary




                         Experimental Economics Program
Matching Game

                                                             Outcome
                  A                  B                         C

                  Reports of Probability Distribution                  Choose player (3) by
                                                                       Max (match function)
   Player 1: q1   q11                  q12                     q13
   Player 2: q2   q21                  q22                     q23

   Player 3: q3   q31                  q32                     q33
        .           .                     .                        .
        .           .                     .                        .
        .           .                     .                        .
     Player 1’s Payoff: (match function)*(C1+C2*Log(q33))
      Match function: f(q1,q2)=(1-0.5*sum(abs(q1i-q2i)))^2
                            Experimental Economics Program
Matching Game



     •   Any match function f(q1,q2) with property

         –   Max when q1=q2

     •   Multiple Equilibria

     •   Payoff increases as entropy decreases

     •   Hopefully, individuals report public information




                               Experimental Economics Program
                Aggregation Function with
               Public Information Correction



Bayes Law with a) Behavioral Correction
               b) Public Info Correction
                                      1             2                 N
                    p1s   p 2s   p Ns 
                     
                    q   q  ... q 
                                         
                            
      Ps | I    1s  1 2s  2  Ns   N
                      p1s   p 2s   p Ns 
                   q   q  ... q 
                       
                  s  1s   2 s 
                                                                                 Normalizing constant
                                       Ns                                          for individual risks




                                  i=r(V i /i)c
                                                                         Holding value/Risk
      “market” risk
                                                                         - measure relative risk of individuals
      ~sum of prices/winning payoff



                                       Experimental Economics Program
Public Information Experiments


    •   5 Experiments
    •   Various Information Structures
        –   All subject received 2 private draws & 2 public draws
        –   All subject received 3 private draws & 1 public draws
        –   All subject received 3 private draws & half of the subjects
            receive 1 public draws
        –   All subject received 3 private draws & 1 public draws. 2
            groups of independent public information.
    •   9 to 11 participants in each experiments


                               Experimental Economics Program
              Correcting for Public Information

                  Kullback-Leibler = 0.291

1.2
          Omnicient
 1        sim aggr
          IAM
0.8
          IAM (true public)
0.6

0.4

0.2

 0
      A   B       C       D   E   F     G    H     I   J



              Public Info Experiment 3, Period 9
                    11 Players Aggregated
KL Measures for Public Info Experiments



                                                                              Nonlinear
                                                                                            Public       Perfect
            Private    Public                    Market             Best     Aggregation     Info       Public Info
     Expt    Info       Info      No Info       Prediction         Player     Function     Correction   Correction

      1     2 draws    2 draws     1.332           0.847            0.932      2.095         0.825         0.279
             for all    for all   (0.595)         (0.312)          (0.566)    (1.196)       (0.549)       (0.254)

      2     2 draws    2 draws     1.420           0.979            0.919      2.911         0.798         0.258
             for all    for all   (0.424)         (0.573)          (0.481)    (2.776)       (0.532)       (0.212)

      3     3 draws    1 draws     1.668           1.349            1.033      2.531         0.718         0.366
             for all    for all   (0.554)         (0.348)          (0.612)    (1.920)       (0.817)       (0.455)

      4     3 draws    1 draws     1.596           0.851            1.072      0.951         0.798         0.704
             for all   for half   (0.603)         (0.324)          (0.604)    (1.049)       (0.580)       (0.691)
                        Two
                       groups
            3 draws      of        1.528           0.798            1.174      0.886         1.015         0.472
      5      for all   public     (0.600)         (0.451)          (0.652)    (0.763)       (0.751)       (0.397)
                        info



                                            Experimental Economics Program
Summary



    •   IAM with public info correction did better than best
        person.
    •   IAM with public info correction did better than markets in
        4 out of 5 cases.
    •   IAM corrected with true public info did significant better
        than all other methods.




                            Experimental Economics Program
                                      $0
                                            -$




                                                            0%
                                                                 5%
                                                                      10%
                                                                            15%
                                                                                  20%
                                                                                        25%
                                                                                              30%
                                                                                                                   35%
                                              10
                                 $1             0   9.
                                   00                 1m
                                      9.
                                        1
                                            -$
                                              10
                                 $1             24
                                   02             .7
                                      4.            m
                                        7
                                            -$
                                              10
                                 $1             38
                                                  .7
                                   03               m
                                      8.
                                        7
                                            -$
                                              10
                                 $1             51
                                   05             .8
                                      1.            m
                                        8
                                            -$
                                              10
                                 $1             64
                                   06             .3
                                      4.            m
                                        3
                                            -$
                                              10
                                 $1             71
                                                  .6
                                   07
                                      1.            m
                                        6
                                                                                                      $1053m




                                            -$
                                              10
                                                                                                    Actual Value




                                 $1             78
                                   07             .8
                                      8.            m
                                        8
                                            -$
                                              10
                                 $1             86
                                                  .0
                                   08               m
                                      6.
                                        0
                                            -$
                                              10
                                 $1             93
                                   09             .2
                                      3.            m
                                        2
                                                                                                    Official




                                            -$
                                              11
                                                                                                    Projection




                                 $1             00
                                                  .4
                                   10               m
                                      0.
                                        4
                                            -$




Experimental Economics Program
                                              11
                                 $1             07
                                   10             .7
                                      7.            m
                                        7
                                            -$
                                              11
                                 $1             20
                                   12             .2
                                      0.            m
                                        2
                                            -$
                                              11
                                 $1             33
                                                  .3
                                   13               m
                                                                                                                         Implied Probabilities of Revenue Bins, September 2003




                                      3.
                                        3
                                            -$
                                              11
                                 $1             47
                                   14             .3
                                      7.            m
                                        3
                                            -$
                                              11
                                                62
                                 $1               .9
                                   16               m
                                      2.
                                        9m
                                              or
                                                 ab
                                                   ov
                                                        e
                                               Implied Probabilities of Operating Profit Bins, September 2003


70%

                                                                                    Official                                              Actual Value
                                                                                    Projection                                              $113m
60%




50%




40%




30%




20%




10%




0%
       $0 -    $37.1 - $46.1m -   $54.4 -   $62.0 -   $69.3 -   $73.6 -   $77.8 -   $82.0 -      $86.2 -   $90.4 - $94.7 - $102.0 - $109.6 - $117.9 - $126.9m
      $37.1m   $46.1m $54.4m      $62.0m    $69.3m    $73.6m    $77.8m    $82.0m    $86.2m       $90.4m    $94.7m $102.0m $109.6m $117.9m $126.9m or above




                                                                   Experimental Economics Program
Supplementary




     Experimental Economics Program
Previous Research

 •   Academic Studies
     –   Information Aggregation in Markets
         • Plott, Sunder, Camerer, Forsythe, Lundholm, Weber,…

     –   Pari-mutuel Betting Markets
         • Plott, Wit & Yang

 •   Real World Applications
     –   Iowa Electronic Markets
     –   Hollywood Stock Exchange
     –   HP Information Markets
     –   Newsfuture
     –   Tradesport.com
     –   …

                                   Experimental Economics Program
                    Risk Attitudes




1.000
0.900
0.800
0.700
0.600                                                              Risk Loving
0.500                                                              Risk Neutral
0.400                                                              Risk Averse
0.300
0.200
0.100
0.000
        A   B   C     D     E        F        G        H   I   J




                          Experimental Economics Program
                 Dealing with Risks Attitudes:
                   Two-Stage Mechanism




       Event 1
                                                  Stage 1: Information Market
       Event 2

       Event 3
                                            Call Market to Solicit Risk Attitudes
Time




       Event 4

       Event 5
                                    Stage 2: Probability Reporting & Aggregation
       Event 6


       Event 7
                                     Individual Report of Probability Distribution
                                            Nonlinear Aggregated Function
       Event 8




                             Experimental Economics Program
Probability Reporting



                                                             Outcome

                 A                  B                          C

                 Reports of Probability Distribution


                 p1                  p2                        p3


                      Pays C1+C2*Log(p3)


                            Experimental Economics Program
Second Stage: Aggregation Function




      Bayes Law with Behavioral Correction

                                                1        2               N
                  Ps | I  
                               p p ...p         1s        2s               Ns

                               p1s p2s ...p
                                 1  2

                                          s
                                                                             N
                                                                             Ns

                                                                                         Normalizing constant
                                                                                          for individual risks




                                     i=r(V i / i)c
                                                                              Holding value/Risk
          “market” risk
                                                                              - measure relative risk of individuals
          ~sum of prices/winning payoff



                                          Experimental Economics Program
Private Information Experiments


    •   5 Experiments

    •   Various Information Conditions

        –   All subject received 3 draws

        –   Half received 5 draws, half received 1 draw

        –   Half received 3 draws, half received random number of draws

    •   8 to 13 participants in each experiments


                               Experimental Economics Program
Next Step


•   Field Test (Fine and Huberman) …




                           Experimental Economics Program

								
To top