Docstoc

2002 Stock Market Predictions

Document Sample
2002 Stock Market Predictions Powered By Docstoc
					 INTELLIGENT MINING FOR TIME SERIES PREDICTIONS
     (AND ITS APPLICATIONS IN STOCK MARKET
                  PREDICTIONS)

                                                 Benjamin W. Wah
                            Department of Electrical and Computer Engineering
                                 and the Coordinated Science Laboratory
                                University of Illinois at Urbana-Champaign
                                       Urbana, Illinois 61801, USA
                                        http://manip.crhc.uiuc.edu

                                                 December 21, 2002




Intelligent Mining for Time Series Predictions                                  Outline


                                                     Outline

    • Market-trend prediction problem
         – Time series predictions
         – Metrics
    • Signal processing of time series
         – Lags in predictable low-frequency components
    • Data mining techniques
         – Intelligent mining and major design issues
         – Prediction agents
    • Constrained optimizations using neural networks
         – Lagrange multipliers for discrete constrained optimization
    • Some sample results
Benjamin W. Wah                                                                      1
Intelligent Mining for Time Series Predictions                                                         Introductions


                                                 Time Series Predictions

    • Prediction of future values based on a sequence of past (and hopefully correlated)

        values

         – Stock market predictions
                                                                                     DJ stock
                                                               52
         – Product failures                                    50            Raw data
                                                               48
                                                               46
         – Occurrence of sunspots


                                                       Price
                                                               44
                                                               42
         – Census data classification                           40
                                                               38
         – Earthquake predictions                              36
                                                               34
                                                                 500   550     600     650 700   750   800
         – plus many others
                                                                                       Day

Benjamin W. Wah                                                                                                   2




Intelligent Mining for Time Series Predictions                                                         Introductions


                                                 Time Series Predictions

    • Prediction of future values based on a sequence of past (and hopefully correlated)

        values

         – Stock market predictions
                                                               52                DJ stock
         – Product failures                                    50         Raw data
                                                               48   8-tap sym filter
                                                               46
         – Occurrence of sunspots
                                                       Price




                                                               44
                                                               42
         – Census data classification                           40
                                                               38
         – Earthquake predictions                              36
                                                               34
                                                                 500 550 600 650 700             750   800
         – plus many others
                                                                                   Day

Benjamin W. Wah                                                                                                   2
Intelligent Mining for Time Series Predictions                                                   Introductions


                                                 Time Series Predictions

    • Prediction of future values based on a sequence of past (and hopefully correlated)

        values

         – Stock market predictions
                                                                                DJ stock
                                                               52
         – Product failures                                    50        Raw data
                                                               48 20-tap sym filter
                                                               46
         – Occurrence of sunspots


                                                       Price
                                                               44
                                                               42
         – Census data classification                           40
                                                               38
         – Earthquake predictions                              36
                                                               34
                                                                 500 550 600 650 700       750   800
         – plus many others
                                                                                  Day

Benjamin W. Wah                                                                                             2




Intelligent Mining for Time Series Predictions                                                   Introductions


                                                 Time Series Predictions

    • Prediction of future values based on a sequence of past (and hopefully correlated)

        values

         – Stock market predictions
                                                               52               DJ stock
         – Product failures                                    50        Raw data
                                                               48 50-tap sym filter
                                                               46
         – Occurrence of sunspots
                                                       Price




                                                               44
                                                               42
         – Census data classification                           40
                                                               38
         – Earthquake predictions                              36
                                                               34
                                                                 500 550 600 650 700       750   800
         – plus many others
                                                                                  Day

Benjamin W. Wah                                                                                             2
Intelligent Mining for Time Series Predictions                                                Introductions


                                                          Metrics

                   a) Sum of squared errors between original and predicted curves


                 PRICE                       Actual
                                             Predicted




                                                                                     DAY
                                       Today




Benjamin W. Wah                                                                                          3




Intelligent Mining for Time Series Predictions                                                Introductions


                                                      Metrics (cont’d)

                             b) Hit rate: fraction of consistent trend predictions
                                (Hit: consistency defined by relative trends)


                                            Hit
         PRICE
                                                                                     Actual
                                                                                     Predicted

                                                                                     Hit
                                                                                     Miss
                               Lag Period

                                      Today T1             T2            DAY




Benjamin W. Wah                                                                                          4
Intelligent Mining for Time Series Predictions                                                Introductions


                                                 Metrics (cont’d)

                             b) Hit rate: fraction of consistent trend predictions
                                (Hit: consistency defined by relative trends)



         PRICE                                        Miss
                                                                                     Actual
                                                                                     Predicted

                                                                                     Hit
                                                                                     Miss
                               Lag Period

                                      Today T1        T2                DAY




Benjamin W. Wah                                                                                          5




Intelligent Mining for Time Series Predictions                                                     Outline


                                                     Outline

    • Market trend prediction problem
         – Time series predictions
         – Metrics
    • Signal processing of time series
         – Lags in predictable low-frequency components
    • Data mining techniques
         – Intelligent mining and major design issues
         – Prediction agents
    • Constrained optimizations using neural networks
         – Lagrange multipliers for discrete constrained optimization
    • Some sample results
Benjamin W. Wah                                                                                          6
Intelligent Mining for Time Series Predictions                                                                                         Signal Processing of Time Series


                     FFT Transformations of 1024 Daily Closing Prices
                               6                                                        6
                              10                                                       10
                                                                  IBM                                                      SUN
                               4                                                        4
                              10                                                       10


                               2                                                        2
                              10                                                       10


                               0                                                        0
                              10                                                       10


                               −2                                                       −2
                              10                                                       10
                                       0    1                2           3                   0            1           2            3
                                   10      10            10             10                  10       10              10           10


                                   4                                                    4
                              10                                                       10
                                                                 MSFT                                                      INTL
                                   3
                              10
                                                                                        2
                                                                                       10
                                   2
                              10
                                                                                        0
                                                                                       10
                                   1
                              10


                                   0                                                    −2
                              10                                                       10
                                  0         1                2           3                   0            1           2            3
                                10         10            10             10                  10       10              10           10

                                                                                                                                                 1
            Random walks, stock-price movements, exchange rates follow the                                                                       f    line



Benjamin W. Wah                                                                                                                                                      7




Intelligent Mining for Time Series Predictions                                                                                         Signal Processing of Time Series


                        Dow Jones Theory for Stock Price Movements

       • Detect

           – Primary trends: changes that                                          Relative Energies          S 2 (f )
                                                                                                                          of Lowest 29 f
                                                                                                              S 2 (0)
                                                                              0
                                                                             10

               are larger than 20%, typically                                                                                                  IBM
                                                                                                                                               SUN
                                                                                                                                               MSFT
                                                                              −1
                                                                             10                                                                INTL
                                                                                                                                               DJ

               lasting more than a year                                       −2
                                                                             10



                                                1        2
           – Secondary trends:                  3
                                                    to   3
                                                                              −3
                                                                             10



                                                                              −4
                                                                             10

               relative change over primary                                   −5
                                                                             10



               trends, typically lasting a few                                −6
                                                                             10




               months                                                         −7
                                                                             10
                                                                                   0             5   10         15          20            25          30




       • Ignore minor trends


Benjamin W. Wah                                                                                                                                                      8
Intelligent Mining for Time Series Predictions                                                                                                                                    Signal Processing of Time Series


                                                               Filtering of Time Series

    • Random noise in time series is not predictable [Zheng99,Hellstrom97]
         – Decompose signals into additive short-term noise and long-term trends, since
           most energy is in low frequencies

                                                                   DJ stock
                                                  52
                                                  50        Raw data
                                                  48 50-tap sym filter
                                                  46
                                    Price



                                                  44
                                                  42
                                                  40
                                                  38
                                                  36
                                                  34
                                                    500 550 600 650 700                                                                            750                      800
                                                                     Day
Benjamin W. Wah                                                                                                                                                                                                 9




Intelligent Mining for Time Series Predictions                                                                                                                                    Signal Processing of Time Series


                                          Illustration of Filtering Process

    • Symmetric FIR filter: g(l) = g(−l)
                                                                                               R(t+L-1)                   R(t+L-2)                 R(t-L)
                                                         R(t+L)                        q −1                q −1                            q −1

                                                                               g(-L)               g(-L+1)                     g(-L+2)                     g(L)

                                                                                                   +                       +                         +                     S(t)


    • Low-pass and high-pass data
         – Prediction need to overcome lag period (10 days here)
                                                                                                                                                                          Autocorrelations
                                                                                                                 54                                             1
                                                                                                                 52
                                                                                                                                                              0.9
                                                                                                                 50
                                                                                                                 48
                                                                                                                                                              0.8
                                                                                                                 46
                                                                                                                 44                                           0.7


                                                                                                   low           42
                                                                                                                 40                                           0.6

                                                                                                   pass          38
                                                                                                                 36
                                                                                                                                                              0.5

                                                                                                                 34                                           0.4
                   55                                    1.2                                                                                                        0         5    10   15   20
                                                                                                                 32
                                                                                                                      0        50    100     150     200
                                                          1
                   50


                                                                                                                           Lag by 10 days                               lag period
                   45
                                                         0.8


                                                         0.6
                                                                                                                                                     Today
                   40
                                                                                                         high    1                                             0.3


                   35
                                                         0.4


                                                         0.2
                                                                                                         pass   0.8                                            0.2

                                                                                                                0.6                                            0.1
                   30
                                                          0
                                                                                                                0.4
                        0      50   100     150    200                                                                                                            0
                                                               0   0.5   1   1.5   2     2.5   3   3.5
                                                                                                                0.2

                            IBM stock prices             Frequency responses of                             −0.2
                                                                                                                 0
                                                                                                                                                              −0.1

                                                                                                                                                              −0.2

                                                         a 20−tap FIR filter bank                           −0.4                                              −0.3

                                                                                                            −0.6                                              −0.4
                                                                                                                                                                      0       5    10   15   20
                                                                                                            −0.8
                                                                                                                      0        50    100    150      200

                                                                                                                                                                          lag period


Benjamin W. Wah                                                                                                                                                                                                10
Intelligent Mining for Time Series Predictions                                         Signal Processing of Time Series


                                         Lags due to Low-Pass Filtering

    • Filtering uses future data to generate low-pass data that lags behind original data
         – High frequency data: random noise and not predictable

                                          A lag of 25 days for a 50-tap filters
                                                          DJ stock
                                         52
                                         50        Raw data
                                         48 50-tap sym filter
                                         46
                                 Price


                                         44
                                         42
                                         40
                                         38
                                         36
                                         34
                                           500 550 600 650 700          750      800
                                                            Day

Benjamin W. Wah                                                                                                     11




Intelligent Mining for Time Series Predictions                                         Signal Processing of Time Series


                                                    Filter Banks

    • Multi-band filter banks
         – Equal width for each band and maximally decimated
    • Multi-resolution wavelet transforms
         – Exponentially larger passbands from low to high frequencies
         – Perfect reconstruction at time t is only related to information at time t of
             different scales, with no error propagation
         – Shift variant: Decomposed outputs depend on the origin for decimations
                 ´
    • Redundant (A Trous) wavelet transforms
         – Similar to multi-resolution wavelet transforms, except on different constraint on
             wavelet function and no decimation (more storage requirement)
         – Shift invariant: statistical estimators are not sensitive to the choice of origin
Benjamin W. Wah                                                                                                     12
Intelligent Mining for Time Series Predictions                                                  Signal Processing of Time Series


                                      ´
                           Redundant (A. Trous) Wavelet Transforms

  Algorithm

                                            set c0(t) = x(t);
                                            set M = total number of channels;
                                            select low-pass filter h(·);
                                            for j ← 1 to M do
                                                cj (t) = h(l)cj−1(t − 2j−1l);
                                                        l
                                              wj (t) = cj−1(t) − cj (t);
                                            end for

  Properties
    • No decimation: redundant transform
                                                                                  M
    • Reconstruction of x(t) with no lag: c0(t) = cM (t) +                             wj (t)
                                                                                 j=1


Benjamin W. Wah                                                                                                              13




Intelligent Mining for Time Series Predictions                                                  Signal Processing of Time Series


                           Redundant WT Using Symmetric LP Filters
  Example: B(2) = {h(−1) = 0.25, h(0) = 0.5, h(1) = 0.25}

                                           c0    ...........
                                           c1     .........                        M =3
                                           c2       .....
                                           c3
                                                                       2 units
                                    w1
                                                                     6 units
                                    w2
                                                                 14 units
                                    w3
                                                                 14 units
                                    c3                             Lag         prediction

                                                   Lags in all bands



Benjamin W. Wah                                                                                                              14
Intelligent Mining for Time Series Predictions                                                            Signal Processing of Time Series


          Examples of Frequency Response Using Symmetric LP Filters

          1                                                                  1
                                                 w1                                                                 w1
                                                 w2                                                                 w2
        0.8                                      w3                         0.8                                     w3
                                                 w4                                                                 w4
                                                 c4                                                                 c4
        0.6                                                                 0.6


        0.4                                                                 0.4


        0.2                                                                 0.2


          0                                                                  0
              0    0.5     1     1.5      2      2.5   3   3.5                    0   0.5    1      1.5         2   2.5     3      3.5


                                                                                  B3 = {h(−2) = h(2) = 0.0625,
                  B2 = {h(−1) = h(1) = 0.25,
                                                                                  h(−1) = h(1) = 0.25,
                  h(0) = 0.5}
                                                                                  h(0) = 0.375}


Benjamin W. Wah                                                                                                                          15




Intelligent Mining for Time Series Predictions                                                            Signal Processing of Time Series


     Example of Applying WT with Symmetric B3 to IBM Stock Trace
         36                                                       1.2
                                                                    1
         34                         x (407 points)                0.8                             w1 (lag 2)
         32                                                       0.6
         30                                                       0.4
                                                                  0.2
         28                                                         0
         26                                                      -0.2
         24                                                      -0.4
                                                                 -0.6
         22                                                      -0.8
         20                                                        -1
              0   50   100 150 200 250 300 350 400 450                  0   50     100 150 200 250 300 350 400 450

        0.6                                                       0.6
        0.4                                                       0.4
        0.2                                                       0.2
          0                                                         0
       -0.2                                                      -0.2
       -0.4                                                      -0.4
                                        w2 (lag 6)                                               w3 (lag 14)
       -0.6                                                      -0.6
       -0.8                                                      -0.8
              0   50   100 150 200 250 300 350 400 450                  0   50     100 150 200 250 300 350 400 450

        0.8                                                       34
        0.6                            w4 (lag 30)                32
        0.4                                                       30                              c4 (lag 30)
        0.2                                                       28
          0                                                       26
       -0.2                                                       24
       -0.4                                                       22
       -0.6                                                       20
              0   50   100 150 200 250 300 350 400 450                  0   50    100 150 200 250 300 350 400 450


Benjamin W. Wah                                                                                                                          16
Intelligent Mining for Time Series Predictions                                                                   Signal Processing of Time Series


                                Corresponding Auto-correlation Functions
          1                                                                1
       0.95                                          x                   0.8
                                                                                                                  w1
                                                                         0.6
        0.9                                                              0.4
       0.85                                                              0.2
                                                                           0
        0.8                                                             -0.2
       0.75                                                             -0.4
              0   5   10   15   20   25    30   35       40   45   50          0   5   10   15    20   25   30   35        40   45   50

          1                                                                1
        0.8                                                              0.8
                                                 w2                                                               w3
        0.6                                                              0.6
        0.4                                                              0.4
                                                                         0.2
        0.2                                                                0
          0                                                             -0.2
       -0.2                                                             -0.4
       -0.4                                                             -0.6
              0   5   10   15   20   25    30   35       40   45   50          0   5   10   15    20   25   30   35        40   45   50

          1                                                               1
        0.8                                      w4
        0.6                                                             0.95                                          c4
        0.4                                                              0.9
        0.2
          0                                                             0.85
       -0.2                                                              0.8
       -0.4
       -0.6                                                             0.75
              0   5   10   15   20   25    30   35       40   45   50          0   5   10   15    20   25   30   35        40   45   50


Benjamin W. Wah                                                                                                                               17




Intelligent Mining for Time Series Predictions                                                                   Signal Processing of Time Series


                                     Relationship Between Lags and ACF


  High-frequency components have lags longer than sequence of correlated points



                                                 WT with Symmetric LP                            Filters
                                          Signal Days with ACF > 0.5                              Lag
                                                   IBM       MSFT
                                           W1        0         0                                  2
                                           W2        1         1                                  6
                                           W3        2         2                                  14
                                           W4        6         5                                  30
                                            C4     50+        50+                                 30




Benjamin W. Wah                                                                                                                               18
Intelligent Mining for Time Series Predictions                                         Signal Processing of Time Series


                          Redundant WT Using Asymmetric LP Filters

                        Example: B(2) = {h(0) = 0.25, h(1) = 0.5, h(2) = 0.25}



                                           c0    ...........
                                           c1      .........
                                           c2          .....
                                           c3
                                  w1
                                  w2
                                  w3
                                   c3                                     prediction




Benjamin W. Wah                                                                                                     19




Intelligent Mining for Time Series Predictions                                         Signal Processing of Time Series


                                                  Comments

    • cs(t) obtained using symmetric LP filters is a shifted version of ca(t) obtained by
       j                                                                j
      corresponding asymmetric LP filters
      For example:
                                      ca(t) = cs (t − 1)
                                       1        1
                                                     ca(t) = cs (t − 3)
                                                      2       2
                                                            ···
       s                    a
    • wj (t) is related to wj (t)
      For example:
                          a
                         w2 (t) =       ca(t) − ca(t)
                                         1        2
                                =       cs (t − 1) − cs (t − 3)
                                         1            2
                                =       (cs (t − 1) − cs (t − 3)) + (cs (t − 3) − cs (t − 3))
                                           1            1             1            2
                                =                                     s
                                        (cs (t − 1) − cs (t − 3)) + w2(t − 3)
                                           1            1

    • Equivalent shifts of corresponding bands filtered by symmetric filters
      =⇒ using these functions alone leads to similar (but smaller) lags

Benjamin W. Wah                                                                                                     20
Intelligent Mining for Time Series Predictions                                                                        Signal Processing of Time Series


    Example of Applying WT with Asymmetric B3 to IBM Stock Trace
         36                                                                1.5
                                                                             1
         34                           x (407 points)                       0.5
         32                                                                  0
         30                                                               -0.5
         28                                                                 -1
         26                                                               -1.5                                w1 (lag 0)
         24                                                                 -2
         22                                                               -2.5
         20                                                                 -3
              0   50   100 150 200 250 300 350 400 450                           0   50   100 150 200 250 300 350 400 450

          1                                                                1.5
        0.5                                                                  1
          0                                                                0.5
                                                                             0
       -0.5                                                               -0.5
         -1                                                                 -1
       -1.5                                                               -1.5
         -2                                 w2 (lag 2)                      -2                                w3 (lag 6)
                                                                          -2.5
       -2.5                                                                 -3
         -3                                                               -3.5
              0   50   100 150 200 250 300 350 400 450                           0   50   100 150 200 250 300 350 400 450

        1.5                                                                34
          1
        0.5                                                                32
          0                                                                30                                 c4 (lag 30)
       -0.5
         -1                                                                28
       -1.5                                                                26
         -2
       -2.5                                w4 (lag 14)                     24
         -3                                                                22
       -3.5
         -4                                                                20
              0   50   100 150 200 250 300 350 400 450                           0   50   100 150 200 250 300 350 400 450


Benjamin W. Wah                                                                                                                                    21




Intelligent Mining for Time Series Predictions                                                                        Signal Processing of Time Series


                                 Corresponding Auto-correlation Functions
          1                                                                  1
       0.95                                            x                   0.8
                                                                                                                      w1
                                                                           0.6
        0.9
                                                                           0.4
       0.85
                                                                           0.2
        0.8                                                                  0
       0.75                                                               -0.2
              0   5    10   15   20   25     30   35       40   45   50          0   5    10   15   20   25    30   35        40   45   50

          1                                                                  1
        0.8                                                                0.8
                                                    w2                                                                w3
        0.6                                                                0.6
        0.4                                                                0.4
        0.2                                                                0.2
          0                                                                  0
       -0.2                                                               -0.2
              0   5    10   15   20   25     30   35       40   45   50          0   5    10   15   20   25    30   35        40   45   50

          1                                                                 1
        0.9
        0.8                                                               0.95                                           c4
        0.7                                         w4
        0.6                                                                0.9
        0.5
        0.4                                                               0.85
        0.3
        0.2                                                                0.8
        0.1
          0                                                               0.75
              0   5    10   15   20   25     30   35       40   45   50          0   5    10   15   20   25    30   35        40   45   50


Benjamin W. Wah                                                                                                                                    22
Intelligent Mining for Time Series Predictions                                       Signal Processing of Time Series


                                                 Key Observations

    • High-frequency components have lags longer than sequence of correlated points

                       WT with Symmetric LP             Filters WT with Asymmetric LP             Filters
                Signal Days with ACF > 0.5               Lag Days with ACF > 0.5                  Lag
                         IBM       MSFT                           IBM       MSFT
                 W1        0         0                     2        1          1                     0
                 W2        1         1                     6        3          2                     2
                 W3        2         2                    14        7          6                     6
                 W4        6         5                    30       14         15                    14
                  C4     50+        50+                   30      50+        50+                    30

    • Additional information within lag, such as price of fluctuations and volume of
      transactions, may be used to augment learning and prediction mechanisms
                                                               S(t)−S(t−5)
    • Transformed objective (e.g. return function                 S(t−5)
                                                                           )   and better filters may help
        improve (short-term or long-term) ACF with respect to lag
Benjamin W. Wah                                                                                                   23




Intelligent Mining for Time Series Predictions                                                               Outline


                                                     Outline

    • Market trend prediction problem
         – Time series predictions
         – Metrics
    • Signal processing of time series
         – Lags in predictable low-frequency components
    • Data mining techniques
         – Intelligent mining and major design issues
         – Prediction agents
    • Constrained optimizations using neural networks
         – Lagrange multipliers for discrete constrained optimization
    • Some sample results
Benjamin W. Wah                                                                                                   24
Intelligent Mining for Time Series Predictions                                                                                    Solution Approaches


                  Existing Models for Nonlinear Time Series
                                            Time Series Models

                   Linear Models                                         Nonlinear Models
                                                                                         General
                                                  Pre−defined                          nonlinearity
        ARMA             state−space              nonlinearity                       (Machine learning)
       & variants          models
        [Box 97]          [Aoki 87]    bilinear AR             TAR
                                        [Granger 78]         [Tong 90]
                                                                      Statistic
                                                                                     Unsupervised Supervised
                                                  time−varying
                                                 parameter models          Reinforcement
                                                                                                          Decision
                                                   [Nicholls 85]                                        tree learning
                                                                         kNN          clustering         [Quinlan 86]
                                                                      [Duda 73]              [Jain 99]
                                                                                                               neural
                                                                              Q−learning                      networks
                                                                              [Watkins 89]                   [Haykin 99]


    • Issues in existing nonlinear supervised learning techniques
         – Single nonlinear objective on training set
         – Cannot enforce individual pattern behavior
    • Constraint on individual pattern behavior is desirable
Benjamin W. Wah                                                                                                                                   25




Intelligent Mining for Time Series Predictions                                                                                    Solution Approaches


                  Ideal Model of Intelligent Mining for Trend Prediction



                              Web

                                                    qualitative/
                   Information/
                                                    quantitative
                  Search Agents                     information
                             Search/assimilation rules
                                                  Feedback                                                                Predicted
                                                                                                                          Trends
                  Learner/Miner                                                   Predictor
                                                   Prediction rules
                               User−defined                                       Prediction
                               Criteria/Profiles                                   Agents




Benjamin W. Wah                                                                                                                                   26


                                                                                                      
                                                                                                      
                                                                                                      
                                                                                                         ¡
                                                                                                         ¡
                                                                                                         ¡
                                                                                                         ¡
                                                                                                          
                                                                                                          
                                                                                                          
                                                                                                              ¡
                                                                                                              ¡
                                                                                                               
                                                                                                               
                                                                                                               
                                                                                                               
                                                                                                              ¡
                                                                                                              ¡   ¡
                                                                                                                  ¡
                                                                                                                  ¡
                                                                                                                  ¡
                                                                                                                   
                                                                                                                   
                                                                                                                   
                                                                                                                      ¡
                                                                                                                      ¡
                                                                                                                       
                                                                                                                       
                                                                                                                       
                                                                                                                       
                                                                                                                      ¡
                                                                                                                      ¡   ¡
                                                                                                                          ¡
                                                                                                                          ¡
                                                                                                                          ¡
                                                                                                                           
                                                                                                                           
                                                                                                                           
                                                                                                                              ¡
                                                                                                                              ¡
                                                                                                                               
                                                                                                                               
                                                                                                                               
                                                                                                                               
                                                                                                                              ¡
                                                                                                                              ¡
Intelligent Mining for Time Series Predictions                                 Solution Approaches


                                                 Major Design Issues

    • Information/search agents to get information
         – Use of wrong, too many, or too little search criteria
              ∗ Possibly inconsistent information from many sources
         – Semantic analysis of (meta-) information
         – Assimilation of information into inputs to predictor agents
    • Learner/miner to modify information selection criteria
         – Apportioning of biases to feedback
         – Developing rules for Search Agents to collect information
         – Developing rules for Information Agents to assimilate information
    • Predictor agents to predict trends
         – Incorporation of qualitative information
         – Multi-objective optimization not in closed form

Benjamin W. Wah                                                                                27




Intelligent Mining for Time Series Predictions                                 Solution Approaches


                             Prediction Agents: Numerical Approaches

  Memory-based approaches: data mining
    • Using historical information to build a model of time-series behavior in order to
      predict future behavior
    • KNN (K-Nearest Neighbor) classification techniques to locate points in
      multi-dimensional space
         – Too much noise in matching original time series
         – Difficulty in overcoming lags in low-frequency data
  Computation-based approaches: neural networks and time-series analysis
    • Formulation: Incorporation of quantitative and qualitative information
    • Training algorithm
         – Window size, sampling lags, network topology, training parameters, training set,
           etc.
Benjamin W. Wah                                                                                28
Intelligent Mining for Time Series Predictions                                                       Outline


                                                 Outline

    • Market trend prediction problem
         – Time series predictions
         – Metrics
    • Signal processing of time series
         – Lags in predictable low-frequency components
    • Data mining techniques
         – Intelligent mining and major design issues
         – Prediction agents
    • Constrained optimizations using neural networks
         – Lagrange multipliers for discrete constrained optimization
    • Some sample results
Benjamin W. Wah                                                                                          29




Intelligent Mining for Time Series Predictions               Constrained Optimizations Using Neural Networks


                   ANN Models for Time Series Predictions

    • Existing architectures

         – Recurrent neural networks (RNN)

         – Memory-based neural networks (TDNN and FIR-NN)

         – Dynamic recurrent neural networks (DRNN): FIR + feedback without delay

         – No consensus on which architecture is better [Horne][Hallas]

         – Training algorithm is more important than architecture [Koskela]

    • Proposed architecture: Recurrent FIR neural network (RFIR)

         – RFIR: FIR + recurrent feedback with time delay

Benjamin W. Wah                                                                                          30
Intelligent Mining for Time Series Predictions                                      Constrained Optimizations Using Neural Networks


                  (A) Proposed Recurrent FIR Architecture


                                                 q −1

                                                 q −1
                                                                          q −1 : Unit delay

                                                             q −1



                  recurrent node                                           y(t)

                          x(t)
                  regular node
                        x(t + 1)                          FIR filter         w(0) w(1)        ...   w(T )

                       bias node


                  Unit delay ⇒ easier to derive gradients as compared with DRNN



Benjamin W. Wah                                                                                                                 31




Intelligent Mining for Time Series Predictions                                      Constrained Optimizations Using Neural Networks


                                            Performance Metrics

    • Normalized mean square error (nMSE):

                                                                t1
                                                           1
                                                        ε= 2           (o(t) − d(t))2,
                                                          σN   t=t0


         – σ 2 is the variance of the true time series in [t0, t1]

         – o(t) is actual output at t; d(t) is desired output

         – N is number of patterns in the measurement

    • Open-loop single-step measurement: external input is true observed data

    • Close-loop iterative measurement: external input is predicted output


Benjamin W. Wah                                                                                                                 32
Intelligent Mining for Time Series Predictions                               Constrained Optimizations Using Neural Networks


                            Traditional Formulations for ANN Training

    • Unconstrained formulation
                                                                n
                                                            1
                                                 min E(w) =           (ot(w) − dt)2
                                                  w         n   t=1


    • Training algorithms

         – BP/BP variants and gradient-based methods
         – Genetic algorithms
         – Simulated annealing

    • Issues

         – No guidance when search reaches a non-zero local minimum of E(w)
         – Nonuniform errors across patterns – not good for training

Benjamin W. Wah                                                                                                          33




Intelligent Mining for Time Series Predictions                               Constrained Optimizations Using Neural Networks


                     (B) Proposed Constrained Formulations

    • Each learning pattern is treated as an additional constraint:

                                                   ht(w) = (ot(w) − dt)2 ≤ τ,

         – τ decreases towards 0 as looser constraints are satisfied

         – Non-zero constraints provide guidance when search reaches a sub-optimum of

             the objective function

    • New constraints added

         – Make the problem more difficult to solve

         – Do not lead to over-training of the neural network

Benjamin W. Wah                                                                                                          34
Intelligent Mining for Time Series Predictions                              Constrained Optimizations Using Neural Networks


                                          Traditional Cross-Validation

    • Divide historical data into two disjoint sets
         – Training set
         – Cross-validation set

                                           Training       Validation        Post−Learning
                                                                               Testing


    • Issues
         – Hard to choose appropriate validation set: how long?
         – Data used for cross-validation cannot be used for training
         – Only one validation set is used at any time: not good when time series is
           multi-stationary
         – Single-objective optimization minimizes errors in validation set: what about
           errors in learning?

Benjamin W. Wah                                                                                                         35




Intelligent Mining for Time Series Predictions                              Constrained Optimizations Using Neural Networks


                               (C) Proposed Cross-Validation Method


    • Multiple validation sets within training set

                                                                                     Post−Learning
                                                 Training Set                           Test Set
                                                 V1                    V3
                                                            V2


    • Iterative and single-step validation errors are added as new constraints

         – Training patterns are fully used

         – Multiple regimes in a multi-stationary time-series are covered

         – Flexibility in choosing validation sets

Benjamin W. Wah                                                                                                         36
Intelligent Mining for Time Series Predictions                                                                                                        Constrained Optimizations Using Neural Networks


                                    (D) Penalties on Incorrect Trend Predictions


             PRICE                                                Actual                                                     Consistent predictions of trend
                                                                  Predicted                                                  Inconsistent predictions




                                                         Lag Period
                                                                                                                                                                              DAY
                                                                      Today


                   Patterns with inconsistent trend predictions are further penalized


Benjamin W. Wah                                                                                                                                                                                   37




Intelligent Mining for Time Series Predictions                                                                                                        Constrained Optimizations Using Neural Networks


                           (E) Predictions of Low-Pass Data in Lag Period



                                                                               data needed to generate low−pass data at point b
                               data needed to generate low−pass data at point a
              53
              52
              51
              50                                                                                                                              b
              49
                                                                                   a
              48
                                                                                                                                                          raw data
              47
              46                                                                                                                                          true low−pass data
              45
              44
                   0                             5                             10                           15                           20                      25               30
                                                                  Low pass ends                                                  Today


Benjamin W. Wah                                                                                                                                                                                   38


                        
                        
                           ¡
                           ¡
                           ¡
                            
                            
                                ¡
                                 
                                 
                                 
                                ¡
                                ¡    ¡
                                     ¡
                                     ¡
                                      
                                      
                                         ¡
                                         ¡
                                          
                                          
                                          
                                         ¡   ¡
                                             ¡
                                              
                                              
                                              
                                             ¡       ¡
                                                     ¡
                                                      
                                                      
                                                      
                                                     ¡    ¡
                                                          ¡
                                                           
                                                           
                                                           
                                                          ¡   ¡
                                                              ¡
                                                               
                                                               
                                                               
                                                              ¡   ¡
                                                                  ¡
                                                                   
                                                                   
                                                                   
                                                                  ¡    ¡
                                                                       ¡
                                                                        
                                                                        
                                                                        
                                                                       ¡   ¡
                                                                           ¡
                                                                            
                                                                            
                                                                            
                                                                           ¡   ¡
                                                                               ¡
                                                                                
                                                                                
                                                                                
                                                                               ¡    ¡
                                                                                    ¡
                                                                                    ¡
                                                                                     
                                                                                     
                                                                                        ¡
                                                                                        ¡
                                                                                         
                                                                                         
                                                                                         
                                                                                        ¡   ¡
                                                                                            ¡
                                                                                             
                                                                                             
                                                                                             
                                                                                            ¡   ¡
                                                                                                ¡
                                                                                                 
                                                                                                 
                                                                                                 
                                                                                                ¡   ¡
                                                                                                    ¡
                                                                                                     
                                                                                                     
                                                                                                     
                                                                                                    ¡   ¡
                                                                                                        ¡
                                                                                                         
                                                                                                         
                                                                                                         
                                                                                                        ¡    ¡
                                                                                                             ¡
                                                                                                              
                                                                                                              
                                                                                                              
                                                                                                             ¡   ¡
                                                                                                                 ¡
                                                                                                                  
                                                                                                                  
                                                                                                                  
                                                                                                                 ¡   ¡
                                                                                                                     ¡
                                                                                                                     ¡
                                                                                                                      
                                                                                                                      
                                                                                                                         ¡
                                                                                                                         ¡
                                                                                                                          
                                                                                                                          
                                                                                                                          
                                                                                                                         ¡   ¡
                                                                                                                             ¡
                                                                                                                              
                                                                                                                              
                                                                                                                              
                                                                                                                             ¡   ¡
                                                                                                                                 ¡
                                                                                                                                  
                                                                                                                                  
                                                                                                                                  
                                                                                                                                 ¡   ¡
                                                                                                                                     ¡
                                                                                                                                      
                                                                                                                                      
                                                                                                                                      
                                                                                                                                     ¡    ¡
                                                                                                                                          ¡
                                                                                                                                           
                                                                                                                                           
                                                                                                                                           
                                                                                                                                          ¡       ¡
                                                                                                                                                  ¡
                                                                                                                                                   
                                                                                                                                                   
                                                                                                                                                   
                                                                                                                                                  ¡
Intelligent Mining for Time Series Predictions                                   Constrained Optimizations Using Neural Networks


                             Previous Work on Handling Lags

    • Extending raw data based on pre-defined assumptions [Masters 95]
         – Flat extension
         – Mirror extension
              53
              52
              51
              50
              49
              48
              47                                                                            raw data
              46
                                                                                            true low−pass date
                                                                                            flat extension
              45                                                                            mirror extension.
              44
                   0                5                10           15          20                  25                30
                                             Low pass ends                  Today

Benjamin W. Wah                                                                                                              39




Intelligent Mining for Time Series Predictions                                   Constrained Optimizations Using Neural Networks


                Issues in Existing Methods for Lag Problem

    • Large mean of absolute errors (M AE) between predictions and targets at the end
      of lag period
         – Need to predict last three data in the lag period


                       1.2
                                                                       Mirror extension
                         1                                               Flat extension
                       0.8
                       0.6
                       0.4
                       0.2
                         0
                             1          2        3        4   5        6     7          8         9        10


Benjamin W. Wah                                                                                                              40
Intelligent Mining for Time Series Predictions                                       Constrained Optimizations Using Neural Networks


                    (F) Constrained Formulation with Cross-Validation

    • Constrained formulation without all closed-form functions
                                                            n
                                                    1
                         minw E(w) =                n           max{(ot(w) − dt)2 − τ, 0}
                                                        t=1
                         s.t.       ht(w) = (ot(w) − dt)2 ≤ τ,
                                    hI (w) = εI ≤ τiI ,
                                     i        i           (iterative validation)
                                     S        S      S
                                    hi (w) = εi ≤ τi ,    (single-step validation)
                                        Errorlag ≤ τlag   (sum of errors in lag period)

    • Transformed into non-differentiable augmented Lagrangian function:
                                                        n
                   L(w, λ) = E(w) +                              λt max{0, ht − τ } + 1 max2{0, ht − τ }
                                                                                      2
                                                     t=1
                                            v
                                                                                      1
                                     +                          λi max{0, εi − τk } + 2 max2{0, εi − τk }
                                                                 k         k
                                                                                i
                                                                                                 k
                                                                                                      i
                                         k=1 i=I,S
                                     + max (0,                   Errorlag − τlag )

Benjamin W. Wah                                                                                                                  41




Intelligent Mining for Time Series Predictions                                       Constrained Optimizations Using Neural Networks


                                         (G) Search for Saddle Points

    • Constrained formulation solvable by Theory of Lagrange Multipliers for Nonlinear
      Discrete Constrained Optimization [Wah & Wu 1999]
    • Discrete-neighborhood saddle point ⇐⇒ constrained local minimum
         – Local minimum of L(w, λ) in w subspace
         – Local maximum of L(w, λ) in λ subspace
                                            minx [f (x) + λT h(x)]↓
                                         Gradient descents in x space
                                         to reduce objective function
                                           and constraint violations
                                                                      Equilibrium point where
                           Penalties are                              constraints are satisfied
                           dynamically                                       (h(x) = 0)
                             varying                                 and objective is minimum
                                                                         ( xL(x, λ) = 0)
                                       λ↑           Gradient ascents in λ      λ↑
                                                 space to increase penalties
                                                   on violated constraints


Benjamin W. Wah                                                                                                                  42
Intelligent Mining for Time Series Predictions                           Constrained Optimizations Using Neural Networks


                          Violation-Guided Back Propagation (VGBP)


    • Gradient descents in w subspace and stochastic acceptance of ascents

         – Using BP to generate approximate gradient direction in L(w, λ)

         – Accepting trial points with Metropolis probability using fixed temperature T

                                                                min(0,L(w)−L(w ))
                                           AT (w , w)|λ = exp           T



    • Gradient assents in λ subspace by deterministic increases of λ

         – Large violation ⇒ increased λ ⇒ more penalty

Benjamin W. Wah                                                                                                      43




Intelligent Mining for Time Series Predictions                           Constrained Optimizations Using Neural Networks


                                          Relax-and-Tighten Strategy

    • Observations

         – Looser constraints
             ⇒ Faster convergence and larger maximum violation at convergence

         – Tighter constraints
             ⇒ Slower convergence and smaller maximum violation at convergence

    • Relax-and-Tighten strategy

         – Loose constraints in the beginning and tighten gradually
             ⇒ Faster convergence, and smaller maximum violation at convergence



Benjamin W. Wah                                                                                                      44
Intelligent Mining for Time Series Predictions                                                  Constrained Optimizations Using Neural Networks


                                                                 Relax-and-Tighten Strategy


                                                        3



                          Average maximum violation
                                                        1

                                                                    R-and-T
                                                       0.3             τ=0
                                                                    τ = 0.05
                                                                     τ = 0.1
                                                       0.1          τ = 0.15
                                                                     τ = 0.2
                                                                                                envelope
                                                      0.03
                                                             1        10          100          1000         10000
                                                                           N (number of iterations)




Benjamin W. Wah                                                                                                                             45




Intelligent Mining for Time Series Predictions                                                                                          Outline


                                                                             Outline

    • Market trend prediction problem
         – Time series predictions
         – Metrics
    • Signal processing of time series
         – Lags in predictable low-frequency components
    • Data mining techniques
         – Intelligent mining and major design issues
         – Prediction agents
    • Constrained optimizations using neural networks
         – Lagrange multipliers for discrete constrained optimization
    • Some sample results
Benjamin W. Wah                                                                                                                             46
Intelligent Mining for Time Series Predictions                                                            Some Sample Results


                                                       Experiments Setup

    • Predictors compared
         – CC: carbon copy the most recently available data
         – AR: Auto-regression
         – FE-NN: Proposed neural network predictor
         – IP: Ideal predictor by using 7 true data in lag and trained by VGBP
             (approximate upper bound for predictions)
         – Results presented in most literatures have next-day hit rates below 55%
             [Gutjahr 97, Hellstrom 2000]
    • Stocks
         – Citigroup (Symbol C), IBM (IBM), Exxon-Mobil (XOM)
         – Duration: 04/1997 to 03/2002

Benjamin W. Wah                                                                                                           47




Intelligent Mining for Time Series Predictions                                                            Some Sample Results


                                           Predictions for Citigroup
    • nM SE                               0.18
                                          0.16             CC
                                          0.14          AR(30)
                                                        FE-NN
                                          0.12              IP
                                 nMSE




                                           0.1
                                          0.08
                                          0.06
                                          0.04
                                          0.02
                                             0
                                                   0             2       4              6     8      10
                                                                              Horizon
    • Hit rate                            75
                                                                                               CC
                                          70                                                AR(30)
                                                                                            FE-NN
                                          65                                                    IP
                                 Rate %




                                          60
                                          55
                                          50
                                          45
                                          40
                                               0            2        4                  6     8      10
                                                                             Horizon


Benjamin W. Wah                                                                                                           48
Intelligent Mining for Time Series Predictions                                                            Some Sample Results


                                                       Predictions for IBM
    • nM SE                               0.25
                                                           CC
                                           0.2          AR(30)
                                                        FE-NN
                                          0.15              IP




                                 nMSE
                                           0.1

                                          0.05

                                               0
                                                   0             2       4              6     8      10
                                                                              Horizon
    • Hit rate                            75
                                                                                               CC
                                          70                                                AR(30)
                                                                                            FE-NN
                                          65                                                    IP
                                 Rate %




                                          60

                                          55

                                          50

                                          45
                                               0            2        4                  6     8      10
                                                                             Horizon


Benjamin W. Wah                                                                                                           49




Intelligent Mining for Time Series Predictions                                                            Some Sample Results


                                     Predictions for Exxon-Mobil
    • nM SE                               0.35
                                           0.3             CC
                                                        AR(30)
                                          0.25          FE-NN
                                                            IP
                                 nMSE




                                           0.2
                                          0.15
                                           0.1
                                          0.05
                                               0
                                                   0             2       4              6     8      10
                                                                              Horizon
    • Hit rate                            80
                                                                                               CC
                                          75                                                AR(30)
                                                                                            FE-NN
                                          70                                                    IP
                                 Rate %




                                          65
                                          60
                                          55
                                          50
                                          45
                                               0            2        4                  6     8      10
                                                                             Horizon


Benjamin W. Wah                                                                                                           50
Intelligent Mining for Time Series Predictions                                      Conclusions


                                                 Conclusions

  Signal processing is useful for

    • Generating frequency components with shorter lags and better correlations

         – Low-frequency components have stronger long-term correlations but long lags
         – High-frequency components are not useful due to long lags and low correlations

  Data mining is useful for

    • Identifying information that can form new constraints or biases in learning

    • Discovering promising input transformations in different frequency bands

  Nonlinear constrained optimization is useful for

    • Nonlinear predictions

    • Multi-stage planning

Benjamin W. Wah                                                                             51

				
DOCUMENT INFO
Description: 2002 Stock Market Predictions document sample