Docstoc

__Ensemble'' learning algorithms

Document Sample
__Ensemble'' learning algorithms Powered By Docstoc
					              “Ensemble” learning algorithms


                  ... last lecture before reading week :-(




(COMP24111)              “Ensemble” learning algorithms      1 / 29
Projects...



Deadline is tomorrow for group H, Friday for group G.


You should submit TWO files:


COMP24111/ex2/lab2.zip ... (all your matlab files)
COMP24111/ex2/lab2.pdf ... (your report in PDF)




      (COMP24111)          “Ensemble” learning algorithms   2 / 29
What is an Ensemble System?



   The usual machine learning approach
        Select a class of models
        Fit a single model to training data
        Predict on test data
   The ensemble approach
        Select a class of models
        Fit multiple models to training data (called base learners)
        Use the models as a committee to VOTE on testing data




     (COMP24111)            “Ensemble” learning algorithms            3 / 29
Creating useful ensembles




    All models identical = no benefit
    Ensemble members need to disagree
    Amount of disagreement is called diversity
    Total disagreement and total agreement are obviously bad - why?




      (COMP24111)          “Ensemble” learning algorithms             4 / 29
Encouraging diversity...



Give each model a slightly different dataset!
    Cross Validation
          Split the data into N folds
          Fit each model using a different fold as training data
    Feature Extraction
          Generate n different feature sets
          Fit each model using a different feature set
    Can you think of any more?




       (COMP24111)            “Ensemble” learning algorithms      5 / 29
Bootstrapping




   Statistical method for generating multiple datasets from an original
   Select N training examples from the total N, with replacement




     (COMP24111)          “Ensemble” learning algorithms                  6 / 29
Bootstrap Example



              Original Dataset                     Bootstrap

                   1    6
                   2    7
                   3    8
                   4    9
                   5 10



     (COMP24111)       “Ensemble” learning algorithms          7 / 29
Bootstrap Example



              Original Dataset                     Bootstrap

                   1    6                               1
                   2    7
                   3    8
                   4    9
                   5 10



     (COMP24111)       “Ensemble” learning algorithms          8 / 29
Bootstrap Example



              Original Dataset                     Bootstrap

                   1    6                               1
                   2    7                               10
                   3    8
                   4    9
                   5 10



     (COMP24111)       “Ensemble” learning algorithms          9 / 29
Bootstrap Example



              Original Dataset                     Bootstrap

                   1    6                               1
                   2    7                               10
                   3    8                               7
                   4    9
                   5 10



     (COMP24111)       “Ensemble” learning algorithms          10 / 29
Bootstrap Example



              Original Dataset                     Bootstrap

                   1    6                               1
                   2    7                               10
                   3    8                               7
                   4    9                               3
                   5 10



     (COMP24111)       “Ensemble” learning algorithms          11 / 29
Bootstrap Example



              Original Dataset                     Bootstrap

                   1    6                               1
                   2    7                               10
                   3    8                               7
                   4    9                               3
                   5 10                                 10



     (COMP24111)       “Ensemble” learning algorithms          12 / 29
Bootstrap Example



              Original Dataset                     Bootstrap

                   1    6                               1   6
                   2    7                               10 10
                   3    8                               7   8
                   4    9                               3   1
                   5 10                                 10 6



     (COMP24111)       “Ensemble” learning algorithms           13 / 29
Bootstrapping




                                                                         1
   Each example in the original dataset has a selection probability of   n
   On average, 36.8% of the datapoints are left unselected.
                                                           1
   ... if you want to know why, consider the limit of (1 − n )n ...
   These form our testing dataset.




     (COMP24111)           “Ensemble” learning algorithms                14 / 29
Bootstrap Example



  Original Dataset          Bootstrap                   Unselected

          1        6               1       6                2
          2        7             10 10                      4
          3        8               7       8                5
          4        9               3       1                9
          5 10                   10 6




     (COMP24111)       “Ensemble” learning algorithms                15 / 29
Bagging




   BAGGING is short for Bootstrap AGGregatING
   Simple ensemble method.
   Generate m bootstraps, train a model on each one.
   When testing data arrives... do a simple majority vote.




     (COMP24111)          “Ensemble” learning algorithms     16 / 29
Bagging Diagram

                                             Dataset




                   Bootstrap   Bootstrap    Bootstrap              Bootstrap




                   Learner     Learner      Learner                Learner
                      1           2             3                     m




                                              Vote




     (COMP24111)                  “Ensemble” learning algorithms               17 / 29
Bagging Example




     (COMP24111)   “Ensemble” learning algorithms   18 / 29
Bagging Example




     (COMP24111)   “Ensemble” learning algorithms   19 / 29
Bagging Example




     (COMP24111)   “Ensemble” learning algorithms   20 / 29
Bagging Example




     (COMP24111)   “Ensemble” learning algorithms   21 / 29
Bagging Example




     (COMP24111)   “Ensemble” learning algorithms   22 / 29
Bagging Example




     (COMP24111)   “Ensemble” learning algorithms   23 / 29
Parallel vs Serial Ensemble Algorithms
Our bagging ensemble generated ensemble members in parallel.

Each was independent of the others.

Can we do a similar thing, but in serial?

                                                  Dataset




                          Bootstrap   Bootstrap   Bootstrap   Bootstrap




                          Learner     Learner     Learner     Learner
                             1           2           3           m




                                                   Vote




       (COMP24111)               “Ensemble” learning algorithms           24 / 29
The “Boosting” Algorithm


  1   Get a dataset.
  2   Take a bootstrap, and train a model on it.
  3   See which examples the model got wrong.
  4   Upweight those ’hard’ examples, downweight the ’easy’ ones.
  5   Go back to step 2, but with a weighted bootstrap.


Each new member focuses on examples that the previous ones got wrong!

If you want a committee of M members, allow this to loop M times.
When a new testing datapoint arrives, vote!




        (COMP24111)          “Ensemble” learning algorithms         25 / 29
Boosting Diagram



                                        Dataset




       Weighted                   Weighted                    Weighted
       Bootstrap   Learner        Bootstrap       Learner     Bootstrap   Learner
                     1                              2                       m




                                    Weighted Vote




     (COMP24111)             “Ensemble” learning algorithms                         26 / 29
Choosing a base learner...



    Not all models are suitable for Ensembles...
    If the model only changes a small amount when the data is changed
         It will not generate a diverse classifier
    Therefore small changes in the input data must produce large changes
    in the fitted model
         To ensure the ensemble is diverse
    This property means the models are unstable
    A decision tree is an example of an unstable classifier




      (COMP24111)             “Ensemble” learning algorithms        27 / 29
Ensemble Learning...




   Ensemble methods treat ML algorithms like a committee.
   ... used to improve the accuracy over a single model
   Two kinds of ensemble methods:
        Independent methods - such as Bagging
        Dependent methods - such as Boosting




     (COMP24111)          “Ensemble” learning algorithms    28 / 29
Finally... g’bye!

This is my last lecture.
I’ll be back nearer to Xmas for a revision lecture.


After reading week: Dr Ke Chen.




Enjoy reading week!


       (COMP24111)           “Ensemble” learning algorithms   29 / 29

				
DOCUMENT INFO