Learning To Predict rare events in event sequences

							Learning To Predict rare events
            in event sequences

               By
    Gary M Weiss and Haym Hirsh
          Presented by
        Veena Raja Rathna
                                      Contents
• Aim & Introduction to the problem
• Basic problem Formulation
• Definitions & Evaluation metrics
• Learning methods
• Example & Results
      TimeWeaver
• Reference
                                AIM
         Introduction to the problem
• To predict rare events from sequences of events
  which contain non numerical features
• An event sequence is a time stamped observations
  described by a fixed set of features
• Why
• Predicting events with categorical data is an
  important real world problem
• Not Suited to be solved by existing statistical
  and machine learning methods
                                   Background
• Classical time series - Predict next „n‟ successive
  observations from a history of past observations
• Statistical techniques not applicable to event
  prediction because
      they require numerical features
     do not support predicting a specific „event‟
  within window of time.
                               Examples

• Telecommunication equipment failure
• Predicting Fraudulent credit card
  transactions
• Start of Transcription in DNA sequences
        Basic Problem Formulation
• Definitions
• Event- Et: is a time stamped observation which
  occurs at time „t‟ and is described by a set of
  feature value pairs.
• Event Sequence- S: is a time ordered sequence of
  events.S=Et1,Et2,….Etn
• Domain Object- D: events are associated with D
  which is a generator of events
                                    Definitions
• Target Event Xt :is the event to be predicted and is
  specified by a set of feature value pairs.
• Warning Time W:is the lead time necessary for a
  prediction to be useful
• Monitoring Time M: determines the maximum
  amount of time prior to target event for which a
  prediction is considered correct.
              Problem now reduces to
• Learning a prediction procedure P that correctly
  predicts the target events.
• P is a function that maps an event sequence to a
  boolean prediction value.
• P:Et1,Et2,Et3,…Etn-->{+,-}
• A target event is predicted if at least one
  prediction is made within its prediction period
• A prediction is correct if it falls within the
  prediction period of some target event
    Evaluation Measures for Event
                      Prediction
• Recall = #Target Events Predicted /Total target
  events
    It is the percentage of target events correctly
  predicted
• Simple Precision = TP/TP+FP
  TP - true predictions       FP - false Predictions
     Simple precision is the percentage of predictions
  that are correct
                     Evaluation Metrics
• Normalized precision = #Target Events
  Predicted/#Target Events Predicted +FP
    Replaces # of correct predictions with target
    events correctly predicted
• Reduced Predictions = #Target Events predicted/#
  Target Events Predicted+Discounted FP
• A prediction is active for a period equal to its
  monitoring time
                  Basic Learning Method
• Identify prediction patterns : The space of PP is searched
  to identify a set „C‟ of candidate PP.Each pattern c  C
  should do well at predicting a subset of target events.
• Generate prediction rules : An ordered list of PP is
  generated from C. Prediction rules are then formed by
  creating a disjunction of the top n PP,thereby creating
  solutions with different precision/recall values.
• PP is a sequence of events connected by ordering
  primitives that define sequential or temporal constraints
  between consecutive events
               Ordering primitives are
• Let A B C D represent individual events
• „Wildcard‟ “*” matches any number of events so
  the PP A*D matches ABCD
• „next‟ “.” matches no events so the PP A.B.C only
  matches ABC
• „unordered‟ “|” allows events to occur in any order
  and is commutative so that the PP A|B|C will
  match ,amongst others,CBA
                                        Example
• “|” has the highest precedence so the pattern
  “A.B*C|D|E” matches A followed by B,followed
  sometime later by a C,D and E in any order.
• Each feature in an event is permitted to take on the
  “?” value that matches any feature value.
• PP also has an integer valued pattern duration
                              Learning Method
• A PP matches an event sequence if
•    1) events within the PP matches events within an event
  sequences
•    2) ordering constraints in the PP are obeyed
•    3) events involved in the match occur within the pattern
  duration
• This language enables flexible and noise tolerant
  prediction rules to be constructed, such as
• if 3(or more) A events & 4( or more B events occur within
  an hour, then predict the target event.
                  GA for identifying PP
• Use GA to identify a diverse set of PP.
• Each individual in the GA‟s population represents
  part of a complete solution and should perform
  well at classifying a subset of the target events.
• GA used is steady state GA where only a few
  individuals are modified each „iteration‟ .
                             Basic steps in GA
•   1. Initialize population
•   2. while stopping criteria not met
•   3. select 2 individuals from the population
•   4. apply mutation operator to both individuals with Pm;
•         else apply crossover operator
•   5. evaluate the 2 newly formed individuals
•   6. Replace 2 existing individuals with the new ones
• 7.done
           Selection and Replacement
                             Strategy
• GA‟s selection and replacement strategy must
  balance two opposing criteria
      • 1. They must focus the search in the most profitable
        areas of the search space.
      • 2. Maintain a diverse population to avoid premature
        convergence and to ensure that the individuals in the
        population collectively cover most of the target
        events.
               Selection & Replacement
                               Strategy
• Fitness of the PP is based on both its precision and
  recall and is given by
         fitness = ((* )+1)precision recall
           -------------------------------------
                (* )precision+recall
• To encourage diversity we use a strategy called
  sharing that rewards individuals based on how
  different they are from other individuals in the
  population
             Selection & Replacement
                             Strategy
• Individuals are selected proportional to their
  shared fitness = fitness/ count
• count measures the degree of similarity of
  individual j to p individuals comprising the
  population
        count j=(1 -distance(j k))3
The similarity of 2 individuals is measured using a
  phenotypic distance metric that measures the
  distance based on the performance of the
  individual
                                Observations
• The more similar an individual to the rest of the
  individuals in the population, smaller the distances
  & greater the count.
• Replacement strategy also uses shared fitness.
• Individuals are chosen for deletion inversely
  proportional to their shared fitness
            Creating Prediction Rules
• Greedy Algorithm shown is used to form a list of
  prediction rules S from a set of candidate patterns
  C
• Precision, recall and prediction vector information
  computed in the first step for each PP are used.
• Step 11 requires access to the training set and is
  the most time-intensive step
                                                    Algorithm
•   1.     C=patterns returned from the GA;S={};
•   2.      while C!=0 do
•   3.       for c  C do
•   4.        if(increase_recall(S+c,S)<=THRESHOLD)
•   5.        then C=C-c;
•   6.        Elseeval=PF*(c.precisionS.precision)+increase_recall(S+c,S);
•   7.      done
•   8.     Best={c C,x  C|c.eval>=x.eval}
•   9.     S=S||best;C=C=best;
•   10.    recompute S.precision on the training set;
•   11.   done
                                  Observations
• Builds solution with increasing recall by
  heuristically selecting the best PP in C using eval
  function in line 6.
• PF controls the importance of precision vs
  recall.PPs that do not increase the recall by at least
  THRESHOLD are discarded.
• Both THRESHOLD AND PF affect the
  complexity of t he learned concept and can
  prevent overfitting of the data
                                         TimeWeaver
• This paper describes timeweaver a GA based ML
  system that predicts rare events
• For AT & T ,the specific task is to predict failure
  of 4ESS h/w components from alarm messages.
• Problem Formulation
             • Each 4ESS generated alarm is an event with 3
               features--device,severity ,code
             • each 4ESS is a domain object that generates an event
               sequence and the target event is any event with code
               set to „FAILURE‟
                                 Experiments
• Training set 110,000 alarms reported from 55
  4ESS switches.
• Test set contained 40,000 alarms from 20 different
  4ESS switches.This data included 1200 alarms
  which indicated equipment failure.
• THRESHOLD was 1% and PF was 10
• Pattern 351:<TMSP,?,MJ>*<?,?,MJ>*<?,?MN>
  indicates that within 351 sec time period a major
  severity alarm on a TSMP device is followed by a
  major alarm and minor alarm
                                         Results

• Varying the warning time demonstrates that it is
  much easier to predict failures when short warning
  time is required.
• Increasing the monitoring time significantly
  improves TW‟s ability to predict failures.
• Larger prediction period leads TW to focus its
  attention on „spurious correlations‟ in data.
 Comparision with other methods
• Tw was compared to C4.5rules and RIPPER - 2 rule
  induction systems and FOIL a system that learns logical
  definitions from relations.
• Class distribution of the generated examples is skewed;
  prevented C4.5rules and RIPPER from predicting any
  failure.
• TW yields results with precision 3-5 times higher for a
  given recall value than various thresholding strategies used
  by ANSWER system. Concept space is much more
  expressive.
                                              Example
• {100:a,b},{104:c,c},{105:d,a},{108:c,c},{110:a,d},
{111:crash,c},{115:d,a},{118:,c,c},{119:a,d},{124:a,b}!
Format: Integer valued timestamp,colon,comma separated list
  of feature values.2 feature values per event and each event
  can take on the values a,b,c or d.The first can also take on
  the value “crash”. The target event is any event with
  „crash‟ as the first feature. Warning time is 2 secs and
  monitoring time is 8 secs.
TW-GA training on this might produce PP: 4:|c,c|*|c,c|.
                                          References
• Learning to predict rare events in categorical time series
  data
• http://paul.rutgers.edu/~gweiss/thesis/timeweaver.html