Estimating by MikeJenny


									Learning and Making Decisions
When Costs and Probabilities are
        Both Unknown
    by B. Zadrozny and C. Elkan
•   Introduce the Problem
•   Previous work
•   Direct Cost Sensitive Decision Making
•   The Dataset
•   Estimating Class Membership Probabilities
•   Estimating Costs
•   Results and Conclusions
• Costs/Benefits are the values assigned to
  classification decisions.
• Cost are often different for different
• Often we are interested in the rare class in
  cost-sensitive learning
• Hence the problem of unbalanced data
     Cost Sensitive Decisions
• Each training and test example has associated
• General optimal prediction
         arg min  P( j | x)C (i, j , x)
             i     j
• Methods differ w.r.t. P( j | x) and C (i, j , x)
• Previous literature has assumed cost are known
  in advance and independent of examples.

         C (i, j, x)  C (i, j, y) x, y
• Estimation of P( j | x)
   – Estimated in training only.
• Estimation of C (i, j )
   – Example independent
• Training changes labelling to its optimum
• Learns a classifier to predict labeling of test
Direct Cost-Sensitive-Decision-
• Estimation of P( j | x)
   – Average of Naïve Bayes and Decision Trees .
   – Estimated on training and test sets.
• Estimation of C (i, j , x)
   – Multiple Linear Regression.
   – Unbiased estimate using Heckman procedure
   – Example dependant.
• Evaluation
   – Evaluate against MetaCost and KDD competition
   – Using a large and difficult dataset, KDD.
    MetaCost Implementation
• For evaluation MetaCost is adapted
  – Probability class estimates found by simple methods
    using decision trees
  – Cost are made example dependant during training
• Adapted MetaCost vs. DCSDM
  – DCSDM uses two models on test example MetaCost
  – Estimation P( j | x) was made in both training and test
    examples in DCSDM.
      The Data Mining Task
• Data on persons who have donated in
  the past to a certain charity, KDD '98
  – Non-donor and donor labelling based on
    last campaign
• The task is to choose which donors to
  ask for new donations
• Training/Test set
  – 95,412 records, labelled donors or non-
    donors and donation amount
  – 96,367 unlabelled records from same
    donation campaign
     Data Mining Task cont.
• Cost of soliciting $0.68
• Donations range from $1-200
• 5% donors and 95% non-donors
  – Very low response rate and varying
    donations make hard to beat soliciting to
• The dataset set is hard
  – Already been filtered to be a reasonable set
    of prospects
  – The task is to improve upon the unknown
    method that produced the set
              Applied DCSDM
• In KDD we will change C(i,j,x) to B(i,j,x)
   – Costs become benefits

                    Actual             Actual
                    Non-donor          Donor
 Predict            0                  0
 Predict Donor      -0.68              y(x)-0.68

• B(1,1,x) is example dependant
   – Replaced by a constant by previous literature
                Optimal policy
• The expected benefit of not soliciting, i = 0
 P( j  0 | x) B(0,0, x)  P( j  1 | x) B(0,1, x)  0
• Expected benefit of soliciting, i = 1
  P( j  0 | x) B(1,0, x)  P( j  1 | x) B(1,1, x)
   (1  P( j  1 | x))(0.68)  P( j  1 | x)( y ( x)  0.68)
   P( j  1 | x) y ( x)  0.68

• Optimal policy: P( j  1 | x) y ( x)  0.68  0
                  P( j | x)
• Optimal decisions require P ( j | x)
• Class sizes may be highly unbalanced
• Two methods proposed
  – Decision Trees - Smoothing Curtailment
  – Naïve Bayes - Binning
   Problems w. Decision Trees
• Decision trees assign as a score to each leaf
  the raw training frequency of that leaf.
• High Bias
  – Decision trees growing methods try to make leaves
    Homogeneous. p’s tend to be over or under
• High Variance
  – When n is small p not to be trusted.
• Pruning is no good for us.
• To make the estimates less extreme lets replace:

                    k       k  b.m
                 p   p' 
                    n        nm
• b – base rate, m – heuristic value (smoothing strength)
• Effect
   – where k, n small p’ essentially just base rate.
   – If k, n larger then p’ is ‘combination’ of base rate and original
Smoothed Scores
• What if the leaves have enough training
  examples to be statistically reliable?
  – Then smoothing seems to be unnecessary.
• Curtailment searches through the tree and
  removes nodes where n < v.
  – V chosen either through cross-validation, or a
    heuristic, like b.v = 10.
Curtailed Tree
Curtailed Scores
       Naïve Bayes Classifiers
• Naïve Bayes
   – Assumes that within any class the attribute values are
     independent variables.
• This assumption gives inaccurate probability
• But, attributes tend to be positively correlated so
  naïve Bayes estimates tend to be too extreme,
  i.e. close to zero or one.
• So, they do rank examples well:
if n( x)  n( y) then P( j  1 | x)  P( j  1 | y)
Calibrating Naïve Bayes Scores
• The Histogram method:
   – Sort training examples by n.b. scores
   – Divide sorted set into b subsets of equal size,
     called bins
   – For each bin compute lower and upper
     boundary n.b. scores
• Given a new data point x
   – Calculate n(x ) and find the associated bin
   – Let P( j | x) = fraction of positive training
     examples in that bin
Averaging Probability Estimates
• If probability estimates are partially uncorrelated
  then it follows that averaging these estimates will
  reduce their variance.
• Assuming all estimates have the same variance
  the average estimate will have a variance given
  by:             2  1   ( N  1) 2
                                 
         2   The individual classifier variances

          N   The number of classifiers
             The correlation factor among all classifiers
   Estimating Donation Amount
• Solicit the person based on policy.
• Policy

         P( j  1 | x) y ( x)  0.68  0

           y (x )   is estimated donation amount
          Cost and Probability
• Good Decisions
  – Estimating Cost well is more important than
    estimating probabilities.
• Why?
  – Relative variation of cost across different examples is
    much greater than the relative variation of
• Probability
  – Estimating Donation probability is difficult.
• Estimating donation amount are easier because
  past amount are excellent predictor of future
       Training and Test data
• Two random process
  – Donate or not to.
  – How much to donate?
  Donation Amount.
                  Donor        Non donor
  Training data   Known         -
  Test data       unknown      unknown

• Method used for estimating donation amount is
  called as Multiple Linear regression (MLR).
    Multiple Linear Regression
Two attributes are used
  – lastgift : dollar amount of most recent gift.
  – ampergift : average gift amount in response to the last
    22 promotions
• Linear Regression equation is used to estimate
  donation amount.
• 46 of 4843 donations recorded have donation
  amount more than $50.
• Donors that have donated at most $50 are used
  as input for linear regression.
   Problem of Sample Selection
• Reasoning outside your learning space.
• Donation Amount
                   Donor                Non donor
Training data      Known                -
Test data          unknown              unknown

• Estimating Donation Amount
  – Any donation estimator is learned on the basis of
    people who actually donated.
  – This estimator is applied to different population
    consisting of donors and non-donors.
Donation Amount and Probability
   Estimates are Negatively
  Solution to Sample Selection
• Heckman’s procedure
  – Estimate conditional probability p( j=1 | x) using linear
    probit model.
  – Estimate y(x) on training dataset for which j (x) = 1
    by including a transformation for each x using the
    estimated values of conditional probability.
• Their own procedure
  – conditional probability is learned using decision tree
    or Naïve bayes classifier.
  – These probability estimates are added as additional
    attributes by estimating y(x).
       Experimental Results
Direct cost sensitive Decision Making

Meta cost
         Experimental Results
• With Heckman
  – profits on test set increases by $484, in all probability
    estimation methods.
  – Systematic improvement indicates that Heckman’s
    procedure solves the problem of Sample Selection
• Meta cost
  – Best result of Meta cost is $14113.
  – Best result of Direct cost sensitive method is $15329.
  – On an average, profit achieved in Meta Cost on test
    set is $1751 lower than the profit achieved in case of
    direct cost-sensitive decision making.
 Statistical Significance of Results
• 4872 donors in fixed test set
• Average donation of $15.62
• Different Test set drawn randomly from same
  probability distribution would expect a standard
  deviation of sqrt(4872)
• Fluctuation will cause a change of about $1090.
             sqrt(4872) * 15.62 = $1090.
• Profit Difference between two methods less than
  $1090 is not significant.
• Cost sensitive learning is better than Meta cost.
• Provides solution to fundamental problem of cost
  being example dependent.
• Identify and solves the problem of Sample
  Selection Bias for KDD’98 dataset

To top