Document Sample

2. Bayes Decision Theory Prof. A.L. Yuille Stat 231. Fall 2004. Decisions with Uncertainty • Bayes Decision Theory is a theory for how to make decisions in the presence of uncertainty. • Input data x. • Salmon y= +1, Sea Bass y=-1. • Learn decision rule: f(x) taking values Decision Rule for Fish. • Classify fish as Salmon or Sea Bass by decision rule f(x). Basic Ingredients. • Assume there are probability distributions for generating the data. • P(x|y=1) and P(x|y=-1). • Loss function L(f(x),y) specifies the loss of making decision f(x) when true state is y. • Distribution P(y). Prior probability on y. • Joint Distribution P(x,y) = P(x|y) P(y). Minimize the Risk • The risk of a decision rule f(x) is: • Bayes Decision Rule f*(x): • The Bayes Risk: • Minimize the Risk. • Write P(x,y) = P(y|x) P(x). • Then we can write the Risk as: • The best decision for input x is f*(x): Bayes Rule. • Posterior distribution P(y|x): • Likelihood function P(x|y) • Prior P(y). • Bayes Rule has been controversial (historically) because of the Prior P(y) (subjective?). • But in Bayes Decision Theory, everything starts from the joint distribution P(x,y). Risk. • The Risk is based on averaging over all possible x & y. Average Loss. • Alternatively, can try to minimize the worst risk over x & y. Minimax Criterion. • This course uses the Risk, or average loss. Generative & Discriminative. • Generative methods aim to determine probability models P(x|y) & P(y). • Discriminative methods aim directly at estimating the decision rule f(x). • Vapnik argues for Discriminative Methods: Don’t solve a harder problem than you need to. Only care about the probabilities near the decision boundaries. Discriminant Functions. • For two category case the Bayes decision rule depends on the discriminant function: • The Bayes decision rule is of form: • Where T is a threshold, which is determined by the loss function. Two-State Case • Detect “target” or “non-target”. • Let loss function pay a penalty of 1 for misclassification, 0 otherwise. • Risk becomes Error. Bayes Risk becomes Bayes Error. • Error is the sum of false positives F+ (non- targets classified as targets) and false negatives F- (targets classified as non-targets). Gaussian Example: 1 • Is a bright light flashing? • n is no. photons emitted by dim or bright light. 8. Gaussian Example: 2 • are Gaussians with means and s.d. . • Bayes decision rule selects “dim” if ; • Errors: Example: Multidimensional Gaussian Distributions. • Suppose the two classes have Gaussian distributions for P(x|y). • Different means but same covariance • The discriminant function is a plane: • Alternatively, seek a planar decision rule without attempting to model the distributions. • Only care about the data near the decision boundary. Generative vrs. Discriminant. • The Generative approach will attempt to estimate the Gaussian distributions from data – and then derive the decision rule. • The Discriminant approach will seek to estimate the decision rule directly by learning the discriminant plane. • In practice, we will not know the form of the distributions of the form of the discriminant. Gaussian. • Gaussian Case with unequal covariance. Discriminative Models & Features. • In practice, the Discriminative methods are usually defined based on features extracted from the data. (E.g. length and brightness of fish). • Calculate features z=h(x). • Bayes Decision Theory says that this throws away information. • Restrict to a sub-class of possible decision rules – those that can be expressed in terms of features z=h(x). Bayes Decision Rule and Learning. • Bayes Decision Theory assumes that we know, or can learn, the distributions P(x|y). • This is often not practical, or extremely difficult. • In real problems, you have a set of classified data • You can attempt to learn P(x|y=+1) & P(x|y=-1) from these (next few lectures). • Parametric & Non-parametric approaches. • Question: when do you have enough data to learn these probabilities accurately? • Depends on the complexity of the model. Machine Learning. • Replace Risk by Empirical Risk • How does minimizing the empirical risk relate to minimizing the true risk? • Key Issue: When can we generalize? Be confident that the decision rule we have learnt on the training data will yield good results on unseen data? Machine Learning • Vapnik’s theory gives a mathematically elegant way of answering these issues. • It assumes that the data is sampled from an unknown distribution. • Vapnik’s theory gives bounds for when we can generalize. • Unfortunately these bounds are very conservative. • In practice, train on part of dataset and test on other part(s). Extensions to Multiple Classes Conceptually straightforward – see Duda, Hart & Stork. The decision partitionsf the feature space into k subspaces ik1 i i j , i j 5 3 2 1 4

DOCUMENT INFO

Shared By:

Categories:

Tags:
loss function, decision rule, decision theory, bayes rule, bayesian decision theory, probability of error, prior distribution, bayes decision theory, decision problem, discriminant functions, risk function, bayes estimator, state of nature, statistical decision theory, pattern recognition

Stats:

views: | 27 |

posted: | 8/29/2010 |

language: | |

pages: | 21 |

OTHER DOCS BY xny14254

How are you planning on using Docstoc?
BUSINESS
PERSONAL

By registering with docstoc.com you agree to our
privacy policy and
terms of service, and to receive content and offer notifications.

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.