# Hypothesis Tests

Document Sample

```					Hypothesis Tests

IEF 217a: Lecture 2.b
Fall 2002
Hypothesis Testing
• Correct models?
• Data similar?
– Use one series to predict another
• Has something changed in the data?
– Quality control, portfolio strategies
Outline
• Proportion changes (Political polls)
• Difference in means (Airline arrivals,
Firestone)
• Testing a distribution (die)
• Causality
• Multiple comparisons and data snooping
• Statistical power
Outline
• Proportion changes (Political polls)
• Difference in means (Airline arrivals,
Firestone)
• Testing a distribution (die)
• Causality
• Multiple comparisons and data snooping
• Statistical power
Hypothesis Testing
• Null hypothesis
– Assumption about how the world works
– Assume this is true
– Could data have come from this
machine/theory/conjecture???
– Do you need more/other data?
• Facts
– Bird normally makes 48 percent of his shots
– Bird has just finished a series of games where
he made only 20 of 57 shots
– Question: Is this the usual Larry Bird, or has
something changed?
– Is he in a slump?
– On to matlab (bird1.m)
Hypothesis Testing Terms
• Null hypothesis
• Test statistic
– Observed statistic (Random variable)
• p-value (probability null is true)
– Prob( shots <= 20 )
Outline
• Proportion changes (Political polls)
• Difference in means (Airline arrivals,
Firestone)
• Testing a distribution (die)
• Causality
• Multiple comparisons and data snooping
• Statistical power
Political Poll
• Gore/Bush 0/1
• Two polls (100 people)
– First 50/50
– Second 55/45
• What is the probability that something has
changed in the population?
• Matlab: pollchange.m
Outline
• Proportion changes (Political polls)
• Difference in means (Airline arrivals,
Firestone)
• Testing a distribution (die)
• Causality
• Multiple comparisons and data snooping
• Statistical power
Differences in Means
• Two samples
• Different means
• Could they be drawn from the same
population?
• Examples
– Has something changed?
• Flights (time)
• Tires (Firestone)
Flight Delays
• Two series (minutes late)
– Before mechanics threat of delays
– After mechanics threat of delays
•   More delays after threat
•   Compare to pooled data
•   Null = two series are the same
•   Could the mean difference between the two
come from the pooled series?
Flight Delays
• Matlab code: airline.m
• Note: Fancy histogram code
Firestone
• Overall tires have a failure rate of 5 in 1000
• You have observed in a sample of 10,000
tires a failure rate of 60
• Is something wrong with Firestone tires?
• Matlab: firestone.m
Outline
• Proportion changes (Political polls)
• Difference in means (Airline arrivals,
Firestone)
• Testing a distribution (die)
• Causality
• Multiple comparisons and data snooping
• Statistical power
Testing a Die
• Problem:
– You’ve observed the following rolls of a die
out of 6000 rolls
• 1: 1014, 2: 958, 3: 986, 4: 995, 5: 1055, 6:992
– Could this have come from a fair die with probs
of 1/6 for each side?
Dietest.m
• Method:
– Think up a test statistic
– Roll 6000 dies with sample
– Check how the value of the test statistic from
the original data compares with the distribution
from the simulations
• dietest.m
Outline
• Proportion changes (Political polls)
• Difference in means (Airline arrivals,
Firestone)
• Testing a distribution (die)
• Causality
• Multiple comparisons and data snooping
• Statistical power
Causality
• Stock returns and weather
• Are returns higher when it is sunny?
• Given some data on weather and returns test
this hypothesis
• on to matlab: sunny.m
Outline
• Proportion changes (Political polls)
• Difference in means (Airline arrivals,
Firestone)
• Testing a distribution (die)
• Causality
• Multiple comparisons and data snooping
• Statistical power
Multiple Tests and Data
Snooping
• In the search for patterns you often look at
many different things
– Different regression runs
– Different drugs
• Each is often tested alone
• Then get excited when 1 is significant
Strategies
• Efficient markets world (no predictability)
• Someone claims to have a buy/sell
(short/long) strategy which generates
significantly large returns
• They pretested 10 strategies and chose the
best out of the 10
• Return sample is independent and normal
Questions
• What is the likelihood that some “best”
strategy beats a buy and hold benchmark?
• What if this strategy were tested to see if it
tests, ignoring that it had been snooped?
• Matlab: snooptest.m
Other Applications
– More later
• Multiple regressions
– Run 20 regressions of y = a + bx for different x
– Report only those with significant b
– Common economist sin
Outline
• Proportion changes (Political polls)
• Difference in means (Airline arrivals,
Firestone)
• Testing a distribution (die)
• Causality
• Multiple comparisons and data snooping
• Statistical power
Hypothesis Tests Again
• P-value or significance level
– Probability of rejecting null hypothesis given
that it is true
P-Value, Size, and Type I error
Observe 2
Prob(x>2)
Null: Normal(0,1)
Hypothesis Tests Again
• Type II error
– Probability of accepting null hypothesis given
that it is false
Hypothesis Tests Again
• Power
– Probability of rejecting null hypothesis when it
is false
– Probability of catching a deviation
Type I and Type II errors
Which do you prefer?
– Null = Mushroom
– Type I: Reject mushroom given mushroom
– Type II: Accept mushroom given toadstool
• Makes a difference
Hypothesis Tests:
Final Word
– Correct Size
– Maximum Power
• Specific situations
– Costs of Type II error (mushrooms)
– Finance:
• Using incorrect model
• Missing risks (LTCM)
Problems for Monte-Carlo Tests
of Power
• Test a null hypothesis under some
alternative
• Need to commit to which alternative
• Power(alternative)
Outline
•   Proportion changes (Political polls)
•   Difference in means (Airline arrivals, Firestone)
•   Testing a distribution (die)
•   Causality (stocks and weather)
•   Multiple comparisons and data snooping
•   Statistical power

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 5 posted: 2/24/2012 language: pages: 33
How are you planning on using Docstoc?