Progress report
Peer effects in BC schools
Brian Krauth, SFU
Jane Friesen, SFU
Review of Project
• Uses individual-level data on BC Grade 4 and 7
students from 1999-2005.
– Home language, sex, aboriginal status, special
needs, French immersion.
– FSA scores in reading and numeracy
• Goal: measure influence of composition of
same-grade schoolmate peer group on exam
performance.
– % male, % aboriginal, % speaking each major
language.
– Cohort-based design to address endogeneity of
school.
Progress since initial report
• We have our new data!
– One more year.
– Principal codes
– Information on grades 5 & 6 peer group composition.
– Updated FSA scores
• What have we done with it? At this point we are
checking/cleaning it
– Updated FSA reading scores don’t match original scores for
1999. Why?
– There’s a major error in the grade 7 FSA math scores.
– Principals: it turns out that there is a lot of principal turnover.
Good!
– Grades 5/6 data in very ugly format.
Next steps
• Finish integrating the new data and
address remaining questions
• Learn more about “IRT” (Item Response
Theory) test scores.
• Estimate basic value-added regressions.
Comments on papers
General comments and advice
• On empirical papers
– Be careful not to overemphasize
• Describing what other people have done.
• Technical details.
• Hypothesis tests.
– Be careful to sufficiently emphasize
• Describing what you have done, and why.
• Measuring a parameter that is of likely interest to readers, in units they understand,
using credible identifying assumptions (as much as possible).
• On theory/policy/methodology papers
– Your paper will probably be more interesting if you choose a point of view on the
question you are discussing.
– However, you must remember that no one cares what you think. You have no
authority to appeal to, so you must use argument to persuade others.
• Provide support for all potentially false statements.
• Acknowledge potential counterarguments, and address them if possible.
• Don’t just say “I think that…”
– Don’t waste words.
Estimation of Treatment
Effects
The idea
• We spend a lot of time in econometrics class
talking about estimating linear models, and
about when we can interpret the coefficients in a
structural/causal fashion.
• There is another approach to measuring causal
effects
– Called “treatment effects”
– Has origins in medical statistics, now used by
econometricians in things like program evaluation
Definitions
• Suppose we have data on
– Some outcome Y.
– Some treatment T.
– Some additional covariates X.
• Standard econometric approach: assume that
– Y= β0 + β1T + β2X + ε.
– E(ε|X,T)=0.
– Then β1 is the effect of T on Y.
• Weaknesses
– Effect is often heterogeneous across individuals
– Linearity might not hold.
– How do we know which covariates to include?
Counterfactuals
• We simplify by assuming T is binary.
– T = 1 : “treated”
– T = 0 : “untreated”
– Extension to continuous T is also possible.
• Suppose that each individual case has actually two
outcomes:
– Y1 = outcome the case would have if treated.
– Y0 = outcome the case would have if untreated.
– We allow both of these to be heterogeneous across cases.
• The “treatment effect” of T is defined as Y1-Y0.
• Sadly, we only observe Y = Y1T + Y0(1-T)
– Either Y0 or Y1 is a counterfactual outcome.
What we might want to know
• Average treatment effect (ATE):
– E(Y1-Y0) = E(Y1)-E(Y0)
– Compares a world in which everyone gets the treatment to one
in which no one gets it.
• Treatment-on-treated effect:
– E(Y1-Y0|T=1)
– Compares the current world to one in which no one gets the
treatment.
• Treatment-on-untreated:
– E(Y1-Y0|T=0)
– Compares the current world to one in which everyone gets the
treatment.
• We could also consider ATE/TOT/TOU for particular
sub-populations (e.g., married working-age women).
Identification and estimation
• What can we estimate from data on
(Y,X,T)?
– We can estimate E(Y1|T=1) and E(Y0|T=0).
– We cannot estimate E(Y1|T=0) or E(Y0|T=1).
• Best case: random selection into treatment
– Then E(Y1|T=0) = E(Y1|T=1) = E(Y1).
– And E(Y0|T=1) = E(Y0|T=0) = E(Y0).
– So ATE = E(Y1|T=1) - E(Y0|T=0).
– In this case ATE=TOT=TOU.
Selection on observables
• Next best case: “selection on observables”.
– E(Y1|T,X) = E(Y1|X)
– E(Y0|T,X) = E(Y0|X)
– This means that cases can be selected into treatment on the basis of X,
as long as they are not selected on the basis of any other outcome-
relevant variable.
– In this case:
• ATE = E(E(Y1|X,T=1) – E(Y0|X,T=0))
• What needs to be in X?
– Covariates that are “balanced” in the treatment and control group do not
need to be in there. For example if % male is the same in both
treatment and control group, controlling for sex will not matter.
– Actually, you only need to control for the scalar random variable
Pr(T=1|X), also known as the propensity score.
– It is possible to estimate the treatment effect by first estimating the
propensity score (by a logit or probit model) and then estimating a
regression of Y on T and Pr(T=1|X).
Instrumental variables
• Suppose that selection into treatment is primarily endogenous, but
there is an exogenous variable that influences selection.
– Example: Vietnam-era draft lottery in US.
– To simplify, suppose it’s a binary variable V.
– We assume exogeneity: E(Y1|V)=E(Y1), E(Y0|V)=E(Y0)
• What can we identify?
– OLS regression of Y on V: the “intent to treat” effect, i.e. the average
treatment effect of V on Y.
– IV regression of Y on T, using V as an instrument: the local average
treatment effect (LATE):
• E(Y1-Y0|would switch from T=0 to T=1 if V switched from 0 to 1).
– If the effect is homogeneous across the population then ATE=LATE.
• Panel data/fixed effects estimates can be interpreted similarly.
Bounds etc.
• Sometimes we can make weaker assumptions
and still get something.
• Suppose E(Y0|T)=E(Y0) (but not E(Y1|T)=E(Y1)).
– Then we can estimate the TOT effect:
• TOT = E(Y1|T=1) – E(Y0|T=1)
= E(Y1|T=1) – E(Y0|T=0)
• We might alternatively assume that E(Y0|T=1) ≥
E(Y0|T=0) or E(Y1|T=1)≥E(Y0|T=1).
– This will allow us to place an upper or lower bound on
some treatment effect of interest.