RN Diff

Document Sample
RN Diff Powered By Docstoc
					   Impact Evaluation Methods
Regression Discontinuity Design and
     Difference in Differences




   Slides by Paul J. Gertler & Sebastian Martinez
               Measuring Impact
    • Experimental design/randomization
    • Quasi-experiments
      – Regression Discontinuity
      – Double differences (diff in diff)
      – Other options




2
     Case 4: Regression Discontinuity
    • Assignment to treatment is based on a clearly
      defined index or parameter with a known
      cutoff for eligibility
    • RD is possible when units can be ordered
      along a quantifiable dimension which is
      systematically related to the assignment of
      treatment
    • The effect is measured at the discontinuity –
      estimated impact around the cutoff may not
      generalize to entire population
3
    Indexes are common in targeting of
             social programs
    • Anti-poverty programs  targeted to
      households below a given poverty index
    • Pension programs  targeted to population
      above a certain age
    • Scholarships  targeted to students with
      high scores on standardized test
    • CDD Programs  awarded to NGOs that
      achieve highest scores

4
     Example: Effect of Cash Transfer
            on Consumption
    • Target transfer to poorest households
    • Construct poverty index from 1 to 100 with
      pre-intervention characteristics
    • Households with a score <=50 are poor
    • Households with a score >50 are non-poor
    • Cash transfer to poor households
    • Measure outcomes (i.e. consumption) before
      and after transfer
5
    80
    75
    70
    65
    60        Regression Discontinuity Design - Baseline




         20    30      40       50       60       70       80
                                Score
6
    80
    75
    70
    65        Regression Discontinuity Design - Baseline




                                    Non-Poor

                            Poor
    60




         20    30      40          50          60   70     80
                                   Score
7
    80
    75
    70
    65        Regression Discontinuity Design - Post Intervention




         20        30       40       50       60       70       80
                                     Score
8
    80
    75        Regression Discontinuity Design - Post Intervention




                                         Treatment Effect
    70
    65




         20        30       40       50       60       70       80
                                     Score
9
      Case 4: Regression Discontinuity
     • Oportunidades assigned benefits based
       on a poverty index


     • Where
     • Treatment = 1 if score <=750
     • Treatment = 0 if score >750


10
     Case 4: Regression Discontinuity
                                             Baseline – No
                                              treatment
                            379.224
            Fitted values




                            153.578
                                      276                                      1294
                                            puntaje estimado en focalizacion



        yi  0  1Treatmenti   ( score)   i

                                                                                      2
11
      Case 4: Regression Discontinuity
                                          399.51         Treatment Period


                         Fitted values




                                         183.647
                                                   276                                         1294
                                                          puntaje estimado en focalizacion




                                                               Case 4 - Regression Discontinuity
                                                                Multivariate Linear Regression
     Estimated Impact on CPC                                                         30.58**
                                                                                      (5.93)
     ** Significant at 1% level


12
      Potential Disadvantages of RD
     • Local average treatment effects – not always
       generalizable
     • Power: effect is estimated at the discontinuity, so we
       generally have fewer observations than in a
       randomized experiment with the same sample size
     • Specification can be sensitive to functional form:
       make sure the relationship between the assignment
       variable and the outcome variable is correctly
       modeled, including:
        – Nonlinear Relationships
        – Interactions

13
       Advantages of RD for Evaluation
     • RD yields an unbiased estimate of treatment effect
       at the discontinuity
     • Can many times take advantage of a known rule for
       assigning the benefit that are common in the
       designs of social policy
        – No need to “exclude” a group of eligible
          households/individuals from treatment




14
               Measuring Impact
     • Experimental design/randomization
     • Quasi-experiments
       – Regression Discontinuity
       – Double differences (Diff in diff)
       – Other options




15
                Case 5: Diff in diff
     • Compare change in outcomes between
       treatments and non-treatment
        – Impact is the difference in the change in
          outcomes
     • Impact = (Yt1-Yt0) - (Yc1-Yc0)




16
     Outcome



                                                       Average
               Treatment Group                         Treatment
                                                       Effect




                                       Control Group




                                                       Time
                                 Treatment
17
     Outcome
                   Average
                   Treatment
                   Effect                         Estimated
                                                  Average
                                                  Treatment
                                                  Effect
               Treatment Group




                      Control Group



                                                   Time
                                      Treatment

18
                      Diff in Diff
     • Fundamental assumption that trends (slopes)
       are the same in treatments and controls
     • Need a minimum of three points in time to
       verify this and estimate treatment (two pre-
       intervention)




19
                       Case 5: Diff in Diff
                         Case 5 - Diff in Diff
                Not Enrolled   Enrolled                                         t-stat
      Mean ΔCPC     8.26         35.92                                          10.31

                                                  Case 5 - Diff in Diff
                                  Linear Regression        Multivariate Linear Regression
     Estimated Impact on CPC        27.66**                         25.53**
                                       (2.68)                          (2.77)
     ** Significant at 1% level




20
                 Impact Evaluation Example –
                     Summary of Results
                                                        Case 2 -                            Case 4 -
                                  Case 1 - Before                           Case 3 -                      Case 5 - Diff in
                                                       Enrolled/Not                       Regression
                                    and After                             Randomization                        Diff
                                                        Enrolled                          Discontinuity
                                    Multivariate                           Multivariate   Multivariate     Multivariate
                                      Linear        Multivariate Linear      Linear         Linear           Linear
                                    Regression         Regression          Regression     Regression       Regression
     Estimated Impact
     on CPC                        34.28**               -4.15             29.79**        30.58**          25.53**
                                      (2.11)              (4.05)             (3.00)          (5.93)           (2.77)
     ** Significant at 1% level




21

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:9/17/2012
language:Unknown
pages:21