Quasi-Experimental Designs by cs49KdsJ


									Quasi-Experimental Designs

   Manipulated Treatment Variable
        Groups Not Equated
 Pretest-Posttest Nonequivalent
         Groups Design
                 N O X O
                 N O            O
• Cannot assume that the populations are
  equivalent prior to treatment.
• Selection and Selection Interactions are
  threats to internal validity.
• Can try to select subjects or intact groups
  in ways that make it likely that the groups
  are similar, but what about unknown
  variables on which the groups may differ.
  Double-Pretest Nonequivalent
         Groups Design
              N   O   O   X    O
              N   O   O        O
• Some control for Selection x Maturation.
• If groups are maturing at different rates,
  that may be shown in the two pretests.
Regression-Discontinuity Design
               C   O   X   O
               C   O       O
• „C‟ indicates subjects are assigned to
  groups based on score on covariate.
• Groups are deliberately nonequivalent.
• I shall illustrate with a hypothetical
Evaluating an Online Tutorial
• IV = Student completes online tutorial or
• DV = Student‟s score on statistics course
• Pretest/Covariate = Student‟s score on a
  test of statistics aptitude.
• How do I assign students to groups?
    Evaluating an Online Tutorial
    Let the students self-select into groups.
•   This gives me a pretest-posttest
    nonequivalent groups design.
•   I use pretest scores as covariate in
•   This does not, however, remove all
    possible confounds.
•   How might the groups have differed other
    than on statistics aptitude???
Evaluating an Online Tutorial
  Randomly assign students to groups.
• This would be an experimentally sound,
  randomized pretest-posttest control group
• And you would live to regret trying it.
  – Students complain.
  – Their parents complain.
  – The Chair of the Department intervenes.
  – The IRB revokes your authority to do
Evaluating an Online Tutorial
  Try a switching-replications design.
• Those who have to wait until the second
  half of the class would be disadvantaged
  – if you don‟t learn the beginning material well,
    the later material will very hard to learn.
• Those who have it taken away at mid-
  semester will complain.
Evaluating an Online Tutorial
  Apply the treatment only to those most
  in need of it, those lowest in statistics
• Those not selected may complain that
  they could benefit from it too.
  – Tough, there is an American tradition of
    favoring the underdog.
Evaluating an Online Tutorial
• Those selected may complain about
  having to do extra work.
  – You can‟t please everybody every time.
    Convince them they need to do it.
• May be cases when you want to give the
  tutorial only to those with highest aptitude
  – purpose of tutorial is to allow brightest
    students to finish course early, allowing prof
    more time to spend in class with others.
Evaluating an Online Tutorial
• Suppose I pick a cutoff on the covariate so
  that ½ get the treatment, ½ don‟t.
• Simulated data are in file RegD0.txt .
• C = control group, T = Treatment group.
• 2nd score is posttest score.
• 3rd score is pretest score.
• I defined the treatment effect to be zero in
  the population.
   Evaluating an Online Tutorial
• In the population,
   Post = 7 + 1.35Pre + error,  = .9,  error  1.71

• In the sample, ignoring group,
  Post = 7.58 + 1.27Pre + error, r = .85, and
  MSE = 2.13
• Look at this plot of the data:
Evaluating an Online Tutorial
• Now I compute two separate regressions,
  one for each group.
• T: Post = 8.09 + 1.17Pre + error, r = .62,
  and MSE = 2.13.
• C: Post = 6.33 + 1.43Pre + error, r = .72,
  and MSE = 2.29.
• The plot shows how little the two lines differ:
Evaluating an Online Tutorial
• I re-simulated the data, with a 3 point
  treatment effect built in.
• The data are at RegD1.txt.
• T: Post = 11.27 + 1.07Pre + error, r =
  .82, and MSE = 1.35.
• C: Post = 7.90 + 1.18Pre + error, r = .82,
  and MSE = 1.25
• The plot shows a clear regression
Evaluating an Online Tutorial
• The dotted line shows the expected
  regression for the treatment group if there
  were no treatment effect.
• Hard to imagine how any threat to internal
  validity would create the observed
  regression discontinuity.
• Caution: This analysis assumes the
  regression is linear, not curvilinear.
        Proxy-Pretest Design
                 N O1 X O2
                 N O1          O2
•   You have a nonequivalent groups posttest
    only control group design.
•   The treatment has already been
•   Now you decide you want a pretest too.
•   Can‟t warp time, can find an archival proxy
PSYC 2210 and Understanding Stats

• Mid-semester, I ask myself “does taking
  2210 improve students understanding of
• I‟ll compare students in current 2210 class
  with those in another class (excluding any
  who have already taken 2210).
• I want a pretest too, but the treatment is
  already in progress.
PSYC 2210 and Understanding Stats

• I use, as a proxy pretest, students‟ final
  averages from PSYC 2101.
• Conduct an ANCOV
  – IV = took 2210 or not
  – DV = end of course stats achievement test
  – COV = the proxy pretest
    Separate Pre-Post Samples
• Pretest subjects different than   N   O
  posttest subjects.                N       X   O
• I want to evaluate online         N   O
  tutorial in stats.                N           O
• Both I and my friend Linda
  taught stats this semester and
  last semester.
• Both of us gave our students a
  end of course standardized
    Separate Pre-Post Samples
• Row 1: My students last        N   O
  semester, no tutorial.         N       X   O
• Row 2: My students this        N   O
  semester, with tutorial.       N           O
• Row 3: Linda;s students last
  semester, no tutorial
• Row 4: Linda‟s students last
  semester, no tutorial.
• Selection problems likely.
Nonequivalent Groups Switching
     Replications Design
           N   O   X   O       O
           N   O       O   X   O
• I am teaching two sections of stats.
• I make the experimental tutorial available
  the first half semester to one class
• and the second half semester to the other.
• Might reduce complaints, until students
  from the two classes meet each other.
       Nonequivalent Dependent
          Variables Design
                      O1    X   O1
                 N    O2        O2

•   Only one group of subject, but two DVs.
•   One DV you expect to be affected by X.
•   The other you expect not to be affected by X.
•   The second DV serves as a control variable.
•   Should be similar enough to 1st DV that it will be
    effected in same way by history, maturation, etc.
     Nonequivalent Dependent
        Variables Design
• I want to evaluate effect of stats remedial
  tutorial given to all PSYC 2210 students.
• DV1 = stats given at start and end of
• DV2 = Vocabulary test given at start and
  end of semester.
• More impressive if have multiple control
  variables and an a priori prediction of
  extent to which each will be effected.
       Nonequivalent Dependent
          Variables Design
•   Stats Knowledge (DV1) – most affected
•   Logical Thinking – next most
•   Verbal Reasoning – same as LT
•   Arithmetic Skills – next most
•   Vocabulary – next most
•   Artistic Expression – least affected by
    Regression Point Displacement
N(n   = 1)   O   X   O
N            O       O
•   Only one subject in the treatment group
•   Several or many in the control group.
•   X = novel economic development plan.
•   Treatment unit = your hometown, in which
    the plan was just initiated.
 Regression Point Displacement
• You consult state economic database.
• Pick 20 cities comparable to your city,
  these serve as the control group.
• Pretest = value of criterion variable (such
  as unemployment rate) last year.
• Posttest = value of same variable two
  years later.
 Regression Point Displacement
• Plot Post x Pre for the Control Group.
• Draw in regression for predicting Post from
• Plot experimental unit data point.
• If it is displaced well away from regression
  line, you have evidence of a treatment

To top