Quasi-experimental Design by u8ONa2

VIEWS: 143 PAGES: 19

									Quasi-experimental
      Design


   CRJS 4466EA
           Introduction
Quasi-experiment
 Describes non-randomly assigned
  participants and controls subject to impact
  assessment
 Most common design involves constructed
  comparisons
     Matching participants and comparisons
     Statistical adjustment
  Rationale for quasi-
  experimental usage
Random assignment not within the
evaluator’s capability
Powerful stakeholders oppose
randomization
              Issues
Quasi-experiment often results from a
randomized experiment
 Program staff subvert the randomization
  process (assigning only those random
  subjects that will yield good results)
 Attrition from treatment

 Problems with data collection
Measuring impacts in
 quasi-experiments
Net effect = gross outcome for an
intervention group – gross outcome for
a constructed control group + or –
uncontrolled difference between
intervention and control groups + or
– design effects and stochastic error
“when there is a possibility that one or
more relevant differences exists
between the members of the
intervention and comparison groups, as
there typically is in quasi-experiments,
then it is also a possibility that these
differences – not the intervention –
cause all or part of the observed effects”
(Rossi, Freeman and Lipsey, 1999)
“In evaluations in which selection bias is at
work, net effects would tend to be over-
estimated because a portion of the difference
between the intervention group and its
comparison would result from the stronger
potential for positive (or, sometimes,
negative) effects inherent in the persons
selected for intervention” (Rossi, Freeman
and Lipsey, 1999)
       Ex ante quasi-
        experiments
Occurs before intervention to plan
selection of the comparisons
Generally maximizes potential for
equivalence
 Considers motivations
 Identifies characteristics
 Locates potential comparisons
 Allows review of prior, like evaluations
       Ex post quasi-
        experiments
Decision to undertake evaluation occurs
after program is underway
 Targets are enrolled
 Insufficient time to enroll a fresh group and
  follow them to termination
 Issues can be managed to some extent
  through statistical controls
  Constructing control
  groups by matching
Participants are sought first and comparisons are matched
afterwards
Matching is based on prior knowledge and theoretical
understanding of the social processes in question
Matching information is often sought in the published literature
Attend to variables that are potentially related to self-selection
processes
Use only as many variables for matching as are necessary
  Pertinent characteristics will tend to be intercorrelated and,
    therefore, somewhat redundant
  Characteristics useful in
devising constructed control
 groups – Exhibit 9A of text
Characteristics of individuals
  Age, sex, educational attainment, socio-economic status, ethnicity,
   etc.
Characteristics of families (households)
  Life-cycle stage, number of members, number of children, etc.

Characteristics of organized units (schools, classes, unions, etc.)
  Size differentiation, levels of authority, growth rate, budget, etc.

Characteristics of communities (territorially organized units)
  Population size, territorial size, industry mix, governmental
   organization, etc.
      Note: not a substitute for priori knowledge directly relevant to
        the phenomena being studied
 Matching procedures
Options are either individual or aggregate matching
Individual matching – draws a “partner” for each
participant from the unexposed pool
Aggregate matching – overall distributions in the
participant and control groups on each matching
variable are made to correspond
Individual matching is usually preferable (the more
characteristics especially) but is more expensive,
time consuming and difficult to execute for a large
number of matched variables (Rossi, Freeman and
Lipsey, 1999)
 Equating groups by
statistical procedures
Statistical procedures, rather than matching, are now generally
used in both ex ante and ex post quasi-experimental evaluations
as the primary approach to dealing with selection bias and other
unwanted differences between groups (Rossi, Freeman and
Lipsey, 1999)
Multivariate statistical methods are commonly used to adjust for
a number of contaminating variables simultaneously
Matched and statistical controls are equivalent ways of
proceeding, with statistical controls possessing some superior
qualities arising from the retention of observations that might
have to be discarded under matching procedures (Rossi,
Freeman and Lipsey, 1999)
Multivariate statistical
        models
Allows for creation of a statistical model to account
for initial measurement differences between the
intervention and comparison groups
The model adjusts the outcome difference between
those groups to subtract the portion attributable
entirely to those initial differences
Whatever difference on outcomes remains after this
subtraction, if any, is interpreted as the net effect of
the intervention (Rossi, Freeman and Lipsey, 1999)
Need also to control for variables dealing with
selection of individuals into the intervention vs. the
control group
Examples of these control variables include:
proximity of individuals to the program site,
motivation to enroll in the program, whether they had
the characteristics program personnel used to select
participants, etc.
Of course, the variables related to selection are only
useful if they also relate to outcome
(hamburger/hotdog example) (Rossi, Freeman and
Lipsey, 1999)
Regression-discontinuity
        designs
 Evaluator is given selection variables up-front
 Limited applications, but most rigourous
 method
 Also called cutting points designs
 “regression-discontinuity designs
 approximate randomized experiments to the
 extent that the known selection process is
 modeled properly, which is generally
 straightforward because it is explicit and
 quantitative” (Rossi, Freeman and Lipsey,
 1999)
     Generic controls
Examples: age, sex, income, occupation and
race; distributions of certain characteristics
and processes (e.g. birth rates, sex ratios,
proportions of persons in various labour force
categories; and, derivatives of these
measures
Best examples of appropriate use of generic
controls are from epidemiological studies
E.g. detection of epidemics rests heavily on
the epidemiologist’s knowledge of ordinary
incidence rates for various diseases
Generic controls used successfully by
epidemiologists because selection processes
are either known or unimportant (i.e., use of
morbidity rates to detect epidemics)
Issue of insufficient norms for using generic
controls (i.e. achievement tests for inner-city
children)
Generic controls are tempting because of low
cost and limited time to collect data
               A final note
Numerous comparisons between randomized designs and
quasi-experimental designs relative to net effects measuring the
same thing
“Lipsey and Wilson (1993) compared the mean effect size
estimates reported for randomized versus non-randomized
designs in 74 meta-analyses of psychological, educational, and
behavioural interventions. In many of the sets of studies
included in a meta-analysis, the effect estimates from
nonrandomized comparisons were very similar to those from
randomized ones. However, in an equal number of cases there
were substantial differences in both directions” (Rossi, Freeman
and Lipsey, 1999)

								
To top