Learning Center
Plans & pricing Sign in
Sign Out

Research Design


									Research Design

       October 22, 2007

   Review Mid-Term
   Extra credit presentations
   External & Internal Validity
   Research Designs
   Top 9 Pitfalls to avoid
   Speaker: Policy Analysis Example
How to select a research design:

   What is study purpose?
    –   To describe vs. to predict/evaluate
   How many data collection points are
    –   Cross-sectional vs. longitudinal
   How many groups are needed?
    –   One group
    –   Two groups who receive different tx
    –   More than two groups
Validity in Research Design

   The extent to which a study allows causal
    inferences to be made about relationships
    between variables.

   To infer causality requires:
    –   statistical association between independent and
        dependent variables;
    –   temporality;
    –   relationship must not be spurious.
Two types of validity in research

   External validity- generalizability;
    –   Will we find same results with different people in a
        different place at a different time?

   Internal validity
    –   How sure are we that the results were caused by
        the independent variable and not something else?

If generalizability or causality are not relevant to your research
question, validity may not be the focus of your research design
Threats to internal validity

   History- other events in time
   Maturation
   Testing
   Instrumentation changes
   Regression to the mean
   Selection biases/ between group differences
   Ambiguity in direction of causal inferences
Campbell & Stanley Experiment

   X
   O
   R
The simplest research design: 01

Examples: Qualitative study
          Descriptive study
          Exploratory study

NOT the right design if purpose is to:
     Evaluate intervention’s effectiveness
     Compare one group to another
Pre-experimental designs
X       O                           O1 X O 2

One-shot case study                 One-group pretest-posttest design

Example studies:                    Example studies:
   What was delinquency rates of       Did attending afterschool
   kids who attended afterschool       program decrease youth truancy
   program?                            rate?
   What is organ donation rate         Did attitudes toward organ
   among people who received an        donation improve after reading
   info brochure?                      brochure?

No correlation between variables.   Not able to rule out spurious
Not able to rule out spurious          causes.
Quasi-experimental design

 X    O
Post-test only with non-equivalent groups

  Did DQ rates differ between youth attending after
  school program and non-attenders?
  Do people who read organ donation brochure report
  higher donation willingness than those that did not?
Not able to rule out spurious causes.
Two groups may be different in unmeasured ways.
True experimental design

  R X O
  R      O
  Post-test only control group design

  Because of random assignment, this is experimental
Other Experimental Designs

R O1 X O2
                           R O 1 X O2
R O1 O2
                           R O 1 O2
Pretest posttest control   R    X O2
  group design             R       O2

Does not control for       Solomon four-group design
  testing/ retesting
Testing alternative treatments
R O1 XA O2
                                          R   O1 XAB   O2
R O1 XB O2                                R   O1 XA    O2
R O1    O2                                R   O1 XB    O2
                                          R   O1       O2
   Which afterschool program is most
   effective at decreasing youth          Example:
   delinquency: Boys & Girls Club, the       Did people who received a
   YMCA, or not attending any                brochure and watch a video
   afterschool program?                      show greater intention to
   Compared to no intervention, is           donate organs than people who
   distributing a brochure or showing a      saw only the video, received
   video more effective at promoting         only the brochure, or had no
   organ donation decisions?                 intervention?

Alternative treatment                     Dismantling studies
   design with pretest

    Random selection vs. random assignment

        Sampling                  Research design
To increase likelihood of comparability:
       When random assignment not
             Comparison group
Quasi-experimental designs

   Use comparison group (not randomly
    selected) rather than control group (randomly
   These are COMMON in social science
    research because some things are hard to
Techniques to improve validity of
quasi-experimental designs

   Use multiple pre-test measures
       O1 O2 X O3
       O1 O2        O3

   Use switching replications
       O1 X O2 O3
        O1 O2 X O3
   Select a comparison group as similar as
    possible to experimental group
Time-series Designs

  Simple
O 1 O 2 O 3 O 4 X O5 O 6 O 7
Example: How did introduction of sentencing
  juveniles as adults impact offending rates in MD?

   Multiple
        O1 O2 O3 O 4 X O5 O6 O7
        O1 O2 O3 O 4 O5 O6 O7
    Example: Compared to a state that has not initiated a policy about
      juvenile sentencing, how did introduction of sentencing juveniles
      as adults impact offending rates in MD?
Thinking about your project…

   Person on Left: share your research question
    and proposed research design.
   Person on Right: Say, “That sounds great,
    but have you thought about [alternative
    research design]?”
   Switch roles.
Top 9 Pitfalls that design logic can’t fix

   Experimenter Expectancies
   Obtrusive observation
   Novelty effects
   Placebo effects
   Control group contamination
   Diffusion of Treatment
   Attrition
   Resistance to case assignment protocol
   Measurement bias
Small Group Activity: Using Empirical
Article Samples

   What research design was used in study?
   Write Campbell & Stanley notation for it.
   If the study was not a true experimental
    design, discuss whether it would be possible
    to design an experiment to answer the
    research question.
Next Week: Qualitative Research I

Required Reading:
 MS&D Ch 9
   – Burstein, Bryan & Chao, 2005: Youth with special
   – Mohatt et al., 2004: Alcoholism among Alaska
   – Morgan et al., 2004: Youth hustling
Guest Speaker

To top