Docstoc

Quasi-experimental designs

Document Sample
Quasi-experimental designs Powered By Docstoc
					 Psychology 242                                                               1
        Quasi-experimental designs
                               Quasi-experiments
  Introduction
  to Research




        Experimental designs for “studies in nature”.


Studying naturally occurring events
        Measurement studies
        Retrospective designs

Evaluate existing groups or program
      Single shot survey or measure
      Non-equivalent groups
      Time series designs
Psychology 242, Dr. McKirnan                 Research Ethics           Home
                                                               Back   page
                                                                              Next
     Psychology 242                                                                                     2
      Introduction
      to Research                  True v. quasi-experimental designs



True experiments:                                                 Quasi-experiments:
Emphasize internal validity                                       Emphasize external validity
 Assess cause & effect (in                                        Describe “real” / naturally
    relatively artificial environment)                              occurring events
   Test clear, a priori hypotheses                                Clear to exploratory hypotheses


Participants assigned to         Existing or non-equivalent
  experimental v. control groups   groups
 Random or matching              Non-random assignment

 Participants & experimenter     Participants not blind

  Blind to assignment             Control group not possible?


Control study procedures          Control often not possible
 Create / manipulate independent  May not be able to manipulate
  variable                          the independent variable
 Control procedures & measures  Partial control of procedures &
                                    measures
    Psychology 242, Dr. McKirnan             Week 12-13, quasi-experimental designs.             Home
                                                                                         Back   page
                                                                                                        Next
  Psychology 242                                                                               3
   Introduction                 Quasi-experiments: naturally occurring events
   to Research




Studying naturally occurring events
        
         Measurement studies
        
         Retrospective designs

Evaluate existing groups or program
       Single shot survey or measure
       Non-equivalent groups
       Time series designs




 Psychology 242, Dr. McKirnan                      Research Ethics                      Home
                                                                                Back   page
                                                                                               Next
  Psychology 242                                                                                                4
                                Measurement studies
   Introduction
   to Research




1. Naturally occurring events; examples

Event                                                                    Study question
(“Predictor”)                                                            (“Outcome”)
Natural disaster / stressor
    3-mile island                                                             Stress -> immune system
    S.F. earthquake                                                           Stress & coping
Crime / trauma
    Iraq service,
    9 / 11 / 01                                                               PTSD & treatment
Historical event
    9/11 & air travel ban                                                     Contrails & climate change
    Economic collapse                                                         Voting patterns
Publicity / cultural event
    Info. re: Hormone replacement                                             Health behavior

 Psychology 242, Dr. McKirnan                Week 12-13, quasi-experimental designs.                     Home
                                                                                                 Back   page
                                                                                                                Next
  Psychology 242                                                                                    5
   Introduction
   to Research                  Naturally occurring events, 2


 Independent variable:
                  Predictor variable (e.g., natural disaster) often assessed after
                   the event (post-hoc).
                  Researcher has little control over dose / type of predictor
 Participant selection
                  No control over who is exposure to event
                  Some control over selection of sample (e.g., via targeted
                   sampling)
 Many potential confounding variables
 Outcome (dependent) variables:
                  No control with archival data
                  Some control with surveys
                  Use retrospective (measured) variables to clarify
                   interpretation of outcomes or test hypothesis.

 Psychology 242, Dr. McKirnan              Week 12-13, quasi-experimental designs.           Home
                                                                                     Back   page
                                                                                                    Next
  Psychology 242                                                                                       6
   Introduction
   to Research            Naturally occurring events: Retrospective designs

Using retrospective (measured) variables to
 clarify interpretation of outcomes or test
 hypothesis.

Retrospective                      Event                                         Outcome
variable(s)                        (“Predictor variable”)                        variable
Social support                     earthquake [v. control city?]                 stress & coping
Psych. history                     crime / trauma                                mental health
[archive? Self-report?]            [v. control people?]

Personal attitudes                 historical event                              voting patterns
Demographics                       cultural event                                health behavior




 Psychology 242, Dr. McKirnan          Week 12-13, quasi-experimental designs.                  Home
                                                                                        Back   page
                                                                                                       Next
 Psychology 242                                                                    7
  Introduction                 Quasi-experiments: Existing groups
  to Research




Studying naturally occurring events
        Measurement studies
        Retrospective designs



Evaluate existing groups or
  program
     
      Single shot survey or measure
      Non-equivalent groups
      Time series designs

Psychology 242, Dr. McKirnan                      Research Ethics           Home
                                                                    Back   page
                                                                                   Next
  Psychology 242
   Introduction                 Existing groups                                                        8
   to Research




Existing groups:
 Single self-selected group; no comparison
  possible
        users of psychotherapy (or any product)
        members of group or cult [contrast with demographically
             matched controls?]
 Two or more groups, with self-selection and / or
  "non-blind" assignment
        Psychological interventions: therapy v. wait list, etc.
 Two or more groups, no random assignment
        Comparing schools / cities / existing groups…

 Psychology 242, Dr. McKirnan                 Week 12-13, quasi-experimental designs.           Home
                                                                                        Back   page
                                                                                                       Next
                                                                                                              9
                           “One shot” case studies
   Psychology 242
    Introduction
    to Research




   Group                  Naturally occurring event or social change                         Observe1

May control selection of                Typically no control                      Dependent Variable(s):
 study group, or must use               over event. Not a                         May or may not have
 Convenience sample.                    true Independent                           control over measures
 Other data may be                      Variable                                   (e.g., surveys v.
 available about group.                                                            archival measures).
 Typical use: Surveys or measures after an event.
 Heuristic value: generating hypotheses for later study
     or confirm controlled data in “real world” setting.
 Internal / External validity:
        No control over selection of people into the event.
        Potentially no control over selection into measurement
             group.
        No control group; uncontrollable event, or other groups
             may not “need” the intervention (e.g., therapy)
  Psychology 242, Dr. McKirnan          Week 12-13, quasi-experimental designs.                        Home
                                                                                               Back   page
                                                                                                              Next
     Psychology 242                                                                              10
      Introduction
      to Research
                                   Example of one-shot case study



Example: Consumer Reports psychotherapy survey
[Click for paper]


Research questions:
   Does psychotherapy “work” from consumer view?
   Who gets therapy / what does it consist of?
   Do consumer responses vary by type of therapy?
Research approach:
   One shot case study / survey
Sampling frame:
   Any therapy or psychological service user
   No real information re: population of therapy users.
Sampling procedure:
   4,100 Consumer reports readers responding to “in
    magazine” mail-back survey form
    Psychology 242, Dr. McKirnan        Week 12-13, quasi-experimental designs.           Home
                                                                                  Back   page
                                                                                                 Next
     Psychology 242                                                                                   11
      Introduction
      to Research                  One shot: Consumer reports survey, 2


Experimental Controls
      Evaluate by gender, type of treatment, medications, to
       provide more differentiated analysis
Negatives:
 Selection bias
             no control over who got therapy (self-selection)
             of those who got therapy, no control over who
    returned a survey (secondary self-selection)
 Cursory outcome measures: satisfaction rather than
  mental health
Positives:
 Huge, national sample
 Wholly anonymous, 3rd party data collection; less bias
 “Real world” assessment of product quality
    Psychology 242, Dr. McKirnan             Week 12-13, quasi-experimental designs.           Home
                                                                                       Back   page
                                                                                                      Next
  Psychology 242                                                                                   12
   Introduction
   to Research                  One shot: Consumer reports survey, 3

Key distinction in psychological interventions:
 “efficacy” v. “effectiveness” research
      Efficacy; “true” experimental design / Lab-basis
          Rigorous controls; High internal validity
          Test basic theory or highly specific technique
 Do the specific ingredients (or theory…) of this
   treatment validly induce the key outcome?

      Effectiveness; quasi-experimental; “natural” or
       applied setting
          Less or no control; naturally occurring treatment
          High external validity
 Does treatment “work” in real patients w/real therapists?
 Psychology 242, Dr. McKirnan             Week 12-13, quasi-experimental designs.           Home
                                                                                    Back   page
                                                                                                   Next
   Psychology 242                                                                                               13
    Introduction
    to Research       Comparisons of study types:                         (Consumer reports 4)


  Efficacy experiment v.                                    Effectiveness research
• One specific diagnosis                                   • Multiple diagnoses & severities
• Rigorous control group                                   • No control group, 2nd controls
   • “no treatment” condition                                 • Archival, via pt. characteristics
   • “attention control”
• Random assignment                                        • Self-selection; “shopping”
• Manualized / uniform treatments                          • Multiple / mixed treatments
   • High Fidelity to treatment method                        • Highly tailored to patient
   • Fixed number of sessions.                                • # sessions is patient based.
• Well operationalized outcomes, e.g.,                     • Diverse, self-referenced outcomes
   • clinician-diagnosed disorder                             • Subjective sense of “wellness”
   • Standard / validated self-report                         • Lessening of “problem” behaviors
     symptom scales                                             or moods
                                                              • Personal assessment of functioning
• “Blind" raters or diagnosticians                         • Self-rated: cannot be "blind"
 ("single-blind“: patient & therapist know
 what the treatment is..)
                                                           • Diverse times since treatment
• Patients followed for a fixed period
                                                           • Retrospective rather than
                                                             prospective
  Psychology 242, Dr. McKirnan       Week 12-13, quasi-experimental designs.                             Home
                                                                                                 Back   page
                                                                                                                Next
  Psychology 242                                                                                   14
   Introduction
   to Research                  Consumer reports survey, 5.


Survey findings on therapy effectiveness:
 People who got more treatment (> 6 months) did better.

 For patients‟ presenting                                     For general ψ health MH
 problem(s) all specialists                                       specialists did best,
 did about the same.                                              marriage counselors worst.




 Psychology 242, Dr. McKirnan             Week 12-13, quasi-experimental designs.           Home
                                                                                    Back   page
                                                                                                   Next
 Psychology 242                                                                                   15
  Introduction
  to Research                  Consumer reports survey, 6.

Other effectiveness / descriptive findings:
  3/4            went to mental-health specialist.
  Patients who rated themselves worse at
   outset made the most progress.
  AA very highly evaluated
  Therapy did as well as medications
  40% got drugs;
                 MDs gave medications to 83% of patients
                 MH pros; 20% drug treatments
                 50% who got drugs got no counseling
                 20% got no information about side effects
                 40% of anti-anxiety drugs given > 1 year
Psychology 242, Dr. McKirnan             Week 12-13, quasi-experimental designs.           Home
                                                                                   Back   page
                                                                                                  Next
  Psychology 242                                                                                            16
                          More complex case studies
   Introduction
   to Research




  Group                         Naturally occurring event                             Observation +
                                                                                     Archival Controls
 Basic selection /                       Dependent Variable(s):
convenience biases,                      Combine survey or other measures with archival or
uncontrollable event.                     ancillary data as Control variables.

Example 1: San Francisco earthquake & coping

Sampling frame: - Randomly selected survey participants

Outcomes:                             - Standardized mental health scales
                                      - Self-reports of stress

Quasi-controls: - population norms on outcomes
                - ancillary measures, e.g., social support

Findings:                             - High rates of stress Rx,
                                      - Social support „buffers‟ stress
 Psychology 242, Dr. McKirnan              Week 12-13, quasi-experimental designs.                   Home
                                                                                             Back   page
                                                                                                            Next
  Psychology 242
   Introduction
                                                                                                              17
                          “One Shot” + archival control: examples
   to Research




Example 2: Stress and immune functioning
Event:                                    - Three-mile Island nuclear accident

Sampling frame: - Randomly selected residents of
                   geographic area around TMI

Outcomes:                                 - Blood draws for immune markers
                                          - Self-reports of stress

Quasi-controls: - Demographically matched sample
                - Archival data on health & illness

Findings:                                 - Long-term suppression of key immune
                                             markers (natural killer cells, T cells)
 Psychology 242, Dr. McKirnan                        Week 12-13, quasi-experimental designs.           Home
                                                                                               Back   page
                                                                                                              Next
  Psychology 242
   Introduction
                                                                                                          18
                                One shot designs with archival controls, 3
   to Research



Example 3: Psych. & Health effects of bereavement

Event:                                 - Loss of spouse

Sampling frame: - Hospital records, self-selected spouses

Outcomes:                              - Blood draws for immune markers
                                       - Standardized mental health scales
                                       - Occupational functioning

Quasi-controls:                        - Population norms on MH scales
                                       - Archival data: occupation & illness

Findings:                              - Long-term immune suppression
                                       - Social support „buffered‟ stress
                                       - impact of bereavement > other stressors

 Psychology 242, Dr. McKirnan                    Week 12-13, quasi-experimental designs.           Home
                                                                                           Back   page
                                                                                                          Next
     Psychology 242                                                                         19
                One shot designs; Summary
      Introduction
      to Research



“One Shot” designs: no control over independent
 variable(s), only partial control over measurement:
           An experiment is not possible
           There cannot be a control group
           “Pre-” measures not possible or practical
     Virtue:
            Assess naturally occurring or uncontrollable socially or
             politically important events
            Provides “real world” look at processes that are typically
             studied in experiments: “Effectiveness” v. Efficacy data
            Archival data can help interpret the findings / “control” some
             alternate interpretations.
     Liability:
            lack of control group creates multiple threats to internal validity
            No pre-measure makes interpretation (e.g., of change…) difficult.
    Psychology 242, Dr. McKirnan   Week 12-13, quasi-experimental designs.           Home
                                                                             Back   page
                                                                                            Next
 Psychology 242                                                                    20
  Introduction                 Quasi-experiments: Existing groups
  to Research




Studying naturally occurring events
        Measurement studies
        Retrospective designs



Evaluate existing groups or
  program
      Single shot survey or measure
     
      Non-equivalent groups
      Time series designs

Psychology 242, Dr. McKirnan                      Research Ethics           Home
                                                                    Back   page
                                                                                   Next
                                                                                                                  21
         One group pre-test — post-test
   Psychology 242
    Introduction
    to Research




   Group                  Observe1                     Intervention or event                      Observe2

Selected or                      Baseline Assessment                                   Outcome Assessment
 convenience                     May or may not have control                           Typically controllable,
 sample.                          over measures (e.g., surveys                          but may be archival.
                                  v. archival measures).
                     Event or intervention May or may not be controllable
                     by researcher, e.g., policy change.

Uses:  Educational & social environments
                 Political or health policy change
                         Not feasible to have a control group
                         System-wide intervention / social change
                          (school, public health campaign..)

  Psychology 242, Dr. McKirnan               Week 12-13, quasi-experimental designs.                       Home
                                                                                                   Back   page
                                                                                                                  Next
  Key design feature: no control group.
  Psychology 242
   Introduction
   to Research
                                                                                             22



  Group                  Observe1              Intervention or event           Observe2

                         Observe1                   Confound                   Observe2


Threats to internal validity (confounds):
                                                  Historical / cultural events occur
 History                                         between baseline & follow-up.

                                                  Individual maturation or growth
 Maturation                                      occurs between baseline & follow-up.
                                                  People respond to being measured or
 Reactive measures                               being a measured a second time.
                                                  Extreme scores at baseline “regress”
 Statistical regression                          to a more moderate level over time.

                                                    People leave the experiment non-
 Mortality /                   drop-out            randomly (i.e., for reasons that may
 Psychology 242, Dr. McKirnan                       affect the results…).
                                      Week 12-13, quasi-experimental designs.
                                                                                 Back
                                                                                      Home
                                                                                         Next
                                                                                      page
  Psychology 242
   Introduction
                                                                                                                23
                                Examples: One group pre- post-, HIV testing
   to Research




Example 1: Effects of HIV testing on sexual risk.

Event:                                        - Receipt of HIV testing & counseling

Sampling frame: - Participants in testing centers

Study structure: - Baseline retrospective interview at
                     testing session
                 - Follow-up interview 3 months later

Quasi-controls:                               - Population characteristics to predict
                                                  between-group differences

Outcomes:                                     - Self-reports of sexual risk

 Psychology 242, Dr. McKirnan                          Week 12-13, quasi-experimental designs.           Home
                                                                                                 Back   page
                                                                                                                Next
    Psychology 242
     Introduction             Example: One group pre- post-, HIV, 2
                                                                                                               24
     to Research



Effects of HIV testing on sexual risk, cont.

  Findings:                          - Significant shifts toward safety
                                     - Few demographic predictors of risk
                                       or risk change

Threats       - Self-selection into testing group
to internal
validity      - Mortality: non-random drop-out(?)
 - History: general shift in norms & behavior during
   study time may account for observed change
 - Instrument change; people may answer more
    conservatively during a follow-up interview

   Psychology 242, Dr. McKirnan                       Week 12-13, quasi-experimental designs.           Home
                                                                                                Back   page
                                                                                                               Next
  Psychology 242
   Introduction                 Examples: One group pre- post- , Education
                                                                                                               25
   to Research




Example 2: Educational reform & “No Child left
 behind” testing requirements.
Intervention:                                - Standardized testing becomes integral
                                               to educational programs & school
                                               evaluation.

Sampling frame & - Longitudinal data across multiple
Study structure: years in target school grades.
                - No control group possible.

Quasi-controls:                              - Population characteristics to predict
                                               between-group differences

Outcomes:                                    - Standardized test scores
 Psychology 242, Dr. McKirnan                         Week 12-13, quasi-experimental designs.           Home
                                                                                                Back   page
                                                                                                               Next
 Psychology 242
  Introduction            Example: One group pre- post-, education, 2
                                                                                                                     26
  to Research                                                              Education reform & test scores.

Findings:
       Modest, statistically significant increase in scores
       Usual demographic predictors of change; more
        affluent, better schools..
Internal validity?:
       Reactive measures; teachers & students do better
        when measured; (they also cheat; see Houston Miracle article)
       Instrumentation: kids get better at taking
        standardized tests, teacher better at teaching them
       History: General cultural shift
                 Education more prominent in city
                 More affluent families sending kids to public schools
Psychology 242, Dr. McKirnan                        Week 12-13, quasi-experimental designs.                   Home
                                                                                                      Back   page
                                                                                                                     Next
              One group pre- post- designs; Summary                                                     27
     Psychology 242
      Introduction
      to Research



One group pre- post- test design useful where:
              An experiment is not possible
              There cannot be a control group
              Researchers have control over measurement and the
               independent variable
   Virtues:
             provide data on naturally occurring socially or politically
              important events
             Pre-measure allows researcher to interpret change & examine
              status of groups at baseline.
                                                                     History
    Liability: lack of control                                      maturation
     group creates multiple                                          statistical regression
     threats to internal validity:                                   reactive measures
                                                                     mortality / drop-out

    Psychology 242, Dr. McKirnan   Week 12-13, quasi-experimental designs.                       Home
                                                                                         Back   page
                                                                                                        Next
                                                                                                            28
            Non-equivalent two-group designs
  Psychology 242
   Introduction
   to Research




 #1; Static Group Design
 Group1                                          Intervention or event                         Observe1
                         (No baseline)
 Group2                                                Contrast group                          Observe1


Groups are not equivalent                                                  Assessments may or may not
  at baseline, due to..                                                      be controlled
 Self-selection                                                            Survey or interviews
 Non-random assignment                                                     Archival / existing data,
 Use of existing groups                                                     e.g., clinic records, grades
 Participants not blind
                                                    Intervention or event may or may not
                                                      be controlled by researcher;
                                                     Existing program
                                                     Experimental intervention
                                                     Naturally occurring event (..9/11..)


 Psychology 242, Dr. McKirnan            Week 12-13, quasi-experimental designs.                     Home
                                                                                             Back   page
                                                                                                            Next
   Psychology 242                                                                                             29
    Introduction
    to Research                  Non-equivalent designs; pre- post-

 #2 Two Group Pre- Post- Design

   Group                  Observe1                   Intervention or event                    Observe2

   Group                  Observe1                         Contrast group                     Observe2


Non-equivalent groups                                 Intervention & Assessments often
 Self-selection                                          controlled by researcher in these
 Non-random assignment                                   designs.
 Use of existing groups
 Participants not blind
                                                                                       Similar to true
Observation1 used to                                                                   experimental
 Assess equivalence of groups at baseline                                           design, except for
 Test for threats to internal validity:                                              non-equivalent
           Reactive measures                                                             groups
           History, mortality effects
           Regression effects
  Psychology 242, Dr. McKirnan             Week 12-13, quasi-experimental designs.                     Home
                                                                                               Back   page
                                                                                                              Next
  Psychology 242
   Introduction                 Examples: Non-equivalent groups, condoms
                                                                                                               30
   to Research



Example Non-equivalent control group design:
Effects of condom distribution on sexual safety
Intervention:                                - Condom education & distribution in
                                                High School health classes

Sampling frame: - Schools in New York & Chicago
                - Schools matched for SES, race, size
Study structure: - NY = intervention schools, Chicago
                    are contrast schools.
                 - Baseline, sexual health programming,
                    end of year Follow-up
Outcomes:                                    - Clinical measures: STDs
                                             - Self-reports: sexual activity & safety
 Psychology 242, Dr. McKirnan                         Week 12-13, quasi-experimental designs.           Home
                                                                                                Back   page
                                                                                                               Next
   Psychology 242
    Introduction
                                                                                                          31
                                 Examples: Non-equivalent groups, condoms, 2
    to Research




Condom distribution, cont.

 Findings:
     NY (intervention) students; lower STD rate, safer sex
     NY and Chicago students; similar levels of sexual
      activity
     Thus; sexual health classes appeared to increase
      safety without increasing sexual activity.
 Internal validity?:
     Reactive measures; Study is not blind; NY students
      know they are the intervention group
     Non-equivalent groups: Possible differences between
      cities = unmeasured confounds

  Psychology 242, Dr. McKirnan                   Week 12-13, quasi-experimental designs.           Home
                                                                                           Back   page
                                                                                                          Next
 Psychology 242                                                                                       32
                               Non-equivalent designs
                                                                         Soloman 4-group design
  Introduction
  to Research




Group 1                 Observe1                           Intervention               Observe2

Group 2                 Observe1                         Contrast group               Observe2

Group 3                                                    Intervention               Observe2

Group 4                                                  Contrast group               Observe2


  Groups 1 & 2:
  Observation1 used to
   Assess equivalence of groups at baseline
   Test threats to internal validity
  Groups 3 & 4:
  Post-test only tests for reactive effects of assessment
   Compare 1+2 versus 3+4
   Test interaction of treatment group x pre- post- versus post- only
Psychology 242, Dr. McKirnan             Week 12-13, quasi-experimental designs.



                                                                                               Home
                                                                                       Back   page
                                                                                                      Next
                                                                                              33
    Non-equivalent 2 group designs: Summary
    Psychology 242
     Introduction
     to Research




    Most common quasi-experimental approach.
    Used where:
            Some form of control or contrast group is possible
            Groups cannot be equivalent:
                      Participants cannot be blind re: group assignment
                      Random assignment not possible
                      Must use existing or self-selected groups.

       Virtue:
           Study natural / “real world” interventions

           Contrast group lessens major threats to internal

            validity
       Liability: non-equivalent groups = possible confound.

Psychology 242, Dr. McKirnan         Week 12-13, quasi-experimental designs.           Home
                                                                               Back   page
                                                                                              Next
 Psychology 242                                                                    34
  Introduction                 Quasi-experiments: Existing groups
  to Research




Studying naturally occurring events
        Measurement studies
        Retrospective designs



Evaluate existing groups or
  program
      Single shot survey or measure
      Non-equivalent groups
     
      Time series designs

Psychology 242, Dr. McKirnan                      Research Ethics           Home
                                                                    Back   page
                                                                                   Next
                                                                                                      35
                          Interrupted time series design
     Psychology 242
      Introduction
      to Research




    Group               Measure1   M2                 M3                     M4   M5      M6 …

                                                       Intervention
                                                         or event


   Test effect of intervention or event on ongoing
    series of measurements.
   Intervention may be experimental or observed
       Policy shift, e.g., educational policy
       Uncontrolled event; e.g., 9/11/01, Media event
 Assessments may be experimental or archival
       Successive cross-sectional surveys
       Traffic data, clinic or crime reports, test scores
    Psychology 242, Dr. McKirnan   Week 12-13, quasi-experimental designs.                     Home
                                                                                       Back   page
                                                                                                      Next
     Psychology 242                                                                                          36
      Introduction
      to Research
                                   Time series designs



    Group               Measure1          M2                 M3                     M4   M5      M6 …

                                                              Intervention
Multiple baseline                                               or event
    Demonstrate highly stable
                                                                       Hypothesis; tested by:


    effect
       long-term crime rates                                             Shift in stable rate after
       disease prevalence                                                 intervention
       economic performance…                                             Increase / decrease in rate
                                                                           of change after intervention
   Show steady rate of change
   Threats to internal validity:
       sensitive to very local history
       Single group possibly prey to confound


   Advantage for internal validity
       Eliminates carryover effects of repeated measurement
       tests maturation, history, reactive measurement, etc

    Psychology 242, Dr. McKirnan          Week 12-13, quasi-experimental designs.                     Home
                                                                                              Back   page
                                                                                                             Next
 Psychology 242                                                                                         37
  Introduction
  to Research          Example of interrupted time series:           Shift in Baboon culture.
Core question: Do baboon troops
 develop and transmit a learned
 “culture”?
Baseline: Long-term observational
 data on aggressiveness in a specific
 baboon troop.
Intervention:
 Tuberculosis outbreak due to infected food.
 Dominant / aggressive males fed first
          are selectively infected
          are naturally culled from troop
 Naturally occurring event in >20yr. ongoing
      field study.
Psychology 242, Dr. McKirnan             Week 12-13, quasi-experimental designs.
                                                                                         Back
                                                                                                 Home
                                                                                                 page
                                                                                                        Next
 Psychology 242                                                                                   38
  Introduction
  to Research                  Baboon culture: findings


Quasi-controls: Parallel data from other baboon
 troops.
                                           Outcome measures:
                                            Standardized indices of
                                            aggression & dominance
                                            behavior
                                           Core finding:
                                            With dominant males gone,
                                             remaining males showed more
                                             cooperative behavior
                                            Enhanced cooperation was
                                             transmitted across generation,
                                             showing learned “culture”.
Psychology 242, Dr. McKirnan             Week 12-13, quasi-experimental designs.           Home
                                                                                   Back   page
                                                                                                  Next
  Psychology 242                                                                                       39
   Introduction
   to Research
                                Example: Interrupted time series data


The “Magic Johnson effect” on HIV testing
Data: Archival records of HIV tests reported to CDC,
    collected monthly
           Data show stable baseline over multiple observations
           Timing of intervention precise relative to data collection
Intervention: Magic reports infection on national TV.
           Uncontrollable, “naturally occurring” event
           Tests hypothesis re: modeling effects in health behavior
Finding: Initial spike in testing rates, followed by
    leveling off at higher base rate.
           Initial increase expected
           Hypothesis tested by longer-term shift in base rate, available
            due to archival time-series data
           Effect found for both genders.
 Psychology 242, Dr. McKirnan                 Week 12-13, quasi-experimental designs.           Home
                                                                                        Back   page
                                                                                                       Next
    Psychology 242
     Introduction
                                                                                                                                               40
                                     Example of time-series data: “Magic” / HIV effect.
     to Research




Time-series data showing shift
 in HIV testing after Magic‟s                                                                                 Magic‟s
 announcement.                                                                                                Announcement



                                                                                                                     Initial spike


                                                                                                                       New, higher
                                                                                                                        base rate

                                                                                                                   Low & variable
                                                                                                                 baserate of testing


                                                                                                                 Multiple (monthly)
                                                                                                                  measures.


 Tesoriero, J.M., Sorin, M.D., Burrows, K.A., LaChance-McCullough, M.L. (1995). Harnessing the heightened public awareness of celebrity HIV
 disclosures: “Magic” and “Cookie” Johnson and HIV testing. AIDS Education and Prevention, 232-250.
   Psychology 242, Dr. McKirnan                        Week 12-13, quasi-experimental designs.                                          Home
                                                                                                                              Back     page
                                                                                                                                               Next
  Psychology 242                                                                                              41
   Introduction
   to Research                  Multiple time series study


Multiple time series data
Group 1              Measure1             M2                  M3                     M4   M5      M6 …

Group 2              Measure1             M2                  M3                     M4   M5      M6 …

                                                                 Intervention
                                                                   or event
Groups typically formed by
    blocking variable measured                                    Hypothesis; tested by
    post-hoc;                                                         interaction of blocking variable
                                                                      by repeated measure:
 Health claims in NYC v. other
  cities post- 9/11/01                                               Is shift in stable rate ( rate of
                                                                      change) greater in one group
 Younger v. older voting                                             than another?
  patterns post- Iraq invasion
 Heterosexual v. gay HIV
  testing rates post- Magic
  Johnson media event.
 Psychology 242, Dr. McKirnan              Week 12-13, quasi-experimental designs.                     Home
                                                                                               Back   page
                                                                                                              Next
     Psychology 242                                                                                     42
      Introduction
      to Research
                                   Blocking variables


Testing blocking variables in the HIV testing
  time-series data.
Core questions:
    Both heterosexuals and Ethnic minorities had low HIV
     testing rates
            May feel HIV is not relevant to them – it is a “white gay” problem.
            They may lack resources or venues for testing.
    Will having a prominent African-American Heterosexual
     disclose HIV+ status may change those perceptions?
Hypotheses:
    Heterosexuals will respond more strongly to the Magic
     Johnson media event than will gay/bisexual men.
     African-American and Latino men and women will
     respond most strongly.
    Psychology 242, Dr. McKirnan               Week 12-13, quasi-experimental designs.           Home
                                                                                         Back   page
                                                                                                        Next
  Psychology 242                                                                                               43
   Introduction
   to Research
                                Blocking variables: sexual orientation, 1.


Testing blocking variables: Gay / IDU data.

                                                                                        High base-line and
                                                                                        high variability in
                                                                                        testing rates among
                                                                                        men with risky
                                                                                        partners, and IDUs.
                                                                                        Gay / bisexual men
                                                                                        show less variable,
                                                                                        but generally lower
                                                                                        baserates.

                                                                                        Risky men & IDUs
                                                                                        slightly increase,
                                                                                        with substantial
                                                                                        variability.
                                                                                        Gay & bisexual men
                                                                                        show no change.



 Psychology 242, Dr. McKirnan                                                                           Home
                                              Week 12-13, quasi-experimental designs.           Back   page
                                                                                                               Next
  Psychology 242                                                                                                      44
   Introduction
   to Research
                                Blocking variables: sexual orientation, 2.


Testing blocking variables: Heterosexuals.

                                                                                       In contrast to gay /
                                                                                         bisexual men or IDUs,
                                                                                         heterosexual show an
                                                                                         initially low baserate.

                                                                                       Followed by a large spike
                                                                                       after the announcement


                                                                                       And a much higher new
                                                                                       baserate.




   The hypothesis that heterosexuals would be more affected by the
    “Magic” announcement was supported by the interaction of
    Time x the blocking variable of sexual orientation.

 Psychology 242, Dr. McKirnan                Week 12-13, quasi-experimental designs.                           Home
                                                                                                       Back   page
                                                                                                                      Next
  Psychology 242                                                                                            45
   Introduction
   to Research                  Blocking variables: ethnic differences

Testing blocking variables: Ethnic differences.

                                                                                     African-Americans
                                                                                     and Hispanics show
                                                                                     low baserates and a
                                                                                     high spike post-
                                                                                     announcement



                                                                                     Both groups go back
                                                                                     toward their
                                                                                     baselines shortly
                                                                                     post-announcement.




 Psychology 242, Dr. McKirnan              Week 12-13, quasi-experimental designs.                   Home
                                                                                             Back   page
                                                                                                            Next
  Psychology 242                                                                                                 46
   Introduction
   to Research
                                Blocking variables: ethnic differences, 2.

Ethnic differences: White participants.


                                                                                        HIV testing among
                                                                                         Whites was similar
                                                                                         to African-Americans
                                                                                         & Hispanics at
                                                                                         baseline,
                                                                                        They showed stable,
                                                                                         much higher testing
                                                                                         rate after Magic‟s HIV
                                                                                         announcement.




 Psychology 242, Dr. McKirnan                Week 12-13, quasi-experimental designs.                      Home
                                                                                                  Back   page
                                                                                                                 Next
  Psychology 242                                                                                   47
   Introduction
   to Research                  Summary: Blocking variables in time series data




A series of measures before & after an event allows us to clearly
  identify patterns of behavior, and to test group differences (via
  blocking variables).

The hypothesis that ethnic groups would differ was supported by
 interaction of Time x the blocking variable of ethnicity (but in a
 direction that was not predicted: Whites showed more change).

 Psychology 242, Dr. McKirnan             Week 12-13, quasi-experimental designs.           Home
                                                                                    Back   page
                                                                                                   Next
    Psychology 242
     Introduction              Time series designs: Summary                                    48
     to Research


     Time series is most common with archival data: existing,
      standard records collected for other purposes.
     Used where:
        The hypothesis concerns changes in long-term trends

        Typically an experiment cannot be run

                      Simple practicality or cost, e.g., health care issues
                      Ethics; crime rates, rates of domestic violence, etc.
                      The target events are not controllable.
       Virtue:
                Study natural / “real world” processes or
                 interventions
                Blocking variables – comparing time trends across
                 groups -- lessens major threats to internal validity
       Liability: lack of control = possible confound.
Psychology 242, Dr. McKirnan          Week 12-13, quasi-experimental designs.           Home
                                                                                Back   page
                                                                                               Next
                                                                                                49
                               Quick quiz
 Psychology 242
  Introduction
  to Research




Researchers often use _____
  to help interpret “single
  shot” surveys
     A = paradigm change
     B = measurement studies
     C = experimental controls
     D = retrospective measures




Psychology 242, Dr. McKirnan           Week 12-13, quasi-experimental designs.           Home
                                                                                 Back   page
                                                                                                Next
  Psychology 242                                                                          50
   Introduction
   to Research
                                Quick quiz, 2



Match:



A = Mortality / drop-out                    Historical / cultural events occur
                                            between baseline & follow-up.
B = Maturation                              Individual maturation or growth
                                            occurs between baseline & follow-up.
C = History                                 People respond to being measured or
                                            being a measured a second time.
D = Statistical
                                            Extreme scores at baseline “regress”
  regression                                to a more moderate level over time.

E = Reactive measures                             People leave the experiment non-
                                                  randomly (i.e., for reasons that may
 Psychology 242, Dr. McKirnan                     affect the results…).
                                    Week 12-13, quasi-experimental designs.        Home
                                                                               Back   Next
                                                                                   page
  Psychology 242                                                                             51
   Introduction
   to Research
                                Quick quiz, 2




 Match:

A = Mortality / drop-out

B = Maturation

C = History                                        People respond to being measured or
                                                   being a measured a second time.

D = Statistical regression

E = Reactive measures
 Psychology 242, Dr. McKirnan       Week 12-13, quasi-experimental designs.           Home
                                                                              Back   page
                                                                                             Next
  Psychology 242                                                                             52
   Introduction
   to Research
                                Quick quiz, 3



Match:



A = Mortality / drop-out
B = Maturation                                     Growth or natural change between
                                                   baseline & follow-up.
C = History
D = Statistical
  regression
E = Reactive measures
 Psychology 242, Dr. McKirnan       Week 12-13, quasi-experimental designs.           Home
                                                                              Back   page
                                                                                             Next
  Psychology 242                                                                             53
   Introduction
   to Research
                                Quick quiz, 4



Match:



A = Mortality / drop-out
B = Maturation
                                               People leave the experiment non-
C = History                                    randomly (i.e., for reasons that may
                                               affect the results…).
D = Statistical
  regression
E = Reactive measures
 Psychology 242, Dr. McKirnan       Week 12-13, quasi-experimental designs.           Home
                                                                              Back   page
                                                                                             Next
  Psychology 242                                                                                           54
   Introduction
   to Research
                                Quick quiz 5



 Group               Measure1           M2                 M3                     M4   M5      M6 …

                                                            Intervention
This is called a:                                             or event
A = Threat to internal validity
B = Manipulation check
C = Multiple baseline
D = ..lot of work.




 Psychology 242, Dr. McKirnan           Week 12-13, quasi-experimental designs.                     Home
                                                                                            Back   page
                                                                                                           Next
           Quasi-experiments; Summary                                                        55
   Psychology 242
    Introduction
    to Research




 1. Study naturally occurring events that could
    not be brought into a lab or a true experiment.
               Measurement studies
               Retrospective designs
2. Evaluate existing groups or program(s)
              Single shot survey or measure of an intervention
                    With or without control variables

              Non-equivalent / pre-existing groups
                    Static group or 2 group pre- post- design
              Time series designs, often with archival data

 Trade off internal for external validity
  Psychology 242, Dr. McKirnan      Week 12-13, quasi-experimental designs.           Home
                                                                              Back   page
                                                                                             Next
 Psychology 242                                                                                   56
  Introduction
  to Research                  Exam issues


    Key exam issues:
           Slide 2: “true” v. quasi –experiments
           Threats to internal validity
           Basic forms of quasi-experiments
                     Single shot
                     Single group pre- post- test
                     “Non-equivalent” two group designs:
                          Self-selection (in or out [mortality])
                          Existing groups
                          Non-blind
                          Non- random assignment
                     Interrupted time-series / group contrasts
           Virtues (external validity) and problems (internal
            validity)

Psychology 242, Dr. McKirnan             Week 12-13, quasi-experimental designs.           Home
                                                                                   Back   page
                                                                                                  Next

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:68
posted:10/17/2011
language:English
pages:56