Chapter 6: The Internal Validity of Research p.157
• Confounds
• Threats to Internal Validity
• Reactivity
• Demand Characteristics
• Experimental Expectancies
• Summary
1
Chapter 6: The Internal Validity of Research
• Extraneous Variables (so what? Who cares?)
– They contaminate the experiment by providing
• Alternative explanations jeopardizing internal validity
– i.e. does the experiment test what is says it tests?
– Esp. for “training” type studies
• Rule them out
– Logically (e.g. cohort age differences in early development)
– Control for them
• random assignment
• direct control (e.g. limit them, “females only”)
• Measure and estimate their effects on DV
• Alternative explanations (2)
– Confounds: Two vars cannot be separated; e.g. black females; white
males
– Artifacts: Something other than the IV is causing changes to DV
2
Internal Validity:
Confounds
• Confound:
– “when two vars are combined so that the effect of one cannot be
separated from the effects of the other” p.156
– ? What are some possible confounding vars in your study?
• How to avoid
– Untangle them, use each as IV (factorial design)
– Eliminate the confound
• Selection of Ps
• Random assignment (how does that do it?)
• Blocking and measuring
• Measure the confounding variable
3
Internal Validity:
Confounds
• Natural Confounds (Vars that naturally occur together)
– Age & maturity; ethnicity & wealth; gender & toy choices
– What are some more?
• Treatment Confounds
– IV is connected with another var or treatment
• E.g. female Es conduct exp. Groups; males, control groups
• Critical question:
– Do Ps experience exactly the same physical, social, temporal
environment except for IV? P. 159
• Measurement Confounds
– DV measures more than one Hypothetical construct
• E.g. depression; anxiety
4
Internal Validity: Threats (8)
Campbell, D. T., & Stanley, J. ‘63
• Time-Related (5)
1. History
2. Maturation
3. Testing
4. Instrumentation change
5. Statistical Regression
– Selection (3)
1. Nonrandom assignment
2. Preexisting groups
3. Mortality
5
Internal Threats:
Time-Related (5)
History:
events outside of research setting (e.g. abortion attitudes)
Maturation:
natural changes over time (e.g. age, experience)
Testing:
Pretesting effects
Instrumentation change:
e.g. with experience, E’s assignment to behavioral category
Statistical Regression
Pre-selection of Ps based on extremes (e.g. anxiety)
(see Huck and Sandler, ’79) Air Force cadets
6
Internal Validity Threats:
Control Groups in Pre-Post
• Control group equivalence
– Random assignment (not a guarantee, though)
– Test by pretesting (not needed most of the time)
• Important to make sure the pretest is the DV or some measure
closely related
• Possible to pretest on several measures related to the DV
– Can use ANCOVA to remove these effects from DV
– Advantages of not using a pretest
• Avoid testing effects
• Less costly
7
Internal Validity Threats:
Selection Threats
– Selection bias: when exp group differs from control group
• Non-random assignment
– Avoid self-selection (e.g. volunteering)
– Avoid data collection bias (collecting one condition first)
• How would that cause bias?
• Preexisting groups
– Ps already self-selected into groups
• E.g. work setting: employees more eager for training in trn group
• What common characteristics do those in preexisting groups have?
• Other examples?
• Mortality
– Survivors = drop outs?
8
Internal Validity Threats:
Reactivity (ouch!)
– Whenever measuring affects DV score
• Sources
– Evaluation apprehension (judgeaphobia)
• Behavior and self-report (faking)
• May inhibit or disinhibit behavior (socially desirable responding)
• Distraction:
– May divert attention from experiment instructions to others
• It’s what’s the Participant thinks, not the researcher that’s important
– Novelty effects
• Participant tries to anticipate what behavior norms should be
– E.g. Milgram, S. Asch
9
Internal Validity Threats:
Controlling Reactivity
• General control measures:
– Hide your identity
• don’t call yourself a psychologist!
– Be informal, friendly, put them at ease
• as much as possible
– Distract them
– Entertain them
– Trick them with deception
• Mislead, lie, whatever it takes to diver them
10
Control Reactivity:
• Control with Behavioral Measures
– Surreptitious, unobtrusive observation (i.e. candid camera)
– Embed IV in environment (Piliavin et al., ’69)
– “Waiting room ploy” ?? (Aronson et al., ’90)
– Use natural” observers (teachers, parents)
• Control with Self-Report Measures
– ?? Student evals? Social influence?
– Anonymity, confidentiality
– Bogus pipeline (Jones & Segall)
• Noting Instances of Reactivity
11
Reactivity: Demand Characteristics (Martin Orne, ’62)
“Purposely behaving in ways that affect the outcome of research” p.171
Sources:
Participant Roles
Good participant
Bad participant
Apathetic participant
Impact of P roles
Controlling Demand
Cue reduction
Motivation
Role-play control groups
Separate DV measure from study
12
Threats:
Experimenter Expectancies
• Types of Experimenter Effects
– Biased Observation
– Influencing P responses
• Techniques of Control
– Rehearsal and Monitoring
– Minimizing E’s Role
– Condition Blindness
– Avoidance of Data Snooping
13
Internal Validity:
Summary
• Confounds
– What are they?
– How do you control them?
• Threats to Internal Validity
– Time related; control groups; selection threats
• Reactivity
– Evaluation apprehension, Novelty
– Controlling reactivity
• Demand Characteristics
– What is it? How do you control/eliminate them?
• Experimental Expectancies
– Types: biased observations; influencing Ps responses
14