Validity of Experimental Designs

Document Sample
Validity of Experimental Designs Powered By Docstoc
					Reliability and Validity of Dependent
Validity of Dependent Variables

   Does it measure the concept?
   Construct Validity: Does DV really capture
    what you want to measure (good operational
   Or does it include mood, culture or gender
    bias, confusing wording, observational bias,
Indicators of Construct Validity

   Face Validity: Does it appear to be a good
    measure (do experts think so?)
   Predictive Validity: Predict later behavior-
    GRE=grad school success?
   Concurrent Validity: Are those known to
    diverge different in scores (Self Monitoring)
Indicators of Construct Validity

   Convergent Validity: do other kinds of
    ratings agree? Similar responses to similar
   Divergent validity: is it different from other
    constructs? (measures intell, not SES or
    gender bias) shy isn’t lonliness
   Reactivity- knowing you are being studied
    changes behavior
Reliability of DV

   Are results repeatable?
   All measurement contains true score plus
    error of measurement

   Not an issue of replication- same
    subjects=same scores
Types of Reliability

   Inter-rater reliability- calculate r for
    observers or Cohen’s Kappa
   Internal consistency- split half reliability
    Cronbach’s Alpha calculates ave of all
    possible corr.
   Temporal consistency- test-retest reliability
    with SAME people
   Restaurant example
   Can a variable be reliable and not valid?
   Valid and not reliable?

   How do you know you have a good DV?
    –   Mental Measurements Yearbook
Validity of Experimental Designs
Survey Design
Internal validity

   Does the design test the hypothesis we want
    it to test? Did IV manipulation cause change
    in DV? Can we infer causality?

   What if internal validity is low?
External validity

   Does your study represent a broad

   Caution with Discussion Section if weak
   Random Sampling
    –   Stratified Sampling
    –   Block Randomization
Ecological validity

Does study reflect the real world-
do people really behave this way?

Can you study anything without changing it?
Threats to Internal Validity:

   In pre-post design:
    –   Test participants
    –   Administer IV
    –   Post test for effect of IV
    –   Compare pre vs. post results to look for effect of

   World events may cause change in attitudes
    or behavior over time.

   Tests of patriotism pre/post 9/11
   Views of President pre/post Katrina
   Attitudes of adolescents pre/post Cobain

   Individuals change over time as they mature.

   Issue for studies of children, but also huge
    growth in freshman year- change of attidues
    and behavior.

   The study you use may cause differences in

   Similar to REACTIVITY, but for entire study
    not just DV. Parenting study for example

   Use of instrument may get better or worse
    with time

   Observation studies
   Testing skill/ interviewing
Regression toward the mean

   Extreme scores do not tend to be repeatable-
    those who score very high or very low on a
    test will be closer to the average if tested

   A big issue for any study where pretest is
    used to select subjects for post test.

   Those who drop out of your study may differ
    from those who choose to continue.
Placebo effect

   If given any treatment, behavior will change,
    even if treatment was not meaningful. (fake
    drugs get some results)
How can we improve internal validity?

   History
   Maturation
   Testing
   Instrumentation
   Regression toward the mean
   Mortality
   Placebo effect
Improved Design

   In pre-post design:                 Two Group design
    –   Test participants               Pretest (do you need to
    –   Administer IV                    do this?)
    –   Post test for effect of IV
                                        RANDOMIZED
    –   Compare pre vs. post
                                         assignment to levels of
        results to look for effect
        of IV                            IV
                                        Compare post test
                                         results of IV and
                                         Control groups
Extraneous Variables

   Any variable that you have not measured or
    controlled (RA) that may impact the results of
    your study
Demand Characteristics

   Participants behave in ways demanded by
    the situation or experimental set-up.
    Behavior does not reflect actual beliefs or

   Issue of Ecological Validity
Subject Bias

   Bias brought on by subjects beliefs
    (Overhead of mood and menstrual cycle)
Social desirability

   Subjects want to do the “right thing” and try
    to guess what the experimenter wants, and
    do not behave naturally.

   How to reduce Subject biases?
Experimenter Bias

   Experimenters’ behavior and expectations
    can sway results of test.

   How to reduce these biases?
Floor & Ceiling Effects

   If measures are too easy or too difficult you
    will not see differences between groups.

   Pilot test with similar subjects!
Order effects

   When using within subjects designs, order of
    presentation can affect results in several ways.
        Practice effects: Subjects get better at task with
               successive trials
        Fatigue effects: Subjects get tired and do worse
               or lose interest
        Carryover effects: subjects experience in one
               condition impacts results of another
               condition- subject bias or anchoring and
               adjustment issues.
How to reduce order effects

   Counterbalancing
    –   Does not get rid of effects, it just makes them
        equal for all groups. Can do complete
        counterbalancing if small number of conditions.
    –   Latin Square counterbalancing
    –   A, B, skip, C, skip, D, etc. then fill back
    –   A, B, N, C, N-1, D, N-2, E etc.
A Latin Square for 6 conditions

1       A   B   F   C    E    D
2       B   C   A   D    F    E

3       C   D   B   E    A    F

4       D   E   C   F    B    A

5       E   F   D   A    C    B

6       F   A   E   B    D    C
Pretest Vs. Pilot test

   When do you use a pilot test?

   When do you use a pre test?
Can a DV be reliable but not valid?
Experimental Validity

   What to do if low Internal Validity?

   What are impacts of low External Validity?

   What if Ecological Validity is low?

Shared By: