Validity and Threats to Validity in Research by dfhdhdhdhjr


									Validity and Threats to
 Validity in Research
  Extraneous Variables and Confounding

• Extraneous variables
• Example
  – An investigation of the effect of parental
    involvement on students GPA.
  – What is the relationship between parental
    involvement and intellect (or socioeconomic
  – What is the relationship between intellect
    (or socioeconomic status) and GPA?
      Confounding variables
• When an extraneous variable
  systematically varies with the
  independent variable it becomes a
  confounding variable.
• Extraneous variables can influence a
  confound in subtle ways.
• Example: Colors and selection of
      Confounding and Validity
• Confounding variables threaten the
  validity of quantitative research results.
• A goal of research design is to guard
  against these threats to validity.
• Four types of validity:
  –   Statistical conclusion validity.
  –   Internal validity.
  –   Construct validity.
  –   External validity.
Statistical Conclusion Validity
• Concerned with the inference that
  there is a relationship between an
  independent variables and a dependent
  variable (and the strength of that
• Questions the appropriateness of the
  statistical analysis used.
• Typically involves computing effect size.
          Internal Validity
• Most research is focused on
  determining whether a causal relation
  exists between independent variables
  and dependent variables.
• Two types of causal relationships:
  – Causal description: describe the
    consequences of manipulating an
    independent variable.
  – Causal explanation: explain the dynamics of
    a causal relationship.
 Criteria for Inferring Causation
• Evidence (usually statistical) of a relationship
  between two variables.
  – Correlation.
  – Comparisons.
• Temporal ordering (knowing which variable
  comes first).
• Elimination of competing causes (confounding
  extraneous variables).
  – Consider the relationship between attendance and
Threats to Internal Validity: History

• Events, other than the independent
  variable(s), that occur between the pretest
  and posttest and influence the postest.
• Example: A study to investigate the effect of
  peer tutoring on spelling performance.
  – Teacher provides instruction on students and
    monitor the tutoring process.
  – Monitoring may influence students’ motivation to
    do well (for both tutor and tutee).
Threats to Internal Validity: Maturation

• Developmental (physical, experiential,
  cognitive) changes that occur between
  the pretest and posttest that affect
  performance on the posttest.
• Example: The effect of SAT training on
  SAT performance.
  – There exists documented evidence that
    SAT scores improve by, on average, 50
    points from one year to the next.
        Threats to Internal Validity:
        Testing and Instrumentation
• Testing: A change in scores on the posttest as
  a result of having taken the pretest.
  – Tests can serve as advanced organizers.
  – Tests can sensitize individuals to particular
    aspects of the treatment.
• Instrumentation: Any change that occurs in
  the way the dependent variable is measure.
  – Test norms can change over time.
  – Raters can get better over time.
 Threats to Internal Validity: Regression

• An artifact of selection according to
  some criterion on the pretest and then
  using the same (or similar) measure for
  the posttest.
• A function of the unreliability of
  measuring instruments.
• Example: remedial instruction for
  children scoring low on the ABC tests.
        Threats to Internal Validity:
           Differential Selection
• Selection of intact groups that may
  have different characteristics to
  receive different treatments.
• Intact groups can differ in many ways,
  Age                Intelligence     Reading ability
  Anxiety Level      Learning style   Religious belief
  Curiosity          Maturity         Self esteem/concept
  Coordination       Motivation       Socioeconomic status
  Hearing ability    Personality      Testwiseness
  Home environment   Eyesight         Time spent on homework
  Threats to Internal Validity: mortality

• Differential attrition.
  – Differential loss of participants from the
    different comparison groups.
  – NOT random attrition.
  – Can occur coincidentally, or
  – As a direct consequence of the treatment
             External Validity
• Concerned with the extent to which a study’s
  results can be extended (generalized) to
  other populations, settings, outcomes, and
  treatment variations.
• Three types of external validity:
  –   Population validity.
  –   Ecological validity.
  –   Outcome validity.
  –   Treatment variation validity.
             External Validity:
 Population validity and ecological validity
• Population validity
  – To which populations can results obtained
    from a sample be generalized?
     • To the Target population?
     • To populations more broadly defined?
• Ecology validity
  – Can results be generalized to other
  – To what extent is subject reactivity a
              External Validity:
  Outcome validity and Treatment variation validity

• Outcome validity.
  – Can the results of a study be generalized
    to different but related dependent
  – E.g., ITED vs CTBS and dependent vars.
• Treatment variation validity.
  – Can the results be generalized across
    variations in the independent variables?
  – E.g., Traditional instruction in one school vs
    traditional instruction in another school?
         Construct Validity
• Concerned with the extent to which
  higher-order constructs are
  represented in a particular study.
  – Can be affected by operational definitions.
    • Identifying specific operations is often
      convenient (and usually necessary), but tends to
      narrow the construct.
    • Bandwidth-fidelity problem.
    • Jingle/jangle fallacies.
      Threats to Internal Validity:
    Additive and Interactive Effects
• Several of the threats to validity can
  combine with differential selection to
  produce additive or multiplicative bias.

  – Selection-history.
  – Selection-maturation.
  – Selection-mortality

To top