Multiple-Group Threats to Internal Validity

							   Multiple-Group Threats to Internal
               Validity




7:02 AM
              The Central Issue

• when you move from single to multiple group
  research the big concern is whether ther groups
  are comparable
• usually this has to do with how you assign units
  (e.g., persons) to the groups (or select them into
  groups)
• we call this issue selection or selection bias


7:02 AM
             The Multiple Group Case
                                    alternative
                                   explanations


          Measure    Administer     Measure
          Baseline    Program      Outcomes

           O            X              O
           O                           O
          Measure      Do not       Measure
          Baseline   Administer    Outcomes
                      Program
7:02 AM                             alternative
                                   explanations
                    Example

• Compensatory Education in Math for 1st
  Graders
• Pre-Post Program-Comparison Group Design
• Measures (O) are standardized achievement
  tests (at start of grade 1 and start of grade 2;
  Forms A & B)




7:02 AM
Selection Threats

                 O            X            O
                 O                         O
          • any factor other than the program that leads to
            posttest differences between groups
          • for example, because of group differences, kids
            in one group watch Sesame Street more
            frequently and pick up more math concepts


7:02 AM
Selection-History
Threat
             O            X            O
             O                         O
     • any other event that occurs between pretest
       and posttest that the groups experience
       differently
     • for example, kids in one group pick up more
       math concepts because they watch more
       Sesame Street

7:02 AM
Selection-Maturation
Threat
             O            X             O
             O                          O
      • differential rates of normal growth between
        pretest and posttest for the groups
      • they are learning at different rates, even
        without program



7:02 AM
Selection-Testing
Threat
                 O              X              O
                 O                             O
          • differential effect on the posttest of taking the
            pretest
          • the test may have “primed” the kids
            differently in each group or they may have
            learned differentially from the test, not the
            program

7:02 AM
Selection-Instrumentation
Threat
              O             X             O
              O                           O
      • any differential change in the test used for each
        group from pretest and posttest
      • for example, change due to different forms of
        test being given differentially to each group,
        not due to program


7:02 AM
Selection-Mortality
Threat
             O             X             O
             O                           O
      • differential non-random dropout between
        pretest and posttest
      • for example, kids drop out of the study at
        different rates for each group



7:02 AM
Selection-Regression
Threat
                 O            X             O
                 O                          O
          • different rates of regression to the mean
            because groups differ in extremity
          • for example, program kids are
            disproportionately lower math scorers and
            consequently have greater regression to the
            mean

7:02 AM
7:02 AM