Improving efficiency of large-scale surveys through design

Document Sample
Improving efficiency of large-scale surveys through design Powered By Docstoc
					Improving efficiency of large-scale
surveys through design, sampling,
           and scaling

         Michael C. Rodriguez
   Quantitative Methods in Education
     Purpose of Data Collection
• Making decisions about individuals
• Making decisions about groups
• Making decisions based on inferences about
  populations or subpopulations
  – Program evaluation
  – Program development
       Targets of Measurement
• Surveys are well suited for collection of
  information regarding
  – Attitudes
  – Opinions
  – Beliefs
  – Behaviors
• Recently, surveys have also been used to
  assess achievement, knowledge, and skills
    Inferences about Populations
• Often, the volume of information we would
  like to collect is so large, no one person could
  possibly respond to all possible questions.
• It is possible to obtain a better estimate of
  how a population of individuals would
  respond to a large population of items from a
  large sample of persons responding to one
  item than a small sample of persons
  responding to many items.
                Long Surveys
• Response rates appear to be affected by
  survey length – but this is moderated by
  salience, affinity to the survey sponsor, and
  other survey characteristics
• Survey length may matter less on e-surveys
• Less research on response quality
  – Fatigue factors are important considerations on
    longer psychological and educational assessments
         Recent Developments

• Matrix Sampling in Large-Scale Surveys

• Sampling Designs to Reduce Error

• Scaling to Improve Measurement
            Matrix Sampling
• Sampling items from a population of items

• Large numbers of items can be distributed on
  several forms and randomly assigned to
  individuals, reducing the response burden on
  any one individual
     Balanced Incomplete Block
• BIB designs are based on blocks (small sets) of
  items that represent one segment of the
  construct being measured.
• A sample of the blocks are assigned to a single
  form.
• Blocks are rotated across forms.
             Rotating 3 Blocks


Form A          Form B       Form C
Background      Background   Background
Block 1         Block 2      Block 3
Block 2         Block 3      Block 1
             Sampling Design
• Sampling design should be done to reduce
  survey error
  – including coverage error
  – sampling error
  – non-response error
             Sampling Designs
• Simple Random Sample
  – Everyone in the population has a known
    probability of being selected
• Stratified Sample
  – Population is divided into groups based on
    important characteristics, samples are drawn from
    each group
• Cluster Sample
  – Natural clusters of individuals are sampled (clinics,
    schools, counties)
               Scaling and IRT
• Scaling
  – Placing scores on a scale that is interpretable,
    stable, and meaningful
• IRT: Item Response Theory
  – Assumes a latent trait is responsible for an
    individual’s responses to a set of items
  – Places items and persons on the same scale
           Rasch Scaling v. IRT
• Rasch Model Scaling
  – A probabilistic model based on the interaction of a
    person and an item.
  – Slight philosophical difference between Rasch
    measurement and IRT – one based on an analogy
    with the properties of physical measurement
     From Numbers to Meaning
• Numbers themselves do not mean much.
  – Is 100 meters a short distance? Long distance?

• We need context to bring meaning to the
  measure: 100 meters.

• However, 100 meters should always be 100
  meters, no matter who takes the measure or
  how it is taken.
    Sample Dependent Statistics
• If 90% of the sample expresses a certain
  attitude, is that attitude really common in the
  population or an easy attitude to adopt
  (support for a certain policy)?
• If a person identifies with 1 out of 10
  characteristics for a certain trait, is that
  person’s trait level low (e.g., depression)?
          Rasch Measurement
• Person-Free item statistics
  – Locates items on the trait continuum
• Item-Free person trait levels
  – Locates persons on the trait continuum


• Places items and persons on the same scale –
  the trait scale
3             .     +
                    |
                    |


                                           Scaling
                    |
                    |
                    |
                    |
2                   +
                    |T   PGB21F
                    |
              #     |
              .    T|
             .#     |
                    |
1            ###    +
           .####    |S
        ########    |
               .   S|
       .########    |    PGB21E   PGB21G
           .####    |
       .########    |    PGB21H
0    ###########    +M
      ##########    |    PGB21C
      ##########   M|
      .#########    |
               .    |    PGB21B
         .######    |    PGB21D
        .#######    |S
-1        ######    +
               .   S|
             ###    |    PGB21A
              .#    |
           .####    |
              .#    |
              ##    |T
-2             .   T+
                    |
            .##     |
                    |
                    |
                    |
                    |
-3                  +
             Added Benefits
• Selecting items to improve measurement
  (potentially reducing scale size)
• Identification of persons who respond in
  inconsistent ways
• Missing Data (items that are not administered)
• Measuring change over time
• Changing items over time (adding or deleting
  items)
             Decision Making
• Data collection methods (tests, assessments,
  surveys, interviews) are designed to inform
  decision making – sometimes at multiple levels.
• The extent to which data can inform decision
  making depends on the quality of each response.
• Statistics and qualitative data should inform
  decision making, under the guidance of
  professional judgment, but data should not make
  decisions for us.
        Online Resources



www.edmeasurement.net/survey

  MN Evaluation Studies Institute
Comments?

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:7/27/2012
language:
pages:21