Slide 1 Institute of Education Sciences by wanghonghx

VIEWS: 6 PAGES: 28

									       Session 20:
Data Collection in the Field
            Ina F. Wallace
           RTI International



   IES/NCER Summer Research Training Institute, June 2007   1
   Data Collection in the Field
           Topics Covered

• Methods of Data Collection
• Use of State-of-the-Art Data Collection
  Procedures
• Ensuring Data Quality
• Conducting Small-Scale Validation Studies
• Ethical Issues in Data Collection

                                         2
    Methods of Data Collection
• Interviewer Administered
  – In Person
     • Surveys
     • Assessments
  – Telephone
• Self-Administered
  – Paper and Pencil
     • Mail-in
     • In-person
  – Computer Assisted Self Interview
  – Web procedures
                                       3
             Data Collection Mode
                Characteristics
                       High Data Collector                  Low Data Collector
                          Involvement                          Involvement
                    Paper             Computer             Paper          Computer
Direct       Face-to-Face         Face-to-Face                           (Audio)
Contact with Paper and            Computer Assisted                      Computer
Respondent Pencil                 Personal Interview/                    Assisted Self-
               Interviewing/      Assessment (CAPI)                      Interview
               Assessment                                                (CASI,
               (PAPI)                                                    A-CASI)
Indirect     Telephone            Telephone             Mail, Fax, E-    Touch-tone
Contact with PAPI                 Computer Assisted     mail             Data Entry
Respondent                        Telephone Interview   questionnare     (TDE), Web,
                                  (CATI)                                 Electronic Mail
                                                                         Survey (EMS)
No Contact     Direct             Computer Assisted     Administrative   Electronic Data
with           Observation        Data Entry (CADE)     Records          Interchange
Respondent                                                               (EDI)

                                                                                     4
 Adapted From Biemer & Lyberg (2003). Introduction to Survey Quality
     Face-to-Face Interviewing
Advantages                  Disadvantages
• Maximum degree of         •High cost due to travel
  communication and         •Can generate social
  interaction                desirability for sensitive
• Can utilize PAPI or CAI    questions
• Allows for long,          •Interviewers can affect
  complex interviews         respondents
• Variety of tactics can    •Can generate
  be used to gain            interviewer variance
  cooperation               •Presence of others can
• Response rates             affect responses
  usually high              •Falsification of
                             interview
                                                      5
         Telephone Interviewing
Advantages                   Disadvantages
• Cost lower than face to    • Less flexible
  face                       • No ability to use visual
• More easy to provide         aids
  training, supervision,     • Must be shorter
  and feedback               • Questions must be less
• Social desirability bias     complicated
  less than with face to     • Less capacity for
  face                         motivating the respondent
• Interviewer variance       • No coverage of non-
  less                         telephone units
                             • Response rates lower
                                                      6
                 Mail Surveys
Advantages                  Disadvantages
• Lowest cost               • Little control over who
• Low social desirability     completes the survey
• Can utilize visual aids   • Item non-response high
• Allows respondent to      • Response rates lower
  take time and look up     • Require a long field period
  information (if             to obtain an acceptable
  necessary)                  response rate
• No interviewer variance   • Respondents must have
                              good literacy skills
                            • Limit to closed-format
                              questions
                                                        7
    Computerized Procedures
• Computer Assisted Personal Interview
  (CAPI)
• Computer Assisted Telephone Interview
  (CATI)
• Computer Assisted Self Interview (CASI)
• Audio Assisted Self Interview (ACASI)
• Telephone Audio Assisted Self Interview
  (T-ACASI)
• Computer Assisted Data Entry (CADE)
• Web Surveys
                                            8
    Computer Assisted Procedures
Advantages                      Disadvantages
• Complex skip patterns and     • Time consuming to develop
  branching are easier to         and program computer
  follow                        • Must be adequately tested
• Previously acquired           • Can’t change things easily
  information can be              once fielded
  programmed                    • Risk of computer crashing
• Ranges, inconsistencies,      • Costly
  and illegitimate values can
  be checked
• Help screens can be
  included
• Data are available for
  analysis more quickly
                                                               9
         Direct Assessment
• Used to obtain psychological and
  educational information about an individual
• Individually administered
• Research driven NOT for clinical or
  diagnostic purposes
• Follows a standardized protocol



                                           10
    Observational Techniques
• Involve a researcher making observations
  within a natural setting
• Direct Observation - individuals know they
  are being observed
  – Continuous Monitoring recording as much
    behavior as possible
  – Time Allocation randomly visiting the sample
    to observe at different times
• Unobtrusive Observation – individuals do
  not know they are being observed
                                                   11
       Observational Techniques
Advantages                   Disadvantages
• No or limited              • Variance associated with
  participation needed         observers is similar to
                               that found with
• May be good for              interviewers
  generating hypotheses      • Can be very costly as
• Sensitive social issues      they are often labor
  can be obtained more         intensive
  accurately                 • Time consuming
• High in validity because   • Reliability can be low
  of the depth of            • Generalizability can be
  information collected        low
                                                     12
              Summary:
    Issues in Choosing a Strategy
•   Type of Population
•   Nature of Research Issue
•   Question Form
•   Question Content
•   Response Rates
•   Costs
•   Length of Data Collection

                                    13
    Methods of Administration
• Standardized Interviewing Approach
  – A protocol in which interviewers interact with
    the respondent in a manner which is
    consistent across all interviewers
• Conversational Interviewing
  – A protocol in which interviewers modify and
    adapt questions to the respondent’s situation
• Hybrid Interviewing
  – Combination of the two approaches

                                                     14
          Factors that Can Explain
            Interviewer Variance
       Interviewer                                               Respondent
•Characteristics                                          •Characteristics
•Appearance                                               •Knowledge
•Motives                                                  •Interest/motivation
•Beliefs/attitudes                                        •Confidence
•Perceptions                                              •Strength of convictions
•Expectations                                             •Expectations
•Skills/Knowledge


    Questionnaire                                          Survey Conditions and
•Definition clarity            Interviewer Effects                  Setting
•Terminology/jargon                                       •Mode of interview
•Question form                                            •Standardization
•Instructions                                             •Interviewer training
•Question wording                                         •Interviewer supervision
•Question topic                                           •Monitoring/observation



                                                                                     15
 From Biemer & Lyberg (2003). Introduction to Survey Quality
            Factors that Can Explain
              Interviewer Variance
       Interviewer                                                    Respondent
•Characteristics                                               •Characteristics
•Appearance                                                    •Knowledge
•Motives                                                       •Interest/motivation
•Beliefs/attitudes                                             •Confidence
•Perceptions                                                   •Strength of convictions
•Expectations                                                  •Expectations
•Skills/Knowledge


    Questionnaire                                               Survey Conditions and
•Definition clarity                      Interviewer Effects             Setting
•Terminology/jargon                                            •Mode of interview
•Question form                                                 •Standardization
•Instructions                                                  •Interviewer training
•Question wording                                              •Interviewer supervision
•Question topic                                                •Monitoring/observation



 From Biemer & Lyberg (2003). Introduction to Survey Quality                              16
Role of the Interviewer/Assessor
• Gaining Cooperation
  – Converting reluctant participants
  – Recognizing and dealing with refusals
• Training and Motivating Respondents
  – Interviewers implicitly and explicitly teach
    respondents how to behave
• Obtaining Quality Data
• Ensuring Confidentiality

                                                   17
         Assessing Children
• Assessor Characteristics
  – Flexibility
  – Vigilance
  – Self-awareness
  – Eye contact
• Establishing Rapport
  – Allaying apprehensions
  – Helping children feel at ease
  – Maintaining limits on permissible behavior
• Arranging the Physical Environment

                                                 18
        Ensuring Quality Data
• Recruiting interviewers/assessors
  – Characteristics of all data collectors
  – Characteristics of interviewers
  – Characteristics of assessors
• Training on standardized procedures
• Certification as data collectors
• Monitoring performance


                                             19
                     Training
Content of Training           Procedures for Training
• Presenting the study        •   Written materials
• Contacting participants     •   Lecture and presentation
• Administering the           •   Computer-based tutorials
  questionnaire/ assessment   •   Planned exercises
• Probing                     •   Role Playing
• Recording responses         •   Observation
• Confidentiality             •   Certification
• Interpersonal relations
• Administrative procedures

                                                             20
      Interviewer Training Agenda
•   Introduction to the Study
•   Role of the Interviewer
•   Confidentiality Procedures
•   Review of Questionnaire and Protocols
•   Standardized Interviewing Techniques
•   Round Robin Interview
•   Gaining Cooperation
•   Presentation of Scenarios
•   Round Robin Demonstrations
•   Paired Mocks
•   Certification
                                            21
         Assessor Training Agenda
Day 1:                      Day 2:                       Day 3:

Large group                 Small groups                 Small groups
• Introduction to Project   • Individual practice with   • Review of
• Role as an Assessor         assessments                  assessments
• Best practices with       • Paired practice with       • Paired practice
  children                    assessments                • Post-training quiz

Small groups                Large group                  Large group
• Pre-training quiz         • Building rapport and       • Concluding remarks
• Introduction to all         gaining cooperation        • Certification
  materials                 • Administrative
• Learn each                  responsibilities
  assessment                • Data security
   –Observation             • Informed Consent
   –Round Robins

                                                                                22
       Monitoring Performance
What to Monitor              How to Monitor
• Detection and prevention   • Telephone verification
  of falsified information     with a sample of
• Compliance with rules        respondents
  and guidelines about       • Direct observation – in-
  administering the            person or by telephone
  questionnaire or           • Audio-recording – CARI
  assessment                 • Review of materials
• Performance on non-        • Performance and
  interview/assessment         production measures
  tasks


                                                          23
       Human Subjects Issues
• Institutional Review Boards
• Informed Consent
  – Who is carrying out research under whose sponsorship
  – Description of purposes and procedures
  – Duration of involvement
  – Risks and Benefits
  – Confidentiality
  – Voluntary nature of research including stopping early
    and skipping questions
  – Compensation
  – Contact information
• Confidentiality Issues
                                                      24
       Human Subjects Issues
• Institutional Review Boards
• Informed Consent
  – Who is carrying out research under whose sponsorship
  – Description of purposes and procedures
  – Duration of involvement
  – Risks and Benefits
  – Confidentiality
  – Voluntary nature of research including stopping early
    and skipping questions
  – Compensation
  – Contact information
• Confidentiality Issues
                                                      25
          Data Collection Evaluation
  Stage of Data           Evaluation Method                       Purpose
   Collection
Design                 Expert review of instrument     Identify problems with
                                                       instrument
Pretest                Behavior coding                 Evaluate one or more stages
                       Cognitive interviewing          of the response process
Pretest/Post-Data      Respondent focus groups         Evaluate instrument and data
Collection             Interviewer debriefings         collection procedures
Pretest/Data           Supervisor observation          Evaluate performance of data
Collection             Telephone monitoring            collector
                       CARI
Post-Data              Experimentation                 Compare alternative methods
Collection             Internal Consistency            of data collection
                       External validation             Evaluate components of MSE
                       Re-interview/retest             Validate estimates
                                                                                26
Adapted From Biemer & Lyberg (2003). Introduction to Survey Quality
Post-Data Collection Evaluations
•   Experiments
•   Observational Studies
•   Internal Consistency Studies
•   External Validation Studies
•   Administrative Record Check Studies
•   Re-interview/Re-Test Studies



                                          27
Post-Data Collection Evaluations
•   Experiments
•   Observational Studies
•   Internal Consistency Studies
•   External Validation Studies
•   Administrative Record Check Studies
•   Re-interview/Re-Test Studies



                                          28

								
To top