Data Collection Tools for Qualitative Data by malj


									          Managing for Results

Module 7: Data Collection Tools for Qualitative Data
Data Collection Tools for Qualitative Data
ADS: Qualitative Indicators: (pg. 14)

   Standard Desk Dictionary: Of or pertaining to
    quality: distinguished from quantitative
   Select most appropriate for the result being
   Consistency in questions, scoring,
    procedures of qualitative indicators supports
    reliability of data over time

   Wine or coffee quality or classification
   Music
   Impact of a new government policy on
    population: positive/neutral/negative
   Value of this training program to participants
   Photo of a major intersection over time as
    indicator of pacification in a post war zone
    Quantifying Qualitative Indicators

 Milestone Scales
 Rating Scales

 Indexes
               Milestone Scale

 Tracks incremental
  progress in a series of
 Clearly defines each
  critical step
 Criteria for assessing
  whether each step            Example:
  achieved                     Key Law is Changed
                               What are some
 Expected timeline for when
  each step achieved
               Rating Scale

 Quantifies a range of subjective
  responses on a single issue or
  single dimension of an issue
 Anchor points defined on the scale
  to reduce subjectivity
    Example: This Session and Three Questions

   What truth should the indicator data tell (job)?
     Participant rating of achievement of the training session
     Participant identification of strengths and potential improvements
      in training
   What pieces of reality can tell that truth (source)?
     Participants completing each session
   What mechanics will capture that truth?
     Rating scale 0 – 100 on objective achievement with anchor
     Question on training strengths that support the value assigned
     Question on training potential improvements that could move the
      score closer to 100%
       Rating Scale of Objective Achievement

   The objective of this session is to enable
    participants to make appropriate decisions on
    selecting qualitative vs. quantitative indicators
   To what degree was the objective achieved for you?
       0 = Not at all; 50 = ½ way; 100 = Completely
    0          25           50            75          100

 What were the training strengths
 that support the value assigned

 What are some potential
 improvements that could move
 the score closer to 100%
                        Rating Scale Example

   Expert Panels: Rating by an expert panel on
    progress in fiscal reform
    Rate each element of progress in the fiscal reform agenda on a scale of –5 to +5,
    where –5 represents serious/significant backward movement; 0 represents no
    movement and +5 represents serious/significant forward movement.
    Fiscal deficit                    -5   -4   -3   -2   -1   0   1   2   3   4   5

    -Taxes                            -5   -4   -3   -2   -1   0   1   2   3   4   5

    -Social sector spending           -5   -4   -3   -2   -1   0   1   2   3   4   5

   Combines one or more data sources into a
    single measure
   Useful to represent multiple dimensions of
   Final summary value may be difficult to
    interpret and changes minimal.
   Disaggregated data may be more informative
                              Index Example

Combination of two or more ratings collapsed into a single measure.
Indicator: score of target CSO on the “CSO Capacity Index”.

1. Organizational Capacity                  1 2 3 4 5
2.   Financial Management                   1 2 3 4 5
3.   Networking                              1   2 3 4 5
4.   Strategic Planning/Vision               1 2 3 4 5
5.   Impact (in selected CSO area)           1   2 3 4 5
                                            Actual Total = 16 / 25 or
-    1 is low and 5 is outstanding
-    Minimum 0, Maximum 25
           Other Examples of Indices

   Couple Years of Protection (CYP)
   AIDS Program Effort Index (API)
   Corruption Perceptions Index (Context)
   Index of Economic Freedom (Context)
   Organizational Capacity Assessment Tool
   Review OCAT PIRS Handout
Understanding Qualitative Context Indicators

   Context indicators measure the
    environment within which each AO is
    implemented at the outcome or impact
   Explore the Doing Business In Indicator for
   Explore the WB Institute Governance
    indicators for Uganda
     Qualitative Subject to Same Questions as

                            Validity       Integrity      Precision    Reliability   Timely
   Data Quality Criteria   7 Ind.
   Truth?                     Does it     Manipulate     Sufficient   Stable and    Frequent
                              measure         d for       detail to    consistent     enough
   Source?                  what was      political or    support        data          for
                            intended to     personal      decision     collection    decision
   What data collection     measure?      reasons?       making?      processes

    method/ mechanics?       Objective                                  analysis
                             Practical                                 over time?
Application: Clinic on Data Collection Tools for
             Qualitative Indicators

   For Selected Qualitative Indicators:

       Review job of data (What truth should they tell?)
       Review source options (What pieces of reality can tell it?)
       Review options for mechanics (What will tell it?) (Milestone
        scale, Rating scale, Index, other?)
       Test against the Data Quality Standards
       Complete Definition, Unit of Measure, and Data Collection
        sections of Performance Indicator Reference Sheet.
        (Attach any forms, tools, protocols)

To top