Item Evaluation Template by yws14770

VIEWS: 7 PAGES: 56

More Info
									DESIGNING EVALUATION
    INSTRUMENTS


       Class X
       AEE 577
• Upon completion of this lesson, students should be able to:
   – List the step by step procedures for developing quality
     evaluation instruments;
   – Describe the errors that must be controlled in evaluation
     instruments;
   – Develop different forms of questions to record outcomes such
     as change in knowledge, attitudes, skills, aspirations, and
     behaviors;
   – Write process evaluation questions;
   – Describe reliability and validity
   – Identify double barreled questions; and
   – Develop an evaluation instrument.
 How to Design Your Data
  Collecting Instrument?


•Where to begin?
Begin with the information
needs of key stakeholders


• Information needs for program improvement
• Information needs for accountability
     Designing Instruments
 Step 1: Identify the type of data and
   information you need to collect.
• Focus on the information needs of key
  stakeholders.
• Clearly identify what data and information are
  needed to collect for this purpose.
• Identify the major categories of information
  that you need to collect.
• List subcategories of information under major
  categories.
     Designing Instruments
     Step 2: Develop the “Sketch” of
             Your Instrument.
• List the major items in your instrument to
  structure it.
• Organize the structure of your instrument to
  collect needed data.
• Organize subcategories under each major topic.
• Include the demographic data collection section
  at the very end of the instrument.
    Designing Instruments
   Step 3: Identify Necessary Scales
             and Questions.

• Determine the types of scales you need to
  include in your instrument.
• Determine the types of questions you
  need to ask.
         Designing Instruments
     Step 4: Be Consistent in Numbering
         Answer Choices and Scales
•    It is a good idea to use low numbers for lower
     manifestation of a measuring variable.
•    Example:
    1.   High school diploma
    2.   Bachelor‟s degree
    3.   Master‟s degree
    4.   Doctorate

•    By using a consistent pattern throughout the
     instrument you can easily interpret results.
     Designing Instruments
      Step 5: Writing Questions
• As a general rule, when writing questions, you must ask “why am I
  asking this question?”
• Remember your evaluation information needs always.
• Think about the answer before you write any question.
• There are two ways to write a question
   – Open-ended
   – Example: What methods do you use to educate farmers on
     sustainable agriculture?
   – Closed ended
   – Example: What methods do you use to educate farmers on
     sustainable agriculture?
       •   Field days
       •   Workshops
       •   Seminars
       •   Printed materials
       •   Electronic materials
       •   Others (please specify)___________________
   Designing Instruments
    Writing Open-Ended Questions
• Things to remember when writing questions:
  – Write questions clearly and concisely.
  – Start with least sensitive or non-threatening questions.
  – Write questions by thinking about the reading level of the
    target population.
  – Avoid double negatives.
  – Avoid double-barreled questions.
  – Example: Are you satisfied with the place and time of the
    program?
Designing Instruments
 Writing Open-Ended Questions
– Open-ended questions are useful to explore a topic in depth.
– However, open-ended questions are difficult to:
    • Respond
    • Analyze
– Therefore, limit the number of open-ended questions to the
  needed minimum.
– When you need to ask a sensitive question it is appropriate
  to use a closed-ended question with response categories for
  the sensitive information.
– Example: Asking income or age (Ask what is your age group
  and provide age categories instead of asking how old are
  you?)
   Designing Instruments
   Writing Closed-Ended Questions
• When writing closed-ended questions:
  – Make sure to include all possible response categories.
  – If you have not included all possible answer categories, it is a
    good idea to include a category called „Other‟ and provide
    instruction to specify what the respondent means under this
    category.
  – Make sure that your answer categories are mutually exclusive.
  – Example: What is your age group?
      • Less than 20 years
      • 20-30 years
      • 31-40 years
      • 41-50 years
      • Above 50 years
Designing Instruments
Writing Closed-Ended Questions
– Closed-ended questions are:
  • Easy to analyze.
  • Not exploratory in terms of searching information.
      Scale Development
• Develop scales if you need to include in your
  instrument.
     Guidelines For Scale
        Development
• Scales are developed for measuring elusive
  phenomena that cannot be observed directly.
  Example: Attitudes, Aspirations.
• Therefore, scale development should be
  based on the theories related to the
  phenomenon to be measured.
• Thinking clearly about the content of a scale
  requires thinking clearly about the construct
  being measured.
   Guidelines For Scale
      Development
    Generate an Item Pool
• The properties of a scale are determined
  by the items that make it up.
• At this stage, you need to develop more
  items than you plan to include in the final
  scale.
Characteristics of Good Items
•   Unambiguous.
•   Avoid exceptionally lengthy items.
•   Consider reading levels of the target respondents.
•   Include positively and negatively worded items. The
    purpose of wording items both positively and
    negatively within the same scale is usually to avoid
    acquiescence, affirmation, or agreement bias.
       Guidelines For Scale
            Development
  Determine the Format for Measurement
• There are different formats
• Identify the format you would like to use with
  your items.
• Determine how many response categories you
  need to include in your format.
     Guidelines For Scale
          Development
Determine the Format for Measurement
• The number of response categories should be limited to
  the respondents‟ ability to discriminate meaningfully.
• Normally 5-7 response categories are adequate for
  extension and education program evaluations.
• Example:
   1.   Strongly disagree
   2.   Disagree
   3.   Neutral
   4.   Agree
   5.   Strongly agree
       Guidelines For Scale
          Development
            Likert Scale
• Named after Rensis Likert.
• This is the most common format
• The response options should be worded so as to
  have roughly equal intervals with respect to
  agreement. That is to say the difference in agreement
  between any adjacent pair of responses should be
  about the same as for any other adjacent pair of
  response options.
• Common choices for a mid point include neither
  agree nor disagree and Neutral.
  Guidelines for Scale
     Development
       Likert Scale
Example for items in Likert format
   1.   Strongly Disagree
   2.   Disagree
   3.   Neutral
   4.   Agree
   5.   Strongly Agree
     Guidelines For Scale
        Development:
    Semantic Differential Scaling
• There are several numbers between the
  adjectives that constitute the response
  options.
• Example: The quality of training session
  Poor 1 2 3 4 5 6 7 Excellent
        Example For a Scale
        (Recording Anxiety)
                    Not at   Some   Moderately   Very
                     all     what                much
1. I feel calm        1        2        3         4

2. I am tense         1       2         3         4

3. I feel upset       1       2         3         4

4. I am relaxed       1       2         3         4

5. I feel content     1       2         3         4

6. I am worried       1       2         3         4
   Instrument Development
Step 6 Provide Necessary Instructions to
         Complete the Survey
• Clear instruction is essential to facilitate
  the responding process.
• Instructions should be clearly and politely
  stated.
• Clear instructions increase your return rate
  as well as accuracy of your data.
       When You Develop a
         Questionnaire:
• Keep it short, simple, and clear
• Include only needed questions for indicators
• Should be compatible with the reading level
  of the respondents
• When you use closed-ended questions make
  sure to include all possible answer choices.
   Instrument Development
    Step 7 Format Your Instrument

• Appearance and editing of your
  instrument are important determinants
  of response rate.
• Therefore, format, structure, and edit
  your instrument professionally.
     Instrument Development
       Step 8 Establish Validity and
      Reliability of Your Instrument
• Reliability refers to the extent to which a measuring
  instrument is consistent in measuring what it
  measures.
   – Test-retest method : We administer the instrument to a
     sample of subjects on two occasions and correlate the
     paired scores to establish the reliability.
• Validity refers to the extent to which an instrument
  measures what it intends to measure.
   – Use experts‟ views to establish validity.
APPLICATION OF STEPS
Determine Your Evaluation
       Questions
• Identify the precise questions need to
  be answered.
• Use the logic model to narrow the focus
  of evaluation.
                           LOGIC MODEL
            Measuring Program Impact
INPUTS                     OUTPUTS                         OUTCOMES - IMPACT
                   Activities Participation             LEARNING            ACTION              IMPACT
What resources    What should      Who should     -     What do you         What do you         What kind of
does your         you do in        participate    -     expect the          expect that         impact can
program need to   order to         be involved?     -   participants will   participants        result if the
achieve its       achieve          be reached?          know, feel or be    will do             participants
objectives?       program goals                         able to do          differently after   behave or act
                  and                                   immediately         the program?        differently?
                  objectives?                           after the
                                                        program?

Staff Volunteer   Workshops        Number of target       Awareness          Behavior           Social
Time & Money      Meetings         clients                Knowledge          Practice           Economic
Materials         Camps            Their                  Attitudes          Decisions          Environmental
Equipment         Demonstrations   characteristics        Skills             Policies
Technology        Publications     Their reactions        Aspirations        Social Action
Partners          Media
                  Web site
                  Projects
                  Field Days
 Possible Question Categories
• Process evaluation questions (These are mostly open-ended)
   – Questions on client characteristics
       • How do you describe your ethnicity?
   – Questions on program delivery
       • What are the strengths of this program?
       • What are the weaknesses?

• Impact evaluation questions
   – Questions on clients satisfaction
       • Did the target clients find the program useful?
   – Outcomes
       • Did the program participants change KASA?
       • Did the program participants change their practices?
   – Impacts
       • Did the participants save money/improve health condition?
        What Data Are Needed for
         Program Improvement?
• Were participants satisfied with:
    –   Information received
    –   Instructors
    –   Facilities
    –   Quality of training
•   What do they like/dislike about the training
•   Did the training meet their expectations?
•   If not, Why
•   Ideas for further improvement
•   Look for data that you can use to fix
    weaknesses and build on strengths.
                           How to Collect Training
                            Improvement Data?
Please circle the appropriate number for your level of response.
                                                  Not Satisfied    Somewhat Satisfied   Satisfied   Very Satisfied
How satisfied are you with:
The relevance of information to your needs?             1                  2               3              4
Presentation quality of instructor(s)?                  1                  2               3              4
Subject matter knowledge of instructor(s)?              1                  2               3              4
Training facilities?                                    1                  2               3              4
The overall quality of the training workshop?           1                  2               3              4
          How to Collect Training
           Improvement Data?
•   Did the training session meet your expectation?
    1. Yes
    2. No
•   Would you recommend this training workshop to others?
    1. Yes
    2. No
•   If not, why:____________________________________
•   What did you like the most about this training?
•   What did you like the least about this training?
•   How could this training be further improved?
        Other Data
Demographics
• What is your gender?
  ____ Male
  ____ Female
• How do you identify yourself?
  ___African American
  ___American Indian/Alaskan
  ___Asian
  ___Hispanic/Latino
  ___Native Hawaiian/Pacific Islander
  ___White
  ___Other
What Data Are Needed for Program
        Accountability?

   • You need impact data
   • To prove that your program achieved
     its objectives
                  How to Document Perceived
                     Knowledge Change?
 Example for Agriculture


How do you rate your knowledge               BEFORE THIS WORKSHOP                          AFTER THIS WORKSHOP
about:
                                 Very   Low        Moderate    High     Very   Very   Low       Moderate     High    Very
                                 Low                                    High   Low                                   High
Conservation tillage systems.     1      2            3             4    5      1      2           3             4    5
Crop rotations.                   1      2            3             4    5      1      2           3             4    5
Weed management under             1      2            3             4    5      1      2           3             4    5
conservation tillage.
       How to Document Levels of
             Aspirations?
• At the end of a successful training session,
  participants will have a heightened level of
  aspirations to apply what they learned.
• They are ready to “taking charge” of what they
  learned.
• Participants are asked whether they intend to apply
  what they learned.
• Example: As a result of this training, do you intend to
  drink reduced fat milk? The answers to this question
  would be:
   –   No
   –   Maybe
   –   Yes
   –   I‟m already doing this
          How to Document Aspirations?
Example for FCS
   Please circle the number that best describes your answer.

                                                                              Already
                                                         No    Maybe   Yes
As a result of this program, do you intend to:                               doing this

1. Eat recommended servings from five food groups?       1       2      3        4

2. Plan meals ahead of time?                             1       2      3        4

3. Consume reduced or non-fat milk and dairy products?   1       2      3        4
        Retrospective Pre and Post
               Evaluations
• Advantages:
   – Simple & Easy to collect data
• Disadvantages:
   – Not appropriate for collecting data from very young
     audiences and low literacy adult audiences. Because
     they will not be able to compare before and after
     situation retrospectively.
       Pre and Post Evaluations
• Pre Evaluation is administrated before your training
  session.
• Post Evaluation is administrated at the end of your training
  session.
• We need to match pre and post evaluations for
  comparison.
• Pre and Post Evaluation will document three impact
  indicators:
   – Change in Knowledge
   – Change in Skills
   – Levels of Aspirations
How to Document Change in
       Knowledge?
• Ask same set of questions before and after
  your educational session and compare their
  answers to document the knowledge gain
  from the program.
                   How to Document Change in
         Example for FCS
                          Knowledge?
Please circle your answer to each of the following statements.                                                             True    False    Don’t
                                                                                                                                            Know


1. According to MyPyramid, the recommended single serving size of a raw, chopped vegetable is 1/2 cup.                      True    False    Don’t
                                                                                                                                             Know
2. According to MyPyramid, the number of servings recommended daily from the Milk, Yogurt & Cheese group is 4 to 5 cups.    True    False    Don’t
                                                                                                                                             Know
3. Daily Values (DV) listed on the bottom of some food labels are the same values for all individuals.                      True    False    Don’t
                                                                                                                                             Know
4. The amount that you need from each group of MyPyramid depends on how many calories you need.                             True    False    Don’t
                                                                                                                                             Know
5. If you eat more food (calories) than your body needs, the extra calories get stored as body fat.                         True    False    Don’t
                                                                                                                                             Know
6. Food high in saturated fat increases risk for heart disease.                                                             True    False    Don’t
                                                                                                                                             Know
7. The Nutrition Facts label on foods tells you how many calories and nutrients are in one serving.                         True    False    Don’t
                                                                                                                                             Know
8. Fruits and vegetables contain fiber which helps prevent constipation.                                                    True    False    Don’t
                                                                                                                                             Know
9. Foods that contain protein are located in the meat, milk, and grain groups of the MyPyramid.                             True    False    Don’t
                                                                                                                                             Know
10. Vitamin A found in many fruits and vegetables helps our bodies absorb iron.                                             True    False    Don’t
                                                                                                                                             Know
 How to Write Knowledge Testing
          Questions:
• Don‟t use general knowledge questions.
• Don‟t include attitudinal or perceptual statements.
   – Example: Growers should practice conservation
     tillage. __True __False ___Don‟t Know
    True and False Questions vs
     Multiple Choice Questions
• True and False questions save your time
  and respondents‟ time.
• Easy to analyze.
• Help you keep your survey short.
    How to Document Change in
             Skills?
• Skill changes are measured indirectly by
  using participants‟ levels of confidence to
  carry out the learned tasks from the program.
  Example: Participants‟ confidence in their
  ability to calibrate a sprayer.
    How to Document Change in
             Skills?
• We record their levels of confidence for
  carrying out specific tasks before and after
  the program on a Likert-type scale.
• Compare pre and post responses to
  document changes in skills.
       How to Document Change in
Example for Agriculture:
                         Skills?

                                               Not        A little   Somewhat                  Very
 How confident are you in your ability to:                                       Confident
                                             confident   confident   confident               confident


 1. Keep waste management records?              1           2           3           4           5
 2. Calculate land application equipments?      1           2           3           4           5
 3. Calculate nutrient removal levels?          1           2           3           4           5
        Pre and Post Evaluations
• Advantages:
  – Appropriate for young and low reading
    audiences.
• Disadvantages:
  – If you want to compare pre and post evaluations
    you must match pre and post evaluations for
    each participant.
  – This is somewhat challenging.
  Change Attitudes

• Difficult to measure
• Need to be very careful in
  designing scales to measure
  attitudes
• Not a practical indicator
• Pre/Post tests
CHECKING ATTITUDES
 To what extent do you agree or disagree with each of the
 following statements
          Statement                     Strongly   Disagree    Undecided   Agree Strongly

                                        Disagree                                  Agree
                                             1             2           3     4        5

a)Conservation Tillage is profitable.       1          2            3       4        5
b) Conservation Tillage is not practical 1             2           3        4        5


(Need to include at least 10-15 items to achieve desired level of validity and
reliability)
       How to Document
       Behavior Change?

• You need to understand the behavior
  change process for designing evaluation
  questions.
     Understanding Behavior
        Change Process
• Behavior change is a process.
• Prochaska and DiClemente developed a model to
  explain the human behavior change process. This
  model is called the Transtheoretical Model.
• According to the Transtheoretical Model, there are
  five stages in behavior change process.
  Prochaska and DiClemente’s
       Stages of Change
                                                                          Characteristics


 Pre-contemplation                                    Not currently considering this change:
 (I’m not considering this)                           "Ignorance is bliss"
 Contemplation                                        Ambivalent about the change: "Sitting on
 (I’m considering this)                               the fence"

 Preparation                                          Some experience with the change and are
 (I’m doing this sometimes)                           trying to change: "Testing the waters"
 Action                            Practicing new behavior or practice
 (I’m doing this most of the time)
 Maintenance                                          Continued commitment to sustaining new
 (This is now a part of my life)                      behavior or practice Stage of Change

Prochaska, J. O. and DiClemente, C. C. (1994). The Transtheoritical Approach: Crossing Traditional Boundaries of Therapy.
         Malabar, Florida: Kerieger Publishing Company.
             Evaluation Template
For each of the following practices, please circle the number that best describes your current behavior.

            Practices                 I am not         I am         I am doing     I am doing    I am doing
                                    considering     considering         this      this most of    this all of
                                         this          this         sometimes        the time      the time

1. Drinking fat free or reduced
   fat milk.                              1              2               3              4             5

2. Doing exercise at least
                                          1              2               3              4             5
   30minutes/day for five days.
3. Eating baked, broiled, or
   grilled foods rather than              1              2               3              4             5
   eating fried foods.
 How to Collect Impact Data
from Multi-Session Programs
• “Benchmark Survey” is administrated before the
  Extension program.
• “End of Program Survey” is administrated at the end of
  the extension program.
• By comparing benchmark and end of program surveys you
  will be able to document the change of participants‟
  behaviors/practices and skills.

								
To top