An Evaluation Plan and Tool Kit for the by whattaman

VIEWS: 11 PAGES: 19

									 An Evaluation Plan and Tool Kit for the
Archibald Bush Innovative Teaching and
      Technology Strategies Grant

          Valerie Ruhe and J.D. Walker,
  Center for Teaching and Learning Services and
               Digital Media Center

             University of Minnesota



                                                  1
             Evaluation Plan
   Evaluation: A formal appraisal of the
    quality of an educational phenomenon
    (Popham, 1993)
   Gathering evidence to determine how
    well the innovation is “working”.
   What does the grant say about the
    evaluation plan?
Foci of the Bush Grant Course-level
Evaluation
   Student Outcomes
   Learning Process:
            Student Engagement
            Reflective Learning
            Responsible Learning
The Bush grant re: Evidence
 Grades—ABC and DFW rates (outcomes).
 Faculty Reflection Logs and conversations
  with consultants (process)
 Surveys e.g. compare engagement across
  groups: with/without innovation and/or:
 Surveys to compare across time, e.g.
  before/after the innovation.
Survey Items: Closed-ended (Examples)
   Student Engagement
   Mostly, I come to class because I want the
    certificate/degree.
   Most of the time, I enjoy this class.

   Reflective Learning
   I enjoy applying theories or concepts from this class
    to new situations.
   I sometimes find myself thinking about the lecture
    after it has ended.

   Responsible Learning
   I usually come to class prepared.
   I usually review my lecture notes after class.
Survey Items: Open-ended
   What are the most important benefits of
    this innovation for you?
   What are the drawbacks?
   What kind of problems have you had
    with the technology? Be specific.
   What suggestions do you have for
    improvement?
Tailoring the Evaluation Plan to
Meet Your Needs
 What do you want to know?
 Collect the kind of evidence to find out what
  you want to know, for example:
 Business: experimental design with two
  treatment groups and long pre- and post-
  intervention surveys.
 Architecture: a series of short surveys,
  interviews and observations, then analyze
  them for recurring themes.
Optional Evaluation Methods
   Scoring Rubrics for student assignments
   Class Observation: take notes, analyze them
    for themes, can inform surveys.
   Narrative: To learn about learners’ reflective
    practices. Triangulation is important.
   Interviews: Open-ended items, can deviate,
    tape-record and analyze for themes.
   Focus groups: Validity checks.
Implementing the Plan
   Decide what you want to know
   Choose your preferred methods/tools
   Write surveys and/or interview protocols
   Collect data
   Data entry: training?
   Data analysis
Tools: Scoring Rubrics
   “Scoring rubrics…guide the analysis of the
    products and/or processes of students'
    efforts…[and] provide a description of what is
    expected at each score level.” (Moskal, 2000)

   a systematic way of evaluating qualitative
    data; a way of reducing subjectivity
Tools: Scoring Rubrics
Development process:

 break general concepts down into evaluation
  criteria
 develop descriptions of degrees to which, and
  ways in which, the criteria can be satisfied
 conduct pilot testing on sample data
Tools: Surveys

Survey questions should:
   be interpreted by respondents in the same
    way;
   ask for information that respondents are
    able to provide;
   provide information that is interpretable by
    you.
 Tools: Questionnaires
 Time estimates: How many hours per day
   do you typically study?

 Less than 1 hour
 1 – 1.5 hours
 1.5 – 2 hours
 2 - 2.5 hours          Less than 2.5 hours
 More than 2.5 hours    2.5 – 3 hours
                         3 – 3.5 hours
                         3.5 – 4 hours
                         More than 4 hours
Tools: Questionnaires
Scale issues: How would you rate your
 instructor’s knowledge of the subject
 matter of this class?

 poor
 fair
 adequate
 good
 very good
 exceptional
Tools: Questionnaires
Scale issues: How effective were the
 small group discussions in helping you
 to learn the course material?

 very effective
 effective
 don’t know
 ineffective
 very ineffective
Tools: Questionnaires
Item order effects: Which of the following
  activities helped you to learn the course
  material?
 Lectures
 Large group discussions
 Small group discussions
 Group assignments
 Course readings
 Studying for quizzes
Tools: Questionnaires
Open versus closed-ended questions:
 How easy or difficult to use did you find
 each of the components of our course
 website?
Tools: Questionnaires
Open versus closed-ended questions:
 How easy or difficult to use did you find
 these components of our course website?

Discussions              very easy
Quizzes                  easy
Content modules          difficult
Audio files              very difficult
PowerPoint files
etc.
Tools: Questionnaires
Interpretability: Which of these statements
  best describes your professor?

 very good teacher, but not very approachable
 about the same on approachability and
 teaching quality
 very approachable, but not a very good teacher
 a born teacher
 extremely approachable

								
To top