Assessing

Document Sample
Assessing Powered By Docstoc
					Assessing K-12 Service-Learning
            Impacts and Quality
                      Shelley H. Billig
                 RMC Research Corporation
                        John Spence
                TX Center for Service-Learning
Overview

• Logic Models and Theory of
  Change
• Measuring Impact
  – Academic impact
  – Civic impact
  – Personal/social impacts
  – Others
• Measuring Quality
Logic Models

• Shows how and why the program
  will work;
• Connects program activities to
  program outcomes through outputs:
  – e.g., activity (input)  Increased
    student engagement in schools
    (output)  Increased student
    academic achievement (outcome).
TX K-12 Learn and Serve Logic
Model
Context (e.g., student, K-12 campus, and community characteristics, support for service-learning, accountability pressures)


     Partner Contributions                     Service-Learning                           Student Outcomes
    LEA and campus investment
                                                  Experience                         Academic performance
    Contributions by community                                                     Civic knowledge, skills, dispositions,
      partners                                   Frequency and type                  and behavior
                                                   of project,                       S.T.A.R.S. outcomes (e.g.,
                                                   implementation                      leadership skills, ethic of service,
                                                   quality, linkage to                 value of lifelong citizenship)
                                                   subject matter,                   School engagement, school and
                                                   effectiveness of                    community attachment
                                                   partnerships
     Support for Service-                        Number and type of
         Learning                                  students
    Training/technical assistance                participating,                 Partner/Community Outcomes
    Recognition of service-learning              intensity and
                                                   duration of                       Benefits for K-12 campuses, LEAs,
      as improvement strategy                                                          community partners
    Teacher buy-in                               participation
                                                                                     Social capital/Community capacity



          Efforts to Promote, Sustain, and Institutionalize Service-Learning Programming
Try It!
• Start with your activities – what will
  you do?
• Specify your goals – what will you
  accomplish?
• Identify those factors that will
  influence results, both positively
  and negatively. Include
  demographics, context,
  implementation issues
Logic Model Use

• Once the logic model has been
  created with stakeholders, review it.
• Use the logic model to design or
  revamp the evaluation.
• Use the constructs identified in the
  logic model to guide the
  development of measures
  (instrumentation).
Logic Model Resources
• University of Wisconsin-Extension
  Program Development and
  Evaluation,
  www.uwex.edu/ces/pdande
• W.K. Kellogg Foundation,
  www.wkkf.org
  – Evaluation Handbook, 1998
  – Logic Model Development Guide,
    2001
Assessing Academic Impact

 – Where is service-learning likely to
   have the greatest academic impact?
   • Knowledge areas (general and specific)
   • Types of skills (general and specific)
   • Moderators


 – What measures are available?
Academic Skills
• Areas of likely impact
   – Higher order thinking skills such as problem solving,
     analysis, transfer of learning
   – Specific knowledge related to service activities, e.g.,
     measurement, writing, synthesizing materials,
     research
• What are the available measures?
   –   Rubrics
   –   Test scores
   –   Writing scores
   –   Problem solving essays
   –   Measures of academic engagement
   –   Absenteeism
   –   Surveys
   –   More!
Sample Measures

• Examine the samples
• What is the best use of each? By
  whom and for what purpose?
• What do you like and dislike about
  each?
Assessing Civic Impacts

• Where is service-learning likely to
  have the greatest civic impact?
  – Knowledge?
  – Skills?
  – Dispositions?
  – Attachment to community?
• Moderators
• What measures are available?
Civic Impacts
• Areas of likely impact
  –   Knowledge of community needs
  –   Skills for planning and implementing projects
  –   Taking responsibility
  –   Ethic of service
• Available measures
  –   Surveys
  –   Focus groups/interviews
  –   Hours of service
  –   Actual changes
Sample Measures

• Examine the samples
• What is the best use of each? By
  whom and for what purpose?
• What do you like and dislike about
  each?
Assessing Social/Personal
Impacts
 – Where is service-learning likely to
   have the greatest impact on
   social/personal areas?
    • Knowledge areas (general and specific)
    • Types of skills (general and specific)
    • Moderators


 – What measures are available?
Social/Personal Outcomes
• Areas of Likely Impact
  – Development of character traits such as
    responsibility, trust, and caring
  – Respect for diversity
  – More bonds with adults
  – More protective factors such as resilience
  – Greater self-efficacy
• Measures
  –   Surveys
  –   Focus groups/interviews
  –   Rubrics
  –   Self-assessments
Sample Measures

• Examine the samples
• What is the best use of each? By
  whom and for what purpose?
• What do you like and dislike about
  each?
Measuring Quality

Quality counts! Higher quality
 service-learning activities have
 stronger impacts.

Which are the critical factors of
 quality?
Quality Indicators That
Predicted Outcomes
• Direct contact with those being
  served
• Linkage with standards
• Character of reflection activities
• Intentionality
Sample Measures

• Examine the samples
• What do you like and dislike about
  each?

• How are these best used??
Characteristics of Good
Assessments
• Valid
• Reliable
• Objective
• Useful
• Items measure only one construct
  at a time (not double barrelled…)
• Related to the logic model or
  assessment questions
Open Discussion

• Questions and Answers
Resources
• National Service-Learning Clearinghouse
  www.servicelearning.org

• Billig, S. H., & Waterman, A. S., eds. (2003).
  Studying service-learning: Innovations in education
  research methodology. Mahwah, NJ: Lawrence
  Erlbaum Associates. Available at
  www.erlbaum.com

• Bringle, R. G., Phillips, M. A., & Hudson, M. (2004).
  The measure of service learning: Research scales
  to assess student experiences. Washington, DC:
  American Psychological Association. Available at
  www.apa.org/books

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:10
posted:11/1/2011
language:English
pages:22