Get documents about "PowerPoint Presentation
Document Sample
How to Focus and Measure Outcomes
Katherine Webb-Martinez
Central Coast and South Region
All Staff Conference
April 23, 2008
Workshop Desired Outcomes
An understanding of how basic logic
model techniques help define outcomes
and focus program evaluation
An understanding of options for data
collection methods to measure outcomes
Workshop Agenda
Introductions
Part 1: Logic Model
• Presentation – Logic Model helps with outcomes
• Activity – “Using a logic model to focus outcomes
evaluation”
Part 2: Measuring Outcomes
• Presentation - Review of Data Collection Options
• Activity – Methods Café
Close
Part 1:
Logic Model as Road Map
Where are you going?
How will you get there?
What will show that “If you don’t know where
you’ve arrived? you are going, how are
you gonna’ know when
you get there?”
Yogi Berra
Everyday example
H
U
N
G
R Get food Eat food Feel better
Y
University of Wisconsin-Extension, Program Development and Evaluation
Logical chain of connections showing
what the program is to accomplish
INPUTS OUTPUTS OUTCOMES
Program Activities Participation Learning Action Impacts
investments (Short- (Medium (Long-
term) -term) term)
What we What Who we
invest we do reach
What results
University of Wisconsin-Extension, Program Development and Evaluation
Outputs vs. outcomes
Example:
Number of patients discharged from state mental
hospital is an output.
Percentage of discharged who are capable of living
independently is an outcome
Not how many worms
the bird feeds its young,
but how well the fledgling flies
(United Way of America, 1999)
University of Wisconsin-Extension, Program Development and Evaluation
What do you Social-economic-
environmental impacts
want to know?
Source: Bennett and Rockwell, 1995,
Targeting Outcomes of Programs
Actions Changes in behaviors
and practices
Learning
Changes in knowledge, attitudes,
skills, aspirations
Reactions Degree of satisfaction with program; level of
interest; feelings toward activities, educational
methods
Participation Number and characteristics of people reached;
frequency and intensity of contact
University of Wisconsin-Extension, Program Development and Evaluation
Needs assessment: Process evaluation: Outcome evaluation: Impact evaluation:
What are the How is program To what extent are What are the net
characteristics, implemented? desired changes effects?
needs, priorities of occurring?
target population?
Are activities delivered
as intended? What are final
What are potential Who is benefiting / consequences?
barriers/facilitators? not benefiting?
Are participants being How?
reached as intended? To what extent can
What is most changes be
appropriate to do? What are unintended attributed to the
What are participant
reactions? outcomes? program?
University of Wisconsin-Extension, Program Development and Evaluation
ACTIVITY: PARENT EDUCATION PROGRAM
What do you (and others) want to know about the program?
INPUTS OUTPUTS OUTCOMES
Assess Parents Parents Reduced
Staff parent ed increase identify stress
programs knowledge of appropriate
child dev actions to
Parents take Improved
Design-
Money of 3-10 child-
deliver Parents better
evidence- year understanding parent
olds their own relations
based Parents use
Partners program attend parenting style effective
of 8 parenting Strong
practices families
sessions Parents gain
Research skills in new
ways to parent
Facilitate
support Parents gain
groups confidence in
their abilities
University of Wisconsin-Extension, Program Development and Evaluation
Logic Model Reflection
How does a logic model help focus on
what outcomes to measure and when
to measure them?
How might you use a logic model in
your own work?
Part 2: Measuring Outcomes
How will you answer your questions?
1. Decide on what evidence answers the questions
2. Determine sources of information
3. Choose data collection methods
Identify evidence
How will you know it when you see it?
What are the specific indicators that will
be measured?
- Often expressed as # or %
- Outputs and outcomes indicators
- Quantitative or qualitative
University of Wisconsin-Extension, Program Development and Evaluation
Logic model with Indicators for Outputs and Outcomes
Outputs Outcomes
Program Targeted Farmers Farm
Farmers practice new profitability
implemented farmers learn techniques increases
Number of Number and Number
Number and Number and
workshops percent of and percent
percent who percent
held farmers who
increase reporting
Quality of attending knowledge practice increased
workshops new profits;
techniques amount of
increase
University of Wisconsin-Extension, Program Development and Evaluation
Remember:
“Not
everything
that counts
can be
counted.”
9 (Quantity) Happy (Quality) Kids
Quantitative: numbers, breadth, generalizability
Qualitative: words, depth, specific
Possible Evaluation questions, indicators
Parents increase
knowledge of Reduced
Staff Develop parent child dev Parents identify stress
ed curriculum appropriate
Parents better actions to take
Money
understand their
own parenting Improved
Deliver series
Partners Parents of style child-
of 8 interactive Parents use
3-10 year parent
sessions effective
olds Parents gain relations
skills in new parenting
Research practices
Facilitate ways to parent
support groups
Strong
Parents gain families
confidence in
EVALUATION QUESTIONS their abilities
What amount How many Who/how many To what extent To what extent
To what extent
of $ and time sessions were attended/did not did knowledge did behaviors
is stress
were held? How attend? Did they and skills change? For
reduced?
invested? effectively? attend all sessions? increase? For whom? Why?
To what extent
#, quality of Supports groups? whom? Why? What else
are relations
support groups? Were they satisfied What else happened?
improved?
– why/why not? happened?
INDICATORS
# Staff # Sessions #,% #,% #,%
#,% attended demonstrating demonstrating demonstrating
$ used held per session increased changes improvements
# partners Quality knowledge/skill
Certificate of Types of Types of
criteria completion Additional changes improvements
outcomes
University of Wisconsin-Extension, Program Development and Evaluation
Determine Sources of Information
Program participants
Existing data
• Program records, attendance logs, etc
• Pictures, charts, maps, pictorial records
Others/Non-participants
• Key informants
• Funders
• Collaborators
• Etc.
Decide on Data Collection Methods
Survey Diaries, journals
Interview Case study
Test Photography,
Observation video
Group techniques Document review
Portfolio review Expert or peer
review
University of Wisconsin-Extension, Program Development and Evaluation
Data collection plan template
Questions Indicators Data collection
Sources Methods Sample Timing
University of Wisconsin-Extension, Program Development and Evaluation
When choosing your methods consider:
Purpose
Participants
Resources Available
Some things to remember…
There is no one right method of
collecting data
Each has a purpose, advantages and
challenges
The goal is to obtain trustworthy,
authentic and credible evidence
Often a mix of methods is preferred
Are the data reliable and valid?
Validity: Are you measuring what you
think you’re measuring?
Reliability: If something was measured
again using the same instrument, would
it produce the same (or near the same)
results?
Logic model and areporting
Components of Success Story
University of Wisconsin-Extension, Program Development and Evaluation
Methods Cafe
What approaches to measure outcomes
work well for CE programs and why?
Is there anything you’ve tried that you would
not recommend?
Methods Reflection
Given what we have discussed, what
might you change or do differently with
the methods that work for you?
Are there any other, new methods that
might work for you and your clientele?
