Evaluating Community-based
Initiatives
May 3, 2005 12:00 noon CST
Holly Ruch-Ross, ScD
David Keller, MD FAAP
Thomas Young, MD FAAP
Declaration of Vested Interest
Holly Ruch-Ross has no interests to
declare
David Keller has no interests to declare
Thomas Young has no interests to
declare
May 3, 2005 American Academy of Pediatrics
Evaluation Overview
Holly Ruch-Ross, ScD
Presentation Objectives
At the completion of the presentation, the
listener will be able to:
1. Identify the distinguishing characteristics of
evaluation research and the benefits of
conducting evaluation.
2. Write clear objectives.
3. Identify significant evaluation challenges
and strategies to address them.
May 3, 2005 American Academy of Pediatrics
Defining Evaluation
Research: to conduct a careful, patient,
systematic, diligent inquiry or examination in
some field of knowledge, to establish facts or
principles; to laboriously or continuously
search after truth.
Evaluate: to determine the worth of; to find
the amount or value of; to appraise.
May 3, 2005 American Academy of Pediatrics
Defining Evaluation
Evaluation is action research, intended to
provide information that is useful for:
Program development and
improvement
Program replication
Resource allocation
Policy decisions.
May 3, 2005 American Academy of Pediatrics
Reasons to Evaluation
Checking Your Process: Are you doing what you
said you would do?
Determining Your Impact: Are you having the
desired effect in the target population?
Creating a Fan Base: Can you generate
information and evidence to share with funders and
other stakeholders?
Replication Justification: Is there evidence to
support replication of this program?
Effective Documentation: Can you collect
information to support your program and meet
funder or other requirements?
May 3, 2005 American Academy of Pediatrics
The Evaluation Cycle
START
Plan program and
evaluation
Adjust program; Implement program and
Refine evaluation begin to collect
evaluative data
Review data.
Are you doing what you
planned? Having the
intended effect?
May 3, 2005 American Academy of Pediatrics
Types of Evaluation
Process Outcome
Is the program being Is the program
implemented the having the intended
way it was designed? effect?
May 3, 2005 American Academy of Pediatrics
Process Evaluation Information
Needs
Describe the program and
implementation, who participates in the
program, what services are received.
Information such as number served,
patient characteristics, number of
contacts with a program, number of
trainings, number of referrals.
May 3, 2005 American Academy of Pediatrics
Outcome Evaluation Information
Needs
Detect whether the intervention made a
difference, what changes can be
measured (knowledge, attitude,
behavior, health status, incidence,
prevalence)
Longer term outcomes may need to be
assessed using shorter term indicators.
May 3, 2005 American Academy of Pediatrics
Outcomes and Indicators:
Examples
Outcome Indicator
Prevent pregnancy Contraceptive compliance
High school graduation School attendance, GPA
Reduce disease incidence Immunization rate
Reduce clinical depression CES-D score
Prevent dental caries Plaque index
May 3, 2005 American Academy of Pediatrics
Goals and Objectives
Goal: broad statement of what the
program would like to accomplish for a
specific target population.
Objective: measurable step toward the
achievement of a goal.
Who will do
What by
When
May 3, 2005 American Academy of Pediatrics
A GOOD OBJECTIVE IS SMART:
Specific
Measurable
Achievable
Realistic for the program
Time specific
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Finding
the Right Tools
Tools need to measure the right construct.
Ideally, tools should be pre-existing and well-
established (valid, reliable and standardized).
Tools must be appropriate for the target
population (in terms of age, culture, language,
literacy, other issues).
Tools must be easy to administer in the setting.
When existing tools are used, they must be
readily available, affordable, and supported by
the author.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges:
Designing Your Own Tools
Adapt an existing tool to be more appropriate
for target population
Review literature
Talk to other grantees
Talk to others with ideas about what you
should ask about: experts, staff, recipients of
services
Pilot test tools with representatives of the
target population.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Using
Qualitative Data
Gain insight into feelings, attitudes, opinions
and motivations.
Study selected issues in depth and detail.
Gather the broadest response possible
without predetermined categories.
Gain rich information about a small number
of people and cases.
Put a human face on the program.
May 3, 2005 American Academy of Pediatrics
Most Common Qualitative Data
Collection Strategies
In-depth interview: usually one-on-one
structured conversation with a person who
has important information about the
program.
Focus group: professionally facilitated,
focused discussion among selected
individuals about program issues. Not all
discussion groups are focus groups, but they
may still be useful.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Data
Collection
The sophistication of data collection should be
appropriate for the scale of the project.
Plan data collection up front, including who,
what and when.
Have a system in place for tracking participants
(particularly if follow up is planned).
Identify the staff person responsible for data
handling.
Protect participant confidentiality.
Do not collect information that you will not use.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Data
Management and Analysis
Budget for expenses associated with data
entry and analysis.
Have a strategy for data management up
front. Begin data entry immediately.
Remember that data analysis should follow
directly from the questions you are trying to
answer about your intervention.
Know what comparative information may be
available.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges:
Technology
Involve technology as early as possible in the
planning process.
Keep it as simple as possible.
Choose technology that will be around at least
five years, and that is compatible with your
current system.
Make certain that your choices match the
technical support that is available to you.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Getting
Help From an Evaluator
Specific evaluation training and applied
research experience
Experience in a human service setting
Professional perspective and
methodological orientation
Interpersonal skills
Self interest (i.e., can he/she put yours
first?!)
May 3, 2005 American Academy of Pediatrics
CATCH Grant Evaluation
Example
Thomas Young, MD FAAP
Presentation objectives
At the completion of the presentation, the
listener will be able to:
Describe a CATCH Project
Describe project’s evaluation plan
Describe project’s data collection and
management techniques
May 3, 2005 American Academy of Pediatrics
Keys to Successful Evaluation
Develop objectives that directly
contribute to your goals for project
Spend time and thought on objectives,
evaluation will follow
Objectives must be
Relevant
Measurable
Feasible/doable
May 3, 2005 American Academy of Pediatrics
CATCH Project Description
CATCH Implementation Grant 2003
Goal: To improve screening, provider
knowledge and access to mental health
care for children and families in a
primary care and continuity clinic setting
This is broad goal for a small grant
We had to develop objectives that were
relevant, measurable, and doable.
May 3, 2005 American Academy of Pediatrics
Developing CATCH Objectives
Process Objective 1: 80%
Implementation of Bright Futures
mental health screening: PEDS, Ped
Symptom Checklist, Edinburgh
Postnatal Depression Screen
Evaluation Method: Chart audit
Findings: Chart audit documented
successful implementation of screenings
in 90% charts
May 3, 2005 American Academy of Pediatrics
Developing CATCH Objectives
Process Objective 2: To provide 120
mental health consultant visits
Evaluation Method: A data file was
developed and completed by the Mental
Health consultant to document the
number and type of mental health
encounters.
Findings: 297 MH sessions
May 3, 2005 American Academy of Pediatrics
Data File and Report
May 3, 2005 American Academy of Pediatrics
Developing CATCH Objectives
Process Objective 3: To provide case
management of mental health referrals
Evaluation Method: Data base
developed with Excel to track referrals,
patient contacts
Findings: 83 children had outside
referrals and case management
documented.
May 3, 2005 American Academy of Pediatrics
Developing CATCH Objectives
Outcome Objective 1: To improve provider
confidence in managing behavioral problems
in children.
Evaluation Method: Development of a
pre/post survey of the providers in clinic on
confidence in management of common
behavior problems (no existing survey tool
found)
Findings: Confidence increased from 40% to
67% in 6 months. Adequate MH resources
improved from 54% to 95%.
May 3, 2005 American Academy of Pediatrics
May 3, 2005 American Academy of Pediatrics
Developing CATCH Objectives
Outcome Objective 2: Identify
resources to sustain mental health
provider
Evaluation: Was funding found to
continue mental health project
Findings: Primary Care Center
funding, grants, and partnership with
community mental health agency
continued and expanded MH services.
May 3, 2005 American Academy of Pediatrics
Summary of CATCH Evaluation
Evaluations are different from scientific
research
Good objectives lead to good evaluations
If possible find reliable tools to measure
outcomes, if not use creative common sense.
Accurately track data pre-grant to the end
Keep it relevant but simple, with larger follow-
up grants hire an evaluator!
May 3, 2005 American Academy of Pediatrics
Healthy Tomorrows Grant
Evaluation Example
David Keller, MD FAAP
Presentation objectives
At the completion of the presentation, the
listener will be able to:
Describe a Healthy Tomorrows Project
Describe project’s evaluation plan
Describe project’s data collection and
management techniques
May 3, 2005 American Academy of Pediatrics
Healthy Tomorrows Project Description:
Family Advocates of Central Massachusetts
A legal-medical collaborative
Focused advocacy on legal issues likely to affect
child health outcomes
Three objectives:
Patient identification and referral from practices
Provider training on legal issues and how they
affect health
Advice and counsel for patients and families in
need
May 3, 2005 American Academy of Pediatrics
A medical-legal collaborative
Legal Assistance Corporation
of Central Massachusetts
and five practices in Central
Massachusetts
• Worcester
• Pediatric Primary Care
• Family Health Center
• Webster
• South County Pediatrics
• Milford
• Milford Pediatrics
• Fitchburg
• CHC Family Health
Center
May 3, 2005 American Academy of Pediatrics
Focused Advocacy
• Housing stability (e.g. lead poisoning,
homelessness, mold and allergens)
• Financial security (e.g. disability benefits, food
stamps, Medicaid)
• Dignity and safety (e.g. immigration status,
domestic violence)
• Access to services (e.g. medical, dental,
mental health, special education services)
May 3, 2005 American Academy of Pediatrics
Model of Intervention
May 3, 2005 American Academy of Pediatrics
Process Evaluation
Based on program objectives:
Are the practices identifying and referring patients
in need of legal services to the program?
Are we able to train medical providers on legal
issues and how they affect health?
Have we provided advice and counsel for patients
and families in need of legal services?
Underlying question:
What can we do better?
May 3, 2005 American Academy of Pediatrics
Process Objective 1:
Are the practices identifying and referring patients in
need of legal services to the program?
Findings in Year 1: 76
referrals
Came from all five sites.
Usually associated with
trainings.
Office hours did not
generate referrals.
Lessons learned:
Time with providers =
more referrals.
Screening instrument
needed.
May 3, 2005 American Academy of Pediatrics
Process Objective 2:
Are we able to train medical providers on
legal issues and how they affect health?
Year 1: 42 trainings
Webster: 9 Satisfaction?
Worcester FHC: 7
Worcester Peds Assoc: 15 1 2 3 4 5
Fitchburg: 6
Milford: 5 Values? 0 0 0 7 9
Feedback
“Code card is great” Useful? 0 0 0 5 12
“More time on cases”
Lessons learned: Materials? 0 0 1 2 12
Providers want practical
tools
Providers want case-based Trainer? 0 0 0 5 11
training
May 3, 2005 American Academy of Pediatrics
Process Objective 3:
Have we provided advice and counsel for patients and
families in need of legal services?
Findings: 58 cases closed for
50 clients.
Counsel and advice in 21
cases.
Brief service in 26 cases.
Full representation in 10
cases.
Referred 1 clients to another
agency.
Lessons learned:
Case mix more varied than
expected based on needs
assessment.
Less full representation
needed than we expected.
May 3, 2005 American Academy of Pediatrics
Outcome Objectives: ???
Practice Outcome
Legal Outcome
Health Outcome
System Outcome
May 3, 2005 American Academy of Pediatrics
Legal Outcome:
Evaluation Methods
Was the case resolved in
favor of the client?
LACCM database
Is that client better off
than before in terms of:
• Housing stability
• Financial security
• Dignity and safety
• Access to services
LACCM outcome tool
From Nasdor, Moving Beyond Funder-Driven
Outcome Measures.
May 3, 2005 American Academy of Pediatrics
Legal Outcome: Problems
What is a case?
• Need to understand “jargon” of legal services.
What is an outcome?
• Need to integrate public health and legal data
• Need an outcome for “brief intervention”
Validity of outcome measure?
• Need to assess inter-observer reliability
May 3, 2005 American Academy of Pediatrics
Practice Outcome
How have the
attorneys changed
their practice?
How have the
physicians changed
their practice?
Is there a more
collaborative
relationship?
May 3, 2005 American Academy of Pediatrics
Practice Outcome: Problems
Counting:
No ready made database
Evaluation:
Utility of self-reported comments
Data entry
Next steps:
Commitment to change
Focus groups on practice change
May 3, 2005 American Academy of Pediatrics
Health Outcome
What is health?
Individual or group?
Compared to what?
Measured how?
What about the
confounders?
May 3, 2005 American Academy of Pediatrics
System Outcome
Changes in policy
Changes in the law
Changes in
legislation
Changes in the
balance of power
May 3, 2005 American Academy of Pediatrics
Evaluation: Lessons Learned
Build evaluation into the program
design
Cross-disciplinary communication is
essential to evaluating multidisciplinary
collaborations
Measure the easy stuff first, but don’t
give up on the hard stuff
May 3, 2005 American Academy of Pediatrics
Where to find more
information…
For an audio file of the call including question
and answer period, go to
http://www.aap.org/commpeds/resources/teleconferences.htm
To submit additional questions, please email
agramiak@aap.org. Your question will be
forwarded to one of the teleconference
speakers.
Please fill out the evaluation form and fax
back to Anne Gramiak at 847-434-8000. A
form must be completed to receive continuing
education credits.
May 3, 2005 American Academy of Pediatrics
Program Staff Contact
Information
CATCH
Lisa Marth: lmarth@aap.org
Kathy Kocvara: kkocvara@aap.org
Aleksandra Stolic: astolic@aap.org
Healthy Tomorrows
Nicole Miller: nmiller@aap.org
Anne Gramiak: agramiak@aap.org
CPTI
Alanna Bailey: abailey@aap.org
Trisha Calabrese: tcalabrese@aap.org
Medical Home
Lauri Levin: llevin@aap.org
May 3, 2005 American Academy of Pediatrics