Docstoc

Evaluation Evaluating Community based

Document Sample
Evaluation Evaluating Community based Powered By Docstoc
					Evaluating Community-based
         Initiatives

   May 3, 2005 12:00 noon CST

      Holly Ruch-Ross, ScD
     David Keller, MD FAAP
    Thomas Young, MD FAAP
Declaration of Vested Interest
 Holly Ruch-Ross has no interests to
  declare
 David Keller has no interests to declare
 Thomas Young has no interests to
  declare



May 3, 2005    American Academy of Pediatrics
Evaluation Overview

  Holly Ruch-Ross, ScD
Presentation Objectives
At the completion of the presentation, the
   listener will be able to:
1. Identify the distinguishing characteristics of
   evaluation research and the benefits of
   conducting evaluation.
2. Write clear objectives.
3. Identify significant evaluation challenges
   and strategies to address them.

May 3, 2005      American Academy of Pediatrics
Defining Evaluation
Research: to conduct a careful, patient,
 systematic, diligent inquiry or examination in
 some field of knowledge, to establish facts or
 principles; to laboriously or continuously
 search after truth.

Evaluate: to determine the worth of; to find
 the amount or value of; to appraise.

May 3, 2005     American Academy of Pediatrics
Defining Evaluation
   Evaluation is action research, intended to
    provide information that is useful for:
       Program development and
        improvement
       Program replication

       Resource allocation

       Policy decisions.


May 3, 2005       American Academy of Pediatrics
Reasons to Evaluation
   Checking Your Process: Are you doing what you
    said you would do?
   Determining Your Impact: Are you having the
    desired effect in the target population?
   Creating a Fan Base: Can you generate
    information and evidence to share with funders and
    other stakeholders?
   Replication Justification: Is there evidence to
    support replication of this program?
   Effective Documentation: Can you collect
    information to support your program and meet
    funder or other requirements?

May 3, 2005         American Academy of Pediatrics
The Evaluation Cycle
              START

                            Plan program and
                               evaluation


      Adjust program;                                 Implement program and
      Refine evaluation                                   begin to collect
                                                          evaluative data
                               Review data.
                          Are you doing what you
                           planned? Having the
                              intended effect?


May 3, 2005                 American Academy of Pediatrics
Types of Evaluation
 Process                           Outcome


   Is the program being               Is the program
    implemented the                     having the intended
    way it was designed?                effect?




May 3, 2005       American Academy of Pediatrics
Process Evaluation Information
Needs
   Describe the program and
    implementation, who participates in the
    program, what services are received.

   Information such as number served,
    patient characteristics, number of
    contacts with a program, number of
    trainings, number of referrals.

May 3, 2005     American Academy of Pediatrics
Outcome Evaluation Information
Needs
   Detect whether the intervention made a
    difference, what changes can be
    measured (knowledge, attitude,
    behavior, health status, incidence,
    prevalence)

   Longer term outcomes may need to be
    assessed using shorter term indicators.

May 3, 2005      American Academy of Pediatrics
Outcomes and Indicators:
Examples
Outcome                              Indicator
Prevent pregnancy                    Contraceptive compliance

High school graduation               School attendance, GPA

Reduce disease incidence             Immunization rate

Reduce clinical depression           CES-D score

Prevent dental caries                Plaque index

May 3, 2005         American Academy of Pediatrics
Goals and Objectives
 Goal: broad statement of what the
  program would like to accomplish for a
  specific target population.
 Objective: measurable step toward the
  achievement of a goal.
    Who will do

    What by

    When


May 3, 2005   American Academy of Pediatrics
A GOOD OBJECTIVE IS SMART:

 Specific
 Measurable
 Achievable
 Realistic for the program
 Time specific




May 3, 2005   American Academy of Pediatrics
Evaluation Challenges: Finding
the Right Tools
   Tools need to measure the right construct.
   Ideally, tools should be pre-existing and well-
    established (valid, reliable and standardized).
   Tools must be appropriate for the target
    population (in terms of age, culture, language,
    literacy, other issues).
   Tools must be easy to administer in the setting.
   When existing tools are used, they must be
    readily available, affordable, and supported by
    the author.

May 3, 2005         American Academy of Pediatrics
Evaluation Challenges:
Designing Your Own Tools
   Adapt an existing tool to be more appropriate
    for target population
   Review literature
   Talk to other grantees
   Talk to others with ideas about what you
    should ask about: experts, staff, recipients of
    services
   Pilot test tools with representatives of the
    target population.

May 3, 2005        American Academy of Pediatrics
Evaluation Challenges: Using
Qualitative Data
   Gain insight into feelings, attitudes, opinions
    and motivations.
   Study selected issues in depth and detail.
   Gather the broadest response possible
    without predetermined categories.
   Gain rich information about a small number
    of people and cases.
   Put a human face on the program.

May 3, 2005        American Academy of Pediatrics
Most Common Qualitative Data
Collection Strategies
   In-depth interview: usually one-on-one
    structured conversation with a person who
    has important information about the
    program.
   Focus group: professionally facilitated,
    focused discussion among selected
    individuals about program issues. Not all
    discussion groups are focus groups, but they
    may still be useful.

May 3, 2005       American Academy of Pediatrics
Evaluation Challenges: Data
Collection
   The sophistication of data collection should be
    appropriate for the scale of the project.
   Plan data collection up front, including who,
    what and when.
   Have a system in place for tracking participants
    (particularly if follow up is planned).
   Identify the staff person responsible for data
    handling.
   Protect participant confidentiality.
   Do not collect information that you will not use.

May 3, 2005         American Academy of Pediatrics
Evaluation Challenges: Data
Management and Analysis
   Budget for expenses associated with data
    entry and analysis.
   Have a strategy for data management up
    front. Begin data entry immediately.
   Remember that data analysis should follow
    directly from the questions you are trying to
    answer about your intervention.
   Know what comparative information may be
    available.

May 3, 2005        American Academy of Pediatrics
Evaluation Challenges:
Technology
   Involve technology as early as possible in the
    planning process.
   Keep it as simple as possible.
   Choose technology that will be around at least
    five years, and that is compatible with your
    current system.
   Make certain that your choices match the
    technical support that is available to you.

May 3, 2005       American Academy of Pediatrics
Evaluation Challenges: Getting
Help From an Evaluator
 Specific evaluation training and applied
  research experience
 Experience in a human service setting
 Professional perspective and
  methodological orientation
 Interpersonal skills
 Self interest (i.e., can he/she put yours
  first?!)

May 3, 2005    American Academy of Pediatrics
CATCH Grant Evaluation
      Example
  Thomas Young, MD FAAP
Presentation objectives
At the completion of the presentation, the
  listener will be able to:
 Describe a CATCH Project
 Describe project’s evaluation plan
 Describe project’s data collection and
  management techniques


May 3, 2005   American Academy of Pediatrics
Keys to Successful Evaluation
 Develop objectives that directly
  contribute to your goals for project
 Spend time and thought on objectives,
  evaluation will follow
 Objectives must be
    Relevant

    Measurable

    Feasible/doable



May 3, 2005   American Academy of Pediatrics
              CATCH Project Description
 CATCH Implementation Grant 2003
 Goal: To improve screening, provider
  knowledge and access to mental health
  care for children and families in a
  primary care and continuity clinic setting
 This is broad goal for a small grant
 We had to develop objectives that were
  relevant, measurable, and doable.


May 3, 2005         American Academy of Pediatrics
Developing CATCH Objectives
 Process Objective 1: 80%
  Implementation of Bright Futures
  mental health screening: PEDS, Ped
  Symptom Checklist, Edinburgh
  Postnatal Depression Screen
 Evaluation Method: Chart audit
 Findings: Chart audit documented
  successful implementation of screenings
  in 90% charts
May 3, 2005   American Academy of Pediatrics
Developing CATCH Objectives
 Process Objective 2: To provide 120
  mental health consultant visits
 Evaluation Method: A data file was
  developed and completed by the Mental
  Health consultant to document the
  number and type of mental health
  encounters.
 Findings: 297 MH sessions



May 3, 2005   American Academy of Pediatrics
Data File and Report




May 3, 2005   American Academy of Pediatrics
Developing CATCH Objectives

 Process Objective 3: To provide case
  management of mental health referrals
 Evaluation Method: Data base
  developed with Excel to track referrals,
  patient contacts
 Findings: 83 children had outside
  referrals and case management
  documented.

May 3, 2005    American Academy of Pediatrics
Developing CATCH Objectives
   Outcome Objective 1: To improve provider
    confidence in managing behavioral problems
    in children.
   Evaluation Method: Development of a
    pre/post survey of the providers in clinic on
    confidence in management of common
    behavior problems (no existing survey tool
    found)
   Findings: Confidence increased from 40% to
    67% in 6 months. Adequate MH resources
    improved from 54% to 95%.

May 3, 2005       American Academy of Pediatrics
May 3, 2005   American Academy of Pediatrics
Developing CATCH Objectives
   Outcome Objective 2: Identify
    resources to sustain mental health
    provider
   Evaluation: Was funding found to
    continue mental health project
   Findings: Primary Care Center
    funding, grants, and partnership with
    community mental health agency
    continued and expanded MH services.
May 3, 2005    American Academy of Pediatrics
Summary of CATCH Evaluation
   Evaluations are different from scientific
    research
   Good objectives lead to good evaluations
   If possible find reliable tools to measure
    outcomes, if not use creative common sense.
   Accurately track data pre-grant to the end
   Keep it relevant but simple, with larger follow-
    up grants hire an evaluator!

May 3, 2005         American Academy of Pediatrics
Healthy Tomorrows Grant
  Evaluation Example
   David Keller, MD FAAP
Presentation objectives
At the completion of the presentation, the
  listener will be able to:
 Describe a Healthy Tomorrows Project
 Describe project’s evaluation plan
 Describe project’s data collection and
  management techniques


May 3, 2005   American Academy of Pediatrics
Healthy Tomorrows Project Description:
Family Advocates of Central Massachusetts

   A legal-medical collaborative
         Focused advocacy on legal issues likely to affect
          child health outcomes
   Three objectives:
         Patient identification and referral from practices
         Provider training on legal issues and how they
          affect health
         Advice and counsel for patients and families in
          need


May 3, 2005             American Academy of Pediatrics
A medical-legal collaborative
Legal Assistance Corporation
  of Central Massachusetts
  and five practices in Central
  Massachusetts
• Worcester
   • Pediatric Primary Care
   • Family Health Center
• Webster
   • South County Pediatrics
• Milford
   • Milford Pediatrics
• Fitchburg
   • CHC Family Health
      Center

May 3, 2005            American Academy of Pediatrics
Focused Advocacy
• Housing stability (e.g. lead poisoning,
      homelessness, mold and allergens)
• Financial security (e.g. disability benefits, food
  stamps, Medicaid)
• Dignity and safety (e.g. immigration status,
      domestic violence)
• Access to services (e.g. medical, dental,
      mental health, special education services)


May 3, 2005        American Academy of Pediatrics
Model of Intervention




May 3, 2005   American Academy of Pediatrics
Process Evaluation
   Based on program objectives:
         Are the practices identifying and referring patients
          in need of legal services to the program?
         Are we able to train medical providers on legal
          issues and how they affect health?
         Have we provided advice and counsel for patients
          and families in need of legal services?
     Underlying question:
         What can we do better?


May 3, 2005             American Academy of Pediatrics
Process Objective 1:
Are the practices identifying and referring patients in
need of legal services to the program?
 Findings in Year 1: 76
  referrals
         Came from all five sites.
         Usually associated with
          trainings.
         Office hours did not
          generate referrals.
   Lessons learned:
         Time with providers =
          more referrals.
         Screening instrument
          needed.


May 3, 2005                American Academy of Pediatrics
Process Objective 2:
Are we able to train medical providers on
legal issues and how they affect health?
   Year 1: 42 trainings
         Webster: 9                                         Satisfaction?
         Worcester FHC: 7
         Worcester Peds Assoc: 15                              1   2   3    4   5
         Fitchburg: 6
         Milford: 5                            Values?         0   0   0    7   9
   Feedback
         “Code card is great”                  Useful?         0   0   0    5   12
         “More time on cases”
   Lessons learned:                          Materials?        0   0   1    2   12
         Providers want practical
          tools
         Providers want case-based             Trainer?        0   0   0    5   11
          training
May 3, 2005                 American Academy of Pediatrics
Process Objective 3:
Have we provided advice and counsel for patients and

families in need of legal services?
   Findings: 58 cases closed for
    50 clients.
         Counsel and advice in 21
         cases.
        Brief service in 26 cases.
        Full representation in 10
         cases.
        Referred 1 clients to another
         agency.
   Lessons learned:
        Case mix more varied than
         expected based on needs
         assessment.
        Less full representation
         needed than we expected.
May 3, 2005                  American Academy of Pediatrics
Outcome Objectives: ???




              Practice Outcome

               Legal Outcome

              Health Outcome

              System Outcome
May 3, 2005     American Academy of Pediatrics
Legal Outcome:
Evaluation Methods
Was the case resolved in
 favor of the client?
    LACCM database
Is that client better off
   than before in terms of:
•   Housing stability
•   Financial security
•   Dignity and safety
•   Access to services
LACCM outcome tool
From Nasdor, Moving Beyond Funder-Driven
   Outcome Measures.



May 3, 2005                  American Academy of Pediatrics
Legal Outcome: Problems

What is a case?
• Need to understand “jargon” of legal services.

What is an outcome?
• Need to integrate public health and legal data
• Need an outcome for “brief intervention”

Validity of outcome measure?
• Need to assess inter-observer reliability


May 3, 2005          American Academy of Pediatrics
Practice Outcome
   How have the
    attorneys changed
    their practice?
   How have the
    physicians changed
    their practice?
   Is there a more
    collaborative
    relationship?
May 3, 2005      American Academy of Pediatrics
Practice Outcome: Problems
   Counting:
      No ready made database

   Evaluation:
      Utility of self-reported comments

      Data entry

   Next steps:
      Commitment to change

      Focus groups on practice change



May 3, 2005       American Academy of Pediatrics
Health Outcome
   What is health?
   Individual or group?
   Compared to what?
   Measured how?
   What about the
    confounders?



May 3, 2005       American Academy of Pediatrics
System Outcome

   Changes in policy
   Changes in the law
   Changes in
    legislation
   Changes in the
    balance of power


May 3, 2005       American Academy of Pediatrics
Evaluation: Lessons Learned
 Build evaluation into the program
  design
 Cross-disciplinary communication is
  essential to evaluating multidisciplinary
  collaborations
 Measure the easy stuff first, but don’t
  give up on the hard stuff

May 3, 2005    American Academy of Pediatrics
Where to find more
information…
   For an audio file of the call including question
    and answer period, go to
    http://www.aap.org/commpeds/resources/teleconferences.htm
   To submit additional questions, please email
    agramiak@aap.org. Your question will be
    forwarded to one of the teleconference
    speakers.
   Please fill out the evaluation form and fax
    back to Anne Gramiak at 847-434-8000. A
    form must be completed to receive continuing
    education credits.

May 3, 2005             American Academy of Pediatrics
Program Staff Contact
Information
         CATCH
            Lisa Marth: lmarth@aap.org

            Kathy Kocvara: kkocvara@aap.org

            Aleksandra Stolic: astolic@aap.org

         Healthy Tomorrows
            Nicole Miller: nmiller@aap.org

            Anne Gramiak: agramiak@aap.org

         CPTI
            Alanna Bailey: abailey@aap.org

            Trisha Calabrese: tcalabrese@aap.org

         Medical Home
            Lauri Levin: llevin@aap.org



May 3, 2005               American Academy of Pediatrics

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:6
posted:10/19/2011
language:English
pages:54