Follow up Working a Case Example Sept - Immunization Program Evaluation (IPE)

Practical Program Evaluation—Follow Up: Working a Case Example By: Thomas J. Chapel, Office of Workforce and Career Development Betty Apt, Division of STD Prevention Centers for Disease Control and Prevention Today…  Review main points from overview by applying to a case example a component to evaluate  Identifying stakeholders that matter most  Developing simple logic models  Choosing a strong evaluation focus  Choosing  Discuss November workshop 2 Why Bother? Because We Want Programs to Improve  Continuous Quality Improvement (CQI) cycle.  Planning—What actions What do we do? Why are we doing well or poorly? will best reach our goals and objectives.  Performance measurement— How are we doing?  Evaluation—Why are we doing well or poorly? How do we do it? How are we doing? 3 What can Program Evaluation do?   Increase Program Knowledge Direct Program Improvement Maximize Resources Provide Accountability 4   Avoid Wasted Effort! 5 Two Pieces of Work 1. Choose a program component to evaluate 2. Apply evaluation steps and standards 6 Choosing a Program Component to Evaluate  Evaluation can focus on any/all parts of a large program like the immunization program 7 Some Potential Criteria..  Where am I spending the most?  Where am I concerned the most?  Where are my big opportunities/new areas?  Where are my big successes?  Other…  Other… 8 Some Potential Components to Evaluate Immunization information systems  Adolescent immunization  Perinatal hepatitis B prevention  AFIX  VFC  9 Today’s Case Example: AFIX— Reasons for Selecting Component Substantial program effort  Research shows it works… But implemented in various ways…  Want to ensure “our” approach is working  10 Next—Apply Evaluation Steps and Standards 11 11 Evaluation Step—Stakeholders 1. 2. 3. 4. 5. 6. Identify and engage stakeholders Describe the program component Focus the evaluation design Gather credible evidence Justify conclusions Ensure use and share lessons learned 12 Which Stakeholders Matter Most? Who is  Affected by the program?  Involved in program operations?  Intended users of evaluation findings? Of these, who do we most need to: Enhance credibility? Implement program changes? Advocate for changes? Fund, authorize, expand program? 13 Some Key Stakeholders Types Definitions Examples Immunization program director, program manager, HD director, health commissioner, legislators Decision makers Decide and direct program operations, including how evaluation findings are used Implementers Involved in program operations Immunization program director, program manager, field staff Physicians, parents, community members Participants Served by the program Partners Support/invested in immunization program or target population Managed care plans, health systems, AAP and AAFP chapters, CBOs 14 Payoffs for Involving Stakeholders Strong input on purpose and design of the evaluation  “Reality check” on utility and feasibility of evaluation plan and methods.  Increase credibility of evaluation findings.  Increase likelihood that evaluation results will be acted upon.  15 Involve Stakeholders from the Start! Potential stakeholder roles at all steps: Input on evaluation design Input on data collection methods Assist with data collection/analysis Outreach to the target population for the evaluation Interpretation and use of results 16 Case Example: AFIX Stakeholders Program staff  Clinical staff (clinic managers, nurses, physicians)  17 Evaluation Step—Description 1. 2. 3. 4. 5. 6. Identify and engage stakeholders Describe the program component Focus the evaluation design Gather credible evidence Justify conclusions Ensure use and share lessons learned 18 Seeking Consensus On…     The activities and outcomes depicted The “roadmap” Which outcomes = program “success” This will facilitate future discussions of:  How much progress on outcomes = program “success”  Choices of data collection/analysis methods 19 Describing the Program: Complete Logic Model Inputs Activities Outputs Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Context Assumptions Stage of Development 20 What the program and its staff actually do Inputs Activities Outputs Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Context Assumptions Stage of Development 21 Results of activities: Who/what will change? Inputs Activities Outputs Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Context Assumptions Stage of Development 22 Simple Logic Model—AFIX Early If ACTIVITIES Later Then If Then OUTCOMES Short-Term Long-Term If Then 23 Case Example: AFIX Logic Model Activities If we do this 1 And then this 2 Then this will occur 3 Outcomes And then this And then this 4 5 And then this 6 Review records and assess coverage level Develop AFIX protocol Provide feedback and recommend strategies for improvement Increase in provider Change in provider motivation behavior Acknowledge and reward improved performance Decrease in missed opportunities Increased vaccination Decreased in invalid dose administration Decreased VPD Increase in timeliness of vaccine receipt Train AFIX staff Set up provider visits Document outcomes of visit in database (eg CoCASA) Increase in provider Change in office knowledge practices and policies Promote information exchange 24 Resource “platform” for the program Inputs Activities Outputs Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Tangible products of Context Assumptions activities Stage of Development 25 AFIX Full Logic Model Inputs 1 Activities 2 3 Outputs 4 Outcomes 5 6 7 8 Budget Develop AFIX protocol Train AFIX staff Set up provider visits Review records and assess coverage level Provide feedback and recommend strategies for improvement Acknowledge and reward improved performance Document outcomes of visit in database (eg CoCASA) Promote information exchange 26 Protocol Trainings Coverage assessments Increase in provider motivation Change in provider behavior Change in office practices and policies Staff Decrease in missed opportunities Increased vaccination Time Immunization information systems CoCASA Increase in provider Recommendations knowledge Decreased in invalid dose administratio n Increase in timeliness of vaccine receipt Decreased VPD Inputs Activities Moderators: Contextual factors that will facilitate Short-term Intermediate Effects/ Effects/ Outputs or hinder getting Outcomes Outcomes our outcomes Long-term Effects/ Outcomes Context Assumptions Stage of Development 27 Evaluation Step—Focusing the Evaluation 1. 2. 3. Identify and engage stakeholders Describe the program component Focus the evaluation design    Tailor evaluation to program and stakeholder’s needs Determine resources and personnel available Develop and prioritize evaluation questions 4. 5. 6. Gather credible evidence Justify conclusions Ensure use and share lessons learned 28 Evaluation Standards Help Choose Focus  “Utility”— Which stakeholders/others want the evaluation findings? How/for what purpose will they use the findings? “Feasibility”— How reasonable are these questions given: How long program has been in existence; program intensity; data collection logistics/constraints. 29  Generating and Prioritizing Evaluation Questions Utility: Questions which:  Are important to stakeholders;  Are consistent with program goals and objectives  Will provide information that can be acted upon Feasibility: Questions which:  Can be answered with available resources and within available timeframe  Are consistent with program maturity and intensity  Will be supported by program decision-makers. 30 Setting Focus: Process and Outcome Evaluation  Process Evaluation  Determines if program activities are implemented as intended.  Tracks who, what, when, and where program information.  Provides feedback loop for program improvement.  Conducted throughout project life.  Outcome Evaluation Measures program effects; changes in target population’s knowledge, attitudes, self-efficacy, skills, intentions, behaviors; or organizational changes (e.g., policy adoption).  Linked to process evaluation.  Provides feedback loop for program improvement  31 Case Example: Setting AFIX Focus Utility  Purpose/User: Program staff  Use: Determine if:  AFIX feedback is actually implemented by providers.  If not, determine barriers and challenges to use of feedback  And, determine how to improve activities and followup to surmount identified barriers and challenges Feasibility  Stage/Intensity: Well-established and multi-faceted program. Should be making these impacts if working  Resources: [Assumption] Evaluation questions of interest have some ready data sources that would be credible for those interested in findings 32 AFIX Logic Model—Focus Activities If we do this 1 And then this 2 Then this will occur 3 Outcomes And then this And then this 4 5 And then this 6 Review records and assess coverage level Develop AFIX protocol Provide feedback and recommend strategies for improvement Decrease in missed opportunities Increase in provider Change in provider motivation behavior Increased vaccination Decreased in invalid dose administration Decreased VPD Increase in timeliness of vaccine receipt Train AFIX staff Acknowledge and reward improved performance Set up provider visits Document outcomes of visit in database (eg CoCASA) Increase in provider Change in office knowledge practices and policies Promote information exchange 33 AFIX—Sample Evaluation Questions Proposed focus: both process and outcomes  Outcome:  Are AFIX recommendations implemented by practices within 6 months of visit? (short term outcome)  Process  Factors that may be related to implementation/non-implementation of AFIX recommendations:  Are AFIX staff appropriately trained?  Are practice contacts the right people?  What is the quality of the feedback? Are practices satisfied with feedback received? 34 Where Next… Indicators Data Collection Sources and Methods Engage Stakeholders Throughout! Data Collection Procedures Data Analysis Data Interpretation Shared Results and Lessons Learned Results used to improve program 35 But…  These steps more likely to be done well because we did a good job on earlier steps!!! 36 Practical Program Evaluation Coming! This Fall… Fall Training Session Reaffirm/reinforce today’s points  Work through remaining steps:   Define indicators  Identify data collection sources and methods  Consider data analysis and interpretation  Maximizing use 38 Pre-Work for November Consider criteria for choosing a program component  Give some preliminary thought to:   Potential program components  Key stakeholders  Purpose/use of evaluation 39 Practical Program Evaluation Life Post-Session Helpful Publications @ www.cdc.gov/eval 41 41 Helpful Resources    NEW! Intro to Program Evaluation for PH Programs—A Self-Study Guide: http://www.cdc.gov/eval/whatsnew.htm Logic Model Sites  Innovation Network:  http://www.innonet.org/  Harvard Family Research Project: http://www.gse.harvard.edu/hfrp/  University of Wisconsin-Extension: http://www.uwex.edu/ces/lmcourse/  CDC/DASH:http://www.cdc.gov/healthyyouth/evaluation/resources.htm #4  CDC/STD: http://www.cdc.gov/std/program/progeval/TOCPGprogeval.htm Texts   Kellogg Foundation Logic Model Development Guide: www.wkkf.org W.K. Kellogg Foundation Evaluation Resources: http://www.wkkf.org/programming/overview.aspx?CID=281  Rogers et al. Program Theory in Evaluation. New Directions Series: Jossey-Bass, Fall 2000  Chen, H. Theory-Driven Evaluations. Sage. 1990 42 Community Tool Box http://ctb.ku.edu 43 43 This document can be found on the CDC website at:  http://www.cdc.gov/vaccines/programs/ progeval/downloads/Eval_Course_2nd. ppt 44