Evaluation 101 Introduction to the Logic Model

Evaluation 101: Introduction to the Logic Model Holly S. Ruch-Ross, ScD The speaker in this session has no relevant financial relationship with the manufacturer of any commercial product and/or provider of commercial services discussed in this CME activity. The speaker will not discuss or demonstrate pharmaceuticals and/or medical devices that are not approved by the FDA and/or medical or surgical procedures that involve an unapproved or "off-label" use of an approved device or pharmaceutical. Objectives: Participants will…  Understand the use of a logic model for program planning, implementation and evaluation  Understand process and outcome evaluation  Identify the components of a logic model  Develop a logic model for your project using the process and steps presented Defining Evaluation  While evaluation is not research, when conducting an evaluation we use research methods to gather information that will be actively used for:  Program development and improvement  Program replication  Resource allocation  Policy decisions Reasons to Evaluate  Checking Your Process: Are you doing what you said you would do?  Determining Your Impact: Are you having the desired effect in the target population?  Building Your Base of Support: Can you generate information and evidence to share with funders and other stakeholders?  Replication Justification: Is there evidence to support replication of this program? Review: The Evaluation Cycle START Step 1: Plan program and evaluation Step 4: Adjust program as data suggests; adjust evaluation Step 3: Review data. Are you doing what you planned? Are you impacting the need you identified? Step 2: Implement program and begin to collect evaluative data Review: Types of Evaluation  Process Evaluation  Is the program being implemented the way it was designed?  Outcome Evaluation  Is the program having the intended effect? Process Evaluation  Describe the program and implementation, who participates in the program, what services are received.  Information such as number served, patient characteristics, number of contacts with a program, number of trainings, number of referrals. Outcome Evaluation  Detect whether the intervention made a difference, what changes can be measured (knowledge, attitude, behavior, health status, incidence, prevalence)  Longer term outcomes may need to be assessed using shorter term indicators. Goals and Objectives  Goal: broad statement of what the program would like to accomplish for a specific target population.  Objective: measurable step toward the achievement of a goal.  Who will do  What by  When A GOOD OBJECTIVE IS SMART:  Specific  Measurable  Achievable  Realistic for the program  Time specific What is a logic model? If you don’t know where you’re going, you might wind up someplace else. -Yogi Berra  Adapted from a business model  A brief (preferably one page) summary of your program  A “snapshot” of the logical sequence of steps connecting resources to intended results  Diverse templates, but the central idea remains the same How is a logic model useful for my program?  Summary that is easy to share with new staff, boards, funders  Helps stakeholders keep focus on what is most important for program  Facilitates program planning, implementation and evaluation  Process of developing LM is itself useful for building consensus by planning together and for identifying gaps or problems in your planning. The Logic Model TARGET POPULATION The characteristics of people or communities you work with and the needs they present. INPUTS The resources required for this program to operate. ACTIVITIES OUTPUTS OUTCOMES Desired changes in the target population as a result of the program activities. Changes in knowledge, attitude, behavior, health status, health care utilization, incidence, prevalence, etc. Strategies you Basic data on use or services program you provide to try participation. to achieve your goal. Provide training, counseling, education, screenings, referrals, develop materials, etc. Number of participants attending a training, number of counseling sessions, etc. Age, gender, SES, Money, staff, ethnicity, language, volunteers, geographical facilities, etc. location, low health care utilization, high cancer incidence, lack of mental health information, etc. Logic Model Example : Sarah and the Prevention First Program  Sarah is the program director of Prevention First  Large multi-agency collaborative  Community has: High mobility Low income Limited/no English Various immigrant groups Logic Model Example : Sarah and the Prevention First Program The program intends to  bring together the diverse resources and expertise present in the collaborative  facilitate the use of preventive health care by this community  increase public awareness of the many free, non-emergency health and dental services available in the community. Prevention First Goals and Objectives Goals  Immigrant families will understand the importance of prevention.  Immigrant families will use preventive health services Objectives  Within the first 6 months of the project, we will conduct a focus group with immigrant parents to explore possible barriers to the use of prevention services.  By the end of year 1, we will have made presentations to staff of at least 4 agencies serving immigrant families to promote preventive health services and encourage referrals.  By the end of year 1, participating immigrant families will schedule and complete an increased number of wellchild visits over base line. Logic Model Example : STEP ONE Prevention First TARGET POPULATION (characteristics and needs) Low income, limited English-speaking community Low use of health care coverage Low use of preventive health services Mostly employed in temporary and/or part-time positions Mostly from cultures without a concept of preventive health care Who does the program serve? What problem does the program address? Be as specific as possible. Logic Model Example : STEP TWO Prevention First TARGET POPULATION characteristics and needs INPUTS resources Low income, limited English-speaking community Low use of health care coverage Low use of preventive health services Mostly employed in temporary and/or parttime positions Mostly from cultures without a concept of preventive health care Coalition members, director, and 2 interns Funding Computers Curriculum and volunteer health educators Prevention media Verbal and written translation What resources are available to the program? Consider resources other than the immediate dollars and staff specified for the program. Logic Model Example : STEP THREE Prevention First TARGET POPULATION characteristics and needs INPUTS resources ACTIVITIES strategies and services Low income, limited English-speaking community Low use of health care coverage Low use of preventive health services Mostly employed in temporary and/or parttime positions Mostly from cultures without a concept of preventive health care Coalition members, director, and 2 interns Funding Computers Curriculum and volunteer health educators Prevention media Verbal and written translation Health care use intake Health care coverage Prevention education sessions Preventive health services in nontraditional locations Focus groups Regular tracking of health care coverage and preventive service use What does the program do in its day to day operations? Logic Model Example : STEP FOUR Prevention First TARGET POPULATION characteristics and needs INPUTS resources ACTIVITIES strategies and services OUTPUTS program participation Low income, limited English-speaking community Low use of health care coverage Low use of preventive health services Mostly employed in temporary and/or parttime positions Mostly from cultures without a concept of preventive health care Coalition members, director, and 2 interns Funding Computers Curriculum and volunteer health educators Prevention media Verbal and written translation Health care use intake Health care coverage Prevention education sessions Preventive health services in nontraditional locations Focus groups Regular tracking of health care coverage and preventive service use Number of new families signed up for coverage Number of lapsed coverage renewed Number attended education Number of contacts in nontraditional settings Number of focus groups What do participants “get” or “do” as a part of their program participation? Logic Model Example Prevention First TARGET POPULATION Characteristics; needs INPUTS resources ACTIVITIES strategies and services OUTPUTS program participation OUTCOMES desired changes in the population Low income, limited English-speaking community Low use of health care coverage Low use of preventive health services Mostly employed in temporary and/or parttime positions Mostly from cultures without a concept of preventive health care Coalition members, director, and 2 interns Funding Computers Curriculum and volunteer health educators Prevention media Verbal and written translation Health care use intake Health care coverage Prevention education sessions Preventive health services in nontraditional locations Focus groups Regular tracking of health care coverage and preventive service use Number of new families signed up for coverage Number of lapsed coverage renewed Number attended education Number of contacts in nontraditional settings Number of focus groups Immigrant families will understand the importance of preventive health services Participating immigrant families will schedule and complete an increased number of well child visits Immunization rates will increase among children in the target population The number of workdays or school days missed due to illness will decrease Logic Model Example: Outputs vs Outcomes  Outputs are the direct products of program activities (number of participants attending, number of sessions, materials produced and distributed).  Outputs reflect program implementation, and are usually measured as a part of process evaluation.  Outcomes are changes in the target population that result from the program (changes in knowledge, attitude, behavior, health status.  Outcomes reflect program impacts, and are usually measured as a part of outcome evaluation. Logic Model Example: Where are my Goals and Objectives?  Your logic model can be seen as an expression of your goals and objectives.  Goals are most often reflected in the target population and outcomes columns.  Objectives are most often reflected in the activities, outputs and outcomes columns. Getting Started on Your Logic Model • Think very specifically about one program. • Keep in mind that your logic model will be useful for planning, implementation, and evaluation. • Consider who your audiences might be. • Work as a team if possible. • Don’t make it too hard – the goal is a “snapshot” of your program; don’t worry too much about the jargon. • It’s okay to work “backwards” if that is easier for you – beginning with the end in mind. Your Logic Model: STEP ONE TARGET POPULATION The characteristics of people or communities you work with and the needs they present. Who does the program serve? What problem does the program address? Be as specific as possible. Step One - Questions and Reflections  Was this step easy or difficult?  What do you need more information about?  Whose help do you need? Your Logic Model: STEP TWO TARGET POPULATION The characteristics of people or communities you work with and the needs they present. INPUTS The resources required for this program to operate. What resources are available to the program? Consider resources other than the immediate dollars and staff specified for the program. Step Two - Questions and Reflections  Was this step easy or difficult?  What do you need more information about?  Whose help do you need? Your Logic Model: STEP THREE TARGET POPULATION The characteristics of people or communities you work with and the needs they present. INPUTS The resources required for this program to operate. ACTIVITIES Strategies you use or services you provide to try to achieve your goal. What does the program do in its day to day operations? Step Three - Questions and Reflections  Was this step easy or difficult?  What do you need more information about?  Whose help do you need? Your Logic Model: STEP FOUR TARGET POPULATION The characteristics of people or communities you work with and the needs they present. INPUTS The resources required for this program to operate. ACTIVITIES Strategies you use or services you provide to try to achieve your goal. OUTPUTS Basic data on program participation. What do participants “get” or “do” as a part of their program participation? Step Four - Questions and Reflections  Was this step easy or difficult?  What do you need more information about?  Whose help do you need? Pre-Step Five Identifying Outcomes  An outcome is a measurable change that occurs in your target community or population beyond the point of service or intervention. Outcomes define the logical and desired results of the services your program provides. Measuring outcomes is a way to detect whether your program is making a difference. Long term outcomes may need to be assessed using shorter term indicators. Limit the number of outcomes so that you can focus resources. Use standard outcomes when appropriate.     Your Logic Model: STEP FIVE TARGET POPULATION The characteristics of people or communities you work with and the needs they present. INPUTS The resources required for this program to operate. ACTIVITIES Strategies you use or services you provide to try to achieve your goal. OUTPUTS Basic data on program participation. OUTCOMES Desired changes in the target population as a result of the program activities. Taking it Home: What comes next?  Who should you share/work on this with? When? Why?  How could this be used to guide program planning and implementation?  How might it inform you evaluation planning and your evaluator? Where to find more information…  Evaluation Resources on the AAP web site: http://www.aap.org/commpeds/resources/evaluation.html You will find information on…  Logic models  Outcomes  Questionnaire Design  Evaluation measures & instruments Also on the web site, you will find:  Evaluating Your Community-Based Program, Part 1: Designing Your Evaluation  Evaluating Your Community-Based Program, Part 2: Putting Your Evaluation Plan to Work (Fall 2008) We’re lost, but we’re making good time. -Yogi Berra