Communicating Evaluation Data Workshop Description This session will provide an overview of how evaluation questions and data collection can be implemented using a communications for sustainability framework. Participants will learn how to use marketing techniques and strategies to promote evaluation data effectively as well as to discover how to use this data to allow for the greatest impact with various target audiences in their communities. Session Objectives At the end of the session, participants will be able to 1. Identify organizations within their community for potential sustainability partnerships 2. Understand how to work with their evaluator to collect and disseminate data based on communications needs 3. Create meaningful communications using evaluation data that is developed and gathered for specific audiences Sustainability Communications Mindset Look down the road not only for sustaining SS/HS initiatives but also for creating an environment that integrates the goals and mission of the initiatives into the fabric of your community 1. Who needs to be at the table? 2. What do we need to say to get them there? 3. What data do we need to provide to make the information relevant to a particular audience? Who Needs To Be at the Table? • Organizations with similar priorities as yours – Think outside of the typical audiences • Current grant partners • Other community agencies/organizations • Civic organizations • Others? Who Needs To Be at the Table? • Things to think about – What are the top three priorities of our current partner agencies? • Where is there natural alignment without money as a motivator? – Within the community, what other organizations/agencies have priorities/vision/mission that align with one or more initiatives of SS/HS? – Follow the money • How does funding funnel into your community? • How are existing youth initiatives funded? What Do We Need To Say to Potential Partners? • Messages must be relevant to your audience • Messages must be in a format that is easy for your audience to read and understand • Messages must be delivered in a timely way by a trusted source What Do We Need To Say to Potential Partners? Questions that you can use to help decide how to present your evaluation data to potential partners – What’s their definition of success? – What are their priorities? – What are they interested in? – What information do they trust? – What level of technical detail do they demand? – What data do you already have? – What data do you need to tell your story? Primary and Secondary Audiences for Communicating Evaluation Data • Primary Users are the individuals that you want to use your results; these people are your chief audience and often provide funding or other support • Secondary Users are individuals who may be associated with your program or have an interest in what you are doing (e.g., city council, neighborhoods, and service recipients) What Information Is Most Relevant to Your Audience’s Priorities? • Description of • Change in – Programs – Expenditures – Cost – Student referrals – Students – School climate – School climate – Academic – Services performance – Consumer satisfaction What Information Do They Trust? • Caregiver-reported data • Teacher-reported data • Student-reported data • Community member-reported data • Administrator-reported data • Agency staff-reported data • Management information system data What Format Do We Use? Once you have completed your evaluation and analysis, you should have information that provides an accurate picture of your program as well as information to make decisions about future program implementation. So, how do you share your findings? Results can be reported not only in a text document, but also in – Oral reports and presentations – Videos – Posters – Press releases – Newsletters – Other forms of communication Exercise: Understanding Your Audience’s Data Needs Understanding Your Audience TO USE: Decide if each audience would prefer quantitative or hard data (QT); qualitative, self-reported, or anecdotal data (QL); or both (B). Then, for each audience, enter a QT, QL, or B in each of the boxes that represent the types of data your target audiences will most want to see. Audience Segments Types of Data Use space under each segment to specifically Descriptive Outcome identify (e.g., does district Students Program Service Satis- Climate Student Academic Service Costs Satis- School Academic leadership include faction Behavior Perform. Array faction Climate Indicators superintendent? School board? Principals?) District Leadership Law Enforcement Mental Health Providers Policymakers Business Leaders Faith-Based Leaders Community Members Other Inventory Your Data • What type(s) of data do you have? • Where did your data come from? • What is the quality of your data? • What message does the data convey? What Data Do You Need? • What do you currently have? What are you currently gathering? • Depending on the status of your program evaluation you can – Work with your evaluator to ensure that evaluation instruments that will provide you with relevant data for communications goals are developed – Use existing evaluation data in a way that is relevant to your audience – Some of both What Type(s) of Data Do You Have? • Qualitative Data • Quantitative Data - Provides answers to open- - Provides answers to close- ended questions ended questions - Derives from notes from - Uses numbers not words observations - Is obtained through - Uses words not numbers structured surveys and - Is obtained through focus interviews, existing records groups and unstructured interviews - Often includes a large - Includes contextual detail number of participants about feelings and - Provides less contextual perceptions in an effort to detail gain a holistic - Is amenable to common understanding statistical procedures - Includes relatively few participants What Type(s) of Data Do You Have? • Descriptive Data • Outcome Data – Is cross sectional – Is longitudinal – Describes – Tracks people, characteristics of organizations, or people, programs over time organizations, or by using the same programs at a measures single point in – Assesses change in time people, programs, or organizations – Has an established baseline Exercise: Do You Have What You Need? Data Inventory Worksheet Data Inventory Worksheet TO USE: Check off the boxes that best represent the types of data you currently are collecting. Note where you might need to collect additional data, based on your findings from the Audience Data Preferences Worksheet. Discuss with your evaluator how you might fill in those gaps. Data Source Type and Quality of Data Qualitative Descriptive Qualitative Outcome Quantitative Quantitative Outcome Descriptive Good Fair Poor Good Fair Poor Good Fair Poor Good Fair Poor Agencies Parents Students Teachers Systems Generated Community Members Academic Indicators External Comparison Other_____________ So Far We’ve - • Identified potential partners/audiences • Analyzed the type of data and information that is relevant to them • Determined the data that you currently have and/or are collecting and identified where there are data needs • So how do we take all of this information and put pen to paper? Planning Your Project: Things To Decide Before Anyone Writes a Single Word • Who is going to read this? • What do they have in common? • When they’re done reading, what do you want this audience to do or know? • What materials do you have to work from? (See Data Inventory Worksheet) Planning Your Project: Things To Decide Before Anyone Writes a Single Word • What kind of research and/or interviews are necessary? (See Audience Worksheet) • Will this piece be printed only, or will it be posted on the Web? • Do you need to design the piece, or must you fit the text into one that’s already designed? • What ―extras‖ do we need, such as sidebars, charts, or photos? Planning Your Project: Things To Decide Before Anyone Writes a Single Word • Internal factors • Who will have input, and who has the final say? • If a lot of people will have input, who will coordinate their comments and resolve any disputes? • Does everyone agree on the number of drafts allowed and when the draft will become final, with no more changes? • What resources are available to help you complete this project? Planning Your Project: Things To Decide Before Anyone Writes a Single Word • Internal factors • Do you have someone who can put in the time necessary for organizing, researching, thinking, drafting, re- writing, and polishing over the life of the project? • What is the budget for this project? • How will your materials be distributed? What Information Should Be Included When Sharing Evaluation Data? 1. State your program objectives a. This is important because, eventually, you will discuss how your evaluation data contributes to your stated objective. What Information Should Be Included When Sharing Evaluation Data? 2. Describe progress toward achieving your objectives a. Describe your activities as they relate to your objectives b. Identify your beneficiaries—the number and characteristics of people you serve c. Describe your desired results—talk about the indicators of success that you have chosen. These are the factors that are most important to your program What Information Should Be Included When Sharing Evaluation Data? 3. Describe your evaluation activities a. Identify the types of instruments you used b. Report on the information you collected c. Detail who administered your instruments, how they were administered (e.g., by mail, in person, or by telephone), and who, as well as how many people, completed them d. Note how you measured your desired result e. Discuss any problems encountered during the evaluation What Information Should Be Included When Sharing Evaluation Data? 4. Describe the results of your evaluation This is the most important section! a. Report your findings—statistics and qualitative information b. Employ graphs, charts, and tables to help convey your message c. Share stories from people who are directly involved in your program and/or your evaluation What Information Should Be Included When Sharing Evaluation Data? 5. Compare the results of your evaluation with your objectives a. Describe how your services addressed the existing need. You can relate the need to baseline data if you have it b. Refer to previous evaluation findings (such as those reported in earlier reports) to provide a more detailed picture of the program progress What Information Should Be Included When Sharing Evaluation Data? 6. State ideas for partner support and ask for commitment This is your “ask” a. What do you want the audience to do? b. Show them a clear path to doing it c. Incorporate message development work to ensure success Pulling It All Together Harrisburg, Pennsylvania Student Survey Data Goal: To create meaningful communications using evaluation data that is developed and gathered for specific audiences Harrisburg, PA Student Survey Data • Purpose – Evaluate prevention programs at the secondary level • Life Skills Training • TimeWise • HealthWise • School Transitional Environment Project (STEP) • Teen Outreach Program – Establish a monitoring system for social, emotional, and behavioral outcomes Harrisburg, PA Student Survey Data • Methods – Conducted annually (beginning in October 2003) – All students in 6th–12th grade – Paper/pencil administration during a required class – 77 percent response rate Harrisburg, PA Student Survey Data • Communications Goals – Give the data back to students before the 2004 survey administration – Integrate data into agency accountability systems – Drive program decisions during grant period Harrisburg, PA Student Survey Data • Target audience: Project director – Communications goal • Drive program decisions during grant period – Messaging • Straightforward report of all data • Not bad vs. good, but pointing how to improve quality – Channel • Bound report Harrisburg, PA Student Survey Data • Target audience: Students – Communications goal • Give data back to students before the 2004 survey to increase participation – Message • Show students that survey data is used and relevant – Channels • Student newsletter • Article from student perspective in school newspaper Harrisburg, PA Student Survey Data • Target audience: Steering committee – Communications goal • Drive program decisions during grant period – Messaging • How do our students compare to national and State averages? • These students are all our responsibility, how can we collaborate to provide the best mix of programs given these data? – Channel • PowerPoint presentation Harrisburg, PA Student Survey Data • Target audience: Superintendent – Communications goal • Integrate data into agency accountability systems – Messaging • The climate of schools is important; we care about the whole student (socially, emotionally, behaviorally, and academically) – Channel • Talking points in PowerPoint of survey highlights for Superintendent’s annual ―state of the union‖ Other Audiences • How would you use this same data for other audiences? – Businesses? – Community organizations? – Parents? – Others? • What different information might you need for these audiences? Questions?
Pages to are hidden for
"Audience Worksheet - PowerPoint"Please download to view full document