"Journey of a Survey � Roadmap to Usable Data"
Frequently Asked Evaluation Questions Tobacco Control Evaluation Center Jeanette Treiber, Robin Kipke and Travis Satterlund April 28, 2011 Where Do We Start? Each evaluation activity is derived from the plan: Objective plan type outcome/process data How will data be used to move the objective forward? Who should the data inform or convince? Map out what pieces of information are needed, and how best to collect it (method, from whom, where, when) 2 What Is End-use Strategizing? Identify your evaluation goals Know the kind of information needed and specific purpose it should serve Understand the best source for your data Figure out when the data should be collected Recognize what type of method is most useful Decide an appropriate sample size 3 How Can We Use Our Data? Your data can inform project staff by: Measuring how your message is being perceived by various stakeholders (subgroups) Indicating whether further community education efforts are needed Assessing which policy options to pursue Identifying where your campaign might encounter opposition Signaling that perhaps it is not yet the right time to pursue policy 4 Are There Other Uses For Our Data? Through a variety of reporting strategies, programs can use data to: Indicate public/stakeholder readiness for a policy to decisionmakers Generate “buzz” by sharing results in meetings, presentations, fact sheets, press releases Build a groundswell of support and motivate more people to join the effort Counteract critics 5 Should We Create New Instruments? Adapt existing instruments (based on end use strategizing): Not always necessary to reinvent the wheel Adapt other data collection instruments for your project’s purposes Start with high quality instrument Add questions you need data on Delete unnecessary questions Make sure it will collect what you need 6 Where Can We Find Instruments? TCEC website has model instruments on limited number of objectives Contact TCEC to access repository of instruments created/used by other projects Ask other projects working on similar objectives to share their instruments 7 What Other Sites Have Instruments? Download instruments from statewide sites CA Health Interview Survey – http://www.chis.ucla.edu/questionnaires.html CA Healthy Kids Survey – http://www.wested.org/cs/chks/view/chks_s/17?x- layout=surveys CDC – http://cdc.gov/tobacco/data_statistics/surveys/ YTPS on STORE Website – http://www.tcsstore.org/stages/1_document/t_yps. pdf 8 How Can We Reach a Large Sample? Utilize high traffic venues/locations Back-to-school packets Church service programs Grocery store flyers Classroom teachers Entertainment tickets (e.g. movie theaters) Organizational newsletters Fairs and local events 9 What Are Other Distribution Avenues? Online surveys (SurveyMonkey) Invite participation via email, social networking media (Facebook, Twitter) Your organization’s website (and those of coalition members, agencies that serve target population) 10 What Are the Pros of Online Surveys? Cost effective Faster delivery Quick response time Better at addressing sensitive issues Ease of layout design and analysis 11 How Can We Increase Response Rates for Online Surveys? Some tips for maximizing the response rates: Request participation in advance Include a cover letter Allow sufficient time frame to complete the survey Provide survey instructions Design a clear survey Send reminders to those who still have not completed the survey Offer incentives for completion 12 Do We Need to Pilot Test the Instrument? Evaluate instrument’s effectiveness Get a sense of how long it takes to complete Create checklist of concerns to address with the pre-test Do respondents/data collectors understand the instrument? Is the wording clear to all segments of sample population? Do any items produce irritation or embarrassment? Have important issues been overlooked? 13 What Should Our Key Informant Interview Protocol Look Like? Identify your goals beforehand Know the kind of information needed and specific purpose it should serve Create a structure to your interview Probe, probe, probe (write probes into your interview protocol) 14 What Kind Of Outcome Data Do We Need For A MUH Objective? Depending on the objective: Observed presence and placement of signage Observation data of smoking and tobacco litter Confirmed smoke-free language in leases Tabulated count of signed leases with smoke- free language 15 How Can We Tap Into Volunteer Labor? Develop relationships with local universities, high schools, programs or clubs Recruit students looking for research experience to do data collection Offer training, internships, course credit, other incentives Entice coalition members to join in the activities 16 How Do We Make Use Of Our Volunteers? To field test instruments/materials for appropriateness to data sources To translate instruments/materials into other languages To collect data To provide access to other distribution points 17 How Do We Write Up the Results From Our Data Collection? Summarize as you go Write “mini-analyses” See if the data can help you meet your objective Incorporate your summaries and analyses into your reports 18 How Can We Utilize TCEC’s Resources? Online coalition satisfaction survey administration and analysis service Additional model data collection instruments on a variety of topics will be developed in coming months Model instruments translated into other languages (Spanish, Vietnamese, Hmong and Russian) We’re working with other statewide projects to develop tools that will help you use and share your data more effectively for your policy work 19 When Should We Contact TCEC? For individualized assistance with: evaluation planning development of data collection instruments advice on analyzing & interpreting data guidelines for reporting & sharing info 20 Resources You Can Tap TCEC archive of data collection instruments Evaluation associates at TCEC who can advise you on developing instruments Recorded webinars on survey development, sampling decisions, conducting POS Tips & Tools on creating interview questions, sampling and end-use strategizing Electronic quarterly newsletter Your colleagues working on similar objectives 21