Measuring Outreach Effectiveness

Measuring Outreach Effectiveness Web4Dev – November 2006 Alex McKenzie - amckenzie@worldbank.org Knowledge & Evaluation Capacity Development Independent Evaluation Group – World Bank Key Points: ►Share IEGWB’s approach for aligning web/online practice with mandate and business strategy ►Share our experience running several outreach campaigns, discuss metrics ►Measuring effectiveness and impact ►Would appreciate hearing about your own experience and/or feedback Business Strategy ► IEG is an independent unit that assesses the effectiveness of World Bank Group’s interventions ► Well regarded for its evaluation capabilities and reports ► Combines Accountability & Learning ► “To be effective, IEG want’s to be an influential and reliable source of knowledge” ► New Outreach and Communications Strategy: – – – – Target key audiences Embed outreach throughout the evaluation process Accelerate learning, encourage utilization Measure results ► Decided on a campaign-based approach, 10 to 12 major evaluation products per year. ► Multi-pronged outreach campaigns: media, print, online, in-person, translations, etc. ► Piloted many new ideas, mainstreamed what worked Key Audiences ►Systematic effort to identify audiences has led to two sets: “core” & campaign-specific (specialists) • Identify decision-makers & influencers • Identify key network nodes ►Evolving audience criteria: from “static” classifications, to others based on opportunity or time-sensitive demand ►Customize products to audience whenever economically feasible (e.g. conference CDROMs, translations, etc.) Embed in Evaluation Process Web support for: Design Phase Communications - Release approach paper Research Phase - Interim data and research - Audience identification - Virtual team workspace - Research surveys Transparency, Efficiency Outreach Phase - Outreach campaign* - Campaign metrics - Feedback, discussion, comments Key messages for relevant audiences Process support Value Created>> Transparency, Inclusion * Initial role for web Encourage Utilization Encourage Utilization Measuring Results ► Results-Based Management causal chain Inputs (Activities) Outputs Intermediate Outcomes Outcomes Area of Control Area of Influence Measuring Results – Causal Chain Source: “Strengthening EAP’s Results Orientation”, April 2004 Measuring Results - Outputs Web Outreach Campaign Comparison Online campaigns focus on key audiences, and generate follow up engagements or requests from relevant parties (numbers are for entire campaign): Global: 4644 requests; Dev.Effect.: 983 requests; Trade: 662 requests. 3000 Email campaign targetting about 15,000 people. 2500 2000 Page Views 1500 Sustained interest through Google advertising Dev.Effectiveness Global Program Trade 1000 500 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Campaign Day (normalized) Measuring Results - Outputs Web Outreach Campaign Patterns 3500 3000 ARDE 2004 Community-based Dev. 2500 Page Views 2000 All campaigns exhibit similar trends: Initial traffic peaks converge quickly to a lower, but stable levels. Dev.Effectiveness HIPC HIV/AIDS Fragile States Natural Disasters 1500 Pension Reform Primary Education 1000 Trade 500 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 Campaign Day (normalized) Measuring Results - Outcomes ►Intermediate Outcomes: perception survey* of IEG evaluation products, covering: – – – – Readership Quality Influence on Bank processes and policies (similar for external clients) Utilization ►Annual Review of Operations Evaluation (AROE) 2006, to be released in November/December. *Source: IEG’s Annual Review of Operations Evaluation (AROE). See http://www.worldbank.org/ieg/aroe Measuring Results - Outcomes ► Survey targeted individuals defined as intended audience: 4285 internal (WBG) and 2759 external contacts. Results are indicative for respondents (22%) but are not generalizable to population Readership Internal External Direct awareness: Overall awareness: heard via email visited web site 42% 56% 64% 12% 55% 76% 69% 28% received hard-copy 20% 15% Perception of Quality – Staff/EDs (E4) 0 1 2 3 4 4.28 4.21 4.13 4.02 4.00 3.87 3.76 3.73 3.69 3.58 4.29 4.71 4.47 4.24 4.76 4.78 5 4.88 4.84 4.74 6 Relevance to your work Ease of understanding Concise presentation of conclusions Timeliness Usefulness of recommendations Unbiased and objective analysis Transparency and clarity of the methodology Strong link between conclusions and evidence Depth of analysis Incorporation of all available information ED 3.55 Staff Perception of Quality – External Clients (E13) 1 Ease of understanding Concise presentation of conclusions Relevance to your work Usefulness of recommendations Transparency and clarity of the methodology Timeliness Unbiased and objective analysis Strong link between conclusions and evidence Depth of analysis Incorporation of all available information 2 3 4 5 4.75 4.73 4.66 4.51 4.49 4.45 4.34 4.34 4.32 4.15 6 Influence – Staff (E8) 1 Helped you gain a clear view of essential lessons learned. Influenced your understanding of the subject area. Helped you gain a clear view of best practices. Influenced the Bank's sector strategy. Influenced the Bank's lending and non-lending services. Influenced the Bank's country strategy. Influenced how outcomes are linked to outputs (Results Framework) in projects. Influenced how outcomes are linked to outputs (Results Framework) in the sector. Influenced how outcomes are linked to outputs (Results Framework) at the country-level within the Bank. 2 3 4 3.80 3.62 3.51 3.30 3.29 3.27 3.23 3.22 3.21 Influence – External Clients (E15) 6 5 4 3 2 1 Helped you better understand the Bank's activities within a sector. Helped you gain a clear view of essential lessons learned. Provided you Influenced with practical your information to understanding help you make of the subject decision on area. strategies and programs. Helped you gain a clear view of best practices. Helped you better understand the Bank's activities within a country. 4.51 4.45 4.31 4.26 4.23 4.12 Staff Use of Evaluations (E9) 1 2 3 4 3.60 3.57 3.57 3.22 3.21 3.05 3.02 2.68 5 6 Commenting on or making inputs into the work of others Making the case for a particular course of action Providing advice to your clients Modifying policies and/or strategies Designing/Modifying your Results Framework Designing new lending operations Designing new non-lending operations Modifying on-going operations External Clients Use of Evaluations (E16) 6 5 4 3 69%* 2 1 70%* 62%* 57%* 4.00 3.95 3.81 3.65 Research Making the case for Refocusing ona particular course going strategies or of action programs Education *Percentages show how many respondents rated 4 or higher Measuring Results – Survey Implications • IEG provides good services to its main client, the Board. External audiences are also highly satisfied with the quality of IEG evaluations and considered them to be influential. • However external audiences would like IEG to broaden its external consultation during evaluations. • Bank staff see IEG products useful with deepening their understanding of a given subject. • However they don’t see them as useful in day-to-day work. IEG needs to focus on influencing ongoing and future operations. Measuring Results – Survey Implications ►All audiences suggest: • High value placed on timeliness to provide information when decisions need to be made • Deepening analysis through more consultation and attention to context • Focus on operational value of findings ►A call for action for IEG’s outreach program • • • • 4-page briefs Customized “findings” brief, upstream, via email More use of non-print media Balance “lecturer” approach with conversations Thoughts Going Forward ►"What information consumes is rather obvious; it consumes the attention of its recipients. Hence, a wealth of information creates a poverty of attention." Herbert Simon, Nobel laureate Your Experience/Feedback ►What has worked in your organization? (audience analysis; relevance/timeliness; measuring outputs, outcomes, impacts, etc.) ►Choices related to channels? High quality content vs. open/collaborative models ►Outreach models used? Worthwhile consultants & advice? ►What challenges do you see ahead?