Measuring Outreach Effectiveness
Web4Dev – November 2006
Alex McKenzie - amckenzie@worldbank.org
Knowledge & Evaluation Capacity Development
Independent Evaluation Group – World Bank
Key Points:
►Share IEGWB’s approach for aligning web/online
practice with mandate and business strategy
►Share our experience running several outreach
campaigns, discuss metrics
►Measuring effectiveness and impact
►Would appreciate hearing about your own
experience and/or feedback
Business Strategy
► IEG is an independent unit that assesses the effectiveness of World Bank
Group’s interventions
► Well regarded for its evaluation capabilities and reports
► Combines Accountability & Learning
► “To be effective, IEG want’s to be an influential and reliable source of
knowledge”
► New Outreach and Communications Strategy:
–
–
–
–
Target key audiences
Embed outreach throughout the evaluation process
Accelerate learning, encourage utilization
Measure results
► Decided on a campaign-based approach, 10 to 12 major evaluation
products per year.
► Multi-pronged outreach campaigns: media, print, online, in-person,
translations, etc.
► Piloted many new ideas, mainstreamed what worked
Key Audiences
►Systematic effort to identify audiences has led to
two sets: “core” & campaign-specific (specialists)
• Identify decision-makers & influencers
• Identify key network nodes
►Evolving audience criteria: from “static”
classifications, to others based on opportunity or
time-sensitive demand
►Customize products to audience whenever
economically feasible (e.g. conference CDROMs,
translations, etc.)
Embed in Evaluation Process
Web support for: Design Phase
Research Phase
Outreach Phase
Communications
- Interim data and
research
- Audience identification
- Outreach campaign*
- Campaign metrics
- Virtual team workspace
- Research surveys
- Feedback, discussion,
comments
Transparency, Efficiency
Key messages for relevant
audiences
- Release approach
paper
Process support
Value Created>> Transparency, Inclusion
* Initial role for web
Encourage Utilization
Encourage Utilization
Measuring Results
► Results-Based Management causal chain
Inputs
(Activities)
Outputs
Intermediate
Outcomes
Outcomes
Area of Control
Area of Influence
Measuring Results – Causal Chain
Source: “Strengthening EAP’s Results Orientation”, April 2004
Measuring Results - Outputs
Web Outreach Campaign Comparison
3000
Email campaign
targetting about 15,000
people.
2500
Online campaigns focus on key audiences, and generate follow up engagements or requests from relevant parties (numbers are
for entire campaign):
Global: 4644 requests;
Dev.Effect.: 983 requests;
Trade: 662 requests.
Page Views
2000
Sustained interest
through Google
advertising
1500
Dev.Effectiveness
Global Program
Trade
1000
500
0
1 2
3 4
5 6
7 8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Campaign Day (normalized)
Measuring Results - Outputs
Web Outreach Campaign Patterns
3500
3000
ARDE 2004
Community-based Dev.
Page Views
2500
All campaigns exhibit similar
trends: Initial traffic peaks
converge quickly to a lower, but
stable levels.
2000
Dev.Effectiveness
HIPC
HIV/AIDS
Fragile States
Natural Disasters
1500
Pension Reform
Primary Education
Trade
1000
500
0
1
3 5
7
9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61
Campaign Day (normalized)
Measuring Results - Outcomes
►Intermediate Outcomes: perception survey* of IEG
evaluation products, covering:
–
–
–
–
Readership
Quality
Influence on Bank processes and policies (similar for external clients)
Utilization
►Annual Review of Operations Evaluation (AROE)
2006, to be released in November/December.
*Source: IEG’s Annual Review of Operations Evaluation (AROE). See
http://www.worldbank.org/ieg/aroe
Measuring Results - Outcomes
► Survey targeted individuals defined as intended audience:
4285 internal (WBG) and 2759 external contacts. Results are
indicative for respondents (22%) but are not generalizable to
population
Readership
Internal
External
Direct awareness:
42%
55%
Overall awareness:
56%
76%
heard via email
64%
69%
visited web site
12%
28%
received hard-copy
20%
15%
Perception of Quality – Staff/EDs (E4)
0
1
2
3
4
5
Relevance to your work
4.88
4.28
Ease of understanding
4.84
4.21
Concise presentation of conclusions
4.13
4.74
4.02
4.00
Timeliness
Usefulness of recommendations
4.76
3.87
Unbiased and objective analysis
3.76
Transparency and clarity of the methodology
3.73
Strong link between conclusions and evidence
3.69
Depth of analysis
3.58
Incorporation of all available information
3.55
ED
Staff
4.78
4.29
4.71
4.47
4.24
6
Perception of Quality – External Clients
(E13)
1
2
3
4
5
Ease of understanding
4.75
Concise presentation of conclusions
4.73
4.66
Relevance to your work
Usefulness of recommendations
4.51
Transparency and clarity of the methodology
4.49
4.45
Timeliness
Unbiased and objective analysis
4.34
Strong link between conclusions and evidence
4.34
Depth of analysis
4.32
Incorporation of all available information
4.15
6
Influence – Staff (E8)
1
2
4
3
3.80
Helped you gain a clear view of essential lessons learned.
3.62
Influenced your understanding of the subject area.
3.51
Helped you gain a clear view of best practices.
Influenced the Bank's sector strategy.
3.30
Influenced the Bank's lending and non-lending services.
3.29
Influenced the Bank's country strategy.
3.27
Influenced how outcomes are linked to outputs (Results
Framework) in projects.
3.23
Influenced how outcomes are linked to outputs (Results
Framework) in the sector.
3.22
Influenced how outcomes are linked to outputs (Results
Framework) at the country-level within the Bank.
3.21
Influence – External Clients (E15)
6
5
4.51
4.45
Helped you
better
understand the
Bank's
activities within
a sector.
Helped you
gain a clear
view of
essential
lessons
learned.
4.31
4.26
4.23
4.12
Helped you
gain a clear
view of best
practices.
Helped you
better
understand the
Bank's
activities within
a country.
4
3
2
1
Provided you
Influenced
with practical
your
information to understanding
help you make of the subject
decision on
area.
strategies and
programs.
Staff Use of Evaluations (E9)
1
2
3
4
Commenting on or making inputs into the work
of others
3.60
Making the case for a particular course of action
3.57
Providing advice to your clients
3.57
Modifying policies and/or strategies
3.22
Designing/Modifying your Results Framework
3.21
Designing new lending operations
3.05
Designing new non-lending operations
3.02
Modifying on-going operations
2.68
5
6
External Clients Use of Evaluations (E16)
6
5
4
4.00
3.95
3.81
3.65
3
69%*
70%*
57%*
62%*
2
1
Research
Making the case for Refocusing ona particular course going strategies or
of action
programs
*Percentages show how many respondents rated 4 or higher
Education
Measuring Results – Survey Implications
• IEG provides good services to its main client, the Board.
External audiences are also highly satisfied with the
quality of IEG evaluations and considered them to be
influential.
• However external audiences would like IEG to broaden
its external consultation during evaluations.
• Bank staff see IEG products useful with deepening their
understanding of a given subject.
• However they don’t see them as useful in day-to-day
work. IEG needs to focus on influencing ongoing and
future operations.
Measuring Results – Survey Implications
►All audiences suggest:
• High value placed on timeliness to provide information
when decisions need to be made
• Deepening analysis through more consultation and
attention to context
• Focus on operational value of findings
►A call for action for IEG’s outreach program
•
•
•
•
4-page briefs
Customized “findings” brief, upstream, via email
More use of non-print media
Balance “lecturer” approach with conversations
Thoughts Going Forward
►"What information consumes is rather obvious; it
consumes the attention of its recipients. Hence, a
wealth of information creates a poverty of
attention."
Herbert Simon, Nobel laureate
Your Experience/Feedback
►What has worked in your organization? (audience
analysis; relevance/timeliness; measuring outputs,
outcomes, impacts, etc.)
►Choices related to channels? High quality content
vs. open/collaborative models
►Outreach models used? Worthwhile consultants &
advice?
►What challenges do you see ahead?