Innovative Approaches
and
New Tools for Evaluation
Hans-Martin Boehmer
Fall Meetings
Evaluation Cooperation Group
October 25, 2013
Washington, DC
So much
information!!
How will we
make sense of it?
How can we give
stakeholders
access to our
data?
How can we
really know
what’s
happening in
the field?
How can we
make our
evaluations
transparent and
participatory?
How can we
disseminate
global
lessons?
2
How can we give
stakeholders
access to our
data?
3
Interesting, Insightful Data is in Demand
4
IEG data is easily graphed with Tableau
5
How can we
disseminate
global
lessons?
6
All staff can access lessons online
Evaluation has produced thousands of lessons since 1996
E-LRN 2.0 will:
• Give WBG staff access lessons from investment, advisory
and MIGA operations
• Stimulate new insights and engage thought leaders
• Recognize subject matter experts by appointing Knowledge
Gurus
• Encourage discussions around topics and themes
• Keep staff engaged and informed through alerts and widgets
How can we
really know
what’s
happening in
the field?
How can make
our evaluations
transparent and
participatory?
9
IEG uses new technology and online media
Approach
Paper
Evaluation
Preparation
/Research
Disclosure
Management
and Board
Discussions
10
IEG collects voices from the Field

Comment from a Facebook user on access to health in Afghanistan:
“ unfortunately most of these works [health improvement projects] are limited and they remain
only on a piece of paper instead of being implemented. A good example is our district, where
nothing has been done yet …” Comment from a user from Ahmad Abad district, Paktia
province. [Comment translated from Pashto]
Comment from LinkedIn user on SME
Definition:
“ Within APEC, there is no region- wide
accepted definition of SMEs. Definitions vary
across members and are based on several
different criteria, including number of
employees and maximum levels of capital,
assets, or sales, which may again vary
according to industry/ sector. APEC economies
adopt either single or multiple criteria to define
SMEs.” – Comment from a staff of the Asia
Pacific Economic Cooperation organization
Emerging big, fast, targeted data sources –
not yet exploited by evaluation
• Twitter feed analysis – Jakarta tweets more
than Tokyo!
• Real-time environmental data – e.g. monthly
global deforestation by province
• Calibrate real-time poverty maps from patterns
of phone air-time purchases
• Conduct targeted surveys by mobile phone
• Better use existing library of high quality
household surveys
• IEG will experiment, on a demand driven
basis, with use of these kinds of data
12
Examples of IEG’s use of outreach
• To help in doing IEG’s evaluations:
– Crowdsource: public comment on
projects slated for ICR reviews,
PPARs, XPSRs (possible collaboration
with Ushahidi)
• For capacity building and popularizing
IEG’s missions
– Working with CLEAR and with
universities, offer training in (part of)
ICR review methodology
So much
information!!
How will we
make sense of it?
14
IEG is striving to achieve improved, more
cost effective portfolio analysis
• IEG spends significant resources on filtering
documents and coding their contents– e.g.
identifying projects that deal with early
childhood development or traffic safety
• IEG will experiment with text analytic software
that can help to rapidly sort and classify
documents – used by lawyers, journalists
• Possibility of outsourcing simple, tedious
analytic tasks via internet (e.g., US State Dept
used internet volunteers to rapidly geocode
project locations)
15
IEG is focusing on outputs (standardized
products & speed)
Linking Databases:
1. Velocity links more than 20 databases including:
Bankscope, MIX Market, MFA, WDI, GDF, IEG
Ratings, Fitch (Rating History) and Institutional
Investor among others.
Standardized Products to Speed our work:
1. Graphs – tables ready to cut and paste.
2. Comparative financial information entity vs. local
and international peers. Easier comparisons.
3. All the country information in one place: CAS,
EIU, Fitch, Central banks.
4. Case studies and best practices.
16
IEG is pursuing ways to improve data
handling and visualization
• IEG is organizing and documenting
ratings and project data
• Integrate with external data – e.g.
national statistics
• Shift from Excel to Tableau for easier
analysis and more powerful
visualization
17
Innovative Approaches
and
New Tools for Evaluation
Hans-Martin Boehmer
Fall Meetings
Evaluation Cooperation Group
October 25, 2013
Washington, DC