Innovations in
Evaluation
IPDET Workshop, Ottawa, June 14 2013
Simon Roy & Louise Mailloux
Outline
 Definitions
 Innovations in Canada
 Innovations on the
International Scene
 Discussion: What’s your
experience?
2
A Definition…

Innovations can be defined as
alternative and new ways of
conducting evaluations (methods,
analyses, governance, etc.)

Many drivers:
−
−
−
−
Methodological challenge affecting data quality or
availability
Opportunities stemming from new technologies
Influence from other disciplines/professions
HR or governance challenges
3
Contextual factors

Innovations are region-specific:
What is innovative in one place
may not be in another area

Some innovations may work in one
country, but not in another
4
Recent Innovations in
Canada
5
Three notable innovations
in last decade In Canada
 Multimode approaches in
surveys
 Focus on cost analyses
 Professionalization of
evaluation: certification of
evaluators
6
Multi-Mode Surveys

Surveys traditionally done in single mode:
mail or phone or fax

Low response rates now major problem

Evaluators have moved to surveys
administered in multiple modes:
respondents offered to complete it online,
by phone or by mail

Advantages: Higher response rates, less
bias in terms of sampling

Disadvantage: There is a bias associated to
the mode
7
Cost Analyses

Many governments moving towards
“value for money” analyses,
including analysis of input, outputs
and outcomes in view of the costs
involved

Innovation is in the refinement of
the approaches to conduct such
analyses
8
Perspectives on Assessing
Resources Utilization and the
Results Chain
Allocative
Efficiency
Operational
Efficiency
Economy
Results
Chain
Inputs
Activities
Outputs
Immediate
Outcomes
Intermediate
Outcomes
Ultimate
Outcomes
The analysis for economy, operational efficiency and
allocative efficiency occurs along the results chain.
Primary focus of analysis
Informs analysis
9
Credentialing

Canadian Evaluators have an association:
The Canadian Evaluation Society (CES)
(http://www.evaluationcanada.ca/)

The CES implemented an Evaluation
Credentialing Program in 2010. Evaluators
can become “Credentialed Evaluators”

This is an association-led initiative. The
Governments of Canada have no direct
control over this credential.

It is not a requirement to conduct
evaluations.
10
Credentialing

Canadian Evaluators can receive a
credential if they meet criteria
(demonstration of competency), including
2 years of evaluation experience and
competencies in 5 areas (see appendix)

Expected benefits: Evaluators gain
recognition. Credentials help evaluation
users select evaluation provider.

About 200 credentialed evaluators to date.
11
Our Overall Lessons to
Date

Evaluation is evolving – becoming more
and more complex

Before discounting new ways, look at the
advantages, especially how they can
compensate for limitations of traditional
approaches (traditional methods have
gaps too!)

Weigh the advantages vs. disadvantages,
manage them to reduce the latter. Have
a backup plan.
12
Innovations
International
Development Context
13
Real Time Evaluations
(RTE)
Digital Data
A definition of RTE
 A real-time evaluation (RTE) is an
evaluation in which the primary
objective is to provide feedback in a
participatory way in real time (i.e.
during the evaluation fieldwork) to
those executing and managing a
humanitarian response.
Source: Real-time evaluations of humanitarian action An ALNAP
Guide Pilot Version, John Cosgrave Ben Ramalingam and Tony
Beck, 2009
Origins of RTEs
 In the humanitarian sector, UNHCR’s
Evaluation and Policy Analysis Unit
(EPAU) was for several years the
chief proponent of RTE

WFP, UNICEF, the Humanitarian
Accountability Project, CARE, World
Vision, Oxfam GB, the IFRC, FAO,
WFP and others have all to some
degree taken up the practice.
Source: ISSUE 32 December 2005 Humanitarian Exchange
Magazine Real-Time Evaluation: where does its value lie? by
Maurice Herson and John Mitchell, ALNAP
RTE vs other types of
evaluations
 RTEs look at today to influence this
week’s/month’s programming
 Mid-term evaluations look at the
first phase to influence programming
in the second phase
 Ex-post evaluations are
retrospective: they look at the past
to learn from it
Key Features/Methods
 Semi-structured interviews
 Purposeful sampling –
complemented by snowball sampling
in the field
 Interviews with beneficiary groups
important
 Observation
Methodological Contraints
of RTE

Limited use of statistical sampling
(sample frame)
 Limited use of surveys
 Lack of pre-planned coordination
between humanitarian actors
 Baseline studies usually inexistent
 Attribution (cause and effect)
difficult given the multiplicity of
actors
Source: Brusset, E., Cosgrave, J., & MacDonald, W. (2010). Real-time
evaluation in humanitarian emergencies. In L. A. Ritchie & W. MacDonald
(Eds.), Enhancing disaster and emergency preparedness, response, and
recovery through evaluation. New Directions for Evaluation, 126, 9–20.
Lessons - Advantages
 Timeliness: RTEs bring in an external
perspective, analytical capacity and
knowledge at a key point in a response.
 Perspective: RTEs reduce the risks that
early operational choices bring about
critical problems in the longer term.
 Interactivity: RTEs enable programming
to be influenced as it happens, allowing
agencies to make key changes at an
intermediate point in programming.
Lessons - Challenges
 Utilisation: Weakness in the follow
up on recommendations

Ownership: workers, managers,
beneficiaries?
 Focus: What are the key questions?
 Meeting each partners’ needs for
accountability and learning
 Few RTEs in complex emergencies
Source:Lessons from recent Inter Agency Real Time
Evaluations (IA RTEs) Riccardo Polastro
Digital data and tools
Rationale behind it
 Explosion in the quantity and diversity of high
frequency digital data e.g. mobile-banking
transactions, online user-generated content
such as blog posts and Tweets, online
searches, satellite images, computerized data
analysis.
 Digital data hold the potential—as yet largely
untapped— to allow decision makers to track
development progress, improve social
protection, and understand where existing
policies and programmes require adjustment
Source: Global Pulse, Big Data for Development: Challenges &
Opportunities May 2012, www.unglobalpulse.org
Big Data – UN Initiative
 1) Early warning: early detection of anomalies in
how populations use digital devices and services
can enable faster response in times of crisis
 2) Real-time awareness: Big Data can paint a finegrained and current representation of reality which
can inform the design and targeting of programs
and policies
 3) Real-time feedback: makes it possible to
understand human well-being and emerging
vulnerabilities, in order to better protect
populations from shocks
Potential Uses and Focus
 ILO, UNICEF and WFP, researching
changes in social welfare, especially with
regard to food and fuel prices, and
employment issues
The number of tweets discussing the price of rice in Indonesia in 2011 follows a
similar function as the official inflation statistics for the food basket.
What is Big Data?
"Big Data" is a popular phrase used to
describe a massive volume of both
structured and unstructured data that is
so large that it's difficult to process with
traditional database and software
techniques.
Types of digital data sources
1.
2.
3.
4.
Data Exhaust
Online Information
Physical Sensors
Citizen Reporting or Crowd-sourced
Data
Lessons Learned to Date
Privacy
 Privacy is an overarching concern that
has a wide range of implications vis-à-vis
data acquisition, storage, retention, use
and presentation
− People routinely consent to the collection
and use of web-generated data by simply
ticking a box without fully realising how
their data might be used or misused.
− Do bloggers consent to have their content
analyzed by publihing on the web?
Lessons Learned to Date
Access and Sharing

Much of the publicly available online data
(data from the “open web”) has potential
value for development, there is a great
deal more valuable data that is closely
held by corporations and is not accessible

“The next movement in charitable giving and corporate citizenship may
be for corporations and governments to donate data, which could be
used to help track diseases, avert economic crises, relieve traffic
congestion, and aid development.”
Source: Data Philanntropy where are we now. Andreas Pawelke and
Anoush Rima TatevossianMay 8, 2013
Lessons Learned to date
Analysis
 “conceptualisation” (i.e. defining categories,
clusters);
 selection bias (representative of general
population?)
 “measurement” (i.e. assigning categories and
clusters to unstructured data, or vice-versa)
 “verification” (i.e. assess how well steps 1 and 2
fare in extracting relevant information)
Discussion:
What’s happening in
your organization/
country in terms of
innovation in
evaluation?
What lessons can you
share about what
works and what does
not work?
31
Thank You!
Louise Mailloux –
lmailloux@ggi.ca
Simon Roy – sroy@ggi.ca
32
Appendix: Competency
Domains in Evaluation
1.0 Reflective Practice: competencies focus on the fundamental
norms and values underlying evaluation practice and awareness
of one’s evaluation expertise and needs for growth.
2.0 Technical Practice: competencies focus on the specialized
aspects of evaluation, such as design, data collection, analysis,
interpretation and reporting.
3.0 Situational Practice: competencies focus on the application of
evaluative thinking in analyzing and attending to the unique
interests, issues, and contextual circumstances in which
evaluation skills are being applied.
4.0 Management Practice: competencies focus on the process of
managing a project/evaluation, such as budgeting, coordinating
resources and supervising.
5.0 Interpersonal Practice: competencies focus on people skills, such
as communication, negotiation, conflict resolution, collaboration,
and diversity.
33
Appendix
RTE Distinguishing Features
Real-time
evaluations
Traditional evaluations
Need
In-the-moment feedback at critical
decision points
In-depth analysis in a detailed
report, with the clarity of hindsight.
Types of
deliverables
Frequent in-person meetings and data
summaries.
Full report at a defined end point
and potentially at mid-point.
End goal
Learning what worked and what
Getting the program to work as efficiently didn’t, and using that information to
as possible, as soon as possible.
inform the next iteration of the
program.
Cost
May be more costly due to multiple
rounds of data analysis and meetings.
Since evaluation activities may evolve to
meet changing information needs, costs
are not always as predictable.
Costs are generally more
predictable because you know what
activities will be conducted at the
evaluation outset.
Trade-offs
The analysis will not be as rigorous
because in-the-moment feedback cannot
achieve the same clarity as hindsight.
The analysis will not be available
until midway through or after a
program’s end. However, with the
additional time available, a higher
degree of rigor is possible.
Source: Getting Real About Real-Time Evaluation, Clare Nolan and Fontane Lo , Non-Profit
Magazine, March 29, 2012
Appendix: Types of digital
data sources

(1) Data Exhaust – passively collected
transactional data from people’s use of digital
services like mobile phones, purchases, web
searches, etc., and/or operational metrics and
other real-time data collected by UN agencies,
NGOs and other aid organisations to monitor
their projects and programmes (e.g. stock levels,
school attendance). These digital services create
networked sensors of human behaviour.

(2) Online Information – web content such as
news media and social media interactions (e.g.
blogs, Twitter), news articles, e-commerce, job
posting. This approach considers web usage and
content as a sensor of human intent, sentiments,
perceptions, and want.
Source : www.unglobalpulse.org
Appendix: Types of digital
data sources

(3) Physical Sensors – satellite or infrared
imagery of changing landscapes, traffic patterns,
light emissions, urban development and
topographic changes, etc. This approach focuses
on remote sensing of changes in human activity

(4) Citizen Reporting or Crowd-sourced Data –
Information actively produced or submitted by
citizens through mobile phone-based surveys,
hotlines, user- generated maps, etc. While not
passively produced, this is a key information
source for verification and feedback
Source : www.unglobalpulse.org