Africare M&E Toolkit
An overview
S. Joshua Volle, DrPH, MEd.
Director of M&E and Learning
Africare
29 October 2014
www.africare.org
Why a toolkit?
•
•
•
•
Support learning
Create a common language
Support consistency
Offer up basic guidance
The history
• Consultant in 2013, Madeleine Gauthier
• Vetted with staff early 2014
• Expanded summer 2014
What is in it?
9 modules
1)
2)
3)
4)
5)
6)
7)
Formative Assessments
Monitoring
Evaluation
Project indicators
Use of information
Pre-award phase
Performance Monitoring
Plans (PM)
8) Putting the PMP to Work
9) Methods for data collection
Annex (pg. 71)
–
–
–
–
–
Glossary
Sample Sizes
Consent forms
Qualitative Methods
Indicator Reference sheet
example
– Illustrative Indicator Table
– Advantages and
disadvantages of different
methods
– Evaluation Protocol
Template
Key messages: Purpose of M&E
•
•
•
Learning about:
– The people we serve
– The project (performance, quality, successes)
Telling our story
– Who
– What
– Where
– When
– How much
Answering our questions
– Are we making a difference?
– We carried out a project, “So what?”
Key messages: M&E Framework
(Introduction, pg iv)
Types of Monitoring and Evaluation:
Formative
Assessments and
Research
(concept and design)
Monitoring
(monitoring
procedures, and
outputs;
assessing quality)
Evaluation
(assessing outcome
and
Impact)
Cost-Effectiveness
Analysis
(including
sustainability
issues)
Questions Answered by the Different Types of Monitoring and Evaluation:
• Is an intervention
needed?
• Who needs it?
• What is the best
design?
• To what extent are
planned activities
actually realized?
• How well are the
services provided?
• What outcomes are
observed?
• What do they mean?
• Does the program
make a difference?
• Should program
priorities be changed
or expanded?
• Should resources be
reallocated?
Key messages: Formative Assessment
(Module 1, pg. 1)
• Prior to program development
• Purpose
–
–
–
–
Better understanding of development/health concern
Begin interaction with community
Gather key information for program design
Begin the M&E process
• Gathering data
– Secondary data
– Mix of qualitative and quantitative
– Beneficiaries, stakeholders, key players
Key messages: Monitoring
Module 2, pg. 5
• Primarily examines process/procedures
• Routine data
• Ask yourself, “what information do I need to “show”
what has been done by the program?
• Monitor:
– Procedures
– Coverage
• Data base
• Routine presentations
• Data Quality Assessments (DQA)
Key messages: What does “E” take?
Module 3, pg. 10
•
•
•
•
•
•
•
•
Looking for change over time, or progress
Levels of “E”
– Process/output (Process Evaluation)
– Outcome
(Program Evaluations)
– Impact
What questions do you want to answer?
Comparisons
– Time (baseline and end-line)
– Exposed/Not exposed (counterfactual)
– Objectives
Methods
Resources (human and financial)
Protocol
Human Subjects Protection Committee
Key messages: Indicators
Module 4, pg. 24
Indicator qualities
• Operational
• Reliable
• Valid
• Specific
• Sensitive
• Affordable
• Proxy
• Project customized
• Internationally
standardized
• Level of results
(input, output,
outcome, impact)
Key messages: Data Quality Assurance
Module 8, pg. 50
•
•
•
•
•
Quality of the indicators
Quality of the tools
Capacity of the front line team
Database structure
Quarterly reviews via staff presentations and
discussion
• Annual data audits
• Annual review of M&E plans/PMP
Key messages: Pre-award/Proposals
Module 6, pg. 34
•
•
•
•
•
•
Results Framework
Narratives
Description of baseline to end line
Special Studies
Illustrative Indicator Table
Budget
Key messages: PMP/M&E Plan
Module 7, pg. 42
•
•
•
•
•
•
•
•
•
•
Project logic model
Id of critical assumptions
Monitoring strategy
Evaluation strategy
Special Studies
Baseline data and targets
Performance management task schedule (data flow)
IPTT
Performance Indicator Reference Sheets
Budget
Key messages: Methods and tools for data collection
Module 9, pg. 57
Start with your questions
Protocol
Budget: What will this cost?
Qualitative
• Focus Groups
• In-depth Interviews
• Direct observations
• Mystery Clients
• Hearsay Ethnography
Information
• What is going on and why
Protection of Human Subjects
Quantitative
• Cluster sampling
• LQAS method
• Time-location
• Convenience
• Snowball
• RDS
Information
• What is happen and to what
degree
Key messages: Protection of Human Subjects
•
•
•
•
•
•
•
International requirement for all research
– Need to get exemption if it is non-research
– Obtain local Ethical Review or letter from corresponding
Ministry
Ensures
– High quality studies
– Supports in-country coordination efforts
– Supports incorporation of research into local government
Protected populations
Informed Voluntary Consent
An Africare policy is forthcoming
Training certificate required for everyone involved
Webinar scheduled for 14 January 2015
Key messages: Next Steps
• Africare Global Database
• Africare Global M&E Meeting, July 2015
• Future Conference calls
–
–
–
–
–
–
–
M&E Framework (19 Nov.)
PMPs (17 Dec.)
Research Ethics (14 January 2015)
Research Protocols (28 Jan.)
Quantitative methods (11 Feb.)
Sampling (25 Feb.)
Focus Group Discussions (11 March)