Historical and emerging
theoretical foundations for
Monitoring and Evaluation
in South Africa:
Implications for practice.
Dr Mark Abrahams, University
of Cape Town, March 2007
Introduction
• Monitoring and evaluation (M&E) relatively
new (Louw, 1998; Potter, 1999; Potter and
Kruger, 2001; Babbie and Mouton, 2001).
• Historical development of programme
evaluation in South Africa
• The emerging theoretical foundations
• Implications for practice.
Analytic Framework
• Shadish, Cook and Leviton (1991);
Shadish (2005)
• Social programmes and policies aimed at
improving the welfare of individuals,
organisations and society .
• Theory - connotes a body of knowledge
that organizes, categorizes, describes,
predicts, explains, and otherwise aids in
understanding and controlling a topic
Questions
• What good is a fine evaluation of an
important attempt to solve a puny
problem?
• What good is a fine evaluation of a puny
attempt to solve a serious problem?
• What good is a fine evaluation of a
program that solves a serious problem if
the results are not stored and used to
ameliorate the problem?
Point (s) of departure
• Evaluation is just one part of a complex,
interdependent, non-linear set of problemsolving activities
• Evaluation research is more than the
application of methods, more than an
isolated academic concern and it operates
in the context of the social policy and
public administration movements.
Five ‘fundamental issues’
•
•
•
Social programming: the ways that social
programs and policies develop, improve,
and change, especially in regards to
social problems.
Knowledge construction: the ways
researchers learn about social action.
Valuing: the ways values can be
attached to program descriptions.
Cont.
•
•
Knowledge use: the ways social science
information is used to modify programs
and policies.
Evaluation practice: the tactics and
strategies evaluators follow in their
professional work, especially given the
constraints they face.
Three periods in S.A.
• Pre - 1994
– Anti-apartheid, anti-government, liberation
• Post – 1994
– Reconstruction and development
• 1999 to date
– Governance, sustainability, accountability
The Drivers
• The Democratic state
–
–
–
–
The Presidential address
Power and accountability
Benchmarking
Legislation – Performance management
• International and regional agreements
–
–
–
–
Millennium goals
NEPAD
Sustainable Development
International Exchange/ globalisation
Drivers (cont.)
•
•
•
•
•
•
•
•
•
•
•
•
Economic growth
Sustainable development
Unemployment
Infrastructure and spatial development
HIV/AIDS and other health issues
Equity and redress
Poverty alleviation
Safety and security
Skills development
Climate change and environmental concerns
Civil and ‘uncivil’ society
Free press
Emerging theories of social
programmes
• Programmes are policy-connected interventions
underpinned by the Constitution and Bill of
Rights
• They are constructed with clear objectives,
timelines and action plans and resources are
allocated to achieve targets
• Integrated approach emphasised, recognising
the multi-faceted nature of social problems.
• Social engineering
Theories of knowledge construction
• Multi-methodological approach with
emphasis on reliable data
• Preference for quantitative data
• Audit and compliance strategies
• Monitoring vs evaluation
Theories of valuing
• Stakeholder dilemma
– Who are the ‘holders’?
– What is the stake ‘steak’ for each holder?
– What is at stake for each holder?
– Who holds?
– Who holds what?
– Who suffers?
– Who gains
– What are the consequences of non-delivery?
Theories of Use
• Centralising structures (GWM&E) to
ensure and enable use.
• Advocacy purposes
• Early warning systems
• Performance measurement
• Policy development
Theories of practice
• Objectivity through indicators
• Elements of the following
– Manipulable solution theory – reducing
uncertainty about programme effects
– Generalisable explanations theory –
knowledge of complex interrelationships
among multiple causal determinants
– Stakeholder service theory - tailored to
information needs of stakeholders
Implications for M&E in
S.A.
Theory of Social Programmes
• Anatomy of social programmes (Pawson, 2006)
–
–
–
–
–
–
–
Intervention as theory
Interventions as active
Intervention chains thickly populated
Intervention chains non-linear can go into reverse
Interventions embedded in multiple social systems
Interventions leaky (change over time)
Interventions are open systems and change the
conditions that make them work in the first place.
Theory of Knowledge
• Ontology, Epistemology and methodology
essential topics
• Knowledge of causation and
generalisation
• All methods are potentially fallible
• The limitations and potential of the single
case study – multiple studies
• Parameters that influence the choice of
methods
Theory of Valuing
• Prescriptive and descriptive valuing
• Agreed upon principles
• Standards of performance and measuring
performance
• Synthesizing results?
• Stakeholder interest vs criteria of merit
Theory of Use
•
•
•
•
•
•
•
•
The challenges of instrumental use
Promoting use
Facilitating use
Influencing use
Conceptual use of findings
Communicating results
Agreement about use
Multiple stakeholders
End here today
• Thank you
Theory of Practice
Historical overview
• In 1999, Potter reported that since the
1970s, an estimated R6 billion of overseas
and local funding had been used by
various NGOs to engage in development
projects in various sectors of society, and
only a small fraction of these programmes
had been evaluated.
History cont.
• Potter and Kruger (2001) illustrate with a 1998
PsychLIT database (Silverplatter International
N.V.) search for references from 1974 to 1997
that produced only “fifteen articles and one book
chapter that were indexed under ‘programme
evaluation’ and ‘Africa’, out of a total of 4 721
articles and books that were indexed under
‘programme evaluation’ (p. 192).” Eleven of the
articles, including the book chapter were from
South Africa, representing less than half a
percent of the total available on that database.
History cont.
• Louw’s (1998) ‘informal’ survey of
programme evaluation and the types of
evaluations produced in South Africa
revealed that qualitative approaches to
programme evaluation was the dominant
one and experimental, including quasiexperimental designs were present but not
in great numbers. He found that most of
the work involved formative rather than
summative evaluation.
Historical Elements of P.E.
• Pre 1994
–
–
–
–
–
–
–
Project Driven
Donor requirements
Mostly formative
Personality driven
Vague objectives
Liberation rhetoric
Dwindling resources
• Post 1994
–
–
–
–
–
–
Programme focused
Planning requirements
Clear objectives
Logical frameworks
Policy frameworks
Accountability
principles –
International agencies
Post 1999
•
•
•
•
•
•
•
•
Policy - programme relationship
Integration at conceptual level
Accountability structures and frameworks
Need for sophisticated information management
systems
Managing development implementation
Large scale implementation
International treaties and agreements
Sustainable development
Historical theoretical foundations
• Theory of social programmes
– Focused projects can bring about desired
change
– Desired change can be brought about within
specified time
– Participants involved and eager for the same
reasons
– Clear lines of contestation/ conflict
– Local knows best, as long as people are
involved
Historical theoretical foundations
• Theory of use
– Instrumental intent