Supporting continuous
improvement in the replication
process
Getting to Grips with Replication
Seminar 3: Monitoring, evaluation & continuous
improvement
21st March 2013
Kerstin Junge, Tavistock Institute of Human Relations
Introduction
• Why is it useful to think of replication in
continuous improvement terms?
• How can we use monitoring and evaluation as
tools for continuous improvement in the
replication process?
• How do these tools affect impact of the
replication process?
• Conclusions
Why is it useful to think of replication
in continuous improvement terms?
Continuous improvement
Replication of social interventions
Incremental or breakthrough process
Incremental process
Phases of small scale testing, measuring, acting
(wider roll-out or ongoing testing) in commonly
used PDSA cycle:
 Plan identify an opportunity and plan for
change
 Do: implement the change on a small scale
 Study: use data to analyse the results of the
change and determine whether it made a
difference
 Act: if the change was successful, implement
t on a wider scale and continuously assess
your results. If the change did not work,
begin the cycle again.
Relies on monitoring data to inform change
generated from exercises
Takes place in three phases:



Creating or identifying an innovation for
replication (including evaluation to show it
works)
Deciding to adopt an innovation for
replication (including identifying funding)
Implementing, replicating, and measuring
success of the intervention (in waves)
Uses both monitoring and evaluation data to
inform adoption and success of replication
process
How can we use monitoring and evaluation as
tools for continuous improvement in the
replication process?
Monitoring and evaluation for continuous improvement in the
replication process: the case of Realising Ambition
Replication Phase 1: Identifying innovations for replication and deciding to adopt
Requirement to
engage with existing
impact evidence as
integral part of
programme design
Monitoring and evaluation for continuous improvement in the
replication process: the case of Realising Ambition
Replication Phase 1: Identifying innovations for replication and deciding to adopt
Requirement to
engage with existing
impact evidence as
integral part of
programme design
“The standards of evidence exercises
in the application forms (…) helped
us to really get to know [the
intervention). They helped us test
[it] in a systematic way to see if we
wanted to use it. It helped me to
look forward and see what the
evidence said. (…) the evidence was
stronger when teachers delivered
the intervention rather than external
people so we are using teachers.”
Application process challenged us,
(…) it asked for some quite robust
information and we had to
organise ourselves to get that. [It]
challenged us to smarten up back
end, e.g. pulling the logic model
together; we knew it and played
around with doing that and had
not expressed that in a document.
(…) we actually said even if we
don’t get that, [the application
process] actually helped us
organise ourselves (…).”
“The application stage forced me to ask
the… team: why are we doing it in this
way? How do we know it works? It was
challenging to ask the organisation. (…)
These are hard questions. Realising
Ambition clarified the thinking in the
organisation.”
Monitoring and evaluation for continuous improvement in the
replication process: the case of Realising Ambition
Replication Phase 1: Identifying innovations for replication and deciding to adopt
Requirement to
engage with existing
impact evidence as
integral part of
programme design
“The standards of evidence exercises in the
application forms (…) helped us to really
get to know [the intervention). They
helped us test [it] in a systematic way to
see if we wanted to use it. It helped me to
look forward and see what the evidence
said. (…) the evidence was stronger when
teachers delivered the intervention rather
than external people so we are using
teachers.”
“The application stage forced me
to ask the… team: why are we
doing it in this way? How do we
know it works? It was challenging
to ask the organisation. (…) These
are hard questions. Realising
Ambition clarified the thinking in
the organisation.”
Application process challenged us, (…)
it asked for some quite robust
information and we had to organise
ourselves to get that. [It] challenged us
to smarten up back end, e.g. pulling
the logic model together; we knew it
and played around with doing that and
had not expressed that in a document.
(…) we actually said even if we don’t
get that, [the application process]
actually helped us organise ourselves
(…).”
Creating preconditions for impact through: better and explicit knowledge of the
intervention and effective delivery, clearer understanding of causal pathways, preparing
the organisation for replication
Monitoring and evaluation for continuous improvement in the
replication process: the case of Realising Ambition
Replication Phase 1: Identifying innovations for replication and deciding to adopt
Requirement to
engage with existing
impact evidence as
integral part of
programme design
“The standards of evidence exercises in the
application forms (…) helped us to really
get to know [the intervention). They
helped us test [it] in a systematic way to
see if we wanted to use it. It helped me to
look forward and see what the evidence
said. (…) the evidence was stronger when
teachers delivered the intervention rather
than external people so we are using
teachers.”
“The application stage forced me
to ask the… team: why are we
doing it in this way? How do we
know it works? It was challenging
to ask the organisation. (…) These
are hard questions. Realising
Ambition clarified the thinking in
the organisation.”
Application process challenged us, (…)
it asked for some quite robust
information and we had to organise
ourselves to get that. [It] challenged us
to smarten up back end, e.g. pulling
the logic model together; we knew it
and played around with doing that and
had not expressed that in a document.
(…) we actually said even if we don’t
get that, [the application process]
actually helped us organise ourselves
(…).”
Creating preconditions for impact through: better and explicit knowledge of the
intervention and effective delivery, clearer understanding of causal pathways, preparing
the organisation for replication
Process evaluation
Sense-making and capturing of lessons learnt to support future replication investments
Analysing programme architecture, programme design and application process
Monitoring and evaluation for continuous improvement in the
replication process: the case of Realising Ambition
Replication phase 2: first wave implementation
Monitoring system
Keeping programme on track, understanding challenges, improving delivery
Monitoring and evaluation for continuous improvement in the
replication process: the case of Realising Ambition
Replication phase 2: first wave implementation
Monitoring system
Preparation for (new)
impact evaluations
Keeping programme on track, understanding challenges, improving delivery
“One of the most
significant things was (…)
looking at how we
constructed the logic
model. (…) Now it's much
easier to be specific.”
Webinars “(…) spark
thinking which can
feed into plans and
keep them on track.”
“Some of the work that is being on
theory of change; understanding
replication and fidelity. What
[intervention owners] do is set up
the programme, train the staff and
give us assistance. But they don’t
engage staff in discussing in
understanding programme fidelity
and replication. “
Monitoring and evaluation for continuous improvement in the
replication process: the case of Realising Ambition
Replication phase 2: first wave implementation
Monitoring system
Preparation for (new)
impact evaluations
Keeping programme on track, understanding challenges, improving delivery
“One of the most
significant things was (…)
looking at how we
constructed the logic
model. (…) Now it's much
easier to be specific.”
“Some of the work that is being
on theory of change;
understanding replication and
fidelity. What [intervention
owners] do is set up the
programme, train the staff and
give us assistance. But they don’t
engage staff in discussing in
understanding programme
fidelity and replication. “
Webinars “(…) spark
thinking which can feed
into plans and keep them
on track.”
Probability of achieving desired impact increases through: greater specificity of intervention
and expected outcomes as well as improved understanding the importance of fidelity in
delivery.
Monitoring and evaluation for continuous improvement in the
replication process: the case of Realising Ambition
Replication phase 2: first wave implementation
Monitoring system
Preparation for (new)
impact evaluations
Keeping programme on track, understanding challenges, improving delivery
“One of the most
significant things was (…)
looking at how we
constructed the logic
model. (…) Now it's much
easier to be specific.”
Webinars “(…) spark
thinking which can feed
into plans and keep them
on track.”
“Some of the work that is being
on theory of change;
understanding replication and
fidelity. What [intervention
owners] do is set up the
programme, train the staff and
give us assistance. But they don’t
engage staff in discussing in
understanding programme
fidelity and replication. “
Probability of achieving desired impact increases through: greater specificity of intervention
and expected outcomes as well as improved understanding the importance of fidelity in
delivery.
Process evaluation
Sense-making and capturing of lessons learnt to support future replication investments
Identification and (early) definition of replication models; overview of types and benefits of impact evaluation and
other support ; replication progress indicators
Effects on impact
• Creating preconditions for impact:
– Better and explicit knowledge of the intervention and
effective delivery
– Clearer understanding of causal pathways
– Preparing the organisation for replication
• Greater probability of achieving desired impact:
– Greater specificity of intervention and expected outcomes
– Improved understanding the importance of fidelity in
delivery
– Data, knowledge and learning to improve replication
delivery
Conclusions
• Understanding replication as a continuous improvement process
offers a ‘practical’ framework to guide activities, as well as tools
• Monitoring and evaluation tools should be an integral part of the
replication process from the start, both at project level and at
programme / policy level
– Creates a structure to support constructive engagement with
evidence, evidence based practice, learning from experience
– And through this supports learning and continuous improvement
• It means understanding monitoring and evaluation at least as much
about learning as about ‘auditing’
• Possibility of not only ‘single loop learning’ but also ‘double loop
learning’ (Argyris and Schön)?
– Not just continuous improvement of implementation (revisiting ‘action
strategies’)
– But also revisiting values [mental models, beliefs, intentions]
underpinning replication actions and strategies (‘governing variables’)