1
USING A MODIFIED
OUTCOME MAPPING
APPROACH TO EVALUATE
HEALTH SYSTEM CHANGE
AND HRH PLANNING
November 2012
Expert Workshop: Cooperation Strategy for the 2nd
Measurement of Regional Goals and Analysis of HRH Policies
Lima, Peru
Gail Tomblin Murphy, RN, PhD
Overview
2
Context
 Boundary partners
 Outcome challenges
 Indicators
 Study design
 Data collection tools
 Stakeholder engagement
 Zambia Case example

Context
3
What change is being proposed?
 Why is it being proposed—what is the
issue prompting it?
 What is the context in which the change…

 Was
conceived?
 Will be implemented?
Boundary partners
4
Who is to be impacted by the proposed
change?
 What policies or other aspects of the
health care system are to be impacted by
the proposed change?

Outcome challenges
5
In what way is the proposed change to impact
the boundary partners?
 Which aspects of their behaviour or
characteristics are expected to be influenced?
What aspects of the system are to be
influenced?
 How will the boundary partners be engaged?

Indicators
6



How can the outcome challenges be measured? What can we
look at to tell whether the proposed change is having an
impact?
 Outcome indicators
What can we measure to determine the degree to which the
change has been implemented? The degree to which
boundary partners have been engaged?
 Process indicators
What else needs to be measured to determine how much any
change in outcomes is due to the proposed change?
 Controls
Study design
7

Can go a long way toward simplifying analysis
and easing interpretation of results (e.g.
distinguishing correlation from causation)


Particularly if randomized control trials, longitudinal,
before-and-after, difference-in-difference designs
are feasible
Highly dependent on the point at which evaluation
is brought into the picture

Earlier the better!
Data collection tools
8






Need to incorporate perspectives of all boundary partners
Need to capture all indicators
 Multiple sources allows for triangulation
Number and type will depend on range of boundary
partners, context, and study design
Use existing, validated tools or adapt/develop new ones
as appropriate
Need to validate with key stakeholders
Pilot testing & psychometric workup important
Stakeholder engagement
9


Evaluation will be most meaningful if those to be
impacted by it are appropriately involved
 Submitting a written report alone not likely to have
much effect
Appropriate partnerships with & engagement of
stakeholders can help ensure:
 Proper consideration of context
 Feasible design
 Good participation/response rates
 Correct interpretation of results
 Increased capacity for research, evaluation
 Meaningful change based on findings
Evaluating the Availability of Adequately
Trained Health Workers in Rural Zambia:
Findings and Recommendations
Zambia HRH Deliberative Forum
Lima Peru
November 14, 2012
Deliberative Forum – Lusaka, ZM
11
About the Project
12

Aimed at informing use of scarce HRH in rural
Zambia
Which retention/recruitment strategies are working?
What changes are needed?
 What are the most pressing health care needs? What
are the largest service gaps?




Partnership between Zambia and Canada
2009-2012
Implemented in two pilot districts: Chibombo and
Gwembe
Funder & Partners


Funded by the Global Health Research
Initiative (GHRI)
Partners:
WHO/PAHO Collaborating Centre
on Health Workforce Planning &
Research
13
ZamCan Team
14


Zambia: Fastone Goma, Miriam Libetwa,
Selestine Nzala, Clara Mbwili, Priscilla
Chisha, Mutale Chimutete, Moses Lungu,
Derrick Hamavhwa, Mercy Mbewe, Ireen
Kabuba, John Mukuka, Viviane Sakanga,
Chilweza Muzongwe
Canada: Gail Tomblin Murphy, Adrian
MacKenzie, Janet Rigby, Amy Gough,
Stephen Tomblin, Rob Alder
Project Milestones
15









November 2009: first ZamCan meeting
May 2010: Instruments drafted
July 2010: Instruments pilot tested/refined
October 2010: Primary data collection
March 2011: Data cleaning and descriptive analysis
October 2011: First deliberative forum
November 2011: Regression and thematic data
analysis
June 2012: Supplementary data collection & future
work planning
October 2012: Second deliberative forum
16
17
18
19
20
21
22
23
Research Objectives
24


Evaluate existing HRH retention and
recruitment strategies in two rural pilot
districts of Zambia (Gwembe and Chibombo)
in terms of their impact on health care
workers and the health care system using
Outcome Mapping
Assess the degree to which the competencies of
the existing health workforce in Gwembe and
Chibombo are suited to specific health needs of
the populations they serve using a needs-based
competency framework
Methods
25

Needs-based competency framework


Outcome mapping


Assess the degree to which the competencies of
workers are suited to specific health needs of the
populations of Gwembe and Chibombo
Evaluate existing HRH retention and recruitment
strategies in Gwembe and Chibombo in terms of
their impact on health care workers and the
health care system)
Capacity building & knowledge transfer
activities integrated throughout
Capacity Building
26





ZamCan team participates in all activities
Team has organized and delivered workshops on
 Outcome mapping/intentional design (May ’10)
 Competency-based framework (May ‘10)
 Quantitative data entry & descriptive analysis (Mar ’11)
 Thematic analysis (November ‘11)
 Regression analysis (November ‘11)
 Analysis using competency-based framework (June ‘12)
 Simulation modeling (October ‘12)
 Regression analysis (October ‘12)
Zambian team led data collection
Graduate student participants in CB workshops, instrument design, data
collection, data analysis
Zambian team members participated in AHSI workshops on NVivo,
monitoring & evaluation, research methodology, policy briefs
Knowledge Translation
27







Policy makers at community, district, and Ministry
levels as part of project team; involved in all activities
Findings validated with health workers in each
district
Presentations/meetings with MoH, CIDA/IDRC,
WHO/AFRO, CHAI, THET
5 presentations at conferences in Canada & Zambia
Knowledge dossier & synthesis developed
Deliberative forum #1 held October ‘11
Deliberative forum #2 planned for October ‘12
Evaluation Design



Outcome mapping
Cross-sectional (necessary as initiatives have all
been in place for some time)
Mixed methods – allows for triangulation



Quantitative/qualitative
Self-report & administrative data
Include perspectives of key stakeholders





Community health committees
Front line health care workers
District administrators
Provincial administrators
National administrators
Competency-Based Health Human Resources Planning Framework
Competencies
Required
Competency Gap
Competencies
Supplied
Level of Service
HHR Gap
Productivity
Epidemiology
(Incidence/
Prevalence etc.)
Need
Demography
(Population Size)
Competency
Prevalence
Activity Rates
Participation Rates
Stock of Providers
Tomblin Murphy, Alder, MacKenzie, Langley, Hickey & Cook (In press)
Requirements
Adapted from Tomblin Murphy, Vaughan, Alder, Alderson, McGeer & Buckley, 2006
and Birch, Kephart, Tomblin Murphy, O’Brien-Pallas, Alder & MacKenzie, 2007
Supply
Research Question #1
30
What formal/informal strategies have been
implemented to retain and recruit health care
workers in rural Zambia?
Zambia Health Worker Retention Scheme (5
components)
Allowances (8 different schemes)
Housing/Utilities (3 different schemes)
Equipment (2 schemes)
Professional development prioritization
Research Question #2
31
What are the outcomes resulting from retention and recruitment
strategies in rural Zambia?
Rated most effective:
1.
Rural hardship allowance
2.
Accommodation/housing
3.
Uniform maintenance allowance
Rated most satisfactory
1.
ZHWRS salary top-up
2.
Water provision
3.
Electrification/solar power
Amounts of allowances deemed insufficient
Retention/recruitment schemes less important than living &
working conditions in predicting health workers’ job satisfaction
and likelihood of leaving their jobs
Research Question #3
32
What are the most pressing health care needs of
people living in rural Zambia?
Administrative data suggested cardiovascular &
respiratory illness were most pressing health
conditions
Focuses mainly on treatment at hospitals
 Not consistent with health workers’ experiences
or community perspectives

Combining admin & community data identified
HIV/AIDS (Chibombo) and malaria (Gwembe) as
most pressing conditions

Research Question #4
33
What are the health care competencies required to
meet the most pressing health needs of people living in
rural Zambia?
Identified list of ~100 competencies for addressing
each condition




Drafted by project team as part of CB workshop
Informed by experience of community and clinical team members
Built from practice protocols where available
Revised based on clinician feedback
Research Question #5
34
What are the current competencies of health care
workers in rural Zambia?
Mostly aligned with population health needs, but
some important gaps exist


e.g. performing lab testing & interpreting results,
performing diagnostic imaging & interpreting
results, taking & interpreting history, performing
physical exam, diagnosing illness, ARV
screening
Most gaps due more to lack of personnel &
number needing care than lack of competency
Research Question #6
35
What strategies have been or are currently being
used to provide health care to people living in rural
Zambia (e.g. ‘skilling up’, task shifting, changes to
team compositions, deployment)?
Task shifting
Overtime
Deployment based on need (i.e. vs.
establishment)
Referral to urban facilities
Some professional development but viewed as
insufficient and largely inaccessible
Limitations



Low numbers of
 Health workers
 Districts
Cross-sectional
‘Stayers’ only
Key Findings
37



In response to concerns about meeting needs of
rural populations, there have been a wide variety of
health worker retention and recruitment strategies
implemented in Zambia over the past few decades.
The level of coordination of these initiatives is
uncertain.
Living and working conditions and the individual
characteristics of health workers are more important
than any of the retention/recruitment schemes in
predicting health worker satisfaction and intention to
stay in their posts.
Key Findings
38

Most health workers interviewed in the pilot
districts report that:
The amounts of allowances need to be updated
to reflect increased costs of living.
 The implementation of retention and recruitment
schemes is viewed as inconsistent.
 Communication between MoH, provinces,
districts, facilities and health workers must be
strengthened.
 Investments in infrastructure to improve living and
working conditions for health workers are needed.

Key Findings
39



HIV/AIDS is the most pressing health condition in
Chibombo; malaria in Gwembe.
Administrative data on mortality and morbidity
conflicts with community-level data and is not
reflective of the full burden of disease
Although the competencies of the existing health
workforces in Chibombo and Gwembe are mostly
aligned with the major health needs of their
populations, some substantial gaps exist. These
are more a result of lack of personnel than lack
of competency among the existing workforce.
Key Findings
40


The most direct strategies undertaken to provide
care to people in rural Zambia have been
undertaken by individual managers & health
workers
 Overtime, task-shifting, deployment
Health workers do not view these as being
sufficient to offset HRH shortages and meet
population health needs
Draft Recommendations Planning
41
1.
2.
Ensure that there are multi-layered orientation processes that
provide newly hired health workers with basic understanding of
context (including pay & incentives), expectations and procedures
of the facility & district to which they are posted (e.g. through
curriculum from the outset of training, basic information from MOH
at graduation, orientation folders; the current induction ceremony at
UTH, group induction on arrival at district office, site visits to
potential posting facility & district).
Explore options for decentralizing HRH recruitment & hiring, e.g. at
provincial and district levels.
 Consider allowing provinces and districts to determine what
establishments should be/which posts are created/which cadres
are hired using approved budgets.
 Perhaps allocate a portion of recruitment funds for this purpose;
districts can specify this in action plans
Draft Recommendations Planning
42
3.
•
4.
5.
Where trained health workers are not available, provide districts with
the authority and allocate resources to facilitate task shifting to provide
necessary services.
Use need and competency assessments to guide task shifting in such
instances.
Utilize a needs-based approach to establishing HRH cadres,
considering differences in needs at the district level.
Utilize needs and competency assessment data to inform ongoing HRH
planning, educational curricula, evaluation and monitoring (e.g. task
shifting guidelines, HRH hiring/recruitment, induction/orientation, and
continuing education)
Draft Recommendations Partnerships
43
Draft Recommendations Resources
44
8.
9.
Facilitate collaboration between government ministries, NGOs (e.g. Area
Development Committees, District Development Coordinating Committees,
traditional authorities etc.), and the private sector to improve infrastructure
(e.g. roads, schools, water/power, telecommunications, housing etc.) in rural
and remote areas such that living and working conditions are improved.
 This process should include the requirement of more explicit and frequent
communication between NGOs and government through Memoranda of
Understanding.
Dedicate resources to allow for targeted continued professional
development activities to be delivered at rural and remote health facilities
without compromising service delivery so that retention/recruitment may be
improved and identified competency gaps reduced.
Draft Recommendations Resources
45
11.
12.
Invest in the creation of a unified health care information
system that includes more systematic collection and
reporting of data on population health needs at the
district level (e.g. consider regular community health
audits, enhanced census methods).
On an ongoing basis, revise incentive amounts to reflect
changes in inflation, cost of living, and ensure they are
appropriate across cadres to duration of training,
experience, and workload/hours worked.
DISCUSSION
46
THANK YOU!
Questions?
GAIL.TOMBLIN.MURPHY@DAL.CA