m-TERG
Modeling mHealth Impact on Neonatal Survival
using the Lives Saved Tool (LiST)
Youngji Jo (Alain B. Labrique)
December 11, 2014
PhD Candidate
Health Systems Program
International Health Department
Johns Hopkins Bloomberg School of Public Health | JHU Global mHealth Initiative
Outline
• Why Lives Saved Tool (LiST)?
• What is LiST?
• Modeling LiST for mHealth Impact on Neonatal Survival in
Bangladesh and Uganda
• Caveats
• Contributions
• Acknowledgements
Motivation
Challenges of mHealth “pilotitis”
VS.
Interest and needs for mHealth scaling up
Increasing mHealth impact evaluation/evidence
•
•
•
•
•
•
Free et al (2013)—Systematic review / Meta Analysis
Zuvorac et al (2009)—RCT study on ART adherence
Lund et al (2013;2014)—RCT study on ANC visits
Higgs et al (2014) – Recommendations for Research and Programs
Anglada-Martinez (2014) – SMS for Adherence
WHO mTERG Review and Recommendations Process
Why LiST? mHealth as catalyst to existing interventions
• Preventive and curative public health interventions of known
efficacy exist and are well described.
“mHealth helps overcome barriers to reaching effective
coverage of interventions of known efficacy”
• Coverage as a potential primary measure of mHealth impact
• By promoting demand-side—eg. SMS text for IEC
• By promoting supply-side—eg. Supply chain management, workflow
management
• By promoting access between the demand and supply
Lives Saved Tool (LiST, www.livessavedtool.org)
an evidence-based modeling tool
What is LiST?
• Initiated by the “Bellagio” modeling exercise and the Lancet Child Survival Series (2003)
• Developed by the Child Health Epidemiology Reference Group (CHERG) for the WHO
and UNICEF;
• Managed by the JHU Bloomberg School of Public Health’s Institute for International
Programs
Objective: Estimate lives saved when introducing or scaling up key interventions
Intervention types
•
•
•
•
•
•
Maternal, fetal, neonatal, child
Periconceptional, antenatal, birth,
immediate postnatal, child
Preventive, curative
Nutritional, vaccination, water/
sanitation, treatment etc.
Risk factors: Cause-of-death specific
External (family planning, AIDS),
internal (all others)
Data needs (default data)
•
•
•
•
Population data and trends
(UN Population Division 1950-2050)
(DemProj)
Mortality rates/ratios (most recent)
Cause of death structure
(WHO/UNICEF/CHERG (2010)
Intervention coverage (0-100%)
(DHS/MICS/JMP/WHO-UNICEF (close
to 2010)
How it works?
country
specific
health
status
coverage of
intervention
1) Select countries
2) Select key interventions
• Individual intervention
• Combined interventions
Efficacy of
intervention
Number of deaths averted
3) Determine target year
• Based on policy goal
• Based on achievable target
4) Determine target coverage
• Absolute target
• Relative target
(by cause; by intervention)
Impact can be categorized by: (i) Year of implementation; (ii) Cause of death; (iii) Population
sub-group (e.g. mothers, newborns, children under 5 years); and (iv) Intervention
Modeling analysis
Select countries
• Bangladesh and Uganda (NMR as 27 and 26 per 1,000 live births as of 2010)
Select key interventions
• Four key interventions—Antenatal care (ANC), Skilled birth attendance and/or Faci
lity delivery (SBA/FD), Breastfeeding promotion (BF), and Postnatal care (PNC)
• Bundled packages of interventions based on common mHealth strategies (“BF & P
NC” and “ANC, SBA/FD, BF & PNC”, called All-combined).
• The optimal mix of services and tradeoffs in coverage by comparing four individual
and two bundled interventions scenarios, as described above.
Determine target year--2015
Determine target coverage
• Multiplying the baseline coverage of each intervention in 2011 by 110%, 130% and
150% in a relative manner, assuming linear trends of coverage increase over time.
LiST Interventions and Coverage Increase Scenarios in Bangladesh and Uganda (input
parameters of baseline year in 2011 and target year in 2015)
LiST Interventions (selected)
Pregnancy
Childbirth
Antenatal care (ANC 4+)
Skilled birth attendance*
Facility delivery*
(Clinic and Hospital)
Unassisted
deliveries**
Home
deliveries**
Assisted
(% of all
deliveries at
deliveries)
home**
Facility
deliveries**
(% of all
deliveries)
Essential care
**
BEmOC**
CEmOC**
Promotion of breastfeeding
Breastfeeding
promotion an Exclusive breastfeeding**
d prevalence Predominant breastfeeding*
*
(<1 month)
Partial breastfeeding**
Preventive postnatal care
(Thermal care, Clean
Preventive
postnatal practice)
Baseline
(2011)
25.5
31.7
28.8
Bangladesh
Projected coverage increase (2015)
10%
30%
50%
28.1
33.2
38.3
34.8
41.2
47.6
31.7
37.4
43.2
Baseline
(2011)
47.6
58.0
57.4
Uganda
Projected coverage increase (2015)
10%
30%
50%
52.4
61.9
71.4
63.8
75.4
87.0
63.1
74.6
86.1
68.3
65.2
58.8
52.4
42.0
36.2
24.6
13.0
2.9
3.1
3.8
4.4
0.6
0.7
0.8
0.9
25.9
15.8
18.7
21.6
14.3
15.8
18.6
21.5
0.0
2.9
36.3
9.5
6.3
39.9
11.2
7.5
47.2
13
8.6
54.5
8.6
34.4
34.8
9.5
37.9
38.3
11.2
44.8
45.2
12.9
51.7
52.2
84.5
5.9
84.9
5.7
85.6
5.5
86.3
5.2
89.9
5.0
90.1
4.9
90.6
4.7
91.0
4.4
9.6
29.6
9.4
32.6
9.0
38.5
8.5
44.4
5.1
2.8
5.0
3.1
4.8
3.6
4.5
4.2
Neonatal Mortality Impacts by Various MNH Interventions and Coverage Scenarios in Bangla
desh and Uganda in 2015
Interventions Illustrative mHealth Strategies
ANC
SBA/FD
Data collection and management
(e.g. Risk assessment and classific
ation, Vital events tracking,
adherence reminder); SMS texting
for health promotion and
scheduled visits reminder
Emergency medical referral (e.g. r
eferral calling)
BF
SMS texting for health promotion
PNC
SMS texting for health promotion
BF & PNC
SMS texting for health promotion
All-combined Data collection and management (
: ANC,SBA/F e.g. Risk assessment and classifica
D, BF &PNC tion, Vital events tracking, adhere
nce reminder); SMS texting for he
alth promotion and scheduled visi
ts reminder; Emergency medical r
eferral (e.g. referral calling)
Coverage
increase
by 2015
Projected number of neonatal lives saved
Bangladesh
Uganda
2013
2014
2015
2012
2013
2014
2012
10%
30%
50%
0
0
0
0
1
1
0
1
2
0 (1)***
1 (1)
2 (1)
1
3
5
2
7
11
3
10
17
5 (0.8)
14 (0.79)
23 (0.78)
10%
30%
50%
10%
30%
50%
10%
30%
50%
10%
30%
50%
10%
30%
50%
1038
1530
2021
4
12
20
98
290
482
102
302
502
1141
1820
2512
2055
3021
3984
8
24
41
194
576
958
202
600
999
2258
3587
4934
3048
4470
5882
12
36
61
289
858
1427
301
894
1487
3346
5298
7262
4016 (0.74)
5877 (0.74)
7717 (0.74)
16 (0.75)
48 (0.75)
80 (0.75)
383 (0.74)
1135 (0.74)
1888 (0.74)
399 (0.74)
1183 (0.74)
1968 (0.74)
4405 (0.74)
6951 (0.74)
9496 (0.74)
381
1141
1892
2
5
8
0
0
0
5
16
27
388
1160
1924
776
2312
3811
3
8
16
6
15
26
11
32
56
790
2349
3874
1187
3512
5753
5
14
26
12
30
54
18
50
86
1208
3569
5847
1611 (0.76)
4738 (0.76)
7714 (0.75)
7 (0.71)
18 (0.72)
35 (0.77)
17 (1)
47 (1)
83 (1)
24 (0.79)
68 (0.76)
118 (0.77)
1639 (0.76)
4814 (0.76)
7839 (0.75)
2015
Skilled birth attendance and increased facility delivery provide the biggest mortality impact
Bangladesh
Uganda
Neonatal Mortality Impacts by Various MNH Interventions and Coverage Scenarios in Bangladesh and
Uganda in 2015—individual intervention
Breast Feeding Promotion & Postnatal Care provide relatively greater mortality impact in
Bangladesh compared to Uganda
Bangladesh
Uganda
Neonatal Mortality Impacts by Various MNH Interventions and Coverage Scenarios in Bangladesh an
d Uganda in 2015—bundled packages
Causes of Neonatal Deaths at Baseline in 2011 and Estimated Causes of Neonatal
Deaths Averted (with 50% coverage scenario) in 2015 in Bangladesh and Uganda
Causes of Neonatal Deaths Avered (2015)
Coverage increase
ANC (50%) SBA/FD (50%)
by 2015
Neonatal-Diarrhea
Neonatal-Sepsis
Neonatal-Pneumonia
Neonatal-Asphyxia
Neonatal-Prematurity
Neonatal-Tetanus
Neonatal-Cogenital a
nomalies
Neonatal-Other
BF (50%)
PNC (50%)
All-combined: AN
BF & PNC (50%) C,SBA/FD, BF &P
NC (50%)
B
U
B
U
17
13
17
13
787
40
1372
723
63
22
63
22
0
0
2382
3693
1082
41
5,627 3370
19
1
35
19
B
0
2
0
0
0
0
U
0
23
0
0
0
0
B
0
625
0
2382
4692
17
U
0
665
0
3693
3338
18
B
17
0
63
0
0
0
U
13
0
22
0
0
0
B
0
787
0
0
1082
19
U
0
40
0
0
41
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Caveats
• LiST modeling assumptions—ex. Single cause of death, delivery care
(BEmOC; CEmOC)
• The level of mortality impact is influenced by the reported initial baseline
coverage level and evidence from standard care practice.
• Like many other modeling tools, the analysis (and LiST) does not
systematically consider health systems constraints in achieving the target c
overage.
• This analysis does not consider effective coverage—quality of
sub-components of a given intervention package.
• It is important to ensure that the findings serve as added guidance for
informed discussion / prioritization and not drive the selection of
strategies in a vacuum.
Contributions
• Impact estimation: Estimates the neonatal mortality impact by
improving coverage of specific MNH interventions around different
target coverage levels, derived from rigorous, efficacy-trial driven
expectations.
• Prioritization of strategy: Determine mHealth programs and
strategies in obtaining the likely highest-impact interventions for
prioritization, considering the unique potential for synergy across m
ultiple areas that mHealth solutions typically allow
• Planning & Evaluation: Enable projects to set benchmarks and
monitor progress against modeled targets, potential endpoints,
and guidance for cost-assessment
Validating the models – next steps…
Empirical test with prospective impact evaluation research
• To validate these models and provide more informed projections on potential
coverage change associated with various mHealth solutions across different
contexts over time.
Effective coverage
• Develop metrics for how mHealth can improve *quality* (completeness) of a given
intervention?
Scaling up/implementation science
• How can mHealth *rapidly* expand a particular intervention?
Barriers to coverage—Heath systems constraints
• Geographical access
• Medical supplies availability
• Provider’s compliance
• Patient’s adherence
Mehl G, Labrique AB, Science 2014
Acknowledgements
•
•
•
•
•
•
Alain B. Labrique, JHSPH
Amnesty E. Lefevre, JHSPH
Garrett Mehl, WHO
Teresa Pfaff, JHSN
Neff Walker, JHSPH
Ingrid K. Friberg, JHSPH
• JHU Global mHealth Initiative
• WHO mHealth Technical Evidence Review Group
References
• USAID. LiST Manual. Available:http://www.jhsph.edu/departments/international-h
ealth/centers-and-institutes/institute-for-international-programs/_documents/ma
nuals/list_manual.pdf.
• Labrique AB, Vasudevan L, Kochi E, Fabricant R, Mehl G (2013) mHealth innovation
s as health system strengthening tools: 12 common applications and a visual frame
work. Global Health: Science and Practice 1: 160-171
• Fischer Walker CL, Friberg IK, Binkin N, Young M, Walker N, et al. (2011) Scaling up
diarrhea prevention and treatment interventions: a Lives Saved Tool analysis. PLoS
Med 8: e1000428
• Walker N, Tam Y, Friberg IK (2013) Overview of the Lives Saved Tool (LiST). BMC Pu
blic Health 13: S1
• Fox MJ, Martorell R, van den Broek N, Walker N (2011) Assumptions and methods i
n the Lives Saved Tool (LiST). Introduction. BMC Public Health 11 Suppl 3: I1
• Bryce J, Friberg IK, Kraushaar D, Nsona H, Afenyadu GY, et al. (2010) LiST as a cataly
st in program planning: experiences from Burkina Faso, Ghana and Malawi. Int J Ep
idemiol 39 Suppl 1: i40-47.
Project Country Organization Interventions
Wired Mo Tanzania
thers
Maternal Ethiopia
and Newb
orn Health
in Ethiopia
Partnershi
p (MaNHE
P)
E-IMCI
Tanzania
mHealth strategies
mHealth benefit/impact evidence on service provision
Danish Internati (i) Family planning (i) Data collection and m “The mobile phone intervention was associated with an increase in antena
onal Developme (ii) Behavior chang anagement (e.g. Risk ass tal care attendance. In the intervention group 44% of the women received
nt Cooperation, es through Inform essment and classificatio four or more antenatal care visits versus 31% in the control group (odds ra
ation, Education a
University of Co
n, Vital events tracking, tio (OR), 2.39; 95% confidence interval (CI), 1.03-5.55). There was a trend
nd Communicatio
penhagen
n (IEC) (iii) Antenat adherence reminder) (ii) towards improved timing and quality of antenatal care services across all s
al care(ANC)/Expa SMS texting for health pr econdary outcome measures although not statistically significant.” [22]
nded Program on I omotion and scheduled “The mobile phone intervention was associated with an increase in skilled
mmunization (EPI) visits reminder (with mo delivery attendance: 60% of the women in the intervention group versus 4
/Postnatal care(PN bile phone voucher com 7% in the control group delivered with skilled attendance. The interventio
C) (iv) Skilled birth
ponents)
n produced a significant increase in skilled delivery attendance amongst ur
attendance(SBA)/F
ban women (OR, 5.73; 95% CI, 1.51–21.81), but did not reach rural women
acility delivery (FD
.” [34]
)
“The perinatal mortality rate was lower in the intervention clusters, 19 per
1000 births, than in the control clusters, 36 per 1000 births. The interventi
on was associated with a significant reduction in perinatal mortality with a
n OR of 0.50 (95% CI 0.27-0.93). Other secondary outcomes showed an ins
ignificant reduction in stillbirth (OR 0.65, 95% CI 0.34-1.24) and an insignifi
cant reduction in death within the first 42 days of life (OR 0.79, 95% CI 0.3
6-1.74).” [40]
(i)
Family
planning
University Resea
(i) SMS texting for healt “Women who had additionally attended 2 or more CMNH meetings with f
rch Co., LLC, Qu (ii) Behavior chang h promotion and schedul amily members and had access to a health extension worker’s mobile pho
ality Improveme es through IEC (iii) ed visits reminder (e.g. p ne number were 4.9 times more likely to have received postnatal care (OR
ANC/EPI/PNC
nt Advisor for th
romotion of community , 4.86; 95% CI, 2.67-8.86; P _.001).” [41] “Notification of health extension
e Maternal and
maternal and newborn h workers for labor and birth within 48 hours was closely linked with receipt
Newborn Health
ealth family meetings an of postnatal care. Women with any antenatal care were 1.7 times more lik
in Ethiopia Partn
d labor and birth notifica ely to have had a postnatal care visit (OR, 1.67; 95%; 95% CI 1.10-2.54; P _
ership
tion)
.001).” [41]
D-Tree
(i) ANC/EPI/PNC (ii (i) Point of care decision “For all ten critical IMCI items included in both systems, adherence to the
) Behavior changes support through complia protocol was greater for eIMCI than for pIMCI. The proportion assessed un
through IEC
nce to IMCI protocols
der pIMCI ranged from 61% to 98% compared to 92% to 100% under eIMC
I (p < 0.05 for each of the ten assessment items).” [25]
Project M Zambia, UNICEF
wana
Malawi
(i) HIV-antiret (i) Data collectio “ SMS delivery of results can increase turnaround times by 50% on averag
roviral therap n and managem e, with a greater positive impact in rural facilities” [42]
y (ART) surveil ent (ii) SMS text
lance and trea ing for health pr
tment
omotion and sch
eduled visits rem
inder
Better Borde Thailand- Mahidol Universit (i) Family plannin (i) Data collection an “ANC/EPI coverage in the study area along the country border improved; numbers of AN
r Healthcare Burma
y, Thailand
g
d management (ii) C and EPI visits on-time as per schedule significantly increased; there was less delay of a
Program
(ii) ANC/EPI/PNC SMS texting for heal ntenatal visits and immunizations” [43]
th promotion and sc
heduled visits remin
der
RapidSMS-M Uganda Ministry of Healt (i) Family plannin (i) Data collection an Study reported “a 27% increase in facility based delivery from 72% twelve months befor
CH
h Uganda, UNICE g (ii) Behavior ch d management (ii) e to 92% at the end of the twelve months pilot phase.” [44]
anges through IE SMS texting for heal
F
C (iii) ANC/EPI/P
th promotion and sc
NC (iv) SBA/FD
heduled visits remin
der
Rural Extend Uganda Ministry of Healt (i) Behavior chan (i) Emergency medic “improved communication and transportation links between the Traditional Birth Atten
ed Services a
h, UN Population ges through IEC ( al referral (e.g. refer dants (TBAs) and the health posts resulted in increased and more timely referrals as well
nd Care for U
Fund and the Uga ii) ANC/EPI/PNC ( ral calling) with tran as the improved delivery of healthcare to a large number of pregnant women”… “The in
ltimate Emer
nda Population S iii) -SBA/FD
sportation services creased number of deliveries under trained personnel and increased referrals to health
gency Relief (
ecretariat
units led to a reduction of about 50 percent in the maternal mortality rate (MMR) in thr
RESCUER)
ee years” [45]
M4RH
Kenya Ta USAID, FHI 360’s (i) Family plannin (i) Data collection an User interviews reported various positive responses including “the text messaging servic
nzania, PROGRESS (Progr g (ii) Behavior ch d management (ii) e was perceived as being private, convenient, and cost-effective.” [33]
am Research for anges through IE SMS texting for heal
Strengthening Se C
th promotion and sc
rvices)
heduled visits remin
der
PREVEN
Aceh Besar
Midwives
MAMA
MOTECH
Peru
Cell-Preven
(i) Sexual and re
productive healt
h surveillance a
nd service delive
ry
(i) Data collection an Lessons include “Two-way information systems are more than just collecting data. They
d management (ii) provide feedback and support to health care workers in the field. Many times, only man
SMS texting for heal agers have information that allows them to monitor and evaluate data but these system
th promotion and sc s do not prove any aggregate value to health care workers in the field. A well-designed i
heduled visits remin nformation system has to support and enhance the performance of all user levels in a se
der
cure environment.” [30] “Prahalad (2005) has reported that health workers in some dev
eloping countries spend as much as 40% of their time filling out forms, compiling and co
pying data from different pro-grams (e.g., tuberculosis, malaria, HIV/AIDS, etc.). By choo
sing the most appropriate information technology, we can avoid duplication and deploy
different devices—i.e., cell phones, Internet—to report from each public health program
.” [30]
Indonesia UNICEF, UNFPA, (i) Behavior chan (i) Data collection an “Findings from the project indicate that the mobile phone has proven to be an effective
and World Vision ges through IEC ( d management (ii) and efficient device for facilitating smoother communication, and allowing speedier eme
ii) ANC/EPI/PNC ( SMS texting for heal rgency response. The system also aids in gathering and disseminating health-related info
th promotion and sc
iii) SBA/FD
rmation to midwives, who in turn convey this knowledge to the patient community.” [13
heduled visits remin
der (iii) Emergency ]
medical referral (e.g
. referral calling)
Banglade mHealth Alliance (i) Family plannin (i) Data collection an MAMA Bangladesh Aponjon project represented “a 37% increase over a 2011 national
sh, India,
g (ii) Behavior ch d management (ii) baseline of 26% attending four ANC visits. It is also important to note that 45% of the Ap
and Sout
anges through IE SMS texting for heal onjon subscribers went to a facility for delivery and 32% chose safe delivery at home” [3
h Africa
C
th promotion and sc 2]
heduled visits remin
der
Ghana
Grameen Founda (i) Family plannin (i) Data collection an Comprehensive observational studies demonstrated lessons learned and key future impl
tion
g (ii) Behavior ch d management (ii) ications. [28] Evaluation is on-going with Grameen Foundation, Healthcare Innovation T
anges through IE SMS texting for heal echnology LAB (HITLAB), and Ghana’s School of Public Health.[29]
C (iii) ANC/EPI/P th promotion and sc
NC (iv) SBA/FD heduled visits remin
der (iii) Emergency
medical referral (e.g
. referral calling)
mHealth Health Service Coverage Increase Impact Model
Mobile phone
Health Information
Systems
Users
Processes
Outcomes
Impact
Information registration through mobile phone : ID, demographic
information, geographic location etc.
Systems:
Calculate due dates for certain care events: child birth, ANC/PNC/EPI visit
scheduling
Identify clients with upcoming delivery dates, those who recently delivered
and those who estimated due dates have passed without delivery
Send alerts to CHWs when event is overdue
Send BCC messages/reminders to mothers
Mothers:
receive SMS
messages
Promoting
Information,
Education and
Communication
(IEC)
Behavior change:
promoting care
seeking behaviors
CHWs:
client management
scheduling visits
send alerts/reminder etc.
Decision
making
supportin
g tool
Facilitatin
g referrals
Community
empowerment:
efficient/effective
service delivery
Physicians:
case identification
Improving health
systems readiness
and quality of care
Improved
responsiveness:
providing timely and
quality care
Increasing uptake (coverage) of health services in ANC,
SBA/FD, BF, or PNC
Effectiveness of Interventions , Affected Fractions, and Assumptions to Neonatal Mortality in Bangladesh and Uganda
Intervention
Effectiveness (<1month)
Affected fraction
Neonatal-Diarrhea
Curative after birth ORS-oral rehydration solution
0.93
1
Neonatal-Sepsis
Pregnancy
0.97
0.006
0.4
0.23
0.28
0.65
0.8
1
1
1
1
1
0.42
1
Injectable antibiotics
Full supportive care
Curative after birth Full supportive care
0.75
0.9
0.05
1
1
1
Preventive
Thermal care
0.2
1
KMC-Kangaroo mother care
0.51
1
Full supportive care
0.28
1
Pregnancy
TT-Tetanus toxiod vaccination
0.94
1
Preventive
Clean postnatal practice
0.4
1
Folic acid supplementation/fortification
0.35
1
0.1
1
Syphillis detection and treatment
Clean postnatal practice
Chlorhexidine
Oral antibiotics
Curative after birth Injectable antibiotics
Full supportive care
Preventive
Oral antibiotics
Neonatal-Pneumonia
Curative after birth
Neonatal-Asphyxia
Neonatal-Prematurity
Curative after birth
Neonatal-Tetanus
Neonatal-Cogenital anomalies Periconceptual
Neonatal-Other
Curative after birth Full supportive care
Developing an evidence base for mHealth solutions
at scale: Monitoring and Evaluation Framework
Amnesty LeFevre PhD MHS
Smisha Agarwal DDS MPH MBA
Alain Labrique PhD MHS MS
Garrett Mehl PhD MHS
Presentation Overview
• Global context for evaluating mHealth soluions
• mHealth in South Africa: 1 million pregnant women
registration initiative
• Monitoring and Evaluation Framework
• Implications and next steps
© 2014, Johns Hopkins University. All rights reserved.
Global Context
• 2011 the Bellagio Call to Action on Global eHealth
Evaluation highlights the need for rigorous evaluation in
mHealth
• US NIH, Global Donors and Academic calls for improved
rigor in mHealth evaluation and reporting
• Significant investment in evidence generation (eg.
Innovations Working Group)
• 2013 WHO mHealth Technical Evidence Review Group:
Working Papers on mHealth Taxonomy, Evaluation,
Indicators and Evidence Grading
© 2014, Johns Hopkins University. All rights reserved.
Considerations for evaluating mHealth solutions
Pre-prototype
Prototype
Feasibility/ Usability
Pilot
Efficacy
Scaled
Demonstration
Integration
Effectiveness
Implementation
Science
• Where the technology is in the stage of development?
• What corresponding stage of evaluation is appropriate for
that strategy?
• What are the evidence claims the project wants to make?
• What is the time point for evaluation initiation?
• Available resources
© 2014, Johns Hopkins University. All rights reserved.
The evaluation process for mHealth solution
• Outcome,
impact
assessments
Is my solution
effective?
Is my solution
good value for
money?
• Cost
effectiveness /
Utility
analysis
• Cost benefit
analysis
• Economic and
financial
evaluations of
a single
program
Is my solution
affordable?
© 2014, Johns Hopkins University. All rights reserved.
Is my solution
scalable?
• Policy analysis
• Economic and
financial
costing of a
single program
• Sector wide
planning for
integration
mHealth in South Africa
• National program to enable approximately one million
pregnant women to register in health facilities using
interoperable mobile health services.
• Three major features of the program
1.
2.
3.
Improve early identification of pregnant women, increase e
arly access to antenatal care, and repeated antenatal visits thr
oughout the pregnancy.
Register all pregnant women in the public health system usi
ng the mobile health system early in their pregnancy;
Subscribe pregnant women in the public health system to re
ceive pregnancy-related information messages.
© 2014, Johns Hopkins University. All rights reserved.
How will this work in practice?
Identification/ registration / subscription
End-u
ser
Facility
provider
CHW
Source: Debbie Rodgers/ Praekelt
© 2014, Johns Hopkins University. All rights reserved.
Monitoring and Evaluation Framework
• Objectives: Develop a monitoring and evaluation framework
that will
• facilitate the measurement of a common set of indicators across
all implementing partners, and
• yet accommodate differentiation across services providers, impl
ementing strategies, etc.
• Challenges
• Accommodating variation in integration
o Programs at various stages of development
o Health systems variability (CHW presence or not?)
•
Program growth
o
o
Potential for linkages with other NDOH programs
mHealth field is dynamic, always evolving
© 2014, Johns Hopkins University. All rights reserved.
Logframe Overview
Inputs/ Proce
sses
Outputs
Outcomes
Partnerships
1. Increased service
utilization
Increased MNC
H services utiliza
tion
Health Promotio
n Messages
2. Strengthening of
human resources an
d facility readiness t
o provide care
Health facility a
nd community in
puts
Technology
3. Improved service
delivery (facility an
d community)
4. Improved technol
ogy use
5. Funding
Funding
6. Supply side
• Pregnancy
• Delivery
• Postnatal/ postp
artum
• Child health
• HIV
• Efficiency in ca
reseeking
• Improved conti
nuity
Improved knowl
edge
Supply side
Funding
Impact
Reduction in nu
mber of
• Stillbirths
• Neonatal mort
ality
• Maternal mort
ality
Monitoring and Evaluation Framework
• Available on GSMA website
http://www.gsma.com/mobilefordevelopment/mhealth-formnch-impact-model
© 2014, Johns Hopkins University. All rights reserved.
How will this work in practice?
• Framework includes ~ 100 indicators across an expansive range
of domains; anticipated that a sub-sample of these applied
• Can be used by organizations that are involved in one or more
components (identification/ subscription/ registration)
o
Example 1: Messaging component alone, no capacity to link w
ith individual patient records
o
Example 2: Programs with CHW identification/ referral, facilit
y based confirmation/ registration, and subscription
© 2014, Johns Hopkins University. All rights reserved.
How will this work in practice?
Example 1: Messaging component alone
INPUTS
OUTPUTS
OUTCOMES
Health promotion messag
es developed Posters/ Fl
iers in place; Accessible to
users
TechnologyNetwork co
verage and power, provider
/ user mobile equipment
Health workers reporte
d use of mobile tools
Platform functionality
functional performanc
e of the service, user sati
sfaction
Improved reported utilization of MCH servi
ces 
• improved ANC attendance; improvements i
n HIV identification / treatment
Woman wit
h a suspecte
d pregnancy
sees a poste
r
Improved knowledge  Subscribers
Women receives staged
base messaging through
out her pregnancy
Dials in Rec
eives Min
message set
Goes to health Registered to
facility; pregna receive full s
ncy confirmed et of messag
es
© 2014, Johns Hopkins University. All rights reserved.
How will this work in practice?
Example 2: Programs with multiple components
INPUTS
OUTPUTS
OUTCOMES
Health promotion messag
es developed Posters/ Fl
iers in place; Accessible to
users
TechnologyNetwork co
verage and power, provider
/ user mobile equipment
Health workers reporte
d use of mobile tools
Platform functionality
functional performanc
e of the service, user sati
sfaction
Improved utilization of MCH services 
• improved ANC attendance; improved early
ANC 1 attendance; improvements in HIV id
entification / treatment
Improved knowledge  Subscribers; provider
s (Phase II)
Supply side  Utilization of HelpDesk; respon
siveness of HelpDesk
Women receives staged
base messaging through
out her pregnancy
CHW identifies
woman with sus
pected pregnan
cy
Dials in Rec
eives MID
SIZE messa
ge set
Goes to health facility;
pregnancy confirmed
Registered to
receive full s
et of messag
es
© 2014, Johns Hopkins University. All rights reserved.
What will this evidence tell us?
Minimum evidence claims
• Effect on critical outcomes MNCH service
utilization, changes in knowledge (users and
provider), and potential for modeled impact
(LiST))
• Information to attract new investors, inform
existing donors
• Draw additional funding; identify avenues of
commercial sustainability
© 2014, Johns Hopkins University. All rights reserved.
What will this evidence tell us?
Optimal use of logframe
• Prospective program monitoring to allow continuous improvement
o Complementary to existing evaluations – avoids duplication wit
h existing evaluation efforts-- allow for tracking at a scale
o Allow improved planning at a national level: linkages with MN
Os, forecasting of resources and health status (MDG targets)
• Support to complementary streams of research (process documentation,
efficiency, economic analyses) cutting across all programs to
o allow for differentiation / improved understanding of implement
ation (what works in some areas and not others?)
o standardized methodology
© 2014, Johns Hopkins University. All rights reserved.
Implementation and next steps
Is implementation feasible?
• Wide array of indicators intended to accommodate a wide array of programs
(established, forthcoming)
• Need for additional modification to accommodate linkages with programs
that have existing streams of data collection / evaluation partners
• What’s the value to providing data on their programs?
o Unique opportunity to contribute to macro-level generation of eviden
ce claims on mHealth
o Opportunity for differentiation


Comparison of different models of implementation / integration
Ability for continued learning given variability in where programs will li
e along the continuum of efficacy  implementation science
© 2014, Johns Hopkins University. All rights reserved.
Acknowledgements
This document is an output from a project funded by the UK
Department for International Development (DFID) for the
benefit of developing countries, managed through HLSP Mott
Macdonald. The views expressed are not necessarily those of
DFID or HLSP.
© 2014, Johns Hopkins University. All rights reserved.
Further information
•
•
•
•
•
Full Logframe  www.gsma.com
Amnesty LeFevre PhD MHS, aelefevre@gmail.com
Alain Labrique PhD MS, alabriqu@gmail.com
Smisha Agarwal DDS MPH MBA, smishaa@gmail.com
Garrett Mehl PhD, mehlg@who.int
© 2014, Johns Hopkins University. All rights reserved.
Supplementary slides
© 2014, Johns Hopkins University. All rights reserved.
Intervention Validation versus Delivery Strategy Evaluation
• Two roles for mHealth strategies: 1. Intervention with
known efficacy, 2. Interventions with an independent effect
on Outcome C
Intervention A
Measles Vaccination
Problem:
Outcome C:
Reduced measles transmissi
on
Measles outbreak
mHealth strategy:
Vaccine stockout n
otification
© 2014, Johns Hopkins University. All rights reserved.
Data sources
Data Source Description
Purpose
Facility
HMIS
Records
Routinely collected
facility-based records for
clients served
Data on facility registration, gestational age,
demographics, receipt of ANC, SBA, PNC
services
CHW
records
Data on community-level client identification,
Routinely collected CHW and referrals for registration (at the health
records for clients
facility) and subscription.
Data on subscribed client’s satisfaction with
Client phone
Cross-sectional phone
mHealth services, knowledge and practice of
survey at
survey addressed to a sub- key behaviors. Will include post-partum
3, 6, 12, 18
sample of subscribers
surveys for data on continuity of care
months
parameters.
Facility and community level provider surveys
Healthcare
Cross-sectional survey
to assess satisfaction with mHealth platform,
provider
addressed to facility and
frequency and regularity of use. May include a
survey
community providers
component of direct observation, if feasible.
© 2014, Johns Hopkins University. All rights reserved.
How will this work in practice?
Supply side
Registered M
om receives c
are at a healt
h facility*
Dials public lin
e
1. Baby/ preg
nancy help
2. Complime
nts and co
mplaints
1.
2.
Receives sy
stem gener
ated messa
ge within 2
4 hours
Compliant logged/ c
ase opened
Thank you response
Standard content SMS m
essage sent
*Separate stream for end-user
s that are not registered
Message sent indicating
question cannot be answe
red advocates referral
© 2014, Johns Hopkins University. All rights reserved.
Responds to m
essage with
1. Baby/ preg
help question
Web Interfa
ce
2. Compliment
or complaints
Employee r
esponds
Complaints/
compliments
Question can be
answered with st
andard content
Question cannot
be answered wit
h standard conten
t
How will this work in practice?
Example 2: CHW identification and referral, facility registration / subscription
to full messaging
INPUTS
OUTPUTS
OUTCOMES
Health promotion messag
es developed Posters/ Fl
iers in place; Accessible to
users
TechnologyNetwork co
verage and power, provider
/ user mobile equipment
Health workers reported use of
mobile tools
Platform functionality functi
onal performance of the service,
user satisfaction
Client Profile Messages sent;
Proportion of and profile of clien
ts that opt-out;
Improved utilization of MCH servi
ces 
• improved ANC attendance;
• improved early ANC 1 attendance;
improvements in HIV identificatio
n / treatment
Improved knowledge  Subscribers
Women receives staged
base messaging through
out her pregnancy
CHW identifies
woman with sus
pected pregnan
cy
Dials in Rec
eives MID
SIZE messa
ge set
Goes to health facility;
pregnancy confirmed
Registered to
receive full s
et of messag
es
© 2014, Johns Hopkins University. All rights reserved.
Vocabulary
• mHealth TECHNOLOGY
• The hardware / software underlying an mHealth solu
tion
• mHealth PROJECT
• The use of a particular technology(ies) to achieve a
particular goal in a specific location(s)
• mHealth STRATEGY
• The generic approach that is being undertaken using
mHealth, agnostic of project or technology.
© 2014, Johns Hopkins University. All rights reserved.
mHealth Summit
11th of December 2014
Hajo van Beijma
hvanbeijma@ttcmobile.com
@hajovanbeijma
Healthy Pregnancy Healthy Ba
by
Partners
•
•
•
•
•
•
Tanzanian Ministry of Health and Social Welfare
mHealth Tanzania Partnership – led by the CDC Foundation
Financial support from the US Government
Airtel Tanzania
Wazazi Nipendeni multi-media campaign
Johns Hopkins Bloomberg School of Public Health
Background
• Healthy Pregnancy Healthy Baby as
part of Tanzania’s efforts in the cam
paign on Accelerated Reduction of M
aternal Mortality in Africa.
• Tanzania has one of the highest mat
ernal mortality rates in the world.
Healthy Pregnancy Healthy Ba
by
SMS Information Categories
• Prevention of mother to child transmission
of HIV/aids
• Ante-natal clinic visit reminders
• Malaria prevention
• Individual birth preparedness plan
• Nutrition
• Danger signs
• Family planning/Birth Spacing
• Fun/Interesting (e.g. fetal development)
• Post-partum care
Healthy Pregnancy Healthy Ba
by
How does it work
TO: 15001
mtoto
One short c
ode for all n
etworks
Healthy Pregnancy Healthy Ba
by
•
•
•
•
•
•
•
HPHB the largest interactive mobile h
ealth program in Africa – to date
We offered 31 million maternal health
and early childcare text messages over
600,000 registrants
On average we count an extra 21,500 r
egistrants every month
Majority of users self-registered, with
only 1.41% of users utilizing the health
-facility assisted registration option
High level of engagement
Users on average opt-in weekly
Goal: 1.000.000 registrants by 2016
Users
51.50% male
37.81% female
10.70% did not want to report gender
Timing
• Day of the Week
 Wednesday > Tuesday > Thursday > Friday > Monday > Satur
day > Sunday
• Time of Day
 Morning 8:00 – 11:00
 Lunch time 11:05 – 14:00
 After work: 17:05 – 20:00
Traffic
Thank you! Any questions?
hvanbeijma@tt
cmobile.com
www.ttcmobile.
com
Using Globa
l South Data
to Improve
mHealth for
All
Global mHealth Forum D
ecember 11, 2014
Meagan Demitz
Hesperian Resources
Mission: Provide informatio
n & tools that help all peopl
e take greater control over t
heir health & work to elimin
ate underlying causes of po
or health.
Resources: Primary health
care; women’s health; repro
ductive health and rights; mi
dwifery; health worker traini
ng; community dentistry; su
pport for women and childre
n with disabilities; occupatio
nal health and safety; enviro
nmental health; early childh
ood development
Collective development to creat
e empowering materials
1. Start from people’s own experiences
2. Offer practical solutions
3. Design materials based on listening carefully, rep
orted needs, data collection and evaluation
4. Encourage participation and action in ways that d
o not depend on literacy, and that encourage criti
cal thinking
5. Collect and incorporate end-user feedback when
ever possible
Adapting the Hesperian mod
el for digital resources
• Feedback from consumers via
field testing, expert review, an
d beta-testing (pre and post)
• On-going collection of global u
ser data (Google analytics)
• Content dev and design, user fl
ow and user experience
• Impact evaluation - qualitative
and quantitative feedback
• On-going adaptation of conten
t/user experience
The HealthWiki
• 13 languages
• 16,000 visitors per day; 2
million over the last year;
majority from Global Sout
h
• Top 10 countries: US, Mex
ico, Brazil, Spain, Colombi
a, India, Peru, Argentina,
UK, Venezuela
• 57% of traffic via mobile d
evices; 243,000 per mont
h
• Most visited: Reproductiv
e health; maternal & child
health; abortion; belly pai
n/worms/diarrhea
Launched in October 2011!
Hesperian’s model
for mHealth
Need: Access to information in th
e field without Internet
Focus: Women’s health based on
HealthWiki traffic; partner feedba
ck
Approach: Applying Hesperian’s
collective development process
Lessons learned: Adapting cont
ent for mobile is time intensive




Editorial, design adaptation
Internal and external review
Field-testing
Beta testing before and after laun
ch
Result: Safe Pregnancy and Birth
app
Mobile App:
Safe Pregnancy and Birth
Over 215,365 apps downloaded fro
m 149 countries
Translations underway in Hindi, Tam
il, Kannada, Malayalam, Bengali, Gr
eek and Nepali.
Field Test:
Compañeros en Salud
Facilitated by CES - PIH health partn
ers
in Chiapas, Mexico:
• Field-testing was conducted with the help of
CHWs, midwives, and patients
• App was used to educate patients about the i
mportance of health, and warning signs duri
ng pregnancy
Feedback:
• Health workers valued skill-building sections
• Images allowed illiterate patients to better u
nderstand information
• CES plans to continue to use the app in pre-n
atal monitoring
• Request for future apps: Communicable and
non-communicable diseases; birth control; i
nfant malnutrition; other common health iss
ues
Applying lessons for future
mHealth resources
• Collaborative development and partners
hips
• Strengths-based, community-centric app
roach
• Time-intensive content development, inc
luding design and user interface
• Ongoing evaluation through field-testing
and beta-testing
Awards: mHIFA working group; Ashoka “She will innovate”
HealthWiki: http://en.hesperian.org/hh
g/Healthwiki
Meagan Demitz
Foundation Relations an
d Grants Manager
Hesperian Health Guides
1919 Addison Street
Berkeley, CA 94704
www.hesperian.org
(510) 845-1447
meagan@hesperian.org
Safe Pregnancy & Birth: http://hesperia
n.org/books-and-resources/safe-pregn
ancy-and-birth-mobile-app/
Data Use and Indicators fo
r the mHealth pragmatist
Smisha Agarwal, Amnesty Lefevre, Lavanya Vasudevan, Alain Lab
rique
Mobile data collection
• Faster, cheaper, easier………. But is it?
Starting with the end in mind
What is the
stage of
development
of the
program
What are the
evidence
claims we
want to
make?
Identifying
specific set of
indicators to
address
claims
Stage of Development
What is the stage
of development
of the program
What are the
evidence claims
we want to
make?
Identifying
specific set of
indicators to
address claims
Stages of
Development
Pre-prototype
Prototype
Pilot
Scaled
Demonstration
Integration
Prioritizing relevant evidence claims
What is the stage
of development of
the program
What are the
evidence claims we
want to make?
Identifying specific
set of indicators to
address claims
Data
Stake-hol
ders
Source- MEASURE Evaluation
Decisions
Key Considerations for Selection of Indicators
What is the
stage of
development
of the program
What are the
evidence
claims we
want to make?
Process
Indicators
Identifying
specific set of
indicators to
address claims
Intervention of kn
own efficacy
Absence of evidence-b
ase for underlying inte
rvention
Barometer of mHealth In
dicators
Outcome
Indicators
Categorization of mhealth Indicators
Does the technolog
y work?
• Technical Facto
rs
•Organizational
Factors
How do people int
eract with technol
ogy?
•User Coverage
• User Respons
e
• User Adoption
How does technolog
y improve implemen
tation process?
• Availability
• Cost
•Efficiency
• Quality
•Utilization
How do improvem
ents in service deli
very affect health?
• Improved hea
lth outcomes
Performance of Routine Information Syst
em Management (PRISM)
 Emphasis on Health
Information Systems
(HIS) performance
 Consideration of Or
ganizational, Technic
al and Behavioral co
mponents
Linking Indicators to Taxonomic Stages of Developm
ent
Stages of
Development
Pre-prototype
Prototype
Pilot
Scaled
Demonstration
Integration
Pre-prototype: This stage includes hypothesis
building, needs/context assessment, and
usability/feasibility testing. This is the first stage
of the development of a project.
Earliest Stage of Developmen
t to Capture Indicators for “D
oes the Technology Work?”
Does The Technology Work?
Metric Area
Indicators
Technical Factors
Connectivity
% of target population with mobile phone signal at time of interview
Power
% of target population who have current access to a power source for recharging
a mobile phone device
Skilled local staf % of mHealth programs with current access* to local technical support for troubl
f
eshooting
% of users who report having access to local technical support systems for troub
leshooting
Maintenance
% devices that are not currently operational (misplaced/broken/not working)
Functionality
% of mobile devices that are operational in the language of the users
% target population who are literate in the language used by the mHealth strateg
y
% of target population who report ever use of Short Message Service (SMS) capa
bilities
% of data fields from original paper based system that technology captures
Organizational
Factors
Training
Total hours of initial training attended by program staff in use and deployment
of technology
Total hours of refresher training attended by program staff in use and deployme
Linking Indicators to Taxonomic Stages of Developm
ent
Stages of
Development
Pre-prototype
Prototype
Pilot
Limited
Demonstration
Integration
Prototype: During this phase, user-focused de
signs are created and tested, and functionality,
stability and usability are tested in an iterative
process. Ways to improve the project are exam
ined to enhance relevance.
Earliest Stage of Development
to Capture Indicators for “How
do People Interact with Techn
ology?”
How do People Interact with Technology?
Metric
Area
Indicator
User Cov % of users who demonstrate proficiency in use of intended mobile application
erage
% intended users observed using the tool in preceding reference period of time
No of transmissions sent by intended users over reference period of time
% of transmissions successfully sent* in ‘x’ period of time
User Res % of users who rate technology as "easy to use"
ponse
% of users rating technology “transmits information as intended”
% of users who report satisfaction with the content of health information received th
rough mobile device
% of users motivated/intend to use technology
User Ado % of messages/amount of data transmission sent from server that are responded to a
ption
ppropriately** by end user within reference period of time
Number of messages/forms/amount of data transmission sent by end-user within re
ference period of time
% of data fields/forms that are left missing/incomplete over specified period of time
*Successful transmission is reflective of the network coverage in the user area
** ‘Appropriately’ could refer to completion of intended action to reflect that the message has b
een read e.g. Acknowledgement of message
Feedback Loop
Technology inputs affects performance
Does the Technology Wor
k?
How do People Interact wi
th the Technology?
User feedback informs technology development proc
ess
Spiral Model of Software Development and Enhancement
Linking Indicators to Taxonomic Stages of Developm
ent
Stages of
Development
Pre-prototype
Prototype
Pilot
Scaled
Demonstration
Integration
Earliest Stage of Development to C
apture Indicators for “How does Te
chnology Improve Implementation
Process?”
Pilot: This stage examines whether or not th
e project has the ability to produce the desir
ed effect under controlled circumstances and
is usually a single deployment. This correspo
nds with the evaluation stage of Efficacy.
How Does Technology Improve Implementation P
rocess?
I. Health syste
ms level
Registration and vital events tracking
Real time indicator reporting
Human Resource management, accountability
Electronic health records
Supply Chain Management
II. Provider le
vel
Decision Support
Scheduling and Reminders
Provider training, service updates
III. Patient le
vel
Client education and Self-Efficacy
Behavior Change Communication
Adherence to Care
Emergency services information
mHealth functions and strategies
Improvements i
n
• Availability
• Costs
•Efficiency
• Quality
•Utilization
Health Systems-level: How Does Technology Improve Imp
lementation Process?
mHealth metri
c by taxonomic
Indicators
constraint
Availability
% of target population who have access intervention ‘X’ over reference period
% of health facilities in a target geographical area that use mHealth platform services
Total number of clients seeking health service “x” at health facility with mHealth platform s
ervices
Efficiency
Total cumulative time in minutes over reference period for all health workers in a facility u
sing mHealth platform to enter data about intervention ‘x’*
Total time taken in minutes for all health workers over reference period to transmit data ab
out intervention ‘x’ from community logs to health facility information systems
Total cumulative time taken in minutes over reference number of events from identification
of an adverse event to care provision for intervention ‘x’ across levels of a health system
Total days in reference period for which a health facility reports stock out of commodity ‘x’
Quality
Total number of healthcare workers observed to be providing clinical services related to m
Health strategy
% change in reported events of “stock out” of commodity ‘x” over reference period **
% change in data entry errors over reference period **
% of target health workers who receive initial training on health intervention ‘x’ using mHe
alth platform
% of target health workers who receive refresher training on health intervention ‘x’ using m
Health platform (initial and refresher training)
Health Systems-level: How Does Technology Improve Imp
lementation Process?
mHealth metri
c by taxonomic
Indicators
constraint
Utilization
Total number of clients seeking health service ‘x’ over a specified period of time
% of clients in a specified area who are receiving health service ‘x’ through mHealth strat
egy over reference period
Costs
Change in costs of transporting paper forms and manual data entry over reference**
Change in costs of human resources for data entry**
Change in costs associated with timely and appropriate management of illness**
Changes in reported out of pocket expenditures over time **
Total population level savings in out-of-pocket payments as a function of timely and appr
opriate care seeking**
* Aggregated facility-level indicator ( Corresponding indicator at provider-level is disaggregated)
** Assumption of data collection at two-points before and after the implementation of the mHealth strateg
y
X: To be replaced by specific health intervention targeted by the mHealth platform
Provider- level: How Does Technology Improve Implemen
tation Process?
mHealth metri
c by taxonomic
Indicator
constraint
Availability
% of targeted health workers who utilize mHealth platform about intervention ‘X’ throug
h their phones over reference period
% of health workers observed to use mHealth platform during their last client contact
% of health workers who use mHealth intervention to connecta with medical staff to receiv
e real-time clinical information and decision support
Total number clients attendedb by a health worker using mHealth platform over reference
period
Efficiency
Total reported/observed time in minutes for last client counseling using mHealth platform
about intervention ‘x’
Total reported/observed time spent on health record keeping about intervention ‘x’ over r
eference period
Total time taken in minutes per health worker over reference period to transmit data abou
t intervention ‘x’ from community logs to health facility information systems
Total individual health provider time taken in minutes over reference number of events fro
m identification of an adverse event to care provision for intervention ‘x’ across levels of a
health system
Total time taken in minutes by health worker to report important adverse events (stock-o
uts)
Provider- level: How Does Technology Improve Implemen
tation Process?
mHealth metri
c by taxonomic
Indicator
constraint
Quality
% of health workers who report adequatec knowledge of topic ‘x’
% of care standards* observed to be met using mHealth intervention about intervention
“x” during a client-provider consultation
% of providers observed to be using mHealth intervention during their patient consultati
on
Costs
Estimated cost savings due to improvement in provider technical efficiencyd
a: “Connect” could be via phone call. E.g. community health workers might call health supervisors for susp
ected complication and received decision support via phone call or other mHealth supported means from
a high level provider
b: “Attended” could be via phone call or personal home-visit or other modes of communication using mHe
alth strategy
c: ‘Adequate’ could be defined by program intervention, eg: % of target health workers who know 3 pregn
ancy danger signs
d: Composite Indicator derived through monetizing time savings for administrative functions
X: To be replaced by specific health intervention targeted by the mHealth platform
Client-level: How Does Technology Improve Implementati
on Process?
mHealth m
etric by tax
onomic con
straint
Indicators
Availability % of target clients who report adequatea knowledge about signs and symptoms requiring care-see
king for health area “X”
% of target clients who report adequatea knowledge about ‘x’ health area
Technical E Incremental time in minutes between mHealth prompt received about intervention ‘x’ and care-see
fficiency
king with provider
Total number of in-person consultation with qualified health provider about intervention ‘x’ by tar
get clients as a result of accessing required services using mHealth strategy over reference period b
Quality
Duration of illness episode in days
Total time in minutes spent with a health provider about health intervention ‘x’ in the last visit
% of messages received through mHealth strategy that clients are able to recall about intervention ‘
x’ during client exit interviews
% of target clients who report correctly adhering to prescribed care protocol about intervention ‘x
’
a: ‘Adequate’ could be defined by program intervention, eg: % of target clients who know 3 pregnancy danger sig
ns
X: To be replaced by specific health intervention targeted by the mHealth platform
b: Required collection at multiple time points to yield estimates of “averted” incidences
Client-level: How Does Technology Improve Implementati
on Process?
mHealth m
etric by tax
Indicators
onomic con
straint
Utilization % of emergency events where mobile phones were used by patients to expedite treatment over re
ference period
% of target clients who report receiving a health information about intervention ‘x’ through their
mobile phone within reference period
% of target clients who report contactc with a qualified health care provider using mobile phone str
ategy about intervention ‘x’ over reference period
Costs
% changes in reported client out-of-pocket payments in illness management over specified period
of time d
c: Contact: To be determined based on mHealth strategy medium of health service delivery. Could include teleph
onic consultation, home visit by health worker, or clinic visit by patient where the use of the mHealth strategy ha
s played a role in the receipt of services.
d: Composite indicator- could be sub-categorized into individual components of interest where cost-savings are i
ntended e.g. travel cost, days wages lost ,