UNDERSTANDING WHAT WE ARE
DOING IN GLOBAL HEALTH:
AN APPROACH TO METRICS
Robert Riviello, MD, MPH
Center for Surgery and Public Health, Brigham and Women’s Hospital
Director, Global Surgery Programs
UT Southwestern Office of Global Health Conference:
Transformative Global Health:
Impact and Accountability
February 9, 2014
•  No conflict of interest
Thanks and Acknowledgements
•  Prof. Fiemu Nwariaku
•  Wendeline Jongenburger
•  Veronica Mason
•  Kristi Thetford
Harvard Medical School
Program for Global
Surgery and Social
Change (PGSSC)
•  Rebecca Maine
•  Eric Nagengast
•  Ainhoa Costas-Chavarri
•  George Baison
•  Charles “Beau” Bush
•  John Rose
•  Rowan Gillies
“No service without a survey
No survey without service.”
Global Surgery Metrics
Introduction
•  Why use metrics:
•  Evaluate current state of surgical/health system
•  Set targets to drive improvement in performance and capacity
•  Establish system of repeated measure to track progress in access and
quality of care and compare across regions and countries
•  Create evaluation tool to allow for sharing best practices.
•  Targeted achievements to drive donors.
Introduction
•  What do metrics measure
•  Process, Outcomes
•  Performance, capacity
•  Different levels of the health system – national, district/region, facility,
individual
•  Examples of current metrics used in global health: MDGs, disease
specific metrics – EmOC, Asthma, Childhood Illness
Introduction
•  Goal of this Presentation
•  Review current metrics used in global surgery
•  Evaluate benefits/drawbacks of those metrics
•  Evaluate metrics used in other areas of global health that might be
relevant to global surgery.
Use of metrics in Global Surgery
•  Much of the information used to assess surgical care in health
systems comes from survey data
•  New survey tools that were developed to assess surgical systems –
•  Often un-validated
•  One-time measurements
•  Examples:
•  WHO – Situational Analysis
•  PIPES
•  HHI
•  Use of data from existing survey (not surgery specific) –
•  Often repeated data
•  Standardized across larger country groups
•  Examples:
•  DHS and Service Provider Survey
•  World Bank Survey
•  United Nations Surveys
•  WHO surveys
•  Health Facilities Census.
Use of metrics in Global Surgery
•  Other methods include snapshots of current state of health care system, facility,
program, etc based on routinely collected data, usually clinical and/or
administrative, though some do prospective audits. Few authors report on
sustained data collection systems.
•  Relies on accuracy of records, consistency in record keeping, ability to easily collect
and record the information.
•  Outcomes and quality data are not often recorded well in this data
•  Can still be time intensive to compile all of the information.
•  Wealthier countries often engage in much more robust prospective data
collection that allowed for the analysis of clinical outcomes
•  Risk stratifications – several systems have been developed to allow for comparison of
outcomes across facilities/sites by predicting risk of developing those complications
based on characteristics of the patient population prior to any intervention
•  NSQIP
•  APGAR
•  POSSUM
•  Clinical database – Registries for patients with particular diagnosis or undergoing
particular procedures are tracked in some countries to evaluated long-term outcomes.
•  This requires significant investment in information systems and analysis of information
that is not possible in most LMICs.
•  Provides robust longitudinal data for analysis of quality and outcomes and establishment
of metrics.
Key Metrics in Global Health
•  Health systems, including surgical systems, can be
evaluated along several different types of metrics, which
all contribute to the ability of the system to deliver high
quality care. We will evaluate the specific metrics used in
each of the following categories to determine which is
most easily measured and which is most important to
assessing the function of the surgical system.
•  Infrastructure, Human Resources, Output, Outcomes, Access/
Effective Coverage, Financial, Governance/Leadership, Burden of
Disease, Education, Supply chain.
•  Few authors have addressed all of these areas in one set of metrics,
through several different surveys attempt to address multiple aspects of
the surgical care system.
Infrastructure
•  Certain equipment is necessary to provide surgical care and
several aspects of infrastructure, especially at the hospital level,
have been evaluated as part of the assessment of the ability to
provide surgical care.
•  Often the metrics use to assess infrastructure are:
•  Available: Yes or no; or Always, sometimes, never
•  Several surveys have noted that equipment may be present but is either not
repaired or the human resources are not available to run the machines.
•  Percent of facilities with that infrastructure available.
•  At the country level: Number of facilities.
•  Assessed via survey and/or national reporting programs
Infrastructure
•  Infrastructure can be evaluated at several levels
•  Hospital Building Infrastructure – This measures the adequacy of the
building to support surgical care at the most basic level.
•  Minimum required: Adequate/safe building, OR, clean water supply, electricity
•  Functional hospital infrastructure – These are the basics functions that a
hospital must be able to perform to support care for surgical patients.
Successful performance of many of these functions is complex, relying
on appropriate facility space, durable equipment, maintain for the
equipment, consumables, and human resources to execute the function.
•  Recovery room not included in any of the metrics, but a stopping point to
using the ORs in hospital settings.
•  Durable equipment – while this ties into the function above, it can also
be thought of as a separate metric in this category.
•  Consumables – Lack of a variety of different types of consumables can
lead to problems delivering essential surgical care.
•  Information systems – These metrics are not as widely referenced but
represent a very important piece of moving forward in measuring the
qua
Infrastructure
•  Recommendations/thoughts
•  Some component of infrastructure should be included in the overall
metrics
•  Straight forward, easy to track and monitor progress.
•  Several tools have been developed
•  Can develop consensus for basics package of subsets of infrastructure from
existing tools and measures that are being used (or endorse/modify one of the
existing tools)
•  Essential – water, electricity, sterilization facility,
•  Strongly recommended
•  Desirable
•  Incorporate that data collection into the routine health systems surveys.
•  Identify who is best to be reporting on this data routinely and how often
these should be evaluated for MOH level reporting
•  Consumable stock outs should be reported on a q 4 month basis?
•  Measurements aggregated to country level
•  Should there be validation of the reported information? Who and how?
Infrastructure
•  This is really the most basic of measures, and while important for
laying a ground work and setting expectations for how facilities
should be equipped, it is not at all sufficient for measuring the
availability of quality of surgical services provided.
Human Resources
•  In surveys focused on a variety of conditions in global
health, including surgery, researchers have been
interested in the human resources available to provide the
essential aspects of providing that care.
•  Basic measures include:
•  Counts of providers for hospitals, regions, countries
•  Rates of providers by population size.
•  Rates of providers for a particular health need.
•  Percentage of types of care that is delivered by different levels of
providers.
•  Availability at a particular facility/region
•  On-call
Human Resources
•  Groups of providers that are important to assess for the
capacity to deliver surgical care.
•  Typically considered:
•  Physicians, General surgeons, Orthopedic surgeons, OB/gyn, other
specialist surgeons (ENT, Plastics, neurosurgery, ect), Nurses
(specialized and not), Anesthesiologist, non-MD anesthetist, other
physician extenders (clinical officers), pharmacists
•  Less frequently or not included: Biomedical engineers, teachers in
training programs, rehabilitation specialists, intensivists, pre-hospital
providers, management, NGOs role in surgical care, community health
workers, other support/maintenance staff, traditional healers
Human Resources
•  Measures that have been address in some studies,
though have not yet been developed into metrics which
are incorporated into repeated measurement or include
targets
•  Brain drains – internal and external brain drain
•  Turn-over at district hospitals
•  Interest in surgical and anesthesia careers – what professions are
medical students choosing and why?
•  Quality of the human resources – Larger investment to track – not
clear what and how to measure
•  Accreditation, training requirement, CME, outcomes (see education
metrics)
•  What do you do if the few available resources are not good?
•  Perhaps a quality of coverage metric which takes into account the availability
of staff of the facility and the “quality” of the employees there
Surgical Output/Clinical Output
•  Outputs move beyond basic infrastructure to measure
how different facilities, regions and countries are able to
apply those that infrastructure to actually deliver care.
Outputs are proxy for the function of the equipment,
Surgical Output/Clinical Output
•  Basic metrics used
•  Number of surgeries
•  by OR, number of beds, hospital/hospital type, region, country
•  By population, surgical disease, catchment area, age of patient, gender, hospital and
outpatient visits
•  Case Mix – attempts to evaluate whether or not the surgical care is sufficient at
a particular level based on the case mix with a percentage of a particular type
of case
•  C-section vs other; emergency vs elective, major vs minor
•  Recommendations of what types of surgical procedures should be available at
different levels of care.
•  Tracer procedures – evaluating the care for a specific common condition in the system
as a proxy for care overall
•  Tracking over time can give sense of treatment of surgical backlog of conditions like
cleft palate.
•  Other measures
•  Geographic distribution of cases – variation by DH vs referral hospital
•  Number of surgical consultations – both inpatient and outpatient.
•  Percentage of time spent in surgical care by providers at DH
•  OR utilization
Surgical Output/Clinical Output
•  Most hospitals will record some type of volume and case
mix
•  Problems with the reliability of those measurement
•  Problems with routine reporting – no systems in place, little
standardization
•  Problems with routine use of the data collected.
•  To consider
•  Set baseline metrics for population – depends on the true burden
epidemiology
•  Requirements for reporting/standardization of
•  International OR registry
•  Efficiency measures
•  Utilization metrics
•  ROI of infrastructure and human resources
Outcomes
•  Measuring quality, often by measuring outcomes, has become a
focus of intense investigation and investment in HIC countries
attempting to evaluate the performance of surgical systems at a
hospital and individual level mostly. However much less work has
been done to establish systems that can track outcomes of surgical
procedures in LMICs.
•  The ability to measure outcomes is an important process measure in
and of itself.
•  There are a lot of differences in the important metrics across different
types of surgical care.
Outcomes
•  Metrics that are used
•  Mortality – this is the big one, in part because it is a clear, dichotomous
outcome. Still in many countries where national death registries do not
exist and standard documentation of in-hospital death may also not
exist, accurate mortality rates are challenging to capture
•  Variety of metrics: 24 hour, in hospital, perioperative, post-operative, child
•  Morbidity
•  Requirement investment in training, collecting and reporting information, or
involves retrospective review where information is not collected well.
•  Definitions can be challenging – NSQIP
•  Post-surgical outcomes
•  QOL measures – not really discussed in the articles reviewed
•  Functional assessment surveys
•  Length of stay
•  Costs/cost-effectiveness
Outcomes
•  Metrics that are used
•  Process measures – proxies for quality care that are more easily
measured than actual clinical outcomes
•  Percent of patients with anesthesia record, use of surgical safety checklist,
Appropriate peri-operative abx, percent of patients with physical injuries
receiving physiotherapy.
•  Risk adjustments
•  When evaluating outcomes it is important to make sure that there are
mechanisms to adjust for risks or it will discourage providers from treating the
most difficult patients.
•  NSQIP, APGAR, POSSUM
•  Ability to report back and use the data collected
•  Morbidity and mortality rounds - not really discussed in these articles
Outcomes
•  Metrics that are used
•  Quality of the health care worker - a few authors attempted to assess
clinical management knowledge during their surveys, mostly with
disappointing results and very few focused on surgical care.
•  Effective coverage- This is a concept in global health that looks at the
actual health gained from surgical care compared to the DALYs do
surgically treated illness. It is a measure of effective quality and
incorporates elements of access, disease burden and quality of surgical
output
•  Still very limited data on an individual country/facility level to assess both the
burden of disease and health gained/DALYs averted from care there, but
overall a good functional metric.
Outcomes
•  Issues to consider
•  Which mortality measures are the most important?
•  Standard for recording and reporting mortality
•  Are there population level mortality metrics (mortality in age of
most injuries?) that can be followed to assess part of surgical care?
Education
•  Few papers have evaluated the adequacy of surgical and
related health care training programs, though several
authors acknowledge the need to train additional
personnel to meet the surgical needs in most LMIC
Education
•  Metrics used
•  Number of training positions in different specialties related to
surgical care
•  Nurses, Surgeons, Anesthesiologist, Anesthetists, Clinical officers
•  Accreditation of providers at each of these levels - accreditation
standards with in a country, pass-rates, recertification can all be
used to assess accreditation
Education
•  Other things to consider for educational metrics
•  Availability and requirements for Continuing Medical Education/
professional development
•  Cost of medical training for specialization
•  Participation in specific focused training programs to support areas
of surgical train
•  Trauma training, pediatric/adult emergency care.
•  Evaluating the efficacy of partnerships in global surgery care.
Education
•  Lack of quality measures
•  Brain drain can possibly be seen as a quality metrics for educations
programs – countries with higher physician exports may have
better training.
•  Very few programs exist to anywhere to truly evaluate technical
skill, but this is very important especially in areas where nonsurgeons are providing surgical care.
Access/Effective coverage
•  Even in some LMICs countries where facilities exist and
trained personnel are available to staff them, rates of
access to surgical care can be low. Thus, it is essential to
be able to assess whether or not patients are able to
access the surgical services in a country
Access/Effective coverage
•  Fundamental measures of access
•  Numbers of facilities that can provide the services in questions
•  Percentage of population with a need that is accessing those
services
•  Best definitions in EmOC – 5 to 15% of pregnant women will need c/s.
•  Depends on both knowing the denominator
•  Can also look at specific care modality (images per population)
•  Type of governance of facilities people are using - public vs private
vs NGO/FBO
•  Variability in available services overtime – estimations of percent of
time in a given period that a facility was actually able to provide all
of the care at that level of interest.
•  This can include specific lists of services recommended by different
groups at each care level.
Access/Effective coverage
•  Assessing barriers to access is an essential part of metrics that
evaluate access to care.
•  Geographic limitations – Percentage of population within a distance and/or
travel time from health facilities; special consideration for variable barriers to
access (conflict, flooding, etc) is also important
•  Financial – Assessing costs to patients (including transport and lost wages for
inefficient health care system) and pay structures is important. (see more in
financial section), impact of costs on care seeking behavior is also important.
•  Knowledge/willingness – Few studies have assessed patient’s knowledge
about the need of surgical intervention for certain conditions, lack of education,
lack of individual empowerment to seek care for some groups, and lack of trust
in government services or satisfaction with treatment by health care workers
can all lead to lower access of available services.
•  Delayed access to health care – access to services, especially elective
procedures can be very slow in many countries. Delays can exist in many
points of contact with the health care system, and loss to follow up can be high.
•  Defining targets for acceptable wait times may help to evaluate how efficiently people
are accessing that care. – delays in seeking care, seeing surgeon, getting recommend
procedure – acute and non-acute.
•  Role of NGOs/visiting teams in providing care and follow-up to that care.
Access/Effective coverage
•  Equity in access to surgical care is a very important
aspect of care that metrics should address, but few
authors have described effective ways of assessing that.
– Access across countries, for different groups (children,
women)
•  Gini index and Lorenzo curves are interesting to evaluate care
across countries/regions.
Leadership
•  Effective function of a health care system and national
and facilities levels require leadership in part to help
develop a trust in the health care system so patients are
willing to seek services. Few authors have used or
proposed metrics to assess the leadership capacity, but
several articles have addressed the issues of leadership,
which could be adopted into some potential metrics.
Leadership
•  Fundamental issues related to leadership
•  Clear organizational structure with power lying with the leadership
to effect changes as problems are identified – presence of
organizational chart, chain of command, and perceived power/
efficacy could be assessed at different levels.
•  Should include assessment of performance and feedback
•  Guidelines and policies –
•  National levels – Do health policies include plans to provide surgical
services including trauma care to patients? Is there accreditation for
hospitals providing surgical care – how often?
•  Facility level: The WHO emphasizes available emergency and essential
surgical care guidelines in their assessment tool. Policies for
sterilization, infection prevention/control, pain control, disaster
management, trauma assessments, as well as several disease/
condition specific policies. Clinician oversight
Leadership
•  Other areas to consider for assessment:
•  Oversight of clinical officers and other task shifting activities.
•  Opportunities/expectations about formal leadership training –
professional management vs physician managers.
•  Number/percentage of surgeons and anesthesiologist in leadership
at national and facility levels.
•  Efficiency of facility function in delivering surgical care – measure of
delays during day/use of facilities; health gained for inputs into
system
•  Financial management – corruption, return on investment
•  Variability in management across MOH/FBO/NGO managed
facilities.
Leadership
•  Potential areas to target
•  Registration numbers
•  M& M rounds
•  National registries to assess care.
•  Checklist for surgical care at facility level
Disease Burden
•  Several authors have tried to assess met and unmet
burden of disease in recent years, including modeling
studies, assessment of output, and community based
assessment – GBD, disease specific (MSK, general
surgery, OB).
•  Actual need is very important to the other metrics to drive
expectations about number of facilities, providers, equipment
needed etc.
Disease Burden
•  Met need
•  This metric is closely linked with surgical outputs, and
improvements in that assessments will help to clarify the current
state of surgical delivery to fully address the known inequities in
distribution of surgical care globally. –
•  Good, if perhaps not totally evidence driven, estimates for C-section
exist based on populations of pregnant women.
Disease Burden
•  Unmet need
•  Estimation techniques – no standard methodologies
•  Population epidemiology
•  Populations surveys
•  Patients may not be good at assessing their need for surgical care
Disease Burden
•  Prevention – Several authors advocate for injury
prevention and access to early surgical care to prevent
morbidity from delayed treatment of complications from
inadequately treated surgical conditions. However, no
articles developed specific metrics for how to assess
participation in such activities.
•  Possible metrics – percent of hernia repairs that are strangulated;
typhoid vaccination coverage, c-section rates, time to surgical
fracture fixation/percent of surgical fixation.
Financials
•  Assessment of the financial function of the surgical
system must also be assessed at multiple levels including
government spending, financing mechanisms, costs to
patients, and cost effectiveness of surgical care. Again,
while these issues have been discussed by several
authors, clear metrics have not yet been developed to
evaluate the function at the different levels of care
provision.
Financials
•  Government financing – Several authors addressing
health systems function have looked at health care
financing mechanisms, but none of the reviewed studies
have focused on how those impact the delivery of surgical
care specifically. Assessment of types of financing
methods specifically on surgery have not been done.
•  Levels of spending on health care
•  Percent of spending by government
•  Measurements of surgical provision by different financing methods
to fuel recommendation.
•  Defining costs of basic surgical package/minimum per capita
spending for surgical care
Financials
•  Patient financing – some studies have proposed max
limits on out of pocket spending of patients in general, a
few have looked at the impact of costs on seeking surgical
care. However, clear guidelines on how much and when
patients should pay have not been made
•  Guidelines for how costs to patients should be measured, levied
and collected.
•  Evaluation of equity of personal spending for surgical care.
Financials
•  Several authors recently have evaluated the cost-
effectiveness of the surgical care (see Cost Effectiveness
Whitepaper).
Supply Chain/Pharmaceuticals
•  Many articles discuss the availability of a list of essential
drugs, including the frequency of availability. A subset of
medications are needed to provide safe surgical care
(anesthesia, pain relief, antibiotics, tetanus vaccine,
typhoid vaccine etc), making this metrics useful to surgical
care as well.
•  Tied closely to infrastructure, finances, planning for emergencies –
access to medications during code.
Supply Chain/Pharmaceuticals
•  Pharmaceutical availability is tied to supply chain issues
that may be best evaluated together with other aspects of
surgical care in relationship to global surgery.
•  Also consider metrics to evaluate appropriate use of
medications in providing surgical care
•  Use of perioperative antibiotics
•  Use and availability of narcotics for pain control
Summary
•  Significant overlap in what several metrics assess, so
selecting key indicators that evaluate several of these
aspects of surgical care delivery is important.
•  Roll out of metrics – Establish if building blocks are present and
then work to achieve the process and outcomes measures?
•  Mix of more easily collected metrics and more difficult metrics to
get a complex picture of care provided.
Summary
•  Much of the important aspects to evaluate have been
reviewed and discussed by many groups, but not specific
consensus has been developed from those
recommendations.
•  Still very few defined benchmarks for most of surgical
care based on limits of epidemiologic knowledge, but
summarizing ranges or making targets, as EmOC did for
c-section rates may be helpful to set targets.
Summary
•  Outcomes/quality are the most difficult to measure, but
have the most relevance for the patients receiving care.
Continuing to investigate how more resource intensive
approaches in HIC may be applicable. Significant training
of how to define and track this information may be
necessary.
Summary
•  Improving overall data collection infrastructure and
standardizing the way data is collected and compiled
could greatly improve the information available about
even more rare conditions – international surgical/trauma/
cancer registry work.
•  Sampling methodologies not addressed in many of these articles,
but could provide a more robust way to assess patient level data in
conjunction with aggregate level data.
•  Data audits to assess quality of reported information are also
needed.
•  Incorporate surgical data into standard survey – DHS, WB, etc
•  Leadership, education, and financial evaluations have the
least developed indicators currently.