Symposium 6: Challenges in Monitoring
Clinical Program Outcomes
Developing an
Institution-Wide M &E paradigm:
The ORBIS experience
Abu Raihan, MD, MPH
Director of Program, Asia
Email: abu.raihan@orbis.org
IAPB 9th GA, Hyderabad, 17-20 September 2012
Outline
 Definition and purpose of M&E
 ORBIS Paradigm: Importance of clinical outcomes
 Principles of choosing and using key outcomes
 Multi-tier challenges & barriers to data collection
 Conclusions
2 | ORBIS International
Monitoring
 Systematic and routine collection of information
from the on-going programs for three main
purposes:
• To have internal and external accountability of the
resources used and the results obtained
• To learn from experience to improve practice and take
informed decisions
• To empower partners to deliver quality care
3 | ORBIS International
Evaluation
 Assessing systematically and objectively the
value of program.
 Evaluations help to draw conclusions about five
main aspects of the interventions:
• Relevance
• Effectiveness
• Efficiency
• Impact
• Sustainability
4 | ORBIS International
Key Elements in Designing the M&E
Framework
 Define relevant indicators and their measures;
 Identify and set benchmarks/standards;
 Determine data collection strategy;
 Analyze data and prepare reports for
dissemination and decision making.
5 | ORBIS International
Purpose of Institution-wide Approach of
M&E
 It is one of the important component of program planning,
development and implementation
 Standardization of data collection and information flow
 Provides checks and balances over program planning
and implementation
 Strong M&E leads to transparency and good governance
for an organization
6 | ORBIS International
ORBIS Paradigm: Importance of Measuring
Clinical Outcomes
 To measure excellence in training and practice for
quality patient care
 To provide evidence for advocacy to persuade policy
makers, partners to promote good quality eye health
 Demonstrate impact of good vision on individuals,
families and on communities
 Sustainability ensured when good outcomes attract
patients
7 | ORBIS International
Principles for Choosing Key Outcomes
 Outcomes must be appropriate and practical to
collect by partners
 Outcomes reflect quality
 Outcome data supports Continuous Quality
Improvement process
8 | ORBIS International
Complexity of Clinical Outcomes
 There is no one standard for all diseases
 For certain complex procedures, outcomes are the
result of expert consultations
– Pediatric cataract, glaucoma
– ROP
– Retina
 Current standards are based on international best
practices - PPP
 This is a good place to start, however, these
outcomes may need to be adapted as we put them
into practice.
9 | ORBIS International
Internal Challenges
• Data collection: who what when how frequent
• Cost to implement strategy: information system
and other resources
• Which software to manage data on the outcomes?
• Willingness and ability of staff / partners to collect
• Analysis, use, and dissemination of resultsrequires additional skills
10 | ORBIS International
External Challenges
 Cultural issues, not valuing the purpose
 Suspicion and fear
 System and resources lacking
 Privacy / legal issues in some countries to patient
data with foreign NGOs
 Partners feel alienated from M&E process.
– Often they have no inputs in the process- “one-size-fits-all”
requirements
– One-way, upward accountability, very little feedback loop.
11 | ORBIS International
Barriers to Outcome Data
Barriers to data availability
•No systems to collect/collate data
•No culture to record
•Suspicion/fear of recording
Sub-Saharan Africa
India
•Limited systems to collate
•Suspicion/Fear of sharing+/-
USA
•Good patient note
•Published and publicly available outcome data
12 | ORBIS International
Lack of information
Practical Approach to Clinical Outcomes
 Invest resources for
• Staff, training, and retention
• System for data collection and management
• Develop capacity to analyze data and give partners
feedback on the results: close the loop!
 Build or strengthen capacity of partners toward clinical
excellence in practice
 Bringing Ophthalmologists on board is the key !
13 | ORBIS International
Areas for Debate
 Should current standards be universal across all
countries?
 What are context specific appropriate standards?
 Ho do we change the culture?
14 | ORBIS International
Conclusions
 Challenges exist, though most can be addressed
through appropriate investment in resources
 Attitudes and perceptions are crucial but are often
difficult to change
 The shift to include measurement of clinical outcomes is
critical to good quality eye care programs for elimination
of blindness.
15 | ORBIS International
Thank you
16 | ORBIS International