The MRP – Development of a
comprehensive CMAM reporting tool
using a set of standardised indicators
CMAM conference London
17th – 18th October 2013
Presentation outline
1. The MRP development – background,
implementation etc.
2. The analysis, results and advantages and challenges
of using the MRP as information system
Background
• SFP review highlighted the inconsistencies, inadequacies and
bias associated with reporting of Supplementary Feeding
Programmes (SFP)
This means….
• Programmes can change the calculation of performance by changing
the denominator
• False “over-performance”
• OTP discharges included in new SFP admissions
• Transfers to TFP or medical care excluded from the
denominator
• Potential for improvement of programme quality is not recognised
and acted upon
• Data is not comparable between programmes/countries
Background cont…
• The ‘Minimum Reporting Package’ (MRP) was developed in response to
this paper with the initial intention of:
– supporting standardised reporting for emergency SFPs, in order to
improve programme management decisions,
– improve accountability
– assist urgently needed learning in the effectiveness of this programme
approach
• Development occurred over a number of years through a consultative
process amongst the global nutrition community
– 2009 – SFP indicators and reporting categories defined by a steering
committee of 12 international agencies (later piloted in 4 countries)
– 2011/12 – OTP and SC reporting categories added following
consultation
The MRP today
• The MRP has developed into a concise and comprehensive management
tool, providing a contextualised overview of the treatment components of
CMAM
• MRP comprises
– a set of guidelines defining indicators and reporting categories (both
basic and advanced)
– an access based software package and software guidelines.
• Some agencies use only the guidelines on indicators and reporting
categories to complement or improve their own systems
• The optional software package which allows rapid programme monitoring
facilitates consistent reporting and reduces the reporting workload of field
workers.
• Currently used by 7 NGOs in 15 countries (4 more countries in pipeline)
Common misconceptions
• Throughout this presentation, we hope to address
some misconceptions around the MRP, the most
common being:
– The MRP is all about software
– The MRP is not suitable for ministries of health as
implementers of CMAM programmes
• The primary goal of the MRP is the
standardisation of indicators and improved
reporting
MRP indicators – basic and advanced
Standardised indicators can be used at a basic or more
advanced level, depending on capacity for all
programmes (both SFP and OTP)
Case study example – Pakistan
• Pakistan has its own national reporting system – the NIS (nutrition
information system), in place since 2009
• Several features of the MRP could potentially improve the NIS for
enhanced utility, improved calculation of performance and to ensure
Pakistan’s CMAM data base is comparable internationally
• Save the Children is currently following a consultative process to work on
harmonisation of systems through introducing standardised indicators:
– A consultation meeting was held in September in Pakistan
– Many of the recommendations made by SCUK following this meeting
have been agreed to be integrated into the NIS
– On-going process integrated with CMAM guidelines revision
Case study example – Yemen
• The MRP software replaced excel sheet reporting which was
prone to many errors and had limited use
– “With the MRP we can see indicators directly when
entering data – a major advantage over the old system”
– “We can take actions if indicators do not reach Sphere in
single feeding sites”
– “Data in the MRP report format is shared with the
MoPH/UNICEF”
– “The MRP helps to improve the quality of the program.
Before starting the MRP, the defaulter rate of the program
was very high. With the MRP it has been easier to monitor
the data and to take corrective actions.”
Data collection through the MRP
• In light of the learning objective of the MRP data is collected
regularly by a group of MRP partners and feeds into a central
database. Analysis is on-going and leading to a larger analysis
planned for early 2014. The aims of these analyses are:
– To describe the characteristics of CMAM programmes
– To describe and assess the effect of CMAM programmes
on rehabilitating malnourished individuals
– To compare programme performance and outcomes
according to contextual factors, differences in protocols
or approaches
Methodology
• SFP and OTP data collected between January 2012 and July 2013 was
analysed
• The length of programme data differs but is generally above 3 months in
order to be able to analyse programme results
• Exclusion of data from analysis
– Data not in MRP format (due to time constraints with this analysis, ongoing analysis includes all formats feeding in data as long as MRP
definitions are used).
– Sites where numbers in charge at the end of one month did not match
the opening number for the next month (difference >5 excluded)
• All analysis is supported by the ERRB (emergency response and recovery
branch) team at CDC
Results – SAM
• OTP data was available from 3 NGOs, supporting 11
programmes in 8 countries (Burkina Faso, Chad, Ethiopia,
Ivory Coast, India, Kenya, Somalia, Yemen).
• After excluding data, a total of 14,995 admissions were
included
• 97.2% of admissions were classed as new admissions (only
four programmes reported relapses or re-admissions).
• MUAC was the most common admission criteria (78.6%) ,
but varied between countries and programmes
Results – SAM performance
•
•
Performance indicators showed
– Overall recovery rates of 80%
– Death rates of 1%
– Defaulter rates of 13.5%
– Transfer to TFP rates of 3.3%
– Non-recovery rates of 2.2%.
5/10 programmes reported
recovery rates above 90%.
SAM
• Some programmes do better than others
SAM
Presentation of data allows real time identification of
sites with problems
Results – MAM
• SFP data was available from 4 NGOs, supporting 10
programmes in 7 countries (Burkina Faso, Chad,
Ethiopia, Ivory Coast, India, Kenya, Somalia)
• After data cleaning, a total of 23,584 admissions and
15,496 were included
• The majority of admissions were new admissions,
(only four programmes reported relapses or readmissions).
• As with SAM, MUAC was the most common
admission criteria (81.3% of admissions)
Results MAM – performance
•
•
Performance indicators showed
– Overall recovery rates of 86.9%
– Death rates of 0.1%
– Defaulter rates of 10.8%
– Transfer rates of 0.9%
– Non-response rates of 1.3%.
6/10 programmes reported recovery
rates above 90%.
Results - MAM
• Some programmes perform better than others
Results – MAM
• Presentation of data allows real time identification of
sites with problems
Lessons learnt
•
The presentation of descriptive data in the standardised MRP format allows:
– real time presentation of programme data
– easy comparison of different programmes, protocols and organisations
– easy access to information on programme background and characteristics
– better reporting of defaulting, or any other discharge not recovered. The
graphs and validation tools highlight problems (including very high and very
low recovery rates) assisting management.
•
This allows programme managers to identify:
– programme characteristics
– the impact of specific events that may be affecting the quality and outcomes of
the programmes, and identify sites in need of supervisory support.
Emerging trends from the descriptive data
•
•
•
•
•
OTPs are implemented widely and overall obtain good results but with wide
variation
Despite similar protocols, the contexts and some characteristics of the
programmes were very different.
Overall results seem positive for SFP and borderline for OTP (high defaulters)
5/10 programmes for SAM and 6/10 programmes for MAM reported recovery
rates above 90%
– Poor programme performance or more accurate reporting and perhaps better
management?
Further investigation is needed into the difference between SAM and MAM data to
determine if programmes are better at recording SAM data, or if high recovery
rates are a true reflection of performance
Data quality
• In response to concerns about data accuracy, a quick analysis of actual
versus reported data was conducted in one SC programme.
• Defaulting rates reported as <1% actually looked closer to 30-40%.
• In another programme, children discharged as recovered were found to
not meet discharge criteria and should have been reported as nonrecovered.
• The same children were later reported as new admissions rather than readmissions.
• One constraint identified was the issue of all hard data being located in
field sites making verification difficult
• In response to this - The MRP is piloting a quality appraisal tool to help
assess the quality of the data.
Limitations and challenges
• Limited number of programmes and reporting duration
• Limited SAM analysis
• Only descriptive statistics are presented in this report, contextual data has
not been included in this analysis
• There is an important lack of data variation in terms of protocols and
performance
• Despite low numbers of defaulting overall, the lack of verification of
defaulting in MAM programmes through home visits may be masking a
higher mortality rate, particularly in programmes with high defaulting rates
• Questions around data quality
• Software often challenging in terms of joining files, bugs, etc
Next steps
• Continued advocacy for the use of standardised indicators in all
programmes addressing acute malnutrition
• Development and roll out of a web based version of the software with offline capability
• Addition of new features in web-based version of software
• Large scale analysis in early 2014
• Data quality audit to apply a level of confidence to how correct our data is
• Ensuring the MRP is a flexible tool which may take different forms
depending on the context whilst still generating comparable and unbiased
reporting.
• Apply results and experiences to improving the reporting indicators and
categories
Conclusion
• The MRP is a useful management tool which uses standardised indicators
to improve the monitoring and reporting of the treatment components of
CMAM
• It provides a comprehensive package for standardised monitoring of
CMAM treatment in both emergency and development contexts
• It can act as a ready-made system in contexts where no other reporting
system exists or elements of the package can be incorporated to
strengthen existing systems
• The MRP allows humanitarian agencies, donors and governments to better
monitor and compare performance of programmes in different contexts as
well as comparison of different CMAM approaches, and enhances the
management of CMAM programmes.