Type of Review: Annual Review Quality Reproductive Health Medicines (QuRHM) Programme Date started: 1 August 2011 Date review undertaken: July – October 2013 Period reviewed: 1 August 2012 – 31 July 2013 (Year 2) Introduction and Context What support is the UK providing? The UK is providing up to £3.5m over 3 years, of which up to £3.2m will go to the implementing partner, Concept Foundation via PATH. The project is implemented under the auspices of the Reproductive Health Supplies Coalition (RHSC). The period of funding is 1 April 2011 – 31 March 2014, although project activities started 1 August 2011. A further £100,000 per annum was allocated for independent external review of the programme. What are the expected results? At impact level, the project will contribute to increased use by women of voluntary family planning methods of an assured quality standard (as per World Health Organisation and UNFPA quality standards). At outcome level, it is likely that by end 2014, the project will have generated savings worth at least the value of its inputs of around £3 million. WHO pre-qualified combined oral contraceptive pills (COC), progesterone only contraceptive pills (POP), emergency contraceptive pills (ECP) and injectables from generic or low-cost manufacturers, will contribute to estimated cost savings to the international community of US$6.4 million (conservative scenario) to US$32.4 million (optimistic scenario) with a medium scenario of $13.7 million savings by the end of 2016. Most of the savings are for lower procurement costs for injectable contraceptives from current cost of US$0.93 (from the international procurers database, the Reproductive Health Interchange) for the branded injectable Depo-Provera (DMPA, depot medroxyprogesterone acetate) to an estimated US$0.50(optimistic estimate) - 0.70 (conservative estimate). The business case is expecting £8m out of £13 million (medium scenario) of savings from injectables alone, over two years 2015 and 2016. This level of savings may be challenging. While several generic injectables are expected to achieve quality assured status by project end, market impact in terms of actual procurement savings may take longer to emerge. Greater cost savings are anticipated over time and can be applied to further procurement, hence increasing overall supply and reducing unmet need, if assumptions in the theory of change on market impact hold. In addition, major procurers and countries will have a wider choice of quality assured and approved suppliers, who can manufacture quality products at lower prices. This should also lead to an improved method mix available in country. What are the planned outputs of the project attributable to UK support? By mid 2014, as an attributable result of DFID funding, a total of 25 generic products (hormonal contraceptives and maternal health medicines such as misoprostal) produced by 15 manufacturers, from a baseline of zero at end 2010 (not including eight prequalified products from originator companies), and seven active pharmaceutical ingredients (baseline zero) will be approved. Six major procurers (including social marketing organizations), which are members of RHSC, will have agreed and adopted common 1 quality assurance definitions, and policies for purchasing essential reproductive health medicines. Four developing country governments will have accepted at least in principle the importance of internationally agreed quality criteria in their own procurement and tendering processes. How will we determine whether the expected results have been achieved? UNFPA manages a database, Reproductive Health Interchange, where major procurers log their activities. The DFID Business Case expects to see a downward trend in unit prices for selected commodities and an increase in volumes procured. Country level reports should indicate that an expanded method mix is available to women. This is being tracked at outcome level in the log frame. In addition, given the high risk nature of this project, DFID in partnership with the Bill and Melinda Gates Foundation will invest in an annual review of progress, contracted to an independent consultancy, HERA (called the Review Team in this report). A full evaluation, to be funded by the Gates Foundation, is planned for 2015/16. What is the context in which UK support is provided? Over 220 million women have an unmet need for family planning. Meeting this need is one of two priorities in DFID’s Choices for women: planned pregnancies, safe births and healthy newborns – the UK’s RMNH Framework for Results - which aims to enable women to access contraceptive services and improve their reproductive choices about when and how many children they have. The Framework includes the UK goal for reaching 10 million new family planning users by 2015. This is a contribution to UN Secretary General’s Global Strategy for Women’s and Children’s Health, and the July 2012 London Summit on Family Planning (FP2020) goal of giving 120 million more women and girls access to lifesaving contraceptives by 2020. The DFID Human Development Department offer for 2015 outlines £20 million savings/per annum to the global reproductive health community through driving down the prices of quality family planning methods. This project is one of several interventions aimed at achieving that saving. Lack of quality, affordable contraceptives is a major barrier to meeting unmet need and increased use. To date, the quality market for hormonal contraceptives and reproductive health medicines is dominated by originator companies and their products remain more costly (e.g. injectables, implants and reproductive health medicines such as oxytocin and misoprostol which are used to prevent bleeding following delivery). Although cheaper products are available from generic companies in India, Brazil and China, they are not always quality assured by international standards1. WHO carries out independent assessment of essential categories of product through its pre-qualification programme (PQP). In addition, where quality assured products are in short supply, and companies may not be ready for a full dossier review but are making progress, a commissioning agency (UNFPA for reproductive health medicines) may request WHO to carry out an expert review panel (ERP) review, which provides a risk assessment for major procurers to use. For reproductive health, the first ERP was held in 2013. Some reproductive health medicines that do not have Stringent Regulatory Authority (SRA) approval or WHO pre-qualification are approved by National Medicines Regulatory Authorities (NMRA). Most low and middle income countries have an NMRA but concerns have been raised about the standards that these agencies apply, and their capacity to perform the required checks of manufacturing practice and product quality is variable. 1 Where quality assured means approval by an SRA (eg USA, Europe); pre-qualified under the WHO Prequalification Programme for Essential Medicines (PQP); or assessed by a WHO Expert Review Panel (ERP) as category 1 or 2. 2 Section A: Detailed Output Scoring Output 1: Increased availability of affordable quality assured RH medicines for supply to less developed countries Output 1 score and performance description: A Concept Foundation has provide the technical assistance to companies, which has enabled a significant increase in the number of quality assured generic products to 17, from a baseline of zero, and the output score is rated A. Output 1 is the most important of the four outputs, with a weighting of 40%. Progress against expected results: Evidence for achievement of the output is the range of WHO prequalified (PQ) or ERP approved reproductive health Finished Pharmaceutical Products (FPP) and Active Pharmaceutical Ingredients (API) that are available for purchase by international and national procurers against a baseline of zero. Concept Foundation’s inputs include dedicated advice and technical support to companies interested in entering the global quality assured market and assessed as being capable of reaching agreed quality standards, for both finished products and the APIs. Concept also supports the preparation of dossiers presented by companies to WHO ERP and PQ assessment panels, and at agreed stages during the assessment process. Concept works very closely with both WHO’s PQP team, and UNFPA (who commission the ERP assessments as needed from WHO). This year the Review Team was able to review internal Concept Foundation documents, and it was able to see the correlation between Concept’s technical assistance with generic suppliers and the eventual success of those same suppliers in securing ERP or PQ approval. The revised 2013 logframe lists 3 milestones for year 2, one delayed 2012 milestone and one planned milestone for 2014. 12 additional generic products approved under ERP, resulting from 2 ERPs 4 RH APIs prequalified or ERP assessed category 1 or 2 Number of RH dossiers accepted [by WHO PQ] over 2 years of the QuRHM project, as compared with 2008/9 Time elapsed between submission of dossiers and product PQ between 2008-2010 (2014) Business case for manufacturers completed/disseminated For the 1st milestone year 2 (12 additional generic products approved under ERP, resulting from 2 ERPs) the Review Team noted 3 new approvals in ERP-2013 in addition to the 12 already achieved in the 2012 ERP. Thus in total 15 products were approved in 2 ERP procedures. Concept provided technical assistance to companies for all these products. Sub-score: A+. As of July 2013,10 quality assured generic products (3 PQ and 7 ERP) were available. This is 3 more than the 7 per 31 July 2012 (2 PQ and 5 ERP). The pipeline is well filled: Concept provided technical assistance for four more products, which then gained WHO PQ status, and three more were granted an ERP extension by September 2013, resulting in a total of 17 quality-assured generic RH products being available for procurement. This is a substantial increase compared to year 1 and a significant achievement from the baseline of zero. (Note that the quality assurance status of an ERP assessed product expires after one year, unless the company has entered it to the PQ process or it has been given an extension by UNFPA. So by September, 10 of the 15 ERP approved products were still valid). With regard to the 2nd milestone year 2 (Four RH APIs prequalified): only one RH API was prequalified per 31 July 2013 as per WHO PQ website (against a baseline of zero on 31 July 2012). The Review Team notes that a 2nd API was prequalified in August 2013. However, 5 RH APIs (all in receipt of technical assistance from Concept) are under assessment and likely to be approved. As 3 such, the milestone is not achieved, but good progress is being made. Sub-score: B. The 3rd milestone year 2 (Number of RH dossiers accepted by WHO over the 2 years of the QuRHM project vs. 2008-2010) is presented as a new Value for Money (VfM) indicator. Concept Foundation has not yet been able to collect (or receive) baseline data from WHO. The WHO/PQ website shows that 11 RH products (generic or originator) were under assessment in WHO PQ as per 8 Aug 2013, but these could have been there for many years. As there is no quantitative proof in this VfM indicator, the Review Team is unable to assess this specific milestone. (No sub-score allocated) The 4th milestone (Time elapsed between submission of dossiers and product PQ between 20082010) is meant for assessment in year 3. The Review Team suggests that the formulation of this indicator is improved before assessment in 2014, and that baseline data is collected. (No sub-score allocated). The 5th milestone (dating from year 1) (Business case for manufacturers completed and disseminated): was not achieved by July 2013. The business or investment case summarises the case for investment in international quality standards by generic companies, providing data on market size and opportunity. A draft report was made available to the Review Team. Generic manufacturers (the main target group) saw presentations of the business case during the “Generic Manufacturers Caucus for Reproductive Health” (GEMS Caucus) meeting in Delhi (9 October 2013), which enabled useful feedback. A separate, WHO-developed business case was also presented by BioBridge in the Delhi meeting. Concept Foundation and WHO have agreed and are developing a joint publication which has delayed progress. Sub-score: B. Concept Foundation has developed a “critical pathway” tool to monitor the timely progress of each product through the quality assurance processes. The Review Team agrees that this is a possible monitoring tool for showing how Concept and WHO/PQP technical assistance and assessment inputs lead to a quality assured product. However, WHO had not yet included its comments and data in the critical pathways documents submitted to the Review Team. Concept Foundation has responded to the recommendation from Year 1, to prioritise and reduce the number of products supported. The total has dropped from 25 (16 product and 9 API manufacturers) with ongoing technical support, to 23 (12 product and 11 API manufacturers), with an appropriate shift to more APIs. CF has successfully brought together around 10 generic manufacturers in the “Generic Manufacturers Caucus for Reproductive Health” (GEMS Caucus), which is a notable achievement, given the previous fragmentation of the industry. A joint WHO/UNFPA/UNICEF meeting with manufacturers was planned for September 2013. The Review Team has noted that Concept will make a presentation. So far, most quality assured products are combined oral or emergency contraceptive pills. However, no injectable contraceptive, oxytocin, mifepristone, or magnesium sulphate, and only one misoprostol product have been prequalified or ERP approved. Recommendations: Concept Foundation should prioritize (in the remaining project months) technical assistance to ensure that injectables, oxytocin, misoprostol, mifepristone, and magnesium sulphate are prequalified, and to monitor closely the composition of the commodities submitted for ERP and PQ, given that the quality assured list comprises mainly oral combined and emergency contraceptives. Concept Foundation and WHO/PQP should continue collaborating on the two business cases to ensure synergies, to produce a joint summary product by end 2013, which includes factual and financial information (such as forecasts per product / country / time period, information on potential donor budgets and future order possibilities). Concept Foundation and WHO/PQP should complete the joint critical pathway tool for each product, to enable progress monitoring (in time for the annual review in 2014) Concept Foundation and DFID should set absolute (rather than relative) numbers for milestones, eg for assessing VfM, by early 2014. Impact Weighting (%): 40 4 Revised since last Annual Review? Yes (logframe) Risk: High changed to Medium Revised since last Annual Review? Yes (the risk is considered less by the Review Team) Output 2: Major international procurement organisations have harmonized quality assurance definitions, policies and practices Output 2 score and performance description: B This output is the second most important, and scored a marginal B (nearly an A). There has been extremely good progress towards bringing major public sector procurers to common quality standards and policies. UNFPA as a leading procurer, together with WHO, have supported this process well, through consensus building workshops and close liaison with USAID. However, UNFPA had not published the procurement guidelines by Year 2 end. Progress against expected results: Achievement of Output 2 is measured by “improved QA policies and procedures in place at international level based upon procurement of only PQ or SRA approved products”. UNFPA’s quality assurance policy for reproductive health medicines (originally published March 2011, and a baseline indicator in 2012) was updated in November 2012. DFID also published a Quality policy for RH medicines in June 2012 (with an update in September 2012) after discussing it with country offices and procurement organisations. Concept Foundation provided input into the finalisation of this policy. Concept Foundation, RHSC, UNFPA and DFID had discussions in 2012 with USAID and KfW to also revise their quality policies. Two milestones were formulated in the revised 2013 logframe for year 2: 1. Two additional procurers to adopt and use policies based upon the UNFPA policy (or evidenced by a transition plan) 2. UNFPA developed and published guidelines for purchasing quality products The 1st milestone year 2 (2 additional procurers to adopt and use policies based upon the UNFPA policy (or evidenced by a transition plan)) is considered achieved as the MSI and IPPF transition plans were accepted by DFID (Sub-score A). The Procurers’ meeting in New York (29 April – 1 May 2013) also contributed to a better understanding of the challenges that procurers face in the target countries, as evidenced through the meeting minutes and agreements by USAID in particular to work more closely with UNFPA and WHO. 2nd milestone year 2 (UNFPA develop and publish guidelines for purchasing quality products) is not completed. There appeared to be confusion about who is responsible for developing these and what form they should take (UNFPA or WHO). UNFPA Procurement Services Branch (PSB) have confirmed that they will publish guidance in line with UN good practice before project closure on 31 March 2014. As the main contracted party, Concept Foundation remains responsible for this milestone not being achieved. (Sub-score B) A “value for money” indicator was added for year 2 (Reduction in the number of separate technical assessments undertaken by UNFPA compared with number undertaken in 2011). 2011 baseline data was not available. Concept reported that no separate technical assessments were undertaken by UNFPA in 2012 and 2013, so the frequency can be assumed to be less in 2012/13 than in 2011. The review found that literally this indicator can be seen as achieved (Sub-score A). Procuring from a quality assured supply source should also reduce/replace the need for annual and 5 repeated risk assessments by individual procurers, in the longer term. Regardless of whether Concept Foundation actually spent more or less on substituting UNFPA visits to companies, fewer technical assessment over time means that VfM will increase. The Review Team recommends that the definition of the VfM indicator is strengthened, as it only measures the value on one side (fewer assessments by UNFPA), and not the money (the costs of Concept Foundation assessments could have been more costly than those of UNFPA assessments). Also, Concept could have replaced the UNFPA assessments with more visits than UNFPA would have undertaken. Recommendations: As the lead organisation, Concept is recommended to better manage/monitor the milestones that are to be achieved by partners, and to provide a short report informing DFID of how this is being done with UNFPA, by end of 2013/early 2014. Concept and DFID should agree on how to improve the VfM indicator with regard to assessing and comparing the actual costs of technical assessments pre and post project, by end 2013/early 2014. Impact Weighting (%): 30 Revised since last Annual Review? Yes (logframe indicators) Risk: Medium Revised since last Annual Review? No 6 Output 3: Commitment to strengthen national procurement criteria, based upon internationally accepted quality assurance definitions, developed by key stakeholders in four countries (Ethiopia, Kenya, Senegal and Nepal), working through RHSC members, WHO and UNFPA. Output 3 score and performance description: C Implementation of the baseline study and in-country activities is seriously delayed, with the country workshops now planned to take place towards the end of the project. Delays are due to the poor quality of subcontractor reports and the need for Concept to do additional data analysis, with constraints on time (given demands on key staff in 2012/13 generated by the unplanned activities arising from the Family Planning Summit and its follow up). Progress against expected results: Establishing current quality assurance practices related to procurement of reproductive health medicines in 4 pilot countries (Ethiopia, Kenya, Nepal and Senegal) was a year 1 milestone, but the four final assessments were still not available end Year 2. The output language was amended as compared to 2012 logframe, in response to the recommendation from Year 1 review, to make the end point more realistic for a three year project. Milestone 1, years 1 and 2 (Completion of situation analysis in 4 countries to provide baseline and documented data on market dynamics and attitudes to quality available): Activities to collect the baseline in the countries started October 2012 with a request for proposals. The contract was awarded to the Royal Tropical Institute in Amsterdam (KIT). By July 2013 only one draft report (Nepal) was available to the RT, although KIT has presented some initial findings to the April procurers’ meeting. The milestone is not achieved. (Sub-score C) Implementation of the baseline study and in-country activities is now seriously delayed. Partially this is due to the four pilot countries only having been selected in August 2012. Some of the delay can be explained by the substantial and unplanned activities around the July 2012 Family Planning Summit, which CF had been requested to undertake. The RFP for the country studies was only published in October 2012. The contract was granted in December 2012 to KIT, with an expected delivery date of 31 March 2013. The delay in final reports appears to be due largely to delayed submission of draft reports and weaknesses in data collection and presentation that Concept has had to address itself. Concept has stated that the data and reports submitted by subcontractor KIT were inadequate, and that CF had to invest considerable time to validate data and review the analysis. In one country (Ethiopia) the data collectors experienced serious problems with official permission to collect the data. Nevertheless, Concept Foundation remains managerially responsible. The delay in generating the country baselines is now also affecting the work at country level. UNFPA has been postponing national workshops as the baseline studies were not yet available, although has started to plan the country workshops with UNFPA country offices. Given the delay in producing country baselines, all country workshops are now planned in the last quarter before project closure (2014Q1). The workshops risk being organised in some haste, and need involvement of the many RHSC members at country level. DFID is also suggesting a greater involvement of UNFPA Commodity Security Branch (Global Programme to Enhance RH Commodity Security) in the country work in order to gain high-level policy involvement. Substantial advocacy efforts may be needed to promote the use of quality assured products in countries, especially if these would be more costly than already registered products. Achieving the planned output in year 3 seems very ambitious. Even when all workshops are delivered by 31 March 2014, this will hardly allow any time for local follow-up. 7 Recommendations: RHSC, Concept and UNFPA are recommended to develop a workplan for each country, including stakeholder workshops and high level policy dialogue, and increase the involvement of RHSC members in the country activities (with UNFPA country office, PSB and CSB), by end December 2013 Concept Foundation is recommended to better manage, supervise and monitor its outsourced activities, and provide a report showing how it is doing this to DFID in early January 2014 Impact Weighting (%): 15% Revised since last Annual Review? Yes (output language) Risk: High Revised since last Annual Review? No Output 4: Awareness of the QuRHM strategy and related quality issues raised among donors and other high-level stakeholders within the RHSC membership Output 4 score and performance description: A There has been good progress on this output, with Concept playing a prominent role in ensuring that quality is now one of the RHSC’s four core pillars in its revised strategy. This reflects Concept’s contribution to generating better consensus among all interest groups in the RHSC (donors, procurers, suppliers, providers, civil society) on the definition of quality assurance standards. Progress against expected results: There are 2 milestones in year 2 for this output: 1. Quality assurance issues included in RHSC working group plans and annual RHSC meeting 2. Key non-procuring donor members of RHSC agree to strengthen funding agreements related to product procurement. Regarding the 1st milestone year 2 (Quality assurance issues included in RHSC working group plans and annual RHSC meeting): Concept Foundation presented QuRHM progress to the Market Development Approaches Working Group (MDA WG) in October 2012, and a written update to the Systems Strengthening Working Group during the annual RHSC membership meeting in October 2012. The RHSC website (www.rhsupplies.org) has three presentations by Concept Foundation on progress of the QuRHM project (although there are no plans, progress reports or other information). MDA WG produced survey results that 43% of members would (17% maybe, and only 9% not) participate in group activities on QuRHM (e.g., regular calls, occasional meetings, joint activities). This shows that the topic of quality is seen as important by MDA WG members. On the other side, a few members commented to theReview Teamthat Concept Foundation has been running the QuRHM project more as a separate vertical project, where they have little influence or interaction. The Technical Advisory Committee met once by phone and once in person, and is agreeing revised terms of reference to build RHSC involvement in the wider quality agenda, with an eye on the new draft strategy and the need for an integrated and systematic approach to quality across the RHSC’s activities. It is very positive that quality will become one of four central pillars in the new RHSC strategic plan being developed. RHSC management confirmed that a Quality Working Group is being considered in RHSC. This will give the issue of quality a prominent discussion platform among RHSC members 8 during and outside RHSC conferences. (Sub-score A+) Regarding the 2nd milestone year 2 (key non-procuring donor members of the RHSC agree to strengthen funding agreements related to product procurement): In year 2 both KfW and DFID included in their procurement policies a preference for prequalified or ERP approved RH products. USAID has also agreed to broadly align with UNFPA policy. (Sub-score A) Recommendations: Concept Foundation (with inputs from DFID) are recommended to make the milestones of output 4 more SMART, by end January 2014 Concept Foundation is advised to work with the RHSC to publish QuRHM project documents, reports and annual review on its website, by end 2013 Concept Foundation (and UNFPA, the co-chair) should continue to ensure that the Technical Advisory Committee is engaged and its role and membership developed in line with RHSC’s new strategy and objectives (by end March 2013) Impact Weighting (%): 15% Revised since last Annual Review? Yes (milestones) Risk: Low Revised since last Annual Review? No Section B: Results and Value for Money 1. Progress and results 1.1 Has the logframe been updated since last review? Yes Although the programme was supposed to last three years, activities only started 1 August 2011, due to late approval. With an end date of 31 March 2014 this means that the project is being implemented over 32 months only (as opposed to 36 months). The 2012 Annual review covered the period 1 August 2011 – 31 July 2012; this review covers the next 12 months, i.e. 1 August 2012 – 31 July 2013. This means that the 3rd year will only cover a period of 8 months. The logframe was adapted by Concept Foundation (on request of, and agreed with DFID) in response to the 2012 Annual Review agreed recommendations, including prioritising and reducing the number of companies provided with technical assistance and strengthening the VfM indicators. The Review Team now feels that some of the new indicators are insufficiently defined. There are some methodological problems with the 2013 logframe. Some indicators are not SMART or can be interpreted in different ways. Others are relative rather than absolute indicators (while the baseline data are not available or collected). 1.2 Overall Output Score and Description: B The overall score in this review is a B. The Review Team found reasonable prospects for an A by project completion next year. In particular, the project is doing well on Output 1, with the significant progress made with regard to the growing number of quality assured generic products, and on Output 4, with growing prominence and consensus on quality standards and assurance across the RHSC’s diverse stakeholders. The results for Outputs 2 and 3 were disappointing, largely driven by delays in delivery by partners and subcontractors, which Concept should have monitored and managed more actively. 1.3 Direct feedback from beneficiaries: 9 While the ultimate beneficiaries would be women, girls and men in developing countries who should gain better access to a wider range of family planning commodities, intermediate beneficiaries can be identified as (i) generic manufacturers, and (ii) members of the RHSC. In this review the Review Team approached the intermediate beneficiaries. Manufacturers of generic medicines supported by Concept are in general very satisfied with the technical assistance they receive. All are supporting the goals of QuRHM. Manufacturers with successful PQ or ERP products presented themselves as the Generic Manufacturers Caucus for Reproductive Health (GEMS) caucus group in the Delhi RHSC conference. RH companies with poorly designed factories or not interested in pre-qualification, have already been deprioritised by CF in last year’s review of all manufacturers. Non-supported manufacturers interviewed last year often stated that other financial, market and strategic reasons prevented them from going for WHO prequalification for RH medicines. The Review Team interviewed RHSC members during and after the annual conference in Paris 1-5 October 2012 and New Delhi 7-11 October 2013. RHSC members aware of the QuRHM project were in general quite positive about the activities of Concept Foundation, but also noted that they are not (yet) seeing the impact, such as quality assured RH products available in target countries, or generic alternatives for originator injectables (or implants, which are not a target for this project). They also listed other barriers for access to quality affordable RH medicines that are currently not included in the project design or Concept Foundation activities, such as national registration, procurement rules, and traditional product choice, for example (and for which Concept would not be best placed to address). In general, they were hopeful that the Concept Foundation activities would in the medium term lead to better choice of quality products. There was no consensus whether this would have an impact on the price level of the products. 1.4 Summary of overall progress The overall year 2 score of the project is a B. This is due to a few indicators in the revised logframe not being achieved or lagging seriously behind. The project has made good progress with generic manufacturers (output 1). The number of PQ or ERP approved products has increased from 0 (baseline) to 7 (as of July 2012), with an additional 10 (by July 2013) rising to 17 (September 2013). The pipeline is showing several promising candidates. An A+ would have been possible, if the business case work had been delivered in time. On the other side, there are still no prequalified generic injectables, oxytocin, magnesium sulphate or implants2. There is one misoprostol, and in emergencies (no availability of the originator product) one generic injectable can be procured under an ERP-3 rating. The Review Team can see from internal confidential documents that CF has been working very hard to get the injectables, oxytocine, misoprostol and magnesium sulphate products to ERP or PQ, and the project completion milestones for Output 1 are likely to be met. The Review Team also noted progress in output 2 (procurers). Procurers are now more optimistic than last year, as they have more options for procurement of quality RH medicines. However, the New York meeting also noted substantial barriers at country level, and procurers wondered who will support the generic manufacturers of the not yet quality approved products after the closure of QuRHM project. Unfortunately the lack of procurement guidelines due to confusion about who is responsible for making them (UNFPA or WHO) meant that this output scored only a B. Output 3 (country activities) is seriously behind schedule: it is yet to start with activities in-country as the ‘baseline’ country studies were delivered late. Even if UNFPA manages to hold workshops in all 4 pilot countries, there will be no time for follow-up. The Review Team is also concerned about the lack of involvement of RHSC members in the country workshop planning. This output scored a C. However, this output is not critical to achieving the outcome, because price savings in the first instance will be 2 Implants are supported by FHI360 funded by BMGF outside the QuRHM project. 10 generated by the global procurers, mainly UNFPA. Output 4 (advocacy) is making good progress and scored an A. Quality will become the fourth pillar of the new RHSC strategy, and RHSC plans to set up a Quality Working Group which can continue the work started by CF under QuRHM. Overall, the project is making progress in enabling more generic RH products to be prequalified or ERP approved, with more products in the pipeline. The overall B score is the result of some milestones not being achieved in outputs 2 and 3, mainly due to delays in producing country reports, the business case (with WHO) and procurement guidelines (with UNFPA). By project completion, the project could still achieve a score of A or A+ in outputs 1, 2 and 4. Output 3 (country work) remains a serious challenge. The project is also at risk of not achieving at the outcome level due to delays in approvals of injectables, upon which the cost savings and value for money case rests. Delays in injectable approvals are largely due to the need for bioequivalence studies, which can take several years and which the Gates Foundation is funding separately. The absence of these study results means that the products cannot be assessed as ERP category 1 or 2, but as category 3 (objections to procurement unless there is no alternative). However, there are signs that more quality assured injectables will drive lower prices. The three month injection (baseline price of $0.86) has increased to over $1 due to quality supply shortages. One product was assessed as ERP category 3. In case procurement is necessary, UNFPA has already negotiated a long term agreement for the product at a price well below the original baseline. While most of the cost savings should be generated through the large procurers (USAID, UNFPA etc), the wider barriers in many developing countries (which are not addressed by this project) still persist (registration, procurement regulations, consumer behaviour and preferences). The Review Team also concluded that the three year time frame of the QuRHM project was too short for gaining full country buyin for procuring only quality approved products. 1.5 Key challenges Although Concept Foundation did well in outputs 1 and 4, they are still not adequately monitoring some deliverables of subcontractor KIT or partner UNFPA in outputs 2 and 3. While there were outside factors (work linked to the London Family Planning Summit 2012) which meant that the country work started late, Concept Foundation as lead contractor remains overall responsible. The Review Team repeats its 2012 recommendation that CF needs to pay more attention to monitoring including accurate forecasting of expenditure on activities and follow-up. Part of this is inherent to the organisation: CF did appoint new management team members, but CF top management remains overloaded with work, and has apparently not delegated the troublesome project areas to other staff in the organisation or outside consultants. The many other barriers down the pipeline (from national registration and procurement at country level to the actual consumers with unmet need) are not yet being addressed and it is not clear that the country level activities were the right strategy to address this longer term change. A “post QuRHM” strategy was discussed at the Technical Advisory Committee in October, in the context of the new RHSC strategic plan where quality is one of four pillars. This needs to continued among RHSC management and development partners, building on current Technical Advisory Committee discussions. Concept Foundation has relied on its partner UNFPA for its country activities. An opportunity is being missed to seek assistance and involvement of RHSC members for its country activities. Output 3 should have been reformulated to take account of (1) what other programmes and partners are already doing at country level and to ensure that relevant linkages are built; (2) the specific expertise in Concept Foundation and RHSC members, and (3) more realistic timeframes. Concept Foundation’s responsiveness and communication to partners has improved somewhat, but is still being noted as sometimes slow and inadequate. 11 The project remains medium to high risk, but the potential outputs are worth the extra wait. The pipeline of generic products contains some badly awaited RH medicines (especially injectables). A more focused approach towards these products is recommended. In retrospect, the log frame should have differentially weighted results for different product types, giving more value to PQ or ERP success in injectables. The project design is still focused on technical issues on the manufacturers’ side, and does not pay enough attention to the economic challenges. The business case for generic RH companies to invest in PQ remains weak. Concept Foundation and WHO PQP explored different pathways to resolve this, and are now working together to maximise the value of their two approaches. Collation of market data, needs and future order quantities is needed. 1.6 Annual Outcome Assessment Given the pipeline of generic RH products being serviced by Concept Foundation, the Review Team is reasonably confident that the number of prequalified or ERP approved generic RH medicines will increase over the 17 products available in September 2013. However, the remaining time until Project Closure (31 March 2014) might be too short to achieve all the year 3 milestones. The country based outputs are unlikely to be achieved, by end March 2014. It is likely that by end 2014, the project will have generated procurement savings worth at least the value of its inputs of around £3 million. Prices for quality assured oral contraceptive products have decreased, in accordance with the theory of change assumptions. This is a positive sign, and it is very likely that QuRHM activities have contributed to these price reductions by enabling more quality assured competition. These price reductions are beginning to deliver procurement savings. UNFPA has completed its first procurement of 0.45m oral pill cycles with a company producing the pre-qualified product at the lower price, generating a saving of $30 ,000. UNFPA is in process of tendering and developing long term agreements with all the newly qualified manufacturers, so it is not yet possible to assess the total cost savings related to those products. Cost savings are also expected from injectable contraceptives. The three month injection (baseline 0.86) has increased to over $1 due to quality supply shortages. One product was assessed as ERP category 3, for procurement only where there is no alternative. In case this is necessary, UNFPA has negotiated a long term agreement for the product at a price well below the original baseline. Given the ambitious time frame, the weak economic incentives (probably underestimated by the donors) and the late development of the business cases (by WHO and Concept Foundation separately) the overall project approach may have difficulties achieving the indicators at outcome level. Any future approach would need to (1) address the economic realities of RH medicines’ markets; (2) be more aware of the downstream factors; (3) involve RHSC members working in/with countries, and (4) strengthen links with existing other programmes working on RH in those countries. Impact would also increase if RHSC members perform activities beyond the four pilot countries and if injectables can be advanced to PQ rapidly. It is unlikely that this project can address all these factors in its remaining time, but it would be essential for any successor activities to address these, across all partners. For longer term change, more attention needs to be given to barriers on the demand side (national registration, procurement rules and systems at country level, conservative attitude of procurers and consumers to stick with existing brands, etc). In some aspects Concept Foundation could play a role (eg national registration), although WHO is perhaps better placed. In other areas (national advocacy, procurement) RHSC members should play a more active role. Recommendations The Review Team recommends that a DFID logframe specialist review the logframe, in particular the VfM indicators, and make recommendations to Concept in early 2014. The project’s in country activities are delayed. Concept and UNFPA should develop a workplan showing how these activities can be delivered and in what time scale, by end 2013. This workplan should include involvement of the RHSC community, to address build consensus in the countries for addressing the downstream barriers. As it stands now, the planned evaluation (scheduled to take place before 31 March 2015) might 12 be too early to show tangible results, as market entry of quality approved products (PQ or ERP) might take more time than envisaged. The Review Team recommend that it takes place in 2015/16. 2. Costs and timescale 2.1 Is the project on track against financial forecasts: No The actual invoices to DFID show 89.3% of the annual budget spent in FY1 (2011/12) and 84.5% in FY2 (2012/13). For the final FY3 a spend rate of 96.8% is predicted. Around half of the predicted underspend (around £400,000 on the whole project) is because it was envisaged that DFID would fund all reviews and evaluation, amounting to £100,000 per year but in fact this component was co-funded with the Bill and Melinda Gates Foundation. The remaining underspend in year 2 is due to the late start and slow spend on activities for Outputs 2 and 3. See 2.3 2.2 Key cost drivers The key cost drivers are staffing, technical experts, travel, and workshop costs. No changes were noted in these cost drivers. 2.3 Is the project on-track against original timescale: No Outputs 1 (working with manufacturers), outputs 2 (procurers) and 4 (advocacy) are slightly behind schedule, but could still complete all activities. Output 3 (country work contracted out to UNFPA) is however seriously delayed due to late country ‘baseline’ reports, and unlikely to complete all activities by 31 March 2014. 3. Evidence and Evaluation 3.1 Assess any changes in evidence and implications for the project No external changes were observed. The Theory of Change was probably too optimistic in its assumption that countries would change behaviours and only allow prequalified or ERP approved generic RH products into their countries. The existing non-prequalified products are already registered by the national medicines regulatory authorities, and the procurement authorities see both groups (prequalified and locally registered products) as “good quality” for procurement. More evidence is needed to make the case that the existing products are or may be poor quality (which is difficult) or that the prequalified products are less costly than the existing products (unlikely at short notice), so that prequalified products will be procured instead of the existing products. Policy decisions to introduce these procurement principles are possible, but will take more time. 3.2 Where an evaluation is planned what progress has been made? The Scope of Work for HERA includes also undertaking an impact evaluation of QuRHM in 2014/15 to be funded by the Bill and Melinda Gates Foundation. This would probably be best scheduled in 2015/16, to allow maximum time for quality assured products to reach their target groups. The terms of 13 reference will be drafted in 2014. 4. Risk 4.1 Output Risk Rating: Medium (changed from High) 4.2 Assessment of the risk level The Review Team reduces the overall risk assessment in the new 2013 logframe from High to Medium. This is mainly due to output 1 performing better. The Review Team recommends changing the Risk Assessment in annex E of the Business Case: Risk 1 (lack of supplier willingness to accept technical assistance): the probability remains Low Risk 2 (perceived lack of transparency of WHO process): this was not assessed in this review although the process is reported as clearer by the manufacturers working with Concept (but remains Medium) Risk 3 (procurement agencies do not agree/abide by quality policy): the probability has been reduced from Medium to Low, given progress made Risk 4 (more PQ products do not lower prices): the probability remains Medium, given the uncertainties on market impact 4.3 Risk of funds not being used as intended The Review Team did not find or observe any indication of improper use of funds. See also VfM discussion below. 4.4 Climate and Environment Risk There are no changes in the risk assessment as per the business case. The project is likely to have a potentially substantial positive environmental impact (increased voluntary family planning use with implications for fertility and family size) and a limited direct detrimental environmental impact. Carbon emissions will depend on frequency of meetings and travel to chosen countries. To mitigate this meetings are held virtually where possible (VCs and teleconferences) and “piggy-backed” on other international meetings. Through supporting and enabling pre-qualification, the project is helping generic manufacturers to achieve internationally approved 'good manufacturing practice', which includes safe disposal of hazardous waste during the manufacturing process. There is also some downstream risk concerning unsafe disposal of unused or expired contraceptives at country level by clinics and users. DFID’s investment with UNFPA’s Global Programme for enhanced Reproductive Health Commodity Security includes an indicator and activities with country partners for implementing guidance on correct disposal. 5. Value for Money 5.1 Performance on VfM measures Economy: With regard to key cost drivers, the financial reports of Concept and partner UNFPA do not provide details of unit costs or volumes, as they collate all costs under high-level budget lines. CF has did not report on VfM in its annual report to RHSC or in its financial reports to DFID via PATH. This makes it difficult for the Review Team to assess VfM in project implementation. However, UNFPA staff salaries and short term assistance fees compare well against comparator 14 benchmarks. UNFPA, in common with most of the UN development system is signed up to the International Civil Service Commission which sets salaries at professional grades based on US civil service comparators (with allowances depending on location)i. For locally engaged staff the basket comparator system comes into effect, where local salaries and allowances surveys are used to competitively benchmark national postsii. UNFPA’s international and national fee schedule for short term technical assistance is competitively priced, with a ceiling of $750 per day for international high level experts (similar to rates offered by the European Commission and below those paid by most international consulting companies)iii. Concept Foundation’s average daily rates are consistent with WHO fees for similar work and well below commercial rates. The CF travel policy was adapted for QuRHM related travel and is in line with DFID travel requirements (economy travel). The issue of “pass-through” funds (DFID-PATH-CF-UNFPA-WHO) has been addressed, and a formal contract will be in place for the next ERP between Concept and WHO for the final process under the current grant (thus avoiding additional overhead costs). Efficiency: In response to the recommendation of the 2012 Annual Review, Concept Foundation included two VfM indicators in the 2013 logframe: Output 1 – (Number of RH dossiers accepted over 2 years of the QuRHM project in comparison with 2008/9) Output 2 – (Reduction in the number of separate technical assessments undertaken by UNFPA in comparison with the number in 2011) The Review Team is not convinced that these are good VfM indicators, as per the specific comments on these VfM indicators in the respective outputs, and recommends that further work is done to strengthen them. Effectiveness and overall VfM: This is largely based on anticipated cost reductions of hormonal contraceptives when more generic products are prequalified. Reported prices for prequalified products have fallen from the baseline, for both combined oral and emergency contraceptive pills. This is a very positive sign, and in accordance with the theory of change assumptions. These price reductions are beginning to deliver procurement savings. UNFPA has completed its first procurement of 0.45m oral pill cycles with a company producing the pre-qualified product at the lower price, generating a saving of $30,000.UNFPA is in process of tendering and developing long term agreements with the newly qualified manufacturers. It is unlikely that these long term agreements will have delivered the anticipated cost savings by project completion. However, if procurement develops as expected, including for injectables, by the end 2014, the project should have generated procurement savings worth at least the value of its inputs of around £3 million. By 2015/16, when the evaluation is expected, it will be possible to assess the total cost savings related to those products. Recommendations: Concept Foundation should provide more detail on price developments of the PQ and ERP approved RH medicines against their branded or generic competitors in its final Project Completion Report (based on RH Interchange and its own intelligence). It should also provide analysis towards the outcome measure of savings achieved, together with UNFPA. Concept Foundation and partner UNFPA should strengthen VfM indicators and report on VfM in their progress and project completion reports. 5.2 Commercial Improvement and Value for Money The DFID accountable grant contract to RHSC/CF through PATH does not include VfM reporting requirements, and the progress and financial reports do not provide enough details allowing the Review Team to draw any conclusions whether VfM is being maximised. As noted above, Concept and UNFPA have competitive rates for short term assistance and salaries. 15 The country studies were awarded after an open international bidding procedure. The Review Team had insight in the tender analysis, and the contract was awarded in compliance of international good procurement practices. The contract with UNFPA was unchanged; the financial reporting by UNFPA is not detailed enough to enable the Review Team to draw conclusions whether VfM is being achieved. However, UNFPA’s consultancy rates for technical experts are competitive, and in line with UNFPA’s overarching policy for consultant costs. The issue of “pass-through” funds (DFID-PATH-CF-UNFPA-WHO) has been addressed, and a formal contract will be in place for the next ERP between Concept and WHO for the final process under the current grant (thus avoiding additional overhead costs). 5.3 Role of project partners Concept Foundation contracted some deliverables of outputs 2 and 3 to partner UNFPA (which subcontracted the ERP assessments to WHO), and it contracted the country baseline studies to the Royal Tropical Institute in Amsterdam (KIT). The Review Team requested but did not receive Concept’s analysis of their subcontractor KIT and partner UNFPA’s performance. CF has probably underestimated the requirements of its role in managing and monitoring the deliverables by its subcontractor and partners. UNFPA was contracted to prepare, and organize the New York procurers’ meetings and the ERP processes in output 2. The 2013 New York meeting helped to get a better understanding of the challenges that procurers face at the country level. The 2013 report was much clearer than the 2012 report. The ERP process is well organised and can be rated as successful, as it increased the number of quality-assured RH medicines accessible to procurers by 12 in 2012 and 3 more by 31 July 2013. Some 2012 communication glitches have been prevented thanks to a new standard operating procedure on communicating ERP results. WHO PQP was contracted by UNFPA to assess the ERP dossiers after initial screening by UNFPA. The Review Team found that WHO did this job well. The approach by UNFPA PSB for country workshops (organising them with the local UNFPA country office) would be enhanced by greater involvement of UNFPA CSB and RHSC members at country level. 5.4 Does the project still represent Value for Money? Spending less money for the same or greater value (in terms of quality assured RH medicines procured) is a key hypothesis in the DFID business case. It is a positive sign that prices of the prequalified generic oral pills have dropped from $0.29 in 2011 to $0.225 in 2013 (23% reduction). Over the same period emergency contraceptive prices dropped from $0.31 to $0.26. These price reductions are beginning to deliver procurement savings. UNFPA has completed its first procurement of 0.45m oral pill cycles with a company producing the pre-qualified product at the lower price, generating a saving of $30,000. UNFPA is in process of tendering and developing long term agreements with the newly qualified manufacturers, so it is not yet possible to assess the total cost savings related to those products. Attribution of these price changes to the QuRHM project is problematic, given limited information about dynamics of the global market and the portion of it which is influenced by QuRHM, but it is likely that the project contributed. The main savings in the Business Case are expected from reduced prices of quality approved generic injectables: unfortunately, these products are not yet available (only one product received an ERP category 3 rating, and so can be procured in exceptional circumstances). However, at least two 16 injectables are expected to receive a favourable rating by project end. The three month injection (with a baseline price of 0.86) has increased to over $1 due to quality supply shortages. One generic product was assessed as ERP category 3, for procurement only where no alternative. In case this is necessary, UNFPA has negotiated a long term agreement for the product at a price well below the original baseline, showing that price reductions are highly achievable. Actual price savings should be generated and calculable by the evaluation in 2015/16. Although the business case mentions Value for Money (VfM), the DFID contract with PATH/Concept Foundation does not include specific VfM reporting requirements, and the new VfM indicators need strengthening. 5.5 If not, what action will you take? The Review Team recommends that DFID request Concept Foundation and partners to report on VfM in the Project Completion Report. The Review Team will then independently assess these reports. Overall VfM in terms of potential cost savings will be assessed as part of the evaluation. 6. Conditionality 6.1 Update on specific conditions Not applicable 7. Conclusions and actions In year 2 of QuRHM, significant progress is being made in output area 1, with 10 generic RH products quality assured by WHO, where all received technical assistance from the project. Prices for quality assured oral contraceptive products have decreased, and attribution to Concept’s activities is plausible although needs more analysis. Progress in the other outputs (mostly contracted to UNFPA, WHO and consultants) is variable, and would benefit from more attention by Concept management, and greater involvement of the RHSC secretariat and community. Achieving prequalification or ERP approval is no guarantee that the quality assured products will be quickly available and procured by countries, and available to consumers and at reduced prices. More attention needs to be given to the downward pipeline problems in countries and the economics of the hormonal contraceptives market. Evidence for a business case for generic RH companies to invest in PQ is weak at present, and will need reassuring data on future needs, markets and orders. The project’s activities are delayed at country level, but continue to be important with respect to building commitment in the four selected countries to quality standards. Concept Foundation’s management is still overburdened, probably by over-reliance on the Project Manager and not enough delegation to other CF staff. CF should also strengthen the monitoring of outsourced or partner’s programmes. Concept Foundation and RHSC members could collaborate more intensely at country level to provide linkages with existing programmes to strengthen RH commodity security, strengthen National Medicines Regulatory Authorities, procurement agencies etc. The potential benefits of the project make taking the risk worthwhile. The Review Team has also recommended that the risk be reduced from high to medium. The pipeline of generic products contains 17 some urgently awaited RH medicines (especially injectables, misoprostol, and oxytocin). A more focused and risk-based approach towards these products would be useful. The Technical Advisory Committee meeting in Delhi took some good initiatives for RHSC to plan for the post-QuRHM period. The Review Team is hopeful that another donor will allow CF to ensure that the much anticipated products in the pipeline reach prequalification status, and that the combined efforts of RHSC members at country level and a new Quality Working Group can continue the good work that CF has done in the QuRHM project. DFID transparency policy requires that Annual Review reports should be made public. The 2012 Annual Review is now available at a UK government website3. On 18 April 2013 BMGF asked RHSC on behalf of DFID and with permission of UNFPA and WHO to make the full 2012 Annual Review report available on the RHSC website. RHSC will now also publish the 2012 annual review report on its website. This Year 2 review has scored the project B (ABCA). Performance has improved in most areas, and the review team judge that the project can score an A. Update on 2012/13 project improvement action plan and recommended 2013/14 actions 1. Ensuring realistic targets are achieved The design of QuRHM was highly ambitious and a number of anticipated results were not delivered in Y1. In response to the AR recommendations, the logframe has been amended to ensure milestones are SMARTer and at the appropriate level of ambition. Concept, with inputs from DFID, are recommended to further SMARTen the indicators, by end 2013/early 2014. VfM indicators, in addition to existing purpose level indicators on commodity costs, will be added to demonstrate efficiencies in procurement behaviour and acceleration of products through PQP/ERP process. The review notes that indicators were added and go some way towards demonstrating improved VfM. Again the reviewers recommend further SMARTening. Concept will work with a lead economist in PD (James Droop) and WHO to improve these further (end 2013/early 2014). Concept Foundation in Y2 will prioritise TA support to suppliers with greatest potential to achieve WHO PQP/ERP, those who supply markets in developing countries, or commodities in neglected categories (namely injectables). This should improve programme focus and delivery. This has been done – the reviewers note that the number of companies engaged with Concept has dropped from 32 to 23. Concept is recommended to continue to prioritise products supported in relation t o market needs. 2. Improving programme management Delays in programme due to contracting and lack of capacity in Y1 have been addressed through recruitment of additional CF management staff and an organisational restructure. Additional staff have been recruited, and financial management and reporting for example have improved. Concept senior management are recommended to continue to build up staff capacity and delegate tasks, and include actions taken on this in the project completion report. Concept Foundation has initiated more regular and planned meetings between implementing partners, to ensure better collaboration. Communications between partners have improved, and Concept will continue to manage partner outputs more effectively, reporting progress to DFID in the project completion report. A joint critical pathway tool has been developed between Concept and WHO to better track products through PQP/ERP process and help manage different roles and responsibilities of respective organisations. The tool is improving Concept’s management of each product pathway though the reviewers recommend that WHO also fully utilise it. Concept will review this with WHO in early 2014, with the intention of agreeing a shared tool. Risk management will be addressed at each programme oversight committee meeting. Risks have been downgraded from high to medium. Major bottlenecks and risks have been addressed in Year 2: Gates is funding bio-equivalence studies for injectables, and WHO has agreed a rolling ERP for injectables. 3 http://devtracker.dfid.gov.uk/projects/GB-1-202825/documents/ 18 Interim review scheduled for summer 2013 to assess progress against recommendations and Y2 milestones, full in-depth review by March 2014 looking at progress in 4 study countries, and impact evaluation by 2015 looking at impact on broader market dynamics. This schedule is on track. The reviewers recommend that that the evaluation is scheduled later in 2015/16 (paid for by the Gates Foundation). 3. Improving value for money Economy – issue regarding pass through funds currently being resolved to deliver savings on overheads. This has been resolved, with the final ERP contracted by Concept to WHO directly. Concept and UNFPA will provide more information on input costs and other VfM indicators in the project completion report. Efficiency and Effectiveness – bottlenecks being identified and documented as part of the critical pathway tool and mitigating actions identified, improved partnership/links with other support (RHSC, BMGF work on DMPA). This is being achieved through the critical pathway tool, although the indicators need improvement (see above recommendation). Overall VfM – this needs to be better tracked through VfM indicators and market analysis. The reviewers found that prices had indeed fallen for both combined oral, and emergency contraceptive pills, by Year 2, which is a positive sign. The 23% reduction in price of generic combined oral contraceptives has generated savings through new UNFPA procurement of 1m cycles. Concept should include more contextual market and pricing data, and manufacturers’ statements, to show the link between price drops and project inputs, in its project completion report. 8. Review Process The TORs for this Annual Review were finalised by DFID on 4 July 2013. The review was conducted by Wilbert Bannenberg of HERA between 9 July and 28 October 2013. The process took longer than expected due to some delay in receipt of the revised logframe. Key staff of Concept Foundation, RHSC, PATH, and DFID were interviewed, and RHSC members and manufacturers were interviewed during the RHSC meeting in Delhi. A Google search was conducted on the keyword “QuRHM”. DFID, PATH and Concept Foundation provided project documents. The Review Team (HERA) signed a confidentiality or non-disclosure agreement with Concept for the review, including the list of companies they are or have been supporting. This enabled the Review Team to have a better insight into the correlation between CF’s activities and progress under milestones 1 & 2 under Output 1. The Review Team was able to verify evidence of CF’s activities as to which generic manufacturers they supported, how and when specific support steps were taken, and when the manufacturer submitted dossiers to PQ or ERP. Internal QA (Marianne Schürmann, HERA) and external QA (Cheri Grace, independent consultant) was applied to the report. Draft conclusions and a summary report were presented to the POC on 7 November 2013 in London. An updated report was submitted on 15 November to DFID. This review is largely based on HERA’s report and scoring assessments, with some amendments and additions made by DFID. There will be a full joint annual review on WHO/PQP and QuRHM and a separate Project Completion Review of QuRHM in April-May 2014. An Evaluation is planned (likely to be re-scheduled from 2014 to 2015/16). i http://icsc.un.org/secretariat/sad.asp and http://www.un.org/Depts/OHRM/salaries_allowances/salary.htm ii iii Manual for Salary Surveys in Duty Stations other than Headquarters and similar duty stations, UN 2013 Personnel Policies and Procedures Manual, UNFPA 2010 19