CRITIQUING A TERMS OF REFERENCE: SAMPLE TORS -- EVALUATION OF THE MULTIPLE INDICATOR CLUSTER SURVEYS (MICS) I. INTRODUCTION A. Background At the World Summit for Children (WSC) on September 30, 1990, heads of state of participating countries agreed to a plan of action to achieve a set of goals in child (and women's) health and well being to be reached by the year 2000. They also committed to measuring progress toward those goals. The UN system was given the task of helping countries work toward all these goals and measure that progress. UNICEF was asked to be the lead agency in this process. As a step toward these goals, countries developed National Programmes of Action. Later, a set of goals for the mid-decade was agreed, first by the members of the Organisation of African Unity (OAU), and subsequently by most other countries. All these activities increased the demand for high-quality, timely data, and focussed attention on the availability and quality of data in developing countries. In many countries, routine reporting systems were found to be inadequate and, in all cases, did not collect information on some of the goals. Although all countries carry out censuses, and many also carry out national household surveys, such surveys and censuses are often expensive, cumbersome, and can take considerable time to yield results. Accordingly, UNICEF led a process to try to develop an affordable, fast, and reliable household survey system that would fill gaps in knowledge and update available data. UNICEF worked closely with other UN agencies, primarily the UN Statistical Division, WHO and UNESCO as well as UNFPA, UNDP and the World Bank and other centres of excellence (the London School of Hygiene and Tropical Medicine and the Centres for Disease Control) to develop a model questionnaire to measure standard indicators for the mid-decade goals (MDG). Data were collected using a probability-sampling cluster survey approach, to take advantage of existing national technical capacity. The surveys were designed to monitor progress at national and sub-national levels, for use in advocacy, policy making and programming, as well as to collect standardised information for international reporting. A handbook on MICS, "Monitoring Progress Toward the Goals of the World Summit for Children", was published by UNICEF in January 1995. This handbook lays out all the stages of carrying out a MICS, from deciding whether a MICS is necessary, to communicating the final results and suggesting ways that those results could be used. The handbook was supplemented by workshops at regional and sub-regional levels, an international network of consultants, and an electronic forum where questions could be posed to a group of people who had been involved in the development of MICS. The methodology was designed to be flexible, to take into account differing country conditions, such as variations in the available resources or quality of existing sample frames. The questionnaire was also intended to be adaptable to local needs. It was designed in "modules" (one module for each goal), which could be added to or deleted from the survey as circumstances dictated. If a separate cluster-sample survey was carried out using some or all of the questionnaire modules for the express purpose of collecting data on the MDG, we refer to it as a "stand-alone MICS." Such surveys were carried out in about 60 countries. In another 40 countries, modules were added to existing country surveys (of whatever type), or were used to modify existing questionnaires on MDG topics. This could cover a variety of surveys, from the Demographic and Health Surveys (which already include measures of many indicators in the MICS questionnaire) to household economic surveys (where all modules could be added from the standard MICS questionnaire to form a separate section of the survey). To make the evaluation more manageable, this assessment will be restricted to "stand-alone MICS", and to comparing this technique to other data collection options. UNICEF M&E Training Resource Module 3.1.3 Critiquing a TOR (MICS) 1/5 The MICS have produced much new data, especially from countries that were notably data-poor. These data are already being used (after a preliminary assessment) in UNICEF publications and planning. However, an in-depth objective assessment of the MICS is needed in order to consider fully the potential uses of this data collection technique in the future, both within and outside UNICEF. B. The rationale for the evaluation This evaluation of the MICS is planned because: There is a political commitment to achieve the WSC Goals and to adhere to the Convention on the Rights of the Child. A cost-effective way of monitoring progress will be important to sustain this commitment; Considerable resources have been invested in the implementation of MICS, both by member states and UNICEF; The development of an effective system for monitoring progress toward goals for children and women is a UNICEF Board priority. We need to assess the future role MICS could play in this system; and National statistical offices, other agencies and organisations are considering or already using the MICS methodology, and want to know about its strengths and weaknesses. II. OBJECTIVES The overall objective is to assess whether and how the MICS could be adapted for future (sustainable) monitoring of the situation of children, especially the WSC goals and the implementation of the Convention on the Rights of the Child (CRC). The evaluation will review the experience gained in different countries in conducting the MICS and utilising the information collected. It will identify the lessons learned — with respect to its effectiveness, efficiency, and capacity building — from the actual implementation of the MICS in order to make rational decisions for future use. More specifically, the evaluation has the following objectives: Examine the preparation, training, co-ordination and follow-up of MICS at the global and regional level; Assess all stages of implementation of MICS at the country level; Identify the conditions within UNICEF Country Offices and counterparts that contributed to their success (or lack of success) in carrying out the MICS; Determine whether information produced by the MICS is of internationally acceptable quality, and the extent to which the data have been used; Document how, and how much, MICS was adapted to local conditions; Assess whether the process of implementing the MICS resulted in national capacity building (and hence to sustainability); Compare the MICS methodology, execution and cost with other data-collection mechanisms; Identify factors that led some countries to decide against doing a MICS; Identify specific areas in which MICS will be useful in the future; and Identify ways to further improve the MICS. III. KEY ISSUES In light of the overall objective of assessing the future possible uses for MICS, the following issues will be addressed: A. How was MICS managed at global and regional level? What is the quality of the MICS tools developed at HQ? How was the MICS introduced in countries, and how much of a sense of ownership was developed in governments? UNICEF M&E Training Resource Module 3.1.3 Critiquing a TOR (MICS) 2/5 What "lead-time" was given in the introduction of MICS, and how did that affect quality and capacity building? How was training carried out? How was information exchanged between and within the different levels of UNICEF, including exchange of advice and instruction as well as questions and results? What was the management process for MICS? For example, how was interaction with countries planned and executed, how did MICS fit in to the country programme evaluation, or how were MICS followed up and commented on? B. How was the MICS implemented? In what ways was the standard MICS instrument adapted to local conditions? And to what degree? Which modules were dropped or added, and why? What strategy was used to implement MICS (e.g. relying mainly on consultants or outside contractors; the amount and breadth of government involvement)? Were the MICS executed according to plan? If not, why not? What problems were encountered in the fieldwork? What were the major cost elements of the MICS? What financial resources did national counterparts, UNICEF, and other agencies and donors invest? C. What were the outputs? Did the survey produce data of good enough quality for the intended use of monitoring MDGs at the international, national and sub-national levels? For example, did the survey encounter statistical problems that affected data quality? Were the results produced in a timely fashion? What problems were faced in analysis and report writing? D. How were the MICS results used? Who (in UNICEF, government, NGO and other donor communities at national and international level) uses the data from the MICS? And who decides not to use MICS data? Are the data used only for MDG reporting, or do uses go beyond that? If so, what? E. Was capacity built? How has survey capacity increased, both in UNICEF (at headquarters, regional and Country Office levels) and in local counterparts? How has understanding of data issues (e.g. the use and abuse of data, the constraints of data) improved in the UNICEF office and among counterparts? Are policy makers and planners better able to make informed decisions for children as a result of MICS? F. Which future monitoring needs could best be served by MICS? How does MICS compare to other possible methods of data collection for monitoring the WSC goals or the CRC? Other methods include administrative data-collection systems, sentinel community surveillance, and other household surveys. Can the MICS be adapted from its MDG focus to be used to monitor other indicators? For what purposes would it most be suited in terms of its appropriateness and cost-effectiveness? IV. EVALUATION METHODOLOGY This evaluation will rely on three complementary methodologies to try to cover both the breadth and depth of the experience: (1) a desktop review of existing documentation; (2) a questionnaire-based survey of all the UNICEF offices regardless of what kind, if any, survey they conducted ; and (3) key informant interviews through field visits to certain countries for more in-depth knowledge. UNICEF M&E Training Resource Module 3.1.3 Critiquing a TOR (MICS) 3/5 A. Desktop review The evaluation will examine all MICS reports and data available in headquarters, including regional or Country Office material, all trip reports (both by HQ and RO staff) to countries in connection with MICS, the database of countries intending to perform MICS, the MICS costing study, and any other documentation available. This will allow the first assessment of data quality (indicating any need for more detailed data analysis), and of cost-effectiveness. B. Questionnaire survey All countries that carried out a MICS will be sent a questionnaire seeking more detail about the MICS process in their country, including questions on capacity building within the UNICEF office, use of data within the office, adaptation and cost analysis of the MICS, timeliness of the results, collaboration with other agencies, and views of UNICEF staff on the usefulness of the exercise. Countries that chose not to do a MICS will be asked what data collection methodologies were used for measuring MDGs, the reasons leading to that decision, and the experience of using those methodologies. The main objective of this questionnaire-based survey is to complement the information available in HQ. No attempt will be made to solicit government counterpart opinions in this survey. C. Key informant interviews Interviews at the HQ, regional and country levels will complement the questionnaires and the desktop review, providing a much deeper understanding of the process in-country, and allowing some issues to be investigated in more detail. Naturally, resources do not permit this kind of analysis to be done in more than a few countries. At the HQ and regional levels All key players — policy makers, technical experts, and programme managers — involved in the development and implementation of the MICS will be interviewed in person or by telephone. They include staff from UNICEF (e.g., Monitoring and Statistics Unit of EPP and selected Programme Sections), UN Statistical Office, CDC, WHO, UNFPA, UNDP and academic institutions. Regional institutions (e.g., ECA, CREDESA) will also be contacted as much as possible. At the country level Interviews with people from UNICEF, NGOs, government (statistical, policy-making and line ministries) and other agencies will be carried out in six selected countries. The countries will be chosen using the following criteria: geography, socio-economic level, efficacy of data collection system, and quality of the MICS. UNICEF Country Offices will need to play a key role in organising and facilitating the field visits, which will be useful for offices’ own assessment of their MICS. V. EXPECTED OUTPUTS This evaluation should produce the following: A first report from the desktop review; An analysis of the questionnaire survey; Individual country reports; One final synthesis report; and A dissemination seminar. UNICEF M&E Training Resource Module 3.1.3 Critiquing a TOR (MICS) 4/5 VI. USE OF THE EVALUATION RESULTS Within UNICEF headquarters, to guide further development of MICS methodology, especially its adaptation to future measurement needs (such as the WSC goals and CRC); In regional and field offices, to help decision makers determine the suitability of the MICS instrument for their monitoring needs; To inform discussion with other agencies on monitoring joint goals (especially in the context of the Common Country Assessment); To provide information to other agencies so they can make informed decisions on the suitability of MICS for their measurement needs; To draw lessons about capacity building in data collection, analysis and use; and To provide input to future discussion of a global monitoring strategy. VII. EVALUATION TEAM MEMBER QUALIFICATIONS A team of four consultants will conduct this evaluation, dividing into two teams for the country visits. The team members should collectively have the following qualifications: Knowledge of survey methodology; Experience in carrying out surveys in developing countries; Good quantitative, interview and analytical skills; Experience in evaluation involving extensive field work; Excellent writing skills; and Appropriate language skills. Also, the team members should not have been involved in the MICS exercise. VIII. COST OF THE EVALUATION The major part of the evaluation will be funded by EPP. Additional resources will be sought from regional and country offices, especially for the field visits. IX. TIMETABLE Event Timing 1. Draft TOR December 1996 – April 1997 2. Selection of Consultants End of May 1997 3. Desktop review (two weeks) Questionnaire design/distribution End of June 1997 End of July 4. Field visits a. Preparation (one week) in NY b. Field work (three weeks) c. Report writing (two weeks) End of July 1997 August 1997 August – September 1997 5. Data analysis and overall report writing (One month) September 1997 6. Review of the report October 1997 7. Final report November 1997 UNICEF M&E Training Resource Module 3.1.3 Critiquing a TOR (MICS) 5/5