MONITORING AND EVALUATION
TRAINER'S NOTES
PURPOSE AND CONTENT
This module gives participants some general guidelines on how to monitor
community drug use interventions, and discusses in more detail how to evaluate.
Evaluation of rational drug use interventions directed towards consumers is rarely
done systematically, and as a result we have little evidence to prove success and/or
failure of various intervention strategies. The module gives several options for
evaluation designs, and argues for the need to combine process and effect
measurements. It gives an overview of evaluation methods. In the activity
participants formulate questions and select methods for the evaluation of a training
programme for drug sellers operating in Ugandan communities.
OBJECTIVES
Upon completion of the module participants will be able to:
1.
2.
3.
4.
Understand the different functions of monitoring and evaluation.
Define what should be monitored and how this should be done.
Formulate appropriate evaluation questions and identify useful methods.
Make a plan for monitoring and evaluation components of an intervention.
PREPARATION
1. Participants should read the Session Notes.
2. The trainer should find out at least a day in advance of the session which of the
participants have been involved in an evaluation of an intervention which aims at
enhancing rational drug use by consumers. If there is more than one participant,
select the case-study which is most interesting from a training point of view
(touches on a variety of evaluation design issues). This participant should be
invited to do a 20 minutes presentation on the objectives and design of the
evaluation. The presented case study need not be an example of an excellent
study. It is in fact better to have a study with flaws as it is easier then to make
comments on the design (for example, the evaluation lacks a control group).
Explain to the participant though, that the aim is not to criticise his or her design,
but rather to learn from the actual practice of doing an evaluation and reflect on
how it could have been done better, considering also available resources.
3. The trainer should read the Session Notes and discuss the presentation with the
participant before the session and review the Trainers Notes to see where the
case-study can be used as an example or point of reference.
Monitoring and evaluation
Trainer's Notes
4. The trainer should check whether there are handouts for the core library and that
they are available for distribution after the session. At the end of the session, the
trainer should briefly explain why the readings are core.
CORE LIBRARY
Marsh VM et al. (1999) Changing home treatment of childhood fevers by training shop
keepers in rural Kenya. Tropical Medicine and International Health, 4(5): 383-389.
ADDITIONAL READING
Feuerstein, M (1986) Partners in evaluation: evaluating development and community programmes
with participants. London, Macmillan.
Fresle DA, Wolfheim C. (1997) Public education in rational drug use: a global survey. Geneva,
World Health Organization, (WHO/DAP/97.5).
Loevinsohn BP (1990) Health education interventions in developing countries: a
methodological review of published articles. International Journal of Epidemiology, 19(4):788794.
Øvretveit J. (1998) Evaluating health interventions. Trowbridge, UK, Open University Press.
Piotrow P et al. (1997) Health communication. Lessons from family planning and reproductive
health. Westport, CT, Praeger Publishers.
Web sites:
International Conference on Improving Rational Drug Use - Poster presentations
http://www.who.int/medicines/organization/par/icium/prgramme.shtml
INRUD/WHO Promoting Rational Drug Use Course. Decision-Making for Rational Drug
Use Interventions module.
http://dcc2.bumc.bu.edu/prdu/HTML_DOCS_TOC.htm
2
Monitoring and evaluation
Trainer's Notes
ORGANISATION OF THE SESSION
The session is planned in four parts:
Structure and timing of the session
A. Case presentation - 20 minutes
and questions for clarification - 10 minutes
B. Interactive lecture on monitoring and evaluation - 45 minutes
TEA BREAK - 15 minutes
C. Activity 1 - 75 minutes
D. Feedback and discussion of expert advice from one group, comments from others - 20
minutes
VISUAL AIDS
1. Title slide
2. Objectives session on monitoring and evaluation
3. Functions of monitoring
4. Functions of evaluation
5. What to monitor
6. How to monitor
7. What to evaluate?
8. Preparation
9. Planning
10. Implementation
11. Effect
12. Four evaluation designs
13. Four evaluation designs fig. no. 1
14. Four evaluation designs fig. no. 2
15. Define key outcome measures
16. Problems in proving effects
17. Methods
18. Examples of key outcome measures
19. Summary guidelines
20. An evaluation plan
21. What can be achieved?
22. Activity 1
3
Monitoring and evaluation
Trainer's Notes
Trainer first briefly introduces the session’s objectives (OH 2)
A. CASE PRESENTATION (30 MINUTES)
The trainer clarifies the function of the case-presentation: to relate design issues to a
concrete example conducted by one of the participants. It is a learning exercise. The
aim is not to criticise the design.
The selected participant gives a detailed presentation on the objectives, methods and
results of an evaluation study (20 minutes).
Participants can ask questions for clarification about the presentation (10 minutes).
B. INTERACTIVE LECTURE ON MONITORING AND EVALUATION
(45 MINUTES)
The interactive lecture deals with the following points:
1. Functions of monitoring and evaluation
The trainer gives a brief introduction on monitoring and evaluation dealing with: the
differences in functions of monitoring and evaluation (OH 3 and 4).
2. What and how to monitor
The lecture continues with a brief introduction on what and how to monitor (OH 5
and 6). Refer the participants to more details in the Session Notes and explain that in
the lecture and activity we focus on evaluation.
3. What to evaluate by intervention phase
Present the various stages of an intervention:
 Preparation
 Planning
 Implementation
 Assessment of effect
Ask participants to give suggestions for key evaluation questions for each phase.
Refer also to the questions in the case-study presented in the beginning of the
session. Use OH 7-11.
4. Evaluation designs
Discuss possible evaluation designs. Ask participants to put forward designs – refer
to the one presented in the case-study. What else could have been used as design?
Present OH 12-14
4
Monitoring and evaluation
Trainer's Notes
Stress the need to measure process variables and effect. Without insight in the
process of the evaluation one cannot explain why there was an effect or not.
5. Outcome measures
Discuss key outcome measures and stress that these are based on the communication
objectives of an intervention. This means that communication objectives have to be
very clear! Give concrete examples of outcome measures. Again refer if relevant to
the case-presentation. Use OH 15 and 16.
6. Methods
Ask participants what methods they think are useful for evaluation.
Use OH 17 to review them.
7. Problems in proving effects
Finally, discuss problems in proving effect. Refer to the results of the casepresentation if relevant: use OH 18.
8. Summing up
End the session with the summary guidelines for evaluation (OH 19) and the
components of an evaluation plan (OH 20); and a reflection on the time-dimension in
effects (OH 21).
BREAK (15 MINUTES)
C. ACTIVITY 1 (75 MINUTES)
Introduce the activity using (OH 22) and refer participants to the description in the
module, see below. This activity tends to be valued by participants as it puts them in
the role of ‘experts’. Address them as such when introducing the exercise. These give
weight to the activity.
5
Monitoring and evaluation
Trainer's Notes
ACTIVITY 1
Monitoring and evaluation
This activity is done in ‘table’ groups. You have 75 minutes to brainstorm on the
questions and to formulate your advice. The responses should be written down and
submitted to the facilitator (maximum of three pages). One group then presents its
plan and the other groups comment on and add to this.
Your group has been invited by the funding agency DANIDA to give expert advice
on how to evaluate a shopkeeper intervention in Uganda to be carried out by the
Uganda Red Cross (URC). The URC has recently received a substantial grant from
DANIDA to conduct the intervention, but has failed to include an evaluation
component in its project proposal. DANIDA wants you to formulate the evaluation
component. The evaluation is to be carried out by a small team of outside experts
and national communications specialists in collaboration with the URC, which is
implementing the intervention. A detailed description of the rationale for the
intervention, its communication objectives, and approach is given below.
Specifically your team is asked to formulate the following:
1. Evaluation questions (process and effect).
2. A limited set of key outcome measures, including their purpose, how they should
be calculated and what methods should be used to collect data for them.
3. Advice on whether to use a quasi-experimental design, or whether to limit the
evaluation to a time-series design or pre-post design.
4. A limited set of data-collection methods to be used.
Description of the intervention that is to be evaluated
A previous investigation of drug use patterns in Uganda revealed that the majority
(72%) of drug treatment is through self-medication, in that it is undertaken without
the advice of a trained health worker. The most commonly used drugs, in order of
frequency, are chloroquine, aspirin, paracetamol, septrin (cotrimoxazole) and
mebendazole. Only about 25% of drugs are obtained from the Government and NGO
units receiving essential drug kits through the Uganda Essential Drugs Management
Programme. This is true not only of common analgesics like aspirin, but also of
antimalarials and antibiotics, which are obtained mostly from drug shops and
general provision stores.
Drug sellers in small shops and storefront clinics were found to be the main
providers of drugs for the common conditions of fever, cough, diarrhoea and worms,
accounting for around 60% of treatments (Odoi et al., 1996). In addition to drugs,
6
Monitoring and evaluation
Trainer's Notes
they also provide advice. Medicines sold in the shops are not labelled. While some of
these ‘informal’ providers have considerable medical experience and may even be
supplementing their meagre incomes in government health units, others have no
formal training (Odoi et al., 1996).
In two communities in Tororo District, the URC aims to implement a pilot
shopkeeper intervention focusing on the appropriate treatment of fever, the most
common condition for which drugs are bought at community drug shops. Malaria is
endemic in Tororo. Most fevers are considered to be malaria by patients and selfmedicated with antimalarials. The malaria control programme in Uganda encourages
the use of chloroquine in adequate dosages as first line treatment.
All drug sellers operating in the intervention communities will be invited to attend a
six-day training workshop; they will receive two supportive supervision visits from
the trainers; and they will be invited to attend a refresher session.
The training approach is to be participatory and problem oriented, with an emphasis
on experience-based learning.
Key messages will be:
 Chloroquine is the first choice of drug for suspected malaria.
 Correct dosages for chloroquine for seven different age groups; these correct
dosages will be listed on a chart and given to all drug sellers participating in the
workshop.
 It is important to label drugs; for chloroquine sales the drug sellers are given
stamps which depict the appropriate dosage of the drug
 The drug sellers are trained to explain to their customers what the stamped
chloroquine labels mean.
 The drug sellers are trained to ask clients to repeat instruction on dosage of the
drugs.
If successful, the URC aims to encourage the Ministry of Health's Malaria Control
Programme to implement a nationwide training programme for drug sellers, to
increase the rational use of antimalarials by consumers. Malaria treatment guidelines
change over time, due to emerging resistance. The URC believes that the most
efficient way of informing the general public of changing guidelines is through the
drug sellers from whom they usually buy medicines for malaria treatment.
D. FEEDBACK AND DISUSSION OF EXPERT ADVICE (20 MINUTES)
Identify the group which has developed the most elaborate advice and has finished
the exercise in the given timeframe. Ask this group to present its results (10 minutes).
Others can add their views and give alternative recommendations.
Introduce and hand out any core library materials.
7