Monitoring and
Evaluation Toolkit
A resource to support the Portfolio of Canadian Standards for Youth Substance Abuse Prevention
Author: Marvin Krank, PhD, University of British Columbia
O
n November 22, 2010, the Canadian Centre on
Substance Abuse (CCSA) released the Portfolio
of Canadian Standards for Youth Substance Abuse
Prevention to support prevention workers in strengthening
the quality of youth-focused substance abuse prevention
initiatives in Canada. One of very few sets of standards for
this area of programming in the world, the Portfolio of
Canadian Standards includes the School-based Standards for
Youth Substance Abuse Prevention and the Community-based
Standards for Youth Substance Abuse Prevention.
In developing the Standards, CCSA consulted extensively
with substance abuse prevention workers (i.e., practitioners,
programmers and policy officials involved in youth substance
abuse prevention), who requested additional knowledge and
tools regarding evaluation activities to aid implementation
of the Standards. In response, CCSA created the Monitoring
and Evaluation Toolkit.
What you will learn
The Monitoring and Evaluation Toolkit provides a brief
introduction to monitoring and evaluation. Key concepts are
defined and exercises are provided to help select the proper
type of evaluation. Building on existing resources, links to
external information sources are also included to provide
further instruction in conducting evaluations.
Why monitor and evaluate?
The prevention of substance abuse is a long-term goal;
monitoring and evaluation allows you to assess whether you
have met your prevention goal, as well as the components of
your initiative that are contributing to behaviour change and
those that could be improved. Additionally, substance abuse
prevention workers are increasingly asked to improve on
what they are doing—and to prove that what they’ve done
has been effective. Monitoring and evaluation can be used to
accomplish both.
Who is the Toolkit for?
The Toolkit is a resource for substance abuse prevention
workers. It is equally valuable to those new to monitoring
and evaluation as well as to prevention teams with mixed
experience in evaluation.
How to use the Toolkit
The Toolkit is written as a workbook; it can be completed
from start to finish or by skipping to the sections most relevant
to you using the links contained within the Toolkit. Inclusion
of a resource in the Toolkit does not imply endorsement
or authorization by CCSA, and any questions regarding
a particular resource should be directed to the author or
publisher.
© Canadian Centre on Substance Abuse 2012
Monitoring and Evaluation Toolkit: A resource to support the Portfolio of Canadian Standards for Youth Substance Abuse Prevention
SECTION 1: What is monitoring and evaluation?
Monitoring provides information as you go—it is the ongoing
collection of information about the context, content, outputs
and outcomes of your initiative. You can use monitoring
information to examine whether your initiative is doing
what you intended, guide you in improving it, and help you
evaluate its effectiveness. For example, if you or your funders
are interested in determining how many students are taking
part in a prevention initiative, you can routinely record the
number of students attending your sessions and at the end
of the day you can calculate how many students attended.
Although monitoring and evaluation can be much more
complex, this simple act is an effective way of monitoring one
potential output of your initiative.
Evaluation is the periodic assessment of how well the
initiative is working and where the gaps are in current
programming. Gaps identified through evaluation tell you
where changes to your initiative or new approaches are
needed. This information forms the raw materials out of
which promising initiatives can be improved and new, more
effective initiatives can be developed.
Evaluation uses data from monitoring activities that is
collected during and after the initiative. There are two
main types of evaluation: formative evaluations, which are
conducted during the planning and implementation phases
of an initiative to help guide it; and summative evaluations,
which are used after the initiative or a phase of the initiative is
complete to evaluate its outcomes.
1
Assess
4
2
Evaluate
FORMATIVE
SUMMATIVE
3
2
© Canadian Centre on Substance Abuse 2012
Coordinate/
Impliment
Organize/
Prepare
Monitoring and Evaluation Toolkit: A resource to support the Portfolio of Canadian Standards for Youth Substance Abuse Prevention
Formative evaluations: Improve delivery of your initiative
Formative evaluations are used to improve what you do by enhancing and adjusting initiative delivery. They do not prove an
initiative’s effectiveness; rather, as the name implies, they serve to guide and direct the form of the initiative. Formative evaluations
assess the need for the initiative and the resources that are available, help with the initial planning of the initiative, and provide
insight into implementation issues.
Types of formative evaluations:
• Needs assessment: A needs assessment is an evaluation of the context of a potential or current program, and helps
determine the scope of the problem being addressed. They are typically conducted before an initiative is implemented
but can also be done during or after to inform revisions to the initiative.
•
Process evaluation: Process evaluations assess how an initiative was delivered and whether it reached the target
audience. They typically occur during implementation of an initiative and are used to modify or make corrections to
the delivery plan. Process evaluations may also be conducted at the conclusion of the initiative to provide context to the
summative evaluation.
Evaluation type
Questions addressed
Needs assessment
Who should the initiative target (target audience)?
What should the initiative target (risk and protective factors)?
What are the potential supports and barriers to implementation?
What resources are available to support the initiative?
Process evaluation
What are the costs of the initiative?
Did the initiative reach its target audience?
How well was the initiative was received?
Was the program delivered as planned?
Notes:
Please note: You may type (save and access) an unlimited amount of text characters in this box, however, only the first
650 characters (including spaces) will be printable.
© Canadian Centre on Substance Abuse 2012
3
Monitoring and Evaluation Toolkit: A resource to support the Portfolio of Canadian Standards for Youth Substance Abuse Prevention
Summative evaluations: Demonstrate your initiative’s effectiveness
Summative evaluations provide a formal appraisal of results following the conclusion of the initiative or one of its phases, and
assess whether the initiative met its goals and if there was a cost-benefit of the initiative. Summative evaluations require expertise
in research design and knowledge of statistical analysis, so it may be useful to work with an evaluation expert when conducting
this type of evaluation. If you hire an evaluation expert, they will rely on you to plan for the evaluation.
Types of summative evaluations:
• Outcome evaluation: Outcome evaluations assess whether the initiative has had its intended effect. In substance abuse
prevention, the outcome evaluation should measure whether a prevention initiative prevents/reduces substance use or
harms arising from that use.
•
Cost-effectiveness and cost-benefit analysis: Cost-effectiveness and cost-benefit analysis address the efficiency of
the initiative by comparing the relative costs of achieving the desired outcomes. Economic evaluation measures the
comparative value of the intervention.
Evaluation type
Questions addressed
Outcome evaluations
Did the initiative prevent youth substance abuse?
Did the initiative meet its short-, medium- and long-term goals?
What were the unintended impacts of the initiative (both positive and negative)?
Cost-effectiveness and
cost-benefit analysis
How much does it cost to produce a positive outcome?
Is the initiative more cost-effective than alternative approaches?
Notes:
Please note: You may type (save and access) an unlimited amount of text characters in this box, however, only the first
650 characters (including spaces) will be printable.
4
© Canadian Centre on Substance Abuse 2012
Monitoring and Evaluation Toolkit: A resource to support the Portfolio of Canadian Standards for Youth Substance Abuse Prevention
SECTION 2: Planning your evaluation
Carefully planning monitoring and evaluation activities can save money and time; the more planning completed upfront, the
easier it will be to address questions posed by funders of the initiative and other stakeholders.
This section will guide you through the initial steps of planning for evaluation. If your prevention initiative does not yet have
a clear documented evaluation plan, it is not too late to start. By planning—even at this point—you will still be able to gather
information that will be helpful in your initiative’s evaluation.
The first tasks in conducting an evaluation are to:
• identify the purpose of your evaluation; and
• identify your stakeholders.
Identify the purpose of your evaluation
Take a moment to reflect on why you are considering an evaluation. Is it to assess the need for an initiative or to justify one?
Assess whether the initiative is being implemented as planned? Assess the impact of the initiative? Examine whether it is costeffective compared to similar initiatives?
Exercise 1
Think about why you are considering an evaluation and the questions you are hoping to answer. Include as many
reasons as you can.
Please note: You may type (save and access) an unlimited amount of text characters in this box, however, only the first
650 characters (including spaces) will be printable.
© Canadian Centre on Substance Abuse 2012
5
Monitoring and Evaluation Toolkit: A resource to support the Portfolio of Canadian Standards for Youth Substance Abuse Prevention
Identify your stakeholders
Stakeholders are individuals, groups or organizations that would have an interest in your initiative and will want to see and use
the results of your evaluation report. Examples include:
• parents;
• school teachers and administrators;
• funding agencies;
• community partners;
• youth/families that will be targeted by your initiative; and
• you—the people who plan and deliver initiatives.
To meet stakeholder needs, you need to know what information they need and for what purpose. Is it to ensure initiative delivery
goes smoothly (formative evaluation)? Is it to demonstrate client satisfaction and improve uptake in other potential clients
(process evaluation)? Or is it to demonstrate a reduction in youth substance use (outcome evaluation) or cost-effective outcomes
for a funding agency (cost-effectiveness evaluation)? It may be helpful to prioritize stakeholders and identify how they may want
to be involved, either by being engaged through the evaluation process or by just receiving the evaluation results.
Exercise 2
Complete the template below to identify your stakeholders and their interests.
Stakeholder
Stakeholder interests
Funders
Administrators
(especially decision makers)
Other program providers
Politicians
Community members
Staff in an institution or agency where
the initiative will be delivered
Potential staff who will work
with the program
Unions
Interest groups
Potential clients or recipients
of the initiative
Other (specify)
6
© Canadian Centre on Substance Abuse 2012
Receive evaluation Type of information they may
results?
want to receive
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Monitoring and Evaluation Toolkit: A resource to support the Portfolio of Canadian Standards for Youth Substance Abuse Prevention
Exercise 3
Review the above list in relationship to the purpose of the evaluation. Who on this list should receive the results?
Write a brief statement about who should receive this information.
Please note: You may type (save and access) an unlimited amount of text characters in this box, however, only the first
650 characters (including spaces) will be printable.
Exercise 4
Identify any questions of interest that your stakeholders may want your evaluation to address.
Please note: You may type (save and access) an unlimited amount of text characters in this box, however, only the first
650 characters (including spaces) will be printable.
© Canadian Centre on Substance Abuse 2012
7
Monitoring and Evaluation Toolkit: A resource to support the Portfolio of Canadian Standards for Youth Substance Abuse Prevention
Identify the type of evaluation
Exercise 5
Considering the questions of interest to you and your stakeholders, what are the primary questions you would like
answered through your evaluation?
Please note: You may type (save and access) an unlimited amount of text characters in this box, however, only the first
650 characters (including spaces) will be printable.
What type of evaluation(s) do your questions correspond to? See the descriptions of each type of evaluation and questions of
interest above.
The questions of interest to you and your stakeholders may span more than one type of evaluation. If this is the case, where you
are in the lifecycle of an initiative may help you prioritize the type of evaluation you start with.
•
In the planning stages, you and your stakeholders will be interested in a needs assessment.
•
During implementation, you and your stakeholders will want to monitor what you are doing and what happens to
improve what you do (process evaluations).
•
At important milestones, you and your stakeholders will want to evaluate against the goals of your initiative and
determine the impact of the initiative (summative evaluations).
Exercise 6
Identify the type of evaluation that is most appropriate for you to begin with and describe why.
Please note: You may type (save and access) an unlimited amount of text characters in this box, however, only the first
650 characters (including spaces) will be printable.
8
© Canadian Centre on Substance Abuse 2012
Monitoring and Evaluation Toolkit: A resource to support the Portfolio of Canadian Standards for Youth Substance Abuse Prevention
If you identified a needs assessment
Go to the Canadian Standards Needs Assessment Toolkit (available mid-2012).
If you identified a process evaluation
The following resources will provide further information and tools to get you started:
The Health Communication Unit. (2007). Evaluating health promotion programs. Toronto: University of Toronto, Centre of Health Promotion. Retrieved from: http://www.thcu.ca/resource_db/pubs/107465116.pdf
Work Group for Community Health and Development. (2011). A framework for program evaluation: A Gateway to tools. Lawrence, KS: University of
Kansas. Retrieved from: http://ctb.ku.edu/en/tab...tents/section_1338.aspx
If you identified an outcome evaluation
Tthe following resources will provide further information and tools to get you started:
Aromaa, S., Guckenburg, S., Rosenbloom, D., & Edwards, E. (2005). How do we know we are making a difference? A community alcohol, tobacco, and
drug indicators handbook. Boston: Join Together. Retrieved from: http://www.promoteprevent.org/resources/how-do-we-know-we-are-makingdifference-community-alcohol-tobacco-and-drug-indicators-hand
The Health Communication Unit. (2007). Evaluating health promotion programs. Toronto: University of Toronto, Centre of Health Promotion. Retrieved from: www.thcu.ca/resource_db/pubs/107465116.pdf
Work Group for Community Health and Development. (2011). Developing an evaluation plan. Lawrence, KS: University of Kansas. Retrieved from: http://ctb.ku.edu/en/tab...b_section_main_1352.htm
If you identified cost-benefit evaluation
The following resource will provide further information and tools to get you started:
The Health Communication Unit. (2007). Evaluating health promotion programs. Toronto: University of Toronto, Centre of Health Promotion. Retrieved from: www.thcu.ca/resource_db/pubs/107465116.pdf
ISBN 978-1-927467-03-9
Canadian Centre on Substance Abuse
75 Albert Street, Suite 500
Ottawa, ON K1P 5E7
Tel.: 613-235-4048 | Fax: 613-235-8101
www.ccsa.ca
© Canadian Centre on Substance Abuse 2012
9