Evaluation at NRCan:
Information for Program Managers
Strategic Evaluation Division
Science & Policy Integration
July 2012
.
Purpose
• The purpose of this document is to provide program
managers with an overview of the evaluation function at
NRCan.
• The TBS Policy on Evaluation (April 2009) requires that
all direct spending, including all G&C Programs, be
evaluated every five years
– most program managers will find themselves participating in an
evaluation at some point.
2
.
What is Evaluation?
• Evaluations are the systematic collection and analysis of evidence on
the outcomes of programs to make judgments about
– their relevance;
– performance; and
– alternative ways to deliver them or to achieve the same results.”
• Evaluations must be neutral, and evidence-based.
• An evaluation is not the same as an audit.
Evaluations
Audits
Focus on whether we are doing the right
things, and the extent to which a program is
achieving its expected outcomes, in a costeffective manner.
Internal audit is looking at financial management,
processes, controls and risk.
Make assessments on the relevance &
performance of programs
Identify strengths and weaknesses in the
management control framework
3
.
Why do Evaluations?
• The objective of evaluation is to create a comprehensive and
reliable base of evidence to support:
– policy and program improvement;
– expenditure management;
– Cabinet decision-making; and
– public accountability.
• Evaluations are often required to support TB submissions and
Memoranda to Cabinet.
• They are also a critical source of evidence for Strategic
Reviews, to support resource reallocation (next review will
occur in 2014).
4
.
Background: Evaluation Stakeholders
Evaluation Reports
Policy & Program
Improvement
Expenditure Management/
Cabinet Decision-making
Accountability/
Public Reporting
NRCan DM
Cabinet
Parliament
Sector ADMs
Strategic
Review
Canadian
Public
Program
Managers
TBS
The challenge is to meet the diverse information needs of many stakeholders
for decision-making AND public accountability in a timely fashion.
5
.
The Evaluation Cycle
Evaluation
Planning
Implementing
Change
Approvals/
Posting
(2-3 months)
Evaluation
Assessment
(1-3 months)
Including planning and
approvals/posting, large
evaluations typically take 12-18
months to complete.
Some phases may overlap.
Management Responses
Contracting
(1-2 months)
Field Work/Analysis
(1 month)
Report &
Recommendations
(6-8 months)
(1-4 months)
6
.
Evaluation Planning
• NRCan must evaluate all direct program spending, including all
ongoing grant and contribution programs, every five years.
• NRCan has developed a five-year Evaluation Plan based on
PAA units that is updated annually and approved by the
Evaluation Committee.
• In most cases, the evaluation of an individual program will be
conducted within the scope of a broader evaluation of a PAA
unit.
• The current plan summary appears on the Strategic Evaluation
Internet site at: http://nrcan.gc.ca/evaluation/plans-eng.php.
7
.
Phases of an Evaluation
1.
2.
Evaluation Assessment (1-3 months)
–
research and planning to understand the programs
–
develop the Terms of Reference
–
obtain approval from the Evaluation Committee
Contracting (1-2 months)
–
3.
Consultants are often used to supplement in-house staff.
Their roles will vary by project.
Fieldwork or Data Collection/Analysis (6-8 months)
–
develop a detailed methodology report
–
methodologies: key informant interviews; focus groups;
file/document/literature reviews; surveys; case studies; and
data and economic analysis
–
analyse information collected from these multiple lines of
evidence to develop conclusions
8
.
Phases of An Evaluation, cont’d
4.
5.
Reporting & Development of Recommendations (2-4 months)
–
prepare preliminary findings and discuss with programs
–
draft report
–
address comments and revisions
–
develop recommendations
Management Responses (1 month)
–
obtain ADM-approved management responses and action
plans to the recommendations
6.
Approvals/Posting of report (2-3 months)
–
recommendation by the Evaluation Committee
–
approval by the DM
–
translation, ATIP review, media lines, release on Internet
9
.
Evaluation Questions and Issues
• Evaluations address relevance and performance.
• Relevance issues focus on:
– continued need for program;
– alignment with government priorities; and
– alignment with federal roles and responsibilities.
• Performance issues focus on effectiveness:
– achievement of expected outcomes; and
– demonstration of efficiency and economy.
• Evaluators work with program managers to develop more
detailed evaluation questions relevant to their program.
10
.
Roles and Responsibilities
• Under the TBS Evaluation Policy, Deputy Ministers are
responsible for the evaluation function.
• NRCan’s Departmental Evaluation Committee – an
ADM-level Committee – is chaired by the DM
• NRCan’s Head of Evaluation – who is also the DG of
Planning and Performance Management Reporting –
reports to the Evaluation Committee.
11
.
The Role of the Evaluation Division
• The Strategic Evaluation Division (SED) is responsible for:
– Proposing a five year departmental evaluation plan to the Evaluation
Committee, and updating it annually;
– Managing and conducting evaluation studies, including managing
contracts and deliverables when consultants are used and issuing reports
in a timely manner.
• Additionally, SED will help program managers develop their
performance measurement strategies, with the goal of ensuring that
good data is collected to support future evaluations.
– Evaluation will work with your team to develop objectives, a logic model, a
performance measurement framework and evaluation requirements.
• NRCan's Strategic Evaluation Division is also responsible for reviewing
and providing advice on the accountability and performance provisions
in Cabinet documents (Memoranda to Cabinet (MCs) and TB
Submissions).
12
.
The Role of Program Managers
• Program managers are key to conducting evaluations.
• They are responsible for developing, implementing and monitoring
ongoing performance measurement – the foundation of evaluation.
• Additionally, during an evaluation, they must be actively involved in:
– explaining how their programs work;
– contributing to evaluation planning, including identifying more detailed
evaluation questions;
– providing performance measurement information on resources used,
activities undertaken and results achieved;
– providing detailed documentation (see next slide) and suggestions on
potential interviewees, case studies etc.;
– participating in working groups to review questionnaires, preliminary
findings, draft evaluation reports, etc.;
– developing management responses and action plans for their ADMs and
implementing them after the evaluation.
13
.
Key Documents for An Evaluation
•
In preparation for an evaluation, program managers will
be asked to provide key documents as early as
possible, including:
–
Legislation, Regulations, MCs, TB Submissions
–
RMAFs, RBAFs or Performance Measurement Frameworks
–
references in budgets, SFTs, DPRs, RPPs
–
briefing notes, reports --including annual and project reports,
studies, databases
–
websites and communications products
–
five-years of financial expenditures for the PAA unit including
G&C expenditures, O&M, and salaries
14
.
Questions and Assistance
• If you have any questions on evaluation, or wish evaluation
assistance in developing performance measurement
information or Cabinet documents, please contact:
• The Director of Strategic Evaluation
(613) 996-9649
• Electronic copies of this document, completed evaluation
reports and the Terms of Reference for the Evaluation
Committee are available at:
http://nrcan.gc.ca/evaluation/index-eng.php
15