Evaluation of the Security
and Prosperity Partnership
of North America (SPP)
Pilot Project on Reducing
Emissions from Vehicles
and Engines (PPRE)
July 2011
Environment Canada
i
Audit and Evaluation Branch
Evaluation of the PPRE
Report Clearance Steps
Planning phase completed
Lessons learned completed
Report completed
Report approved by Departmental Evaluation Committee
February 2011
April 2011
April 2011
July 2011
Acronyms used in the Report
CACs
CAP
CARA
CEPA
FCM
FCO
FTE
G&C
GHGs
HDD
MSC
NCDC
NGO
NOx
NRCan
O&M
PAA
PM
PPRE
SPP
U.S. EPA
Criteria air contaminants
Clean Air Partnership
Clean Air Regulatory Agenda
Canadian Environmental Protection Act
Federation of Canadian Municipalities
Fleet Challenge Ontario
Full-time equivalent
Grants and contributions
Greenhouse gases
Heavy-duty diesel
My Sustainable Canada
National Clean Diesel Campaign
Non-governmental organization
Nitrogen oxides
Natural Resources Canada
Operations and maintenance
Program Activity Architecture
Particulate matter
Pilot Project on Reducing Emissions from Vehicles and Engines
Security and Prosperity Partnership of North America
United States Environmental Protection Agency
Acknowledgements
The Evaluation Consultant Team of Stratos, Inc. and Alison Kerry would like to thank
Nicola Scahill and Manjit Kerr-Upal for their ongoing input, provision of
documentation, and review of key deliverables during the evaluation process. They
would also like to thank the interviewees who provided their insights on the pilot
project.
Input from the federal government’s Evaluation Division was provided by Robert
Tkaczyk under the direction of the Environment Canada Evaluation Director, William
Blois.
The evaluation was conducted by Stratos, Inc. in association with Alison Kerry.
Audit and Evaluation Branch
Evaluation of the PPRE
Table of Contents
Executive Summary .................................................................................................... i
1. Introduction ......................................................................................................... 1
2. Background......................................................................................................... 2
2.1 Profile ............................................................................................................ 3
2.1.1 Objectives and Expected Outcomes ....................................................... 3
2.1.2 Stakeholders and Beneficiaries .............................................................. 6
2.2 Governance .................................................................................................. 6
3. Evaluation Design ............................................................................................... 8
3.1 Purpose and scope ....................................................................................... 8
3.2 Evaluation Issues and Questions .................................................................. 8
3.3 Evaluation Approach and Methodology....................................................... 11
3.4 Limitations ................................................................................................... 12
4. Findings by Evaluation Issue ............................................................................ 13
4.1 Rating of Findings ....................................................................................... 13
4.2 Relevance – Continued Need for the Program ........................................... 14
4.3 Relevance – Alignment with Federal government Priorities ........................ 17
4.4 Relevance – Consistency with Federal Roles ............................................. 18
4.5 Performance – Design and Delivery – Performance Information ................ 19
4.6 Performance – Design and Delivery – Program Design .............................. 21
4.7 Performance – Design and Delivery – Barriers ........................................... 22
4.8 Performance – Design and Delivery – Communications ............................. 23
4.9 Performance – Design and Delivery – Best Practices ................................. 24
4.10
Performance – Effectiveness – Outputs and Outcomes .......................... 25
4.11
Performance – Effectiveness – Intended Outcomes ................................ 28
4.12
Performance – Efficiency ......................................................................... 28
4.13
Performance – Economy ......................................................................... 30
5. Conclusions ...................................................................................................... 33
6. Lessons Learned .............................................................................................. 35
Annex 1. Bibliography of Documents Reviewed ......................................................... i
Annex 2: Literature Review ........................................................................................iv
Annex 3. Interviewees and Interview Guides ........................................................... xvi
Annex 4. Summary of Findings (Table) .....................................................................xx
Audit and Evaluation Branch
Evaluation of the PPRE
Executive Summary
Scope and Objectives of the Evaluation
A formal, independent evaluation is required to assess the relevance and
performance of the Pilot Project on Reducing Emissions from Vehicles and Engines
(PPRE). The scope of the evaluation is to cover the three years of the program and
all related project activities. This requirement fulfills the commitment made in the
original program design for the PPRE. As per the requirements presented in the
Directive on the Evaluation Function, this evaluation will address value for money by
including clear and valid conclusions about the relevance and performance of the
pilot program. As the funding is not ongoing, and there is no decision required on
program renewal, recommendations are not part of this evaluation. Rather, the focus
is on lessons learned that may inform related future work and regulatory
development.
Pilot Project on Reducing Emissions from Vehicles and Engines
The PPRE was an Environment Canada initiative, under the Security and Prosperity
Partnership of North America (SPP), funded at the level of $1.2 million over three
years (2008-11). The objectives of this pilot project were for Environment Canada
and the United States Environmental Protection Agency (U.S. EPA) to collaborate
on initiatives aimed at reducing emissions from vehicles and engines in order to
better utilize the capabilities of each agency, and to share information on innovative
programs such as retrofitting diesel engines for lower emissions. Funding was
required to work on collaborative initiatives with the U.S. EPA, and to work with
stakeholders to reduce emissions from diesel buses and heavy-duty trucks that are
currently on the road.
Evaluation Methodology and Design
The evaluation addressed five key issues: (1) Continued need for the program; (2)
Alignment with government priorities; (3) Alignment with federal roles and
responsibilities; (4) Achievement of expected outcomes; and (5) Demonstrated
efficiency and economy. To examine these issues, and the specific evaluation
questions related to each issue, the evaluation employed three lines of inquiry: (1)
document review; (2) literature review and comparison of similar programs; and (3)
key informant interviews. The evaluation scope focused on Environment Canada’s
pilot project to work with stakeholders (within Canada) to reduce emissions from
diesel buses and heavy-duty trucks that are currently on the road.
Evaluation Conclusions
The key findings from the evaluation, detailed in the report, are summarized below.
Relevance
1. The pilot project and its respective activities are relevant and continue to be
needed to address heavy-duty diesel (HDD) greenhouse gas emissions and
criteria air contaminants produced by public fleets.
2. The pilot project is directly aligned with federal government priorities and the
strategic outcomes of Environment Canada to take action on climate change and
reduce air emissions.
Environment Canada
i
Audit and Evaluation Branch
Evaluation of the PPRE
3. The activities of the pilot program are aligned with federal roles and
responsibilities.
Performance – Project Design and Delivery
4. Performance information is provided by funded partners, primarily at the output
level, in a timely and relevant manner. Data pertaining to the project’s
performance indicator are currently being collected and will be reviewed at the
end of the pilot. The evaluation did not find evidence of a formal internal reporting
mechanism to senior managers or federal partners, though information was
shared throughout the project on an informal basis.
5. The project was designed appropriately and included a clear and plausible link
between program activities, outputs and expected results. The resources
allocated to this project were commensurate with the expected results.
6. The pilot project identified that access to funding and the ability to work with the
variety of players who are involved with municipal fleets (e.g. ranging from
municipal governments to subcontractors and third parties) may be barriers to
implementing a similar project on a national scale.
7. On behalf of Environment Canada, project partners engaged municipal fleet
operators regularly and communicated the outputs from project activities in a
timely and relevant manner. Correspondence and communication within the
federal community was informal and in some cases inconsistent.
8. The pilot project was successful in developing a best practices guide and a
resources guide for municipal fleet managers, and in providing advice on how to
move forward with addressing emissions from school buses.
Performance – Effectiveness
9. The pilot project successfully generated a number of important outputs, including
a best practices guide and a resources guide, fact sheets, and multiple reports
that increased the level of knowledge about and number of approaches to
addressing the on-road HDD municipal fleet and school bus sectors. One
planned output (a baseline and profile of municipal fleets in Canada) was not
achieved. Data are not yet available to assess the achievement of the project’s
expected outcome.
10. The evaluation did not find any negative unintended outcomes.
11. Environment Canada successfully partnered with organizations that were experts
in the subject matter and/or maintained a comprehensive understanding of the
key organizations and individuals within the municipal fleet sector. In one case,
the Department successfully leveraged funding from an organization to support
the project.
12. The evaluation found that the project is on track to achieving its intended
outcome in an economical manner.
Environment Canada
ii
Audit and Evaluation Branch
Evaluation of the PPRE
Lessons Learned
The evaluation identified a number of lessons learned, and suggestions on how to
improve the delivery of similar projects complementary to regulatory development.
These are detailed in Section 6, and are summarized below.
Key Lessons Learned from the PPRE:



Allocating an appropriate amount of time to designing a
program/project can lead to relevant and valuable results. Environment
Canada allocated one year to design a successful pilot project that was
relevant and met a demonstrable need that was not duplicative of other
federal efforts.
Partnering with organizations that maintain expertise and existing
networks can lead to effective results. Environment Canada worked with
non-governmental organizations (NGOs) that maintained existing networks
and a comprehensive understanding of municipal fleets across Canada. This
approach enhanced program efficiencies by creating immediate access to
existing networks, and provided greater assurance that project deliverables
were credible and were useful to the municipal fleet community.
When provided with the appropriate tools and funding, Canadian
municipalities can engage in actions to reduce emissions from their
fleet vehicles. The PPRE has shown that by building the capacity of
municipalities, providing tools, and providing assistance in accessing funds,
emission reductions can be supported and achieved.
Suggestions for Future Projects:


A one-window approach, providing information and resources to reduce
emissions from all types of vehicles in the municipal fleet, would be
beneficial. There is merit in addressing the emissions from all vehicle types
that are present in the municipal fleet more broadly, rather than just focusing
on HDD. Fleet managers deal with multiple types of vehicles and make
integrated decisions that can best be informed and supported by a more
comprehensive approach to addressing emissions from all vehicle types.
Long-term funding is required to ensure action is taken to reduce fleet
emissions. The pilot project has increased the general level of awareness of
municipal fleet managers and has provided a number of resources to reduce
the emissions associated with municipal fleets. However, the pilot program
has also underlined a need for long-term funding to implement solutions to
reduce emissions produced by municipal fleets. This pilot project has
demonstrated that long-term funding from sources with municipal jurisdiction
should be identified.
Environment Canada
iii
Audit and Evaluation Branch




Evaluation of the PPRE
The Government of Canada can play an important role in identifying
innovative technologies for reducing vehicle emissions. The Government
of Canada can play an important role in testing of technologies for reducing
on-road vehicle emissions. The PPRE-funded project in Ontario led to the
identification of retrofit technology that has a proven ability to reduce
emissions produced by school buses.
Environment Canada is best placed as an enabler for reducing vehicle
emissions. The evaluation highlighted that this pilot project was a unique
opportunity for Environment Canada to lead a pilot project that identified a
gap, was complementary to HDD regulations, and addressed greenhouse
gases and air pollutants. However, other federal departments, such as
Natural Resources Canada, are now better placed to lead comprehensive
capacity-building programs in this area, because of their mandate and
emerging work under the FleetSmart program.
A national emissions baseline for municipal fleets is required. The pilot
project identified the need for a national emissions baseline for municipal
fleets, but found it challenging to complete. In order to target cost-effective
emission reduction activities and to measure change with municipal fleets, a
baseline is required.
The impact of a capacity-building / outreach program is maximized
when delivered jointly with a suite of like-minded programs, including
regulations/standards. Although education and capacity building are
important tools to help municipalities take action to reduce their emissions, a
suite of complementary and reinforcing initiatives, including regulations,
provides them with the push and pull required for taking substantive action
within their jurisdictions.
Environment Canada
iv
Audit and Evaluation Branch
Evaluation of the PPRE
1. Introduction
The evaluation of the Security and Prosperity Partnership of North America (SPP)
Pilot Project on Reducing Emissions from Vehicles and Engines (PPRE) was
undertaken from January to March 2011, in response to a commitment to conduct an
evaluation of the original program design documentation submitted to Treasury
Board Secretariat officials. The intent of this evaluation is to inform management in
Environment Canada, as well as the Treasury Board Secretariat, about the success
of the pilot project and lessons learned that may inform related future work.
For the PPRE evaluation, program staff in the Strategic Transportation and Policy
and Analysis Section of Environment Canada’s Transportation Division, along with a
representative from the Department’s Evaluation Division, led the evaluation and
performed an oversight role that involved providing input and feedback to the
consultant hired to conduct the independent evaluation; approved all deliverables
outlined in the contract; and coordinated the management response to the
evaluation report.
This report summarizes the evaluation process and findings, conclusions and
lessons learned. The report is organized into five sections:





Section 2 describes the pilot project, including the governance structure,
allocated resources and expected outcomes
Section 3 indicates the evaluation objectives, scope, issues and approach
taken
Section 4 documents the evaluation findings
Section 5 presents the evaluation conclusions
Section 6 outlines the lessons learned
The report also includes a number of annexes that form the information base for the
evaluation’s key findings and conclusions, including a list of reference material,
literature review details, interviewees, and a summary findings table.
Environment Canada
1
Audit and Evaluation Branch
Evaluation of the PPRE
2. Background
The SPP is an Industry Canada–led initiative that provides a flexible means for
dialogue, priority setting, collaboration and action on issues affecting the security,
prosperity and quality of life of Canadians, Americans and Mexicans. It addresses
diverse issues, such as border facilitation, the environment, food and product safety,
and includes measures to improve overall North American competitiveness.
A total of $39.2 million was allocated to the SPP over three years (2008-09, 200910, 2010-11). Of this allocation, $6.2 million was provided to Environment Canada
for three projects:
1. Chemicals assessment ($4 million)
2. A North American air emissions inventory ($1 million)
3. The PPRE ($1.2 million)
An overview of the funding provided to Environment Canada can be viewed in Table
1 below.
Table 1: SPP Funding to Environment Canada
Initiative
Chemicals Assessment
N.A. Emissions Inventory
PPRE
TOTAL
2008-09
$1,190,000
$163,000
$120,000
$1,473,000
2009-10
$1,390,000
$418,000
$540,000
$2,348,000
2010-11
$1,420,000
$419,000
$540,000
$2,379,000
Total
$4M
$1M
$1.2M
$6.2M
This evaluation relates solely to the PPRE.
Table 2 summarizes the resources allocated to the Transportation Division for PPRE
activities.
Table 2: Resources allocated to the Transportation Division for PPRE activities
Funded
Activities
FTE
Salaries
Employee
Benefits
O&M
G&C
PWGSC
Accommodation
Total
2008-09
Environment Canada
Fiscal Year
2009-10
Total
2010-11
0.5
$50,549
$10,109
1
$107,764
$21,553
1
$107,764
$21,553
$266,077
$53,215
$52,770
$6,572
$73,009
$323,664
$14,010
$73,009
$323,664
$14,010
$198,788
$647,328
$34,592
$120,000
$540,000
$540,000
$1.2M
2
Audit and Evaluation Branch
2.1
Evaluation of the PPRE
Profile
The PPRE complements the Clean Air Regulatory Agenda (CARA) by addressing
emissions from existing vehicles and engines that could remain on Canada’s roads
for up to 20 years.1 It targets on-road heavy-duty diesel (HDD) vehicles such as
trucks and school buses, and explores how best to voluntarily reduce their emissions
(greenhouse gases [GHGs] and criteria air contaminants [CACs]). Emissions from
these pre-2007 diesel engines contribute significantly to local, regional and global air
pollution. The Government of Canada carries out various emission reduction
programs targeting HDD vehicles (mainly Class 8 freight vehicles), such as
Transport Canada’s ecoFREIGHT program and Natural Resources Canada’s
(NRCan’s) FleetSmart program. This initiative complements these programs by
focusing on Canada’s existing on-road public HDD vehicles, such as municipal fleet
vehicles.
2.1.1 Objectives and Expected Outcomes
The objective of the PPRE, as noted in its program design, is as follows:
“Environment Canada and the United States Environmental Protection Agency (U.S.
EPA) will collaborate on initiatives to reduce emissions from vehicles and engines to
better utilize the capabilities of each agency, as well as share information on
innovative programs such as retrofitting diesel engines for lower emissions. Funding
is required to work on collaborative initiatives with the U.S. EPA, and to work with
stakeholders to reduce emissions from diesel buses and heavy-duty trucks that are
currently on the road.”
The first year of the SPP pilot (2008-09) was dedicated to developing the program
design, which included consultations with Transport Canada, NRCan and the U.S.
EPA to gain knowledge and advice on how the pilot project should be designed.
These consultations allowed Environment Canada to better understand what work
was already being carried out in this area (e.g., ecoFREIGHT and FleetSmart) and
gaps or opportunities that could be addressed through the pilot project. Based on
this design phase, two initiatives emerged:
Reducing Emissions from Canada’s On-Road Heavy-Duty Diesel Municipal Fleet
The objective of this project was to increase knowledge, awareness and supporting
tools for municipalities to reduce their emissions of GHGs and CACs, by highlighting
emission reduction opportunities and identifying barriers and challenges to be
In response to the Government of Canada’s On-Road Vehicle and Engine Emission Regulations, which
introduced a new regulatory framework under the Canadian Environmental Protection Act, 1999,
emission-reduction technologies have been introduced on HDD engines (starting with the 2007 model year).
New HDD engines will emit significantly fewer air pollutants, including particulate matter (PM) and nitrogen
oxides (NOx). However, diesel vehicles built before 2007 will continue to emit comparatively high levels of NO x
and PM over the next 20 years.
1
Environment Canada
3
Audit and Evaluation Branch
Evaluation of the PPRE
overcome during the implementation of emission reduction strategies for fleet
vehicles. To realize this objective, Environment Canada signed a Contribution
Agreement with the Federation of Canadian Municipalities (FCM) in January of
2010, in order to enable the following activities:
 Establish partnerships with municipal fleet managers’ networks, federal and
provincial agencies, environmental and non-governmental organizations
(NGOs) and private sector organizations.
 Create a baseline of the municipal heavy-duty fleet within Canada, including a
fleet profile, an overview of best practices, and identification of barriers and
challenges for initiatives to green fleets.
 Identify and create an online resource in both official languages for municipal
fleet managers to access information on best practices available to achieve
emission reductions.
 Promote in both official languages the online resource and other products and
services (e.g., publications) to reduce HDD emissions.
 Deliver education and training workshops in collaboration with partners to
increase the capacity of municipal fleet managers to make decisions about
reducing HDD emissions within their fleet profile.
 Develop a business case for emission reduction initiatives in heavy-duty
fleets, to be included in a resources guide for fleet managers.
In addition to the work completed with the FCM, Bronson Consulting was contracted
to conduct a scoping study of Canadian school bus fleet characteristics, in order to
identify key considerations in the development of any clean air programs for school
bus fleets.
Furthermore, Environment Canada’s Ontario Region provided funds to two not-forprofit groups to conduct similar activities:
1. The Clean Air Partnership (CAP) – Funding was allocated to CAP to update
information about the level of awareness of school bus emissions and actions
that Ontario groups have taken to address these emissions. CAP also
examined projects undertaken by the federal government and provincial
governments, and policies adopted by school boards in other provinces, to
reduce emissions and exposures associated with school buses.
2. My Sustainable Canada (MSC) – Funding was allocated to MSC to implement
two diesel-retrofit pilot programs in Ontario (for small municipal school buses
and garbage trucks).
Environment Canada
4
Audit and Evaluation Branch
Evaluation of the PPRE
The expected outcome for this pilot project was2: Increased implementation by
Canadian municipalities of actions to reduce emissions from municipal on-road HDD
vehicles.
The main performance indicator for this outcome was: “Population weighted count of
municipalities that participate or report through the FCM that they are implementing
greening fleet initiatives consistent with the pilot project (i.e., pre-2007 HDD
vehicles).”
Table 3 outlines the resources allocated to this initiative.
Table 3: Allocated Resources
Funded Activities
Fiscal Year
2008-09
2009-10
2010-11
Reducing Emissions from Canada’s On-Road HDD Municipal Fleet
FTEs
0.5 FTE
0.6 FTE*
0.6 FTE*
O&M
$52,770
$73,009
$73,009**
G&C
$148,664
$148,664
* 0.1 FTE for the evaluation
** Includes $21,000 for the evaluation
Total
$198,788
$297,328
Advancing Regulatory Action to Reduce Vehicle and Engine Emissions
Canada and the United States have agreed to work together under the Canada-U.S.
Air Quality Agreement to reduce transportation-related emissions. Under the SPP,
Environment Canada and the U.S. EPA proposed options to improve information
sharing and build upon existing informal mechanisms through the Agreement.
The objective of this initiative was to reduce emissions from vehicles and engines,
while enhancing pre-regulatory cooperation between Canada and the United States.
Furthermore, this project aimed to demonstrate how pre-regulatory collaborative
work between Canada and the United States could work, as well as whether there
are lessons that could apply to future regulatory work completed by both countries.
More specifically this project aimed to identify and complete testing and technical
assessments for reducing emissions from vehicles and engines. Based on emerging
priorities and the recognized importance of complementary policy and regulatory
regimes for North America’s transportation sector, this project was designed to
support the alignment of future regulatory measures for the on-road vehicle fleet.
The expected outcome of this project was intended to be:
 Canada and United States collaboration on technical studies to support future
regulatory actions for the on-road fleet.
– Performance Indicator: Number of technical studies developed in
collaboration with the U.S. EPA to support future regulatory actions for the
on-road fleet.
2
A Results Based-Management Framework was not developed for this program; therefore a logic model does
not exist.
Environment Canada
5
Audit and Evaluation Branch
Evaluation of the PPRE
Table 4 outlines the resources for this initiative.
Table 4: Allocated Resources
Funded Activities
Fiscal Year
2008-09
2009-10
2010-11
Advancing Regulatory Action to Reduce Vehicle and Engine Emissions
Salaries
0.4 FTE
0.4 FTE
O&M
G&C
$175,000
$175,000
Total
$350,000
Given that the grants and contributions (G&C) funding mechanism could not be used
and that regulatory alignment with the U.S. EPA on the heavy-duty fleet was quickly
evolving into a major focus of work for Environment Canada (i.e., collaborative work
and information sharing between the two governments was already taking place
under the Canada–U.S. Air Quality Agreement), a decision was taken to not
implement this project and to transfer some of the funds to the first initiative.
2.1.2 Stakeholders and Beneficiaries
The PPRE aims to reduce emissions from on-road HDD vehicles and engines within
Canada. Its reach extends to a broad range of stakeholders and beneficiaries that
include:
 Canadian municipalities who own and operate fleets
 Environment Canada staff and management involved with vehicle regulatory
development and policy analysis
 Canadian environmental NGOs
 Other government departments (including Transport Canada and NRCan)
2.2
Governance
The PPRE was implemented and managed by Environment Canada’s
Transportation Division, Energy and Transportation Directorate (ETD). The project
was implemented by a policy analyst from the Strategic Transportation and Policy
and Analysis Section hired as part of the PPRE, and management oversight was
provided by the Manager, Strategies and Outreach, Strategic Transportation and
Policy and Analysis Section.
No separate committee or management structure was established within
Environment Canada to further oversee or coordinate the PPRE initiatives.
The overall accountability structure for this pilot is the same as for other
Transportation Division programs, i.e., this project reports through the Director
General, ETD, to the Environmental Protection Board. Results are included in the
Program Activity Architecture (PAA) element:
 Strategic Outcome 3, Clean Environment
Environment Canada
6
Audit and Evaluation Branch



Evaluation of the PPRE
Program Activity 3.2, Climate Change and Clean Air
Program Sub-Activity 3.2.1, Climate Change and Clean Air Regulatory
Program
Program Sub-Sub-Activity 3.2.1.2, Transportation Sector Emissions
Environment Canada
7
Audit and Evaluation Branch
Evaluation of the PPRE
3. Evaluation Design
3.1
Purpose and scope
A formal, independent evaluation is required to assess the relevance and
performance3 of the SPP PPRE. The scope of the evaluation covers the three years
of the program from 2008 to 2011 and all related project activities. This requirement
fulfills the commitment made in the program design for this pilot, which was
submitted to and approved by Treasury Board officials in the summer of 2009.
As per the requirements presented in the Directive on the Evaluation Function, this
evaluation addresses value for money by including conclusions about the relevance
and performance of the pilot program. As the funding is not ongoing, and there is no
decision required on program renewal, recommendations are not part of this
evaluation. Rather, the focus is on lessons learned that may inform related future
work and regulatory development.
3.2
Evaluation Issues and Questions
The evaluation provides conclusions on the following broad evaluation questions:
 Relevance: Did the pilot project remain consistent with, and contribute to, federal
government priorities and responsibilities and address actual needs?
 Performance – Project Design and Delivery: Was the PPRE project designed
and delivered appropriately?
 Performance – Effectiveness: Has the project achieved its intended outcomes?
 Performance – Efficiency and Economy: Were the most appropriate,
economical and efficient mechanisms used to achieve outcomes?
As this pilot project concluded at the end of March 2011, and all results are therefore
not yet available, the evaluation report focuses on outputs and early outcomes
achieved, as well as lessons learned.
Table 5 presents the evaluation framework, which outlines the evaluation questions
and indicators that were explored as part of this evaluation, as well as the methods
utilized to collect evidence.
Table 5: Evaluation Framework
EVALUATION FRAMEWORK
Evaluation Issue & Question
Relevance
1. Was there a need for the
Indicators
-
Demonstration of environmental need
Methods
-
Document
3
As per the 2009 Treasury Board (TB) Policy on Evaluation, the evaluation should assess: Relevance: The
extent to which the Program addresses a continued need, is aligned with Government priorities, and is aligned
with federal roles and responsibilities; and Performance: The extent to which the Program has achieved its
expected outcomes, and demonstrates efficiency and economy.
Environment Canada
8
Audit and Evaluation Branch
Evaluation of the PPRE
EVALUATION FRAMEWORK
Evaluation Issue & Question
project? Is there a need for the
project to continue?
2. Is the project aligned with
federal government priorities?
3. Is the project consistent with
federal roles and responsibilities?
Indicators
- Demonstration that the pilot
complements, and does not duplicate,
other similar programs (CARA,
ecoFREIGHT, etc.)
- Demonstration that the pilot addresses
gaps in environmental need
- Project activities and reach are
connected to environmental need
- Views of stakeholders and project
managers on the connection of project
objectives with environmental need
- Project’s objectives correspond to
recent/current federal government
priorities
- Project’s objectives are aligned with
current departmental strategic
outcomes
- Evidence of alignment with SPP,
CARA, ecoFREIGHT and FleetSmart
- Project mandate aligned with federal
government jurisdiction
- Views on the appropriateness of
federal involvement
Performance – Project Design and Delivery
4. Are appropriate performance
- Existence of effective performance
data being collected, captured and
measurement plan
safeguarded? If so, is this
- Extent to which performance data are
information being used to inform
reliable, timely and relevant
senior management / decision
Extent to which performance data
makers?
inform/support EC’s decision-making
processes
5. Is the program design
- Plausible link between program
appropriate for achieving expected
activities, outputs, and intended
program results?
outcomes
- Clearly defined and understood
governance structure, including
program processes, roles,
responsibilities and accountabilities
- Program resources/capacity
commensurate with expected program
results
- Views on the appropriateness of
program activities, processes and
governance structures
Environment Canada
Methods
review
- Literature
review
- Interviews
-
-
-
-
Document
review
Interviews
Document
review
Interviews
Literature
review
Document
review
Key informant
interviews
Document
review
Key informant
interviews
9
Audit and Evaluation Branch
Evaluation of the PPRE
EVALUATION FRAMEWORK
Evaluation Issue & Question
6. Are there any barriers or
challenges that could affect the
success of this project if it was to
be implemented nationally or
integrated into an existing national
project?
Indicators
- Views of stakeholders and project
managers
- Evidence of documented barriers or
challenges
- Evidence of solutions or approaches to
addressing barriers or challenges
7. Have the results of the pilot
project been communicated
effectively? How could
communications be improved?
-
-
8. What are the best practices and
lessons learned as a result of the
project?
Performance – Effectiveness
9. To what extent have the
intended outputs and outcomes
been achieved as a result of the
project?
-
Views of project managers, partners
and recipients
Extent to which planned
communications activities have been
implemented as intended
Evidence of information sharing,
distribution of project publications and
reports, etc. to public fleet operations
and other fleets
Identified lessons learned and shared
best practices among partners
Identified strengths and weaknesses
Views on strengths / best practices,
weaknesses and lessons learned
Views of project managers, partners
and recipients on achievement of
outputs and outcomes
- Analysis of available performance data
on project’s expected results and
performance indicator
- Population-weighted count of
municipalities, through FCM, that
report implementing green fleet
initiatives
10. Have there been any
- Presence/absence of unintended
unintended outcomes, either
outcomes
positive or negative?
- Views on whether unintended
outcomes occurred
Performance – Efficiency and Economy
11. Has the project undertaken its
- Analysis of project operational costs in
activities and delivered products
relation to the production of outputs
(e.g., technical studies) in the most - Views on whether the cost of
efficient manner?
producing project outputs is as low as
possible
- Evidence of / views on whether there
are alternative, more efficient ways of
delivering project activities and outputs
12. Has the project achieved, or is
- Views on whether good value is being
Environment Canada
-
Methods
- Interviews
- Document
review
-
Interviews
Document
review
-
Interviews
Document
review
-
Document
review
Interviews
-
-
-
-
Document
review
Interviews
Document
review
Interviews
Literature
review
Financial
analysis
Document
10
Audit and Evaluation Branch
Evaluation of the PPRE
EVALUATION FRAMEWORK
Evaluation Issue & Question
it on track to achieve, its intended
outcomes in the most economical
manner?
3.3
Indicators
obtained with respect to the use of
public funds
- Evidence of / views on whether there
are alternative program models that
would achieve the same expected
outcomes at a lower cost
Methods
review
- Interviews
- Literature
review
Evaluation Approach and Methodology
Three lines of inquiry were completed to collect evidence in order to address the
evaluation questions and develop the evaluation findings:
1. An in-depth review of all relevant documents provided by the organizations
engaged in the PPRE was completed, and additional documents were
requested where gaps were identified or where additional information was
needed to supplement the existing evidence base. The project team reviewed
each document to understand the mandated requirements, goals and
objectives, governance structure, authorities, activities, outputs and outcomes
for the pilot project and its components. The overall adequacy of the
document evidence base (e.g., availability of performance information) was
also assessed. All evidence from the documentation was examined against
the defined evaluation issues. A bibliography of the documents reviewed is
included in Annex 1.
This data collection method addressed evaluation questions 1-12.
2. A literature review was conducted to: (1) better understand the federal
government’s approach to addressing HDD vehicle emissions compared to
the federal approach in the United States; and (2) compare the pilot to other
programs in Canada that aim to reduce HDD vehicle emissions through
capacity building. Details on the literature reviewed are included in Annex 2.
This data collection method addressed evaluation questions 1, 2, 3, 11 and
12.
3. Key informant interviews with representatives from Environment Canada as
well as key partners and stakeholders were either conducted in person (e.g.,
with Environment Canada staff) or by phone. Interviews provided information
on the program components’ relevance and performance. A total of 8
interviews with 11 individuals were conducted. A list of the categories and the
number of informants is included in Annex 3, as are the interview guides.
Environment Canada
11
Audit and Evaluation Branch
Evaluation of the PPRE
This data collection method addressed evaluation questions 1-12.
Draft findings from the above lines of inquiry were integrated and presented to the
managers overseeing the evaluation in Environment Canada, in order to validate key
facts and address omissions and errors before completion of the final evaluation
report.
3.4
Limitations
The PPRE was not complete at the time of the evaluation (the evaluation findings
were arrived at in February 2011 and the pilot project ended in March 2011). As
such, the evaluators have commented exclusively on the pilot’s ability to deliver its
intended outputs and achieve its intended outcomes. Also, the PPRE will not receive
any additional funding beyond this fiscal year. Therefore, the evaluation did not
include a series of recommendations to improve the delivery and effectiveness of the
PPRE. Instead, a summary of lessons learned has been included to inform the
design and delivery of additional pilot projects that focus on addressing vehicle
emissions.
The evaluation did not address the second initiative (i.e., Advancing Regulatory
Action to Reduce Vehicle and Engine Emissions), as it was not implemented and no
G&C funds were spent on this project.4 Therefore, the evaluation scope does not
reflect the original program design as the second initiative was not examined.
4
The Evaluation Team was informed that the second initiative was never launched formally and was not
announced by the Minister of the Environment. Program staff indicated that G&C funding allocated to this
initiative was not utilized for this project, although some of the funding was reallocated to the pilot project and the
remainder was lapsed.
Environment Canada
12
Audit and Evaluation Branch
Evaluation of the PPRE
4. Findings by Evaluation Issue
This section outlines evaluation findings for each of the defined evaluation issues
and questions. Findings and ratings are presented by evaluation issue for the PPRE,
in keeping with the protocol of Environment Canada’s Evaluation Division, with
specific comments included under each evaluation question.
4.1
Rating of Findings
Ratings have been provided to indicate the degree to which the PPRE has
addressed the key evaluation criteria. The rating is assessed according to the
following chart provided by Environment Canada’s Evaluation Division. A summary
table of these ratings/findings is provided in Annex 4.
Table 6: Evaluation Ratings Legend
Rating
Significance
Achieved
Progress Made;
Attention Needed
Little Progress;
Priority for Attention
N/A
~
The intended outcomes or goals have been achieved or met
Considerable progress has been made to meet the intended outcomes or
goals, but attention is still needed
Little progress has been made to meet the intended outcomes or goals
and attention is needed on a priority basis
A rating is not applicable
Outcomes achievement ratings are based solely on subjective evidence
Environment Canada
13
Audit and Evaluation Branch
4.2
Evaluation of the PPRE
Relevance – Continued Need for the Program
Evaluation Issues (and
related questions)
1. Was there a need for the
project? Is there a need for
the project to continue?
Indicators
-
-
-
Demonstration of environmental
need
Demonstration that the pilot
complements and does not
duplicate other similar programs
(CARA, ecoFREIGHT, etc.)
Demonstration that pilot
addresses gaps in environmental
need
Project activities and reach are
connected to environmental need
Views of stakeholders and
project managers on the
connection of project objectives
with environmental need
Overall
Rating
Achieved
SUMMARY:
The evaluation found that there was a clear need for the pilot and a continued need
to reduce emissions from municipal fleets. These vehicles are a significant
contributor to local air pollution and GHG emissions, and pose a health risk to
vulnerable populations (e.g., school children). The evaluation found that there are
regional programs that target this need, but they are not national in scope, do not
focus exclusively on on-road HDD vehicles, and remedial actions are not
widespread. There are also similar federal programs, but the pilot was designed to
complement these initiatives and address a specific area (i.e., on-road HDD
including GHG and CAC emissions). The PPRE was designed to target an identified
gap in meeting the needs of municipal fleet operators nationally. Due to the noted
need, the overall rating has been assessed as Achieved.
Environment Canada
14
Audit and Evaluation Branch
Evaluation of the PPRE
Municipalities operate HDD vehicles as a normal part of their operations, for services
such as the transport of school children, garbage collection and snow removal.
However, municipal fleets are a key contributor to the environmental and health
impacts of municipal operations. Burning diesel fuel creates GHG emissions that
contribute to climate change, and CACs that
contribute to smog and acid rain. On average,
Emissions from School Buses
Canadian municipal fleets are responsible for
With approximately 800,000 Ontario
children being transported on school
three to five percent of a municipality’s total GHG
buses each year, school bus exposures
emissions, and consume billions of litres of fuel
represent a significant public health
each year. In addition, HDD vehicles such as
concern. Diesel exhaust has strong
health impacts, particularly for children
school buses can expose on-board children to
where early exposures can have long
elevated levels of air pollutants (see sidebar),
term health impacts. Exposures studies
adding to their environmental exposure. In
have found that exposures on-board
school buses can be significantly
particular regions (e.g., British Columbia and
reduced, even under idling conditions,
southern Ontario), local air quality is of particular
by retrofitting school bus tailpipes with
concern and international agreements have been
emission control devices (School Board
Factsheet (Health School Buses), Sept.
established to address air quality (e.g., Canada–
2010).
U.S. Air Quality Agreement).
Although federal and provincial programs address emissions from HDD vehicles
(see Annex 2), the PPRE was designed to complement these programs and target
on-road legacy public vehicles that will continue to emit GHGs and CACs for many
years. The pilot was designed not just to promote technology and tools to reduce
emissions, but also to address needs specific to municipalities with respect to
obtaining support for action (e.g., demonstrated economic benefits, business cases
for action, etc.).
As part of the PPRE program design, Environment Canada consulted with federal
partners and examined current initiatives to ensure the pilot project filled an
identified gap. Partners indicated that “municipalities are committed to reducing
emissions from their fleets, but are unfamiliar with the appropriate technological and
behavioural adaptations.” Furthermore, the literature review completed for this
evaluation noted that regional programs in British Columbia, Ontario and Nova
Scotia work with industry and/or fleet managers to reduce emissions; however,
these programs are not national in scope. Table 7 summarizes these regional
capacity-building programs and compares them to the PPRE (see Annex 2 for
details).
Environment Canada
15
Audit and Evaluation Branch
Evaluation of the PPRE
Table 7: Comparison of Capacity-Building Programs
PROGRAMS
Clean Nova Scotia - Fleetwiser Program
BC Air Action Plan (Action #10 only)
Funding source
Delivery partner
Fleet Challenge Ontario (Municipal Fleet
Review)
Provincial government
One NGO
Provincial government
One NGO
Federal government
One NGO
Target audience
Municipalities
Types of vehicles
On-road vehicles
Off-road vehicles (vehicle selection)
To support green management decisions
and demonstrate ways to improve
operations, thereby reducing costs and
GHG emissions
Municipalities
Commercial sector
On-road vehicles
Municipalities
Utilities (energy)
On-road vehicles
To help owners of commercial and public
sector vehicle fleets improve their fuel
efficiency and reduce emissions
To create a cleaner, healthier environment
by informing, enabling and inspiring Nova
Scotian fleet operators to respect and
consider the environment in all their choices
Objective
Mechanisms/activities
Fleet audits
Workshops for fleet managers
Networking/forums for fleet
managers
Toolkits and guides
Links to other resources on
website
Testing of reduction technologies
Duration
Source
Environment Canada








2008 (pilot project)
2009 to 2012
http://www.fleetchallenge.ca/content/fleetreview
PPRE Pilot Project
Federal government
Three NGOs
One consulting firm
Municipalities
School bus operators and districts
On-road HDD vehicles
Heavy-duty diesel engines
To reduce emissions (GHGs and CACs)
from Canada’s on-road HDD municipal fleet
by increasing knowledge and awareness
and supporting tools for municipalities








2007 to 2011
2009 to 2011
2008 to 2011
http://www.bcairsmart.ca/transportation/hea
vyduty.html
http://www.greenfleetsbc.com/
http://www.clean.ns.ca/content/Sustainable_
Fleet
http://Fleetwiser.norex.ca/
Evaluation Work Plan, February 2011
http://fmv.fcm.ca/Enviro-Fleet/
16
Audit and Evaluation Branch
4.3
Evaluation of the PPRE
Relevance – Alignment with Federal government Priorities
Evaluation Issues (and
related questions)
2. Is the project aligned with
federal government priorities?
Indicators
-
-
-
Project’s objectives correspond
to recent/current federal
government priorities
Project’s objectives are aligned
with current departmental
strategic outcomes
Evidence of alignment with SPP,
CARA, ecoFREIGHT and
FleetSmart
Overall
Rating
Achieved
SUMMARY:
The PPRE’s objectives are clearly aligned with current federal government priorities,
and with Environment Canada’s strategic outcomes to take action on climate change
and reduce air emissions. The PPRE was found to be complementary to other
federal programs such as CARA and ecoFREIGHT. However, it was found to be
similar to NRCan’s FleetSmart program, although FleetSmart focuses broadly on
GHG emissions and public and private vehicles, while the PPRE focused on both
GHGs and CACs specifically for on-road public municipal fleets and school buses.
Due to the alignment with federal priorities and outcomes, the overall rating has
been assessed as Achieved.
When designed, the PPRE was aligned with the Government of Canada priority,
identified in the 2008 Speech from the Throne, of “tackling climate change and
preserving Canada’s environment by reducing greenhouse gas emissions 20
percent by 2020.”5 This commitment to addressing climate change is maintained in
the 2010 Speech from the Throne.
The PPRE is aligned with Environment Canada’s priorities and strategic outcomes,
as evidenced in the following Reports on Plans and Priorities (RPPs):


5
At the program design stage, the PPRE was aligned with the 2008-2009 RPP
priority “reducing GHG emissions and air pollution” and contributed to the
Department’s strategic outcome, as noted in the 2008 PAA: Canadians and their
environment are protected from the effects of pollution and waste (Transportation
Sector Emissions); and
Currently, the PPRE is aligned with the 2010-2011 PAA strategic outcome:
Threats to Canadians and their environment from pollution are minimized
(Climate Change and Clean Air Program Activity, Climate Change and Clean Air
Regulatory Program Sub-Activity, Transportation Sector Emissions Sub-sub
Link to 2008 Speech from the Throne: http://www.speech.gc.ca/eng/media.asp?id=1378
Environment Canada
17
Audit and Evaluation Branch
Evaluation of the PPRE
Activity – which has the expected result: Reduced air pollutant emissions from
Canadian motor vehicles and fuels).
The PPRE complements the Government of Canada’s CARA by addressing
emissions from existing vehicles and engines that could remain on Canada’s roads
for up to 20 years. CARA focuses on the On-Road Vehicle and Engine Emission
Regulations to reduce emissions from new HDD engines (starting with the 2007
model year). However, diesel vehicles built before 2007 will continue to emit
comparatively high levels of nitrogen oxides (NOx) and PM over the next 20 years.
The PPRE targets the on-road legacy public HDD vehicle fleet, such as municipal
fleet vehicles, and explores how best to voluntarily reduce their emissions. Reducing
emissions from the in-use public HDD fleet addresses two important Government of
Canada priorities: reducing emissions (GHGs and CACs) and improving the health
of Canadians.
The PPRE was designed to complement other federal programs such as Transport
Canada’s ecoFREIGHT and NRCan’s FleetSmart (details of these programs are
provided in Annex 2). It clearly complements ecoFREIGHT, which works with the
freight transportation industry to increase the uptake of technologies and practices
that reduce fuel consumption and emissions of CACs and GHGs. However, there is
some overlap with the FleetSmart program, which offers free practical advice on how
energy-efficient vehicles and business practices can reduce commercial and
municipal fleet operating costs, improve productivity and increase competitiveness.
NRCan’s transportation programs, in keeping with their PAA, indicate that they “aim
to reduce GHG emissions from on-road transportation by encouraging drivers and
fleet managers to use energy-efficient purchasing, driving and vehicle maintenance
behaviours.” This is similar to the Environment Canada strategic outcomes.
However, the focus of the FleetSmart program is GHGs (not air pollutants as well),
and more on training and retrofits to promote energy efficiency specifically related to
GHG emissions, while the PPRE focused particularly on capacity building and tools
specific to HDD public municipal fleets and addressed both GHG and CAC
emissions. Also, the PPRE focuses solely on public fleets (municipal fleets and
school buses), while the FleetSmart program focuses on public and private vehicles.
4.4
Relevance – Consistency with Federal Roles
Evaluation Issues (and
related questions)
3. Is the project consistent
with federal roles and
responsibilities?
Environment Canada
Indicators
-
Project mandate aligned with
federal government jurisdiction
Views on the appropriateness of
federal involvement
Overall
Rating
Achieved
18
Audit and Evaluation Branch
Evaluation of the PPRE
Evaluation Issues (and
Indicators
Overall
related questions)
Rating
SUMMARY:
Managing emissions is a partnership between multiple government jurisdictions and
stakeholders. There is a federal role and responsibility for the Government of
Canada to regulate emissions from vehicles and engines, and thus engaging in
pilots such as the PPRE to complement such regulations is appropriate providing
that federal authorities focus their efforts on facilitating and enabling such programs
and projects with relevant partners. Responsibilities for building capacity with and
reducing emissions from the in-use municipal fleet are also strongly aligned with
provincial and municipal roles and responsibilities. Stakeholders viewed the federal
role in this pilot as appropriate. Therefore, the overall rating has been assessed as
‘achieved’.
All levels of government have roles to play in addressing emissions from HDD
vehicles. The environment is a matter of shared jurisdiction between the provincial
and federal governments. Both levels of government derive their jurisdiction from
distinct powers granted by the Constitution Act 1867. Both levels of government
may have the authority to regulate the same emissions, albeit based upon different
Constitutional authorities. Municipalities derive their power from the provinces.
The PPRE builds on the federal role to address emissions nationally (through a
group such as the FCM) and to target regions where cross-border air emissions are
an issue (e.g., southern Ontario). However, the specific roles for working with
municipal fleet operators and funding retrofits to reduce emissions from diesel buses
and heavy-duty trucks is more strongly aligned with provincial and municipal roles.
This is evidenced by the existence of similar provincially funded programs in British
Columbia and Ontario (see Annex 2 for details). In fact, the literature review, in
comparing the PPRE to other relevant capacity-building programs in Canada,
highlighted that these programs were mostly designed for a regional audience and
implemented at the regional and local levels by NGOs.
Views from interviewees indicated that this kind of pilot was unique for Environment
Canada to undertake, as complementary measures to regulations are of interest to
the Department but not its primary focus. Stakeholders believed that the appropriate
federal role is primarily to regulate air quality, and secondarily to provide national
capacity building in order to support municipalities as well as to provide incentives
for change (adoption of new technology).
4.5
Performance – Design and Delivery – Performance
Information
Evaluation Issues (and
related questions)
4. Are appropriate
performance data being
collected, captured and
safeguarded? If so, is this
Environment Canada
Indicators
-
Existence of effective
performance measurement plan
Extent to which performance
Overall
Rating
Achieved
19
Audit and Evaluation Branch
Evaluation Issues (and
related questions)
information being used to
inform senior management /
decision makers?
Evaluation of the PPRE
Indicators
-
Overall
Rating
data are reliable, timely and
relevant
Extent to which performance
data inform/support Environment
Canada’s decision-making
processes
SUMMARY:
The evaluation found that a performance indicator was included in the original
program design for the PPRE, but has yet to be reported on. Performance
information has been provided by funded partners, primarily at the output level, in a
timely and relevant manner. However, there is little evidence of a formal internal
performance reporting mechanism to senior management and/or federal partners for
the entire pilot project. As the pilot has ended, performance data are not required to
support future decisions, but it is relevant to determine whether the pilot has
achieved its expected outcome. As it is expected that Environment Canada / FCM
will report on the stated performance indicator at the end of the pilot, and other
performance data have already been reported, the overall rating has been assessed
as Achieved.
A comprehensive performance measurement plan was not developed for the PPRE.
Rather, the program design document presented the following anticipated results
and performance indicator for the pilot project:
o Anticipated Result: Increased implementation by Canadian municipalities of
specific actions to reduce emissions from municipal on-road HDD vehicles.
o Performance Indicator: Population-weighted count of municipalities
that participate or report through FCM that they are implementing
greening fleet initiatives consistent with Project 1.
Although this level of detail was considered appropriate by Treasury Board to report
on this initiative, no performance information on the defined indicator has yet been
compiled. However, the FCM is conducting a final survey of all participants within its
Enviro-Fleets Pilot Project, which will produce the required performance information
for this indicator.
The work conducted by the FCM under the PPRE had a number of defined
performance measures to track outputs and outcomes. These were reported on
regularly within its quarterly reports. In addition, the FCM is planning to conduct an
end-of-pilot survey with municipalities to better assess the impact of the pilot project.
This information was not available for the evaluation but is expected by the end of
March 2011. The performance information provided by the FCM thus far was
assessed to be reliable, timely and relevant to the project.
The other initiatives funded under the PPRE (i.e., CAP, MSC, Bronson Consulting
study) did not have specific performance measures, but were required to submit
Environment Canada
20
Audit and Evaluation Branch
Evaluation of the PPRE
deliverables as part of their contracts. Interviews with program staff confirmed that
all deliverables were completed appropriately and in accordance with the terms and
conditions presented in the contracts between Environment Canada and those who
received PPRE funding.
Although performance information was collected and reported by the FCM, and
contract deliverables were produced as planned, there is no evidence of formal
internal reporting mechanisms to senior management or other federal partners on
the status of the pilot project, including all of its activities (i.e., those completed by
the FCM, CAP, MSC and Bronson Consulting). Program managers indicated that
they shared the PPRE outputs and results with management and partners informally
through meetings/briefings or by distributing the final deliverables to relevant
colleagues.
4.6 Performance – Design and Delivery – Program Design
Evaluation Issues (and
related questions)
5. Is the program design
appropriate for achieving
expected program results?
Indicators
-
-
-
-
Plausible link between program
activities, outputs and intended
outcomes
Clearly defined and understood
governance structure, including
program processes, roles,
responsibilities and
accountabilities
Program resources/capacity
commensurate with expected
program results
Views on the appropriateness of
program activities, processes
and governance structures
Overall
Rating
Achieved
SUMMARY:
The evaluation found a plausible link between program activities, outputs and
expected results for the pilot project. There was a clearly defined, understood and
appropriate program design and governance structure for managing projects,
including their deliverables, timing, funding, and reporting requirements. In addition,
program resources and the capacity of internal staff and funded partners were
commensurate with the expected results. Therefore, the overall rating has been
assessed as Achieved.
The program design document is the foundation of the PPRE design and delivery. It
clearly lays out the rationale; anticipated results and performance indicator;
activities, deliverables, timelines and funding; and the management of the projects
and requirements for evaluation. The first year of the pilot was spent consulting with
Environment Canada
21
Audit and Evaluation Branch
Evaluation of the PPRE
partners and determining the most appropriate activities to be undertaken, not only
to meet the goals but to fill necessary gaps. The program design document was
produced after this first year. Interviews indicated that it was important to take this
time to investigate what had been done, what was being done by others, and what
was needed to ensure the PPRE could provide valuable outputs.
As a result, the PPRE was appropriately designed to use the available resources
and capacity of funded partners, as expert delivery agents, to achieve the intended
results. Although no logic model was developed, the link between the planned
activities, outputs and intended outcomes was plausible.
Although there was little formal management or governance structure for the pilot,
this is seen as appropriate given the small amount of funds, the oversight provided
by the Strategic Transportation and Policy and Analysis Section, and the controls put
in place to effectively manage the funded partners (e.g., via contracts or contribution
agreements).
4.7
Performance – Design and Delivery – Barriers
Evaluation Issues (and
related questions)
6. Are there any barriers or
challenges that could affect
the success of this project if it
were implemented nationally
or integrated into an existing
national project?
Indicators
-
Views of stakeholders and
project managers
Evidence of documented barriers
or challenges
Evidence of solutions or
approaches to addressing
barriers or challenges
Overall
Rating
Achieved
SUMMARY:
The evaluation found that the key barriers to implementing a national program to
address HDD emissions resulting from municipal and school bus fleets are: a)
funding; and b) ability to work with all components of a municipal / school bus fleet
(sometimes composed of both public and private vehicles). There were specific
challenges identified in component pieces of the pilot that could be addressed by a
broader, national strategy, but some elements, such as determining the amount of
GHGs produced by HDD vehicles within a municipal / school bus fleet, are difficult to
determine due to the heterogeneous composition of municipal / school bus fleets. As
the barriers were clearly identified, and some recommendations for how to address
them were included in component reports, the overall rating has been assessed as
Achieved.
Two reports produced from the PPRE identified specific challenges that should be
considered in developing a broader program to reduce emissions from on-road
municipal and school bus fleets. The Bronson Consulting report noted that any clean
air program for school buses should be part of a comprehensive school bus health
and safety program in order to achieve high success rates, and should be linked with
Environment Canada
22
Audit and Evaluation Branch
Evaluation of the PPRE
a larger national strategy. Implementation challenges included operators’ knowledge
gaps regarding emission issues, and funding constraints. The report noted that
program delivery agents from various sectors identified that the optimal role for the
Government of Canada in a national clean air for school buses program is that of
facilitator rather than being the driving force.
The MSC report on school bus retrofit programs noted that the key barrier is funding,
and that another barrier is the reluctance of some school bus companies to consider
diesel retrofit technologies for their buses due to time constraints and, in some
cases, confidence in specific technologies. Some operators believe that the
availability of new technologies, and legislation requiring the transport sector to
reduce diesel emissions from school buses over the next several years, will
significantly reduce air pollutants and that other efforts are therefore not necessary.
All stakeholders interviewed believed that funding was the key constraint in rolling
out a more comprehensive national program. In addition, the variety of players that
make up municipal / school bus fleets (e.g., vehicles owned and operated by
municipalities vs. vehicles subcontracted by municipalities), along with the variety of
information sources and technologies that currently exist, were seen as barriers to
action. Busy fleet managers need one integrated and easy-to-access source of
information to facilitate adoption. Also, multiple jurisdictions need to be engaged to
establish a longer-term and more comprehensive approach.
4.8
Performance – Design and Delivery – Communications
Evaluation Issues (and
related questions)
7. Have the results of the pilot
project been communicated
effectively? How could
communications be
improved?
Indicators
-
-
Overall
Rating
Views of project managers,
Achieved
partners and recipients
Extent to which planned
communications activities have
been implemented as intended
Evidence of information sharing,
distribution of project publications
and reports, etc. to public fleet
operations and other fleets
SUMMARY:
The evaluation found that there was extensive communication and information
sharing with the target audience (municipal fleet operators) to promote and facilitate
emission reduction actions, particularly as part of the FCM project, with FCM leading
the majority of the communication and information-sharing activities. Therefore, the
overall rating has been assessed as Achieved. It was noted, however, that
information sharing within the federal community was informal and at times
inconsistent.
Environment Canada
23
Audit and Evaluation Branch
Evaluation of the PPRE
Table 8 summarizes how the FCM and CAP have shared information with
organizations.
Table 8: Evidence of Information Sharing by FCM and CAP as part of the PPRE
Delivery Organization and Information Shared
FCM
Number of surveys completed by municipal fleets
Number
Number of inquiries received from municipalities seeking
support on municipal fleets
50
Number of participants attending the webinars and workshops
207*
Number of municipal contacts in the network that was
established
324
Number of contacts on the list generated by the FCM’s
ClearVantage that were sent information on resources and
educational opportunities**
1300
Number of resources guides distributed
60
Number of municipalities responding to impact survey planned
for March 2011
CAP
Number of organizations to which the report was sent directly
Figures not yet
available
Number of views on the CAP web page dedicated to the
school bus project (e.g., views of fact sheets)
1146 page views***
21
10
* 140 municipal representatives and 67 non-municipal representatives. Does not include participants who attended
the FCM’s EnviroFleets Workshop, who were not registered; estimated at 50 participants.
** Information was also sent to subscribers to the FCM weekly newsletter and Partners for Climate Protection
members.
*** 818 views from unique IP addresses were reported.
In addition, Environment Canada indicated that the Department distributed project
outputs (e.g., Bronson Consulting report, Clean Air Agenda’s Clean Air Partnership
Report) to other federal partners (Transport Canada and NRCan) as a means of
sharing information. However, some interviewees stated that they had not been
informed of some deliverables, such as the program design document or FCM
publications such as the best practices guide.
4.9
Performance – Design and Delivery – Best Practices
Evaluation Issues (and
related questions)
8. What are the best practices
and lessons learned as a
result of the project?
Indicators
-
Environment Canada
Identified lessons learned and
shared best practices among
partners
Identified strengths and
weaknesses
Views on strengths / best
Overall
Rating
Achieved
24
Audit and Evaluation Branch
Evaluation Issues (and
related questions)
Evaluation of the PPRE
Indicators
Overall
Rating
practices, weaknesses and
lessons learned
SUMMARY:
The evaluation found that best practices and lessons learned were provided as part
of the component pieces of the pilot. A best practices guide was produced for
municipal fleet managers, and advice was provided for addressing emissions from
school buses. Therefore, the overall rating has been assessed as Achieved.
Elements of the pilot project identified best practices, lessons learned and/or
challenges as a result of the pilot activities. The FCM produced a best practices
guide that aims to help municipal fleet managers find cost-effective ways to reduce
emissions from heavy-duty vehicles. The guide focuses on best practices relative to
overall fuel use (e.g., life cycle management, idle-reduction technologies, etc.). The
FCM also produced a resources guide to help fleet managers access the resources
they need to improve their fleet efficiency. The CAP, MSC and Bronson reports
completed under the pilot provided advice and recommendations for moving forward
in addressing CAC emissions from school buses.
In terms of moving forward from this pilot, a summary of the best practices and
lessons learned is provided in Section 6 of this evaluation.
Interviewees highlighted that it is important to take time at the beginning of a new
pilot project such as this to investigate what others are doing, and what is needed, to
ensure the pilot/program meets a clear need and builds on existing programs and
initiatives. Many indicated that pilot projects such as this are important for
demonstrating what can be done with a finite amount of resources.
4.10 Performance – Effectiveness – Outputs and Outcomes
Evaluation Issues (and
related questions)
9. To what extent have the
intended outputs and
outcomes been achieved as a
result of the project?
Indicators
-
-
-
Environment Canada
Views of project managers,
partners and recipients on
achievement of outputs and
outcomes
Analysis of available
performance data on the
project’s expected results and
performance indicator
Population-weighted count of
municipalities through FCM that
report implementing green fleet
initiatives
Overall
Rating
Progress
Made;
Attention
Needed
25
Audit and Evaluation Branch
Evaluation of the PPRE
Evaluation Issues (and
Indicators
Overall
related questions)
Rating
SUMMARY:
The evaluation found that all but one of the planned outputs have been achieved
under the pilot project (the FCM did not obtain sufficient responses to establish a
baseline of municipal HDD fleets across Canada). There was a consensus from the
evidence and interviews that the pilot project achieved its intent. As the pilot recently
ended, it is too early to determine the extent to which this intended outcome has
been achieved, although the evaluation did find some evidence to suggest that
progress is being made. Information on the performance indicator
“population-weighted count of municipalities through FCM that report implementing
green fleet initiatives” is not yet available, although it is anticipated that this
information will be received through a final assessment survey to be carried out by
the FCM in March 2011. Therefore, the overall rating has been assessed as
Progress Made; Attention Needed.
Table 9 (presented on the next page) summarizes the planned and delivered outputs
under the PPRE pilot project. It is evident from this table that nearly all of the
planned outputs were delivered.
Table 9: Planned and Delivered Outputs for the PPRE
Planned Outputs
FCM
Establish partnerships required to deliver an effective program that will meet
the goal. Examples include municipal fleet managers’ networks, federal and
provincial agencies, environmental NGOs and private sector organizations.
Create a baseline of municipal heavy-duty fleet within Canada including a
fleet profile.
Overview of best practices and identification of barriers for greening fleet
initiatives.
Identify and create an online resources guide for municipal fleet managers
that will provide information on best practices available to achieve emissions
reductions.
Promote the resources guide and other products and services.
Deliver two webinars and four education and training workshops in
collaboration with partners, using resources identified to increase the
capacity of municipal fleet managers to act.
Create a resources guide for reference and use by municipal fleet managers.
Develop a business case for emissions reductions initiatives in heavy-duty
fleets, for inclusion in the resources guide.
CAP
Develop a brief report that:
a. outlines steps taken to collect information on the demographics of school
buses in Ontario, and summarizes the data collected;
b. outlines the steps taken to determine familiarity with, and action on, the
2005 Ontario Public Health Association report School Buses, Air
Pollution & Children’s Health and/or its findings, among key staff within
school boards, school bus operators, the public health sector, and health
associations; and summarizes the findings;
c. outlines the steps taken to research the language being used in school
board policies and/or contracts related to school bus emissions in
Ontario, British Columbia and Nova Scotia, and summarizes the findings;
d. includes model language that can be incorporated into school board
Environment Canada
Delivered Outputs

X







26
Audit and Evaluation Branch
Evaluation of the PPRE
Planned Outputs
policies and/or contracts, related to school bus retrofits and other
emission reduction options, for use in Ontario and across Canada.
Develop a four-page fact sheet on school bus retrofits, directed at school
board staff and to be disseminated across Ontario.
Develop a four-page fact sheet on school bus retrofits, directed at school bus
operators and to be disseminated across Ontario.
My Sustainable Canada
Investigate diesel retrofit technologies to determine which technology is most
suitable for use in school buses, and test it in a pilot program for school
buses in a community in northern or eastern Ontario.
Bronson Consulting
Develop a report that features a review of school bus characteristics and
North American school-bus-focused clean air programs, provides an
understanding of how clean air programs could affect the Canadian school
bus sector, and summarizes the key considerations from these programs.
Delivered Outputs




The anticipated result/outcome for the first initiative under the pilot project was:
“Increased implementation by Canadian municipalities of specific actions to reduce
emissions from municipal on-road heavy-duty diesel vehicles,” with the following
performance indicator: “Population-weighted count of municipalities that participate
or report through FCM that they are implementing greening fleet initiatives consistent
with the pilot project (i.e. first initiative).” As the pilot has just ended, the degree to
which the anticipated outcome has been achieved is not yet documented. The FCM
will be conducting an end-of-pilot survey of municipalities that is expected to provide
information to address this outcome. In addition, future FCM reports (expected circa
2012-2013) will provide information on how municipalities took actions with their onroad HDD vehicles (e.g., new applications to the FCM’s Green Municipal Fund and
its Partners for Climate Protection annual measures report on actions taken related
to the HDD fleet).
The following information indicates that progress has been made toward achieving
the stated outcome:
 The FCM indicated that it has increased the capacity of municipal fleet
managers and other officials to adopt measures that reduce GHGs and
CACs, reduce fuel consumption, and reduce costs in municipal fleets through
training and knowledge resources. Evaluation of their webinars indicated that
a vast majority of participants felt that the webinar met or exceeded their
expectations and increased their knowledge in the topic area, and agreed that
they would apply the discussed ideas, tools and strategies to their work.
Evaluation of the FCM workshops indicated that all respondents agreed that
the sessions were worthwhile.
 The FCM reported that 39 fleet-related measures have been undertaken by
municipalities, with an aggregated amount of $8 million invested in greening
the fleet measures (this is part of the broader Partners for Climate Protection
program and is not directly attributable to the PPRE). In addition, the FCM’s
Green Municipal Fund has changed its criteria to allow for fuel switching and
Environment Canada
27
Audit and Evaluation Branch


Evaluation of the PPRE
retrofit projects in municipal fleets; it is anticipated that projects of this type
will increase based on the work initiated in the PPRE.
Information on best practices and resources has been widely distributed to
the target audience (as noted under Section 4.8) to increase awareness and
facilitate the implementation of actions to reduce emissions.
MSC implemented retrofits on 60 school buses and 6 garbage trucks in
southwestern Ontario. This resulted in substantial emissions reductions from
these vehicles (approximately a 40% reduction of PM, 75% reduction of
hydrocarbons, and 60% reduction of carbon monoxide).
4.11 Performance – Effectiveness – Intended Outcomes
Evaluation Issues (and
related questions)
10. Have there been any
unintended outcomes, either
positive or negative?
Indicators
-
Presence/absence of unintended
outcomes
Views on whether unintended
outcomes occurred
Overall
Rating
N/A
SUMMARY:
The evaluation found no negative unintended outcomes.
There was no evidence of unintended outcomes identified in the documentary
evidence. Interviewees noted the following two key unintended outcomes:
 increased understanding of the challenges facing municipal fleet managers in
implementing changes to address emissions; and
 enhanced relationships and networks between non-traditional partners (e.g.,
EC, NRCan and the FCM).
4.12 Performance – Efficiency
Evaluation Issues (and
related questions)
11. Has the project
undertaken its activities and
delivered products (e.g.,
technical studies) in the most
efficient manner?
Indicators
-
-
-
Environment Canada
Analysis of project operational
costs in relation to the production
of outputs
Views on whether the cost of
producing project outputs is as
low as possible
Evidence of / views on whether
there are alternative, more
efficient ways of delivering
project activities and outputs
Overall
Rating
Achieved
28
Audit and Evaluation Branch
Evaluation of the PPRE
Evaluation Issues (and
Indicators
Overall
related questions)
Rating
SUMMARY:
The evaluation found evidence that Environment Canada partnered with other
organizations to achieve the program’s objectives in an efficient manner while
maximizing limited resources to manage the program within the Department and
leveraging funding from partner organizations (i.e., the FCM).
Therefore, the overall rating has been assessed as Achieved.
Table 10 presents the expenditures of the PPRE against original allocations. These
expenditure figures were provided by program staff.
Table 10: PPRE Allocations and Expenditures
2008-09
2009-10
2010-11
Allocated Spent
Allocated Spent
Allocated
Spent*
Reducing Emissions from Canada’s On-Road HDD Municipal Fleet
FTEs
0.5 FTE 0.5 FTE
1 FTE
1 FTE
1 FTE
1 FTE
O&M Allocation
$52,770
$73,009
$73,009
O&M Expenditure:
Program Design
$52,000
--Bronson Consulting
-$25,000
-CAP
--$25,000
MSC
-$42,000
-Other**
-$6,000
$23,000
Evaluation
--$25,000
TOTAL O&M
-G&C Allocation
$148,664
$223,664***
G&C Expenditure:
FCM
$32,000
$140,000
MSC
$83,664
TOTAL G&C
Total 2008-11
Allocated
Spent
$198,788
$52,000
$25,000
$25,000
$42,000
$29,000
$25,000
$198,000
$198,788
$372,328
$172,000
$83,664
$255,664
$372,328
* Projected expenditures; in some cases figures were rounded by program staff.
** Includes costs associated with program travel, translation, and a co-op student.
*** Includes $75,000 that was reallocated to the first initiative.
The financial data provided highlight that $116,664 in G&C funding available for the
pilot project’s first initiative in 2009-10 was not spent. Program staff explained that
this funding was not provided to Environment Canada until late in the fiscal year
2009-10 (released in September), and therefore the program partner (FCM) only had
a limited part of the year to spend the contributions effectively. As a result, some of
the allocated funds available to the pilot project were lapsed.
In terms of comparable federal government approaches to reducing diesel
emissions, the literature highlighted that the United States maintains a more
comprehensive approach to delivering its objectives concerning diesel emissions,
through the U.S. EPA’s National Clean Diesel Campaign (NCDC), which provides
access to approximately $200 million (between 2007 and 2011) in funding to
manufacturers, fleet operators, air quality professionals, environmental and
community organizations, and state and local officials, to reduce diesel emissions
(see Annex 2 for details). In the absence of such a large funding envelope to
Environment Canada
29
Audit and Evaluation Branch
Evaluation of the PPRE
address diesel emissions in Canada, the PPRE utilized partnerships with not-forprofit and/or expert organizations to achieve the project objectives.
The evaluation highlights that the PPRE was implemented by funding not-for-profit
and/or expert organizations to undertake specific studies and initiatives. The funded
partners were chosen based on their expertise in the required area and their ability
to deliver work effectively and efficiently. For example, the FCM was selected as an
appropriate organization to work with based on their ability to access municipalities
and fleet managers through the FCM’s existing networks. The funds provided to
these groups were relatively small (ranging from $25,000–$170,000) and all the
deliverables were produced on time and met the Department’s expectations. The
project was managed efficiently by one FTE in Environment Canada and was
designed to complement existing initiatives.
In addition, some leverage was achieved through the FCM initiative (e.g., $37,000
was provided to the project by the Green Municipal Fund, which resulted in a 14%
increase in the original $230,000 in G&Cs allocated for the project). Leverage was
also achieved by the FCM collaborating with Fleetwiser to co-promote projects, with
FPInnovations, and with Fleet Challenge Ontario (FCO) to deliver the Ontario
workshop (FCO contributed $1,000).
Interviewees noted that the funded projects and the work managed by Environment
Canada were undertaken in an efficient and collaborative manner. For these
reasons, the PPRE was assessed to be efficient.
4.13 Performance – Economy
Evaluation Issues (and
related questions)
12. Has the project achieved,
or is it on track to achieving,
its intended outcomes in the
most economical manner?
Indicators
-
-
Views on whether good value is
being obtained with respect to
the use of public funds
Evidence of / views on whether
there are alternative program
models that would achieve the
same expected outcomes at a
lower cost
Overall
Rating
Achieved
SUMMARY:
The evaluation found that the project is on track to achieving its intended outcomes
in an economical manner. All stakeholders viewed the PPRE as a good use of public
funds, and believed that the delivery approach was economical. Therefore, the
overall rating has been assessed as Achieved.
As noted above, the delivery approach was seen as economical, as it used expert
non-profit groups that completed the required tasks for a relatively low cost and in a
timely manner. The FCM project was able to reach a wide audience across Canada,
to ensure a strong and equitable reach.
Environment Canada
30
Audit and Evaluation Branch
Evaluation of the PPRE
Interviewees believed that the federal government, on its own, could not have
completed the work in the time required and for the available budget. For instance, it
would not have been able to reach the target municipalities economically without the
FCM, which had a pre-existing national network and channel in place. The FCM also
had established credibility with the target audience, facilitating uptake and
engagement. In addition, the FCM built on existing regional initiatives in British
Columbia, Quebec and Ontario, and leveraged its broader networks and resources
through the Partners for Climate Protection program and Green Municipal Fund.
One alternative suggested for enhancing the economy and reach of the FCM
workshops was to implement a train-the-trainer approach rather than hold individual
workshops.
As well, alternatives could have been used to conduct the studies completed by
MSC, CAP and Bronson Consulting, but these would likely not have been as
economical, as these groups had successfully and recently conducted previous
related studies and were able to build on this work in an economical and timely
fashion. One interviewee noted that an alternative such as funding municipalities to
do outreach or retrofits may be problematic, as they may not have the same drivers
for action (they may be more concerned about fuel savings than environmental
benefits).
The literature review (see Annex 2) indicated that the PPRE’s delivery is consistent
with the delivery of regional programs that are being implemented in Ontario, British
Columbia and Nova Scotia. Key highlights include:



other capacity-building programs maintain a similar temporal scope of 2–4
years;
implementation by a non-governmental partner is the preferred delivery
vehicle for capacity-building programs; and
the target audience for capacity-building programs continues to be
municipalities, although in at least one case a program is working directly with
the commercial sector.
The literature review also demonstrated that there are multiple options for
developing capacity with the goal of reducing HDD emissions, including the
following:
 Conducting audits of municipal fleets, which focus on identifying retrofit
opportunities by conducting an on-site audit of a municipal fleet.
 Delivering technical workshops to fleet managers, which educate the
managers on fuel efficiency or how to green their heavy-duty vehicle fleet.
 Providing networking opportunities, which include enabling networking
opportunities for fleet managers through online forums, workshops,
informal/formal meetings, and so forth.
Environment Canada
31
Audit and Evaluation Branch
Evaluation of the PPRE
 Developing guidelines and toolkits for fleet managers (on a range of topics),
such as tools for identifying appropriate technologies, and information or data
on best practices / approaches for reducing HDD emissions.
 Product testing, which focuses on supporting demonstration projects for
emission reduction technologies.
Environment Canada
32
Audit and Evaluation Branch
Evaluation of the PPRE
5. Conclusions
The evaluation found that the pilot project was relevant and addressed a clear need
to reduce emissions from HDD municipal fleet vehicles. Furthermore, the PPRE’s
objectives were found to be clearly aligned with current federal government priorities
and Environment Canada’s strategic outcomes to take action on climate change and
reduce air emissions, as well as complementary to other federal programs such as
CARA and ecoFREIGHT. By maintaining an exclusive focus on reducing GHGs and
CACs from on-road public HDD vehicles, the PRRE addressed a unique gap.
However, it is similar to initiatives under NRCan’s FleetSmart program.
Recognizing that air quality management is a shared responsibility between multiple
government jurisdictions and stakeholders engaging in complementary measures
like the PPRE pilot was appropriate for the federal government. Beyond the pilot
stage, responsibilities for building capacity and reducing emissions from the in-use
municipal fleet are also strongly aligned with provincial and municipal roles and
responsibilities.
The evaluation found a logical and appropriate program design for the pilot project,
one that was built on an identified need and utilized organizations with relevant
expertise to assist in implementation. There was evidence of a clearly defined and
understood structure for managing projects, including a clear articulation of
expectations for deliverables and reporting requirements to be prepared by
organizations who received PPRE funding. The pilot was successful in achieving its
intended outputs: all but one of the planned outputs for the pilot project was
completed. Regarding the planned output that was not completed, Environment
Canada hoped that the FCM could help establish a national baseline of the
municipal HDD fleet across Canada by designing and distributing a survey to
municipal fleet managers, but the response to this survey was too low to be
quantified.
The evaluation found evidence that Environment Canada partnered with existing
organizations to achieve the program’s objectives in an efficient manner, and to
leverage a small amount of funding (i.e., from the FCM) for project activities.
Furthermore, all stakeholders viewed the PPRE as a good use of public funds, and
that the delivery approach taken was economical.
The expenditure information provided for this evaluation indicates that the majority of
PPRE funds were spent through internal departmental activities, or by distributing
PPRE funds (through contracts or G&C agreements) to consultants and nongovernmental organizations.
Despite developing a specific performance indicator for this project, the evaluation
did not find evidence of formal reporting against this indicator, nor was a formal
internal reporting mechanism to senior management or federal partners developed
to communicate results of the project and report on its performance. Rather,
Environment Canada
33
Audit and Evaluation Branch
Evaluation of the PPRE
performance information was provided by funded partners in a timely manner, and
focused on outputs. As the pilot has come to a close, performance data are not
required to support future decisions, but it is relevant to determine whether the pilot
has achieved its expected outcome. Environment Canada is expecting that the
results of an FCM follow-up survey, to be made available in the near future, will
allow the department to report on the project’s performance indicator.
Environment Canada
34
Audit and Evaluation Branch
Evaluation of the PPRE
6. Lessons Learned
The evaluation identified a number of lessons learned, and suggestions to improve
the delivery of similar projects that are complementary to regulatory development.
Key lessons learned from the PPRE:



Allocating an appropriate amount of time for designing a
program/project can lead to relevant and valuable results. The evaluation
noted that the first year of program delivery was dedicated to designing a
program that was relevant and met a demonstrable need that was not
duplicative of other federal efforts. In this case, Environment Canada
consulted with relevant federal partners and commissioned a study to identify
an appropriate gap to be addressed by the pilot project.
Partnering with organizations that maintain expertise and existing
networks can lead to effective results. For this project, Environment
Canada elected to work with NGOs that maintained existing networks and
had a comprehensive understanding of municipal fleets across Canada.
Smart decision making around who to partner with can not only enhance
program efficiencies by creating immediate access to existing networks, it can
also provide greater assurance that project deliverables will be useful and
credible and that they can effectively achieve results.
When provided with the appropriate tools and funding, Canadian
municipalities can engage in actions to reduce emissions from their
fleet vehicles. According to the FCM, up to 45% of Canada’s GHG emissions
are under the direct/indirect control of municipal governments, and municipal
fleets are responsible for 3–5% of total municipal GHG emissions as well as
significant local health impacts. The PPRE has shown that by building the
capacity of municipalities, providing tools, and providing assistance with
accessing funds, emissions reductions can be supported and achieved. One
of the key drivers for municipalities is the cost efficiencies that can result from
fuel savings, along with emissions reductions.
Suggestions for future projects:

A one-window approach, providing information and resources to reduce
emissions from all types of vehicles in the municipal fleet, would be
beneficial. There is merit in addressing the emissions from all vehicle types
that are present in municipal fleets more broadly, rather than focusing on
HDD. Fleet managers deal with multiple types of vehicles and make
integrated decisions that can best be informed and supported by a more
Environment Canada
35
Audit and Evaluation Branch





Evaluation of the PPRE
comprehensive approach. Developing a one-window approach / portal that
consolidates multiple information sources and provides information that is
applicable to emissions from all vehicle types would be beneficial to fleet
managers.
Long-term funding is required to ensure action is taken to reduce fleet
emissions. The pilot project has increased the level of awareness of a group
of municipal fleet managers and provided them with resources for reducing
the emissions from their municipal fleet. However, municipalities continue to
be forced to do more with less and an enhanced understanding alone is not
sufficient to ensure that action is taken to reduce emissions. This pilot project
has demonstrated that long-term funding sources from other sources should
be identified.
The Government of Canada can play an important role in identifying
innovative technologies for reducing vehicle emissions. The Government
of Canada can play an important role in testing technologies for reducing onroad vehicle emissions. The PPRE-funded project in Ontario identified retrofit
technology that has a proven ability to reduce emissions produced by school
buses.
Environment Canada is best placed as an enabler for reducing vehicle
emissions. The evaluation highlighted that this was a unique opportunity for
Environment Canada to lead a pilot project that addressed an identified gap,
was complementary to HDD regulations, and addressed both GHGs and air
pollutants. However, other federal departments such as NRCan are now
better placed to lead comprehensive capacity-building programs in this area,
because of their mandate and their emerging work under programs such as
FleetSmart.
A national emissions baseline for municipal fleets is required. The pilot
project was unable to develop a national emissions baseline for municipal
fleets. In order to target cost-effective emission reduction activities and
measure change with municipal fleets, a baseline is required. However, the
municipal fleet operating environment includes multiple players, with some
municipalities working with partners/third parties (e.g., private fleets) to
provide services to their jurisdiction. Therefore, calculating an estimate of the
emissions produced by municipal fleet vehicles across Canada requires an indepth study of municipal fleet management and operating systems. Although
emissions data are available for the 12 largest urban areas in Canada, little is
known about the level of emissions in the remaining municipalities.
The impact of capacity-building / outreach programs is maximized when
delivered jointly with a suite of like-minded programs, including
regulations/standards. Although education and capacity building are
important tools to help municipalities take action to reduce their emissions, a
suite of complementary and reinforcing initiatives, including regulations,
Environment Canada
36
Audit and Evaluation Branch
Evaluation of the PPRE
provides them with the push and pull required for taking substantive action
within their jurisdictions.
Environment Canada
37
Audit and Evaluation Branch
Evaluation of the PPRE
Annex 1. Bibliography of Documents Reviewed
Bronson Consulting Group. Contract with Environment Canada, December 2009.
Bronson Consulting Group. School Bus Scoping Study: Fleet Statistics and Clean
Air Programs, March 2010.
Clean Air Partnership. Contract with Environment Canada, June 2010.
Clean Air Partnership. Healthy School Buses for School Bus Operators, September
2010.
Clean Air Partnership. Healthy School Buses: School Board Factsheet, September
2010.
Clean Air Partnership. School Buses, Air Pollution and Children’s Health: Follow-up
Report, October 2010.
Environment Canada. Clean Diesel Workshop: Solutions for Municipalities,
March 2008.
Environment Canada. National Clean Diesel Campaign Workshop Agenda, June
2008.
Environment Canada. A Guide to Understanding the Canadian Environmental
Protection Act, 1999, 2009. Via: www.ec.gc.ca/lcpecepa/default.asp?lang=En&n=E00B5BD8-1&offset=2&toc=show
Environment Canada. SPP Program Design, June 2009.
Environment Canada. Contribution Agreement between FCM and EC, September
2009.
Environment Canada. Contribution Agreement Amendment (between FCM and EC),
May 2010.
Environment Canada. Reports on Plans and Priorities, 2008-2009, 2009-2010,
2010-2011.
Via: www.tbs-sct.gc.ca/rpp/index-eng.asp
Federation of Canadian Municipalities. Proposal to Environment Canada, August
2009.
Federation of Canadian Municipalities. Enviro-Fleet Progress Report 2009-2010,
April 2010.
Federation of Canadian Municipalities. Green Your Municipal Fleet (Enviro-Fleet
brochure), May 2010.
Environment Canada
i
Audit and Evaluation Branch
Evaluation of the PPRE
Federation of Canadian Municipalities. Enviro-Fleet Activities Report First Quarter,
July 2010.
Federation of Canadian Municipalities. Enviro-Fleet Activities Report Second
Quarter, October 2010.
Federation of Canadian Municipalities. Enviro-Fleet: Reducing municipal heavy-duty
vehicle emissions – A guide to best practices, November 2010.
Federation of Canadian Municipalities. Enviro-Fleet Activities Report Third Quarter,
January 2011.
Federation of Canadian Municipalities. Feedback forms for the Quebec, Ontario and
BC Workshops, 2009-2010.
Fleet Challenge Ontario. Municipal Fleet Review Program website
(www.fleetchallenge.ca/content/review-program).
Fleetwiser. Fleetwiser website (www.Fleetwiser.ca).
Government of British Columbia. BC Air Action Plan. Via:
www.bcairsmart.ca/transportation/heavyduty.html.
Government of British Columbia. BC Air Quality – Levels of Government Involved.
Via: www.env.gov.bc.ca/epd/bcairquality/regulatory/levels-of-government.html.
Government of Canada. Speech from the Throne, 2008. Via:
www.speech.gc.ca/eng/media.asp?id=1383.
Green Fleets BC. Green Fleets BC website (www.greenfleetsbc.com).
My Sustainable Canada. Diesel School Bus Retrofit Pilot Project – Report, March
2010.
My Sustainable Canada. Proposal to Environment Canada, June 2010.
Natural Resources Canada. FleetSmart website
(http://fleetsmart.nrcan.gc.ca/index.cfm?fuseaction=fleetsmart.home&attr=16).
Natural Resources Canada. Evaluation of the Transportation Sub-Sub Activity, 2010.
Via: www.nrcan-rncan.gc.ca/evaluation/reprap/2010/e20100706eng.php?PHPSESSID=f080598211f8d16c01c04fc1d6ef428b.
Transport Canada. ecoFREIGHT website (www.tc.gc.ca/eng/programs/environmentecofreight-menu-1131.htm).
Transport Canada. Sustainable Development Strategy, 2001-03. Via:
www.tc.gc.ca/eng/policy/acs-sd-strategy0103-challenges5-673.htm.
Environment Canada
ii
Audit and Evaluation Branch
Evaluation of the PPRE
United States Environmental Protection Agency. National Clean Diesel Campaign
website (www.epa.gov/cleandiesel/basicinfo.htm).
Environment Canada
iii
Audit and Evaluation Branch
Evaluation of the PPRE
Annex 2: Literature Review
A literature review was conducted by the Evaluation Team to help assess the
broader relevance and success of the PPRE compared to other similar or related
initiatives in the United States and Canada, and to identify alternative delivery
mechanisms or best practices to be considered for similar programs that may be
delivered in the future. The literature review was conducted alongside the document
review and included a review of Internet-based information.
Two areas were explored:
1. Comparison of the Government of Canada’s approach to addressing
emissions from HDD vehicles to the approach delivered by the U.S. federal
administration:
o A review of the U.S. NCDC and the Government of Canada’s
ecoFREIGHT and FleetSmart programs was completed, to provide a
more general understanding of the approaches taken by both countries
to address vehicle emissions.
2. Comparison of the PPRE to other related capacity-building programs in
Canada:
o The Evaluation Team, in conjunction with the Project Authority,
selected three capacity-building programs that aim to reduce vehicle
emissions: (1) Fleet Challenge Ontario (Municipal Fleet Review); (2)
BC Air Action Plan (Action #10); and (3) the Fleetwiser Program. The
criteria used to select these programs included:
o comparability of program’s scope and objectives to the PPRE
objectives; and
o comparable program structure and delivery mechanisms.
1. Comparison of the Government of Canada’s approach to addressing
emissions from HDD vehicles to the approach delivered by the U.S. federal
administration
The NCDC was initiated in 2000 by the U.S. EPA to promote clean air strategies, by
working with manufacturers, fleet operators, air quality professionals, environmental
and community organizations, and state and local officials to reduce diesel
emissions. The NCDC focuses on reducing emissions from on-road and off-road
diesel engines, through regulations, information-sharing mechanisms, regional clean
diesel collaboratives, sector-specific programs and clean diesel funding. It is
expected that the NCDC will award $200M USD in funding from 2007 to the end of
September 2011. Among other programs, the NCDC administers a Clean School
Bus program, which aims to reduce children's exposure to diesel exhaust and the
amount of air pollution created by diesel school buses. Clean School Bus USA
brings together partners from business, education, transportation, and public health
Environment Canada
iv
Audit and Evaluation Branch
Evaluation of the PPRE
organizations, focusing on anti-idling efforts, retrofitting old buses and replacing the
oldest ones with new, less-polluting buses.
In Canada, ecoFREIGHT and FleetSmart are the two major programs with a
mandate to reduce emissions generated by the transportation sector. ecoFREIGHT,
established in 2007 and managed by Transport Canada, supports the freight
transportation industry toward a greater uptake of technologies and practices that
reduce fuel consumption, CACs and GHGs. The program was expected to receive
$61 million from 2007 through to 2011. FleetSmart, a part of the ecoENERGY for
Fleets suite administered by NRCan, seeks to provide practical advice and
consultation on how energy-efficient vehicles and business practices can reduce
fleet operating costs, improve productivity and increase competitiveness.
Table 11 provides a high-level overview of NCDC, ecoFREIGHT and FleetSmart. An
overview of the PPRE has been included to highlight similarities and differences
between the programs.
This analysis has highlighted that the United States has a comprehensive approach
to reducing diesel emissions through the NCDC. The NCDC has significant funding
($200 million in clean diesel funding alone from 2007 to 2011) and is an integrated,
umbrella system to channelling initiatives aimed at reducing diesel emissions. The
target audience for this initiative is heterogeneous and includes manufacturers, fleet
operators, air quality professionals, environmental and community organizations,
and state and local officials, which allows the initiative to address a variety of
systems and vehicle types, including on-road vehicles, existing and new off-road
vehicles, marine vessels, and locomotives. Similarly, the NCDC includes multiple
delivery mechanisms, including regulatory development, funding assistance, and
development of publications and materials to develop the capacity of vehicle
operators to reduce emissions caused by diesel vehicles.
The ecoFREIGHT and FleetSmart programs provide funding and capacity-building
activities to reduce emissions, but they do not focus exclusively on diesel emissions.
They each target a specific audience—the freight transportation industry for
ecoFREIGHT, and the commercial highway freight sector for FleetSmart—and
address off-road and on-road vehicles. Delivery mechanisms include capacitybuilding initiatives for both, but ecoFREIGHT also provides funding mechanisms and
incentives in addition to establishing voluntary agreements with the freight
transportation industry.
In addition to these programs, Canada has developed and will continue to develop a
series of regulations to reduce smog-forming air pollutant emissions from vehicles
and engines in alignment with the standards of the U.S. EPA. Currently, there are
regulations in place to reduce air pollutant emissions from passenger cars and lightduty trucks, heavy-duty vehicles, motorcycles, small marine engines such as
outboards and personal watercraft, recreational vehicles as well as construction and
agricultural equipment, and garden equipment such as lawnmowers and chainsaws.
Canada will continue to amend and update existing regulations to maintain
alignment with the U.S.
Canada addresses diesel emissions under CEPA 1999 through:
Environment Canada
v
Audit and Evaluation Branch
Evaluation of the PPRE

The Sulphur in Diesel Fuel Regulations which came into force in 2003 set
maximum limits for sulphur in on-road, off-road, rail (locomotive) and marine
(vessel) diesel fuels.

The On-Road Vehicle and Engine Emission Regulations which came into effect
in 2004, control emission of hydrocarbons, nitrogen oxides, carbon monoxide,
benzene, and a number of other toxic substances from the operation of on-road
vehicle engines. These regulations aligned Canadian emission standards with
those of the United States for light-duty passenger vehicles, light-duty trucks
(such as vans, pickup trucks and sport utility vehicles), heavy-duty vehicles (such
as heavy trucks and buses), and motorcycles.

The Off-Road Compression-Ignition Engine Emissions Regulations which came
into force in 2006 and introduced emission standards for diesel engines such as
those found in construction, agricultural and forestry machinery. The standards
were aligned with US EPA tier 2 & 3 emission standards. On February 12, 2011,
the proposed Regulations Amending the Off-Road Compression-Ignition Engine
Emissions Regulations were published in the Canada Gazette Part I. These
amendments will result in further reductions of smog-forming emissions from the
off-road diesel engine sector.
The Government of Canada has also recently announced regulations to reduce
greenhouse gas emissions from vehicles including:
 The Passenger Automobile and Light Truck Greenhouse Gas Emission
Regulations were published in Canada Gazette, Part II in October 2010 and
establish progressively stringent fleet average GHG emissions standards for new
vehicles sold in model years 2011 through 2016. These regulations are aligned
with those in the U.S. and apply to companies that manufacture new cars and
light trucks in Canada, or import these vehicles into Canada for the purpose of
sale. In October 2010, the Government of Canada published a Notice of Intent
in Canada Gazette, Part I, to develop standards for new passenger automobiles
and light trucks of model years 2017 to 2025, in coordination with the U.S. EPA.
 On May 21, 2010 Canada and the U.S. announced their respective intentions to
develop regulations to limit GHG emissions from new on-road heavy-duty
vehicles such as buses and combination tractors. The Canadian regulations
would be developed under CEPA 1999 and would align Canadian standards with
those of the U.S. for the 2014 and later model year vehicles.
Environment Canada
vi
Audit and Evaluation Branch
Evaluation of the PPRE
Table 11: Comparison of U.S. and Canadian Approaches to Reducing Emissions from HDD Vehicles
Overall objective
Funding
Lead organization
Delivery partner
Target audience
Types of vehicles
Activities
U.S. Approach
National Clean Diesel Campaign
To promote clean air strategies by working with
manufacturers, fleet operators, air quality
professionals, environmental and community
organizations, and state and local officials to
reduce diesel emissions
N/A
($200 million for clean diesel funding from 2007
to 2011)
U.S. EPA
U.S. EPA
Manufacturers, fleet operators, air quality
professionals, environmental and community
organizations, and state and local officials
On-road vehicles
Off-road vehicles
Large ocean vessels
Diesel engines (including locomotive and marine
engines)
 Regulations
 Information-sharing through website (e.g.,
tool kits, information on technologies, etc.)
 Regional clean diesel collaboratives (through
public-private partnerships)
 Sector-specific programs (clean school buses,
clean ports, clean construction, clean
agriculture, SmartWay Transport)
 Clean diesel funding, including:
o National Clean Diesel Funding
Assistance Program (50% of funding
dedicated to public fleets)
o National Clean Diesel Emerging
Technologies Program
o SmartWay Clean Diesel Finance
Environment Canada
FleetSmart (part of ecoENERGY for Fleets)
To offer free practical advice on how energyefficient vehicles and business practices can
reduce fleet operating costs, improve
productivity and increase competitiveness
Canadian Approach
ecoFREIGHT
To support the freight transportation industry
toward a greater uptake of technologies and
practices that reduce fuel consumption, CACs
and GHG emissions
N/A
$61 million
$1.2 million for 2008 to 2011
NRCan
NRCan
Transport Canada
Transport Canada
Commercial highway freight sector (commercial
and municipal fleets)
Freight transportation industry (trucking industry,
companies in the air, rail, road and marine
sectors, users of the freight system)
Off-road vehicles
On-road vehicles
Environment Canada
FCM
MSC
CAP
Municipalities
School bus operators and districts
On-road vehicles
 Capacity-building initiatives
o SmartDriver training
o Fuel Management 101 workshops
 Information-sharing through website
(fuel-saving information, tool kit, publications,
links, newsletter)
vii
 Six initiatives, including:
o Freight Technology Demonstration
Fund
o Freight Technology Incentives Program
o Marine Shore Power Program
o National Harmonization Initiative for the
Trucking Industry
o ecoFREIGHT Partnerships
o ecoENERGY for Fleets (NRCan
program)
 Voluntary agreements with freight industry
associations
 Information-sharing through website (case
studies, ecoFREIGHT announcements and
newsletters)
PPRE Pilot Project
To reduce emissions from Canada’s on-road
HDD municipal fleet by increasing knowledge,
awareness and supporting tools for
municipalities
On-road vehicles (heavy-duty only)
Diesel engines
 Capacity-building initiatives:
o Enviro-Fleets project with the FCM
(resources guide, best practices guide,
webinars and workshops for municipal
fleet managers)
o Southwest Ontario school bus and
garbage truck retrofit pilot
 Studies and information-collection activities:
o Scoping study on the HDD school bus
fleets within Canada
o Study on the number and age of school
buses operated by or for school boards
in Ontario
o Research on policies and/or contract
language that can be used to reduce
Audit and Evaluation Branch
o
Duration
Source
Program
State Clean Diesel Grant Program
Ongoing since 2000
http://www.epa.gov/cleandiesel/basicinfo.htm
Environment Canada
Evaluation of the PPRE
emissions of GHGs or air pollutants
from school buses
Ongoing since 1997
http://fleetsmart.nrcan.gc.ca/index.cfm?fuseactio
n=fleetsmart.home&attr=16
viii
2007 to 2011
http://www.tc.gc.ca/eng/programs/environmentecofreight-menu-1131.htm
2008 to 2011
Evaluation Work Plan
Documentation Review
Audit and Evaluation Branch
Evaluation of the PPRE
2. Comparison of the PPRE to other related capacity-building programs in
Canada
Fleet Challenge: Delivered by Fleet Challenge Ontario (FCO), an Ontario-based
NGO, the municipal fleet review program is funded by the Government of Ontario.
The objective of the program is to reduce GHG emissions from municipal fleets by
supporting green management decisions and demonstrating ways to improve
operations. The program has helped renew fleets in approximately 40 municipalities,
by supporting initiatives such as developing tool kits and guidelines for municipal
fleet operators, completing retrofit audits of on-site municipal fleets, and providing
networking opportunities to municipal fleet operators by delivering workshops and
hosting meetings. The program was launched as a pilot program in 2008 and is set
to continue until 2012.
B.C. Action Plan: This is part of the B.C. provincial government’s broader strategy to
lead the world in sustainable environmental management. The BC Air Action Plan
contains eight actions aimed at making heavy-duty vehicles cleaner. For example,
Action #10, “Green vehicle fleets: we’re making it happen,” pertains to Green Fleets
BC, a partnership initiative led by the Fraser Basin Council. The goal of the initiative
is to help owners of commercial and public sector vehicle fleets improve their fuel
efficiency and reduce emissions. Under this initiative, the Council has been a key
information hub for the latest on green technologies for private and public sector
fleets, including taxis, emergency vehicles, delivery vans and commercial freight
trucks. Green Fleets BC has also supported demonstration projects on trucking
technologies and on the use of biodiesel in British Columbia’s public sector and
commercial vehicle fleets, and sponsored a new network of fleet managers.
Fleetwiser: The Fleetwiser program in Nova Scotia is funded by NRCan and
delivered by Clean Nova Scotia, a regional NGO. The program is currently helping
four municipalities and one utility reduce their emissions and make environmentally
friendly choices, by providing participating fleet managers and operators with free
consultation, training and resources. These resources and training have included
Fleet Efficiency Tool Kits developed for fleet managers, operators and technicians,
and Fuel Management 101 workshops.
A summary of each program is provided in Table 12. In addition to these programs,
a brief overview of the PPRE has been included to highlight the similarities and
differences across these programs.
Environment Canada
ix
Audit and Evaluation Branch
Evaluation of the PPRE
Table 12: Comparison of the PPRE to other capacity-building programs in Canada
Funding (source and
amount)
Partners
Target audience
Fleet Challenge Ontario (Municipal Fleet
Review)
Supported by the Government of Ontario
(Ontario Ministry of Transportation and Ontario
Ministry of Finance)
$ - N/A
FCO (Canadian Energy Efficiency Alliance)
The FCO program has been made possible
through the Canadian Energy Efficiency
Alliance in partnership with the Ontario Ministry
of Transportation and the Ontario Ministry of
Finance, through the Strengthening Our
Partnerships initiative.
Ontario municipalities: 11 municipalities
selected to participate in 2009; 8 municipalities
in 2010; 10 municipalities in 2011
BC Air Action Plan
Clean Nova Scotia (Fleetwiser Program)
PPRE Pilot Project
Funded by the BC Ministry of Environment ($28.5 million
over three years to implement the plan, in partnership
with industry, communities and other levels of
government)
Funded by NRCan (ecoENERY for Fleets)
Environment Canada (through
Industry Canada’s SPP)
$1.2 million for 2008 to 2011
Action #10: Green Fleets BC, a partnership initiative led
by the Fraser Basin Council
Clean Nova Scotia (not-for-profit organization)
FCM
MSC
CAP
For Actions 5–13: a variety of stakeholders
(municipalities, school districts, transit organizations,
etc.)
Action #10: Private and public sector fleets
Municipalities and utilities
Municipalities
School buses – school bus
operators and districts
The BC Air Action Plan is part of the BC provincial
government’s broader strategy to lead the world in
sustainable environmental management, with the best
air and water quality and the best fisheries
management. The 28 actions outlined complement the
work under way across B.C. to address climate change,
improve air quality, protect water, enhance public transit,
improve local planning, and make the province electricity
self-sufficient by 2016.
“Solutions are available today that greatly reduce
vehicle fuel consumption and harmful emissions,
what's needed is the will to make the change.”
The PPRE complements CARA
by addressing emissions from
existing vehicles and engines
that could remain on Canada’s
roads for up to 20 years. It
targeted on-road HDD vehicles
such as trucks and school
buses, and explores how best
to voluntarily reduce their
emissions.
$ - N/A
Between 2008 and 2010, over 7000 vehicles in
Ontario’s municipal fleets have completed
FCO’s Fleet Review.
Context (rationale and
need for the program)
FCO also works with governments, private
business, and private and public sector fleets.
It is estimated that approximately 26% of
Canadian GHG emissions are released from
the transportation sector (GHGs are directly
related to fuel consumed). Automotive
technology is improving in order to reduce
emission intensity, but the number of vehicles
on Canadian roads continues to grow.
Municipal fleet operations usually perform
according to fixed annual cycles. Vehicles are
outfitted to complete specific tasks and sustain
municipal services. The Municipal Fleet Review
Program delivers a comprehensive set of
reports that support green management
decisions and demonstrate ways to improve
operations, thereby reducing costs and GHG
Environment Canada
Rationale for action on heavy-duty vehicles: older
commercial and industrial vehicles can emit up to 60
times more fine PM than those with new, modern
engines. And modern engines can run even cleaner
when they switch from 100% diesel fuel to a biodiesel
blend.
x
Audit and Evaluation Branch
Objective/goal
Mechanisms/activities
completed
Evaluation of the PPRE
Fleet Challenge Ontario (Municipal Fleet
Review)
emissions.
The Fleet Review is a green fleet optimization
program designed with principles similar to
LEED© for buildings, and is Canada’s leading
initiative in this area. Supported by
the Government of Ontario, FCO works with
each municipality’s fleet management team to
evaluate a broad range of management
options, including fuel efficiency, vehicle
selection, maintenance techniques, asset
management, computer systems and end-ofcycle disposal of surplus vehicles.
BC Air Action Plan
Clean Nova Scotia (Fleetwiser Program)
PPRE Pilot Project
The BC Air Action Plan contains eight actions aimed at
making heavy-duty vehicles cleaner (Actions 5–13).
Fleetwiser seeks to create a cleaner, healthier
environment by informing, enabling and inspiring
Nova Scotian fleets to respect and consider the
environment in all their choices.
Fleet Review Process
 The Fleet Review Program is applicable to
licensed, on-road leased or owned vehicles
using any fuel type.
 Participating municipalities are required to
contribute a nominal fee to the Municipal
Fleet Review Program based on the fleet size
(from $1,250 to $2,500).
 Steps for the fleet review:
1. A compilation of fleet data by the fleet
department. FCO employs a unique
software system with an extensive
database of current, relevant fleet
information. A fleet’s data are entered
and a series of computer-generated
reports is produced. Results
are further analyzed and interpreted by
FCO experts in accordance with a fleet’s
unique features.
Actions:
Action #5: Retrofit HDD vehicles
Action #6: Retrofit transit buses
Action #7: Clean up school buses (funding to school
districts to buy new, clean buses)
Action #8: Get AirCare ON-ROAD out to more
communities
Action #9: Get big diesels to stop idling
Action #10: Green vehicle fleets: we’re making it happen
Action #11: Use biodiesel in government diesel vehicles
Action #12: B.C. Buys Green
Action #13: Support greener ports and marine vessels
Clean Nova Scotia’s Fleetwiser Pilot Project is
working with four municipalities and one utility to
improve vehicle fleet efficiency and reduce fuel
consumption, GHG emissions, and business costs.
The objective is to reduce
emissions from Canada’s onroad HDD municipal fleet. The
project aims to increase
knowledge, awareness and
supporting tools for
municipalities to reduce their
emissions, by highlighting
emission reduction potentials
and identifying barriers to the
widespread adoption of
emission reduction strategies
for fleet vehicles.
Environment Canada worked
with the FCM to enable the
following activities:
2. Participants receive a set of detailed
Environment Canada
Action #10, “Green vehicle fleets: we’re making it
happen”, pertains to the Green Fleets BC initiative (a
partnership with the Fraser Basin Council). The goal is
to have Green Fleets BC be a key information hub for
the latest on green technologies for private and public
sector fleets, including taxis, emergency vehicles,
delivery vans and commercial freight trucks. Initiatives
include:
 Green Trucking Technologies: Green Fleets BC
will support demonstration projects on trucking
technologies to help fleet managers make good
xi

Participating fleet managers and operators receive
free consultation, training and resources to produce
measurable efficiency improvements. The program
delivers fleet energy efficiency through education,
technology, outreach, and driver awareness
initiatives. Fleet baseline data, NRCan’s online Fleet
Emissions Tool accounts, and individual fleet action
plans are in development. The knowledge and tools
gained from the project will be used to improve the
fuel efficiency of other fleets that are interested.
Initiatives to date:

Nova Scotia municipal and utility fleet
managers have participated in Fuel
Management 101 workshops.

Fleet Efficiency Tool Kits have been
developed for fleet managers, operators
and technicians, and are accessible


Establish partnerships with
municipal fleet managers’
networks, federal and
provincial agencies,
environmental NGOs and
private sector
organizations.
Create a baseline of
municipal heavy-duty fleets
in Canada, including a fleet
profile, an overview of best
practices, and an
identification of barriers to
greening fleet initiatives.
Identify and create an
online resource in both
official languages for
Audit and Evaluation Branch
Evaluation of the PPRE
Fleet Challenge Ontario (Municipal Fleet
Review)
reports and a synopsis; contents
include recommendations and a
comprehensive roadmap to green
and improve fleet efficiency. FCO
experts then meet with the fleet
operators and related personnel at
their location to review and discuss
the results and answer questions.
Forums and workshops
As part of the Ontario Municipal Fleet Review
Program, and with the support of the
Government of Ontario, FCO presents forums
and workshops that keep fleet managers on the
cutting edge of green transportation trends,
technology developments and green fleet
initiatives. These have included FCO
Networking Breakfasts (two each in 2009 and
2010), an FCO-FCM workshop for fleet
managers, etc.
BC Air Action Plan
Clean Nova Scotia (Fleetwiser Program)
choices for the best results.


BC BioFleet: Green Fleets BC is sponsoring
demonstration projects on the use of biodiesel in
the B.C. public sector and commercial vehicle
fleets, through branding, website resources, training
workshops, outreach and incentives.
Hybrid Experience Report:,The report documents
the real-life experiences and fuel efficiency results
of fleets that use hybrid vehicles.

Green Fleets Network: Green Fleets BC is
sponsoring a new network of fleet managers.

Canada’s Green Fuels Map: The Canada's Green
Fuels Map provides an online lookup of both retail
and key-lock stations, coast to coast.

Emissions calculator tool.


through Fleetwiser’s website, as are other
key resources.
Strategies implemented by municipalities /
utilities have included: anti-idling and
efficient driving promotion and education,
in-car training, preventive maintenance, and
green procurement planning to optimize
fleet equipment, technology and vehicles.
Customized vehicle-tracking systems and
idle-reduction technologies are being tested
on vehicles such as snow plows and salt
trucks, bucket-lift trucks, buses, and lightduty vehicles.
PPRE Pilot Project



municipal fleet managers
that will provide information
on best practices available
to achieve emission
reductions.
Promote, in both official
languages, the resources
guide and other products
and services.
Deliver education and
training workshops in
collaboration with partners,
using resources identified to
increase the capacity of
municipal fleet managers to
act.
Develop a business case
for emission reduction
initiatives in heavy-duty
fleets that can be included
in a resources guide.
In addition to the work
completed in conjunction with
FCM, Environment Canada
worked with other Government
of Canada ecoTRANSPORT
funding programs (i.e.,
Transport Canada’s
ecoFREIGHT, NRCan’s
FleetSmart) to ensure they
were aware of these activities
for the on-road public HDD
fleets. Furthermore,
Environment Canada worked in
conjunction with CAP and MSC
Environment Canada
xii
Audit and Evaluation Branch
Fleet Challenge Ontario (Municipal Fleet
Review)
Time frame / duration
Pilot project in 2008
Extension from 2009–2012
Source
http://www.fleetchallenge.ca/
http://www.fleetchallenge.ca/content/fleetreview
Environment Canada
Evaluation of the PPRE
BC Air Action Plan
Launched in 2007
Website features information from 2007 to 2009, but
should have funding from 2008 to 2011
http://www.bcairsmart.ca/transportation/heavyduty.html
http://www.greenfleetsbc.com/
xiii
Clean Nova Scotia (Fleetwiser Program)
PPRE Pilot Project
From 2009 to 2011
to collect information about,
and build capacity to address,
emissions from school buses in
select communities in Ontario.
From 2008 to 2011
http://www.clean.ns.ca/content/Sustainable_Fleet
http://Fleetwiser.ca/
http://Fleetwiser.norex.ca/
Evaluation Work Plan,
February 2011
Audit and Evaluation Branch
Evaluation of the PPRE
This analysis has demonstrated that all selected capacity-building programs have
a similar temporal scope of two to four years, in line with the PPRE’s three-year
time frame. In addition, all programs partnered with an NGO to deliver their
activities. All target audiences for the programs included municipalities, although
the Green Fleets BC initiative also targeted the commercial sector, whereas
Fleetwiser worked with one utility and the PPRE’s scope included school bus
operators and school districts.
Although none of the programs were found to have a specific objective to build
the capacity of municipal fleet operators responsible for on-road HDD vehicles,
the literature review demonstrated that there are multiple alternatives to building
capacity in order to reduce vehicle emissions more generally, including the
following:
 Conducting audits of municipal fleets, which focus on identifying retrofit
opportunities by conducting an on-site audit of a municipal fleet
 Workshops, such as for fleet managers regarding fuel efficiency or how to
green their heavy-duty vehicle fleet
 Networking, which includes networking opportunities through joining
various fleet manager networks
 Developing guidelines and toolkits, which are provided to fleet managers
and cover a range of topics ranging from best practices to emerging
technologies
 Links to other resources, primarily through the organizations’ websites
 Testing of reduction technologies, which focuses on supporting
demonstration projects and purchasing reduction technologies
Environment Canada
xv
Audit and Evaluation Branch
Evaluation of the PPRE
Annex 3. Interviewees and Interview Guides
Interviewees
Internal:
1 Senior manager from Environment Canada
2 Staff from Environment Canada (HQ)
3 Staff from Environment Canada (Ontario Region)
External – Other Government Department:
1 Staff from NRCan
External – Partners/NGOs:
1 Manager from FCM
1 Staff from FCM
1 Staff from CAP
1 Staff from MSC
Interview Guides
Letter of Introduction
Date:
February XX, 2011
To:
Key Informant
Subject:
Evaluation of the Pilot Project on reducing emissions from vehicles
and engines
Environment Canada (EC) is currently conducting an independent evaluation of
the Pilot Project to reduce emissions from vehicles and engines (PPRE). You
should have already received an email from program staff in EC requesting your
participation in this study; along with a call from the Evaluation Team to schedule
this interview. I am writing to confirm your scheduled interview for February xx,
2011 at xx am/pm. We very much appreciate your valuable contribution to this
research.
The planned interview, to be conducted by a consultant from Stratos, will take
around one hour and the information you provide will not be directly attributed to
you, rather your input and candid response will help inform our evaluation by
contributing to our formulation of conclusions and recommendations. Interviews
can be conducted in either English or French according to your preference, and
you will have the option to answer questions you are most comfortable with. If
your schedule has changed or you are not available, we would appreciate if you
could suggest another time that is more convenient or nominate another
individual who could participate in this evaluation in your place.
Attached, you will find a series of questions that will serve as the basis for the
scheduled interview.
Environment Canada
xvi
Audit and Evaluation Branch
Evaluation of the PPRE
If you have any comments or concerns, please feel free to contact Nicola Scahill,
Transportation Division, Environment Canada, 819-997-9284
(nicola.scahill@ec.gc.ca), or Michael Gullo, Project Manager, Stratos, 613-2411001x231 (mgullo@stratos-sts.com ) .
Interview Guide – Internal
Relevance
1) What is your involvement and/or role with the PPRE? (Introduction)
2) In your opinion, to what extent does the PPRE address an actual need?
What ‘environmental need’ or gap did it address and how was this
determined? (EQ1)
3) Does the PPRE address recent/current federal priorities? If so, which ones
and to what degree? (EQ2)
4) How does the PPRE help meet EC’s current strategic outcomes and
expected results? If so, which ones and to what degree? (EQ2)
5) What role/responsibility is there for the Federal Government to:
a) encourage voluntary reduction of emissions from Canada’s on road
heavy duty diesel municipal fleet, and
b) to work on collaborative technical studies with the US to reduce GHG
emissions from on-road fleets in support of future regulatory actions?
(EQ3)
6) Can you identify other similar programs offered by the Government of Canada
that compliment or duplicate the objectives or activities of the Pilot projects?
(EQ1)
Project Design, Delivery and Effectiveness
7) To what extent are the roles and responsibilities clear to all involved parties?
(EQ5)
8) How was the project designed to meet its expected results (e.g. what
performance data is collected and reported)? How effective has the program
been in achieving its expected results? Can you comment on whether the
project did not achieve any of its expected results? (EQ 4 & 5)
9) What have been the key outputs and results from the projects to-date? (e.g.,
number and population of municipalities implementing green fleet initiatives,
number of technical studies done in collaboration with U.S. EPA, etc.) (EQ9)
10) How has information been shared with other public fleet operations or other
fleets that operate within a municipal context? (EQ7)
Environment Canada
xvii
Audit and Evaluation Branch
Evaluation of the PPRE
11) What have been the main strengths, weaknesses, and lessons learned in
implementing the projects – what worked best, what was critical to success,
what were the main challenges? (EQ8)
12) Do you foresee any potential barriers for expanding this project or replicating
it a national scale? (EQ6)
13) Have there been any unintended outcomes as a result of the pilot (positive or
negative)? (EQ10)
Project Efficiency and Economy
14) What measures have been implemented to enhance the efficiency of the
PPRE (i.e., keep costs as low as possible) (EQ11)?
15) Do you think that the investments made in the PPRE are a good use of public
funds? Why? (EQ11 & 12)
16) Are there alternative means or models to enhancing efficiency and economy
with these kinds of projects? (EQ5 & 11)
Interview Guide – External
Relevance
1) What was your involvement in project #X/Y? (Introduction)
2) In your opinion, what ‘environment need’ does Project X/Y address? Why
was this project necessary? (EQ1)
3) In your opinion what is the appropriate role for the federal government to play
with respect to reducing emissions from vehicles and engines?
Project Design, Delivery and Effectiveness
4) What outputs or information has the project provided you with (e.g. technical
studies, new information on retrofits, etc.)? Why is this information valuable
and has it helped you contribute to other activities or initiatives? (e.g.,
implementation of green fleet initiatives, school bus retrofits, ongoing
contributions to research, etc.) (EQ1, EQ9)
5) What have been the main strengths, weaknesses, and lessons learned in
implementing the projects – what worked best, what was critical to success,
what were the main challenges? (EQ8)
6) Do you foresee any potential barriers for expanding this project or replicating
it at a national scale? (EQ6)
7) Have there been any unintended outcomes as a result of the pilot (positive or
negative)? (EQ10)
Environment Canada
xviii
Audit and Evaluation Branch
Evaluation of the PPRE
Project Efficiency and Economy
8) Do you think that the investments made in this Pilot Project are a good use of
public funds? Why? (EQ11 & 12)
9) Are there alternative ways to implement projects such as this? What would
be the pros and cons of these alternatives? (EQ5 & 11)
10) Are there more effective or efficient ways to deliver the program, or obtain
similar results? (EQ5 & 11)
11) What would have been the impact if this Pilot Project did not exist? (EQ1)
Environment Canada
xix
Audit and Evaluation Branch
Evaluation of the PPRE
Annex 4. Summary of Findings (Table)
Evaluation Issue
Relevance:
Continued need for the program
1. Was there a need for the project? Is there a
need for the project to continue?
Alignment with federal government priorities
2. Is the project aligned with federal government
priorities?
Consistency with federal roles
3. Is the project consistent with federal roles and
responsibilities?
Performance – Project Design and Delivery:
4. Are appropriate performance data being
collected, captured and safeguarded? If so, is
this information being used to inform senior
management / decision makers?
5. Is the program design appropriate for achieving
expected program results?
6. Are there any barriers or challenges that could
affect the success of this project if it were to be
implemented nationally or integrated into an
existing national project?
7. Have the results of the pilot project been
communicated effectively? How could
communications be improved?
8. What are the best practices and lessons
learned as a result of the project?
Performance – Effectiveness:
9. To what extent have the intended outputs and
outcomes been achieved as a result of the
project?
10. Have there been any unintended outcomes,
either positive or negative?
Performance – Efficiency and Economy:
11. Has the project undertaken its activities and
delivered products (e.g., technical studies) in
the most efficient manner?
12. Has the project achieved, or is it on track to
achieving, its intended outcomes in the most
economical manner?
Environment Canada
Achieved
Progress
Made;
Attention
Needed
Little
Progress;
Priority for
Attention
N/A












xx