MS Word

Evaluation of the

National Vehicle

Scrappage Program

Final Report

July 12, 2011

Audit and Evaluation Branch Evaluation of NVSP

Report clearance steps

Planning phase completed

Report completed

Report approved by Departmental Evaluation

Committee

November 2010

July 2011

July 2011

Environment Canada 2


Acronyms used in the report

AQLPA Association québécoise de lutte contre la pollution atmosphérique

ARC Automotive Recyclers of Canada

CAA

EC

G&Cs

GHG

KI

NFP

NO x

NVSP

O&M

Clean Air Agenda

Environment Canada

Grants and contributions

Greenhouse gas

Key informant

Not-for-profit organization

Nitrogen oxide

National Vehicle Scrappage Program

Operations and maintenance

PM

RYR

SI

Particulate matter

Retire Your Ride

Summerhill Impact (named Clean Air Foundation until the fall of

2009)

SPARC SPARC is a program database used to track retired vehicles and disbursed incentives

TB

VOCs

Treasury Board

Volatile organic compounds

Acknowledgements

The Evaluation Project Team would like to thank those individuals who contributed to this project, particularly all the interviewees who provided insights and comments crucial to this evaluation. We would also like to thank the members of the Evaluation Committee for their guidance and support: Gwen Goodier, Josée Maillette, Caitlin Horrall and Phil

Caron, all with the Outreach Division, Environmental Stewardship Branch.

The Evaluation Project Team was led by Michael Callahan, under the direction of

William Blois, Acting Director, Evaluation, and included Ryan Brown, Monicah Malusi and Robert Tkaczyk.

The evaluation was conducted by Goss Gilroy Inc. in association with David Redmond and Associates, who prepared this report under the direction of the Evaluation Division,

Audit and Evaluation Branch.



Table of Contents

EXECUTIVE SUMMARY ....................................................................................... i

1.0

INTRODUCTION......................................................................................... 7

2.0

BACKGROUND .......................................................................................... 8

2.1

Program Profile ........................................................................................ 8

2.2

Governance Structure ............................................................................ 12

2.3

Resource Allocation ............................................................................... 13

2.4

Expected Results and Program Logic Model ......................................... 14

3.0

EVALUATION DESIGN ............................................................................. 16

3.1

Purpose and Scope ............................................................................... 16

3.2

Evaluation Approach and Methodology ................................................. 16

3.3

Limitations .............................................................................................. 18

4.0

FINDINGS ................................................................................................. 19

4.1

Relevance .............................................................................................. 20

4.2

Performance

– Design and Delivery ...................................................... 27

4.3

Performance – Effectiveness ................................................................. 48

4.4

Performance – Efficiency and Economy ................................................ 65

5.0

CONCLUSIONS ........................................................................................ 72

6.0

LESSONS LEARNED ............................................................................... 75

Annex 1: Bibliography ......................................................................................... 76

Annex 2: Network of Regional NFP Delivery Organizations ............................... 79

Annex 3: Evaluation Issues and Questions ......................................................... 80

Annex 4: Summary of Findings ........................................................................... 86

Annex 5: Comparison with Other Jurisdictions ................................................... 89

Environment Canada


EXECUTIVE SUMMARY

INTRODUCTION

In order to meet the information requirements of senior management at

Environment Canada (EC) and fulfill a Treasury Board commitment, an evaluation of the National Vehicle Scrappage Program (NVSP) was conducted.

This evaluation was part of EC’s Risk-Based Evaluation Plan for 2010-2011 to

20142015. The evaluation was overseen by EC’s Evaluation Division, Audit and

Evaluation Branch, in 2010-2011. The Department commissioned Goss Gilroy

Inc. in association with David Redmond and Associates to carry out the data collection and analysis.

The NVSP (known as Retire Your Ride / Adieu bazou) is one of a range of complementary clean transportation initiatives undertaken in partnership with other federal departments as part of the Clean Air Agenda. The program encouraged owners of old cars (model-year 1995 and older) to retire their vehicle in exchange for an incentive. The NVSP’s primary goal was to reduce air pollutants by removing these older vehicles from the road; its secondary goals were to reduce greenhouse gases (GHGs) by promoting sustainable transportation alternatives and to prevent the release of toxic substances through responsible vehicle recycling. EC also worked with the Automotive Recyclers of

Canada to develop the national code of practice for vehicle recycling in order to ensure high environmental standards and consistent practices for vehicles retired under the program.

EC set up a separate contribution authority with a not-for-profit organization

(NFP) to deliver the NVSP beginning in 2007. Funding of up to $92 million was provided to the program over five years (2007-2008 to 2011-2012) from funds set aside in Budget 2005 and Budget 2007. The program ended on March 31, 2011, and, as intended, was not renewed.

METHODOLOGY

The evaluation’s objective was to assess the relevance and performance of the

NVSP. The evaluation focused primarily on the four-year time frame from 2007-

2008 to 2010-2011, but also encompassed an examination of 2011-2012 program data on the total number of vehicles retired and final planned program expenditures. The evaluation relied on multiple sources of evidence, including a literature and document review, secondary data analysis, 20 key informant (KI) interviews, and three case studies that focused on specific aspects of the program: communications, the use of incentives, and the vehicle recycling code of practice.

Environment Canada i


FINDINGS AND CONCLUSIONS

Relevance

Based on the evidence gathered for this evaluation, there was an identified need for the NVSP. Model-year 1995 and older vehicles produce significantly more smog than newer models, and there is scientific evidence that smog is highly toxic for the environment and human health. When the NVSP began in 2007, there remained 4.6 million model-year 1995 and older models on the road. There was also a need to improve the environmental practices of the vehicle recycling industry in Canada. Although larger vehicle recyclers have generally had good environmental practices, many small and medium companies have struggled in this regard because of insufficient knowledge or misconceptions about the costs associated with good environmental practices.

Although there were regional vehicle scrappage initiatives with similar objectives when the NVSP was implemented, none were of similar scope. EC strived to build on these initiatives in order to reduce duplication. The program was consistent with fed eral priorities and EC’s mandate and priorities related to air quality and climate change. The NVSP was aligned with EC’s strategic outcome,

“Threats to Canadians and their environment from pollution are minimized.”

Performance: Design and Delivery

The NVSP objectives were well defined and targets were clear, though the original target for the total number of retired vehicles became unattainable because the full national program was only launched in January 2009, 21 months after first being announced in Budget 2007. This pre-launch period was needed for program design and obtaining the necessary financial approvals. However, annual targets were met or exceeded once full implementation was achieved.

The objectives for the vehicle recycling code of practice were realistic, well defined, and successfully achieved.

The program was delivered through a national NFP (Summerhill Impact [SI]) using a contribution agreement with EC. The NFP then worked with a network of regional NFPs —a delivery system that proved to be effective and efficient overall. The advantages included faster implementation, flexibility when making adjustments, and reduced duplication. Program funding for cash incentives and program operations was adequate. Program participants were satisfied with the program, and the level of satisfaction increased as the program matured.

Program results were monitored using an online data system (SPARC), quarterly and annual reports from SI to EC, a mid-program review, surveys of participants and the general public, and various program management tools such as spreadsheets to monitor vehicle recycler audits. The program monitoring and

Environment Canada ii

Audit and Evaluation Branch Evaluation of NVSP performance data were used extensively by program managers to direct and fine-tune the program.

Communications activities were successful at reaching key stakeholders and the general public. Outreach and promotional materials for Retire Your Ride (RYR) were linked to two other federal government initiatives: ecoENERGY for Personal

Vehicles (which includes, for example, driving tips and information on buying a fuel-efficient vehicle), and Q&As on admissibility of the RYR transit incentive for the federal transit pass tax credit.

Many lessons were learned, including: the strengths of an approach in which an

NFP delivers the program through a contribution agreement; the need for handson involvement by EC in managing a large, complex contribution agreement; the need to allow adequate time to reach full implementation of a complex program involving many partners; and the importance of clearly defined roles, responsibilities and lines of communication.

Performance: Effectiveness

The NVSP achieved its intended short-term outcomes. The program was made available across all regions, except in the territories due to a lack of recycling infrastructure and low demand. The level of public awareness increased steadily as the program matured. Many program users recognized that their older vehicle was less environmentally friendly than new models, but many users also overestimated the role of car maintenance as a determinant of the level of pollution produced by older automobiles.

Regarding the medium-term outcomes, as of March 31, 2011 a total of 138 600 vehicles were retired as a result of the program. However, these results cannot be entirely attributed to the NVSP, because some participants indicated in a survey that they would have retired their vehicle in the absence of the program.

The fact that the NVSP was not fully implemented nationally until 21 months after program announcement explains why it did not reach its initial overall target of

200 000 vehicles (i.e., 50 000 for each of four years). Also, some regions had difficulties reaching their targets due to the lack of recycling infrastructure and other logistical issues.

Most of the participants (93%) preferred the cash incentive of $300 over a noncash incentive (e.g., transit passes, discounts on bicycles). More than 75% of participants replaced or indicated they would replace their vehicles with another one. The manufacturer rebates had a direct impact on the type of vehicle purchased, as most program users who obtained rebates opted for larger vehicles than the ones they retired.

Environment Canada iii


Regarding the long-term outcomes, EC data indicate that the program contributed to reduced emissions of nitrogen oxide (NO x

), volatile organic compounds (VOCs) and greenhouse gases (GHGs). Total reductions are estimated to be approximately 5000 tonnes of VOCs and NO x

, and approximately

37 500 tonnes of GHGs.

The vehicle recycling code of practice is considered an important and valuable legacy of the program. The code, published in print and electronic formats, was widely and readily available to auto recyclers across the country. Environmentally safe practices for the recycling of vehicles during the program were ensured by requiring all participating recyclers to adhere to the code. A survey indicates that

99% of participating recyclers plan to continue applying the code of practice after the program ends.

Many factors had an impact on the program’s success, including the influence of a similar U.S. program (Cash for Clunkers) and the auto manufacturer rebates, both of which contributed to increased profile and use of the NVSP. At the same time, the program had limited ability to influence the transportation choices of program users after the incentive was provided —an outcome that was not expected. In terms of unintended or indirect impacts (i.e., results that were not formally expected of the program), the NVSP helped build NFP capacity, and led to economic and social benefits by removing older, less safe vehicles and replacing them with new, safer vehicles.

Performance: Efficiency and Economy

Evidence indicates that program resources were well spent. EC and SI monitored performance and introduced corrective measures to ensure program efficiency. A cost analysis indicates that, compared to other similar programs, the NVSP can be considered cost-efficient. In addition, significant resources were leveraged by the program. The total value of incentive rewards delivered to program participants was $126 million, of which $42 million was federal funding, and thus the program achieved its target of $3 in incentive value for every $1 in federal funding toward incentives.

Few alternative program models were identified by KIs. A review of foreign programs indicates that an alternative approach would have been to offer more significant incentives, with conditions on the types of substitute vehicles purchased.

Lessons Learned

As the NVSP has ended, the focus of this evaluation was on lessons learned, not recommendations for improvement. The following lessons learned from the

NVSP may help inform the design of future similar initiatives:

Environment Canada iv




Incentives can help encourage the removal of technologies (i.e., vehicles in this case) that are harmful to the environment . In particular, a moderate cash incentive (such as $300 to retire an older vehicle) can be effective. It was originally assumed that $300 cash would not be as popular a reward as other incentives of higher value (transit passes, discounts on bikes, memberships in car-sharing programs, etc.). This assumption was proven to be inaccurate, as more than 90% of program participants chose the cash incentive.



The use of a third-party delivery partner (through a contribution agreement with an NFP) can make a program efficient, effective and timely . According to evaluation findings, the use of a third-party deliverer allowed EC to implement a cost-efficient and effective program by capitalizing on the agility of a national NFP.



When using large, complex contribution agreements, care must be taken to establish a tailored contribution agreement, with precise clauses on roles and responsibilities . The NVSP was delivered through a national NFP that managed a significant amount of program dollars. An agreement of this size and complexity should be tailored to the type of program and clarify the roles and responsibilities of each partner.



It is important to thoroughly assess all aspects of a program up front, and to manage the risks that they represent . The evaluation evidence indicates that the NFP which received the contribution did not have the full capacity up front to manage such a large contribution. For example, EC did not sufficiently investigate the financial management capacity of the NFP before the program’s launch, and consequently the financial reporting and monitoring had to be strengthened throughout the program. In a situation such as this (a large contribution to a single organization), a risk assessment should be conducted. The assessment should then be used to identify and address weaknesses and gaps on the part of the recipient, and/or to design a tailored monitoring strategy.



Effective performance monitoring can help improve programs and the quality of evidence for evaluations .

The NVSP’s performance monitoring system, featuring an online reporting database, allowed program management to assess the program’s results on a daily basis. The monitoring system allowed EC to monitor progress and make adjustments to the delivery of the program in a timely manner. It also allowed the evaluation team to assess the program’s success with reliable and representative quantitative measures. The availability of this performance data therefore strengthened the evidence for the evaluation.

Environment Canada v


Environment Canada vi


1.0 INTRODUCTION

In order to meet the information requirements of senior management at Environment

Canada (EC) and to fulfill a Treasury Board (TB) commitment, an evaluation of the

National Vehicle Scrappage Program (NVSP) was conducted. This evaluation was part of EC’s Risk-Based Evaluation Plan for 2010-2011 to 2014-2015. The evaluation was overseen by EC’s Evaluation Division, Audit and Evaluation Branch, in 2010-2011.

The evaluation’s objective was to assess the relevance and performance of the NVSP.

The evaluation focused primarily on the four-year time frame from 2007-2008 to 2010-

2011. The NVSP ended on March 31, 2011.

This document presents the findings and conclusions of the evaluation of the NVSP, and is organized in the following way: section 2.0 provides background information on the NVSP; section 3.0 presents the evaluation design, including the evaluation’s purpose and scope, and the methods used to conduct the evaluation; section 4.0 presents the evaluation’s findings; sections 5.0 and 6.0 lay out, respectively, the conclusions and lessons learned.



2.0 BACKGROUND

This section provides the background and main parameters of the NVSP. It is based on program documentation and financial data (see Annex 1 for bibliography).

2.1 Program Profile

The main goal of the NVSP (known as Retire Your Ride [RYR] / Adieu bazou) was to accelerate the removal of older, higherpolluting vehicles from Canada’s roads.

Participants retiring their vehicle through the program received a reward worth at least

$300. The program’s target was 50 000 retired vehicles per year.

Secondary goals of the NVSP were to reduce greenhouse gas (GHG) emissions by encouraging the uptake of sustainable transportation choices, and to prevent the release of toxic substances into the environment through responsible vehicle recycling.

Targeted Vehicles

The NVSP defined an “older” vehicle as being model-year 1995 or older. This cut-off date was selected because new vehicle emission standards were tightened in 1996 (as they were again in 2004). Because of these changes, older vehicles pollute more than newer ones. For example, vehicles from model-year 1995 and previous years produce

19 times more smog-forming pollutants than vehicles from model-year 2004 and subsequent years.

When the NVSP was announced in 2007, 4.6 million of the estimated 18 million personal vehicles in use in Canada were model-year 1995 or older (Figure 1). In 2009

(the most recent year for which these data are available), 3 million older vehicles were still being driven and still polluting the air.



Figure 1: Number of Model-year 1995 and Earlier Vehicles on the Road, by Year

(in millions)

7

6.28

6

5.45

5

4.56

4 3.72

3.01

3

2

1

0

2005 2006 2007 2008 2009

Source: Statistics Canada, Canadian Vehicle Survey; Annual , 53-223-X

Environmental, Economic and Health Impacts of Smog

Smog is the noxious mixture of ground-level ozone and particulate matter (PM).

Ground-level ozone, a respiratory irritant, is formed through a complex series of reactions involving volatile organic compounds (VOCs) and nitrogen oxide (NO x

), two substances found in the exhaust gases of personal vehicles. Although significant health impacts arise from PM as well, the NVSP focused on NO x and VOCs reductions, because personal vehicles are not large emitters of PM.

In many well-designed short- and long-term exposure studies, higher levels of smog were associated with increased human mortality and morbidity. Smog also had negative environmental impacts, as vegetation health and productivity is harmed by pollutants like ground-level ozone (Environment Canada March 2011a). Furthermore, smog can harm wildlife, their habitat, and the availability and quality of their food supply

(Environment Canada, 2011b).

Personal vehicles also emit GHGs, which have been linked to human-influenced climate change. GHGs are emitted through the burning of fossil fuels, alongside the smogforming pollutants discussed above. However, unlike smog-forming pollutants, GHG emissions cannot be reduced significantly through emission-reduction technology;

GHGs are reduced by burning less fuel. This can be achieved by reducing personal vehicle usage, driving vehicles with improved fuel efficiency, or switching to more sustainable modes of transportation like public transit and cycling.



Vehicles cause further damage to the environment if they are not recycled with high environmental standards. Toxic or harmful substances like oil, antifreeze and mercury need to be removed and stored properly in order to prevent their release into the environment.

Program History and Financing

The federal government has been involved in accelerated vehicle retirement initiatives for over a decade. Between 2001 and 2008, EC provided approximately $500,000 per year for the delivery of seven local vehicle scrappage programs in six provinces. The

NVSP was the first scrappage program at the national level and, when fully implemented, had an annual budget of $26

–$30 million.

Funding of up to $92 million was provided to the NVSP over five years (2007-2008 to

2011-2012) from funds set aside in Budget 2005 and Budget 2007. The NVSP ended on March 31, 2011, and, as intended, was not renewed.

Table 1: NVSP 5-year Budget Allocation (in $ millions)

Salary and benefits

Operations and maintenance (O&M)

Grants and contributions (G&Cs)

Corporate support

Total

$1.70

$8.63

$79.98

$1.70

$92.00

In 2007, EC set up a separate contribution authority for the NVSP. In 2008, Summerhill

Impact (SI) (named Clean Air Foundation at that time), a not-for-profit organization

(NFP) specializing in environmental public engagement campaigns, signed a contribution agreement with EC to receive $62.2 million in funding over four years for program delivery and incentives.

Funding to SI was used to support its role as national program coordinator, which included establishing partnerships, distributing incentives to Canadians, and promoting the program at the local level. SI delivered the program through a network of regional

NFPs that also delivered public education and outreach programming (see Annex 2). As well, these NFPs provided a network of call centres and automotive recyclers, and carried out the daily interaction with participants in RYR.

The NVSP was one of a range of complementary clean transportation initiatives undertaken in partnership with other departments as part of the Clean Air Agenda

(CAA). The CAA’s spending for the first three years of its four-year mandate was $1.34 billion; the planned spending was $1.55 billion. Total planned spending for all federal departments in 2010-2011 was reported as $951.3 million.

Participation Process

To participate, vehicle owners submitted an online application or called the program’s toll-free number. Once the application was approved, the collection of the vehicle was arranged by the NFP in partnership with local vehicle recyclers. In most cases the


Audit and Evaluation Branch Evaluation of NVSP owner signed the vehicle over to the recycler, who then confirmed that the vehicle had been collected. In other cases, the owner could drive the vehicle to a pre-approved drop-off location. The NFP then provided the participant with an incentive of their choice.

Eligibility

Eligible vehicles were personal automobiles and light-duty trucks of model-year 1995 and earlier. They were required to be in running condition, and to have been registered and insured in the owner’s name in Canada for the previous consecutive six months

(12 months in B.C.). In accordance with program guidelines, vehicles were required to be:



able to start and move;



under 3856 kg in gross weight;



complete with no parts removed;



for personal use; and



free from liens (parking tickets, tolls for highway usage, etc.).

Incentives

A cash incentive of $300 was available everywhere that the program was offered. Noncash incentives, which varied among the provinces and communities, included rebates on newer vehicles, transit passes, discounts on bicycles and e-bikes, and memberships in car-sharing programs. The non-cash incentives were all worth more than the $300 cash incentive and were offered to encourage participants to move toward more environmentally friendly methods of transportation. Many of the non-cash incentives were leveraged by SI with the $300 allotted to each vehicle accepted into the program.

SI also negotiated partnerships with automobile manufacturers who offered rebates on the purchase of a new vehicle. The manufacturers absorbed the full price of these offers, because the terms and conditions of RYR prohibited the use of federal incentive dollars for vehicle rebates. For RYR participants, this meant that auto manufacturer rebates were a “top-up” to the RYR incentive of their choice.

Other EC Activities

EC worked with the Automotive Recyclers of Canada (ARC) to develop a national code of practice for vehicle recycling, in order to ensure consistent and high environmental practices for vehicles retired under the program. Participating recyclers were required to follow this code, and on-site audits were conducted (under contract) to verify their compliance.

EC conducted a financial audit of SI to verify that it was complying with the terms of the contribution agreement and that its financial system was effective. Preliminary findings of this audit were used to adjust financial monitoring by SI. Final audit results were not available for the current evaluation.



EC also provided funding to the Canadian Urban Transit Association for the development and delivery of a public awareness campaign for the program. The campaign placed special emphasis on transit incentives.

In addition, EC ’s Biosphère museum developed exhibits and short movies on vehicle recycling and sustainable transportation.

Horizontal / Cross-departmental Integration

The NVSP, part of the CAA, complemented the Agenda’s GHG regulations (which take effect starting with modelyear 2011 vehicles), Transport Canada’s ecoAUTO rebates that encouraged the purchase of fuelefficient vehicles, Transport Canada’s ecoMOBILITY program that increased the use of public transit and alternative transportation, and Natural Resources Canada’s ecoENERGY for Personal Vehicles program that encouraged better purchasing, driving and maintenance practices.

2.2 Governance Structure

reporting.

In EC’s Program Activity Architecture, the NVSP was intended to contribute to the strategic outcome “Threats to Canadians and their environment from pollution are minimized, ” under the Architecture’s sub-sub-activity 3.2.2 “Climate Change and Clean

Air Partnerships.

”

Within EC, the NVSP was managed and delivered by the Outreach Division of the

Environmental Stewardship Branch. The NVSP team was responsible for managing the contribution agreement with SI, including the monitoring of its activities and results.

Figure 2a shows the chain of contribution agreements and contracts put into place for program delivery. EC signed a contribution agreement with SI, which, in turn, signed separate agreements with provincial NFPs. Either through a contract with a third party or through their existing agreements, SI and the NFPs delivered the incentives, the auto recycling services and the call centre services. Reports from these relationships fed

EC’s reporting obligations. Figure 2b shows EC’s internal and intergovernmental



Figures 2a and 2b: Schematic of Program Delivery and EC Reporting

2a. Program Delivery

EC

SI

NFP

Incentive

Partners

Recyclers

Call

Centres

2b. EC Reporting

Clean Air Agenda

Secretariat

- EC

- HMARF*

- Clean Transportation Theme

- Transport Canada

EC

* Horizontal Management Accountability and Reporting Framework

2.3 Resource Allocation

Information on the program ’s financial resources over the five-year time frame from

2007-2008 to 2011-2012 is presented in Table 2.

Table 2: NVSP Financial Profile: Actual and Planned Expenditures (in $ millions)

2007-08 2008-09 2009-10 2010-11 2011-12 All years

1.50 13.15 34.76 41.39 1.20 92.00 Treasury Board funding

Direct program support

Salaries

O&M

G&Cs

Subtotal

Actual

0.24

0.22

0.09

0.55

0.18

Actual Actual Actual Planned

0.23

0.75

5.34

6.32

Corporate support, benefits, accommodations

0.60

Total EC expenses

0.73 6.92

Difference between projection and actual

Lapse and adjustment

0.77 6.05

Reallocated to other priorities

0.00 0.18

0.26

0.76

26.93

27.95

0.57

28.52

1.21

5.03

0.24

0.60

28.85

29.69

0.55

30.24

0.13

11.02

0.00

0.00

1.20

1.20

0.00

1.20

N/A

N/A

0.97

2.33

62.41

65.71

1.88

67.59

8.18

16.23

Source: EC financial information



2.4 Expected Results and Program Logic Model

The expected program results to be achieved over the four-year life of the NVSP included the following:



Retirement of 50 000 in-use vehicles per year of program operation



The following annual reductions in smog-forming emissions: approximately 2250 tonnes of NO x and VOCs, and 54 000 tonnes of GHGs (the reduction of NO x

and

VOCs was a primary program goal, and reduction of GHGs was a secondary goal)



A reduction of GHG emissions through increased use of sustainable transportation



An increase in awareness and understanding among Canadians of the environmental impacts of older vehicles



A reduction in the release of toxic substances



Development of a code of practice for vehicle recycling that will lead to higher standards nationally and that may be adopted by provinces and territories



A 3:1 leveraging of federal funding by partners for incentives

The performance measures and data sources used to assess these outcomes are identified in the Accountability, Risk and Audit Framework developed for the program

(Environment Canada 2008). The outcomes of the program are illustrated in the logic model represented in Figure 3 , which also presents the program’s key activities and outputs.



Figure 3 : National Vehicle Scrappage Program Logic Model

INPUTS

What we invest

ACTIVITIES

What we do

Manage contributions with recipients (NFPs)

OUTPUTS

What we produce

PARTICIPANTS

Whom we reach



Contribution agreement(s)



Financial and progress reports



Incentives



Leveraging



Database



National NFP



Local program delivery NFPs



Incentive partners



Owners of older vehicles



Canadians

SHORT TERM

OUTCOMES

(By March 2009)

What are the immediate outcomes?

Canadians in all provinces have access to incentives to reward them for removing older vehicles from the road

$92M of funding over five years (2007-2008 to 2011-2012)

FTEs







Promote program and evaluate the effectiveness of this promotion

Public awareness and promotion material (NFP and EC) including a

Museum Exhibit

Owners of older vehicles

Canadians

Promotional partners

Increased awareness, by the public, of

 impact of older vehicles

 program itself

Develop and implement recycling code of practice and conduct related training and audit activities

Code of practice



Vehicle recyclers

Increased awareness, by recyclers, of environmentally-safe recycling practices

MEDIUM TERM

OUTCOMES

(By March 2010)

What are the medium term outcomes?

LONG TERM

OUTCOMES

(March 2011 and beyond)

What are the long term outcomes?

Fewer older, high polluting personal vehicles on the road

Air pollutants are reduced

More sustainable transportation modes being used for personal transportation

Greenhouse gas emissions are reduced

Environmentally-safe practices in use for recycling of vehicles scrapped through the program

Release of harmful and toxic substances into the environment is prevented



3.0 EVALUATION DESIGN

3.1 Purpose and Scope

In order to meet a TB commitment, an evaluation of the NVSP was conducted in 2010-

2011. This evaluation was part of EC’s Risk-Based Evaluation Plan for 2010-2011 to

2014-2015.

The evaluation’s objective was to assess the relevance and performance of the NVSP.

The evaluation focused primarily on the four-year time frame from 2007-2008 to 2010-

2011, but also encompassed an examination of program data in 2011-2012 on the total number of vehicles retired and final planned program expenditures.

3.2 Evaluation Approach and Methodology

The evaluation examined the following two key issues, in accordance with the 2009 TB

Policy on Evaluation:



Relevance: the extent to which the program addressed a continued need, was aligned with federal government priorities, and was aligned with federal roles and responsibilities



Performance: the extent to which the program achieved its expected outcomes, and demonstrated efficiency and economy

A total of 16 evaluation questions supporting these two broad issues were addressed in the evaluation. These evaluation questions, along with the associated indicators, data sources and data collection methods, are listed in Annex 3. The methodological approaches are briefly described below.

Literature and Document Review

A literature and document review was conducted to address most of the evaluation questions. The evaluation team, in cooperation with program representatives, began the process by selecting key documents and literature to be reviewed. The team conducted additional searches via online databases to complete the information, and reviewed documentation about similar programs offered in other jurisdictions within Canada and abroad in order to compare the features of each program and assess their strengths and weaknesses. The full list of documents appears in Annex 1.

The documentation was reviewed using a grid (based on the evaluation issues) that summarized the source and reliability of the information. After completing the grid, the evaluation team summarized the findings from the documentation per evaluation issue in a separate matrix. This summary served as a base for the draft report.



Secondary Data Analysis

The evaluation team also reviewed program data to address the evaluation questions.

This included financial data from the program and performance data from SPARC, a database to track RYR vehicles and incentives.

The data were reviewed to obtain descriptive information about the program in order to establish its profile, and to gather data in order to address the evaluation issues. Tables were developed to describe the financial aspects of the program

—in particular, the financial information was analyzed to assess the program’s cost efficiency compared to similar programs in Canada (ecoAUTO rebate) and abroad (Cash for Clunkers in the

United States). The SPARC database was used to establish the reductions in smog, which were calculated based on EC and Natural Resources Canada emissions information.

Key Informant Interviews

A series of 20 in-depth, one-on-one key informant (KI) interviews was conducted using an interview guide, which included open-ended questions. The distribution of respondents to these interviews is outlined in Table 3 below:

Table 3: Distribution of Key Informant Interviews

Category

EC program management and staff

NFP delivery organizations

Vehicle recycler associations

Provincial governments

Completed

Interviews

4

12

2

2

Population of

Stakeholders

5

30

2

3

TOTAL 20 40

The interviews were used to address most evaluation issues and were conducted in the respondents’ official language of choice. Interviews were conducted in person in the

National Capital Region, and by phone for other areas. A visit to the Toronto area was also carried out, for in-person interviews with SI representatives and other stakeholders.

Interviews were conducted using tailored guides, presented under separate cover in a technical appendix.

The results of the interviews were entered into a matrix, organized by evaluation issue for analytical purposes. After an overall analysis, the evaluation team summarized the results for the purposes of this report.

Case Studies

Certain aspects of the program were assessed as separate case studies. This allowed the evaluation team to conduct interviews and review documents focused on key aspects of the delivery of the program. The three cases are described below.



1. Vehicle Recycling Code of Practice

As part of the NVSP, EC dedicated funding to help the automotive recycling sector develop a recycling code of practice. The code was intended to help recyclers improve their environmental performance and profitability while delivering valuable service to society. It was expected that the code would create a legacy of good environmental performance for the automotive recycling sector after the program sunsets. Seven individuals were interviewed and various documents were reviewed, including audit reports.

2. Working with Partners to Provide Non-cash Incentives

In exchange for their vehicle, Canadians who qualified for the program could choose a reward from a suite of optional incentives. These varied from province to province, and included free transit passes, discounts on bicycles or e-bikes, memberships in car-sharing programs, and rebates on new or newer vehicles. SI and the NFPs arranged these higher-value non-cash incentives by negotiating and/or leveraging the

$300 base incentive with local and national partners. Six individuals were interviewed, including NFPs and partners. Descriptive information from online sources was also reviewed.

3. Communications

The scope and complexity of RYR provided many communications challenges (and opportunities) for EC and the program delivery partners. On a functional level, communications issues touched all aspects of the program and were at multiple levels, including internal and external communications (e.g., communication of information, advertising, service standards, bilingualism, federal funding). The case study was based on four interviews and a review of program documentation and statistics, including general population surveys, user surveys and outreach materials.

Overall, interviews and documentation were used to gain a better understanding of the program’s performance on each case study topic, as well as challenges, best practices and lessons learned. Interviews were conducted using guides tailored for each case study. The key results of the case studies are highlighted in the body of this report under the relevant evaluation questions. The case study guides and case reports are presented under separate cover in a technical appendix.

3.3 Limitations

As with any other evaluation, this study involved a number of limitations, including the following:



Although the program ’s performance was well monitored, certain aspects of the evaluation depended on information that was not documented (it is not unusual for some information to not be documented). To supplement the available performance data and documentation, the evaluation team relied on KI interviews to address a number of evaluation questions. KI interviews potentially provide in-depth


Audit and Evaluation Branch Evaluation of NVSP responses, yet are also limited by a number of factors, including respondent bias and memory gaps. The KI interviews also covered only a selection of respondents, not the entire population of stakeholders involved in the program. The evaluation team is confident, however, that an appropriate mix of respondents was interviewed

(i.e., an appropriate balance between knowledgeable respondents representing each key stakeholder group and respondents capable of providing an objective view of the program). The interpretation of interview findings also considered the other lines of evidence, including program documentation as well as literature and program data, all of which were extensive. Integrating the results from multiple lines of evidence helped to ensure the validity of the evaluation findings and conclusions.



Although there are excellent data on the number of vehicles retired, there remain data gaps on the environmental impact of these removals. To assess the reductions in NO x

, VOCs and other pollutants, the evaluation team used various assumptions, including assumptions based on Statistics Canada data (i.e., estimates of kilometres driven by typical vehicles) and the specifications of the retired vehicles and how they were replaced. The team based its assumptions on the best information available, but it is likely that the environmental impact is different than what is reported in this evaluation, as there was no assessment of the environmental performance of each vehicle upon its retirement. The evaluation team remained conservative in its assumptions, however, to help ensure that the reductions in pollutants were not overestimated.

4.0 FINDINGS

This section presents the evaluation findings for each issue and the related evaluation questions. At the beginning of each section, a summary of the key findings is presented, with an assessment based on standardized ratings. The ratings and their definitions are outlined in Table 4, and a summary of the ratings for each evaluation question is presented in Annex 4.

Table 4: Ratings and Definitions

Rating

Achieved

Progress Made;

Attention Needed

Little Progress; Priority for Attention

Not Applicable

Definition

The intended outcome or goal has been achieved or met.

Considerable progress has been made to meet the intended outcome or goal, but attention is still needed.

Little progress has been made to meet the intended outcome or goal, and attention is needed on a priority basis.

A rating is not applicable.



4.1 Relevance

Evaluation Issue: Relevance

Overall Findings:

Overall, the evidence indicates that there was an identified need for the NVSP.

Model-year 1995 and older vehicles produce significantly more smog than newer models, and there is scientific evidence that smog is highly toxic for the environment and human health. When the NVSP was developed, there remained about 4.6 million model-year 1995 and older models on the road. There was also a need to promote environmentally safe vehicle recycling practices in Canada.

The program is generally consistent with federal priorities and EC’s mandate and priorities. Although there were regional initiatives with similar objectives when the

NVSP was implemented, none were of similar scope. However, EC made efforts to build on these regional efforts in order to reduce duplication.

The evaluation evidence suggests that there is not a significant continued need for the NVSP past March 2011.



Relevance

1. a) Did the Vehicle Scrappage Program respond to an identified need?

What was that need?

Rating: Achieved b) Is there a continued need for the program?

Rating: Not Applicable

The evaluation examined these related questions using data collected through the document and literature review and the KI interviews.

Findings: Evidence indicates that there was an identified need for the

NVSP. Model-year 1995 and older vehicles produce significantly more smog than newer models, and there is scientific evidence that smog is highly toxic to the environment and human health.

When the NVSP was announced in 2007, there remained 4.6 million model-year 1995 and older models on the road. There was also a need to improve the environmental standards of vehicle recycling practices in Canada. However, the evaluation evidence suggests that there is not a significant continued need for the

NVSP past March 2011. a) Did the NVSP respond to an identified need? What was that need?

According to evidence, there was an identified need for the NVSP. In 2007, there were

4.6 million model-year 1995 and older vehicles on the road; these vehicles produce

19 times more smog-forming pollutants than 2004 and newer models (Environment

Canada 2007a, p. 2). These pollutants include VOCs, NO x

, PM and carbon monoxide.

Accelerating removal of these vehicles from the road reduces these emissions. As well, by encouraging the use of sustainable transportation (e.g., public transit, bicycles), the program offers the potential to reduce GHG emissions.

Smog is the noxious mixture of ground-level ozone and PM, and is considered highly toxic to human health. Research indicates that higher levels of smog are associated with increased mortality and morbidity (Health Canada 2000), and increased risk rates for chronic bronchitis, asthmatic attacks and restricted activity days. Overall, these events translate into increased emergency room visits and hospital admissions for respiratory illness and cardiovascular complications. Another Health Canada document

(Health Canada 2003) indicates that smog can worsen existing heart and lung problems, especially among seniors who are most at risk. Children can also be more sensitive to the effects of air pollution, because of their lifestyle and the fact that their respiratory systems are still developing.

Smog also has negative environmental impacts, as vegetation health and productivity are harmed by pollutants like ground-level ozone (Environment Canada 2011a).



Furthermore, smog can harm wildlife, their habitat, and the availability and quality of their food supply.

Smog-forming emissions from individual vehicles have been greatly reduced through government regulation and the resulting technology improvements in exhaust systems.

In Canada, a 1995 or older-model vehicle produces 19 times more smog-forming emissions than a 2004 or newer model.

Replacing an older vehicle with a newer one does not necessarily reduce GHG emissions. Unlike smog-forming pollutants, GHG emissions cannot be reduced drastically through improved exhaust technology; GHGs are reduced by burning less fuel. This can be achieved by reducing personal vehicle usage, driving more fuel-efficient vehicles, or switching to sustainable modes of transportation like public transit and cycling. So, replacing an older vehicle with a newer and larger one will reduce smog emissions but could increase GHG emissions due to higher gas consumption. Thus, because the NVSP did not prescribe the replacement vehicle (or type of transportation), GHG reduction was only a secondary goal of the program.

In addition to reducing the impact of emissions from these older cars, there was a need to improve environmental practices among vehicle recycling companies. Harmful substances such as oil, antifreeze and mercury can be released into the environment if proper recycling measures are not implemented. Although there are municipal, provincial and federal laws that apply to this industry, there was evidence that some vehicle recyclers were not aware of the various acts and bylaws that were applicable to their operations.

The 2007 Budget recognized the value of scrappage programs, by providing funding to support and develop an initiative that removed older vehicles from the road. b) Is there a continued need for the program?

Although there was a documented need for the program in 2007, there is no longer any documented evidence to support a renewal of the program (with its current parameters) after March 2011, especially considering the natural retirement rate of the vehicles of these model years (see Figure 1 in section 2.1).



EC staff who were interviewed for this evaluation reported that, at program inception, there was a recognized need to accelerate the retirement of older high-polluting passenger vehicles, promote sustainable transportation alternatives through incentives, and promote responsible recycling of retired vehicles through a national code of practice that would improve standards and vehicle disposal practices. The NVSP was seen as supporting the CAA

’s clean transportation theme, and as complementing other nonregulatory initiatives previously announced under the ecoTRANSPORT Strategy (e.g., ecoAUTO rebate for fuel-efficient vehicles, the ecoMOBILITY program to help increase transit ridership and alternative transportation, and the ecoENERGY for Personal

Vehicles program to encourage Canadians to make better purchasing, driving and vehicle maintenance decisions). A national program was also considered to be a good way of consolidating smaller provincial programs.

A majority of the EC staff reported that there was not a continued need for the program beyond its end date. As of April 2011, the targeted vehicles (model-years 1995 and earlier) were more than

15 years old, near the end of their “natural” lives, and facing retirement through normal attrition. Continuation of the program was not seen as the best use of federal funds, a view confirmed by the number of participants who reported that they would have retired their vehicle irregardless of RYR (a finding presented with the results on the achievement of outcomes in section 4.3).

NFP respondents had a somewhat different perspective: they believed there were still more cars to be retired, particularly in rural areas where take-up of the program had not been extensive. NFPs believed that demand for the program is still high and that consumer readiness to retire their cars is not always in line with the timing of government programs. The provincial representatives also believed that there was a continued need for the program, as there are many model-year 1995 and older vehicles still on the road.

NFPs did not, however, necessarily express the view that a program should continue to exist at the national level, and at least three of the NFPs intend to continue their program at the provincial level just as they had before the national program was implemented. It was also acknowledged that if the program were to be reinstated in the future, there would need to be more emphasis on GHG reductions and more outreach about the importance of environmentally safe vehicle recycling. Education for the public and the vehicle recycling community was seen as being more critical than offering incentives. SI expressed the view that the political landscape is changing with respect to the way that the auto industry deals with older cars. In their view, NFPs and provincial governments would like to see a greater emphasis on product stewardship and manufacturer responsibility.

Need for code of practice . Respondents representing the vehicle recycling organizations believed that there was a continued need for the recycling code of practice. ARC and its provincial affiliates will continue to promote the code, and ARC committed to conducting audits for the next two years without EC funding. Beyond those two years, it is not clear if there will be funding to continue the audits. The Ontario

Automotive Recyclers Association has made adherence to the code a condition of membership in the association.



Relevance

2. Did the program complement or duplicate other federal and/or provincial government programs (i.e., Transport Canada programs, provincial emission testing programs, etc.)?

The evaluation examined this question using data collected through the document review and KI interviews.

Rating: Achieved

Findings: Although there were regional initiatives with similar objectives when the NVSP was implemented, none were of similar scope.

However, EC made efforts to reduce duplication by building on these initiatives.

There is no documented evidence of a similar program of national scope, and interview respondents indicated that there were no other federal programs duplicating the NVSP at the regional level. There were, however, a number of programs run by NFPs, such as

Car Heaven (national), the Lung Association in New Brunswick and Manitoba, BC

Scrap-

It, Faites de l’air in Quebec, and the Kidney Car Program (a fundraising initiative of the Kidney Foundation of Canada).

Many of these existing programs had different approaches to the NVSP and/or did not have the same scope. The environment was not always the focus, and the cars did not always get recycled (in some cases they were resold). For example, Car Heaven

(undertaken by the Clean Air Foundation / SI) targeted both

“live” (functioning) and

“dead” (non-functioning) vehicles (“dead” vehicles do not emit smog-forming pollutants).

It offered rebates on the purchase of a new vehicle exclusively with one vehicle manufacturer, free towing, and a nominal charitable income tax receipt. Car Heaven retired approximately 8000 live vehicles in 2006 and did not receive federal funding. The

Manitoba Lung Association and New Brunswick Lung Association ran programs in their provinces that offered a variety of incentives (e.g., transit passes, vehicle rebates) but no cash reward, and they did receive federal funding. The Kidney Foundation of

Canada’s program, which operated in seven provinces, collected mostly dead vehicles and offered free towing and a small income tax receipt (the amount of which varied by geographic area).

Seven of the pre-existing programs received funding from EC, but the scale of these operations was much smaller than the NVSP. For example, in 2006 only 3700 live vehicles were retired through the EC-funded programs. This represented a very small percentage of the estimated 5.5 million model-year 1995 and older vehicles on the road at that time, and there was an expressed need to accelerate the retirement rate of these highly polluting vehicles.



There were also a number of other programs (outside the CAA) that complemented the

NVSP, including:



Transit pass tax credit (federal)

 British Columbia’s Air Care mandatory vehicle inspection and maintenance program

(provincial)

 Ontario’s Drive Clean mandatory vehicle inspection and maintenance program

(provincial)

Relevance

3. Was the Vehicle Scrappage Program aligned with departmental and federal government priorities?



Findings: The program was generally consistent with federal priorities, and

EC’s mandate and priorities, related to air quality and climate change.

Documents indicated that the NVSP was aligned with departmental and federal government priorities. It contributed to the Government of Canada commitment to reduce emissions that contribute to air pollution and climate change, as stated in the

Government’s 2007 plan Turning the Corner: An Action Plan to Reduce Greenhouse

Gases and Air Pollution, which identified air emission targets for 2020 (from all sources combined). Through its CAA, the Government announced a series of regulatory actions in sectors such as industry, transportation, and consumer and commercial products.

The NVSP is one of the non-regulatory program measures under the clean transportation theme (see box below for a description of the CAA).

The Clean Air Agenda

Through the CAA, the Government of Canada has been working toward making tangible improvements in Canada's environment by addressing the challenges of climate change and air pollution. Since 2007, the CAA has been supporting regulatory initiatives in the industrial, transportation, and consumer and commercial sectors, and has been supporting a range of complementary measures designed to reduce GHG emissions, improve indoor air quality, mitigate the impacts of climate change, and engage at the international level.

The CAA is managed through a horizontal management, accountability and reporting framework. The 45 programs of the CAA are organized within eight themes: Clean Air Regulatory Agenda, clean energy, clean transportation, indoor


Audit and Evaluation Branch Evaluation of NVSP air quality, international actions, adaptation, partnerships, and management and accountability. The shared outcomes are as follows:



The health, economic and environmental benefits for Canadians have been realized.



The risks to the health of Canadians and the environment resulting from exposure to air pollution have been reduced.



The risks to communities, infrastructure and the health and safety of

Canadians resulting from climate change have been reduced.

Source: http://www.tbs-sct.gc.ca/hidb-bdih/initiative-eng.aspx?Hi=12

The NVSP complemented actions announced under the Regulatory Framework for Air

Emissions, such as the mandatory fuel-efficiency standard beginning with 2011 modelyear vehicles and the proposed notice under the Canadian Environmental Protection

Act, 1999 to require the preparation and implementation of pollution prevention plans for mercury released from end-of-life vehicles. It also complemented initiatives under the ecoTRANSPORT Strategy, and was intended to be cross-promoted with other programs designed to encourage emission reductions from transportation (ecoAUTO rebate, ecoENERGY for Personal Vehicles, and ecoMOBILITY).

Relevance

4. Was the Vehicle Scrappage Program consistent with the D epartment’s mandate and with federal roles and responsibilities?



Findings: The NVSP was aligned with EC’s strategic outcome related to pollution minimization, and was consistent with departmental and federal roles and responsibilities.

The NVSP was aligned with EC

’s strategic outcome “Threats to Canadians and their environment from pollution are minimized” under sub-sub-activity 3.2.2 “Climate Change and Clean Air Partnerships” in EC’s Program Activity Architecture. EC staff indicated that the NVSP was consistent with the Department

’s mandate and its federal roles and responsibilities. It complemented actions announced under the Regulatory Framework for Air Emissions, such as the mandatory vehicle fuel-efficiency standard and implementation of pollution prevention plans for mercury released from end-of-life vehicles. The program responded directly to key government priorities and Canadian concerns about air quality.

The federal government was seen as being uniquely positioned to fund national programs of the scope required by the NVSP. Federal support allowed for a national approach that was more effective than individual vehicle scrappage programs


Audit and Evaluation Branch Evaluation of NVSP implemented locally at the provincial or regional level (as was the case before the launch of the NVSP).

4.2 Performance – Design and Delivery

Evaluation Issue: Design and Delivery (under Performance)


Evidence indicates that the program objectives were well defined up front, including precise overall program targets (such as 200 000 vehicles retired over four years). However, these four-year targets became unattainable because the full national program was not launched until January 2009. Annual targets (such as 50 000 vehicles per year) were met or exceeded once full implementation was achieved. The objectives for the recycling code of practice were realistic, well defined, and successfully achieved.

One of the key design features of the program was the use of a contribution agreement with a national NFP to administer the program, and the use of a network of provincial NFP partners to deliver the program. These features were shown to be effective and efficient overall. The advantages included faster implementation, flexibility, and reduced duplication with other programs in other jurisdictions. As a result of this delivery approach, participant satisfaction with program administration and the incentives was high during the program’s early stages, and increased to a very high level as the program matured.

Evidence indicates that the program achievements were captured effectively and efficiently through an online data system, comprehensive surveys, and ongoing tracking of vehicle recycler audits. A wide range of communications activities undertaken to communicate program performance and the achievements of the program were successful at reaching key stakeholders and the general public.

Comprehensive program performance data were collected through the SPARC system, quarterly and annual reports, surveys, and various program management tools such as spreadsheets to monitor vehicle recycler audits.

Key lessons learned included: the effectiveness of using an NFP for program delivery; the need for hands-on involvement by EC in managing a large, complex contribution agreement; the need to allow adequate time to reach full implementation of a complex program of this size and scope and involving many partners; and the importance of clearly defined roles, responsibilities and lines of communication.



Performance – Design and Delivery

5. Were the objectives of the program realistic and adequately defined?

The evaluation examined this question using data collected through the document and literature review and KI interviews.

Rating: Progress Made; Attention Needed

Findings: Program objectives were well defined. However, the four-year program target of retiring 200 000 vehicles overall became unattainable because the full national program was not launched until January 2009, 21 months later than initially expected.

Annual targets (50 000 vehicles per year) were met or exceeded once full implementation was achieved. The objectives for the recycling code of practice were realistic, well defined and successfully achieved.

As presented in section 2 of this report, the program’s objectives were clearly defined in program documentation. According to documentation, the NVSP’s primary goal was to reduce air pollutants by removing older, high-polluting personal vehicles from the road.

Secondary goals were to reduce GHG emissions by promoting low-emission alternative transportation to prevent the release of toxic substances into the environment

(Environment Canada 2008, p. 4). As presented earlier in this report, the program had developed a logic model, and specific targets (e.g., removal of 50 000 vehicles per year).

EC staff agreed that the program’s objectives were clear and that the annual target of scrapping 50 000 cars per year was reasonable for the program once it was fully operational. In fact, this target was exceeded for 2009-2010 and 2010-2011. The original four-year target of retiring 200 000 vehicles over the life of the program was not attainable because RYR was fully operational for only 27 months, not 48. It took close to one year to receive policy and funding approval, and almost another year to develop and launch the program and to achieve full implementation.

Overall, NFPs thought that the objectives of the program were realistic and well defined, and that there were good linkages between the objectives, activities and outcomes set for the program. One NFP described the program targets as “ambitious” and noted that setting high targets “was the right thing to do” because they were achieved and exceeded.

Similarly, provincial government representatives thought that the program objectives were realistic, and agreed that there are strong linkages between RYR program objectives and activities. One provincial government representative attributed the good linkages between program objectives and activities to the pre-existing programs that provided a good basis from which to develop RYR targets. Another thought that the pollution reduction targets could have been more aggressive.



Vehicle recycling code of practice . The development and implementation of a national recycling code of practice was a realistic and practical fit with the program objective of recycling vehicles using environmentally safe practices. Prior to the launch of the NVSP, ARC had been interested in developing a national code of environmental practices and had approached the Car Heaven program to discuss the opportunities for a national code. The auto recycling industry already had a code of practice in place in

B.C. and a set of best practices in Quebec. Industry representatives also were aware of similar work being carried out in the United States. This previous work provided a solid platform for ARC, EC and other stakeholders to work on the development of a new recycling code of practice. The EC-funded contracts with ARC greatly accelerated and expanded the process of developing and implementing the recycling code of practice nationwide.

All KIs reported that the development and implementation of the recycling code of practice represented a large and successful step toward achieving the objective of using environmentally sound practices to recycle cars. The program required all participating auto recyclers to comply with the code, which was verified by independent audits. Some

350 of the approximately 370 auto recyclers who participated were audited through the program, and of those 350, 92% passed their audits. The reasons that not all recyclers could be audited were a higher participation by recyclers than anticipated and a delay in awarding the contract to conduct the audits.

Performance – Design and Delivery

6. a) Was the design of the Vehicle Scrappage Program appropriate for achieving expected program results? b) Was the funding of the program commensurate with its objectives and the needs identified?

The evaluation examined these related questions using data collected through the document and literature review, KI interviews and case studies.


Findings: Key design features were as follows: EC funding through a contribution agreement with a national NFP to administer the program, and use of a network of provincial NFP partners to deliver the program. These features were proven to be effective and efficient overall. The advantages included faster implementation, flexibility when making adjustments, and reduced duplication. Program funding for cash incentives and program operations was adequate.


Audit and Evaluation Branch Evaluation of NVSP a) Was the design of the Vehicle Scrappage Program appropriate for achieving expected program results?

Overall, EC and other key partners and stakeholders believe that the design of the program was appropriate for achieving the expected results.

At the program’s outset, EC established linkages between key features of the program design and expected outcomes, shown in Table 5.

Table 5: Linkages Between Program Objectives and Activities

Program Outcomes/Targets

Scrappage of 50 000 in-use vehicles per year of operation

Annual reduction of smog-forming emissions: approximately 2250 tonnes of

NO x

and VOCs, and 54 000 tonnes of

GHGs

Program Activities

Accelerating the retirement of model-year 1995 and older vehicles is the main program activity.

A calculation methodology was developed to estimate impact on emissions from the retirement of an older vehicle and its replacement.

Incentives related to sustainable transportation were promoted.

A reduction of GHG emissions through increased use of sustainable transportation

An increase in awareness and understanding among Canadians of the environmental impacts of older vehicles

A reduction in the release of toxic substances

SI, provincial NFPs and EC undertook outreach and promotion activities

Implementation and use of a recycling code of practice is a key element of the program.

Development of a vehicle recycling code of practice that will lead to higher standards nationally and that may be adopted by provinces and territories

A 3:1 leveraging of federal funding by partners for incentives

Offering attractive incentives is a key element of the program.

Source: NVSP Accountability, Risk and Audit Framework, 2008

One of the program’s key features was the involvement of one national NFP and eight provincial NFPs to deliver the NVSP. There was widespread agreement among KIs that successful implementation would not have been possible without partner contributions, including NFPs and provincial government support. In fact, delivery of the program involved many partners that played key roles in its success:



The national-level NFP (SI) had knowledge and experience gained from running Car

Heaven, first in Ontario and then nationally. SI also used its marketing and public education expertise to attract auto manufacturers and the Bicycle Trade Association of Canada as incentive providers. There was a consensus that an NFP national


Audit and Evaluation Branch Evaluation of NVSP coordinator was able to be more flexible and respond more quickly than a federal department in certain situations.



Some provincial-level NFPs had prior experience running vehicle scrappage programs, i.e., BC Scrap-It, Climate Change Central (Alberta), the Manitoba Lung

Association, L’Association québécoise de lutte contre la pollution atmosphérique

(AQLPA) and the New Brunswick Lung Association. These NFPs were able to contribute their expertise and experience to EC for program design and planning, create partnerships with local incentive providers (especially transit authorities), provide knowledge of their local market, and capitalize on relationships with local media.



The B.C., Quebec and Manitoba governments also took an active role in supporting and developing the program, contributing approximately $14 million for additional incentives (based on program data).



ARC played a key role in vehicle recycling, having been contracted by EC to develop the recycling code of practice in order to ensure environmentally responsible disposal of all vehicles retired through the program. All partners recognize that this code has become a valuable program legacy. ARC also provided strategic knowledge about the vehicle recycling industry in Canada.



Vehicle manufacturers started to offer rebates on the purchase of a new vehicle in

August 2009. These rebates had a positive impact on program participation: the number of daily applications received doubled after the rebates were introduced

(based on program data).



The Canadian Urban Transit Association provided their transit expertise to develop and deliver a public awareness campaign in 62 transit systems across the country.

The campaign aimed to raise awareness about the environmental impacts of older vehicles and linked RYR and public transit use with clean air benefits.



More than 350 vehicle recycling companies were RYR partners.



Other participating incentive providers included bicycle retailers and public transit companies.

EC representatives believe that the program’s successes have proven that its design and approach, especially the contribution to SI, were appropriate. From EC ’s perspective, working with a national-level NFP made it easier because of the single point of contact for the NFP network, and working under a contribution agreement provided flexibility for SI to conduct program operations. The contribution to a single national organization also allowed EC to leverage the expertise and resources of an organization with experience in the area. The other key aspect was the involvement of the provincial NFP network, which leveraged local resources and knowledge while reducing potential duplication.



According to interviews, the governance structure, which involved hands-on management by EC, was highly effective. A departmental representative commented that

“this is not a hands-off process.”

SI representatives strongly agreed that using a network of partners is a good way to set up this type of program. From their perspective, the challenge of operating a national program was accommodating provincial programs and allowing the different programs to operate in different ways (e.g., with different databases). Provincial NFPs also strongly agreed that it was beneficial to have local participation in each province instead of the program being conducted solely by a national organization. EC provided overall program direction and policy advice, while promoting the program through the departmental website and news releases (which helped give the program a national presence). SI provided national leadership and coordination. The advantage of having provincially based organizations was that they could address the differences in each province and adapt to provincial needs. Pre-existing programs meant that provincial

NFPs had already established relationships with auto recyclers and incentive providers as well as a public profile. These NFPs also had experience with program promotion and advertising.

ARC reported that the design and planning involved in developing the recycling code of practice was effective. The project depended on partnerships between ARC and its provincial affiliates, SI, EC (which funded two contracts with ARC related to code of practice development and implementation), and auto recycling companies.

Operational Challenges Associated With Program Design

Roles and responsibilities . Two operational assessments were conducted for SI during the program: one overall by CBW Associates, and one by the Taylor Research

Group for the call centre, in 2010. According to the CBW study, the roles, responsibilities and expectations were not clearly defined. According to the call centre assessment, the responsibilities of the call centres (and who gives them direction) were neither clear nor consistently understood. Some people were not clear on whom to interface with at SI to obtain help or resolve issues, and thus they were not confident that their issues were being addressed. According to the reports, NFPs wanted to be consulted more extensively (e.g., on national incentives), and communicated with in advance rather than after decisions had been made.

From a provincial perspective, some NFPs thought that SI, during the initial stages of the program, was not flexible enough in adapting to their existing operations and that the roles and responsibilities were not clear. They said that some staff turnover at SI added to the problem.

Other issues pertaining to roles and responsibilities as well as communications arose from EC having had previous relationships with NFPs. EC had relationships with some of the provincial NFPs before SI became involved with the introduction of RYR, which created some communications problems for people who were used to dealing with EC directly instead of following the RYR reporting relationships and communicating with SI.

The mid-program review by CBW helped identify this problem, and the provincial NFPs


Audit and Evaluation Branch Evaluation of NVSP were given clear instructions to communicate with SI about any administrative, procedural or contractual issues.

There were also communications problems that resulted from the reporting relationships between SI and NFPs that contracted with external call centres (i.e., in Ontario,

Saskatchewan and the Atlantic provinces). SI introduced monthly conference calls with the call centres and NFPs to deal with processing issues and complaints.

The mid-program review identified some of these above issues, and steps were taken by EC and SI to address them. Most NFPs agreed that the actions taken following the review were effective at addressing problems that occurred during the initial stages of the program. These actions included the following:



EC clarifying the process for exceptions to vehicle eligibility criteria



SI establishing standardized hours for call centres across the country



EC clarifying the official language responsibilities of SI and the other NFPs



Implementing service standards



Establishing an overflow call centre

Incentives . Some NFPs expressed concerns about the impacts of certain incentives on the GHG reduction goal (which was more important to some NFPs than was stated in the program design). Their concern was that rebates on the purchase of a new vehicle led to higher GHG emissions, as most manufacturers provided higher incentives for purchasing larger vehicles (e.g., larger cars and light trucks).

Reach . The program was implemented in all regions except in the three territories. EC had wanted to launch a pilot project in the north and asked SI to develop a plan, but SI determined that it would be too difficult due to the absence of a suitable NFP network to administer the project, a lack of auto recyclers, and a limited number of auto dealerships or outlets to enable provision of incentives to more remote areas. Therefore, a decision was made to not proceed with a pilot project. b) Was the funding of the program commensurate with its objectives and the needs identified?

According to interview findings, program funding through the contribution agreement between EC and SI was sufficient to deliver the program. As part of the program design, a broad objective was set to split the overall spending under the contribution agreement, with a maximum of 25% to cover operating costs and the remaining funding to be allocated to incentives. By program end, the cost split was 30% and 70%, respectively.

SI

’s increased operating costs were attributed to the extra advertising that it delivered.

This additional program promotion was a partial replacement for advertising cancelled by EC.

EC

’s O&M budget was sufficient for the cost of typical expenses for EC staff (e.g., telecommunications, office supplies, training), to support EC monitoring activities such as audits of vehicle recyclers for adherence to the recycling code of practice, a recipient audit of the contribution agreement between EC and SI, vehicle data, some travel, and program evaluation.



The funding agreements between EC and SI and between SI and provincial NFPs required the establishment of separate bank accounts for funds allocated to incentives and for program delivery costs. This requirement allowed for close monitoring of program funds and was particularly useful for managing operating costs. EC also provided funding directly to local NFPs for program delivery on an interim basis between

April 1, 2008, and December 31, 2008, until SI’s exclusive arrangement with an auto manufacturer ended.

Overall, NFPs reported that funding for the program was sufficient to achieve its objectives. One provincial NFP described the program as “one of the cheapest programs to run.

” Some NFPs, however, reported that the amount of work to run the program was greater than what they expected at the beginning. Some reported that the implications of requests by SI and EC requirements were not fully understood, with the result being increased costs for the NFPs. One provincial NFP reported that it “struggled on the operational side,

” particularly with human resources and covering the costs of staff time. Another reported that management of the program was more demanding than expected.

Amounts for incentives . Initial budget targets for incentives were based on the population of the province or on the estimated number of older vehicles in the province.

Allocation of operational funds was proportional to the number of vehicles to retire. Two

NFPs said that the budgeting formula did not work as well for their organizations because they operated in smaller provinces. They reported that the amount of funding did not allow them to achieve their provincial targets for vehicles recycled.

Advertising . Despite the decision by EC to not invest in a national advertising campaign, EC and SI believe that the advertising and promotion activities conducted by

SI and local NFP partners, along with media coverage, were sufficient to generate broad public awareness of RYR.




7. a) Were participants satisfied with the administration of the program by both the Department and the not-for-profit organizations? b) Were participants satisfied with the incentives provided to reward them

for scrapping their vehicle (type and value)?

The evaluation examined these related questions using data collected through participant surveys and case studies.


Findings: Participant satisfaction with program administration and the incentives was high during the program’s early stages, and increased to a very high level as the program matured. a) Were participants satisfied with the administration of the program by both the

Department and the not-for-profit organizations?

Participant satisfaction was monitored through various means, including surveys of program participants. According to the survey reports, satisfaction with the NVSP steadily increased over the program lifespan. Overall satisfaction (the proportion of respondents completely and mostly satisfied) rose from 77% in 2009 to 89% in

December 2010 (see Figure 4).

Figure 4: Level of Program Participants ’ Satisfaction with the NVSP

64

48 51

2009

January 2010

December 2010

29 30

25

12 12

7 7

5

2

4 3 2

Completely satisfied

Mostly satisfied

Somewhat satisfied

Not very/ not at all satisfied

Too early to say/ dk/na

Source: Surveys of NVSP Participants, Environics Research (1000 respondents for each time period)

As indicated in the survey responses in Figure 5, the level of satisfaction with call centre staffs ’ ability to answer questions, their level of knowledge, and their capacity to provide bilingual services generally increased between 2009 and December 2010.



Figure 5 : NVSP Participants’ Views on Program Delivery at Call Centres

RYR rep able to answer questions

2009 43 39 7 11

January 2010 62 32 4 2

December 2010 64 29 4 1

2009

RYR rep was knowledgeable

40 38 6 15

January 2010 62

64

30 4 5

30 33 December 2010

Able to receive service in language of choice

2009

January 2010 63

December 2010

2009

55

3213

38 41

58 37 23

January 2010 63 3213

December 2010 66

Strongly agree

Agree

Disagree/strongly disagree dk/na


*

Less than 1% for “disagree/strongly disagree” timely fashion also increased as the program matured.

Strongly agree that vehicles were picked up in a timely fashion and that incentives were received in a

Agree




Figure 6: NVSP Participant s’ Views on Timeliness of Vehicle Pickup and Receipt of Incentives

2009

January 2010

December 2010

42

52

55

33 9 15

35 11 3

33 8 4

2009

January 2010

December 2010

41

39

55

42

46 11 1

172

37 7 1

Strongly agree

Agree


Strongly Agree

Agree

Disagree/Strongly Disagree dk/na


Overall, these findings indicate that the level of satisfaction increased as the program matured. The increase in participant satisfaction coinci ded with the program’s efforts to address initial operational issues, uncovered by the mid-program review.

Vehicle recyclers . For ARC, the two contracts with EC made it possible for them to develop the code of practice, and train and deploy auditors of vehicle recyclers across the country. A third contract with SI allowed ARC to design and deliver training for the code. ARC was satisfied with how the contracts were administered by EC and SI.

Auto recycling companies were satisfied with the program’s administration, including the process to receive vehicles. One company reported that some minor

“administrative


Audit and Evaluation Branch Evaluation of NVSP glitches ” at the program’s beginning were resolved once the online system was implemented, stating that “There were no issues at the end.”

Auto recycling companies also indicated that they found the recycling audit process fair and that it helped them to improve their businesses. b) Were participants satisfied with the incentives provided to reward them for scrapping their vehicle (type and value)?

Although the incentives package varied among regions, all participants could choose the $300 cash incentive, and many had access to other forms of incentives such as public transit passes or discounts on new bicycles. Most participants opted for the cash incentive, according to program data. The number of participants who opted for the cash incentive rose through the life of the program (from 66% in 2009 to 85% in

December 2010). Satisfaction with the incentives among the participants surveyed (i.e., those indicating very satisfied or somewhat satisfied) was 94% in December 2010. Four in ten participants received a new vehicle manufacturer ’s rebate or discount in 2010, and were more likely (64%) than those who did not receive a rebate or discount (57%) to be

“very satisfied” with the value of their incentive.

Performance

– Design and Delivery

8. Were the achievements of the Vehicle Scrappage Program effectively and efficiently captured and disseminated (disseminated to appropriate audiences, i.e., key stakeholders and the general public)?

The evaluation examined this question using data from multiple sources, including the document review, KI interviews and case studies.


Findings: Program achievements were captured effectively and efficiently through in-depth monitoring of real-time transactions and program performance with the SPARC system, comprehensive surveys of participant satisfaction and public awareness, and ongoing tracking of vehicle recycler audits. A wide range of communications activities undertaken to communicate program performance and achievements were successful at reaching key stakeholders and the general public. These activities included workshops, webinars and conference calls among the NFP network, providing all partners with access to the SPARC system, presenting results such as the Clunker Counter on the

RYR website, and press releases from EC and NFPs.

SI and EC undertook a number of initiatives to effectively and efficiently capture and disseminate achievements:





An online survey and select in-person interviews were conducted with all provincial

NFP partners, SI, EC, and the call centres (as part of the CBW Associates assessment study).



A Call Centre Assessment Study (conducted by the Taylor Research Group) assessed the effectiveness of the interface with the participants through call centres.



Program process and results monitoring through the SPARC system was carried out. NFPs and call centres entered program delivery information directly into

SPARC. Performance information and information on vehicles registered and retired through the program was available in real time via the SPARC database. The

SPARC database was initially developed by SI for the Car Heaven program, and was expanded and modified to meet tracking needs for RYR, including opening the database for use by local delivery partners. To ensure data accuracy and reliability,

SI provided support and training to provincial NFPs in the use of SPARC.



SI hosted NFP partnership workshops to discuss key issues and build rapport within the network. Topics reviewed included marketing and outreach, financial processes, improved service standards, program evaluation, new database functionality training, environmental impact of the program, and exit strategy.



In addition to monthly conference calls with the regional NFPs, calls with the call centres and finance managers were conducted by SI. Additional webinars were scheduled to discuss one-off initiatives as needed.

According to interviews, communications to stakeholders about program achievements were extensive. EC presented program data at workshops and by email to the NFPs, including, for example, data on the number of vehicle rebates chosen, projected program participation, and recycler audit results. SI also provided NFPs with monthly reports on the numbers of applications accepted, vehicles permanently retired and other indicators. SI hosted semi-annual workshops with NFPs, held regular conference calls, and communicated by email and telephone with NFPs to disseminate program information.

Communications to the general public about the program and program achievements took place in a variety of ways:



The RYR website included the Clunker Counter, which displayed the number of cars retired and tonnes of smog reduced in real time.



Customer service standards were posted on the RYR website. The standards were implemented in mid-2010 after the mid-program review made it evident that there was a need to standardize processing times and communicate these standards to the public. These standards were described by one KI as

“crystal clear.”





Quarterly “results so far” were added to the website for the general public, including the number of applications accepted, vehicles retired by each province, and participating recyclers.



News releases were issued, including national news releases from EC

(announcing the program in June 2008, celebrating the launch in January 2009 and publicizing the receipt of 50 000 applications in November 2009), and from SI

(announcing the program in April 2009, publicizing the new bicycle incentive in

June 2009, and celebrating 100 000 vehicles retired in October 2010). Local news releases covered the activities of provincial organizations. A provincial NFP reported that SI was very good at preparing news releases and engaging incentive partners in public relations efforts.



Facebook and social media were utilized, including a Facebook account created in

June 2010 and updated regularly, and local groups creating Facebook and Twitter accounts and posting videos on YouTube.



NFP outreach was carried out. Each NFP developed an outreach program that included various activities, such as holding events to raise awareness and having a program booth at summer fairs, festivals and auto shows.



NFP advertising was carried out. The local NFPs did regular print, online and radio advertising; limited television ads ran in certain provinces.



EC cancelled the original plan for a national advertising campaign regarding the pollution impacts of older vehicles. However, SI conducted two national advertising campaigns on radio, online and in print (one in February and March of 2010 to drive up participation, and the other in February 2011 to remind Canadians that the program was ending on March 31, 2011). SI also ran a series of smaller public relations campaigns.

NFPs expressed favourable opinions about the dissemination of information overall.

The website was characterized as “very good” and the marketing as “good,” and one

NFP described the information sharing as

“fantastic.” Public survey data on how people were learning about the program (e.g., by ads, word of mouth or websites) helped some provincial NFPs develop marketing strategies.

It was noted, however, that the NFPs would have preferred more credit being given to the provincial providers in SI’s national marketing initiatives. Some NFPs indicated that they would have liked more sharing of best practices with other provinces so that provinces would know what was taking place nationally.

Some NFPs also indicated that they would have benefited from more timely receipt of information about

“some of the broader issues,” such as how the vehicle manufacturers' incentives affected the program and how potential participants were learning about the program. These NFPs believed that they may have been able to adapt their marketing initiatives more quickly with more timely information.



Performance


9. Have appropriate performance data been collected, captured and safeguarded? If so, has this information been used to inform senior management / decision makers?

The evaluation examined this question using data from multiple sources, including the document review, a review of program performance data, and

KI interviews.


Findings: Comprehensive program performance data were collected through the SPARC system, quarterly and annual reports from SI to EC, a mid-program review, participant and general public surveys, and various program management tools such as spreadsheets to monitor vehicle recycler audits. The program monitoring and performance data were used extensively by program managers to direct and fine-tune the program. The value of the monitoring and performance data and the effectiveness of

EC managers

’ use of these data were demonstrated by the significant increases in program performance and participant satisfaction as the program progressed.

Two public opinion surveys of the general public were conducted to assess awareness of the NVSP, and three surveys of program participants were conducted to assess program delivery and results. This allowed the program to monitor the awareness campaigns as well as program delivery and results.

Financial monitoring was conducted in accordance with the contribution agreement. In the program’s early stages, EC had concerns about SI’s capacity to monitor and report the financial performance of its activities, which SI eventually acknowledged, and for which SI then developed capacity. SI had concerns about some of the provincial NFPs ’ capacity to report financial results and use SPARC: SI reported that provincial NFPs initially did not understand the financial complexity of a program of this type and size, and SI believed that they needed to strengthen their human resources and financial management capacity. After substantial efforts by SI and EC, SI and the NFPs made significant improvements in meeting contractual deadlines and deliverables.

Financial monitoring for a program of this size and complexity requires appropriate resources and expertise. Although problems with expense monitoring issues noted during the financial audit were largely corrected, projecting participation remained problematic, and in March 2011 the program was uncertain that the allocated funding would be sufficient. The funding turned out to be sufficient, but SI and EC had to develop contingency plans.



All partners and stakeholders agreed that EC was effective at program monitoring.

There was a consensus that EC program staff were diligent and effective at working with NFPs, and other partners such as the automotive recycling industry, to ensure that the program remained on track and that the results were being achieved. One provincial

NFP described the monitoring procedures as “much more stringent than any other program we’ve been involved with.”

Overall, EC project monitoring was more intensive than for other typical contribution agreements, because of the program’s complexity, the amount of funding, and the risks associated with delivery by an NFP that did not have previous experience in a project of this scope and size. According to interviews with EC staff, EC monitoring activities included the following:



A third-party financial audit examined SI

’s financial management and verified compliance with the contribution agreement.



Daily monitoring of the SPARC database provided program results in real time.



Daily phone calls and emails between EC and SI took place to discuss operational issues.



In-person meetings with SI for longer-term planning took place approximately six times per year.



Six workshops were held with SI and provincial/regional NFPs for more in-depth discussions of program-wide issues: two in Toronto (fall of 2008 and spring of 2009), and one each in St. John ’s (fall of 2009), Vancouver (February 2010), Montreal (April

2010) and Ottawa (September 2010).



Automatic Google Alerts provided EC with information about program coverage by electronic, print and broadcast media.

SI also monitored program activities:



Quarterly reports (nine) and annual reports (two) were provided to EC by SI in order to document ongoing program activities and meet the requirements of the contribution agreement.



Webinars were conducted by SI to instruct provincial NFPs on using the SPARC database.



Three surveys of participants were conducted.



A mid-program review was conducted that focused mainly on identifying areas of possible improvement in NFP and call centre operations (assessment studies by

CBW Associates and the Taylor Research Group).



Two public opinion sur veys were conducted to gauge the Canadian public’s awareness of scrappage programs and related issues.

Monitoring was used for decision-making purposes. For example: EC monitored the value of scrapped steel to ensure that recyclers were paying fair prices for cars picked up through the program; data on call centre volumes helped determine the amount to charge NFPs for services provided by the overflow call centre; and the assessment studies conducted by CBW Associates and the Taylor Research Group led to significant operational changes, such as a review of roles and responsibilities, development of standard operational procedures, and the opening of a national call centre to handle overflow.



The results of the recycling audits were summarized on an ongoing basis in spreadsheet files. These results, which were aggregated periodically by EC, confirmed to the Department that auto recyclers were adhering to the recycling code of practice in the great majority of cases. Close to 400 recycler audits were conducted.

The case study of communications found that multiple ads were placed on local radio stations, local and national websites, and in various written publications across the country. According to an RYR Media Coverage Report (November 2010), there were 24 million media impressions 1 from the

“Spring Cleaning” campaign.

Performance


10. a) What are the lessons learned (both strengths and weaknesses) from the program? b) Are there follow-up activities or programs that could build on the success achieved by this program?

The evaluation examined these related questions using data collected through the document review, KI interviews and case studies.


Findings: Many lessons were learned about program design and planning, implementation and delivery, and ways to increase effectiveness.

Key lessons included validation that the use of an NFP organization can be an effective approach for program delivery; the need for hands-on involvement by EC in third-party delivery arrangements; the need to allow adequate time to reach full implementation of a complex program involving many partners; and the importance of clearly defined roles, responsibilities and lines of communication. Follow-up activities that could build on the program

’s success include support to continue the legacy of the vehicle recycling code of practice. a) What are the lessons learned (both strengths and weaknesses) from the program?

Delivery model . The delivery model of one national NFP and local delivery partners worked very well. The use of a contribution agreement instead of a contract (or set of contracts with NFPs) allowed for flexibility in program operations, and meant that SI could be responsive to partners and participants while still having oversight from EC.

The provincial NFPs added credibility and were better able to build regionally specific partnerships. NFPs described this program model as “the way for the non-profit sector

1 Media impressions are measured by the number of people who have seen an article, heard something on the radio or in a podcast, watched something on television, or read something on a web page or blog.


Audit and Evaluation Branch Evaluation of NVSP and government to work together.

” To improve the implementation of this delivery model, SI recommended developing a process map at the beginning to give all partners a clear understanding of their roles and responsibilities from the outset.

For the delivery model to be successful, there was a need for personnel training across the NFP network, ensuring capacity for financial management, and service delivery in both official languages. Program managers reported they now have a much better understanding of the skill set required by NFPs to deliver national programs and that there should be a

“checklist” of these requirements in future contribution agreements with NFPs.

A call centre to handle overflow .

An overflow call centre was established to provide more consistent and responsive customer service (by substantially reducing the need for voice mail), and improve services in both official languages across the country. This was a very successful initiative, and SI recommended that a national call centre be used in any future programs of this type.

Variety of incentives . The number and variety of incentive partners helped to create higher-value incentives for participants. Provincial government involvement especially helped support sustainable transportation initiatives and made these more attractive to program participants. The cash incentive was also attractive to many participants, possibly because it was the only incentive available to everyone regardless of whether they were rural or urban, or whether or not they were transit riders.

Some NFPs found it difficult to understand the administrative procedures of large potential incentive-provider organizations (e.g., the Toronto Transit Commission).

Importance of service standards . Service standards are very important, both to participants and partners. Clear service standards (e.g., for processing times, car pickup, payments) were introduced and communicated in August 2010 following the mid-program review. KIs agreed that it would have helped to have service standards established at the outset, even if the standards evolved over time.

Need to allow time for start-up and transition to full implementation . KIs agreed that setting up and implementing a national program needs a longer lead time than was originally anticipated. EC and SI signed the contribution agreement in March/April 2008, and envisioned that everything would be in place by June 2008. For SI, the provincial

NFP partner negotiations took longer than expected, and their exclusive arrangement with an auto manufacturer under the Car Heaven program until December 2008 also created delays in launching the program. This meant that an interim approach was adopted for the fall of 2008, where EC worked directly with six provincial NFPs.

EC managers believe that implementing a program like the NVSP takes approximately

18 months, which includes time to complete the TB Submission. This means that a program with 48 months of funding will have approximately 30 months of full program activity. This delay was not accounted for when program targets were set (50 000 retired vehicles per year for four years).



Ongoing monitoring . The program implemented effective monitoring tools that allowed timely improvement of the program, such as the mid-program review, which led to the introduction of a national call centre; the financial audit of SI, which led to greater financial accountability; and participant surveys, which led to service improvements. The use of an online database (SPARC) allowed real-time and accurate monitoring of program results.

SI also introduced regular and frequent communications with the NFP network to deal with problems and conflicts that emerged. For example, hosting NFP partnership workshops was , according to SI’s 2010 annual report, an effective way to discuss key issues and build rapport within the team.

Exit strategy . The Treasury Board Secretariat

’s requirement for an exit strategy led to a proactive program wind-down. Some funding was transferred to 2011-2012 to support program wind-down when it was understood that the program could not end and wrap up completely by March 31, 2011. The 2011-2012 funding will be used to disperse final cheques to participants, create final reports from NFPs and SI, and ensure all vehicles are picked up. EC and SI described the planning for the exit strategy, which was in place from the start, as very valuable.

Marketing and promotion by NFPs . The marketing and advertising guidelines created by EC ’s Communications Branch gave NFPs the flexibility to develop advertising and promotion without ongoing oversight by EC. KIs indicated that many of the advertisements and communication materials from provincial/regional NFPs were very strong, and that their local-level outreach and public relations have been effective at promoting the program and raising awareness. KIs said that it would have been useful if there had been some mechanism or forum for sharing advertisements and promotional materials across the NFP network.

Communications . The NVSP involved many stakeholders at different levels. A number of lessons were learned with respect to communications among these stakeholders, including the following:



Certain roles and responsibilities were unclear, such as those for the approval of news releases and the division of responsibilities between SI and the local NFPs.



Although having a third-party partner (SI) leading the communications was effective, some government staff involvement should still be expected, given th is activity’s strategic aspect. This involvement needs to be clarified up front, ideally in the contribution agreement. The agreement needs to be clear about roles and responsibilities, especially with respect to approvals and their timelines. The agreement should also specify that all communications need to be consistent with federal guidelines and standards.



The involvement of EC program staff as the main contact with the third-party deliverer (SI) (with EC ’s Communications Branch playing more of a supporting role) was an effective arrangement.



The third-party deliverer (SI) was fully engaged and had experience in marketing and program delivery. Without this training and expertise, the risk to the program would have been greater given its high reliance on third-party delivery and its exposure to the general public.


Audit and Evaluation Branch Evaluation of NVSP b) Are there follow-up activities or programs that could build on the success achieved by this program?

SI and its network of local delivery organizations were given the option to continue delivering the National Vehicle Scrappage Program after April 1, 2011, without federal funding. To date, SI has announced its intention to restart its Car Heaven national program (which they regard as an SI legacy program), and delivery organizations in

British Columbia and Quebec will run provincial programs, likely with support from their provincial governments.

Continuing the legacy of the vehicle recycling code of practice . All KIs consider the recycling code of practice to be an important and valuable legacy of the NVSP, and would like it to continue.

In 2011-2012, SI will continue working with the code of practice, by promoting the code to recyclers across Canada and informing Canadians of the environmental and health benefits of selecting a code-of-practice-approved recycler. They will also be investigating means of expanding the code to other complementary industries (e.g., steel recyclers).

ARC will continue supporting the code of practice after the program ends. ARC would like to increase the pass threshold for the code over time as the industry becomes more familiar with the code and as auto recyclers ’ environmental practices improve. ARC has also committed to conducting recycling audits for 18 months in Ontario. As well, audits are expected to continue in Alberta and the Atlantic provinces. ARC believes that the developmental stage is complete , having stated that “90% of the heavy work has been done ,” and now the priority is to engage in continuous improvement of the code. ARC stated that “continuity is key to have growth in the code of practice.”

For the longer-term viability of the code of practice, ARC stated that it needs a commitment from all partners, including EC, to keep improving the code and make it a continuing success. ARC stated that the work of recruiting and hiring the auditors of vehicle recyclers on a continuous basis will be a challenge, and that they are not sure how this will happen or how it will be financed. ARC recommended setting up an advisory board with representatives of ARC, SI and EC, to upgrade the code of practice on an ongoing basis.

In ARC’s opinion, the Government of Canada (specifically EC) should be involved in two ways: 1) on a standards committee (which would have a low cost and would need no financial support); and 2) on an investigative committee to direct the audits (which would need financial support).

Provincial NFPs agreed that continued support for the vehicle recycling code of practice and for the audits would be helpful in ensuring that the momentum from the good work carried out to date is sustained.



Auto manufacturer rebate programs . ARC believes that there are opportunities for

ARC, SI and EC to become involved in auto manufacturer rebate programs that provide incentives for fuel-efficient vehicles.



4.3 Performance – Effectiveness

Evaluation Issue: Effectiveness (under Performance)


The NVSP achieved its intended short-term outcomes. The program was made available across all regions, except in the territories due to a lack of recycling infrastructure and low demand. In terms of program awareness among the public, survey evidence indicates that the level of awareness increased steadily as the program matured. The recycling code of practice is considered an important and valuable legacy of the program.

According to program data, approximately 138 600 vehicles were retired through the program. Evidence indicates, however, that some of these would have been retired in the absence of the program. As the NVSP was not fully implemented nationally until 21 months after program inception, it was unable to reach its initial target (200 000 vehicles over four years).

Most of the program’s users preferred the cash incentive and replaced their vehicles with another one. The structure of the manufacturer rebates (which were beyond the program’s control) affected the type of replacement vehicle purchased.

Environmentally safe vehicle recycling practices during the program were ensured by requiring all participating recyclers to adhere to the code of practice.

ARC reported that 99% of the participating recyclers plan to continue applying the code of practice after the program ends.

With respect to long-term outcomes, based on EC data the program led to approximately a 5000-tonne reduction in NO x

and VOCs emissions, and a

37 500-tonne reduction in GHG emissions.

Many external factors had an impact on the

NVSP’s success, including the existence of previous programs, the influence of a scrappage program in the

United States (which increased take-up through major awareness campaigns that reached Canadians), the auto manufacturer rebates, and the state of the economy (recession).

As for the unintended benefits, the program helped build NFP capacity, and the positive effects of the recycling code of practice on other stakeholders (steel producers) were beyond expectations. The program also had economic benefits in the context of the recession (by supporting demand for new car purchases), and safety benefits by removing older, less safe vehicles and replacing them with new safer vehicles.



Performance – Effectiveness

11. To what extent have the intended outcomes been achieved as a result of the Vehicle Scrappage Program?

Short-term outcomes: a) Canadians in all provinces have access to incentives to reward them for removing older vehicles from the road b) Increased awareness by the public of the program and the environmental impact of older vehicles c) Increased awareness by recyclers of environmentally safe recycling practices

The evaluation examined this question using KI interviews, a review of documentation and performance data, and case studies.


Findings: The program was made available across all regions, except in the territories due to a lack of recycling infrastructure and low demand. Regarding program awareness by the public, survey evidence indicates that the level of awareness increased steadily as the program matured. Many program users recognize that their older vehicles are less environmentally friendly than new models, but also believe that car maintenance is a key determinant of the level of pollution produced by automobiles.

The vehicle recycling code of practice is considered an important and valuable legacy of the program. The code was published in print and electronic formats, and was widely and readily available to auto recyclers across the country. The positive impact of the code on auto recycler s’ awareness of environmentally safe recycling practices was one of the program’s successes. a) Canadians in all provinces have access to incentives to reward them for removing older vehicles from the road

According to documentation and KI interviews, the program was made available in all

Canadian regions except the territories. A feasibility study of launching RYR in the territories revealed that the existing program would not be practical, as the territories did not have recycling infrastructure, and derelict vehicles presented a much larger issue than retiring older, live cars.

EC managers and NFPs indicated some difficulties with access and participation for people living in rural or remote areas who may not have access to high-speed Internet


Audit and Evaluation Branch Evaluation of NVSP or nearby auto recyclers. Despite the vehicle recovery fee that was implemented to subsidize pickup in remote areas, an NFP in a smaller province still identified problems arranging pickup throughout the province when the distance to the nearest auto recycler could be up to several hundred kilometres. b) Increased awareness by the public of the program and the environmental impact of older vehicles

Program Awareness

Advertising and outreach campaigns were conducted to raise awareness about the program and encourage participation. EC had initially planned to spend about $5 million on a national advertising campaign to increase public awareness of the pollution caused by older vehicles, but the campaign was cancelled and only ad concepts were created.

However, EC and SI believe that local advertising and promotion activities increased public awareness of the pollution from older vehicles to some extent. NFPs organized outreach events at venues such as fairs, festivals, parades, auto shows, environmental shows and markets. Displays varied among provinces but would often include

RYR-branded booths, promotional vehicles, representatives in RYR-branded clothing, and sustainably produced giveaways (e.g., RYR bike lights, water bottles).

Survey results indicate that program awareness increased significantly as the program was implemented and as it matured. In 2008, a survey of model-year 1995 and older car owners found that 33% of older-car owners were aware of vehicle scrappage programs in Canada. The survey was conducted again in 2010 and results indicate that, following the launch of RYR, the proportion of model-year 1995 and older vehicle owners who have seen or heard something about a vehicle scrappage program had more than doubled.

The program participants were asked how they heard about the program, which is another indicator of the effectiveness of the marketing and outreach activities. The percentage of program participants who heard about the program through ads decreased from 22% in June 2009 to 15% in January 2010, then rose to 28% in

December 2010. Between June 2009 and December 2010, word of mouth as a source declined from 34% to 24%, and the Internet as a source declined from 15% to 6%. In all cases, the marketing efforts of 2010 (including program ads, partner ads and outreach activities) appeared to have a significant impact.



Awareness of Environmental Impacts of Older Vehicles

The 2008 survey of model-year 1995 and older vehicle owners indicated the extent to which these owners were aware of the environmental impacts of their vehicle. When asked what makes some vehicles more polluting than others, based on what they know or may have heard, the most frequently cited factor was how a vehicle is maintained

(45%), followed distantly by its age (19%), fuel type (16%), vehicle or engine size

(14%), its type of pollution reduction technology (11%), and other factors. However, the majority (65%) of drivers of older vehicles recognized that model-year 1995 and older vehicles pollute more than new ones.

Program participants were also asked to explain the rationale for their participation in the program. In 2009, 21% of participants cited concerns about environmental impact as the reason for retiring their vehicles, while in December 2010 only 10% cited this reason. The primary reason that participants chose to dispose of their vehicle was that it was no longer reliable/practical to keep (37% of respondents in December 2010). c) Increased awareness by recyclers of environmentally safe recycling practices

Almost all KIs described the vehicle recycling code of practice as an important and valuable legacy of the program. The code was drafted by ARC, the national umbrella organization representing the auto recycling industry, under a contract with EC and other stakeholders. The idea for a recycling code of practice and recycling audits, which originated from ARC, was based on a pre-existing code in B.C., a best practices guide in Quebec, and best practices in other countries such as the United States. The code was published in print and electronic formats and was widely and readily available to auto recyclers across the country. ARC operates a website that includes the code and related training materials. ARC also made presentations to and attended meetings with auto recyclers, to promote the code.

Interview respondents stated that the positive impact of the code on auto recycler awareness of environmentally safe recycling practices was one of the program’s successes. Over 350 vehicle recyclers were RYR partners, and according to program documentation, efforts were made to increase the recyclers’ awareness of the importance of environmentally safe recycling practices and of the code. SI conducted inperson training on the code with recyclers across Canada. To maximize the code’s influence, training was free and recyclers were not required to be part of RYR in order to attend. In-person recycler training on the code was held in 16 locations in seven provinces covering all regions of the country. Over 200 companies attended the training sessions, with many companies sending two or more participants. Approximately 200

DVD copies of the training were distributed, and 78 companies took the online training course.

Implementation of the code in the North was explored, but found to be not feasible. The

North is far from markets for scrap metal, and attracting partners is more difficult given that transportation costs are higher than the scrap metal value. Moreover, in the

Northwest Territories, there are only a few vehicle recyclers, and they were not in a position to meet the code’s requirements.



In addition to the training, 92% of participating RYR recyclers were audited to verify their adherence to the code.

The vast majority (92%) were found to be in compliance. The four auditors hired to conduct on-site vehicle recycling audits played an important role in informing and educating recyclers about the code and how to make improvements to their businesses. Recyclers were aware of the website, although they appreciated the site visits by auditors and the information provided in person.

EC managers cautioned that it is difficult to know the extent to which recyclers not involved with the NVSP are aware of the code and are following its recommendations.

However, audits of the recycling companies that were involved with the program demonstrate that the great majority are following the code. ARC estimates that larger and mid-sized auto recycling companies that follow the code, whether or not they are participating in the program, could be recycling up to 80% of the vehicle components (in most regions of the country).

SI respondents expressed concern about the long-term sustainability of the code after the program ends. A provincial NFP also expressed a concern that auto recyclers say there is no overarching regulatory body to oversee their practices. This NFP further wondered if the auto recyclers will continue with environmentally safe practices. More optimistically, SI stated that they have established “deeper relationships” with auto recyclers and that these relationships will help promote awareness of evolving environmentally safe auto recycling practices.





The evaluation examined this question using KI interviews, a review of documentation and performance data, and case studies.

Medium-term outcomes: d) Fewer older, high-polluting personal vehicles on the road

Rating: Progress Made; Attention Needed e) More sustainable transportation modes being used for personal transportation

Rating: Little Progress; Priority for Attention f) Environmentally safe practices in use for recycling of vehicles scrapped through the program


Findings: 138 600 vehicles were retired by the program, although evidence indicates that some of these vehicles would have been retired in the absence of the program.

As the program was not fully implemented until 21 months after program inception, the program did not reach its initial overall target of 200 000 vehicles retired over 48 months. Some regions had difficulties reaching their targets due to a lack of recycling infrastructure and other logistical issues.

Most of the pro gram’s users preferred the cash incentive and replaced their vehicles with another one. The manufacturer rebates affected the type of replacement vehicle purchased, as more program users who obtained rebates opted for larger vehicles than the ones they retired.

Environmentally safe practices in the recycling of vehicles were promoted during the program by requiring all participating recyclers to adhere to the recycling code of practice. ARC reported that it now has approximately 450 member companies nationally, most of whom adhere to the code. d) Fewer older, high-polluting personal vehicles on the road

As indicated earlier, the program goal was to retire 50 000 vehicles per year, for a total of 200 000 vehicles over four years. As of March 2011, approximately 138 600 vehicles


Audit and Evaluation Branch Evaluation of NVSP were retired as a result of the program (see Table 6). Although the target of 50 000 vehicles per year was reached, the program was not fully operational until 21 months after program inception and thus did not reach its initial overall target.

Table 6: Program Use Statistics (from January 2009 to March 31, 2011)

Province Number of

Applications

Received by the

Program

Number of

Vehicles

Permanently

Retired*

Number of

Active

Recyclers at

Program End

Alberta

British Columbia

Manitoba

New Brunswick

Newfoundland and

Labrador

Nova Scotia

Ontario

14 692

19 527

6313

2239

1585

3199

57 355

11 692

19 030

5519

1806

1269

2575

50 485

23

27

4

11

10

23

120

Prince Edward Island 959

Quebec 47 363

Saskatchewan

Total

6239

159 471

758

40 419

5089

138 642

4

109

16

347

* Number of vehicles off the road and handed over to recyclers

Source: RYR statistics (website)

According to documentation, Alberta and Newfoundland and Labrador experienced challenges reaching their original 2009-2010 targets. These provinces were challenged by a lack of recyclers who were willing to participate in the program due to geographic challenges and market forces. As there are no steel processors in either province, there is an additional logistical and financial burden of shipping processed vehicles out of province, thus making the financial model for recyclers less attractive and making it more difficult for the program to recruit recyclers (SI 2010d, p. 26).

Nevertheless, the program exceeded its annual targets for the 2009-2010 period

(calendar years) by approximately 18%. However, the impact cannot be entirely attributed to the program: according to survey results, 61% of participants stated that they retired their vehicles earlier than expected as a result of the program (39% said the opposite) (Environics Research Group 2011). This finding indicates that the results cannot all be attributed to the program, as it is likely that some of the impacts would have occurred in the program’s absence. However, the exact level of attribution would be difficult to assess considering the following:



The survey question is a proxy based on a hypothetical question (participants were asked “Did this Retire Your Ride program prompt you to get rid of this vehicle sooner than you might otherwise have done?”). The percentage of participants who would have

“gotten rid” of their vehicle in the program’s absence may be different than what the respondents stated (e.g., respondents may have delayed their decision for various reasons).





The survey question and response do not indicate exactly what would have happened to the vehicle in the program’s absence. Getting rid of a vehicle may simply mean an exchange with a dealer, a donation to a family member, or a sale to a recycler. In all three cases, it may mean that the vehicle changed owners and was not retired. Thus, one cannot assume that the 39% of participants would in fact have recycled their vehicle in the program’s absence.

Despite the above factors, it is still likely that a significant percentage of participants would have retired their vehicle in the program’s absence, which leads the evaluation team to conclude that not all observed impacts are attributable to the program. e) More sustainable transportation modes being used for personal transportation

According to program statistics, more than 90% of program participants chose the $300 cash incentive after retiring their vehicle, while only 7% of participants selected a sustainable transportation reward (e.g., a transit pass, bicycle discount, car share). The majority of participants replaced or were planning to replace their retired vehicle with another one (75% in 2009 and 86% in 2010, according to survey results), while 22% of participants in 2009 and 12% in December 2010 had no plans to replace their vehicles.

British Columbia provided the highest-value non-cash alternative transportation rewards, and therefore made the biggest impact in shifting participants away from vehicles. This was likely due in part to the significant contribution of additional incentive funds from the provincial government (e.g., $750-value car-sharing membership, up to

$2,000 in free transit passes, rebates on the purchase of new fuel-efficient vehicles).

As shown in Table 7, the majority of those who replaced / were planning to replace their vehicle in 2009 were considering replacing it with a car (74%), while 18% were considering a truck, sport utility vehicle (SUV) or van. Unfortunately, the percentage of those planning to purchase a car rather than an SUV or van fell to 46% in December

2010.

2

2 The Canadian market for new vehicles shifted during the program. At the start (January 2009), passenger cars accounted for the majority of new vehicle sales (54% cars and 46% light-duty trucks [van, pickup, SUV]) in Canada. At the end of the program (March 2011), light-duty trucks accounted for the majority of sales (44% cars and 56% light-duty trucks).



Table 7: Replacement Plans of Program Participants (%)

Type of Vehicle 2009 Jan.

2010

Car

Truck

74

7

57

21

Dec.

2010

46

34

SUV 8 11 12

Van or crossover 3 3 2

Don’t know / not sure

8 7 5

Source: Survey of Program Participants, Environics Research (1000 respondents for each time period)

Approximately 3% of participants replaced their vehicle with a model-year 1995 or older vehicle, and approximately 13% replaced it with a vehicle manufactured between 1996 and 2003 (not shown).

The effect of the above substitutions (vehicles replacing retired vehicles) depends on the type of vehicle. As noted above, approximately 16% of the vehicles replacing the retired ones are pre-2004 models, which reduces the smog emissions reductions achieved by the program given that these vehicles emit approximately eight times more smog-forming emissions than 2004 and newer vehicles. Data also show that many of the replacement vehicles are in the medium to large size categories and would likely produce larger amounts of GHGs than the vehicles they are replacing. These factors were taken into account in the calculations of the program’s reductions of smog and

GHG emissions (reported in Table 8).

KIs acknowledged the difficulty of shifting people to other modes of transportation.

NFPs noted that, although there are fewer older, high-polluting cars on the road, the program was not effective at encouraging Canadians to use more sustainable modes of transportation. One stated that “the vehicle manufacturers took up all the space” and that more incentives could have been related to sustainable transportation if the vehicle manufacturers had not been involved.

One lesson learned by EC was that without providing any funding for vehicle rebates, there was no control over what incentives were offered by auto manufacturers. Because the manufacturers offered bigger discounts on large vehicles, purchases of these vehicles were higher. SI consulted with the auto manufacturers about their incentive structure but the industry was reluctant to make changes. f) Environmentally safe practices in use for recycling of vehicles scrapped through the program

Environmentally safe practices for the recycling of vehicles during the program were promoted by requiring all participating recyclers to adhere to the recycling code of practice. EC respondents indicated that the code’s main objective was to use it and the



RYR program to “start the conversation” with smaller providers that may not have the infrastructure to follow the code and to start them on a path to better practices.

ARC reported that, traditionally, the mid-sized recyclers were not concerned about anything “environmental.” Many recyclers wanted to become involved in RYR for financial reasons (i.e., access to a supply of cars). However, following dissemination of information about the program and code, ARC received an increased number of applications to join the association from recycling companies agreeing to adhere to the code’s requirements. Auto recycling industry representatives reported that development of the code has led to “an amazing change in the industry.” The biggest positive change was among mid-sized companies; the larger auto recyclers already were following environmentally safe practices for the most part.

Application of the code was enforced by on-site audits conducted to assess adherence.

ARC was responsible for administering the audits and managing the auditors. Some

348 recyclers were audited at least once:



321 recyclers (92%) passed their inspections, with 309 recyclers receiving a pass grade after a primary initial audit and 12 recyclers being upgraded to pass during a repeat audit after receiving a conditional pass during their primary audit.



10 recyclers (3%) failed their inspections, with 8 recyclers receiving a failed grade after their primary audit and 2 recyclers being downgraded to fail during a repeat audit after receiving a conditional pass during their primary audit (these 10 were excluded from the program).



17 recyclers (5%) received a conditional pass during their primary audit, but were not re-audited later to receive a final grade. These recyclers continued to receive vehicles until the end of the program.

Results of the 2011 Vehicle Recycler Survey indicate that RYR succeeded with participating recyclers on numerous levels:



The program achieved high levels of overall satisfaction (62% very satisfied, 31% somewhat satisfied).



The program increased the number of vehicles processed for 78% of participants.



92% of participants applied the code of practice to all vehicles they processed, not just qualified RYR vehicles.



Virtually all participants (99%) indicated that they will continue applying the code after the program ends.



92% of participants said that they are very likely to take part in a similar program in the future.

In the initial period of time following introduction of the recycling code of practice, ARC and auditors of vehicle recyclers reported that many recycling companies were

“nervous” about the code and did not wish to participate. Through contact with recycling companies, the four auditors influenced these companies to realize that the code is a

“work with” tool about training and making improvements. The auditors played a very important role and were responsible for showing many auto recyclers how the program and the code worked. This has led to increased adherence to the code.



ARC reported that it now has approximately 450 member companies nationally, most of whom adhere to the code. Significant developments in the provinces are as follows:



The Ontario Automotive Recyclers Association is now using the code and is auditing all members, not just those participating in RYR.



The Automotive Recyclers Association of Atlantic Canada has agreed to make successful code audits a requirement of membership starting in 2011.



The New Brunswick and Saskatchewan governments have shown interest in using the code to support some of their regulatory work.

NFPs reported that some auto recyclers are proud to advertise themselves as environmentally conscious. One stated that “recyclers are proud to say they got the certificate from passing the Retire Your Ride audit.

”



The evaluation examined this question using a review of performance data, documentation and literature.

Long-term outcomes: g) Air pollutants are reduced

Rating: Achieved h) Greenhouse gas emissions are reduced

Rating: Progress Made; Attention Needed i) Release of harmful and toxic substances into the environment is prevented


Findings: EC data indicate that the program contributed to reduced emissions of NO x

, VOCs and GHGs. Total reductions are estimated to be approximately 5000 tonnes of VOCs and NO x

, and about 37 500 tonnes of GHGs.

Based on differing emissions from model-year 1995 and older vehicles that were scrapped, and their replacement by either newer units or alternative means of transportation, EC calculated the total annual emission savings based on program participation, Statistics Canada information on average kilometres driven, and other


Audit and Evaluation Branch Evaluation of NVSP information sources (EC 2010b). Estimated total reductions are approximately 5000 tonnes of VOCs and NO x

, and 37 500 tonnes of GHGs (see Table 8). Annual targets were exceeded for reductions of VOCs and NO x

, but were not met for GHG reductions.

Table 8: Emission Savings

Number of vehicles retired

NO x

+ VOCs

Annual

Target

50 000

Annual Results

(12-month periods)

2009-10

(Fiscal Year)

65 200

2010-11

(Fiscal Year)

69 900

Emission Reductions (Tonnes)

2250 2350 2520

Total for RYR

(27 months)*

138 600

5000

GHGs (CO

2

eq) 54 000 17 700 18 900 37 500

* Total reductions for RYR include an additional three-month period (January-March

2009) not reflected in the annual targets.

Source: EC project database

The level of emission savings is affected by the age and types of vehicles being purchased to replace the older vehicles. According to survey results, about 30% of participants replaced their vehicles with used vehicles and/or trucks and SUVs.

3

3 Emission reductions are the difference between emissions from the retired older vehicle and its replacement. The calculations considered only vehicles retired through the program and only the year following vehicle retirement. Surveys of program participants conducted 6 –12 months after their participation were used to assess transportation behaviour after the retirement of the older vehicle.

Survey data included whether a replacement vehicle was purchased (and the model year and category of this replacement vehicle), or if other means of transportation were used (e.g., transit, car sharing). The calculations also used published data for emission factors, annual vehicle usage, and transit data.




12. Did this program lead people to make use of other related federal and

provincial government programs or vice-versa?

The evaluation examined this question using KI interviews.

Rating: Not Applicable (Insufficient Evidence)

Findings: Outreach and promotional materials for RYR were linked to two other federal government initiatives: ecoENERGY for Personal

Vehicles (e.g., driving tips, buying a fuel-efficient vehicle), and

Q&As on the admissibility of the RYR transit incentive for the federal transit pass tax credit. In addition, the original plan was to cross-promote RYR and another federal program (ecoAUTO), but ecoAUTO was terminated in March 2009. The evaluation team was unable to find evidence of the degree to which people made use of other related programs.

Outreach and promotional materials for RYR were linked to two other federal government initiatives: ecoENERGY for Personal Vehicles (e.g., driving tips, buying a fuel-efficient vehicle), and Q&As on admissibility of the RYR transit incentive for the federal transit pass tax credit. In addition, EC managers reported that the original plan was to cross-promote RYR and the ecoAUTO Rebate Program. However, ecoAUTO was terminated in March 2009, only three months after RYR was launched.

On a related point, three provinces (B.C., Manitoba and Quebec) contributed to the program by topping up the incentives for participants. According to EC interview respondents, program efficiencies were achieved by combining provincial and federal funding rather than setting up separate programs.




13. Were there any external factors outside of the Vehicle Scrappage

Program that influenced (positively or negatively) the success of the program?

The evaluation examined this question using KI interviews, documentation and case studies.


Findings: Many factors had an impact on the NVSP’s success, including the existence of previous programs, the influence of a similar

U.S. program, the auto manufacturer rebates, and the economic recession. The U.S. program involved a major awareness campaign that reached Canadians, which contributed to increased take-up for RYR. The manufacturer rebates also increased take-up, but some focused on larger vehicles. The recession had both positive and negative effects.

A number of external factors were identified by respondents and confirmed by documentation as influencing program success:

Pre-existing provincial programs . The fact that a number of NFPs were already involved in vehicle scrappage and had existing expertise and partners made it easier to launch a national program. However, the existence of provincial scrappage “program names” resulted in lower visibility for the RYR name. There were also operational challenges related to provincial NFPs wanting to retain their modes of operation, which led to some “growing pains” when the national program was launched.

U.S. Cash for Clunkers program . Cash for Clunkers was launched midway through

RYR. This American program (and to a lesser extent other international programs) received extensive Canadian media coverage, which led to increased awareness of

RYR. It also led to some criticisms of RYR, because the U.S. cash rebates were larger than the Canadian rebates (i.e., $4,000 per vehicle in the U.S. vs. $300 in Canada).

Cash for Clunkers was designed as an economic stimulus program and only offered vehicle rebates, while the Canadian program was designed as a clean air / environmental program and offered a range of incentives.

Auto manufacturer rebate programs . According to multiple sources, the combined rebates offered by the auto manufacturers had a significant impact on the program and its performance. GM, Ford, Hyundai, Chrysler, VW and Mazda all launched incremental rebates ranging from $500 –$3,000. These rebates were offered in addition to the federal rewards. Participant applications would peak during these offers, although manufacturers had the option in their contracts to

“go dark” (i.e., stop offering rebates) at certain periods, often with on ly several days notice. Ford and GM “went dark” in early

March 2009, which resulted in a decrease in applications despite a communications campaign by SI (SI 2010d, p. 13).



Although the combining of NVSP rewards with the manufacturer rebates stimulated program use, it also had an impact on program GHG emissions reduction results. In general, the value of the rebate offered by the manufacturers increased with vehicle size, which led to more participants purchasing replacement vehicles that were larger than those retired.

Because the NVSP’s terms and conditions did not allow the use of incentive funds for vehicle rebates, the manufacturers designed and solely funded this part of the program, which meant that neither EC nor SI had any leverage to influence the rebate structure or encourage greater incentives for fuel-efficient vehicles. This is why program participants were more likely to purchase trucks, SUVs and vans in 2010 than in 2009 (discussed earlier).

Provincial government incentives . The financial participation of provincial governments, particularly B.C. and Quebec, increased program take-up. Provincial funding was mostly directed toward transit incentives, which helped to increase the popularity of non-cash incentives.

Impact of recession on participation . The recession / weak economy may have led some potential participants to keep their old cars due to a reluctance to invest in a newer vehicle during difficult economic times. The NFPs believed that, for some potential participants, the incentive was not enough to motivate them to give up their cars. NFPs also noted that the recession made it more difficult to leverage funds for incentives from some organizations.

Low scrap steel prices . The economic recession during the early stages of the program led to low prices for recycled steel, which affected the number of recyclers who wanted to participate in RYR. NFPs reported that low steel prices made it challenging at the beginning to recruit enough auto recyclers to ensure good geographic and market coverage to serve all participants.

Market for recycled auto parts . The reluctance of the insurance industry and consumers to use recycled auto parts reduces revenues for the recycling industry, which may limit the widespread adoption of environmentally safe practices by auto recyclers. Environmentally safe practices involve added costs (or at least additional investments), which can appear less attractive if revenues are lower due to lower resales of recycled parts. On the positive side, the insurance industry, the customers of auto recyclers, and the suppliers of cars to auto recyclers have become more knowledgeable about recycling practices and increasingly want to deal with companies that employ environmentally safe practices.




14. Were there any unintended positive or negative outcomes of the Vehicle

Scrappage Program?

The evaluation examined this question using KI interviews, documentation and case studies.


Findings: The program had limited ability to influence program user transportation choices after the incentive was provided. As a result, many participants replaced their vehicles with large or older vehicles. On the positive side, the program helped build

NFP capacity, and the impacts of the recycling code of practice were beyond expectations. The program also had economic benefits in the recession context, as well as social benefits by removing older, less safe vehicles and replacing them with new, safer vehicles.

The following unintended outcomes were identified in documentation and interviews:



Increased purchases of larger vehicles . The program was not allowed to provide financial support for vehicle rebates, and therefore had limited ability to encourage vehicle manufacturers to focus their rebates on fuel-efficient vehicles rather than their full range of vehicles, including larger, more expensive models or older secondhand vehicles. Although vehicle rebates increased program participation and helped achieve smog reduction goals, they may have reduced the program

’s ability to achieve GHG reductions (Government of Canada 2011, p. 3).



NFP capacity building . The creation of the NFP network to deliver the program in each region, and the increased management and financial capacity of the NFPs for program delivery, are positive outcomes of the program. The program brought together the NFPs and other partners (such as auto recyclers), built relationships, and developed networks (even among organizations with different missions). As one

NFP reported:

“We’ve made great partners. We get together for conference calls and physical meetings. Those relationships are very positive.

”



Broader impacts on environmentally safe vehicle recycling practices . A vehicle recycling company reported that large steel recyclers not directly involved with the program have been influenced by the recycling code of practice. For example, some steel companies have stopped accepting metal from vehicles unless the mercury switches have been removed.



Impacts on Economy

Considering the timing of the program, new vehicle purchases are a major unintended impact of the NVSP. Based on participant surveys, between 37% (2009 survey) and

61% (December 2010 survey) of participants were planning to replace or did replace their old vehicle with a new one. Assuming a new car purchase rate of 45%, this equates to 43 875 new vehicle purchases (0.45 X 97 500 participants) linked to the program. This finding can be interpreted as a significant impact on the Canadian economy, in particular considering that the economy was at a low point in 2009 and

2010. It is beyond the scope of this evaluation to assess the exact value of this impact, but for illustrative purposes, Table 9 provides an approximate estimate (although it is agreed that the program only contributed partially to this impact).

Table 9: Approximate Estimate of Unintended Economic Impact

A. Number of

New Vehicles

Purchased

(2009 and

2010)

B. Approximate

Average Value per Vehicle (1)

C. Total Value of Vehicles

Purchased

(Benefit for

Canadian

Economy)

(A x B)

D. Approximate

Tax Revenues for the Federal

Government

(Based on 5%

GST)

(C x 0.05)

Totals 43 875 $25,000 $1,096,875,000 $54,843,750

(1) Based on Statistics Canada, New Motor Vehicle Sales (December 2009), Catalogue no. 63-007-X.

These numbers, which would be less significant in an economy growing at “normal” rates, become significant considering that the purchases were made in a period when auto manufacturers were experiencing a severe cutback in demand, and government was in a deficit period. It is also important to note that these impacts are not entirely attributable to the program, given the significance of auto manufacturers

’ rebates (which were not subsidized by RYR) and other contributing factors.

Impacts on Safety

It is likely that the program also had positive social impacts by removing older cars and replacing them with new vehicles equipped with better safety features. According to

Statistics Canada (2008), many competing factors increase or decrease the risk of motor vehicle accidents and injuries. Technological advances such as anti-lock braking, airbags, improved seatbelts and child restraints make vehicles safer. For example, sources indicate that the gradual implementation of airbags in the 1990s has significantly reduced fatality rates. Although airbags are still not a mandatory requirement by Transport Canada, most automakers had gradually adopted this technology for their products by the end of the 1990s. According to a Transport Canada study (Transport Canada 2001), the number of saved lives associated with airbags significantly increased beginning in 1997: it is estimated that approximately 50 lives were saved annually between 1997 and 2000 due to airbags. Other technological improvements such as ABS braking systems and electronic stability control have also


Audit and Evaluation Branch Evaluation of NVSP saved lives, according to American sources (National Highway Traffic Safety

Administration 2010). So it is likely that the program has helped reduce fatalities through the replacement of less safe vehicles.

4.4 Performance – Efficiency and Economy

Evaluation Issue: Efficiency and Economy (under Performance)


Evidence indicates that program resources were well used. EC and SI monitored performance and introduced corrective measures to ensure program efficiency.

An efficiency analysis indicates that the proportion of total program costs utilized for administrative costs was low for the NVSP compared to a similar program, the ecoAUTO rebates. As well, significant resources were leveraged by the program: based on an analysis of all incentives provided to program participants divided by the total cost of incentives provided by the federal government, a 3:1 leveraging ratio was achieved.

Few alternative program models were identified by KIs. An alternative approach, used in certain foreign programs, would have been to offer more significant incentives and/or prescribe the types of substitute vehicles that could be purchased.



Performance – Efficiency and Economy

15. a) Did the Vehicle Scrappage Program undertake its activities and deliver its products in the most efficient manner?

Rating: Achieved c) How could the efficiency of the program ’s activities be improved?


16. Did the Vehicle Scrappage Program achieve its intended outcomes in the most economical manner?


The evaluation examined these questions using KI interviews, a review of literature and documentation, and an analysis of program financial data.

Findings: Evidence indicates that program resources were well used. As well, evidence shows that EC and SI monitored performance and introduced corrective measures to ensure program efficiency. An efficiency analysis indicates that the proportion of total program costs utilized for administrative costs was low for the NVSP compared to a similar program, the ecoAUTO rebates.

For the purposes of this evaluation, the same data sources were used to assess efficiency and economy, including qualitative and quantitative information on the use of program resources.

EC respondents provided views on the extent to which program resources were well used. At the time of the interviews, they noted that the program was coming in under cost: the program had a budget of $92 million, whereas costs are projected to be approximately $68 million. The program also leveraged significant amounts of resources

(see below). Some amounts advanced to NFPs remained unspent, mainly in marketing.

At the other end of the spectrum, some NFPs reported that they were in over-budget situations.

Some NFPs and auto manufacturers reported problems concerning efficiency. Although most NFPs thought that the process was efficient, a few mentioned that there were missed opportunities, including better data systems. It was reported that SI struggled to find the appropriate resources (in human resources) when launching the program.

Some of the systems used by the provincial organizations did not work very well, especially at the beginning, and there was poor liaison between SI and regional delivery partners.

As discussed earlier, monitoring systems uncovered difficulties, and the program implemented a number of successful corrective measures regarding delivery.



Financial Analysis

To assess efficiency and economy, the evaluation team calculated the percentage of total program costs used for administrative costs. This is a standard approach in evaluation of efficiency, and should involve a comparison with similar programs to assess the program ’s level of administrative costs. The evaluation team established this percentage for the NVSP and compared it with a similar federal program, the ecoAUTO rebates.

The analysis of the NVSP’s cost structure indicates that, compared to other similar programs, the NVSP can be considered cost-efficient. Table 10 presents the cost structure for 2009-2010, which would be representative of the program in full delivery mode. As shown, the program involves EC internal administrative costs (salaries, benefits, O&M and corporate support), SI administrative costs, and a significant amount in G&Cs (mainly for SI but also for other NFPs involved in delivery of the program).

Table 10: NVSP Efficiency Analysis (2009-2010 Fiscal Year)

Cost Components

Salaries

Employee benefits plan

O&M

Corporate support

SI administrative costs

Total administrative costs (A)

G&Cs

Total Costs (B)

% Administrative Costs (A/B)

2009-2010 (Actuals)

$260,962

$52,192

$760,400

$481,543

$339,000

$1,894,097

$26,594,268

$28,488,365

6.6%

Source: Program financial data, SI financial statements

The percentage of administrative costs was calculated by dividing the administrative costs of EC and SI by the total costs. As indicated above, the proportion of administrative costs is 6.6%, which is considered highly acceptable for a program of this type.

As a comparison, the evaluation team assessed the same percentage for another federal program, Transport Canada’s ecoAUTO Rebate Program, which also provided rewards to promote more sustainable personal transportation behaviour, namely the purchase of a fuel-efficient vehicle. However, ecoAUTO was primarily delivered by

Service Canada, whereas the NVSP was primarily delivered by SI and other NFPs. The financial analysis is provided in Table 11.



Table 11: ecoAUTO Rebate Program Efficiency Analysis (2008-2009 Fiscal Year)

(in $ Thousands)

Cost Components 2008-2009

Salaries and operating costs (Transport Canada)

Salaries and operating costs (Service Canada)

$1,897

$7,717

Total administrative costs (A)

G&Cs

$9,614

$119,864

Total Costs (B)

% Administrative Costs (A/B)

$129,478

7.4%

Source: Program data (reported in Transport Canada 2010)

As shown above, the administrative costs for ecoAUTO represented 7.4% of total program costs. Although this percentage is more than acceptable, the NVSP demonstrated a slightly lower proportion for administrative costs, despite the fact that it has a smaller program budget (larger programs, i.e., ecoAUTO in this case, can benefit from economies of scale and reduce their administrative costs).

Further evidence indicates that the contribution funding used by regional NFPs for program delivery was approximately 30% of the total funding, with the remaining amount being used for incentives. This exceeded the objective of allocating no more than 25% of the contribution funding to program delivery. The cost of extra advertising

(used by some NFPs as a partial replacement for advertising cancelled by EC) is responsible for most of the larger program delivery cost.

Another indicator that can be used to assess efficiency is the government administrative cost per grant (in this case, per vehicle). Table 12 compares the above programs and

Cash for Clunkers, and provides the costs of the incentives for the governments and the average value of the incentives received by participants.



Table 12: Comparison of Average Cost to Government per Vehicle

Number of vehicles

Federal government’s administrative cost per vehicle

NVSP

(2009-2010)

65 200

CAN $126 ecoAUTO

(entire program)

169 255

CAN $98

Cash for Clunkers

(entire program)

677 842

USD $114

Average incentive cost to federal government

Average incentive

CAN $300 CAN $1,124 USD $4,209

CAN $934 CAN $1,124 USD $4,209 value to participant

Sources: Program financial data; SI financial statements; Transport Canada; U.S.

Department of Transportation

As shown above, the administrative costs per vehicle for all three programs are similar.

The NVSP’s costs are the highest but it also had a smaller economy of scale (it was a smaller program in terms of total funding).

An indicator of the

NVSP’s cost efficiency is a comparison of the government’s incentive cost to the incentive value delivered to program participants. For the NVSP, federal incentive costs were leveraged via partnerships into incentive values of a much higher level (over 3:1); the NVSP’s administrative costs reflect the work involved in developing partnerships. The other two programs did not leverage any funding, which means the federal incentive costs were the only driver of value to participants.

In its evaluation of the clean transportation theme of the CAA, Transport Canada (2010) concluded that the NVSP and ecoAUTO are efficient compared to similar programs, but not cost-effective for reducing GHG emissions (based on an analysis of the estimated cost to the federal government for each tonne of GHGs reduced). However, the reduction of GHG emissions was only a secondary goal of the NVSP.



Performance – Efficiency

15. b) Was federal funding leveraged to the extent planned?


d) Are there alternative, more efficient ways of delivering the program?


The evaluation examined these questions using KI interviews and a review of documentation and literature.

Findings: Evidence indicates that significant resources were leveraged by the program. Based on an analysis of the value of all incentives provided to program participants in relation to the total cost of incentives provided by the federal government, a 3:1 leveraging ratio was achieved.

Few alternative approaches were identified by KIs. Foreign programs show that alternative approaches exist, including the use of more significant incentives with conditions on the types of substitute vehicles purchased.

Documentation and interviews with EC staff indicate that significant resources were leveraged by the program. For example, it is estimated that recyclers paid approximately $7 million to call centres in order to contribute to the centres’ operating costs. In addition, SI was funded by vehicle manufacturers to provide accelerated processing of registrations of their customers. Provincial contributions of nearly

$10 million were also received. In terms of investments disbursed by program partners, an estimated $84 million was leveraged in the form of incentives from partners. Overall, as indicated in Table 13, the leveraging ratio achieved was approximately 3:1, which was the program’s goal.

Table 13: Leveraging of Federal Funding for Incentives

Funding Partners

–

Contribution to Incentives

Provincial governments

Vehicle manufacturers and dealers

Transit authorities

Bicycle retailers, car-share programs, etc.

$ Millions

$9.7

$70

$3.1

$1.3

$84.1 Subtotal

Federal Government – Contribution to Incentives

Environment Canada (A) Subtotal

Total Incentive Value (B)

Leveraging Ratio of Total Incentive Value to EC Contribution

(B:A)

$41.6

$125.7

3:1



Alternative Approaches

Interview respondents (NFPs) identified various alternative delivery approaches, including having separate contribution agreements with each regional NFP. Many NFP respondents said that SI did not have the capacity to deliver the program without regional partners. According to EC respondents, there were no alternative ways of delivering the same program with the same level of success at significantly less cost.

For example, a centralized “1-800” program with solely national incentives may have been more cost-efficient but would have lacked many valuable local/regional contributions and financial leveraging, e.g., local incentives, local outreach, and local brand recognition.

Another alternative would have been to lower the amount of the incentive. Program staff indicated that the $300 cash incentive was already a bare minimum, suggesting that a lower incentive may have reduced the program’s incremental impact.

The program did not implement the originally planned advertising campaign, reducing

EC costs by approximately $5 million, but SI increased its initial budget allocation for advertising by approximately $2 million, for a net cost saving in the area of $3 million.

Foreign Programs . Other programs were implemented abroad and are considered here as potential alternatives. Programs reviewed include those in the United States,

United Kingdom, Germany and France (judging the efficiency of these programs is beyond the scope of this study). Details of each program are provided in Annex 5. The review found the following:



Unlike the NVSP, all the foreign programs were intended to be economic stimuli for national economies by encouraging the sales of new vehicles. An environmental angle may have been present but it was secondary.



The programs had a very short lifespan (typically one year).



The value of incentives was larger than the NVSP, ranging from $1,500

–$4,500.



All programs prescribed the type of vehicle purchased, with most requiring the purchase of a new vehicle. Three countries prescribed minimum environmental efficiency criteria (e.g., fuel-efficient cars).



All programs report positive environmental impacts.



Environment Canada

5.0 CONCLUSIONS

The evaluation evidence leads to the conclusions indicated below.

Relevance

Based on the evidence gathered for this evaluation, there was an identified need for the

NVSP. Model-year 1995 and older vehicles produce significantly more smog than newer models, and there is scientific evidence that smog is highly toxic for the environment and human health. When the NVSP began in 2007, there were 4.6 million model-year

1995 and older models on the road. There was also a need to improve the environmental standards of vehicle recycling practices in Canada.

Although there were regional initiatives with similar objectives when the NVSP was implemented, none were of similar scope. However, EC made efforts to build on these efforts to reduce duplication. The program was consistent with federal priorities and

EC’s mandate and priorities related to air quality and climate change. The NVSP contributes to EC’s strategic outcome related to the minimization of pollution.

The evaluation assessed three dimensions of program performance: design and delivery, effectiveness, and efficiency/economy.


The NVSP objectives were well defined and targets were clear. The initial program target for vehicles to be retired became unattainable because the full national program was only launched in January 2009, 21 months later than initially expected. But annual targets were met or exceeded once full implementation was achieved. The objectives for the recycling code of practice were realistic, well defined and successfully achieved.

The program was delivered through a national NFP using a contribution agreement with

EC. The national NFP then worked with a network of regional NFPs. This delivery system proved to be effective and efficient overall. The advantages included faster implementation, flexibility when making adjustments, and reduced duplication. Program funding was adequate for cash incentives and program operations. Program participants were generally satisfied with the NVSP and the level of satisfaction increased as the program matured.

Program achievements were monitored using the online SPARC data system, quarterly and annual reports from SI to EC, a mid-program review, surveys of participants and the general public, and various program management tools such as spreadsheets to monitor vehicle recycler audits. The program monitoring and performance data were used extensively by program managers to direct and fine-tune the program.

Communications activities were successful at reaching key stakeholders and the general public. Outreach and promotional materials for RYR were linked to two other federal government initiatives, ecoENERGY for Personal Vehicles (e.g., driving tips,

72

Audit and Evaluation Branch Evaluation of NVSP buying a fuel-efficient vehicle), and Q&As on admissibility of the RYR transit incentive for the federal transit pass tax credit.

Many lessons were learned, including the strengths of an approach based on third-party delivery, the need to allow adequate time to reach full implementation of a complex program involving many partners, and the importance of clearly defined roles, responsibilities and lines of communication.


The program was made available across all regions, except in the territories due to the lack of recycling infrastructure and low demand. The level of public awareness increased steadily as the program matured. Many program users recognized that their older vehicle was less environmentally friendly than new models, but many users also overestimated the role of car maintenance as a determinant of the level of pollution produced by older automobiles.

As of March 2011, approximately 138 600 vehicles were retired through the program, although some would have been retired in its absence. As the program was implemented 21 months after program inception, it was unable to reach its initial target.

Some regions had difficulties reaching their targets due to the lack of recycling infrastructure and other logistical issues.

Most of the participants (93%) preferred the cash incentive. More than 75% of participants replaced or indicated they would replace their vehicles with another one.

The auto manufacturer rebates had a direct impact on the type of replacement vehicle purchased, as most program users opted for larger vehicles due to their coming with the biggest incentives and discounts.

EC data indicate that the program contributed to reduced emissions of NO x

, VOCs and

GHGs. Total reductions are estimated to be approximately 5000 tonnes of VOCs and

NO x

, and approximately 37 500 tonnes of GHGs.

The recycling code of practice is considered an important and valuable legacy of the program. The code was published in print and electronic formats and was widely and readily available to auto recyclers across the country. Environmentally safe practices for the recycling of vehicles during the program were ensured by requiring all participating recyclers to adhere to the code. ARC reports that it has approximately 450 member companies nationally, most of whom now adhere to the code.

Overall, many factors had an impact on the program’s success, including the influence of a similar U.S. program and the auto manufacturer rebates, both of which contributed to increase the profile and take-up of the NVSP. The program

’s limited ability to influence program users ’ transportation choices after the incentive was provided was also unexpected. In terms of unintended impacts, the NVSP helped build NFP capacity, led to economic benefits, and generated social benefits by removing older, less safe vehicles and replacing them with new, safer vehicles.




Evidence indicates that program resources were well spent. EC and SI monitored performance and introduced corrective measures to ensure program efficiency. A cost analysis indicates that, compared to other similar programs, the NVSP can be considered cost-efficient.

As well, significant resources were leveraged by the program. Based on an analysis of the value of all incentives provided to program participants in relation to the total cost of incentives provided by the federal government, a 3:1 leveraging ratio was achieved.

Few alternatives were identified by KIs. Foreign programs show that an alternative approach would have been to offer more significant incentives, with conditions on the types of substitute vehicles purchased.



6.0 LESSONS LEARNED

As the NVSP has ended, the focus of this evaluation was on lessons learned, not recommendations for improvement. The following lessons learned from the NVSP may help inform the design of future similar initiatives:



Incentives can help encourage the removal of technologies (i.e., vehicles in this case) that are harmful to the environment . In particular, a moderate cash incentive (such as $300 to retire an older vehicle) can be effective. It was originally assumed that $300 cash would not be as popular a reward as other incentives of higher value (transit passes, discounts on bikes, memberships in car-sharing programs, etc.). This assumption was proven to be inaccurate, as more than 90% of program participants chose the cash incentive.



The use of a third-party delivery partner (through a contribution agreement with an NFP) can make a program efficient, effective and timely . According to evaluation findings, the use of a third-party deliverer allowed EC to implement a cost-efficient and effective program by capitalizing on the agility of a national NFP.



When using large, complex contribution agreements, care must be taken to establish a tailored contribution agreement, with precise clauses on roles and responsibilities . The NVSP was delivered through a national NFP that managed a significant amount of program dollars. An agreement of this size and complexity should be tailored to the type of program and clarify the roles and responsibilities of each partner.



It is important to thoroughly assess all aspects of a program up front, and to manage the risks that they represent . The evaluation evidence indicates that the

NFP which received the contribution did not have the full capacity up front to manage such a large contribution. For example, EC did not sufficiently investigate the financial management capacity of the NFP before the program’s launch, and consequently the financial reporting and monitoring had to be strengthened throughout the program. In a situation such as this (a large contribution to a single organization), a risk assessment should be conducted. The assessment should then be used to identify and address weaknesses and gaps on the part of the recipient, and/or to design a tailored monitoring strategy.



Effective performance monitoring can help improve programs and the quality of evidence for evaluations. The NVSP’s performance monitoring system, featuring an online reporting database, allowed program management to assess the program’s results on a daily basis. The monitoring system allowed EC to monitor progress and make adjustments to the delivery of the program in a timely manner. It also allowed the evaluation team to assess the program’s success with reliable and representative quantitative measures. The availability of this performance data therefore strengthened the evidence for the evaluation.



Annex 1: Bibliography

CBW Associates. Colleen Brydon William White. (no date). Retire Your Ride Operations

Assessment.

CBW Associates. Colleen Brydon William White. (no date). Retire Your Ride Program:

Operations Assessment Project. Presentation Overview for EC.

Clean Air Foundation. (June 2009a). Retire Your Ride Program: Participant Survey:

Results Presentation.

Clean Air Foundation. (June 2009b). Retire Your Ride Program: Participant Survey.

Environics Research Group.

Clean Air Foundation. (February 2009). National Vehicle Scrappage Program: Baseline

Public Opinion Survey.

Clean Air Foundation. (March 2009). Final Report & Questionnaire E&F.

Environics Research Group. (January 2010). Retire Your Ride Survey: January 2010

Preliminary Topline Results.

Environics Research Group. (April 2010a). Retire Your Ride Program: 2010 Participant

Survey Preliminary Results.

Environics Research Group. (April 2010b). Retire Your Ride Program: 2010 Participant

Survey.

Environics Research Group. (January 2011). Retire Your Ride: 2010 Participant

Survey –Wave 2. Draft Report.

Environment Canada. (May 17, 2007). Vehicle Recycling in Canada. Government

Consulting Services, PWGSC.

Environment Canada. (December 19, 2007). Vehicle Recycling Best Management

Practices. Government Consulting Services, PWGSC.

Environment Canada. (April 23, 2008). National Vehicle Scrappage Program

Accountability, Risk and Audit Framework (ARAF).

Environment Canada. (October 2010). National Vehicle Scrappage Program: Program

Evaluation Start-up Briefing.

Environment Canada. (2010). Estimating impact on emission reductions from the national vehicle scrappage program (unpublished technical report).



Environment Canada website. (March 2011a). Environmental Issues. http://www.ec.gc.ca/Air/default.asp?lang=En&n=62D68640-1

Environment Canada website. (March 2011b). Wildlife. http://www.ec.gc.ca/air/default.asp?lang=En&n=945F2849-1

Government of Canada. (2009). Clean Air Agenda Horizontal Performance Report

2008-2009.

Government of Canada. (2010). Clean Air Agenda Horizontal Report on Plans and

Priorities 2009-2010.

Government of Canada. (2011). Clean Air Agenda Horizontal Report on Plans and

Priorities 2010-2011.

Health Canada. (January 2000). Climate Change and Health Economic Advisory Panel

–Final Report on Health Impacts of the Greenhouse Gases (GHG) Mitigation Measures. http://www.hc-sc.gc.ca/ewh-semt/pubs/climat/greenhouse_gases-gaz_effet/translationtraduction-eng.php

Health Canada. (December 2003). Smog and Your Health. http://www.hc-sc.gc.ca/hlvs/alt_formats/pacrb-dgapcr/pdf/iyh-vsv/environ/smog-eng.pdf

Health Canada. (2007). Climate Change 2007: Impacts, Adaptation & Vulnerability, The

Intergovernmental Panel on Climate Change (IPCC). / Climate Trends and Variation

Bulletin, EC website. / From Impacts to Adaptation: Canada in a Changing Climate

2007, Natural Resources Canada. / Human Health in a Changing Climate: A Canadian

Assessment of Vulnerabilities and Adaptive Capacity, Health Canada. Cited in http://www.ec.gc.ca/indicateurs-indicators/default.asp?lang=en&n=EBBA9434-1 .

National Highway Traffic Safety Administration (U.S.). (2010). An Analysis of the

Significant Decline in Motor Vehicle Traffic Fatalities in 2008.

Summerhill Impact (SI). (January 1, 2009). Incentive Provider Agreement (template document).

SI. (March 2010a). National Vehicle Scrappage Program Scoping Exercise Report.

SI. (April 2010b). Retire Your Ride. Call Centre Operational Guide.

SI. (September 2, 2010). Retire Your Ride: Annual Report to EC for Year April 1st to

March 31st 2010.

SI. (September 2010). Exit Strategy: National Vehicle Scrappage Program Update.

SI. (no date). Contracts Summary Document.

Statistics Canada. (2008). Motor Vehicle Accident Deaths, 1979 to 2004.



Taylor Research Group. (2010). Retire Your Ride Call Centre Assessment.

Transport Canada. (October 2001). Evaluation of the Effectiveness of Air Bags and

Seat Belts, TP13187E. Fact Sheet RS 2001-03 E, Road Safety and Motor Vehicle

Regulation.

Transport Canada. (November 2010). Evaluation of the Clean Transportation Theme of the Clean Air Agenda. Transport Canada, Evaluation and Advisory Services.

U.S. Department of Transportation. (December 2009). Consumer Assistance to Recycle and Save Act of 2009. National Highway Traffic Safety Administration.



Annex 2: Network of Regional NFP Delivery Organizations

Region NFP

National SI

BC BC Scrap-it

Society

Prior Experience

Delivering

Scrappage

Initiatives

Car Heaven

Has been operating

BC Scrap-it since

1996 / Received EC funding between 2000 and 2007

Received EC funding between 2003 and

2007

None

Call Centre

Operator (1)

GMI

Scrap-it Society

AB

SK

Climate

Change

Central

Saskatchewan

Environmental

Society

Climate Change

Central

Canadian Diabetes

Association

MB Manitoba Lung

Association


2007

Car Heaven

Manitoba Lung

Association

ON SI Ontario Automotive

Recyclers

Association

AQLPA QC

NB

PE

NS

NL

AQLPA

New Brunswick

Lung

Association

Clean Nova

Scotia

Newfoundland and Labrabor


2007


2008

None

None

Canadian Diabetes

Association

(centralized call centre for Atlantic provinces)

Lung

Association (2)

(1) Starting in June 2010, an overflow call centre came into operation to accept calls to the program outside of regular business hours and when local call centres were lacking capacity.

(2) After December 2010, program operations were taken over by the New Brunswick

Lung Association.



Annex 3: Evaluation Issues and Questions

Relevance

Was the Vehicle Scrappage Program consistent with, and did it contribute to, the federal government priorities, and did it address actual needs?

Question

1. a) Did the Vehicle

Scrappage Program respond to an identified need? What was that need? b) Is there a continued need for the program?

2. Did the program complement or duplicate other related federal and/or provincial government programs (i.e.,

Transport Canada programs, provincial emission testing programs, etc.)?

3. Was the Vehicle

Scrappage Program aligned with departmental and federal government priorities?

Indicators

 Demonstration of societal/environmental need

 Views on connection of program objectives with societal/environmental need

 Degree to which gaps would exist in addressing societal/environmental need in absence of the program

 Views on whether there is a continued need for the

Vehicle Scrappage Program

 Presence/absence of other government programs

(federal and provincial) that complement or duplicate the objectives of the program

 Views on whether the program complemented or duplicated other related programs

Data

Sources/Methods



Document review



KI interviews



Case studies (recycling code of practice)



Document review



Literature review

(including statistics on age of vehicles in use)



KI interviews



Case studies (recycling code of practice)



Document review





Literature review

KI interviews



Document review



Literature review



KI interviews

TB Policy Issue

Addressed



Issue #1:

Continued Need for Program



Issue #2:

Alignment with

Federal

Government

Priorities

4. Was the Vehicle

Scrappage Program consistent with the

D epartment’s mandate and with federal roles and responsibilities?

 Extent to which the program’s objectives corresponded to recent/current federal government priorities (e.g.,

CAA objectives)

 Extent to which the program’s objectives aligned with recent/current departmental strategic outcomes

 Views on the alignment of the program’s objectives with recent/current departmental and federal government priorities

 Extent to which program mandate was aligned with departmental mandate and federal government jurisdiction

 Views on the appropriateness of departmental and federal



Document review



Literature review



KI interviews



Issue #3:

Alignment with

Federal Roles and

Responsibilities


Audit and Evaluation Branch Evaluation of NVSP involvement

Performance

Has the Vehicle Scrappage Program achieved its intended outcomes? Were the most appropriate, efficient and economic means used to achieve outcomes?

Question

5. Were the objectives of the program realistic and adequately defined?

6. a) Was the design of the Vehicle

Scrappage

Program appropriate for achieving expected program results?

b) Was the funding of the program commensurate with its objectives and the needs identified?

7. a) Were participants satisfied with the administration of the program by both the

Department and the not-for-profit organizations? b) Were participants satisfied with the incentives provided to reward them for scrapping their vehicle (type and value)?

Indicators

 Demonstration that the Vehicle

Scrappage Program had realistic objectives

 Degree to which the program’s objectives are clearly understood by key stakeholders

 Views on the viability and clarity of program objectives

 Plausibility of link between program activities, outputs and intended outcomes

 Extent to which program processes and governance structure, including roles, responsibilities and accountabilities, were clearly defined and understood

 Views on appropriateness of program activities, processes and governance structure

 Evidence of the degree to which program resources and capacity were commensurate with expected program results

 Views on adequacy of program funding

 Level of participants’ satisfaction with administration of the program by the Department and the NFPs

 Level of satisfaction of participants with the incentives provided through the program

Data Sources/Methods



Document review





Literature review

KI interviews



Document review



Literature review



KI interviews



Case studies



Document review



Literature review



KI interviews



Secondary data analysis

(tracking research on

program participants)








Case studies

TB Policy Issue

Addressed



Issue #4:

Achievement of

Expected

Outcomes

[Design and

Delivery]



8. Were the achievements of the Vehicle

Scrappage

Program effectively and efficiently captured and disseminated to appropriate audiences (i.e., key stakeholders and the general public)?

9. Have appropriate performance data been collected, captured and safeguarded? If so, has this information been used to inform senior management / decision makers?

10. a) What are the lessons learned

(both strengths and weaknesses) from the program? b) Are there follow-up activities or programs that could build on the success achieved by this program?

11. To what extent have the intended outcomes been achieved as a result of the

Vehicle

Scrappage

Program?

 Degree to which program activities and results were effectively communicated to key stakeholders

(including provision of input into CAA

Horizontal Management

Accountability and Reporting

Framework)

 Availability of information on program results for the Canadian public

 Views on extent to which program achievements were effectively and efficiently captured and shared with key stakeholders and general public

 Existence of an effective performance measurement strategy

 Extent to which performance data are reliable, timely and relevant

 Extent to which performance data informed/supported EC’s decisionmaking processes

 Views on strengths and weaknesses of the Vehicle Scrappage

Performance Monitoring Framework

 Views about the program’s strengths, best practices and weaknesses (what worked well, what has not worked well, and what could have been done better)

 Extent to which program activities could lead to further future opportunities to meet the overall

CAA objectives or other environmental priorities

 Number of provinces and territories

(agreed upon in approved operating plan) in which program was available

 Number of provinces and territories

(agreed upon in approved operating plan) in which trained recyclers were available

 Views on extent to which outcome has been achieved



Document review



Literature review



KI interviews



Case studies























Document review


(program performance data)

KI interviews

Document review

KI interviews

Case studies

Document review

Literature review

KI interviews

Case studies


(program database)



Document review



KI interviews



Case studies




Issue #4:

Achievement of Expected

Outcomes

[Effectiveness]

Audit and Evaluation Branch Evaluation of NVSP f)

Short-term outcomes a) Canadians in all provinces have access to incentives to reward them for removing older vehicles from the road b) Increased awareness by the public of the program and the environmental impacts of older vehicles

 Change in level of awareness among Canadians (the general population and owners of older vehicles) about the environmental impacts of older vehicles

 Change in attitude toward the use of older vehicles

 Change in level of awareness among Canadians about the program

 Degree to which the public believed the program was a good idea

 Number of requests for information about incentives received via the website and/or 1-800 O-Canada

(Service Canada)


 Recyclers’ level of acceptance of and attitude toward the recycling code of practice

 Views on extent to which outcome has been achieved c) Increased awareness by recyclers of environmentally safe recycling practices

Medium-term outcomes d) Fewer older, high-polluting personal vehicles on the road e) More sustainable transportation modes being used for personal transportation

Environmentally safe practices in use for recycling of vehicles

 Number of older high-polluting personal vehicles scrapped as part of the program

 Reported influence of the program on participants’ decision to recycle their vehicle


 Uptake of incentives that do not involve the ownership of another vehicle

 Use of sustainable transportation during the incentive life (e.g., one year of free transit passes)

 Potential use of sustainable transportation beyond incentive life


 Level of adherence with code of practice by vehicle recyclers

 Level of support from provincial recycling associations and provincial




(tracking research on public)



Document review



KI interviews










(Vehicle Recyclers

Survey)



KI interviews



Document review

Case studies


(program database and tracking research on program participants)



Document review



KI interviews



Case studies


(program database and tracking research on program participants)



Document review



KI interviews



Case studies



KI interviews



Document review



Literature review



Case studies


Audit and Evaluation Branch Evaluation of NVSP scrapped through the program

Long-term outcomes g) Air pollutants are reduced h) GHG emissions are reduced governments for code of practice

 Extent of adoption of code of practice by provinces and other stakeholders


 Calculated reductions in smogforming emissions


 Calculated reductions in GHG emissions

 Views on extent to which outcome has been achieved i) Release of harmful and toxic substances into the environment is prevented

12. Did this program lead people to make use of other related federal and provincial government programs or vice-versa?

13. Were there any external factors outside of the

Vehicle

Scrappage

Program that influenced

(positively or negatively) the success of the program?

14. Were there any unintended positive or negative outcomes of the Vehicle

Scrappage

Program?

15. a) Did the Vehicle

Scrappage

Program undertake its activities and deliver its

 Number of vehicles recycled according to code of practice


 Level of involvement of program participants in the use of other related programs

 Views on extent to which outcome has occurred

 Evidence of factors outside the program which have influenced the achievement of intended outcomes

 Views on whether there were any external factors that helped or hindered the program’s success

 Presence or absence of unintended outcomes

 Views on whether unintended outcomes occurred as a result of the program

 Analysis of actual program operational costs in relation to the production of outputs

 Analysis of program administrative costs in relation to total program costs

 Views on whether the cost of




(program database)



Document review



Literature review



KI interviews




(program database)



Document review



Literature review



KI interviews










(program database)



Document review

Literature review

KI interviews






KI interviews



Case studies



Document review



Literature review



KI interviews



Case studies



Document review



KI interviews



Case studies



Document review



Financial analysis



Literature review



KI interviews



Issue #5 :

Demonstration of Efficiency and Economy

[Efficiency]


Audit and Evaluation Branch Evaluation of NVSP products in the most efficient manner? b) Was federal funding leveraged to the extent planned? c) How could the efficiency of the program be improved? d) Are there alternative, more efficient ways of delivering the program?

16. Did the Vehicle

Scrappage

Program achieve its intended outcomes in the most economical manner? producing program outputs was as low as possible

 Amount of financial commitments from incentive partners

 Amount of financial commitments and other support from provinces and territories toward the program’s delivery

 Number of partnerships achieved by national NFP with incentive partners that supported the program’s goals

 Views on how the efficiency of program activities could be improved

 Comparison of program activities and products delivered by other similar programs, including U.S.

Cash for Clunkers

 Views on whether there are alternative, more efficient ways of delivering program activities and outputs

 Extent to which the progr am’s intended outcomes have been achieved at the least possible program cost

 Views on whether good value was obtained with respect to the use of public funds

 Evidence of / views on whether there are alternative program models that would achieve the same expected outcomes at a lower cost



Document review



Financial analysis



Literature review



KI interviews



Issue #5:

Demonstration of Efficiency and Economy

[Economy]



Annex 4: Summary of Findings

Evaluation

Question

Achieved

Progress

Made;

Attention

Needed

Little

Progress;

Priority for

Attention

Relevance

1. a) Did the Vehicle Scrappage Program respond to an identified need?

What was that need?

1. b) Is there a continued need for the program?



2. Did the program complement or duplicate other related federal and/or provincial government programs (e.g.,

Transport Canada programs, provincial emission testing programs, etc.)?

3. Was the Vehicle Scrappage Program aligned with departmental and federal government priorities?

4. Was the Vehicle Scrappage Program consistent with the D epartment’s mandate and with federal roles and responsibilities?


5. Were the objectives of the program realistic and adequately defined?

6. a) Was the design of the Vehicle

Scrappage Program appropriate for achieving expected program results?

6. b) Was the funding of the program commensurate with its objectives and the needs identified?













Not

Applicable



7. a) Were participants satisfied with the administration of the program by both the Department and the not-forprofit organizations?

7. b) Were participants satisfied with the incentives provided to reward them for scrapping their vehicle (type and value)?







Evaluation

Question

Achieved

Progress

Made;

Attention

Needed

Little

Progress;

Priority for

Attention

Not

Applicable

8. Were the achievements of the Vehicle

Scrappage Program effectively and efficiently captured and disseminated to appropriate audiences (i.e., key stakeholders and the general public)?



9. Have appropriate performance data been collected, captured and safeguarded? If so, has this information been used to inform senior management / decision makers?

10. a) What are the lessons learned

(both strengths and weaknesses) from the program?

10. b) Are there follow-up activities or programs that could build on the success achieved by this program?


11. To what extent have the intended outcomes been achieved as a result of the Vehicle Scrappage Program?

Short-term outcomes

11. a) Canadians in all provinces have access to incentives to reward them for removing older vehicles from the road

11. b) Increased awareness by the public of the program and the environmental impacts of older vehicles

11. c) Increased awareness by recyclers of environmentally safe recycling practices

Medium-term outcomes

11. d) Fewer older, high-polluting personal vehicles on the road

11. e) More sustainable transportation modes being used for personal transportation

11. f) Environmentally safe practices in use for recycling of vehicles scrapped through the program

Long-term outcomes

11. g) Air pollutants are reduced

11. h) GHG emissions are reduced

























Evaluation

Question

Achieved

Progress

Made;

Attention

Needed

Little

Progress;

Priority for

Attention

Not

Applicable

11. i) Release of harmful and toxic substances into the environment is prevented

12. Did this program lead people to make use of other related federal and provincial government programs or vice-versa?



~ 

13. Were there any external factors outside of the Vehicle Scrappage

Program that influenced (positively or negatively) the success of the program?

14. Were there any unintended positive or negative outcomes of the Vehicle

Scrappage Program?


15. a) Did the Vehicle Scrappage

Program undertake its activities and deliver its products in the most efficient manner?

15. b) Was federal funding leveraged to the extent planned?

15. c) How could the efficiency of the program ’s activities be improved?

15. d) Are there alternative, more efficient ways of delivering the program?





16. Did the Vehicle Scrappage Program achieve its intended outcomes in the most economical manner?



~ Due to insufficient evidence, a complete assessment cannot be carried out.











Annex 5: Comparison with Other Jurisdictions

Comparative Summary of Key Features of Selected International Programs

Program

Feature

Program objectives

Program duration

U.S. UK

Economic stimulus

Environmental:

 improve overall fuel efficiency by modernizing vehicle fleet

Economic stimulus

Environmental

(not a priority):

 Remove older vehicles from the road

 Encourage purchase of new, safer and potentially more environment ally friendly vehicles

2009 2009-2010

Total program cost (approx. in $CDN)

$3B

($1B increased to

$3B after high early demand)

Cash incentive per vehicle

(approx. in

$CDN)

$3,500 –

$4,500

($USD)

Rebate for replacement vehicle increases with fuel efficiency

$700M

($500M increased by

$200M to add another

100 000 vehicles)

$3,500

(50% government and 50% mandatory industry contribution)

Germany

Economic stimulus

Environmental

(not a priority): although the primary program objectives were economic, the program was named “ecopremium

” to achieve broad political and social support

Economic stimulus

Environmental:

 CO

2 reduction

(secondary)

2009-2010

(end dates of

December

2009 for purchase of new vehicle and June

2010 for scrapping of old vehicle)

$8B

($2.4B raised to $8B after high early demand)

$4,000

France

2008-2009

(a scrappage incentive introduced in

2007 was substantially improved in

December

2008)

$950M

(initial $350M increased to

$950M)

$1,500

A bonus was paid to purchasers of lower-emitting vehicles: e.g.,

$1,500 for a car rated



Program

Feature

U.S. UK Germany France

(e.g., $4,500 for new vehicles with improved fuel efficiency of 5 or more MPG)

Age of vehicles targeted for scrappage

Replacement vehicle purchase required

Type of replacement vehicle required

< 25 years

Yes

New



10 years

Yes

New

Environmental criteria for replacement vehicles

Other restrictions / requirements

Scrappage requirements

Program takeup: maximum number of vehicles

Environmental impacts: CO

2 reduction –

Fuel efficiency must be a minimum of 18

MPG (approx.

13 km/100)

Maximum retail price

(MSRP) of

$45,000

Permanent disabling of the engine

678 000

N/A

None

Vehicle rebate must be matched by industry

None

400 000

280 843

(The average

CO

2

value of

133 g/km for



9 years

Yes

Maximum 14 months old

(with prior ownership in

Germany)

Vehicle must meet 2004 emission standards

None

2 000 000

1 150 204

(The average

CO

2

value of

142 g/km for under

101gallons per kilometre

(g/km); $7,500 for a car rated under 60 g/km

(electric cars).

The basic incentive was reduced in steps in 2010, i.e., to $750 in

June.



10 years

Yes

New

Vehicle must meet stricter

CO

2

standards of 160 g/km or less

None

600 000

276 782



Program

Feature cumulative annual abatement by

2011 (tonnes

CO

2

)

U.S.

Environmental impacts: fuel efficiency improvements

–average reduction per vehicle

9.2 MPG

(58% improvement overall: 10.3

MPG for passenger cars, 5.5 for

Category 1 trucks, 2.1 for

Category 2 trucks)

Other impacts Overall, 84% of consumers traded in trucks and

59% purchased passenger cars.

One of the most popular deals under the program was the exchange of an old full-size pickup truck for a new fullsize pickup.

Government reports estimate the program contribution to

U.S. 2009

GDP at $3.6B

(Council of

UK new cars purchased was

10.9% below the average value for nonincentivized new car purchases and

27% below the average value for scrapped cars)

N/A

Germany new cars purchased under the program was

8% below the average value for nonincentivized new car purchases)

N/A

France

N/A

The number of new cars from foreign car makers sold under the scrapping program was much higher than the number scrapped (i.e., the market share of non-

German manufacturers increased).

Sales of cars emitting less than 130 g/km increased from

30.4% in 2007 to 55.5% in

2009 as a result of scrappage incentives.

Sales of cars emitting more than 250 g/km

( “gas guzzlers ”)

“nosedived” from 24.3% in

2007 to 9.3% in 2009.



Program

Feature

U.S. UK Germany France

Economic

Advisors,

Sept. 2009) to

$3.8B (NHTSA

Report to

Congress,

Dec. 2009).

Other comments, critiques of government estimates of impacts

A University of

Delaware study concluded that each vehicle trade had a net cost of approx. $2,000 and that total program costs exceeded economic benefits by

$1.4B.

A University of

Michigan study found that the incremental improvement in the average fuel economy of all vehicles purchased under the program improved by a maximum of

0.7 MPG.

There was some fraud from resale of scrapped cars because there was no requirement to permanently disable the engine (i.e., an estimated

50 000 scrapped vehicles were exported to

Africa and

Eastern

Europe).

Sources of Information (International Comparison):

Assessment of the Effectiveness of Scrapping Schemes for Vehicles: Economic,

Environmental and Safety Impacts , IHS Global Insight, prepared for the European

Commission, DG Enterprise and Industry, Automotive Industry, March 2010, Final

Report and Country Profile Annex.

Evaluation of the Clean Transportation Theme of the Clean Air Agenda , Transport

Canada Evaluation and Advisory Service, November 2010.



A sample of international vehicle scrappage programs (September 2009), table of comparative statistics provided by the NVSP Outreach Division.

Une évaluation du bonus malus automobile écologique , Commissariat Général au

Développement Durable, N o 53, Mai 2010.

U.S. Department of Transport, August 26, 2009 press release on C.A.R.S. program statistics.

Is CARS a Clunker?, Burton A. Abrams and George R. Parsons, University of

Delaware, The Berkley Electronic Press, Vol. 6 (2009), Issue 8.

The Effect of the “Cash for Clunkers” Program on the Overall Fuel Economy of

Purchased New Vehicles , Michael Sivak and Brandon Schoettle, University of Michigan,

Transportation Research Institute, Report No. UMTRI-2009-34, September 2009.


MS Word

Evaluation of the

National Vehicle

Scrappage Program

Final Report

Table of Contents

EXECUTIVE SUMMARY

1.0 INTRODUCTION

2.0 BACKGROUND

2.1 Program Profile

2.2 Governance Structure

2.3 Resource Allocation

2.4 Expected Results and Program Logic Model

3.0 EVALUATION DESIGN

3.1 Purpose and Scope

3.2 Evaluation Approach and Methodology

3.3 Limitations

4.0 FINDINGS

4.1 Relevance

4.2 Performance – Design and Delivery

4.3 Performance – Effectiveness

4.4 Performance – Efficiency and Economy

5.0 CONCLUSIONS

6.0 LESSONS LEARNED

Annex 1: Bibliography

Annex 2: Network of Regional NFP Delivery Organizations

Annex 3: Evaluation Issues and Questions

Relevance

Performance

Annex 4: Summary of Findings

Annex 5: Comparison with Other Jurisdictions

Related documents

Products

Support

MS Word

Evaluation of the

National Vehicle

Scrappage Program

Final Report

Table of Contents

EXECUTIVE SUMMARY

1.0 INTRODUCTION

2.0 BACKGROUND

2.1 Program Profile

2.2 Governance Structure

2.3 Resource Allocation

2.4 Expected Results and Program Logic Model

3.0 EVALUATION DESIGN

3.1 Purpose and Scope

3.2 Evaluation Approach and Methodology

3.3 Limitations

4.0 FINDINGS

4.1 Relevance

4.2 Performance – Design and Delivery

4.3 Performance – Effectiveness

4.4 Performance – Efficiency and Economy

5.0 CONCLUSIONS

6.0 LESSONS LEARNED

Annex 1: Bibliography

Annex 2: Network of Regional NFP Delivery Organizations

Annex 3: Evaluation Issues and Questions

Relevance

Performance

Annex 4: Summary of Findings

Annex 5: Comparison with Other Jurisdictions

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib