Environment
Canada’s Invasive
Alien Species
Partnership Program
October 2009
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Report Clearance Steps
Planning phase completed
November 2008
Report sent for management response
September 2009
Management response received
October 2009
Report completed
October 2009
Report approved by Departmental Evaluation Committee
(DEC)
October 2009
Acronyms used in the report
IAS
Invasive Alien Species
IASPP
Invasive Alien Species Partnership Program
IASSC
Invasive Alien Species Strategy for Canada
TRC
Technical Review Committee
RMAF
Results-based Management and Accountability Framework
Acknowledgments
The Evaluation Project Team would like to thank those individuals who contributed to this
project, particularly members of the Evaluation Committee as well as all interviewees and
survey respondents who provided insights and comments crucial to this evaluation.
The Evaluation Project Team was led by Robert Tkaczyk, under the direction of the
Environment Canada Evaluation Director, Shelley Borys, and included William Blois and
Janet King.
This evaluation report was prepared by TDV Global Inc. and the Evaluation Division,
Audit and Evaluation Branch.
Environment Canada
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Table of Contents
Section
Page No
EXECUTIVE SUMMARY ................................................................................................... i
1.0
INTRODUCTION ..................................................................................................7
2.0
BACKGROUND ....................................................................................................7
2.1
3.0
Profile ...............................................................................................................7
EVALUATION DESIGN ......................................................................................12
3.1
Purpose and Scope ........................................................................................12
3.2
Evaluation Approach and Methodology ...........................................................13
3.2.2
4.0
Limitations ...................................................................................................14
FINDINGS ..........................................................................................................16
4.1 Relevance .............................................................................................................16
4.2
Success ..........................................................................................................24
4.3
Cost-Effectiveness ..........................................................................................37
4.4
Design and Delivery ........................................................................................41
5.0
CONCLUSIONS .................................................................................................50
6.0
RECOMMENDATIONS.......................................................................................51
7.0
MANAGEMENT RESPONSE .............................................................................53
Annex A National Invasive Alien Species Strategy for Canada Logic Model...................55
Annex B Evaluation Matrix .............................................................................................57
Annex C List of Background Information and Supporting Documentation ......................59
Annex D List of Interviewees ..........................................................................................60
Annex E Summary of Findings .......................................................................................61
Environment Canada
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
EXECUTIVE SUMMARY
Background
Environment Canada’s Evaluation Division, Audit and Evaluation Branch, conducted an
evaluation of Environment Canada’s Invasive Alien Species Partnership Program
(IASPP) in the 2008-2009 fiscal year. The evaluation was identified as part of the
Departmental Audit and Evaluation Plan approved in April 2008 and was undertaken to
fulfil Treasury Board evaluation requirements for program renewal as the program’s
terms and conditions are expiring at the end of 2009-2010. The evaluation was initiated
in October 2008 and completed in September 2009.
The IASPP is one part of a larger Government of Canada initiative on invasive alien
species (IAS). Administered by Environment Canada, the IASPP is a contribution
program with a budget of $5 million over five years (from fiscal year 2005-2006 to 20092010) designed to reduce the risk of future introductions and to help manage the spread
of existing IAS within Canada.
The evaluation examined the IASPP both within the context of EC priorities, as well as
within the context of the objectives of the joint national Invasive Alien Species Strategy
for Canada (IASSC), of which the IASPP is a component. The scope of this evaluation
was limited to examining EC’s participation in the IASSC through the IASPP. The
evaluation focused on the time frame of the fiscal years 2005-2006 to 2008-2009. The
main objectives of the evaluation were to provide conclusions and recommendations to
be applied in the context of program renewal.
Findings Summary
Relevance
Although there is a continued need for a contribution program in Canada to address IAS
at the grassroots level, there remain questions with respect to the effectiveness of the
program to achieve its overall objectives given issues related to a lack of concise and
focused program priorities and limited program funding. Some interviewees felt that
diverting the funding to other programs under the national strategy could have a greater
impact on addressing IAS-related concerns nationally.
Furthermore, although the IASPP was well-aligned with government-wide priorities in
2005, when the program was created, as of the present, there is only an indirect link
between IASPP objectives and current government-wide economic priorities.
The IASPP does contribute to the national strategy, and complements other programs
under the national strategy, in that it engages stakeholders beyond the federal and
provincial governments, targeting community-level and grassroots organizations.
Environment Canada is perceived as the best situated department to administer the
IASPP and coordinate the national strategy, given that it is seen as a neutral party
without a vested interest with respect to IAS.
Success
The program is perceived to be contributing to some extent to resolving the challenges
related to IAS in Canada. However, given the present limited funding of the program
these contributions have resulted in marginal gains at the local/community level, not
Environment Canada
i
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
major impacts at the national level. Furthermore, although there have been marginal
gains at the local/community level in raising the awareness of Canadians through
training and outreach, as well as through the generation and dissemination of products
related to IAS, these do not necessarily indicate that Canadians have become better
informed or more engaged in activities on priority IAS issues as a result of the program.
Finally, certain design and delivery limitations, such as uncertainty about the accuracy of
available performance data, further limit the possibility of determining the level of
success of the program in achieving its expected outcomes and overall objectives.
Cost-Effectiveness
Given the uncertainty about the accuracy of available performance data for the IASPP, a
cost-effectiveness analysis could not be conducted. Consequently, cost-effectiveness
was measured indirectly by examining program efficiency and key stakeholders’
perceptions of the program’s efficiency.
Evidence collected indicates that the program is being delivered in a cost-efficient
manner through its low administrative costs (13.4%), as well as its significant leveraging
of matching funds ($1.19:1 ratio).
No alternative approaches to a contribution program were identified that could be more
cost-effective or efficient at achieving the same objectives as the IASPP, although some
areas of improvement to the present approach were suggested, including the need to
focus more on an action-oriented approach to addressing IAS in Canada, as opposed to
focusing on education and outreach through the dissemination of information products
and training.
It was felt by most interviewees that limited operational funding had a negative impact on
the overall effectiveness of the program to achieve its objectives, in that it affected both
the management of IASPP contribution agreements, as well as Environment Canada’s
ability to fulfil its coordinating role with respect to the national strategy.
Design and Delivery
Overall, the program is being delivered as designed and continues to make operational
improvements (e.g., creation of a website, addition of an online application process).
There are a number of areas that need to be addressed in the delivery of the program,
including the implementation of an effective performance measurement strategy,
communications to proponents related to the overall objectives of the program, and
timeliness of funding to successful proponents.
Recommendations
These recommendations are directed to the ADM, ESB.
Recommendations 1 and 2:
Overall, although there is a continued need for a contribution program in Canada to
address IAS at the grassroots level, interviewees questioned the effectiveness of the
IASPP to achieve its overall objectives given issues related to a lack of concise and
focused program priorities and limited program funding. Most felt that although the
individual projects funded by the program were generally successful in achieving their
respective objectives, these results were at the local/community level. It was therefore
felt that the contributions of the program resulted in marginal gains at the
Environment Canada
ii
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
local/community level and not major impacts towards addressing IAS-related concerns at
the national level. Some interviewees felt that given these results, the limited funds could
be diverted to other programs under the national strategy to provide a greater impact at
addressing IAS-related concerns at the national level.
Interviewees spoke of the need to re-examine the overall objectives of the program, in
order to determine a few key priorities for the program where it could be expected to
make a difference. At present, it was felt that the current objectives were too broad,
resulting in the program being spread too thinly across the country, providing nominal
funding to small localized projects, instead of focusing on a few key priorities that would
result in the program making a bigger impact at addressing IAS-related concerns
nationally.
Furthermore, limited program funding also had a negative impact on the overall
effectiveness of the program to achieve its objectives, as it affected both the
management of IASPP contribution agreements and Environment Canada’s ability to
fulfil its coordinating role with respect to the national strategy. Many identified the need
for a more effective coordination of the national strategy, something they felt was lacking
at present.
Recommendation 1:
In the context of the upcoming renewal of the IASSC, the ADM, ESB should liaise
with counterparts in participating departments and agencies to determine the
appropriate role for the IASPP within the context of both the IASSC and EC
departmental priorities, along with commensurate funding for the IASPP.
Recommendation 2:
Based on the decision taken with respect to the appropriate role and
commensurate funding of the IASPP as per Recommendation 1, it is
recommended that the program’s objectives be re-examined with an aim to
develop a concise and more focused set of key priorities for the program to
ensure that the IASPP contributes in an effective manner to both the national
strategy and EC departmental priorities.
Recommendation 3:
Evidence collected as part of this evaluation illustrated a lack of clear understanding by
project proponents about the overall objectives of the IASPP, thereby making it difficult
for them to develop projects that link more directly with the program’s objectives. The
application guidelines do provide some guidance, but it is at a very high level, with more
emphasis on the evaluation criteria and listing examples of the types of activities that are
funded, than specifically stating the objectives or the expected results of the IASPP.
Another key weakness of the IASPP lies in its limited ability to demonstrate achievement
of its intended outcomes. This is due in part to the absence of a formal mechanism to
validate the performance data that are collected by project proponents and provided to
the program at the completion of individual projects. Thus, there is uncertainty about the
accuracy of the performance data provided. Furthermore, the program presently collects
information on approximately 114 key performance indicators, used to measure the level
of achievement related to 56 project-level outputs and outcomes. The sheer number of
performance indicators, as well as their diverse nature (i.e., environmental, economic,
Environment Canada
iii
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
community capacity, outreach, innovation, engaging Canadians, training) has made it
difficult for the program to consolidate performance data and use this data to report on
the overall success of the program in achieving its outcomes. Furthermore, in some
cases, there are issues with respect to the link between these performance indicators
and the project-level outputs and outcomes that they are intended to measure. Thus, the
program cannot use the indicators to determine whether IASPP-funded projects have
been successful in achieving specific project results and overall program objectives.
As a result, it is quite difficult to determine the extent to which the program has been
successful in achieving its immediate and intermediate outcomes, as identified in the
IASPP logic model.
Based on the decision taken with respect to the appropriate role and
commensurate funding of the IASPP as per Recommendation 1, as well as the
articulation of program priorities as per Recommendation 2, it is recommended
that:
a. the program clearly communicates its key priorities and objectives to potential
project proponents to enable their understanding and thus ensure that
projects which are developed are more directly linked to program objectives.
b. the program’s current performance measurement strategy be re-examined with
the objective to develop and implement a set of intended project-level
outcomes and performance indicators that would allow the program to be
better able to demonstrate its results.
Management Response
The Assistant Deputy Minister, Environmental Stewardship Branch, Environment
Canada (ADM, ESB) agrees with these three recommendations. Should there be a
government decision to renew the IASPP, the program commits to the following actions
in response to the three recommendations.
Recommendation 1:
In the context of the upcoming renewal of the IASSC, the ADM, ESB should liaise
with counterparts in participating departments and agencies to determine the
appropriate role for the IASPP within the context of both the IASSC and EC
departmental priorities, along with commensurate funding for the IASPP.
If renewed, the specific role of and priorities for the IASPP should focus on the IASCC
and EC departmental priorities, the priorities of the Canadian Food Inspection Agency
(CFIA) and Fisheries and Oceans Canada (DFO) with whom EC shares governance of
the IASPP, and the advice of other involved federal departments and agencies. The
following action will be taken by the ADM, ESB to ensure this recommendation is
addressed:
DATE
December
2009
ITEM

Environment Canada
ADM, ESB to discuss role of, priorities and commensurate funding
for the IASPP in implementing the renewed IASSC with other
federal departments and agencies.
iv
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Recommendation 2:
Based on the decision to be taken with respect to the appropriate role and
commensurate funding of the IASPP as per Recommendation 1, it is
recommended that the program’s objectives be re-examined with an aim to
develop a concise and more focused set of key priorities for the program to
ensure that the IASPP contributes in an effective manner to both the national
strategy and EC departmental priorities.
Following confirmation of the role and commensurate funding for the renewed IASPP,
the program’s objectives will be re-examined to create a more focused set of key
priorities aligned with both the national strategy and EC departmental priorities. In doing
so, EC will consult with the Canadian Food Inspection Agency, the Department of
Fisheries and Oceans and other key involved federal departments and agencies with
respect to their priorities for IASPP.
Specific actions to be taken to address this recommendation include:
DATE
ITEM

December 2009
Subject to renewal of the IASPP, identify strategic outcomes,
objectives and key priorities for the renewed IASPP in
consultation with other federal departments and agencies.
 Program to return to Environmental Sustainability Board with
recommendations for future implementation of the IASPP.
Recommendation 3:
Based on the decision to be taken with respect to the appropriate role and
commensurate funding of the IASPP as per Recommendation 1, as well as the
articulation of program priorities as per Recommendation 2, it is recommended
that:
a. the program clearly communicates its key priorities and objectives to potential
project proponents to enable their understanding and thus ensure that
projects which are developed are more directly linked to program objectives.
Subject to a renewed IASPP, EC will refine and update the IASPP application guidelines
including project selection criteria to ensure greater clarity about the overall objectives of
the IASPP, thereby ensuring projects link directly with the Program’s objectives.
Specific actions to be taken to address this recommendation include:
DATE
ITEM
January 2010

Refine the application guidelines in line with revised Program
objectives.
 Provide guidance such as a list of eligible and ineligible projects to
project proponents to ensure clarity with respect to IASPP
objectives and priorities.
b. the program’s current performance measurement strategy be re-examined with
the objective to develop and implement a set of intended project-level
outcomes and performance indicators that would allow the program to be
better able to demonstrate its results.
Environment Canada
v
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
The performance measurement strategy of a renewed IASPP will be revised in line with
the objectives and priorities to be established for the Program. Performance indicators
will be streamlined to ensure collection and consolidation of meaningful data which can
be used to report on the overall success of the program in achieving its outcomes.
To meet this goal, the following specific actions will be taken.
DATE
ITEM
March 2010

March 2011
March 2012
Refine current logic model to reflect the new objectives (outcomes)
for the Program
 Complete a Performance Measurement Strategy with the aim to
develop performance indicators most relevant to program
objectives
 Improve the reporting format, implement an appropriate reporting
tool and make other program adjustments as determined by the
departmental G&C risk assessment strategy developed as part of
the Departmental Action Plan for G&C Reform (CAPE Optimization
Initiative)
 Adopt the departmental online application and information
management system, which will enhance client service and
facilitate application, monitoring and reporting processes for both
clients and program staff (CAPE Optimization Initiative)
Contact persons: Bob McLean, Elizabeth Roberts
Environment Canada
vi
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
1.0 INTRODUCTION
Environment Canada’s Evaluation Division, Audit and Evaluation Branch, conducted an
evaluation of Environment Canada’s Invasive Alien Species Partnership Program
(IASPP) in the 2008-2009 fiscal year. The evaluation was identified as part of the
Departmental Audit and Evaluation Plan approved in April 2008 and was undertaken to
fulfil Treasury Board evaluation requirements for program renewal (as the program’s
terms and conditions expire at the end of 2009-2010).
This document presents the findings and recommendations of the evaluation and is
organized in the following way. Section 2 provides background information on the
IASPP. Section 3 presents the purpose of the evaluation and the methods used to
conduct the evaluation. Section 4 presents the evaluation’s findings. Sections 5 and 6
lay out, respectively, the conclusions and recommendations. The management response
is found in Section 7.
2.0 BACKGROUND
Invasive Alien Species (IAS) represent the second greatest threat to biodiversity, after
habitat loss. Alien species are species of plants, animals (including fish), and microorganisms introduced by human action outside their natural past or present distribution.
They are also known as exotics or specified as being foreign or non-native. Introductions
of alien species may be deliberate or accidental and may be beneficial, as in the
examples of corn, wheat, and domestic livestock, or damaging, such as in the cases of
leafy spurge, zebra mussels and wild boars.
IAS are those harmful alien species whose introduction or spread threatens the
environment, the economy, or society, including human health. Alien bacteria, viruses,
fungi, aquatic and terrestrial plants, mammals, birds, reptiles, amphibians, fish, and
invertebrates (including insects and molluscs) can all become invaders. In their new
ecosystems, IAS become predators, competitors, parasites and diseases of native and
domesticated plants, animals and marine life. The impact of IAS on native ecosystems,
habitats and species is severe and often irreversible, and can cost billions of dollars
each year.1
2.1
Profile
In September 2001, federal, provincial and territorial Ministers responsible for forests,
fisheries and aquaculture, wildlife, and endangered species chose IAS as one of four
priorities of the Canadian Biodiversity Strategy. With this direction, the federal
government, led by Environment Canada, along with the provinces, territories, industry,
and non-government groups developed a joint national Invasive Alien Species Strategy
for Canada (IASSC), which was approved in September 2004. At this time, direction was
given to the federal-provincial-territorial working groups to develop associated
implementation strategies for aquatic invasive species, invasive alien plants and plant
pests, and terrestrial animals.
1
Environment Canada. Invasive Alien Species Partnership Program website:
http://www.ec.gc.ca/eee-ias/default.asp?lang=En&n=C4637128-1
Environment Canada
7
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
The purpose of the IASSC is to minimize the risk of IAS to the environment, economy,
and society through a hierarchical approach that prioritizes prevention, early detection,
rapid response and management. Specifically, the IASSC aims to prevent new
invasions; detect and respond rapidly to new IAS; and manage established and
spreading IAS through eradication, containment, and control. The strategy also includes
measures to help prevent introductions of IAS from other countries or from species
which have moved from one ecosystem to another within Canada. Similar to other
countries, the focus of Canada’s IAS policy and management framework is on priority
pathways2 of unintentional and intentional introductions. The strategy is applicable to a
wide range of sectors, including agriculture, fisheries and aquaculture, wildlife, forests,
transportation, industry, and human health.
In October 2005, the implementation strategies developed by the working groups were
approved by federal, provincial and territorial Ministers. These strategies focused on risk
analysis, science, legislation and regulation, education and outreach, international
cooperation, and priority setting and included: The Canadian Action Plan to Address the
Threat of Aquatic Invasive Species; The Action Plan for Invasive Terrestrial Plants and
Plant Pests; and Canada’s National Wildlife Disease Strategy.
In the 2005 Federal Budget, four departments/agencies received funding of $85M in new
resources to implement the IASSC: Natural Resources Canada (Canadian Forest
Service) received $10M over five years; Fisheries and Oceans Canada received $20M
over five years; Agriculture and Agri-Food Canada (Canadian Food Inspection Agency –
Plant Health Division) received $50M over five years; and Environment Canada received
$5M over five years for the Invasive Alien Species Partnership Program (IASPP).
2.1.1 Program Description
Environment Canada’s IASPP is one part of a larger Government of Canada initiative on
IAS. Administered by Environment Canada, the IASPP is a contribution program ($5
million over five years) designed to reduce the risk of future introductions and to help
manage the spread of existing IAS within Canada, by funding initiatives intended to
address priority pathways of invasion for invasive alien plants, plant pests and aquatic
invasive species. Funded activities encourage stakeholders to prevent the introduction of
invasive species by increasing their understanding and awareness of invasive species
issues and facilitating responsible decision-making that minimizes the risk of unwanted
introductions.
The IASPP is administered through an allotment held in Environment Canada’s
reference levels. Funding is used to establish partnerships with the provinces and
territories, as well as with industry and non-governmental organizations to promote
compliance with regulatory approaches and establish voluntary initiatives for pathways
not amenable to regulation.
The Invasive Alien Species Strategy for Canada (2004) defines pathways as the “different ways
that IAS can be introduced” or “pathways of invasion” which may include transportation vectors
(e.g., boats, trains, planes) or in packaging material.
2
Environment Canada
8
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
2.1.2 Expected Project Results
The activities funded through the IASPP are directly linked to the education and outreach
component of the IASSC, specifically through the “Plan for Engaging Canadians and
Sectors” output in the national IASSC Logic Model (presented in Annex A). Activities are
intended to promote voluntary approaches and complement regulatory compliance
activities.
Expected results of approved projects fall into three main areas:

Active engagement of Canadians in minimizing the risk of invasive alien species;

Development of products and tools to reduce unintentional introductions through
specific pathways of invasion; and

Development of products and tools to increase general public awareness and
understanding of invasive species issues.
2.1.3 IASPP Governance Structure
IASPP is overseen by a Steering Committee, which is comprised of the Deputy Minister
of Environment Canada, the President of the Canadian Food Inspection Agency, and the
Deputy Minister of Fisheries and Oceans Canada, or their designates. Roles and
responsibilities of the Steering Committee, defined in the Interdepartmental
Memorandum of Understanding between Environment Canada, Canadian Food
Inspection Agency and Fisheries and Oceans Canada, include:

Approve projects for funding to be released by Environment Canada and, subject to
Treasury Board and Finance approval, recommend the amount of funding to be
reprofiled between fiscal years;

Review annual results to assess progress on outputs and outcomes;

Review annual cash flow forecasts and expenditures to date;

Report on results and achievements; and

Approve changes to the objectives of the Interdepartmental Memorandum of
Understanding between Environment Canada, Canadian Food Inspection Agency
and Fisheries and Oceans Canada, as appropriate.
An interdepartmental Director General Committee was established to coordinate the
overall implementation of the national IASSC including the IASPP.
A Technical Review Committee, coordinated by Environment Canada, was established
to provide technical, scientific, and policy advice for the review, selection, and
management of projects for the IASPP. The committee is comprised of experts from the
three signatory departments and, when appropriate, may include representatives of
Natural Resources Canada (Canadian Forest Service); Agriculture and Agri-Food
Canada, Parks Canada Agency, and Canada Border Services Agency in an advisory
role. The Terms of Reference of the committee were also established in the
Interdepartmental Memorandum of Understanding between Environment Canada,
Canadian Food Inspection Agency and Fisheries and Oceans Canada.
The activities of the Technical Committee include:
Environment Canada
9
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program

Review project proposals and develop recommendations annually for the selection of
projects;

Develop an annual costed workplan with recommended projects, milestones, targets,
and indicators;

Submit the proposed workplan with recommendations to the Steering Committee;

Monitor annual results to assess progress on outputs and outcomes;

Monitor annual cash flow forecasts and expenditures to date; and

Report annually on results and achievements to the Steering Committee.
2.1.4 Resource Allocation
As part of the 2005 Federal Budget, the IASPP received a total of $5 million for the
period FY 2005-2006 until 2009-2010 ($1 million per year) for the contribution
component of the national IASSC. No specific funding for program administration was
allocated to the program (i.e., Operations & Maintenance, salary); all operating expenses
for the IASPP were covered by Environment Canada’s A-base budget.
2.1.5 IASPP Logic Model
The 2008 Results-based Management and Accountability Framework (RMAF) includes
the logic model of the IASPP (Figure 1), which identifies the linkages between the
program’s activities and intended outcomes.
The key activities of the IASPP are two-fold: program-related and project-related. These
include planning and coordination, project assessment and review, as well as managing
agreements (i.e., monitoring progress, reporting on results). Key outputs of the program
identified in the RMAF (though not specifically in the logic model) include the following:

Contribution agreements with project proponents who undertake activities related to
the prevention, early detection, rapid response and management of IAS;

Communications products relating to both aquatic and terrestrial IAS developed and
disseminated for national use;

Education modules developed and utilized where possible and appropriate to raise
awareness regarding IAS; and

Other key products (e.g., codes of conduct, consumer information pamphlets, best
practices) developed in partnership with key stakeholders to minimize the risk of
introduction of IAS through priority pathways.
Environment Canada
10
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Figure 1: Invasive Alien Species Partnership Program Logic Model
Environment Canada
11
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
3.0 EVALUATION DESIGN3
The following sections outline the evaluation purpose and scope, along with the data
collection approach and methods used.
3.1
Purpose and Scope
The evaluation examined the IASPP both within the context of EC priorities, as well as
within the context of the objectives of the joint national IASSC. The main objectives of
the evaluation were to provide conclusions and recommendations to be applied in the
context of program renewal. The scope of this evaluation was limited to examining EC’s
participation in the IASSC through the IASPP. The evaluation focused on the time frame
of the fiscal years 2005-2006 to 2008-2009.
The following evaluation issues were addressed as part of this evaluation:
Relevance
Q1: Does the IASPP continue to be consistent with government-wide priorities and the
departmental mandate?
Q2: Is there a continued need for the IASPP?
Success
Q3: Are planned activities being implemented and producing expected outputs?4
Q4: Are immediate and intermediate outcomes being achieved as a result of the
program?
Q5: Have there been any unexpected outcomes (positive or negative)? Were any
actions taken as a result of these?
Cost-effectiveness/Alternatives
Q6: Are the most appropriate and efficient means being used to achieve objectives
relative to alternative design and delivery approaches?
Design and Delivery
Q7: Are program deliverables and expected outcomes identified clearly, and is the
program being delivered as designed?
Q8: Are activities and outputs linked to shared outcomes and the overall governmentwide objectives?
3
The design for the IASPP evaluation was carried out in the 2008-09 fiscal year, prior to
implementation of the new Treasury Board Policy on Evaluation (http://www.tbs-sct.gc.ca/pol/doceng.aspx?id=15024). The current evaluation reflects those issues outlined in the 2001 evaluation
policy that was in effect at the time this evaluation was conducted.
4
Given that the information gathered to address this question is similar in nature to the information
gathered to address Question 7 under Design and Delivery, the decision was taken to present the
findings for both these questions together, under Question 7.
Environment Canada
12
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Q9: Are performance data collected against program activities/outcomes and used for
informed decision making?
Q10: Are there appropriate governance structures in place to meet the objectives?
Q11: Has there been an assessment and strategic use of lessons learned?
The evaluation matrix, mapping each question to the related indicators, data sources
and methodologies, is presented in Annex B.
3.2
Evaluation Approach and Methodology
This section describes the methods that were used to conduct the evaluation of the
IASPP as well as the limitations of the evaluation.
3.2.1 Methods
Document Review
Existing documentation was reviewed early in the data collection phase in order to
establish a sound basis for the interviews and survey. Performance monitoring
information on specific performance indicators related to program outputs and results
was collected and analyzed. The performance measurement system as a whole was
also assessed. Gaps, indicator weaknesses and/or data vulnerabilities were identified.
Please see Annex C for the list of documentation reviewed.
The goal in this step was to answer as many of the questions within the evaluation
matrix as possible and then to validate the findings where necessary and appropriate
through other data collection methods.
This data collection method addressed evaluation questions 1 through 11.
File Review
A total of 8 out of 71 completed project files were reviewed at the initiation of the
evaluation to determine the extent to which project-related performance information was
available, as contained in the “Key Indicators of Success Report,” which is prepared and
submitted by each recipient at the end of their project. The report template consists of a
large range of indicators categorized under various themes such as prevention,
monitoring, and community capacity building, among others. Performance data from all
71 completed projects, which program staff had recently consolidated, were used to
assess the achievement of program outcomes.
This data collection method addressed evaluation questions 4, 5, 9 and 10.
Key Informant Interviews - Round I
A total of nineteen interviews were conducted with Environment Canada program
management and staff, other government departments (OGD), and provincial and nongovernmental organization stakeholders which were successful project proponents (a list
of interviewees can be found in Annex D). A set of interview guides were prepared,
which were provided to interviewees beforehand. These guides explained the purpose
and provided the questions that were to be asked during the interview. For each
interview, a set of interview notes was prepared. Interviews were conducted either in
person or by telephone, and in the interviewee’s language of choice.
This data collection method addressed evaluation questions 1 through 11.
Environment Canada
13
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Key Informant Interviews - Round II
A total of 12 additional key informant interviews were conducted in July and August
2009, with program staff and Environment Canada departmental senior management
(4), Interdepartmental Directors General IAS Steering Committee members (4), and
federal department and agency partners (4). A list of interviewees can be found in
Annex D. Interviews focused on the following issues:
This data collection method addressed evaluation questions 1, 2, 4 and 6.
Survey
An online survey of funding applicants (both successful and unsuccessful) was
conducted between February 18, 2009 and March 9, 2009. The survey served as a
validation tool for findings that were identified through interviews and document review.
The survey was distributed to 100 proponents (72 successful, 28 unsuccessful) – a total
of 37 responses were collected for a response rate of 34.7 per cent and 42.9 per cent,
respectively (25 successful, 12 unsuccessful).
This data collection method addressed evaluation questions 2, 3, 6, 7, 9 and 11.
3.2.2
Limitations
There are five specific limitations associated with this evaluation.
Performance Information
Evaluators relied on consolidated performance information submitted by each recipient
once their project is completed (71 out of 108 approved projects at the time of the
evaluation) contained in the “Key Indicators of Success Report,” as well as on
testimonies from key informants to assess the success of the program in meeting its
immediate and intermediate outcomes. The information that was submitted by each
recipient once their project is completed has not been validated by the program or by this
evaluation; therefore, there is uncertainty about the accuracy of the performance data.
Specifically, there are issues of duplication (i.e., double-counting) and inconsistency of
performance data, as well as a disconnect between some performance indicators and
the project-level outputs and outcomes that they are intended to measure. Thus, the
evaluators’ ability to report on the achievement of program outcomes was limited.
Furthermore, performance information contained in the “Key Indicators of Success
Report” regarding targets and percentage achieved of targets can be misleading. In
many cases, targets were not established at the beginning of each project; therefore, it
is difficult to accurately determine the level of success of a project in achieving its
objectives.
Financial Information
The IASPP received a total of $5 million in funding over five years (approximately $1
million per year) for the contribution component of the program. No additional funding for
program administration was approved (i.e., Operations & Maintenance and salary) and
all operating expenses for the IASPP were covered by Environment Canada’s A-base
budget. The financial information with respect to program administration was provided by
the program lead. The breakdown of Operations & Maintenance costs is derived from
the financial system but the allocation of salary costs to the program is a best estimate.
Therefore, total operational expenses may not be completely accurate, which would
Environment Canada
14
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
affect the comparison of operational expenses to contributions used to examine program
efficiency.
External Perspective
The input of individuals and groups who have not benefited from or been involved in
delivering the program would have been useful to provide an external perspective on the
program’s relevance, to explore possible limitations to the program’s reach, and to gain
external views on the program’s perceived impacts to date. Given the scope of the
evaluation (based on the low materiality of the program), such individuals were not
engaged as part of the evaluation.
Scope of the Evaluation
Each of the participating departments and agencies undertook an evaluation of their
respective IAS-related programs as per the national 2008 IASSC RMAF. The RMAF
indicated that each department would conduct an evaluation of their respective
programs. Thus, the scope of this evaluation was limited to examining EC’s participation
in the IASSC through the IASPP. The evaluation did not examine any aspects related to
the IASSC itself (e.g., shared governance, shared outcomes as per the 2008 IASSC
RMAF).
Survey Sample Response
Although the survey response rate was 34.7 per cent for successful project proponents
and 42.9 per cent for unsuccessful project proponents, actual numbers of respondents
for each category are relatively low. Given the small number of respondents which
limited the ability of the evaluation to generalize survey findings, the survey data should
not be interpreted in isolation from other sources of evaluation data. The results of the
survey, however, closely matched the findings in the key informant interviews and
support the conclusions reached from other sources of evaluation evidence.
Environment Canada
15
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
4.0 FINDINGS
This section presents the evaluation findings by evaluation issue (relevance, success,
cost-effectiveness, design and delivery) and by the related evaluation question. The
findings at the overall issue-level are presented first, followed by the findings for each
evaluation question.
A rating is also provided for each evaluation question based on a judgment of the
evaluation findings. The rating symbols and their significance are outlined below in
Table 1. A summary of ratings for the evaluation issues and questions is presented in
Annex E.
Table 1: Rating Symbols and Significance
Symbol
Significance
Achieved
The intended outcomes or goals have been achieved or met
Progress Made;
Attention Needed
Considerable progress has been made to meet the intended
outcomes or goals, but attention is still needed
Little Progress; Priority
for Attention
N/A
~
Little progress has been made to meet the intended outcomes or
goals and attention is needed on a priority basis
Items where a rating is not applicable
Outcomes achievement ratings are based solely on subjective
evidence
4.1 Relevance
Evaluation Issue: Relevance
Overall Findings: Interviewees felt that there is a continued need for a contribution
program in Canada to address IAS at the grassroots level. However, although many felt
that the individual projects funded by the program were successful and effective, they
questioned the effectiveness of the program to achieve its overall objectives given
issues related to a lack of concise and focused program priorities and limited program
funding. Some interviewees felt that the funding could be diverted to other programs
under the national strategy to provide a greater impact at addressing IAS-related
concerns nationally.
Although the IASPP was well-aligned with government-wide priorities in 2005, when the
program was created, as of the present, there is only an indirect link between IASPP
objectives and current government-wide priorities (many interviewees spoke of the
negative economic impact of IAS on various industries including fisheries, agriculture
and forestry).
Overall, the IASPP does contribute to the national strategy. It complements other
programs under the national strategy, in that it engages stakeholders beyond the federal
and provincial governments, targeting community-level and grassroots organizations.
The program is unique in that it is the only contribution program which provides funding
to these organizations to take action related specifically to IAS.
Interviewees felt that if the IASPP did not exist, there would be also a decrease in
Environment Canada
16
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
coordination of IAS-related activities across the federal government. Environment
Canada is perceived as the best situated department to administer the IASPP and
coordinate the national strategy, given that it is seen as a neutral party without a vested
interest with respect to IAS.
Evaluation Issue:
Relevance
Q1: Does the IASPP continue to
be consistent with governmentwide priorities and the
departmental mandate?
Indicator(s)
 Program objectives that are
aligned to government-wide
priorities and departmental
mandate.
 Inter/intra-governmental
relations and coordination
(F/P/T and First Nations).
Methods
Rating
 Document
review
 Interviews
Progress
made,
attention
needed
Alignment to Government-wide Priorities
A review of relevant documents, including the Speech from the Throne (2005) and
Budget 2005, reveals that the IASPP was consistent with government-wide priorities in
2005 when the program was created as part of the national strategy. Budget 2005
provided $85 million in funding for the national strategy for 5 years ($5 million was
allocated for the IASPP over the same period). However, more recent documents do not
provide any link between the IASPP and current government-wide priorities.
The majority of Round II interviewees felt that the objectives of the IASPP aligned to
some extent with current government economic priorities. Specifically, many mentioned
the negative economic impact of IAS on various Canadian industries, most importantly
fisheries, agriculture, and forestry with respect to natural resources (e.g., lumber, crops),
trade, as well as operating costs. In a 2003 study, examining 16 species, it was
estimated that cumulative annual costs related to IAS within various sectors was
between 13.3 and 34.5 billion dollars.5 A few individuals noted that although IAS is not
“making the headlines” as it did when the national strategy was developed in 2004, the
issue has not disappeared and continues to be a concern. One individual specifically
discussed IAS in the context of the Species at Risk Act (SARA), stating that SARA is an
important issue for government given the significant amount of funding that the federal
government continues to provide to address species at risk issues. In 2002, it was
estimated that about 24 percent of species at risk in Canada listed by the Committee on
the Status of Endangered Wildlife in Canada may be threatened with extinction by IAS.6
Given that IAS is a significant threat to species at risk, it therefore continues to be an
indirect government priority.
5
Government of Canada. An Invasive Alien Species Strategy for Canada. September 2004. Pg. 10.
6
Government of Canada. An Invasive Alien Species Strategy for Canada. September 2004. Pg. 9.
Environment Canada
17
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Contribution to IASSC
Round II interviews indicated the IASPP does contribute to the joint national IASSC. The
national strategy aims to target investment of new funding to support science-based
regulatory initiatives (including risk assessment and import controls), strengthen national
surveillance for early detection, and raise public awareness and understanding of
harmful practices that introduce IAS into Canada. The IASPP was included in the
national strategy to complement the activities of other federal department and agency
partners which primarily focused on science-based regulatory initiatives and national
surveillance, while the IASPP was intended to promote public compliance with regulatory
approaches and to establish voluntary initiatives to address pathways not amenable to
regulation. Specifically, the activities funded through the IASPP are directly linked to the
education and outreach component of the IASSC, specifically through the “Plan for
Engaging Canadians and Sectors” output in the IASSC Logic Model.
Most interviewees indicated that the IASPP does complement the other activities of the
national strategy. The IASPP specifically targets communities and organizations in order
to develop IAS-related activities at the local level, thereby engaging stakeholders beyond
the federal and provincial government, which fills a gap outside the scope of the
activities of other federal departments and agencies. As one individual noted, “it finds a
niche that has not been covered by the jurisdictions (federal or provincial),” focusing on
raising awareness at the community-level and engaging grassroots organizations to take
action related to IAS within their communities.
Alignment to Departmental Priorities
In terms of current departmental priorities, Environment Canada program staff and
senior management discussed alignment to SARA (discussed above), the Migratory
Birds Convention Act (i.e., impact IAS have on nesting areas of native birds, destroying
habitat), Wild Animal and Plant Protection and Regulation of International and
Interprovincial Trade Act (WAPPRIITA) and the United Nations Convention on
Biodiversity.
WAPPRIITA is the legislative vehicle by which Canada meets its obligations under the
Convention on International Trade in Endangered Species of Wild Fauna and Flora
(CITES), by regulating the international and interprovincial trade in animals and plants.
Environment Canada is the federal department responsible for the implementation of the
Act, which provides the authority to designate species as invasive or harmful. While
Environment Canada is responsible for terrestrial animals from a regulatory sense,
Fisheries and Oceans Canada is responsible for regulating most aquatic species (in
certain cases, Canadian Food Inspection Agency regulates aquatic plants7). The
Fisheries Act does not provide authority to designate aquatic species as invasive or
harmful (the only possible mechanism to do so is WAPPRIITA). However, as of present,
no aquatic species has been designated under WAPPRIITA as invasive or harmful.
Furthermore, one interviewee stated that the focus of WAPPRIITA is on endangered
species not necessarily IAS.
The document review revealed that the IASPP is aligned with the requirements under
the United Nations Convention on Biological Diversity, of which Canada is a signatory
member. Canada joined the United Nations Convention on Biological Diversity in 1992
7
Plant and plant pests are regulated under the Plant Protection Act and Animal Quarantine Act.
Environment Canada
18
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
and developed the Canadian Biodiversity Strategy in 1995, with the goal of monitoring
and controlling importation of alien species. Article 8 (h) of the Convention indicates that
participating states must “prevent the introduction of, control or eradicate those alien
species which threaten ecosystems, habitats or species.”
A review of Environment Canada’s corporate documents (Report on Plans and Priorities
2008-2009; Departmental Performance Report 2007-2008) provides evidence that the
IASPP remains consistent with the department’s mandate. All interviewees also felt that
the IASPP is most strongly aligned with the department’s priority related to the protection
of biodiversity. IAS could lead to population declines in native species and reduced
biodiversity. For instance, in the Great Lakes, sea lamprey have been implicated in the
extinction of the Deepwater Cisco, and zebra mussels have extirpated native mussels
from some areas.8 The objectives of the IASPP align with the departmental intermediate
outcome “Biodiversity is conserved and protected,” which is linked to the departmental
strategic outcome “Canada’s natural capital is restored, conserved and enhanced.” The
approved departmental Program Activity Architecture for fiscal year 2010-2011 has also
identified biodiversity as a departmental intermediate outcome, linked to the
departmental strategic outcome “Canada's natural environment is conserved and
restored for present and future generations.”
Interviews with other government departments also indicate the IASPP continues to be
relevant. All representatives indicated that the IASPP aligns well with their own efforts
and goals. Interviewees from Natural Resources Canada felt that the IASPP does align
well with its own priorities. Specifically, the Canada Forest Service has a Forest IAS
Program and a National Forest Pest Strategy that are based on the same priorities of the
IASSC. The Canadian Food Inspection Agency sees the IASPP as complementary to its
own activities and supportive of its departmental strategic outcome, “A safe and
sustainable plant and animal resource base.” One CFIA interviewee stated that the
“goals and objectives of the IASPP are very similar in that they want to reduce the
spread of IAS in Canada.”
Evaluation Issue:
Relevance
Indicator(s)
Q2: Is there a continued need for  Utility/rationale for the program
the IASPP?
 Ongoing need for capacity or
activity in the area of
departmental jurisdiction
Methods
Rating
 Document
review
 Interviews
 Survey
Progress
made,
attention
needed
Continued Need for the IASPP
The prevention, early detection, rapid response and management of IAS are a
continuing challenge in Canada. According to the International Union for Conservation of
Nature9, an international organization dedicated to natural resource conservation, IAS
8
http://www.ec.gc.ca/eee-ias/default.asp?lang=En&n=805A5D5C-1.
9
The International Union for Conservation of Nature is one of the largest global environmental
networks, with more than 1,000 government and NGO member organizations, and almost 11,000
volunteer scientists in more than 160 countries.
Environment Canada
19
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
are the second most significant threat to biodiversity after habitat loss. They often
dominate the ecosystems they invade, causing losses to productivity and market access
and increasing costs to producers.
Overall, Round II interviewees felt that there is an ongoing need for a contribution
program in Canada to address IAS at the grassroots-level. All interviewees felt that IAS
is an ongoing relevant issue and that there is a need for a contribution program aimed at
engaging grassroots organizations and the public to undertake activities related to
raising awareness of IAS. However, most interviewees stated that there are two
significant related issues that the IASPP needs to address going forward: re-examining
the priorities of the program, and program funding.
A majority of Round II interviewees stated that, although the goals of the program were
relevant and the projects funded by the program were making a difference in their
communities, they questioned the overall impact the program was making with respect to
raising awareness at the national level. As one interviewee stated, there is a “need for a
more structured approach focusing on critical issues and not funding tiny projects
scattered across the country.” Most interviewees felt the program was using a “shotgun
approach”, thereby spreading itself too thinly (i.e., providing nominal funding to many
small localized projects spread across Canada instead of focusing on critical issues of
importance related to IAS at the national level), which had a negative impact with respect
to the overall effectiveness of the program to address IAS-related concerns nationally.
As another interviewee stated, “there is a need to focus on a small number of priorities
and make a bigger impact, not on many projects with little likelihood of really making a
difference nationally.” At most, interviewees felt that there were marginal gains from the
individual projects funded by the program at the local level (i.e., individuals reached, IAS
addressed), not major changes.
Many interviewees discussed the need to determine a few key priorities for the program,
where it can really make a difference. These included among others a focus on priority
pathways and IAS (e.g., Emerald Ash Borer, Asian Longhorn Beetle), linking priorities of
the program to the priorities of federal department and agency partners (i.e., Canadian
Food Inspection Agency, Fisheries and Oceans Canada, Natural Resources Canada),
linking IAS priorities to certain species (e.g., species at risk), or focusing on only one of
the four key strategic goals of the joint national strategy, namely prevention, early
detection, rapid response and management.
Furthermore, although most felt that the approach used by the program (i.e., projects
funded through contribution agreements) was an effective approach to raising public
awareness and engaging grassroots organizations, they also felt the overall
effectiveness of the program was hampered by the total amount of program funding.
Most felt that, at one million dollars per year of available contribution funding, the
program could not achieve its overall objectives. As one individual noted, “to say that the
program would fully address all aspects of IAS [i.e., prevention, early detection, rapid
response, and management] is trying to make too much of the program because it is so
small.” A few interviewees questioned whether the money could be used elsewhere to
have a greater impact with respect to IAS in Canada. One individual stated that “a million
dollars is a drop in the bucket. At a million dollars, cutting this program will have no effect
on the national strategy.” Another individual noted that “for one million dollars, you’re not
even in the game. You might as well spend it elsewhere as you can’t make enough of a
difference.” Similarly, another individual stated that with such limited resources available,
the funding could be used elsewhere to provide a greater impact on addressing IASrelated concerns. Although the relevance of the program was not questioned in and of
Environment Canada
20
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
itself, the ability of the IASPP to achieve its overall objectives given present funding
levels was raised as a concern.
Gaps/Complementarity/Duplication of IASPP
Round II interviewees also discussed the gaps that would emerge if the IASPP did not
exist. Most discussed the important contribution of the IASPP to the national strategy in
terms of engaging local and community-level organizations. If the IASPP did not exist,
interviewees felt that an important capacity to detect IAS at the community-level would
be lost (e.g., monitoring program in Manitoba focusing on detecting Dutch Elm Disease,
Brown Spruce Longhorn Beetle in Halifax, Emerald Ash Borer in Windsor). Without
these grassroots organizations, it was felt that the strategy would lose an important
component to address IAS. It was also felt that it would create a perception that
government was not engaged at the community level.
Furthermore, interviewees felt that there would be a decrease in Environment Canada’s
ability to fulfil its coordinating role with respect to IAS-related activities across the federal
government. Most interviewees mentioned that Environment Canada was the best
placed of all federal departments and agencies to take on the coordination role with
respect to the national strategy. Most perceived the department to be an “honest broker
with respect to national coordination and delivery.” Whereas the other departments and
agencies were seen to have a vested interest with respect to IAS, either species-related
(Fisheries and Oceans – aquatic; Canadian Food Inspection Agency – plant and plant
pests) or sector-related (Agriculture and Agri-Food Canada – agriculture; Natural
Resources Canada – forestry), Environment Canada, on the other hand, was perceived
to have the broadest mandate to address IAS-related issues and to be a neutral party.
As a result, interviewees felt that there was no other federal government department or
agency that could take on the coordinating role with respect to the national strategy.
Most interviewees also felt that Environment Canada should continue to administer the
IASPP. By not having a vested interest, Environment Canada was considered to be the
best situated to ensure that there existed one global vision and message related to IAS
issues, whereas if it was parceled off, each department and agency would focus on their
own specific interests.
Most Round II interviewees indicated that the IASPP is the only funding mechanism
which is national in scope and focuses on engaging grassroots-level organizations to
address issues related to IAS, through both action-oriented (e.g., pulling weeds) and
information-oriented (e.g., developing and distributing pamphlets and brochures)
projects. Stakeholder interviews also highlighted that there are limited funding
opportunities to address IAS challenges at local/regional levels and that the IASPP was
often the only funding source for IAS-related activities. This has created a high demand
for the IASPP as evidenced by the number of applications received during each request
for proposals, which average over 150 per year (to date, a total of 108 projects have
been approved for funding, an average of 27 per year).
Table 2 provides some details with respect to the demand on the IASPP. The requestfor-proposals for 2005-2006 and 2006-2007 were combined into one process due to
delays in starting up the program and getting funds disbursed. On average, there are
almost $8 million worth of proposals submitted each year for a program that provides, on
average, $1 million per year in contributions. The amounts approved do not always
reflect the planned distribution of $1 million per year. As mentioned, delays in the first
year caused a combined call, and some of those projects received multi-year funding
which decreased the amount available for 2007-2008.
Environment Canada
21
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Table 2: Result of Call for Proposals
Combined Call
2005/06
2006/07
Number of
310
Applications
Number of
58
Approvals
Percentage
18.7%
Approval
Total Request
$19m
(approx.)
Total
Approved
$2,368,139
Funding*
2007/08
2008/09
123
192
18
32
14.6%
16.6%
$7m
$7.3m
$466,000
$844,320
* Values are for total approved funding – see Table 13 for disbursed amounts.
Furthermore, without IASPP funding, many IAS-related projects are not implemented.
Survey results revealed that of those project proponents who were not funded by the
IASPP, 36% (4 respondents) stated the project was cancelled, 27% (3 respondents)
stated the project was postponed, and 36% (4 respondents) said the project continued
but at a reduced level.
Overall, interviewees felt that these gaps could not be filled by any other entity.
Specifically, interviewees mentioned that provinces do not have the available resources
to fund local projects. Furthermore, most felt that parceling off the program to specific
federal departments/agencies would increase the likelihood of duplication of effort (e.g.,
invasive alien plants could be considered either the responsibility of the Canadian Food
Inspection Agency or Fisheries and Oceans Canada) and could lead to gaps in the
national strategy (one individual stated that Environment Canada “kept the federal family
together” by bridging the gap between federal departments and agencies who would
otherwise continue to work in silos).
When asked whether the program duplicates other programs in Canada, most
interviewees stated that the program is unique. Although other programs may exist at
the federal or provincial level which fund IAS-related activities (e.g., the Ontario
Stewardship Fund, which does occasionally fund IAS-related projects, Agriculture and
Agri-Food Canada’s Growing Forward10), the IASPP is the only contribution program
which specifically addresses the issue of IAS at the national level. Most interviewees felt
that the program is complementary to other programs in Canada, including Habitat
Stewardship Program (HSP), which focuses on species at risk, and EcoAction, both
administered by Environment Canada. These three programs coordinate their processes
to ensure, for the most part, that IAS-related projects are not funded by either HSP or
EcoAction (though the HSP has funded IAS projects related to species at risk).
EcoAction, although a community-based contribution program also has limited funds to
address IAS-related projects. However, a few interviewees felt that there is a clear
10
Growing Forward focuses on building a profitable agricultural sector through its three strategic outcomes:
a competitive and innovative sector, a sector that contributes to society’s priorities, and a sector proactive in
managing risks).
Environment Canada
22
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
overlap between the IASPP and the HSP. One individual indicated that the scope of the
HSP could be marginally expanded to include the objectives of the IASPP. Specifically,
the most important concern surrounding IAS in Canada is the negative impact they have
on species which are listed as at risk, and therefore IAS-related projects should focus on
those IAS which negatively affect species at risk.
Environment Canada
23
Audit and Evaluation Branch
4.2
Evaluation of the Invasive Alien
Species Partnership Program
Success
Evaluation Issue: Success
Overall Findings: Overall, interviewees noted the establishment of the
regional/provincial councils as the program’s key accomplishment to date. These
councils are important in that they facilitate partnerships between various key
stakeholders and provide an important contribution to the development of regional
priorities related to IAS.
The overall consensus of interviewees is that the IASPP is contributing to some extent
to resolving the challenges related to IAS in Canada. However, given the present
funding level of the program, most felt that these contributions resulted in marginal gains
at the local/community level (with respect to informing and engaging Canadians, as well
as addressing IAS), not major impacts at the national level.
In addition, although the program has been successful to some extent (at the
local/community level) in raising the awareness of Canadians through training and
outreach, as well as through the generation and dissemination of products related to
IAS, these do not necessarily indicate that Canadians have become better informed or
more engaged in activities on priority IAS issues as a result of the program. Without
baseline data, it is difficult to determine the program’s impact in getting Canadians
better informed and engaged in IAS-related activities. There is also less evidence to
suggest that the program has been successful in achieving the immediate outcome
“threat to natural capital reduced.”
Furthermore, given issues related to the accuracy of available performance data, it is
also difficult to determine the level of success the program has achieved.
Evaluation Issue: Success
Q4: Are immediate and
intermediate outcomes being
achieved as a result of the
program?
Indicator(s)
 As per indicators in the
performance measurement
plan and reporting strategy
Methods
Rating
 Document
review
 File
review
 Interviews
~ Progress
made,
attention
needed
Overall, Round II interviews indicated the establishment of the regional/provincial
councils11 as the program’s key accomplishment to date (including Alberta, Ontario,
Manitoba and Saskatchewan). Before the inception of the program, only BC had
established an IAS-related council. These councils have facilitated the establishment of
partnerships between federal and provincial governments, industry, non-governmental
11
Regional/provincial councils generally work in a specific region or province to minimize the
negative ecological, social, and economic impacts caused by the introduction, establishment, and
spread of invasive plants. They are non-profit organizations whose members are involved in all
aspects of invasive plant management. Members may include technical specialists working for
government and industry, weed committee coordinators, First Nations representatives, foresters,
forest technologists, biologists, ranchers, horticulturists, recreation enthusiasts, gardeners, and
other concerned individuals.
Environment Canada
24
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
organizations and aboriginal groups. These councils have provided an important
contribution to the development of regional priorities related to IAS.
Other key accomplishments mentioned by interviewees included raising awareness with
targeted communities on priority pathways (e.g., recreational boaters and anglers
regarding cleaning the hulls of boats and transferring baitfish between bodies of water,
horticulturalists, and aquariums). However, it was mentioned that these
accomplishments have not occurred in a comprehensive fashion nationally.
Another key accomplishment cited by interviewees was the creation of a network across
the country and sharing of products between various key stakeholder groups (e.g.,
creation of a particular pamphlet related to a specific IAS).
Given the funding level of the program, however, most interviewees agreed that the
program did not have the resources to achieve its objectives. Although there was
success at the individual project level, overall, the program at most had marginal gains at
the local level, not major accomplishments at the national level.
There are four immediate outcomes and four intermediate outcomes identified in the
2008 IASPP logic model. The following sections will address each one separately.
Performance information is derived from the consolidation of the “Key Indicators of
Success Report,” which is part of the final report for each project. In total, 71 projects out
of the total of 108 projects that have been funded are complete and have submitted this
report. As already noted, the information in these reports is not verified or audited so the
accuracy of available performance data is uncertain. It can, however, provide some
indication of the results of the program as a whole.
Evidence exists that projects are generating outputs which are contributing to immediate
and intermediate outcomes. A project file review and an analysis of available
performance information clearly show significant production of outputs. These are
presented in the following sections under the respective outcome to which they relate.
Almost all interviewees indicated that the program is achieving the intended results to
some extent. A few interviewees noted that they are not able to assess the results of the
program given that performance information is not distributed. For example, members of
the Technical Review Committee do not have any feedback as to the results of the
projects that they reviewed and approved. They mentioned this would be helpful when
reviewing the next round of applications in order to help them determine whether an
organization was successful in the past in achieving its objectives and thus a more
reliable proponent for future projects.
In terms of the survey results (Table 3), almost 80% of successful project proponents felt
the IASPP was contributing to the reduction of IAS in Canada through its activities to at
least a moderate extent. Unsuccessful proponents who were able to respond to the
question also felt the program was contributing to at least a moderate extent. No
respondents felt that the IASPP was having no impact.
Environment Canada
25
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Table 3 – Survey Results for Overall Success of IASPP
“To what extent has the IASPP contributed to the
Successful
reduction of invasive alien species in Canada
n=25
through its activities?”
1.
2.
3.
4.
To no extent
To a moderate extent
To a tremendous extent
Don’t know
Unsuccessful
0
11
8
5
n=12
0
4
0
7
Immediate Outcome 1: Threat to natural capital reduced
Evaluation Issue: Success
Q4.a: Is the threat to natural
capital reduced as a result of
the program?
Indicator(s)
 As per indicators in the
performance measurement
plan and reporting strategy
Methods
Rating
 File
review
 Interviews
 Survey
~ Little
Progress,
Priority for
Attention
Interviewees were generally of the opinion that reducing the threat to natural capital was
a less important outcome of the program, explaining that the main objective of the
program was to increase Canadians’ awareness with respect to IAS. It was felt that
raising awareness would lead to a reduction of the threat to natural capital. Project
proponents interviewed felt that most of their project activities were focused on
increasing awareness, which would then lead to action on IAS. Project proponents
themselves were unclear as to whether this outcome was being achieved. Furthermore,
in many cases in the interviews across all categories, it was commented that “threat to
natural capital reduced” was very hard to measure as there was no performance
information available on which to make an assessment.
The following project outputs (Table 4), however, demonstrate that action at the local
level has contributed to some degree to reduce the threat of IAS. In total, the projects
supported by the IASPP have undertaken activities related to a range of invasive alien
species (215) and pathways (31).
Table 4: IAS targeted by projects
Plant
Weed
Plant
Pest
Aquatic
Invertebrate
Aquatic
Vertebrate
Amphibian
Others
64
72
11
18
42
1
7
Table 5 presents performance data related to environmental indicators. There is a large
amount of activity that is occurring across a wide range of IAS. Although the data do
show projects are carrying out activities related to this outcome, given that the data have
not been validated, it is difficult to determine the level of success in achieving this
outcome. Furthermore, without baseline data providing a clear indication of the present
threat (related to each project-level outcome used to measure the level of success in
achieving this program outcome), it is difficult to determine the degree to which this is an
issue and the extent to which the program has been successful in addressing this issue.
For example, available performance data may state that 100 square hectares of forest
Environment Canada
26
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
has had an invasive plant removed, but give no indication as to how many hectares of
land are suffering from this invasive plant.
Table 5: Environmental Indicators
Measures
Environmental management
manuals or plans
implemented / signed
General Habitat improvement
Vegetation and exotics
removal
Native plants, trees and
shrubs planted
Wildlife habitat created or
restored
Species at risk protected or
restored
Other native species protected
or restored
Non-native species removed
Wetlands, land ecosystems
created, restored or
rehabilitated
Indicators
Hectares
Number of plans
Number of
participants
Number of
hectares improved
Number of
participants
Number of
hectares improved
Number of plants /
trees / shrubs
Area M2
Hectares
Target
7,060
Achieved
15,050
%
213%
11
15
136%
1,007
1,320
131%
10,757
30,163
280%
859
1,205
140%
186
693
373%
100,175
350,494
350%
1,045
2,799
268%
57
624
1095%
2
16
800%
21
27
129%
27
254
941%
442
452
102%
Number of species
Number of species
Number of species
Hectares
Note: Performance data contained in the table are as submitted by projects and have not been validated by
the program or this evaluation.
Immediate Outcome 2: Improved capacity to detect and respond to IAS
Evaluation Issue: Success
Q4.b: Is there improved
capacity to detect and respond
to IAS as a result of the
program?
Indicator(s)
 As per indicators in the
performance measurement
plan and reporting strategy
Methods
Rating
 File
review
 Interviews
 Survey
~ Progress
made,
attention
needed
This immediate outcome is very similar to the intermediate outcome of “improved
community capacity, skills and knowledge with respect to IAS initiatives”. Both are
equally hard to measure given there is no baseline of existing capacity which could
provide a comparison against which to measure improvements. It is therefore difficult to
assess the achievement of this outcome.
Environment Canada
27
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
EC program staff could not speak to the issue of increased capacity at the local level
(although it was mentioned that smaller communities have been made more aware of
issues related to detecting and responding to IAS). Comments from other government
departments were also very limited due to a lack of knowledge with respect to
performance information related to the projects. One interviewee stated that there was
most likely an improved capacity to detect IAS as a result of the program, although not
necessarily to respond to IAS (as funding was targeted more towards improving capacity
to detect IAS). In another case (related to unwanted plants targeted by the horticulture
industry), the respondent was very positive and stated that the increased information
with respect to IAS had indeed resulted in improved capacity to detect IAS.
Despite a lack of baseline data, the projects have reported on outputs in this regard, as
demonstrated in Table 6 (Training). Performance data indicate that 3,617 individuals
have received training. It is important to note that it is possible that duplication in these
numbers has occurred (i.e., the same individual could have received training several
times). The number of people who have received training more than once is likely low
however given that the data are derived from 71 projects spread across the country.
Although projects have provided training opportunities for community members, and the
projects have been successful in terms of the number of training sessions held and the
number of individuals reached, an increase in training and the information provided
through training does not necessarily indicate an improved capacity to detect and
respond to IAS. These figures only indicate the potential for improved capacity.
Other performance information used to measure the level of success with respect to
improved community capacity (Table 7) included how the projects were able to establish
means and mechanisms to combat IAS and how effectively they were able to engage
people in the process. However, there is a disconnect between the project-level outputs
and outcomes identified for this immediate outcome and the performance indicators
identified to assess their level of achievement. Although the program may have been
successful in reaching and engaging individuals, these performance indicators will not
allow the program to measure the level of achievement with respect to establishing a
coordinated public monitoring network, establishing capacity of diagnostics and
taxonomic expertise, or monitoring results and impacts of introductions. Therefore, it is
difficult to determine whether the program has indeed been successful in improving the
capacity to detect and respond to IAS.
Table 6: Training on IAS
Measures
Training opportunities for
community members
Indicators
Number of training
sessions held
Number of days of
courses
Number of people
Target
Achieved
%
63
140
222%
27
89
328%
1,560
3,617
232%
Note: Performance data contained in the table are as submitted by projects and have not been validated by
the program or this evaluation.
Environment Canada
28
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Table 7: Community Capacity
Measures
Established a coordinated
public monitoring network to
detect and report IAS
Established capacity of
diagnostics and taxonomic
expertise to identify IAS.
Monitored results and impacts
of introductions
Indicators
Number of people
engaged
Number of people
reached
Number of people
engaged
Number of people
reached
Number of people
engaged
Number of people
reached
Target
Achieved
%
11,465
20,875
182%
48,099
105,011
218%
197
5,796
2942%
676
26,593
3934%
225
270
120%
1,420
1,822
128%
Note: Performance data contained in the table are as submitted by projects and have not been validated by
the program or this evaluation.
Immediate Outcome 3: Canadians better informed about IAS
Evaluation Issue: Success
Q4c: Are Canadians better
informed about IAS as a result
of the program?
Indicator(s)
 As per indicators in the
performance measurement
plan and reporting strategy
Methods
Rating
 File
review
 Interviews
 Survey
~ Progress
Made,
Attention
Needed
All interviewees stated that the program was very successful at informing Canadians
about IAS. EC management and staff, as well as other government departments all
agreed that the IASPP had increased awareness related to IAS. Some EC staff
mentioned that the support to IAS Plant Councils has been a major achievement and
that, generally, one of their principal roles is to increase awareness of IAS in their
respective regions. Interviewees across all categories mentioned the quantity of
information products that have been produced and disseminated. Proponents were also
confident that the program had achieved this outcome.
Performance data indicate that of the 71 projects that have been completed, 60 of them
(or 84.5%) produced and disseminated information products related to IAS. Almost all
(92%) of successful project proponents who responded to the survey indicated that the
main objective of their project was to “Improve Canadians’ understanding and
awareness of IAS and the individual actions and choices that contribute to their spread.”
Table 8 presents the performance information on information products produced,
although it is not clear from these figures the actual number of products generated
versus distributed.
Environment Canada
29
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Table 8: Information Products
Measures
Indicators
Public information
Number of
generated/disseminated
products
generated/
disseminated
Target
Achieved
%
26,410
60,072
227%
Note: Performance data contained in the table are as submitted by projects and have not been validated by
the program or this evaluation.
While production of information products is one area of considerable strength of the
IASPP, outreach figures (Table 9) should be treated with some caution. There was very
limited orientation given to proponents on the definition of the performance indicators
(e.g., defining people reached versus people engaged). Thus, the reliability of the data
being provided by individual proponents at the end of their respective projects is brought
into question. Furthermore, the project-level outputs and outcomes, and their related
performance indicators, point towards an increase in outreach and information
generated and disseminated, not necessarily to Canadians being better informed.
Although the program has been successful in increasing available information related to
IAS through its production and outreach activities, it does not necessarily mean that
Canadians are better informed, only that more information is publicly available.
Table 9: Outreach Indicators
Measures
Worked with media
General public outreach
Perform targeted outreach
Indicators
Number of media
interviews
Number of reports
on projects
Number of people
reached
Number of people
engaged
Number of people
reached
Number of people
engaged
Target
Achieved
%
47
119
253%
27
57
211%
307,749
290,811
94%
10,410
11,524
111%
138,817
194,696
140%
21,387
34,152
160%
Note: Performance data contained in the table are as submitted by projects and have not been validated by
the program or this evaluation.
Immediate Outcome 4: Partners achieve measurable results
Evaluation Issue: Success
Q4d: Partners12 achieve
measurable results as a result
of the program?
12
Indicator(s)
 As per indicators in the
performance measurement
plan and reporting strategy
Partners in this instance refer to the actual project proponents.
Environment Canada
30
Methods
Rating
 File
review
 Interviews
~ Achieved
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Out of the 108 projects that have been approved since the program began in FY 20052006, 71 have been completed. The “Key Indicators of Success Reports” from these
completed projects provide evidence that project proponents are achieving measurable
results13, despite concerns surrounding the accuracy of performance data.
From interviews with EC senior management and staff, the overwhelming response was
that the key result of the program has been an increased awareness of IAS in Canada.
Comments such as “awareness is the really big contribution” were common among
these interviewees; many mentioned the quantity of information products produced by
the projects. Stakeholders interviewed commented that they have seen many good
results and provided the establishment of IAS Plant Councils as an example and how
that has helped bring all the IAS-related groups together.
A few of the interviewees from all three categories commented that the size of the fund
may limit the results that can be achieved. The projects are relatively small in size and
disbursed widely across the country. Issues around the timely transfer of funds were also
raised as having the potential to limit results (i.e., some projects have a small window of
opportunity related to their geographic location or the time of year they need to be
implemented).
Intermediate Outcome 1: Improved jurisdictional coordination and communication
of community response to IAS
Evaluation Issue: Success
Q4e: Is there improved
jurisdictional coordination14 and
communication as a result of
the program?
Indicator(s)
 As per indicators in the
performance measurement
plan and reporting strategy
Methods
Rating
 File
review
 Interviews
~ Achieved
The IASPP has contributed to improved jurisdictional coordination and communication.
The program has provided funding to provincial government organizations and three
provincial IAS Plant Councils—namely British Columbia, Alberta and Manitoba. Another
funded provincial organization is the Native Plant Society of Saskatchewan. Smaller
regional councils have also been supported, for example, the Central Kootenay Invasive
Plant Committee in British Columbia. Two projects are also presently being supported
whose objectives are to establish IAS Councils—one in Nova Scotia and one in the
Yukon. The provincial and regional IAS Councils themselves play a large role in
jurisdictional coordination and communication by networking IAS-related stakeholders at
the provincial, regional and local levels, and sharing information and research.
13
Performance data are presented in Tables 5, 6, 7, 8, 9, 10, 11 and 12.
14
For the purposes of this evaluation, jurisdictional coordination was defined as inter-departmental
coordination at the federal level, federal and provincial/territorial coordination,
municipal/community coordination, or any mix thereof.
Environment Canada
31
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Interviews with EC program staff, other government departments and proponents all
mentioned the important contribution of the program in establishing these councils. In
some cases, the IASPP has helped draw provincial attention to the issues of IAS; it has
also allowed provinces to become more engaged in this area. No study of other IAS
funding sources at the provincial level was undertaken, but comments from one
provincial partner mentioned that the IASPP had allowed them, through their funded
projects, to start activities related to IAS. One survey respondent mentioned the following
as a key strength of the program: “the program is one of the only funding avenues for
groups doing invasive alien species work. The IASPP has enabled many Canadian
provinces to establish invasive species councils, which will empower them to better help
the federal government meet the priorities of the IASSC.”
Other government department interviewees also raised the point that the IASPP, through
the review of proposals process, has provided their staff at headquarters offices the
opportunity to improve their understanding and knowledge of priority IAS issues at the
local and regional levels.
While it is beyond the scope of the IASPP program, it is worth mentioning that some
discussions have occurred about establishing a national IAS council. If and when such
an organization is formed, it would be an appropriate mechanism to share the products
and outputs that have come from the IASPP so that information can be disseminated
nationally instead of just provincially or regionally.
Intermediate Outcome 2: Development and application of innovative techniques,
practices, and/or processes involving introductions through unintentional
pathways
Evaluation Issue: Success
Q4f: Is there development and
application of innovative
techniques, practices and/or
processes involving
introductions through
unintentional pathways as a
result of the program?
Indicator(s)
 As per indicators in the
performance measurement
plan and reporting strategy
Methods
Rating
 File
review
 Interviews
~ Progress
Made,
Attention
Needed
Some Environment Canada staff commented that science was not a major focus of the
program during this initial five-year funding period, and that, in the future, it might be
worthy to examine the relationship between the IASPP and the academic/scientific
sector to increase activities in the area of developing new and innovative methods to
address IAS. Program staff commented that they are aware of tools that have been
developed based on their own knowledge of the projects. One specific example provided
was an IAS database developed in Ontario.
Table 10 presents the performance indicators related to the development of new
techniques and approaches. Information was collected with respect to the development
of new tools to better predict IAS, new methods/technologies for preventing the spread
of IAS and on new diagnostic tools to identify IAS. It is possible that what has been
reported here is “new” in the sense that it has been developed and applied in a region for
the first time, but not necessarily new in the sense that the specific method or tool may
Environment Canada
32
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
have already been used in some other region. An example would be the use of GPS
systems to map out areas of IAS infestation. It is difficult to determine the actual quality
of the outputs reported below. There also seems to be some inconsistency in the
performance data. For instance, under the measure “Conducted and supported research
aimed at developing tools to better predict IAS,” the number of reports distributed (1,217)
is greater than the number of people reached (1,000), which may again be a result of
duplication (i.e., double-counting), although it was not possible for the evaluation to verify
this. Another example of an inconsistency from the performance data on innovation
relates to the measure “Develop new diagnostic tools to identify IAS.” There were 36
new tools developed (forty-three percent of the target), while the number of new tools
put into action is 483.
Table 10: Indicators on innovation
Measures
Indicators
Conducted and supported
Number of
research aimed at developing stakeholders
tools to better predict IAS
involved
Number of tools
developed
Number of reports
distributed
Number of people
reached
Conducted and supported
Number of
research on new
stakeholders
methods/technologies for
involved
preventing spread of IAS
Number of people
reached
Number of new
methods or
technologies
developed or used
Develop new diagnostic tools
Number of tools
to identify IAS
developed
Number of people
engaged
Number of people
reached
Number of new
tools put into
action
Target
Achieved
%
49
58
118%
21
28
133%
405
1,217
300%
0
1,000
100%
221
237
107%
8,245
8,308
101%
12
13
108%
83
36
43%
298
364
124%
1,166
472
40%
231
483
209%
Note: Performance data contained in the table are as submitted by projects and have not been validated by
the program or this evaluation.
Environment Canada
33
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Intermediate Outcome 3: Canadians are engaged in activities on priority IAS
issues
Evaluation Issue: Success
Q4g: Are Canadians engaged
in activities on priority IAS
issues as a result of the
program?
Indicator(s)
 As per indicators in the
performance measurement
plan and reporting strategy
Methods
Rating
 File
review
 Interviews
 Survey
~ Progress
Made,
Attention
Needed
Interviews with program staff did not provide any concrete examples, but all staff felt that
there was increased engagement of Canadians as a result of the communication and
outreach activities of the projects. One proponent mentioned that since the IASPP has
been operational, there has been a “groundswell of interest” on IAS issues. Another
proponent commented that since his involvement from the inception of the program,
there has been a progression from “almost no awareness to many groups now getting
involved.”
Table 11 presents performance data related to engaging Canadians. While the figures
below have not been validated, survey information indicated that almost all of the
organizations that received IASPP funding rely on volunteers – only 12% of respondents
indicated that they seldom use volunteers, and almost 30% of respondents stated they
use over 50 volunteers on their projects. Performance data indicate that 3,831
volunteers participated in the projects funded by the program, an average of 35 per
project. However, these figures are not identical to the figures related to volunteer
participation provided in Table 12 related to community capacity. Although the number of
volunteers is similar, the number of person-years is significantly different.
IASPP projects have also involved federal departments, as well as provincial and
territorial departments. However, there seems to be duplication (i.e., double-counting)
with respect to available performance data presented in Table 11. For instance, a total of
88 federal departments (62 being the target) and 116 provincial and territorial
departments were reported to be involved in the projects. Similarly, nothing precludes
more than one project proponent reporting that their project affected the same
individuals, indicating that some of the people affected by a project may have been
counted twice, which may explain why that particular project-level outcome achieved
684% of its target.
It is clear that one of the key strengths of the program has been raising Canadians’
awareness of IAS issues. Although the program has been successful in raising the
awareness of Canadians through training and outreach, as well as through the
generation and dissemination of products related to IAS, and it has made progress in
achieving the immediate outcome, “Canadians better informed about IAS,” these do not
necessarily indicate that Canadians have become engaged in activities on priority IAS
issues as a result of the program. These only speak to the potential for Canadians to
become more engaged because they are more aware and better informed. Furthermore,
without baseline data, it is difficult to determine the program’s impact in getting
Canadians engaged in IAS-related activities.
Environment Canada
34
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Table 11: Indicators on engaging Canadians
Measures
Volunteers participating
directly in project
Federal departments involved
in project
Provincial or Territorial
departments involved in
project
Municipal governments
involved in project
People affected by project
Indicators
Number of
volunteers
Number of personyears
Number of
departments
Number of
departments
Number of
governments
Number of people
Target
Achieved
%
2,005
3,831
191%
1,010
518
51%
62
88
142%
93
116
125%
88
191
217%
161,268
1,102,793
684%
Note: Performance data contained in the table are as submitted by projects and have not been validated by
the program or this evaluation.
Intermediate Outcome 4: Improved community capacity, skills and knowledge with
respect to IAS initiatives
Evaluation Issue: Success
Q4h: Is there improved
community capacity, skills and
knowledge with respect to IAS
as a result of the program?
Indicator(s)
 As per indicators in the
performance measurement
plan and reporting strategy
Methods
Rating
 File
review
 Interviews
 Survey
~ Progress
made,
attention
needed
Interviews with program staff indicated that they believe community capacity has
increased mostly as a result of the communication and outreach activities that have
occurred.
The survey collected information on the organizational profile of the proponents.
Approximately 42% have annual budgets of less than $1 million, whereas 36% have an
annual budget over $1 million (22% were not able to respond). Approximately 56% have
fewer than 20 staff members, whereas 38% have over 20 staff (about 6% were not able
to respond). Given that many IASPP recipients are thus of a relatively smaller size, these
organizations are more dependent on IASPP funding to carry out activities; therefore,
IASPP funding can have an important impact on their capacity to undertake projects
related to IAS.
Performance data presented in Table 12 below indicate that project proponents are
engaged in their communities through public events, responding to inquiries through the
provision of services, getting volunteers engaged and training them. However, these
figures only indicate the potential for improved capacity. As was the case with the
intermediate outcome related to Canadians becoming engaged in IAS-related activities,
an increase in the amount of training, outreach, as well as the generation and
dissemination of products related to IAS indicates a potential for improved capacity, skills
Environment Canada
35
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
and knowledge with respect to IAS. These performance indicators do not speak
necessarily to improved capacity being achieved. Although the program has been
successful in producing these outputs, the impact they have had on community capacity,
skills and knowledge is uncertain.
Table 12: Indicators on community capacity
Measures
Indicators
Public attendance at project
Number of events
events
Number of people
Inquiries
Services provided in response
to requests
Volunteer participation
Recognition of community
participation
Training opportunities for
community members
in attendance
Number of
inquiries about
projects
Number of
services provided
Number of people
served
Number of
volunteers
Number of
volunteer personyears
Number of awards
Target
Achieved
%
330
468
142%
128,450
146,392
114%
4,985
10,479
210%
467
744
159%
2,353
4,614
196%
2,207
3,661
166%
29
108
379%
6
6
100%
63
140
222%
27
89
328%
1,560
3,617
232%
Number of training
sessions held
Number of days of
courses
Number of people
Note: Performance data contained in the table are as submitted by projects and have not been validated by
the program or this evaluation.
Evaluation Issue: Success
Q5: Have there been any
unexpected outcomes (positive
or negative)? Were any actions
taken as a result of these?
Indicator(s)
 Documented unexpected
outcomes
 Lessons learned or best
practices have been
documented
No unintended outcomes were identified through this evaluation.
Environment Canada
36
Methods
Rating
 Document
review
 File
review
 Interviews
N/A
Audit and Evaluation Branch
4.3
Evaluation of the Invasive Alien
Species Partnership Program
Cost-Effectiveness
Evaluation Issue: Cost-Effectiveness
Overall Findings: Evidence collected indicates that the program is being delivered in a
cost-efficient manner through its low administrative costs (13.4%), as well as its
significant leveraging of matching funds ($1.19:1 ratio).
Overall, interviewees indicated that the program’s approach was appropriate. No
alternative approaches to a contribution program were identified that could be more
cost-effective or efficient at achieving the same objectives as the IASPP. However, a
few modifications to the approach presently implemented by the program were
suggested, including the creation of two separate streams of IASPP funding, one for
provincial governments and the other for non-governmental organizations and industry.
A few interviewees also discussed the need to focus more on an action-oriented
approach to addressing IAS in Canada.
It was felt by most interviewees that limited operational funding had a negative impact
on the overall effectiveness of the program to achieve its objectives. Specifically, the
lack of operational funding affected both the management of IASPP contribution
agreements (i.e., timely transfer of funds, verifying individual project results), as well as
Environment Canada’s coordinating role with respect to the national strategy.
Evaluation Issue: CostEffectiveness
Q6: Are the most appropriate
and efficient means being used
to achieve objectives, relative to
alternative design and delivery
approaches?
Indicator(s)
Methods
Rating
 Financial management system
lined to results management
process.
 Consideration of alternate
approaches
 Document
review
 Interviews
Achieved
Program Efficiency
Cost-effectiveness is ideally measured by calculating the costs of achieving the
measured program results and comparing them to the costs of achieving the same
results through different means. Given the challenges encountered in identifying
accurate performance data, effectiveness was measured indirectly by examining
program efficiency and key stakeholders’ perceptions of the program’s efficiency and not
a subjective assessment of cost-effectiveness.
Table 13 below presents a financial summary of the IASPP. It should be noted that
financial information related to salaries is a best estimate provided by the program (given
that no specific funding was earmarked for the administration of the program). In
addition, the actual value for contributions reported here is different from Table 2. Table
2 presented approved values, while Table 13 presents disbursed values.
Environment Canada
37
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Table 13: Cost of the IASPP
Cost Items
2006-07
2007-08
2008-09
Total
Salaries and Benefits (estimated)
$97,804
$120,107
$97,160
$315,071
Operating Costs
$8,996
$122,070
$30,705
$161,771
Sub-Total (A):
$106,800
$242,177
$127,865
$476,842
$624,894
$1,958,592
$984,185
$3,567,671
A. Actual Operating Costs:
B. Non-Administrative:
Contributions15
Sub-Total (B):
$3,567,671
TOTAL COSTS (A+B):
$4,044,513
Ratio of total actual operating costs to
contributions
17.1%
12.4%
12.9%
13.4%
OGD In-Kind Estimates (person/days)
88
35
54
177
The IASPP was planned to be a $1 million per year contribution program. No additional
funding for program administration was approved and all operating expenses for the
IASPP have been covered by Environment Canada’s A-base budget. Given the level of
resources currently applied to administer the IASPP, the program has proven to be costefficient at requesting, reviewing, and identifying projects that contribute to the overall
goals of the program. It must also be noted that significant in-kind contributions are
made by the TRC members (estimated at 177 person days thus far) related to the review
and assessment of project proposals which occur on an annual basis.
It is difficult to match approved values with disbursed values and with the $1 million in
funding for contributions that the IASPP was allocated each year over the five-year
period. The delays in the first request-for-proposals in FY 2005-2006 resulted in a
combined call that permitted multi-year funding. This, therefore, also had an impact on
subsequent financial years.
Over the three years for which financial information is available, the percentage of total
actual operating costs compared to total actual contributions is 13.4%. This ratio is
similar to that of the Habitat Stewardship Program, whose administrative costs represent
13% of its contributions for the period 2004-2005 to 2007-2008 (budgeted amounts).
The workload of such a program is created by the volume of applications received and
the subsequent administration of the approved projects. The request-for-proposal
process is a time-consuming process, even with the implementation of an online
application system. Program staff commented that they receive many queries from
proponents during this period and do their utmost to respond to them all. There is also
the screening and selection of an average of 150 proposals which are submitted per
15
Contribution expenses in fiscal years 2007-08 and 2008-09 include subsequent payments of
multi-year contribution agreements approved in fiscal years 2005-06 and 2006-07.
Environment Canada
38
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
year. Other government departments contribute their staff time and resources to this
process, as noted in the above table.
There is also significant leveraging of funds. Projects were selected based in part on
their ability to leverage partnerships and funds. Information collected from the program
indicates that for every dollar of IASPP funding, $1.19 is leveraged by the proponent.
With respect to the overall efficiency of the program, the overriding concern is with
delays in transferring funds to project proponents. This was raised by almost all
interviewees across all categories, as well as by survey respondents. Given the timesensitive nature of IAS field activities, which usually have to be undertaken in the April to
October time period, delays in funding can have a major impact on budgets and
therefore project workplans. The need to revise workplans, budgets and contribution
agreements due to delays in final approvals is perceived to be a major weakness in the
program.
Effects of Lack of Operational Funding
Interviewees discussed various issues to emerge as a result of the program not having
specific operational funding. Overall, many felt that a lack of operational funding had a
negative impact on the overall effectiveness of the program to achieve its objectives.
Specifically, the lack of operational funding affected both the management of IASPP
contribution agreements, as well as Environment Canada’s ability to fulfil its coordinating
role with respect to the national strategy. As one individual stated “without question,
operational funding with dedicated personnel and in-house resources would certainly
increase the effectiveness of the program.”
A few interviewees felt that this lack of operational funding was especially problematic for
the program at the beginning. For instance, there were major delays in the approvals
process for the first round. Those who were approved did not see funding until the
second round was well underway. In some cases, given timing issues, projects could not
undertake the activities that they proposed. Other issues noted by interviewees included:

getting contribution agreements out in a timely fashion;

an inability on the part of the program to follow-up with recipients to determine
whether or not projects had really achieved the expected results they described in
their “Key Indicators of Success Report”; and

delays in providing feedback to non-recipients due to capacity issues (as resources
were used to prepare contribution agreements for recipients).
Furthermore, the lack of operational funding also detracted from the ability of the
program to coordinate the national strategy. Environment Canada was tasked with the
responsibility to provide overall leadership and coordination of the national strategy.
Given a lack of resources and capacity, the program has spent much of its time
administering the IASPP contribution agreements and, as a result, has not been able to
direct resources to its coordination role. Participants16 who attended the National
Invasive Alien Species Forum, held in June 2009, felt that there was a gap in national
governance and that a national coordinating/governance mechanism needs to be
implemented (this included the need to establish priorities, the need to clarify roles and
16
Participants included provincial/territorial and federal government representatives, as well as
representatives of jurisdictional IAS councils.
Environment Canada
39
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
responsibilities related to the national strategy and the need to create a mechanism to
ensure partnerships among the key stakeholders). A few Round II interviewees also
noted the lack of an interdepartmental coordination and governance structure for the
national strategy as a result of the lack of operational funding.
Alternative Approaches
In terms of alternative approaches, most interviewees felt the program’s approach was
appropriate. For the most part, no alternatives were identified that could be more costeffective or efficient at achieving the same objectives as the IASPP. In some cases,
interviewees suggested that modifications could be made in a subsequent iteration of
the program. For example, one stakeholder suggested that it is not appropriate for
provincial governments to be competing alongside small organizations (e.g., a First
Nations community organization), and that provincial needs should be addressed in a
different manner. For example, two streams of competitive funding could be created, one
for provincial governments and the other for non-governmental organizations and
industry.
A couple of interviewees discussed the need for a more action-oriented approach to
addressing IAS in Canada. As one interviewee noted, “at some point you’ll saturate the
market with pamphlets and brochures. It’s still better than nothing, though we do not
have the impact we could have.” Another interviewee felt that there was a need for more
action-oriented projects to deal with introductions that have already occurred in order to
minimize the impact of those introductions. The IASPP was designed to focus on IASrelated projects which improved the understanding and awareness of Canadians of IAS,
as well as enabling Canadians to become actively and concretely involved in projects
that address the threat of IAS.
A few questioned the mix of funding between prevention, detection, response and
management. Most EC management and staff and OGDs were comfortable with the
emphasis on prevention and detection as this falls within the federal jurisdiction. A few of
the stakeholders voiced their interest in seeing more work on management, although
they were in the minority.
One final area for improvement mentioned was the distribution of the performance
information to other government departments involved in the program, for example, the
TRC members. This would allow them to know the performance of project proponents on
past projects and assist in the project review and selection process.
Environment Canada
40
Audit and Evaluation Branch
4.4
Evaluation of the Invasive Alien
Species Partnership Program
Design and Delivery
Evaluation Issue: Design and Delivery
Overall Findings: The program is being delivered as designed and continues to make
operational process improvements (e.g., creation of a website, addition of an online
application system). The program’s activities, outputs and expected outcomes are
clearly identified although they could be better communicated to all stakeholders. A
governance structure exists that is consistent with the overall objectives of the program
and the IASSC.
There appear to be areas for improvement regarding the design and delivery of the
program. Specifically:
1) Accuracy of available performance data
There is uncertainty about the accuracy of available performance data for the IASPP,
which is used to assess the level of success achieved for particular outcomes. Although
performance data are being collected and reported by project proponents at the
completion of their projects, the information has not been validated by the program.
There are issues of duplication (i.e., double-counting) and inconsistency of available
performance data, as well as a disconnect between some performance indicators and
the project-level outputs and outcomes that they are intended to measure. Thus it is
difficult to determine the level of success of the program in achieving its expected
outcomes and overall objectives.
2) Full implementation of performance management plan
The performance management plan has not yet been fully implemented. Although the
program does collect performance data, this information has not been analyzed to
determine the extent of the program’s achievement of its expected outcomes and overall
objectives. Therefore, performance data are not currently being used to inform program
decision-making.
3) Timeliness of funding disbursement
Providing successful proponents with funding for approved projects in a timely manner
was identified as an issue by all three categories of key informants as well as by a
majority of survey respondents.
4) Communications to proponents with respect to the overall objectives of the IASPP
Interviews with project proponents revealed that they were less clear on the overall
objectives of the IASPP. The application guidelines do provide some guidance, albeit at
a very high level. They do not specifically state the objectives or the expected results of
the IASPP.
Environment Canada
41
Audit and Evaluation Branch
Evaluation Issue: Design
and Delivery
Evaluation of the Invasive Alien
Species Partnership Program
Indicator(s)
Methods
Rating
Q7: Are program deliverables
and expected outcomes
identified clearly, and is the
program being delivered as
designed?
 Demonstration of the
program’s expected
deliverables and results
 Consistency with documented
design and
delivery/implementation
 Document
review
 Interviews
 Survey
Progress
made,
attention
needed
Q3: Are planned activities being
implemented and producing
expected outputs?
 As per indicators in the
performance measurement
plan and reporting strategy
The program deliverables and expected outcomes are clearly identified in the RMAF and
IASPP logic model. Interviewees from other government departments indicated that they
have a clear understanding of the expected outcomes of the program. Program staff and
other government departments commented that the program was generally being
delivered as designed.
For instance, with respect to the contribution process, there have been four requests for
proposals that have taken place since FY 2005-2006, and 625 projects have been
reviewed, of which 108 have been approved and for which contribution agreements have
been prepared and signed (up to FY 2008-2009). Although proponents do provide a
“Key Indicators of Success Report” at the end of their project, performance data have
only recently been consolidated. No analysis of the data has been conducted to date,
and the performance data have not been used to inform program decision-making (this
issue is discussed in further detail in Question 9).
In FY 2007-2008, the program implemented an online application system. EC program
staff commented that this system was a vast improvement to the previous process where
applications were submitted in hard copy to the program. One possible area for
improvement noted by the evaluators during the document review was the lack of clear
proposal evaluation criteria included in the application documents. Understanding the
evaluation criteria and ranking system related to the assessment of project proposals
would assist proponents in the development of proposals and improve the transparency
of the selection process.
A new website has also been developed that provides information and support to
Canadians on the IASSC and details on the IASPP, including a summary of previously
funded projects, success stories and the program’s objectives and goals. Further
improvements are planned and will be implemented pending the availability of time and
resources. The improvements include introducing an electronic reporting system within
the current online application system. This would allow project proponents to report
online and assist in the capture and analysis of performance information. Discussions
pertaining to automating processes for the drafting of contribution agreements, which
should reduce the turn-around time needed to produce and/or revise agreements, have
also taken place.
Interviews with proponents revealed that they were less clear on the overall objectives of
the IASPP. The application guidelines do provide some guidance, albeit at a very high
level, and they place an emphasis on the evaluation criteria and list examples of the
Environment Canada
42
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
types of activities that are funded. They do not specifically state the objectives or the
expected results of the IASPP. However, all program staff and other government
department representatives interviewed felt the objectives of the program were clear.
This perception of clarity could be due to a greater familiarity with government IAS
programs themselves, either through their own departmental programs or close
interaction with the IASPP.
Survey questions focused specifically on the IASPP’s application, selection and funding
processes. The results (Table 14) reveal that, in general, the application and selection
processes are operating well, while there were issues surrounding the funding process.
Most successful proponents (91%) felt that, overall, the application process was clear,
while 64% of unsuccessful proponents felt it to be clear. The survey revealed that both
successful and unsuccessful proponents were clear on eligibility requirements17 (91%
and 88%, respectively). Results were significantly lower when respondents were asked
whether they received information, guidance and support provided from program staff
during the application process. While 65% of successful proponents responded that they
had received information, guidance and support, only 9% of unsuccessful proponents
felt the same. This is at odds with interviews of program staff and other government
department representatives, who stated that great effort is made to respond to all
queries received during the request-for-proposal process.
Regarding the selection process, the majority of successful and unsuccessful
proponents were clear on the selection criteria18 (78% and 64%, respectively). Feedback
as to why projects were not funded seemed to be an issue. Only 27% of unsuccessful
proponents felt they received an explanation from the program on why their project was
not funded.
The timely transfer of funds was identified in almost all of the interviews from all three
categories as an area for improvement (i.e., getting “money out the door”). This was also
highlighted in the survey. Although 74% of successful proponents responded that the
funding process was clearly articulated in the contribution agreement, only 35% of
successful proponents stated that they received the funds in a timely fashion. This is a
critical issue given the time-sensitive nature of IAS activities that generally occur during
the field season, roughly April to October each year. The delays in getting funding to
recipients has had an effect on the workload of proponents and program staff because
of the need to revise workplans and budgets due to funding arriving late in the season in
which field work for IAS can be conducted.
17
Eligibility requirements are both project-specific (e.g., must be less than $50,000, cannot be on
federal land) and proponent-specific (e.g., organizations can be non-governmental organizations,
provincial, territorial or municipal crown agencies).
18 Examples of selection criteria are technical feasibility, proponent capacity, public education
value, and risk reduction value.
Environment Canada
43
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Table 14: Survey results related to application process
1. Based on your experience preparing an
application for IASPP funding, was the overall
application process clear?
2. Did you receive information, guidance or support
from IASPP representatives during the
application process?
3. Was the information and guidance provided on
the IASPP website sufficient to help your prepare
your IASPP application?
4. Based on your review of program eligibility criteria
in available program documents, did you find
these criteria to be easy to understand?
5. Based on your review of program selection
criteria in available program documents, did you
find these criteria to be easy to understand?
6. Was the decision regarding whether your funding
proposal was successful or not communicated to
you in a timely manner?
7. Were the timelines related to your project’s key
deliverables clearly articulated in the contribution
agreement?
8. Was the funding process clearly articulated in the
contribution agreement?
9. Were the funds provided to you in a timely
manner?
Successful
Unsuccessful
n=25
n=12
Y
N
DK
Y
N
DK
21
91%
2
0
7
64%
3
1
15
65%
7
1
1
9
82%
1
21
91%
2
0
6
55%
3
2
21
91%
2
0
8
88%
1
2
18
78%
3
2
7
64%
2
2
18
78%
3
2
6
55%
4
1
19
83%
3
1
17
74%
8
3
3
15
65%
0
7
64%
8
73%
1
10. Did you receive an explanation from EC on why
this project was not funded through the IASPP?
N/A
11. Were the reasons your IASPP funding application
was not successful made clear to you?
N/A
N/A
N/A
N/A
3
2
Note: Tallies may not add up to total respondents as some respondents skipped certain questions.
Evaluation Issue: Design
and Delivery
Q8: Are activities and outputs
linked to shared outcomes and
the overall government-wide
objectives?
Indicator(s)
 Links between activities,
outputs and outcomes
supporting joint national
objectives
Methods
Rating
 Document
review
 Interviews
Progress
made,
attention
needed
Although the logic model describes program activities and outcomes, the relationships
between the activities of the program and the outcomes are unclear. Furthermore, key
outputs are not included in the logic model although they are described in the RMAF
itself. Therefore, the format of the logic model does not allow for the individual activities
and outputs to be linked to the program’s immediate outcomes. It also makes it difficult
to link program activities not only to the program’s expected outcomes but also to the
Environment Canada
44
1
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
overall government-wide objectives. As a result, the program’s underpinning logic is not
expressed in the logic model as clearly as it should be.
Program staff and other government departments noted areas for improvement related
to the expected outcomes as they are presented in the program’s logic model,
specifically the placement of certain outcomes at either the immediate or intermediate
level. For example, it could be argued that the main focus of the program has been to
produce information as well as to engage and raise the awareness of Canadians related
to IAS.
The IASPP was designed and is delivered to support the implementation of the IASSC
with an emphasis on raising the awareness of Canadians and engaging them in IAS
activities, both of which are identified outcomes of the program. In addition, project
eligibility and selection is consistent with the IAS priorities of prevention, early detection,
rapid response and management. A review of the projects indicates that all projects
address at least one, and usually multiple, IAS priorities. Table 15 presents the
alignment of projects to IAS priorities. The strongest emphasis has been on prevention
and detection (86% and 91.5% of approved projects address those IAS priorities,
respectively), while 20% and 50% of projects address rapid response and management,
respectively. As stated in the IASSC, “as predominant landowners and managers of
agriculture, forests, freshwater fisheries, and wildlife, the provinces and territories
could be seen to have broad responsibilities for the management - eradication,
containment, and control - of established and spreading invaders”. The emphasis on
prevention and early detection is, therefore, consistent with the federal role.
Table 15: IAS Priorities
Priority
Prevention
Early
Detection
Response
Management
Number of
projects
61
65
14
35
Percentage
86%
91.5%
20%
50%
Total number of projects = 71
Evaluation Issue: Design
and Delivery
Q9: Are performance data
collected against program
activities/outcomes and used for
informed decision making?
Indicator(s)
 Measuring and reporting
strategies and procedures
 Presence of populated
performance data system
 Decisions based on
performance information
Methods
Rating
 Document
review
 File
review
 Interviews
 Survey
Little
progress,
priority for
attention
Performance data are collected against program activities/outcomes as evidenced by
quarterly expenditure reports, interim project progress reports, as well as the end-ofproject reports which include the “Key Indicators of Success Report.” Based on an
examination of consolidated performance data from these reports, the program presently
collects information on approximately 56 project-level outputs and outcomes with 114
indicators.
Environment Canada
45
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
There were issues raised in the “Success” section of the present evaluation with respect
to the accuracy of available performance data used to assess the level of success
achieved for particular outcomes. Specifically, there appear to be instances of
duplication (i.e., double-counting) and inconsistency in performance data (e.g., under the
intermediate outcome “Development and application of innovative techniques, practices,
and/or processes involving introductions through unintentional pathways,“ the number of
reports distributed is greater than the number of people reached). There is also a
disconnect between the project-level outputs and outcomes and the performance
indicators identified to measure their level of achievement (e.g., under the immediate
outcome “improved capacity to detect and respond to IAS,” measuring the number of
individuals reached and engaged does not allow the program to measure the level of
achievement with respect to establishing a coordinated public monitoring network,
establishing capacity of diagnostics and taxonomic expertise, or monitoring results and
impacts of introductions). Finally, without baseline data providing a clear indication of the
situation at the time of the program’s inception, it is difficult to determine the extent to
which the program has been successful in achieving its outcomes. Thus, it is quite
difficult to determine the extent to which the program has been successful in achieving
its immediate and intermediate outcomes as identified in the IASPP logic model.
There have also been challenges pertaining to reporting based upon the performance
management plan outlined in the IASPP RMAF. The plan called for a report, which was
to include a summary of interim reports for ongoing projects, as well as a summary by
strategic priority and by habitat. This information is provided annually after the results of
each request for proposals are determined in terms of which projects are to be funded.
Furthermore, each individual proponent was to provide a project-end “Key Indicators of
Success Report.” In total, 71 projects out of the total of 108 approved projects are
complete and have submitted this report thus far. These reports were then to be
consolidated to provide an overall analysis of the level of success of the program in
terms of achieving its intended results. While that information has recently been
consolidated, the program has not undertaken any analysis of the information.
Although data are being collected, the reporting requirements, as per the RMAF, are not
being met. The number of performance indicators (114) as well as their diverse nature
(i.e., environmental, economic, community capacity, outreach, innovation, engaging
Canadians, training) has made it difficult for the program to consolidate performance
data and to use this data to report on the overall success of the program in achieving its
outcomes. Analysis of performance information has been a challenge in the face of
managing the day-to-day delivery of the program. Therefore, performance data are not
currently being used to inform program decision-making.
The results from the survey of proponents (Table 16) reveal that, for the most part,
performance measurement and reporting requirements were articulated in the
contribution agreements. Contribution agreements are generally very similar in their
content. What may vary are the specific activities, budgets and expected results. The
fact that not all proponents responded that expected results (74%) and reporting
requirements (83%) were clearly articulated in the contribution agreement raises some
questions as to whether the contribution agreements are consistently including all these
items, or whether the respondent was not familiar with the terminology or contents of the
contribution agreements themselves.
Environment Canada
46
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Table 16 Measuring Results
Performance Reporting
1. Was an approach to measure the expected results of your
project clearly articulated in the contribution agreement?
2. Were reporting requirements established for your project?
3. Were reporting requirements clearly articulated in the
contribution agreement?
Successful
Y
N
DK
17
3
3
74%
21
1
1
91%
19
2
2
83%
Interviews with program staff revealed that they are aware of the deficiency in collecting
and analysing performance information and hope that further automation of the system,
including the development of an online reporting system to match the online application
system, will assist in improving data collection and analysis.
Evaluation Issue: Design
and Delivery
Indicator(s)
 Established governance
Q10: Are there appropriate
structure
governance structures in place to
 Roles, responsibilities and
meet the objectives?
accountabilities are well
defined, well communicated,
and understood.
 Roles, responsibilities and
accountabilities are functioning
as intended.
Methods
Rating
 Document
review
 File
review
 Interviews
Achieved
Governance structures are in place to meet the objectives of the IASPP. A Steering
Committee, comprised of the Deputy Ministers of Environment Canada and Natural
Resources Canada as well as the President of the Canadian Food Inspection Agency,
has been established, and the roles, responsibilities and accountabilities for the Steering
Committee are defined in the Interdepartmental Memorandum of Understanding
between Environment Canada, Canadian Food Inspection Agency and Fisheries and
Oceans Canada. Furthermore, an interdepartmental Director General Committee
coordinates the overall implementation of the IASSC including the IASPP. A Technical
Review Committee, comprised of experts from the three signatory departments, is
responsible for reviewing and assessing project proposals. The Terms of Reference of
the committee were established in the Interdepartmental Memorandum of Understanding
between Environment Canada, Canadian Food Inspection Agency and Fisheries and
Oceans Canada
Interviews with senior management indicated that the governance structure appeared
adequate. Interviews with program staff also revealed that there is clarity regarding the
roles, responsibilities and accountabilities of the program. There was some
acknowledgement that, in terms of accountability, the review of project outputs could be
strengthened in order to improve performance management and to utilize that
Environment Canada
47
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
information for decision-making. Interview responses from the other government
departments were also similar; most felt that their roles and responsibilities were clear.
Furthermore, during interviews with proponents, most mentioned that their roles and
responsibilities were also clear. Some mentioned that there could be a potential benefit
of better coordination/participation with the provinces, for example, in identifying regional
IAS priorities for funding. Exploring means by which the provinces could be integrated
into the TRC were also mentioned, but only if they are not potential project proponents.
Evaluation Issue: Design
and Delivery
Q11: Has there been an
assessment and strategic use of
lessons learned?
Indicator(s)
Methods
Rating
 Identified lessons learned and
best practices
 Identifies strengths and
weaknesses
 Factors that contribute to / detract
from the achievement of results
 Document
review
 Interviews
 Survey
Progress
made,
attention
needed
There has been some assessment and use of lessons learned from an operational
perspective. In response to proponent feedback on the application system, the
application process was automated. There is a desire to continue to evolve the online
system to include reporting as well as the automation of the contribution agreement
development process. From a more strategic perspective, there are no documented
lessons learned although, recently, a process has been put in place, and an interdepartmental working group formed to examine options for the future of IASPP within the
context of renewal of the national strategy.
Both successful and unsuccessful proponents were asked to comment on what they
perceived as the IASPP’s key strengths and weaknesses presented in Figures 2 and 3,
respectively. Of the 37 survey respondents (successful n=25, unsuccessful n=12), 31
responded to the question on strengths and 33 responded to the question on
weaknesses. Overall, the key program strengths are that it is recognized as an important
funding source (48%), and that the program has worthwhile goals and objectives (39%).
The key program weaknesses are its approval process (52%), insufficient funding for the
program (27%) and funding delays (24%). In two cases, the level of funding was raised
as a main contributing factor to ensuring / limiting the effectiveness of a program. These
are consistent with earlier findings.
A key weakness mentioned in the interviews was that many quality proposals go
unfunded as a result of the size of the IASPP fund and the geographic coverage
required of the program. As presented in Table 2, the program has received an average
of over $7 million in proposals each year and can fund only $1 million per year. On
average, only 16% to 19% of proposals are approved. As one proponent commented in
the survey, when asked about the weaknesses of the program, it is “extremely poorly
funded and as a result, good proposals are not fully funded, if at all.”
Environment Canada
48
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Figure 2: Survey Response to Program Strengths (n=31)
Important funding source for work in relation to
Invasive alien species
3
12
Worthwhile program goals and objectives, and
staff
9
3
8
Recognized public need for program
1
2
On line system
1
1
Don't know/other
2
0
Unsuccessful
2
4
6
8
10
12
14
Successful
Figure 3: Survey Response to Program Weaknesses (n=33)
12
Approval process
5
Funding delays
8
0
Insufficient IASPP funding
Reporting requirements unclear and onerous
7
0
5
IASPP staff difficult to contact
1
Lack of bilingual resources (phone and written)
1
Program goals and objectives
1
Don't know
1
0
Successful
Environment Canada
7
2
2
2
4
Unsuccessful
49
6
8
10
12
14
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
5.0 CONCLUSIONS
This section concludes on the findings of the evaluation.
The evaluation of the IASPP revealed that the program remains relevant given that there
is a continued need for a contribution program to address IAS-related issues in Canada
at the grassroots level within the context of the national strategy. There are however
issues with respect to the appropriate role for the IASPP that need to be addressed.
Specifically, there remain questions with respect to alignment of the IASPP to current
government priorities, as well as the effectiveness of the program in achieving its overall
objectives given issues related to a lack of concise and focused program priorities and
limited program funding.
Although the IASPP was well-aligned with government-wide priorities in 2005, when the
program was created, there is only an indirect link between IASPP objectives and
current government-wide priorities. With respect to departmental priorities, the
evaluation found that the IASPP remains consistent with Environment Canada’s
mandate, most importantly the protection of biodiversity.
The IASPP does contribute to some extent to the national strategy and complements the
activities of other federal department and agency partners which primarily focus on
science-based regulatory initiatives and national surveillance. The IASPP, on the other
hand, focuses on promoting public compliance with regulatory approaches and
establishing voluntary initiatives to address pathways not amenable to regulation.
Environment Canada is also perceived as the best situated department to administer the
IASPP and coordinate the national strategy, given that it is seen as a neutral party
without a vested interest with respect to IAS.
The overall objectives of the program were found to be too broad, resulting in the
program being spread too thinly across the country. According to the evidence collected,
although the individual projects funded by the program were generally successful in
achieving their respective objectives, these results were at the local/community level. As
a result, program contributions have resulted in marginal gains, not major impacts at the
national level. Most interviewees felt that there is a need to determine a concise and
more focused set of key priorities for the program, where it could be expected to be able
to make a difference.
The evaluation also found that the overall effectiveness of the program was hampered
by the total amount of program funding. According to the evidence collected, the
program could not achieve its overall objectives given limited resources available.
Although the relevance of a contribution program was not questioned in and of itself, the
ability of the IASPP to achieve its overall objectives given present funding levels was
raised as a concern. Given this concern, some interviewees felt that diverting the funding
to other programs under the national strategy could have a greater impact on addressing
IAS-related concerns nationally. As a result, there is a need to examine the appropriate
role for the IASPP, along with commensurate funding needed to effectively carry out this
role for the program.
The evaluation also found that there are a number of areas that need to be addressed in
the delivery of the program, including the implementation of an effective performance
measurement strategy and the need to clearly communicate key priorities and objectives
to potential proponents.
Environment Canada
50
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
6.0 RECOMMENDATIONS
These recommendations are directed to the ADM, ESB.
Recommendations 1 and 2:
Overall, although there is a continued need for a contribution program in Canada to
address IAS at the grassroots level, interviewees questioned the effectiveness of the
IASPP to achieve its overall objectives given issues related to a lack of concise and
focused program priorities and limited program funding. Most felt that although the
individual projects funded by the program were generally successful in achieving their
respective objectives, these results were at the local/community level. It was therefore
felt that the contributions of the program resulted in marginal gains at the
local/community level and not major impacts towards addressing IAS-related concerns at
the national level. Some interviewees felt that given these results, the limited funds could
be diverted to other programs under the national strategy to provide a greater impact at
addressing IAS-related concerns at the national level.
Interviewees spoke of the need to re-examine the overall objectives of the program, in
order to determine a few key priorities for the program where it could be expected to
make a difference. At present, it was felt that the current objectives were too broad,
resulting in the program being spread too thinly across the country, providing nominal
funding to small localized projects, instead of focusing on a few key priorities that would
result in the program making a bigger impact at addressing IAS-related concerns
nationally.
Furthermore, limited program funding also had a negative impact on the overall
effectiveness of the program to achieve its objectives, as it affected both the
management of IASPP contribution agreements and Environment Canada’s ability to
fulfil its coordinating role with respect to the national strategy. Many identified the need
for a more effective coordination of the national strategy, something they felt was lacking
at present.
Recommendation 1:
In the context of the upcoming renewal of the IASSC, the ADM, ESB should liaise
with counterparts in participating departments and agencies to determine the
appropriate role for the IASPP within the context of both the IASSC and EC
departmental priorities, along with commensurate funding for the IASPP.
Recommendation 2:
Based on the decision taken with respect to the appropriate role and
commensurate funding of the IASPP as per Recommendation 1, it is
recommended that the program’s objectives be re-examined with an aim to
develop a concise and more focused set of key priorities for the program to
ensure that the IASPP contributes in an effective manner to both the national
strategy and EC departmental priorities.
Environment Canada
51
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Recommendation 3:
Evidence collected as part of this evaluation illustrated a lack of clear understanding by
project proponents about the overall objectives of the IASPP, thereby making it difficult
for them to develop projects that link more directly with the program’s objectives. The
application guidelines do provide some guidance, but it is at a very high level, with more
emphasis on the evaluation criteria and listing examples of the types of activities that are
funded, than specifically stating the objectives or the expected results of the IASPP.
Another key weakness of the IASPP lies in its limited ability to demonstrate achievement
of its intended outcomes. This is due in part to the absence of a formal mechanism to
validate the performance data that are collected by project proponents and provided to
the program at the completion of individual projects. Thus, there is uncertainty about the
accuracy of the performance data provided. Furthermore, the program presently collects
information on approximately 114 key performance indicators, used to measure the level
of achievement related to 56 project-level outputs and outcomes. The sheer number of
performance indicators, as well as their diverse nature (i.e., environmental, economic,
community capacity, outreach, innovation, engaging Canadians, training) has made it
difficult for the program to consolidate performance data and use this data to report on
the overall success of the program in achieving its outcomes. Furthermore, in some
cases, there are issues with respect to the link between these performance indicators
and the project-level outputs and outcomes that they are intended to measure. Thus, the
program cannot use the indicators to determine whether IASPP-funded projects have
been successful in achieving specific project results and overall program objectives.
As a result, it is quite difficult to determine the extent to which the program has been
successful in achieving its immediate and intermediate outcomes, as identified in the
IASPP logic model.
Based on the decision taken with respect to the appropriate role and
commensurate funding of the IASPP as per Recommendation 1, as well as the
articulation of program priorities as per Recommendation 2, it is recommended
that:
a. the program clearly communicates its key priorities and objectives to potential
project proponents to enable their understanding and thus ensure that
projects which are developed are more directly linked to program objectives.
b. the program’s current performance measurement strategy be re-examined with
the objective to develop and implement a set of intended project-level
outcomes and performance indicators that would allow the program to be
better able to demonstrate its results.
Environment Canada
52
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
7.0 MANAGEMENT RESPONSE
The Assistant Deputy Minister, Environmental Stewardship Branch, Environment
Canada (ADM, ESB) agrees with these three recommendations. Should there be a
government decision to renew the IASPP, the program commits to the following actions
in response to the three recommendations.
Recommendation 1:
In the context of the upcoming renewal of the IASSC, the ADM, ESB should liaise
with counterparts in participating departments and agencies to determine the
appropriate role for the IASPP within the context of both the IASSC and EC
departmental priorities, along with commensurate funding for the IASPP.
If renewed, the specific role of and priorities for the IASPP should focus on the IASCC
and EC departmental priorities, the priorities of the Canadian Food Inspection Agency
(CFIA) and Fisheries and Oceans Canada (DFO) with whom EC shares governance of
the IASPP, and the advice of other involved federal departments and agencies. The
following action will be taken by the ADM, ESB to ensure this recommendation is
addressed:
DATE
December
2009
ITEM

ADM, ESB to discuss role of, priorities and commensurate funding
for the IASPP in implementing the renewed IASSC with other
federal departments and agencies.
Recommendation 2:
Based on the decision to be taken with respect to the appropriate role and
commensurate funding of the IASPP as per Recommendation 1, it is
recommended that the program’s objectives be re-examined with an aim to
develop a concise and more focused set of key priorities for the program to
ensure that the IASPP contributes in an effective manner to both the national
strategy and EC departmental priorities.
Following confirmation of the role and commensurate funding for the renewed IASPP,
the program’s objectives will be re-examined to create a more focused set of key
priorities aligned with both the national strategy and EC departmental priorities. In doing
so, EC will consult with the Canadian Food Inspection Agency, the Department of
Fisheries and Oceans and other key involved federal departments and agencies with
respect to their priorities for IASPP.
Specific actions to be taken to address this recommendation include:
DATE
December 2009
ITEM

Subject to renewal of the IASPP, identify strategic outcomes,
objectives and key priorities for the renewed IASPP in
consultation with other federal departments and agencies.
 Program to return to Environmental Sustainability Board with
recommendations for future implementation of the IASPP.
Environment Canada
53
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Recommendation 3:
Based on the decision to be taken with respect to the appropriate role and
commensurate funding of the IASPP as per Recommendation 1, as well as the
articulation of program priorities as per Recommendation 2, it is recommended
that:
a. the program clearly communicates its key priorities and objectives to potential
project proponents to enable their understanding and thus ensure that
projects which are developed are more directly linked to program objectives.
Subject to a renewed IASPP, EC will refine and update the IASPP application guidelines
including project selection criteria to ensure greater clarity about the overall objectives of
the IASPP, thereby ensuring projects link directly with the Program’s objectives.
Specific actions to be taken to address this recommendation include:
DATE
ITEM
January 2010

Refine the application guidelines in line with revised Program
objectives.
 Provide guidance such as a list of eligible and ineligible projects to
project proponents to ensure clarity with respect to IASPP
objectives and priorities.
b. the program’s current performance measurement strategy be re-examined with
the objective to develop and implement a set of intended project-level
outcomes and performance indicators that would allow the program to be
better able to demonstrate its results.
The performance measurement strategy of a renewed IASPP will be revised in line with
the objectives and priorities to be established for the Program. Performance indicators
will be streamlined to ensure collection and consolidation of meaningful data which can
be used to report on the overall success of the program in achieving its outcomes.
To meet this goal, the following specific actions will be taken.
DATE
ITEM
March 2010

Refine current logic model to reflect the new objectives (outcomes)
for the Program
 Complete a Performance Measurement Strategy with the aim to
develop performance indicators most relevant to program
objectives
March 2011
 Improve the reporting format, implement an appropriate reporting
tool and make other program adjustments as determined by the
departmental G&C risk assessment strategy developed as part of
the Departmental Action Plan for G&C Reform (CAPE Optimization
Initiative)
March 2012
 Adopt the departmental online application and information
management system, which will enhance client service and
facilitate application, monitoring and reporting processes for both
clients and program staff (CAPE Optimization Initiative)
Contact persons: Bob McLean, Elizabeth Roberts
Environment Canada
54
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Annex A
National Invasive Alien Species Strategy for
Canada Logic Model
Strategic
Outcome
Making Canada a leader in the prevention and management of IAS in a manner
that ensures environmental sustainability, economic competitiveness and
societal well being.
Prevention of unintentional introductions of IAS in Canada.
Ultimate
Outcome
Intermediate Outcomes
Application of risk analysis
to unintentional pathways of
introduction.
Development and application of
innovative technologies,
practices, and/or processes
regarding unintentional pathways
of introduction.
Application of technologies,
practices and/or processes
to minimize the risks of
unintentionally exporting IAS
to other countries.
Immediate Outcomes
Improved capacity
to identify risks
associated with
unintentional
pathways of
introduction
Improved capacity
to detect and
respond to IAS
Integrated
legislative and
policy framework
applied
Increased
awareness of IAS
in specific
resource sectors
and the public
Increased
influence of
international
technical
standards and
policies
Outputs
Risk Analysis
Development
Develop a risk
analysis for
unintentional
pathways of
introduction
Science &
Technology
Development
Legislation,
Regulation, Policy
Framework
Establish a Canadian
Develop a National
Plant Protection
Invasive Plant
Network, and a national Strategy
aquatic invasive
species surveillance
network
Environment Canada
55
Plan for Engaging
Canadians
and
Sectors
Strategy for
International
Cooperation
Development of a
joint industrygovernment-NGO
communications
campaign
Develop a strategy
to influence polices/
standards of int’l
standard-setting
bodies (IMO, IPPC,
OIE)
Audit and Evaluation Branch
Evaluation of the Invasive Alien Species Partnership Program
Annex B
Evaluation Matrix
Evaluation Issue
Indicators
Data Source
Methodology
Relevance
Q1: Does the IASPP continue to be
consistent with government-wide
priorities and the departmental
mandate?
•
Q2: Is there a continued need for
the IASPP?
•
•
Program objectives that are aligned to governmentwide priorities and departmental mandate.
Inter/intra-governmental relations and coordination
(F/P/T and First Nations).
•
•
•
•
Utility/rationale for the program
Ongoing need for capacity or activity in the area of
departmental jurisdiction
•
As per indicators in the performance measurement
plan and reporting strategy
•
•
Invasive Alien Species Strategy for
Canada, RPP, DPR, Speech from
Throne
Terms of Reference and records of
decision from coordinating bodies,
MoUs, other agreements
Program staff
Professional journals, papers
Environmental groups and other
stakeholders
•
Program management documentation
MoUs, Agreements, Terms of
References, Records
Project files
Stakeholder interviews
Survey
Project files and reports
Stakeholder interviews
Survey
•
Project files and reports
Program documentation
Program staff and stakeholders
•
•
•
•
•
Document review
Interviews
Document review
Interviews
Survey
Success
Q3: Are planned activities being
implemented and producing
expected outputs?
•
•
•
•
•
Q4: Are immediate and intermediate
outcomes being achieved as a
result of the program?
•
Q5: Have there been any
unexpected outcomes (positive or
negative)? Were any actions taken
as a result of these?
•
As per indicators in the performance measurement
plan and reporting strategy
•
•
•
•
Documented unexpected outcomes
Lessons learned or best practices have been
documented
•
•
•
•
•
•
•
•
•
•
Environment Canada
57
Document review
Interviews
Survey
Document review
File review
Interviews
Document review
File review
Interviews
Audit and Evaluation Branch
Evaluation of the Invasive Alien Species Partnership Program
Evaluation Issue
Indicators
Data Source
Methodology
Cost effectiveness/alternatives
Q6: Are the most appropriate and
efficient means being used to
achieve objectives, relative to
alternative design and delivery
approaches?
Design and Delivery
•
Q7: Are program deliverables and
expected outcomes identified
clearly, and is the program being
delivered as designed?
•
Q8: Are activities and outputs linked
to shared outcomes and the overall
government-wide objectives?
Q9: Are performance data collected
against program activities/outcomes
and used for informed decision
making?
Q10: Are there appropriate
governance structures in place to
meet the objectives?
•
Q11: Has there been an
assessment and strategic use of
lessons learned?
•
•
•
•
•
•
Financial management system lined to results
management process.
Consideration of alternate approaches
•
Demonstration of the program’s expected
deliverables and results
Consistency with documented design and
delivery/implementation
•
Links between activities, outputs and outcomes
supporting joint national objectives
•
Measuring and reporting strategies and procedures
Presence of populated performance data system
Decisions based on performance information
•
•
Financial management system
Finance department staff
•
•
•
•
•
•
Program documentation (RMAF,
NIAS, etc.)
Program staff and stakeholders
•
Program documentation
Program staff and stakeholders
•
Program Management System
Program staff
•
•
•
•
•
•
•
•
•
•
•
•
Established governance structure
Roles, responsibilities and accountabilities are well
defined, well communicated, and understood.
Roles, responsibilities and accountabilities are
functioning as intended.
Identified lessons learned and best practices
Identifies strengths and weaknesses
Factors that contribute to / detract from the
achievement of results
•
•
•
•
Program documentation, MoUs,
Agreements, ToRs
Program staff and stakeholders
•
Program documentation
Program staff
•
•
•
•
•
Environment Canada
58
Document review
Interviews
Survey
Document review
Interviews
Survey
Document review
Interviews
Document review
File review
Interviews
Survey
Document review
File review
Interviews
Document review
Interviews
Survey
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Annex C
List of Background Information and Supporting
Documentation
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
Document/Files
List of Funded IASPP Projects – 05-06
List of Funded IASPP Projects – 06-07
List of Funded IASPP Projects – 05-06
Treasury Board Submission – Implementing an Invasive Alien Species Strategy
for Canada
EC IASPP RMAF March 2008
RMAF – Invasive Alien Species Strategy for Canada May 2008
An Invasive Alien Species Strategy for Canada
IASPP Online Application Process – Survey Results
IASPP Summary Data (Revised Dec 16 - summary of projects)
National Aquatics Invasive Species Committee – Terms of Reference
MOU IAS Partnership Program
Contribution Agreement Development Process and Tips
EC Common Indicators – Grants and Contributions
Memo to DM – call 2006-07 and annexes
Memo to DM – call 2007-08 and annexes
Memo to DM – call 2008-09 and annexes
Technical Review Committee (TRC) Terms of Reference
IASPP Project Performance Summary – updated January 9, 2009
IASPP Successful Applicants (Call 1-4)
IASPP Unsuccessful Applicants (Call 4)
2008 TRC meeting summary data
Evaluation Summary 2008 TRC version
IASPP Minutes and Comments
Report IASPP Technical Review Committee Meeting May 3-4 2007
Report IASPP Technical Review Committee Meeting March 08
Technical Review Committee – 0708 selected projects
IASPP Summary Data – revised December 19
Financial reports
Environment Canada RPP 2008-2009
Environment Canada DPR 2007-2008
Project files (8)
IASPP Application Guidelines September 2008
Environment Canada
59
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Annex D
List of Interviewees
Key Informant Interviews - Round I
Number of Regions
Represented
Number of Individuals
Interviewed
1
8
1
5
Provincial
3
3
Non-Governmental
Organizations
3
3
Department
Environment Canada
Other Government Departments
CFIA; NRCan; DFO; AAFC
Other Stakeholders
19
Total
Key Informant Interviews - Round II
Number of Regions
Represented
Number of Individuals
Interviewed
Environment Canada
program staff and senior
management
1
4
Interdepartmental Directors
General IAS Steering
Committee members
2
4
Department
Other Federal Government Departments and Agencies
CFIA; NRCan; AAFC
1
12
Total
Environment Canada
4
60
Audit and Evaluation Branch
Evaluation of the Invasive Alien
Species Partnership Program
Annex E
Summary of Findings19
Evaluation
Question (EQ)
Achieved
Relevance:
Q1: Does the IASPP continue to be consistent
with government-wide priorities and the
departmental mandate?
Progress
Made,
Attention
Needed
Little
Progress,
Priority for
Attention
Not
Applicable


Q2: Is there a continued need for the
Department’s contribution?
Success:

Q3: Are planned activities being implemented
and producing expected outputs?
Q4: Are immediate and intermediate outcomes
being achieved as a result of the program?
~
Q5: Have there been any unexpected
outcomes (positive or negative)? Were any
actions taken as a result of these?
N/A
Cost Effectiveness:
Q6: Are the most appropriate and efficient
means being used to achieve objectives,
relative to alternative design and delivery
approaches?

Design and Delivery:
Q7: Are program deliverables and expected
outcomes identified clearly, and is the program
being delivered as designed?

Q8: Are activities and outputs linked to shared
outcomes and the overall government-wide
objectives?

Q9: Are performance data collected against
program activities/outcomes and used for
informed decision making?
Q10: Are there appropriate governance
structures in place to meet the objectives?



Q11: Has there been an assessment and
strategic use of lessons learned?
19
The rating symbols and their significance are outlined in Table 1.
Environment Canada
61