Evaluation of the Great
Lakes Action Plan IV
Final Report
June 10, 2010
Audit and Evaluation Branch
Evaluation of the Great Lakes Action Plan IV
Report Clearance Steps
Planning phase completed
Report completed
Report sent for management response
Management response received
Report approved by Departmental Evaluation Committee (DEC)
December 2009
March 31, 2010
April 29, 2010
May 25, 2010
June 10, 2010
Acronyms used in the report
AOC
BUI
CESD
COA
DFO
ECB
GLAP
GLAP IV
GLBEI
GLEC
GLSF
GLWQA
G&C
IJC
MC
MNR
MOE
MOU
NGOs
NWRI
OGDs
O&M
PWGSC
RAP
RDG
TB MAF
Areas of Concern
Beneficial use impairments
Commissioner of the Environment and Sustainable Development
Canada–Ontario Agreement Respecting the Great Lakes Basin Ecosystem
Fisheries and Oceans Canada
Environmental Conservation Branch
Great Lakes Action Plan
Great Lakes Action Plan for Areas of Concern (Phase IV)
Great Lakes Basin Ecosystem Initiative
Great Lakes Executive Committee
Great Lakes Sustainability Fund
Great Lakes Water Quality Agreement
Grants and Contributions
International Joint Commission
Management Committee
Ministry of Natural Resources (Ontario)
Ministry of the Environment (Ontario)
Memorandum of understanding
Non-governmental organizations
National Water Research Institute
Other government departments
Operations and maintenance
Public Works and Government Services Canada
Remedial Action Plan
Regional Director General
Treasury Board Management Accountability Framework
Acknowledgments
The Audit and Evaluation Branch, Evaluation Division Project Team led by William Blois,
under the direction of the Director, Shelley Borys, would like to thank those individuals
who contributed to this project, particularly members of the Evaluation Committee as
well as all interviewees who provided insights and comments crucial to this evaluation.
This evaluation was conducted and reported by EKOS Research Associates Limited.
Environment Canada
Audit and Evaluation Branch
Evaluation of the Great Lakes Action Plan IV
Table of Contents
EXECUTIVE SUMMARY ....................................................................................i
1.0
Introduction .............................................................................................. 1
1.1
2.0
Evaluation Design ................................................................................ 13
2.1
2.2
2.3
2.4
3.0
Objectives and Scope....................................................................................13
Approach and Methodology .........................................................................13
2.2.1 Document and Literature Review ............................................ 14
2.2.2 File Review .................................................................................. 14
2.2.3 Key Informant Interviews........................................................... 15
Challenges and Limitations ..........................................................................16
Reporting Note.................................................................................................17
Findings.................................................................................................... 19
3.1
3.2
3.3
3.4
3.5
4.0
Program Profile................................................................................................. 1
1.1.1 Program Goals.............................................................................. 2
1.1.2 Program Activities ........................................................................ 4
1.1.3 Stakeholders and Recipients...................................................... 8
1.1.4 Governance ................................................................................... 9
1.1.5 Resources ................................................................................... 10
1.1.6 Program Logic Model ................................................................ 10
1.1.7 Performance Reporting and Evaluations................................ 12
Relevance .........................................................................................................19
3.1.1 Continued Need.......................................................................... 20
3.1.2 Alignment with Federal and Departmental Priorities ............ 24
Program Performance: Design and Delivery ..........................................29
3.2.1 Engagement of Partners ........................................................... 31
3.2.2 Work Planning............................................................................. 33
3.2.3 Roles and Responsibilities........................................................ 34
3.2.4 Resources ................................................................................... 36
3.2.5 Monitoring and Reporting.......................................................... 38
3.2.6 GLSF Program Delivery ............................................................ 40
Program Performance: Achievement of Program Outcomes ............42
3.3.1 Program Outcomes .................................................................... 44
Program Performance: Efficiency ..............................................................53
Performance: Program Economy ..............................................................58
3.5.1 Cost-Effectiveness ..................................................................... 59
3.5.2 Alternative Approaches ............................................................. 61
Conclusions ............................................................................................ 62
4.1
Relevance .........................................................................................................62
Environment Canada
4.2
4.3
4.4
4.5
4.1.1 Continued Need for the Program ............................................. 62
4.1.2 Alignment with Federal and Departmental Priorities ............ 62
4.1.3 Consistency with Federal Roles and Responsibilities .......... 62
Program Performance: Design and Delivery ..........................................63
Program Performance: Achievement of Program Outcomes ............64
Program Performance: Cost-Efficiency....................................................64
Program Performance: Cost-Effectiveness ............................................65
5.0
Recommendations ............................................................................... 66
6.0
Management Response ..................................................................... 69
ANNEXES (UNDER SEPARATE COVER):
Annex A: GLAP IV Committee Structure
Annex B: Evaluation Issues and Questions
Annex C: Bibliography
Annex D: Sample and Population of GLSF Files
Annex E: File Review Template
Annex F: Profile of Agreement Files
Annex G: Descriptive Profile of Sample of Federal Projects
Annex H: Interview Guides
Annex I: Summary of Findings
Annex J: Status of Beneficial Use Impairments and AOCs
Environment Canada
Audit and Evaluation Branch
Evaluation of the Great Lakes Action Plan IV
EXECUTIVE SUMMARY
Background
In the 2009–2010 fiscal year, Environment Canada’s Audit and Evaluation Branch,
Evaluation Division, commissioned an evaluation of the Great Lakes Action Plan IV
(GLAP IV). This program was selected for evaluation prior to the end of the program’s
terms and conditions at the close of the 2009–2010 fiscal year.
The GLAP for Areas of Concern (AOCs) is the primary vehicle under the Great Lakes
Basin Ecosystem Initiative (GLBEI) for the federal government, with Environment
Canada as the lead department, to act to restore AOCs around the Great Lakes area
and fulfill Canadian commitments under the Great Lakes Water Quality Agreement
(GLWQA) with the United States. The current fourth phase1 of the Action Plan, GLAP IV,
was approved in 2005 with a budget of $40 million ($8 million per year over five years) to
restore, protect and conserve AOCs around the Great Lakes.
The goals of GLAP IV are:
›
›
to make progress on federal2 actions in eight AOCs: St. Mary’s River,
St. Clair River, Toronto and Region, Hamilton Harbour, Jackfish Bay,
Detroit River, Niagara River and Port Hope; and
to complete federal actions in seven AOCs: Thunder Bay, Nipigon Bay,
Peninsula Harbour, Bay of Quinte, Wheatley Harbour, St. Lawrence River
and Spanish River.
To achieve these goals, GLAP IV is based on an ecosystem approach: remedial efforts
targeted to AOCs under GLAP IV deal with interrelated environmental and sustainable
development issues based on beneficial use impairments (BUIs) identified in these
areas. As laid out in the GLWQA, remediation in each AOC is guided by Remedial
Action Plans (RAPs).
Program funding is allocated to remedial actions (e.g., sediment remediation, habitat
restoration), science (e.g., assessment and monitoring of ecosystem health and status of
BUIs), and engagement/governance (e.g., engagement of community members, support
for RAP committees). Over the life of the program, approximately 40 science and
monitoring projects led by federal partners and over 200 grants and contributions (G&C)
projects under the Great Lakes Sustainability Fund (GLSF) were approved for funding.
1
The federal government has provided GLAP funding since 1989 ($125 million over 5 years in
1989; $150 million over 6 years in 1994; $40 million over 5 years in 2000; and $40 million over
5 years in 2005). Identification, evaluation and remediation of AOCs have been a major
element of GLAP funding since 1989 and the single focus of GLAP funding since 2000.
2
Federal actions reference the Government of Canada’s actions in response to its commitments
under the COA (as opposed to Ontario’s). These vary by AOC.
Environment Canada
i
Evaluation Issues
The evaluation covers GLAP IV program activities from the 2005–2006 to 2009–2010
fiscal years. The evaluation issues include the following:
›
›
›
›
›
Relevance: Is there a continued need for a GLAP? Is GLAP IV aligned to
federal government priorities? Is GLAP IV consistent with federal roles and
responsibilities?
Design and Delivery: Is the GLAP IV design appropriate for achieving
expected program results?
Achievement of program outcomes: To what extent have intended outcomes
been achieved as a result of GLAP IV? Have there been any unintended
(positive or negative) outcomes?
Efficiency: Is GLAP IV undertaking activities and delivering products in the
most efficient manner?
Economy: Is GLAP IV achieving its intended outcomes in the most
economical manner?
Methodology
Data were collected for the evaluation using multiple lines of evidence. These included a
document review, a review of 39 GLSF G&C agreement files, a qualitative analysis of 21
federal-partner project files, and a total of 46 key informant interviews with departmental
program managers and federal partners (n=11); representatives of various committees
(n=10); federal project proponents (n=4); GLSF project proponents (n=10); unsuccessful
applicants (n=4); RAP committee members (n=4); and international or academic experts
on aquatic ecosystem remediation (n=3). Key challenges for the study included a
reliance on internal sources of evidence and limited availability of program activity
information and financial data with respect to GLAP IV.
Evaluation Findings
Relevance
GLAP IV remains a relevant program that enables the federal government to address
the continued need for restoration and maintenance of the Great Lakes AOCs. While
recent trends in Great Lakes’ ecosystem conditions are variable, historical sources of
stress, combined with new challenges, are leading to negative impacts in many areas of
the lakes. Environmental need, together with the social and economic benefits of the
Great Lakes, supports the continued relevance of the program. Public opinion is
consistent with scientific information and key informant views on the environmental and
societal importance of the Great Lakes and the continued need for the GLAP.
There is little redundancy risk associated with GLAP IV. The program uniquely
addresses federal commitments in the GLWQA and the Canada–Ontario Agreement
Environment Canada
ii
Respecting the Great Lakes Basin Ecosystem (COA) by providing a framework to
address BUIs linked to AOC RAP priority areas. Unlike other public funding (federal,
provincial and municipal) and non-public funding (industry, foundations,
non-governmental organizations [NGOs], etc.), GLAP IV is targeted exclusively to AOCs.
Key informants express few concerns about the possibility of duplication or overlap
between GLAP IV and other programming. Rather, GLAP IV is seen as being highly
complementary and as a catalyst in orienting other funding sources to provide support in
the AOCs.
GLAP IV is consistent with federal and departmental priorities. The program is part of the
GLBEI and is one of six ecosystem initiatives at Environment Canada under the Priority
Ecosystem Initiatives Sub-Activity. The GLBEI supports the Department’s Strategic
Outcome that “Canada’s natural capital is restored, conserved and enhanced.” GLAP IV
outcomes and priority investments also support a number of other sub-activities within
Environment Canada’s program activity architecture, including Aquatic Ecosystems,
Wildlife, Priority Ecosystems, Ecosystems Sustainability, and Assessment and
Ecological Monitoring.
Environment Canada’s GLAP IV is aligned to support federal government commitments
and obligations under the GLWQA and COA, as well as federal government priorities
outlined in the 2007 Speech from the Throne and two recent federal budgets. Legislative
authorities for the GLAP IV include the Canada Water Act, the International Boundary
Waters Treaty Act, and the Canadian Environmental Protection Act.
Design and Delivery
Overall, GLAP IV was viewed as a sensible model to achieve intended outcomes.
Engagement of partners represents a critical and beneficial aspect of the GLAP IV
program delivery, with partnerships occurring at many levels and in many ways. Formal
agreements between Canada and the United States (GLWQA) and Canada and Ontario
(COA) represent the foundation for commitments of the jurisdictional parties with respect
to the AOCs. Implementation involves horizontal partnerships within Environment
Canada and across the federal government. RAP committees provide a forum for
jurisdictions (federal, provincial, municipal and First Nations) and key stakeholder groups
(conservation authorities, community groups and NGOs) to coordinate their efforts
through to the delisting process. The funding structure of GLSF projects promotes
partner engagement at the local and regional level with partner contributions including
cash and in-kind funding, assistance with delivery, and participation in advisory or
research capacity.
The implementation of GLAP IV diverged from the original design of the program in
several areas. Although the five-year work plans required by federal partner departments
to access GLAP IV funds were considered to be a positive and worthwhile exercise, the
work planning, annual reporting, review and updating processes did not occur as
intended, which resulted in a loss of flexibility, responsiveness and accountability.
Due to the horizontal nature of the initiative and broad changes at Environment Canada,
GLAP IV governance and management activities do not reflect those outlined in the
Environment Canada
iii
management framework developed for the program. Areas identified as a potential
source of confusion or concern by federal key informants include an inadequate
instrument to ensure that partner contributions to program goals are fulfilled, and clarity
of roles and responsibilities with respect to delisting AOCs.
For federal partners who received their recommended funding, the amount of GLAP IV
funding was seen to be adequate, with the caveat that they would like future funding to
be indexed for inflation. However, some federal partners report receiving significantly
fewer resources than their original approved funding levels, which they feel impacted
negatively on their ability to implement planned activities. For federal partners that
received their GLAP IV funding allocation in full, a more pressing concern was the high
ratio of operations and maintenance (O&M) to salary dollars and inability to convert
these dollars (i.e., from O&M to salary), which left many groups with insufficient salary
dollars to hire scientific and technical staff. GLSF project proponent and committee key
informants tended to suggest that the GLAP IV focus and resources be expanded to
include priority areas and activities that extend beyond the designated AOCs.
GLAP IV monitoring and reporting activities are undertaken to meet annual reporting
obligations associated with both the GLWQA and the COA. At the level of funded
projects, GLSF project reporting is quite complete and the majority of reviewed GLSF
files provided clear evidence of intended output achievements, however, limited
evidence was provided on the outcomes of remedial actions taken in the AOCs. While
federal proponents were supportive of increasing reporting requirements for GLAP IV
projects to enhance performance measurement and accountability, regular reporting of
federal science and monitoring projects is not a formal requirement and reporting to the
GLAP IV program is ad hoc.
Project proponent key informants indicate a high level of satisfaction with the GLSF
program. Program priorities are considered to be clear and appropriate. The application
process is regarded as clear and transparent by most funding recipients and selection
criteria are viewed to be explicit and well communicated. Overall, the GLSF proposal
review process is perceived to be logical and GLSF funding decisions are viewed as
strategic and fair. Program staff are seen by project proponents to be accessible and
supportive. Modifications to the GLSF program (including a shift from the use of
memoranda of understanding to contribution agreements), implemented by Environment
Canada in 2007, have challenged the timely delivery of the program, reduced flexibility
and increased uncertainty among partners. Project proponents indicate a preference for
streamlining the application and approval process, for multi-year agreements and for
more straightforward funding agreements.
Achievement of Program Outcomes
In general, key informants had favourable impressions on the performance of the
program with respect to the achievement of immediate and intermediate outcomes, a
finding that is supported by the program documentation and review of GLSF and federal
project files. As indicated, engagement of partners at the local level and across
jurisdictions and scientific communities is perceived to be a strength of the program. This
is supported by GLAP IV funds allocated to federal partners to facilitate coordination and
Environment Canada
iv
management of GLAP IV (e.g., with the provinces and First Nations). Engagement of
partners and participation at the local level is facilitated through committees, informal
networks and the RAP structure (which also receive GLSF funding support). With
respect to remedial actions, addressing pollution (through sediment remediation, support
to municipal infrastructure improvements and reducing non-point sources of pollution) is
a significant priority for the program and is allocated almost one half of the GLAP IV
dollars. While the success of federal partners’ efforts is difficult to discern due to gaps in
the federal project files, GLSF projects that supported this objective were found to be
well documented in terms of outputs and experienced few challenges in implementation.
BUI assessment and monitoring work is occurring on many fronts to assess the status of
BUIs, effectiveness of restoration activities and defining BUI goals/targets. Fewer federal
projects were funded for habitat restoration (though this is a particular focus of GLSF).
Habitat projects are more easily documented in terms of outputs (e.g., plantings, wetland
acreage), but these projects are also more subject to implementation challenges.
Overall, program investments advance activity in each AOC to some degree (though
some coordination projects target all AOCs generically). Note that while group 1 AOCs
(those closest to delisting) were intended to be assigned a higher priority for GLAP IV
investments, the number of projects funded in these group 1 AOCs is lower than in the
group 2 AOCs.
Achievement of the program’s longer-term outcomes is much less evident, and the
majority of BUIs that were originally identified in AOCs still exist. Of the over 100 BUIs
identified, only 20 per cent have been restored. In consideration of the status of the
BUIs, the program has identified outstanding priority actions for each AOC and projected
timelines for delisting. Only two AOCs have been delisted and one AOC designated an
area in recovery and these occurred prior to GLAP IV. The original program goal of
completing federal actions in seven group 1 AOCs has not been achieved. Three AOCs
are expected to be delisted or designated as Areas in Recovery within the next one to
two years, and a fourth, Wheatley Harbour, has just been delisted (announced on
April 16, 2010). With respect to the remaining AOCs, the picture is much more complex
and the time frames for delisting are longer-term, with most of these AOCs estimating
delisting as occurring between 2015 and 2020. For committee and expert key
informants, the overall importance of the contribution made by GLAP IV to the
restoration and maintenance of AOCs is viewed as being very significant, if not critical.
For key informants, positive external factors that support program success include:
leveraged funding, particularly infrastructure funding; United States investments; and
community/political engagement (e.g., the Great Lakes and St. Lawrence Cities
Initiative). External factors that interviewees identified as having the potential to
negatively affect the success of GLAP IV include: new and changing ecosystem issues;
and the economic downturn. Unintended outcomes of GLAP IV cited by key informants
tend to be positive and focus on unexpected interest and engagement of non-targeted
groups (e.g., landowners) and the general public, as well as on unanticipated
opportunities for collaboration and knowledge transfer.
Environment Canada
v
Efficiency
The cost-efficiency of GLAP IV is difficult to determine with existing information due to
the unbundling3 of GLBEI funds and activities and the introduction of a shared,
results-based accountability approach to environmental initiatives. The impact on
financial data availability resulting from the changes implemented to Environment
Canada’s resource allocation and accountability structure is that GLAP IV funds are not
specifically coded at the program level and the A-base support allocated to GLAP IV is
not known.
Financial analysis of the GLSF program component indicates that for every contribution
dollar, $0.24 is spent on program administration (salary and O&M). This amount
compares favourably to Environment Canada’s EcoAction Community Funding Program,
which has a cost-efficiency ratio of $0.39, but unfavourably with the Habitat Stewardship
Program ($0.08) and the Invasive Alien Species Partnership Program ($0.13). Like
EcoAction, the delivery of the GLSF involves additional staff responsibilities such as
participation in RAP committees.
For key informants, there is an impression of efficient program delivery both overall and
at the project level, with a number of factors identified that support cost-efficiency at the
program and project levels. Some suggestions to improve efficiency were nonetheless
received.
Economy
There is consensus among key informants that GLAP IV is a good investment of public
funds and that results are achieved in an economical way. Design aspects that are seen
as contributing to the program’s cost-effectiveness include: a focused effort on AOCs;
cost-sharing among jurisdictions and key stakeholders; local involvement and delivery;
and continuity of intervention and knowledge acquisition over successive GLAP
programs. A few key informants also noted the economic benefits of remediation, as well
as the cost of not intervening.
No viable alternative approaches to achieve the same or better outcomes at a lower cost
were identified. The prevailing sentiment among key informants is to “stay the course” in
that GLAP IV provides a solid foundation for work in the AOCs. Fundamental changes to
the program were discouraged by key informants as they may jeopardize the impact of
work completed to date.
Recommendations
Recommendations for the current or future iterations of GLAP are based on the findings
and conclusions of the evaluation. Overall, GLAP IV continued to be a relevant program
to address ongoing needs for ecological restoration in the AOCs and is well-aligned with
departmental and federal priorities. At the end of its five-year term, GLAP IV has not fully
3
Unbundling refers to the process of realigning temporary funding allocations (originally intended
to deliver programs for various specific purposes) toward other departmental priorities.
Environment Canada
vi
achieved its intended longer-term outcomes in the areas of addressing BUIs and
delisting AOCs, due in large part to the complexity and long-term nature of the task. The
following recommendations focus on improvements to the management of the program
in three areas: delivery and oversight; financial management and accountability; and
performance measurement and reporting.
Delivery and Oversight
Oversight of GLAP IV was challenged by several changes that occurred within
Environment Canada during the program’s five-year term. The changes affected both
the overall management and the coordination of the program, and the GLSF. One
recommendation pertains to improving coordination and accountability of the program,
and one pertains to ensuring the timeliness and responsiveness of the GLSF.
1) The Regional Director General (RDG)–Ontario should develop a suitable
instrument to enhance coordination of federal partners involved in
GLAP and the accountability of their efforts toward achieving GLAP
goals. GLAP IV is a complex initiative that requires the contribution of many
federal partners to achieve program goals. With the demise of the work plan
review teams and process, GLAP IV lacked the instruments to ensure that
these efforts were coordinated and that federal partners were implementing
work plans to achieve program goals. The work planning process, convened
annually as it was envisioned, could be re-instituted to enhance coordination
and accountability.
2) The RDG–Ontario should explore means of streamlining both the GLSF
funding approvals process and the GLSF application process. While
mostly satisfied with the GLSF program, proponents identified two issues with
respect to administration: timeliness of approvals and an overly complex
application process. Consideration should be given to scrutinizing the timing
of the funding cycle to ensure that funding approvals coincide with seasonal
activities, and that the application process is streamlined to promote
efficiency (perhaps by exploring adoption/applicability of common application
forms/guidelines being developed as part of the wider federal G&C reform
exercise). Although most contribution agreements are single-year only, the
majority of projects are of an ongoing nature. The use of multi-year
agreements may be appropriate in some of these instances.
Financial Management and Accountability
There is limited information on GLAP IV program expenditures. Financial tracking of
GLAP IV funds was weak during this iteration of the program due to limited use of
codes for expenditures at the program level. One recommendation pertains to financial
monitoring of GLAP funds at Environment Canada.
3) The RDG–Ontario should explore ways to enhance the financial
accountability of the program. The direct transfer of funds to internal
partners and absence of program-specific coding of expenditures within the
Environment Canada
vii
Department has weakened financial accountability of the program. Diligent
coding of expenditures at the program level is a key element to ensure
transferred funds are received and utilized within Environment Canada by
departmental partners to meet program goals and priorities. The merit and
potential disadvantages of direct transfers of program funds within
Environment Canada to departmental partners for financial accountability
should be examined.
Performance Measurement and Reporting
GLAP IV monitoring and reporting was not guided by a performance measurement
framework. As a result, a broad spectrum of performance measures for federal partners
and the GLSF were not identified early in the program and reporting requirements,
particularly for federal partners, were not clearly articulated. Five recommendations
pertain to improving performance measurement and reporting.
4) The RDG–Ontario should develop a more regular and robust reporting
approach for GLAP projects conducted by federal partners. Performance
reporting for projects conducted by federal partners was unspecified and
sporadic. A more robust reporting approach should be based on clear terms
and conditions for reporting on funds allocated to federal partners. Reporting
should link activities/deliverables to approved work plans, recognizing that, in
some cases, federal projects are undertaken as part of an integrated science
program.
5) The RDG–Ontario should develop a performance measurement
framework for the program, and include a spectrum of more sensitive
performance measures (shorter- and longer-term). The program does not
have a performance measurement framework and is not currently
well-positioned to demonstrate progress and interim results because
measures such as delisting AOCs are long-term. A more nuanced approach
must balance the benefit of performance measurement with the cost of
monitoring and assessing BUIs in the AOCs. Recent efforts by the program to
document accomplishments and assign roles, responsibilities and timelines
for outstanding actions and priorities are acknowledged as a strong
foundation for this ongoing work.
6) The RDG–Ontario should support the development of information
management tools to enhance implementation and monitoring of GLSF
and federal projects. No systematic process exists for monitoring and
reporting the activities, outputs and performance for GLAP-funded projects or
the program overall. A Department-wide Management Information System for
Environment Canada’s community funding programs, including the GLSF, is
in development, although this system is not expected to be implemented until
2012. Opportunities for interim tracking of activities and project and partner
contributions should be identified.
Environment Canada
viii
7) The RDG–Ontario should support the development of informationsharing tools to facilitate broader access to the research generated by
GLAP IV by the scientific community and the public. There is no ongoing
mechanism available for technical transfer of research and results across the
AOCs. This is an outstanding commitment in Annex 4 of the COA.
8) The RDG–Ontario should continue efforts to define delisting criteria for
AOCs and clarify responsibilities with respect to delisting
decision making. Roles and responsibilities for aspects of delisting AOCs
are a grey area and may be variable across the AOCs. As more AOCs
approach restoration, clarifying the steps to delisting will increase in
importance.
Management Response
The RDG–Ontario accepts the evaluation and all of its recommendations, and the Great
Lakes Division has provided a plan to implement the following management actions in
response to the evaluation recommendations, within the context of the program’s
renewal.
1. The RDG–Ontario should develop a suitable instrument to enhance
coordination of federal partners involved in the GLAP and the accountability of
their efforts toward achieving GLAP goals. GLAP IV is a complex initiative that
requires the contribution of many federal partners to achieve program goals. With the
demise of the work plan review teams and process, GLAP IV lacked the instruments
to ensure that these efforts were coordinated and that federal partners were
implementing work plans to achieve program goals. The work planning process,
convened annually as it was envisioned, could be re-instituted to enhance
coordination and accountability.
The RDG–Ontario agrees with this recommendation.
A GLAP five-year work plan will be developed by Environment Canada, Great Lakes
Division, with input from all federal departments and Responsibility Centres within
Environment Canada contributing to the restoration of Great Lakes AOCs and accessing
GLAP funds. The GLAP Work Plan Review Team will be re-established and will annually
conduct a review of progress achieved, identify actions required to complete the
restoration of AOCs, determine priorities for the coming year, and propose adjustment of
the work plan and funding allocations as required. The GLAP Work Plan Review Team
will be led by the Great Lakes Division of Environment Canada and will comprise
representatives of Environment Canada, the Department of Fisheries and Oceans,
Public Works and Government Services Canada and other federal government
departments as necessary. Recommendations of the GLAP Work Plan Review Team
will be presented to the Director, Great Lakes Division, for approval. A GLAP work
planning template will be established specifying the project rationale, (i.e., how the
proposed project contributes to AOC delisting), scope of work, deliverables, schedule
and resource requirements.
Environment Canada
ix
Timeline
August 2010
Deliverable(s)
GLAP Work Plan Review Team established
October 2010
Establish GLAP work planning and
reporting template
Five-year work plans for all federal
departments and groups within
Environment Canada accessing GLAP
funds contributing to the remediation of
AOCs
Review and revision of GLAP Work Plan,
Annual Reports on Results, funding
allocation decision
November 2010
March–April
2011, 2012,
2013, 2015
Responsible Party
Director, Great Lakes
Division
Director, Great Lakes
Division
Director, Great Lakes
Division
Director, Great Lakes
Division; and
GLAP-funded groups
2. The RDG–Ontario should explore means of streamlining both the GLSF
funding approvals process and the GLSF application process. While mostly
satisfied with the GLSF program, proponents identified two issues with respect to
administration: timeliness of approvals and an overly complex application process.
Consideration should be given to scrutinizing the timing of the funding cycle to
ensure that funding approvals coincide with seasonal activities, and that the
application process is streamlined to promote efficiency (perhaps by exploring
adoption/applicability of common application forms/guidelines being developed as
part of the wider federal G&C reform exercise). Although most contribution
agreements are single-year only, the majority of projects are of an ongoing nature.
The use of multi-year agreements may be appropriate in some of these instances.
The RDG–Ontario agrees with this recommendation.
Environment Canada’s Action Plan to Reform the Administration of Grants and
Contributions will improve the efficiency of the GLSF application and approval process.
In addition, improvements have been made to the GLSF application and proposal
evaluation processes to simplify and improve the efficiency and consistency of the GLSF
application and approval process. Multi-year agreements will be used where appropriate.
Timeline
June 2010
Deliverable(s)
Adopt application and reporting processes
from the Departmental Action Plan to
Reform the Administration of Grants and
Contributions
Responsible Party
Director, Great Lakes
Division
3. The RDG–Ontario should explore ways to enhance the financial accountability
of the program. The direct transfer of funds to internal partners and absence of
program-specific coding of expenditures within the Department has weakened
financial accountability of the program. Diligent coding of expenditures at the
program level is a key element to ensure transferred funds are received and utilized
within Environment Canada by departmental partners to meet program goals and
priorities. The merit and potential disadvantages of direct transfers of program funds
Environment Canada
x
within Environment Canada to departmental partners for financial accountability
should be examined.
The RDG–Ontario agrees with this recommendation.
The Great Lakes Division will take steps to ensure that departmental resources
expended in the delivery of the GLAP are adequately documented. To this end, the
Great Lakes Division will take the necessary steps to adopt program-specific authority
codes for all Environment Canada internal salary, O&M and G&C expenditures of GLAP
funds. Furthermore, all GLAP funds will be managed by the Great Lakes Division and
distributed to other federal departments and Responsibility Centres within Environment
Canada in accordance with the GLAP five-year work plan annual updates.
Timeline
May 2010
(done)
May 2010
(done)
Deliverable(s)
Request for the establishment of a
GLAP-specific authority code in the
Financial Information System
Establish program-specific authority codes
for GLAP salary, O&M and G&C funds
Responsible Party
Director, Great Lakes
Division
Environment Canada
Finance Directorate
4. The RDG–Ontario should develop a more regular and robust reporting
approach for GLAP projects conducted by federal partners. Performance
reporting for projects conducted by federal partners was unspecified and sporadic. A
more robust reporting approach should be based on clear terms and conditions for
reporting on funds allocated to federal partners. Reporting should link
activities/deliverables to approved work plans, recognizing that, in some cases,
federal projects are undertaken as part of an integrated science program.
The RDG–Ontario agrees with this recommendation.
Environment Canada, Great Lakes Division, will require annual reports on all work plan
activities from Environment Canada and other federal partners participating in the
restoration of Great Lakes AOCs. Reporting information will be considered in the annual
review and revision of work plans. Clear guidelines will be established for monitoring and
reporting of GLAP funds by Environment Canada and federal partners. This will include
the establishment of a short reporting template, including financial reporting, and
pertinent performance information.
Timeline
October 2010
March–April
2011, 2012,
2013, 2015
Environment Canada
Deliverable(s)
Establish guidelines for monitoring and
reporting of GLAP funds; GLAP work
reporting template established
An annual report on the status of GLAP
work plan commitments will be prepared
and reviewed through the annual GLAP
work planning process
Responsible Party
Director, Great Lakes
Division
Director, Great Lakes
Division
xi
5. The RDG–Ontario should develop a performance measurement framework for
the program, and include a spectrum of more sensitive performance measures
(shorter- and longer-term). The program does not have a performance
measurement framework and is not currently well-positioned to demonstrate
progress and interim results because measures such as delisting AOCs are longterm. A more nuanced approach must balance the benefit of performance
measurement with the cost of monitoring and assessing BUIs in the AOCs. Recent
efforts by the program to document accomplishments and assign roles,
responsibilities and timelines for outstanding actions and priorities are acknowledged
as a strong foundation for this ongoing work.
The RDG–Ontario agrees with this recommendation.
A performance measurement framework will be developed that will consider feasible
measures to provide a more detailed and incremental assessment of progress toward
restoration of BUIs and delisting of AOCs. An Assessment of the Status of Remaining
Actions to Delist AOCs has been completed and will serve as an activity-based,
short-term performance measurement framework. A Beneficial Use Impairment Status
and Progress Report has also been completed and provides a longer-term, result-based,
performance measurement framework. The Assessment and Report will be reviewed,
revised and reported on biennially in alternating years.
Timeline
March 2011
February 2012,
February 2014
February 2011,
February 2013,
February 2015
Deliverable
Performance measurement framework
developed for the program
Updated Beneficial Use Impairment Status
and Progress Report
Updated Assessment of the Status of
Remaining Actions to Delist AOCs
Responsible
Party
Director, Great
Lakes Division
Director, Great
Lakes Division
Director, Great
Lakes Division
6. The RDG–Ontario should support the development of information management
tools to enhance implementation and monitoring of GLSF and federal projects.
No systematic process exists for monitoring and reporting the activities, outputs and
performance for GLAP-funded projects or the program overall. A Department-wide
Management Information System for Environment Canada’s community funding
programs, including the GLSF, is in development, although this system is not
expected to be implemented until 2012. Opportunities for interim tracking of
activities and project and partner contributions should be identified.
The RDG–Ontario agrees with this recommendation.
Environment Canada’s Action Plan to Reform the Administration of Grants and
Contributions (the Department’s G&C reform initiative) is developing an online
application and information management system for G&C programs. This system will
improve program efficiency, enhance alignment with departmental priorities and improve
the ability to report collectively on the results of departmental funding programs. The
Environment Canada
xii
system is scheduled to be in operation by 2011–2012. In the interim, the Great Lakes
Division has established an Excel-based system to track GLSF project proposals,
proposal reviews and selection, project recipients and contribution agreements, and
project financials, products and outcomes.
Timeline
2010
Deliverable
Adopt information management system from
the Departmental Action Plan to Reform the
Administration of Grants and Contributions.
Responsible
Party
Director, Great
Lakes Division
7. The RDG–Ontario should support the development of information-sharing
tools to facilitate broader access to the research generated by GLAP IV by the
scientific community and the public. There is no ongoing mechanism available for
technical transfer of research and results across the AOCs. This is an outstanding
commitment in Annex 4 of the COA.
The RDG–Ontario agrees with this recommendation.
Information on AOCs is provided through the Environment Canada website and other
communications products. Research findings are regularly published in scientific journals
and presented at scientific forums, including the Great Lakes State of the Lakes
Ecosystem Conference and the International Association for Great Lakes Research
Conference.
Additionally, AOC Progress Reports have been developed jointly with the Province of
Ontario for release in 2010. This tool will be built upon and modified to publicly report on
progress regarding remediation of Great Lakes AOCs, and to enhance sharing of
research results and other information. Reporting on AOCs will be implemented on a
three-year cycle, consistent with other Great Lakes reporting.
Timeline
Deliverable
2013
Area of Concern Progress Reports will be
issued every three years and will communicate
research results to the scientific community and
general public
Responsible Party
Director, Great Lakes
Division
8. The RDG–Ontario should continue efforts to define delisting criteria for AOCs
and clarify responsibilities with respect to delisting decision making. Roles and
responsibilities for aspects of delisting AOCs are a grey area and may be variable
across the AOCs. As more AOCs approach restoration, clarifying the steps to
delisting will increase in importance.
The RDG–Ontario agrees with this recommendation.
The GLWQA clearly states that the parties (Canada and the United States) are
responsible for leading all aspects of the AOC process “in cooperation with State and
Provincial Governments.”
Environment Canada
xiii
Environment Canada’s January 2010 Great Lakes RAP Workshop, which included
federal and provincial representatives involved in Great Lakes AOCs as well as the local
RAP coordinators, addressed the delisting process issue as an agenda item. The
principle outcome was a commitment to develop a Canada–Ontario Guide to the
Designation of Beneficial Use Impairments, Preparing Stage 3 Reports and Delisting
Areas of Concern. The guide, which is being developed collaboratively with the Ontario
Ministry of the Environment, will clarify the responsibilities with respect to delisting
decision making.
Timeline
Deliverable
December Canada–Ontario Guide to the Designation of
2010
Beneficial Use Impairments, Preparing Stage 3
Reports and Delisting Areas of Concern
Environment Canada
Responsible Party
Director, Great Lakes
Division
xiv
Audit and Evaluation Branch
Evaluation of the Great Lakes Action Plan IV
1.0 Introduction
In the 2009–2010 fiscal year, Environment Canada’s Audit and Evaluation Branch,
Evaluation Division, commissioned an evaluation of the Great Lakes Action Plan IV
(GLAP IV). This program was selected for evaluation prior to the end of the program’s
terms and conditions at the close of the 2009–2010 fiscal year.
This Final Report presents the findings of the evaluation. Chapter One provides
background information on GLAP IV. Chapter Two discusses the objectives of the
evaluation and the methodological approach. Chapter Three presents the evaluation’s
findings related to relevance, design and delivery, program outcomes, cost-efficiency
and cost-effectiveness. Conclusions are included in Chapter Four and recommendations
in Chapter Five.
1.1
Program Profile
The five Great Lakes—Superior, Michigan, Huron, Erie and Ontario—hold one fifth of the
freshwater on the Earth’s surface and 80 percent of the lake and river water in
North America. Approximately 30 per cent of Canadians live within the Great Lakes
Basin and 45 per cent of Canada’s industry is located in this area. Ensuring
environmental quality in this important region has implications for the natural
environment, the health and well-being of Canadians, and Canada’s economic
competitiveness.
The overarching mechanism for protecting and restoring the Great Lakes is the Great
Lakes Water Quality Agreement (GLWQA) (1972, 1978, 1987), which is a binational
agreement between Canada and the United States. The Agreement sets out a series of
commitments for both countries with respect to the Great Lakes. The International Joint
Commission4 (IJC) assists the parties (Canada and the United States) in the
implementation of the GLWQA. The most recent amendment to the Agreement was in
1987 and negotiations are under way for a renewed GLWQA. A key feature of the 1987
Agreement was the requirement that Canada and the United States take remedial action
in heavily degraded locations or “areas of concern” (AOCs) around the lakes.5 AOCs
were identified in 1985 by the IJC and relevant federal, provincial and state
4
The IJC was established in 1909 by the Boundary Waters Treaty between the United States
and Canada. The role of the IJC is described in detail in Article VII of the GLWQA.
5
The 1987 amendments also mandated the development and implementation of lakewide
management plans, and committed both countries to control pollution from non-point sources,
identify the nature and extent of sediment pollution, and develop methods to evaluate the
impact of contaminated sediments and the technological capabilities of programs to clean
them up.
Environment Canada
1
governments. In total, 17 AOCs6 were identified in Canada—twelve in Ontario and
another five (along connecting channels) shared by Canada and the United States.
Canada’s responsibility for managing and protecting the Great Lakes involves both
federal and provincial (Ontario) jurisdictions. Both levels of government share authority
to protect the environment, and are involved in aspects such as water, agriculture,
species and spaces, and fisheries. Coordination of federal and provincial efforts with
respect to the Great Lakes is accomplished through the Canada–Ontario Agreement
Respecting the Great Lakes Basin Ecosystem (COA). The goals and objectives of the
COA parallel those of the GLWQA, and thus the Agreement includes an Annex on
AOCs.
At the federal level, the Great Lakes Basin Ecosystem Initiative (GLBEI) is Environment
Canada’s mechanism for coordinating and delivering on federal commitments stemming
from the GLWQA and the COA with respect to the Great Lakes Basin ecosystem.
Remediation of domestic and binational Great Lakes AOCs is one of three priorities of
the GLBEI.7
The Great Lakes Action Plan for Areas of Concern is the primary vehicle under the
GLBEI for the federal government, with Environment Canada as the lead department, to
act to restore AOCs around the Great Lakes area and fulfill Canadian commitments
under the GLWQA. The current fourth phase8 of the Action Plan, GLAP IV, was
approved in 2005 with a budget of $40 million ($8 million per year over five years) to
restore, protect and conserve AOCs around the Great Lakes.
1.1.1
Program Goals
The GLWQA states that “ … the Parties, in cooperation with the State and Provincial
Governments and the Commission, shall identify and work toward the elimination of …
Areas of Concern.... Remedial Action Plans … shall embody a systematic and
comprehensive ecosystem approach to restoring and protecting beneficial uses in Areas
of Concern…. The Parties shall cooperate with State and Provincial Governments to
6
Two of these AOCs have since been delisted: Collingwood Harbour in 1994 and Severn Sound
in 2003. Although officially announced on April 16, 2010, the delisting of Wheatley Harbour
was not factored into the analysis of outcomes achievement, as it occurred outside the
time frame for GLAP IV, which ended on March 31 2010, and subsequent to data collection
and analysis for the present evaluation.
7
Development and implementation of binational lake-wide management plans for the
restoration, conservation and protection of the ecosystems of each of the five Great Lakes and
Lake St. Clair, and implementation of the Great Lakes Binational Toxics Strategy (which deals
with the development and implementation of challenge goals for the reduction of priority
persistent toxic substances within the Great Lakes Basin), are the other two priorities of the
GLBEI (Treasury Board Secretariat 2008 Strategic Review of GLBEI).
8
The federal government has provided GLAP funding since 1989 ($125 million over 5 years in
1989; $150 million over 6 years in 1994; $40 million over 5 years in 2000; and $40 million over
5 years in 2005). Identification, evaluation and remediation of AOCs have been a major
element of GLAP funding since 1989 and the single focus of GLAP funding since 2000.
Environment Canada
2
ensure that Remedial Action Plans are developed and implemented for Areas of
Concern.”
The goals of GLAP IV are:
›
›
to make progress on federal9 actions in eight AOCs: St. Mary’s River,
St. Clair River, Toronto and Region, Hamilton Harbour, Jackfish Bay,
Detroit River, Niagara River and Port Hope; and
to complete federal actions in seven AOCs: Thunder Bay, Nipigon Bay,
Peninsula Harbour, Bay of Quinte, Wheatley Harbour, St. Lawrence River
and Spanish River.
To achieve these goals, GLAP IV is based on an ecosystem approach: remedial efforts
targeted to AOCs under GLAP IV deal with interrelated environmental and sustainable
development issues. As laid out in the GLWQA, remediation in each AOC is guided by
Remedial Action Plans (RAPs). The original RAP reports were developed in 1987 by the
federal and Ontario governments, with extensive public consultation. RAP teams with
scientific/technical expertise for each AOC identified the nature and causes of
environmental problems, and recommended actions, implementation plans and
monitoring strategies. RAP reports are issued and updated for three stages of
implementation: Stage 1 – Problem definition; Stage 2 – Selection of remedial
measures; and Stage 3 – Restoration of beneficial uses.
The definition and characterization of the environmental problems in the AOCs is
founded on identification of the beneficial uses that are impaired, the degree of
impairment and the geographic extent of such impairment. Beneficial use is described as
“the ability of living organisms to use the ecosystem without adverse consequence.”
There are 14 beneficial use impairments (BUIs) that are assessed:
1.
2.
3.
4.
5.
6.
Restrictions on fish consumption
Tainting of fish and wildlife flavour
Degradation of fish and wildlife populations
Fish tumours or other deformities
Bird (or other animal) deformities or reproduction problems
Degradation of benthos (organisms living in or near marine sediment
environments)
7. Restrictions on dredging activities
8. Eutrophication or undesirable algae
9. Restrictions on drinking water consumption, or taste and odour problems
10. Beach closings / water contact-sports restrictions
11. Degradation of aesthetics
12. Added costs to agriculture or industry
13. Degradation of phytoplankton and zooplankton populations
14. Loss of fish and wildlife habitat
9
Federal actions reference the Government of Canada’s actions in response to its commitments
under the COA (as opposed to Ontario’s). These vary by AOC.
Environment Canada
3
Prior to the renewal of the GLAP in 2005, a total of 98 BUIs were identified across the 15
Canadian AOCs, and 14 additional beneficial uses required further evaluation. Using this
information, RAP reports were updated and the priority work required in each AOC for
delisting was identified. Removal of the designation as an AOC and monitoring of
recovery indicates that the identified beneficial uses are no longer impaired and the area
is restored. An Area in Recovery is a designation used to indicate that all remedial
actions have been taken and the area must enter a period of natural recovery.
1.1.2
Program Activities
Upon renewal of the GLAP in 2005, Environment Canada, with federal partners,
undertook a work planning and priority-setting effort. The federal partners that
participated were the National Water Research Institute (NWRI) of Environment Canada,
the former Environmental Conservation Branch (ECB),10 the Department of Fisheries
and Oceans (DFO), and Public Works and Government Services Canada (PWGSC).
The recommendations of the GLAP Work Plan Review Team were approved by the
federal Great Lakes Executive Committee (GLEC) and signed off by the Environment
Canada GLAP program director. GLAP IV funding was distributed to Environment
Canada, DFO and PWGSC in April 2006.
›
Priority Investments
Three broad categories of priority investments under GLAP IV within AOCs are: :
◊ Remedial actions – Remedial actions within AOCs, consistent
with the mandate of the Department, typically administered
through the Great Lakes Sustainability Fund (GLSF):
¤ contaminated sediment assessment activities, such
as decisions on sediment management measures,
and completion of contaminated sediment risk
management strategies or engineering studies;
¤ habitat restoration activities, such as supporting
habitat creation projects, supporting stewardship
and habitat acquisition, and habitat identification,
restoration and protection;
¤ support to municipal infrastructure improvements,
including pre-engineering design studies for
10
When it existed, the ECB included a number of divisions, four of which received GLAP IV
funding: EHD (Ecosystem Health Division), RPD (Restoration Programs Division), GLSFD
(Great Lakes Sustainability Fund Division), and CSD (Conservation Strategies Division).
During departmental transformation, the divisions in the former ECB were restructured as
follows: EHD went to Science and Technology Branch; part of CSD went to the Wildlife and
Landscape Science Division within the Science and Technology Branch, while the other part of
CSD became the Canadian Wildlife Service–Ontario Region (CWS–ON) within the
Environmental Stewardship Branch; and both RPD and GLSFD went to the Regional Director
General’s Office, and subsequently were restructured into the Great Lakes Management and
Reporting Section and the Great Lakes Areas of Concern Section.
Environment Canada
4
infrastructure upgrades, actions to secure partner
funding and municipal commitment, and evaluation
of cost-effective technologies for municipal
wastewater treatment.
◊ Science – Assessment and monitoring of BUIs:
¤
assessments of ecosystem health based on the 14
BUIs identified in the GLWQA, to determine the
need and effectiveness of remedial actions and
eventually to confirm restoration of ecosystem
health and therefore support official delisting of the
AOC.
◊ Engagement/Governance – Cooperative development and
updating of RAPs with partners, consultation with the public:
¤ management of federal programs in AOCs and
coordination with Ontario and local governments,
the private sector, non-governmental organizations
(NGOs) and individuals engaged in the remediation
of AOCs.
Excluding GLSF projects, approximately 40 federal science and monitoring projects
were recommended for implementation over the life of the program. Table 1.1 displays
these projects by AOC priority grouping and Table 1.2 by investment priority. This
information is drawn from the recommended work plans for federal partners; information
on actual project expenditures is not known, due to gaps in financial tracking of this
information within Environment Canada (discussed in section 3.2.4).
Table 1.1: Federal Partner Projects Recommended for Funding: Priority AOC
Grouping*
Environment Canada –
ECB
Environment Canada –
NWRI
PWGSC
DFO
Total
Group 1 Priority AOCs
11
Group 2 Priority AOCs
11
Total**
22
6
5
11
3
8
28
3
14
33
6
22
61***
Source: Environment Canada. GLAP IV Total Work Plan, August 21, 2008.xls. Internal document.
* The priority groupings are based on those identified by the internal Review Committee recommendations and not
those identified in the GLAP IV program documentation. The only difference is the Review Committee group 1 AOCs
includes the St. Clair River and excludes the Spanish River.
** Spanish Harbour (in recovery) and Port Hope were not identified as priority (group 1 or 2) AOCs, and so are not
represented in Table 1.1. None of the GLAP IV projects led by other government departments (OGDs) and
Environment Canada branches between 2005 and 2009 targeted these two AOCs. Environment Canada’s Science and
Technology Branch studied Spanish Harbour in 2005 and 2009, and the Department’s Water Quality Monitoring and
Surveillance group conducted water quality monitoring there in 2008.
*** Projects can address multiple AOCs falling into both of the priority groupings.
Environment Canada
5
Table 1.2: Number and Value of Federal GLAP IV Projects Recommended for
Funding by Priority Area and Partner
GLAP IV
Priority Area
Habitat
Restoration
BUI
Assessments
Monitoring
Municipal
Infrastructure
Contaminated
Sediment
Assessment
and
Remediation
Coordinating
Actions
among
Partners
Total
NWRI
No. of
Approved
Projects
Funding
No. of
Projects
ECB
Approved
Funding
No. of
Projects
DFO
Approved
Funding
PWGSC
No. of
Approved
Projects
Funding
--
--
1
$7,169,000
2
$220,000
--
--
$7,389,000
4
$2,665,000
3
$3,375,000
14
$4,223,000
--
--
$10,263,000
1
$250,000
1
$604,000
--
--
--
--
$854,000
2
$422,000
1
$5,080,000
--
--
--
--
$5,502,000
1
$150,000
1
$12,871,000
--
--
4
$650,000
$13,671,000
--
--
2
$1,626,000
--
--
--
--
$1,626,000
8
$3,487,000
9
$30,725,000
16
$4,443,000
4
$650,000
$39,305,000
Source: Environment Canada. GLAP IV Total Work Plan, August 21, 2008.xls. Internal document.
›
Great Lakes Sustainability Fund (GLSF)
A component of GLAP IV, the GLSF is a contribution funding mechanism to foster
partnerships with other agencies and local community stakeholders to advance the goals
of GLAP IV. Projects carried out by DFO and PWGSC were also funded through the
GLSF.
The GLSF provides technical and financial support (up to one third of the total cost) to
projects that implement remedial actions to support cleanup and restoration in four key
priority areas: fish and wildlife habitat rehabilitation and stewardship, contaminated
sediment assessment and remediation, innovative approaches to improve municipal
wastewater effluent quality, and the elimination of non-point source pollution from
agricultural sources.
GLSF funding priority is given to those submissions undertaking remedial actions
identified in the COA and in RAPs as federal government responsibilities, as well as
science and monitoring activities essential to supporting the design and evaluation of
these actions.11 First priority for funding is given to those AOCs with the greatest
potential for delisting in the short to medium term (group 1 AOCs as defined in the COA),
11
Total
The COA assigns federal or provincial leadership for each AOC. According to the COA Annex:
a) Canada and Ontario will co-lead the RAP process in the Toronto and Region, St. Mary’s
River, St. Clair River, and Detroit River AOCs; b) Canada will lead the RAP process in the
Thunder Bay, Hamilton Harbour, Port Hope and St. Lawrence River AOCs; and c) Ontario will
lead the RAP process in the Nipigon Bay, Jackfish Bay, Peninsula Harbour, Spanish Harbour,
Wheatley Harbour, Niagara River and Bay of Quinte AOCs.
Environment Canada
6
and second priority is assigned to AOCs with less potential for delisting (group 2
AOCs).12
The GLSF uses a directed approach in soliciting project proposals: all proposals must
link closely with intended goals and results listed in the COA, which in turn identify work
required to delist AOCs. The proposal assessment criteria further include:
›
›
›
›
›
›
›
›
›
›
the need for federal participation through legislated mandate, existing federal
policy or declared federal interest (the GLSF does not support funding for
capital or operating costs of municipal infrastructure);
advances the completion of federal actions toward the recovery and delisting
of AOCs;
the benefit that a project has towards specific measurable achievements;
environmental impact of a project (e.g., results of environmental
assessments);
project’s actual or potential technical merits;
application of the “polluter pays” principle, and two-thirds funding provided by
other financial supporters;
input and endorsement by review agencies and peers, including RAP
Implementation Teams where projects support remedial measures;
opportunities for technology transfer to other AOCs and the Great Lakes
ecosystem;
project’s ability to promote innovation and support new technologies that feed
economic growth, create new opportunities and provide long-term
improvements; and
achievement of environmental benefits at the lowest possible cost.
GLSF proposals are submitted to a multi-disciplinary review, which can include
subject-matter experts and AOC committee or RAP management team members, to
provide external opinion on the proposed project in the form of a technical review of the
proposed project and its alignment with regional priorities. A total of 267 GLSF
contribution agreements worth nearly $16 million have been supported through GLAP IV
since 2005, with the majority (n=158) targeting group 2 AOCs (those with less potential
for delisting)—an activity focus that appears to be at odds with the stated intention of the
program to focus on group 1 AOCs.
12
In the 2007–2010 Canada–Ontario Agreement, Group 1 AOCs are Nipigon Bay, Jackfish Bay,
Wheatley Harbour and St. Lawrence River (Cornwall).
Environment Canada
7
Table 1.3: Value and Number of GLSF Contribution Agreements, 2005–2010
Group 1
Number
Value
< $25,000
$25,000–
$74,900
$75,000+
Total
1.1.3
Group 2
Number
Value
Group 1 & 2
(multiple AOCs) / No
group identified
Number
Value
Number
Total
Value
30
$363,732
49
$706,017
7
$54,003
86
$1,137,062
33
32
95
$1,431,265
$3,953,542
$5,748,539
71
38
158
$3,148,331
$5,809,144
$9,663,492
5
2
14
$125,390
$207,000
$424,703
109
72
267
$4,704,986
$9,969,686
$15,836,734
Stakeholders and Recipients
The achievement of GLAP IV program objectives depends on the engagement of a
variety of stakeholders. As indicated above, Environment Canada is the lead department
for GLAP IV, and the program is the responsibility of the Strategic Integration and
Partnerships Division in the Ontario Region. Scientific support for GLAP IV is provided
by internal Environment Canada partners, particularly the NWRI, the Canadian Wildlife
Service and Environmental Protection Operations Directorate within the Environmental
Stewardship Branch, and the Water Science and Technology Directorate within the
Science and Technology Branch. Although Environment Canada has primary
responsibility for the delivery of GLAP IV, the program nonetheless relies on the
technical and scientific expertise of partner OGDs (other government departments) to
undertake key activities for the remediation of AOCs and to provide guidance (through
their participation on various oversight committees, such as GLEC, etc.) on priorities for
the remediation of these areas. Federal partners include DFO, Health Canada,
Agriculture and Agri-Food Canada, Transport Canada, Parks Canada Agency, Natural
Resources Canada and PWGSC.13
Other government jurisdictions are engaged through formal agreements: Canada and
the United States are signatories to the GLWQA, and six federal departments and three
provincial ministries are signatories to the COA.14
Partnerships are a requirement of GLSF projects. GLSF partners include provincial and
municipal governments, Conservation Authorities, NGOs, First Nations, educational
institutions, local community volunteers, agriculture, industrial and other business
sectors, and academia. These organizations may be proponents of GLSF-funded
projects or act as partners in GLSF projects (such as in a funding, advisory or
implementation role).
13
Some of these departments were invited to submit proposals for GLAP IV but declined, and
were therefore not represented on the internal Review Committee. However, all OGDs listed
here are members of the GLEC.
14
Federal partners include Environment Canada, Parks Canada Agency (not a separate
signatory, as the EC Minister is also responsible for Parks Canada), Agriculture and Agri-Food
Canada, Fisheries and Oceans Canada, Health Canada, Natural Resources Canada, and
Transport Canada. Provincial signatories to the agreement include the of the Environment, the
Ministry of Natural Resources and the Ministry of Agriculture, Food and Rural Affairs.
Environment Canada
8
Finally, the development and implementation of RAPs involves various partners, similar
in type to those involved in GLSF projects. Local implementation teams or councils
typically include federal representatives (e.g., DFO, Environment Canada), other
government representatives (Ontario Ministry of the Environment [MOE] or Ministry of
Natural Resources [MNR], municipal/regional, First Nations), Conservation Authorities,
community groups and NGOs.
1.1.4
Governance
The Great Lakes Areas of Concern Section, Strategic Integration and Partnerships
Division within Environment Canada, is responsible for the management and
coordination of GLAP IV.15 While the program is managed in the Ontario Region under
the Regional Director General, it is functionally accountable to the Department’s
Assistant Deputy Minister, Environmental Stewardship Branch. Further management
oversight of GLAP IV is provided by the Great Lakes Environment Office, which is also
within the Strategic Integration and Partnerships Division.
Several horizontal, federal/provincial and binational committees oversee and manage
GLAP IV (described in more detail in Annex A under separate cover):
›
›
›
›
›
›
15
GLEC: ensures Canada’s commitments under the GLWQA are met through
the effective and efficient delivery of GLAP IV.
GLAP Workplan Review Team: annual review of five-year work plans
submitted by federal departments, in consultation with the GLEC.
COA Management Committee: responsible for the oversight and overall
administration of the COA, it helps to coordinate federal/provincial work
carried out in AOCs (as per Annex 1 of the COA) and to provide direction and
decision making for AOC work.
COA Annex Implementation Committee: coordinates development and
implementation of the work planning process across multiple agencies.
Great Lakes Binational Executive Committee: coordinates binational
programs and activities (including in five shared AOCs targeted by GLAP IV).
RAP committees: maintain working-level links to community and provincial
stakeholders and ensure that environmental needs at the AOC level are
addressed. Each RAP committee has a federal project lead who reports RAP
activities directly to the COA Annex Implementation Committee.
The Areas of Concern Section, Great Lakes Environment Office, and Great Lakes
Management and Reporting Section now report through the Great Lakes Division.
Environment Canada
9
1.1.5
Resources
Budget 2005 provided federal funding of $40 million ($8 million per year over five years)
for GLAP IV. Table 1.4 describes the original allocation of these funds across fiscal
years.
Table 1.4: Allocation of GLAP IV Funds ($000s)
Salary1
Employee
Benefit Plan
Operating
Total
2005–06
659
2006–07
659
2007–08
659
2008–09
659
2009–10
659
Total
3,295
132
132
132
132
132
660
7,209
8,000
7,209
8,000
7,209
8,000
7,209
8,000
7,209
8,000
36,045
40,000
1
Most full-time equivalent staff are situated in a Burlington, Ontario facility owned by Environment
Canada.
Table 1.5 summarizes the allocation by organization. Funds were transferred from
Environment Canada to relevant departments through interdepartmental settlements on
the basis of approved work plans, while the work plans prepared for Environment
Canada projects specified that GLAP IV resources would be allocated directly to these
organizations. Information on actual GLAP IV expenditures for the remaining years of
the program is not available due to gaps in financial reporting of expenditures at
Environment Canada.
Table 1.5: Summary of Work Plans – Recommended Funding ($000s)
Annual Recommended
Funding
Total Recommended
Funding
2005–10
573
2,866
EC-ECB
1,193
9,568
EC-GLSF
3,784
18,918
EC-NWRI
1,730
8,648
Total
8,000
40,000
Organization
DFO
1.1.6
Program Logic Model
A logic model is a visual representation of a program/initiative that identifies the linkages
between an initiative’s activities and the achievement of its outcomes. Figure 1 shows
the program logic model, which presents a graphical depiction of how the activities and
outputs of GLAP IV relate to immediate, intermediate, long-term and ultimate outcomes.
Environment Canada
10
Audit and Evaluation Branch
Evaluation of the Great Lakes Action Plan IV
Figure 1: Logic Model – GLAP IV
Environment Canada
11
Audit and Evaluation Branch
1.1.7
Evaluation of the Great Lakes Action Plan IV
Performance Reporting and Evaluations
There are a number of reporting exercises related to federal Great Lakes ecosystems
programs:
›
GLWQA: The parties to the agreement (Canada and the United States) report
biennially to the IJC on progress in developing and implementing RAPs. The
Commission reports to federal, state and provincial governments every two
years on progress in relation to achievement of GLWQA goals and
objectives, and provides ongoing assessment and advice to governments
through semi-annual meetings and special reports. RAPs for each AOC are
also submitted to the Commission for review and comment at three stages:
◊ Stage 1 RAPs – upon completion of definition of the
problem and definition of the causes of the use impairment
(including a description of all involved known sources of
pollutants and an evaluation of other possible sources);
◊ Stage 2 RAPs – submitted when remedial and regulatory
measures are selected, and are to include a schedule for
their implementation, as well as identification of the
persons or agencies responsible for remedial measures
implementation; and
◊ Stage 3 RAPs – submitted when monitoring indicates that
identified beneficial uses have been restored.
›
COA: A multi-year work plan is developed and updated annually for the COA,
based on input from all the participating federal departments and provincial
ministries that are parties to the COA. Reporting on activities and
achievements in the work plans is analyzed and evaluated in order to ensure
that key contributions are being made by participating agencies to enable
achievement of COA commitments.
While the GLAP programs themselves have not been the subject of a program
evaluation, other performance reporting initiatives have included:
›
›
›
a mid-term review of GLAP IV;
a review of the GLBEI by the Commissioner of the Environment and
Sustainable Development (CESD) completed in 2008 (a follow-up to the 2001
CESD audit); and
participation of the GLBEI in a Treasury Board Management Accountability
Framework (TB MAF) exercise.
Environment Canada
12
Audit and Evaluation Branch
Evaluation of the Great Lakes Action Plan IV
2.0 Evaluation Design
2.1
Objectives and Scope
The evaluation covers GLAP IV program activities from the 2005–2006 to 2009–2010
fiscal years. The evaluation findings will be used to assist the Great Lakes AOC program
and other departmental stakeholders in the ongoing management of GLAP IV, and will
contribute to efforts (beginning in February 2010) related to the renewal of the program
for another five-year period. The evaluation issues include:
1. Is there a continued need for a GLAP?
2. Is the GLAP IV aligned with federal government priorities?
3. Is the GLAP IV consistent with federal roles and responsibilities?
4. To what extent have intended outcomes been achieved as a result of
GLAP IV?
5. Is the GLAP IV design appropriate for achieving expected program results?
6. Have there been any unintended (positive or negative) outcomes?
7. Is GLAP IV undertaking activities and delivering products in the most efficient
manner?
◊ How could the efficiency of the program’s activities be
improved?
◊ Are there alternative, more efficient ways of achieving the
program’s objectives?
8. Is GLAP IV achieving its intended outcomes in the most economical manner?
The complete matrix of evaluation questions, indicators and data sources is presented in
Annex B (under separate cover).
2.2
Approach and Methodology
Multiple lines of evidence were used to increase the reliability and robustness of the
analysis. The evaluation included a review of documentation and literature, a review of
GLSF and federal GLAP IV project files, and key informant interviews. Data collection
occurred between February 3 and March 5, 2010.
Environment Canada
13
2.2.1
Document and Literature Review
Secondary documentary sources were reviewed to develop a detailed program profile
and contribute to addressing several of the evaluation questions, including relevance
and program performance (achievement of program outcomes and the cost-efficiency
analysis in particular). Environment Canada provided documents such as:
background/planning and scoping materials; performance documents (e.g., Evaluation
Plan for the Great Lakes Action Plan, Mid-term Review of GLAP IV); corporate/policy
documents; reports on the Great Lakes (e.g., IJC reports); program work plans /
progress reports; and other material. A document review template was developed to
summarize findings in the documents pertaining to the evaluation questions. A
bibliography of key documents is included in Annex C (under separate cover).
2.2.2
File Review
A review was conducted of a sample comprising 39 of a total of 234 files for contribution
projects funded between 2005–06 and 2008–09 by the GLSF program component.16 The
sampling strategy was devised by Environment Canada, taking into consideration fiscal
year, AOC priority group, and project value strata17 so that the sample reflected the
population. Annex D (under separate cover) provides a profile of the sample and
population of GLSF files.
A data collection template was used to capture file information in a consistent manner,
and to ensure that the content of the files was well-documented and linked to the specific
evaluation questions and indicators (see Annex E under separate cover). The file review
addressed issues related to the consistency of contributions to departmental objectives,
the achievement of outputs and outcomes, unintended impacts, and project costs and
leveraging. A profile of the characteristics of the files that were reviewed is included in
Annex E (under separate cover).
The GLSF files were generally well documented, though there were some gaps.
Contents typically included the project proposal (92 per cent of files), memorandum of
understanding (MOU) or contribution agreement (90 per cent of files), financial
information or records (e.g., invoices; 77 percent of files) and proposal review / technical
review information (69 per cent of files). Almost 80 per cent of files had some output or
outcome reporting (i.e., an annual report, progress report or final report). It is not clear
why some files do not have output or outcome reporting. Table 2.1 indicates the extent
to which key documents were included in the files that were reviewed.
16
GLSF files from 2009–2010 were not sampled due to the recency of their approval and the
limited reporting available for these projects during the evaluation data collection period.
17
A homogeneous subgroup of members of the population.
Environment Canada
14
Table 2.1: Distribution of Documents in GLSF Files Reviewed
Type of document
Proposal
MOU / Contribution Agreement
Financial files/records
Proposal review/approvals
Final report
Correspondence
Technical screening
Progress reporting / activity reporting
Annual report
Meeting/committee agendas/minutes
Audit/evaluation reporting
Other products
Other (e.g., brochures, media releases, special
technical reviews, photographs, CDs)
Per cent of files
92%
90%
77%
69%
62%
56%
31%
28%
18%
8%
5%
26%
49%
Files for 21 projects sampled from those conducted by federal partners were also
requested. All funded partners, including DFO, the former ECB, NWRI and PWGSC
provided project-related documentation. However, the documentation was in various
formats (e.g., conference presentations, reports, scientific publications,
invoices/contribution agreements) and reporting was not always linked to a single
project, thus was not amenable for capture using a standardized file review
methodology. As a result, descriptive analyses were undertaken to summarize the
contents of the federal files (see Annex F under separate cover). To better understand
the reporting requirements and project-level documentation for federal projects, the key
informant interview guides for federal partners and federal project proponents were
expanded to address this issue.
2.2.3
Key Informant Interviews
In total, 46 key informant interviews were completed with Environment Canada
personnel, committee members, other government representatives, project proponents
(funded and unfunded) and experts. The targeted distribution of interviews was met in all
cases and is as follows:
›
›
›
›
›
›
Departmental program managers and federal partners involved in the
management and delivery of GLAP IV (n=11);
Representatives from federal OGDs and Ontario who are members of
committees (GLEC, GLAP Workplan Review Team, COA Management
Committee, and COA Annex Implementation Committee (n=10);
Federal project proponents (n=4);
GLSF project proponents (n=10);
Unsuccessful applicants (n=4);
RAP committee members (n=4); and
Environment Canada
15
›
International or academic experts on aquatic ecosystem remediation (n=3).
The interviews were conducted by phone and the duration of the interviews varied by
respondent group, ranging from 30 to 90 minutes. Key informants were asked their views
and opinions on a number of issues, including the continued need and relevance of the
program, its design and delivery, the extent to which the program is meeting immediate,
intermediate and longer-term outcomes, and the efficiency and economy of the program
in achieving outputs and results. A master key informant interview guide is included in
Annex G (under separate cover), from which individual interview guides were further
tailored to be suitable for the various respondent groups.
To ensure a common understanding of the terms used in the analysis and reporting of
interview results, the following guidelines have been used:
›
›
›
›
›
2.3
“A few / a small number of interviewees” = less than 25 per cent
“Some / a minority of interviewees” = 25–49 per cent
“A majority of interviewees” = 50–75 per cent
“Most interviewees” = over 75 per cent
“Almost all interviewees” = 95 per cent or more
Challenges and Limitations
A number of methodological caveats should be noted for this evaluation. First, much of
the information that was gathered for this review is drawn from internal sources: program
documentation as well as interviews with program managers or federal partners,
committee members and project proponents. With respect to program documentation,
the main disadvantage is that these secondary sources often reflect stated intentions of
a program (not implementation) and are not necessarily produced for the explicit
purposes of the evaluation, thus are often not organized to present, or do not touch
upon, issues related to those investigated through an evaluation. With respect to the
interviews, respondents frequently had some prior connection to the program (e.g.,
unfunded applicants who were funded for another project, or experts who worked at
Environment Canada in the past). Although a small number of interviews were
conducted with unfunded GLSF applicants and experts, and priority was given to
interviewing key informants who had not been awarded GLSF funding, the interview
feedback benefits from respondents’ knowledge of the program but lacks a high level of
objectivity.
Second, the availability of program activity information and financial data with respect to
the GLAP IV is limited. Specifically:
Environment Canada
16
›
›
›
Due to the departmental transformation in 2005–2006 and unbundling of
resources and activities of the GLBEI,18 it is not possible to track Environment
Canada departmental GLAP IV expenditures. Expenditures were being
tracked at the ecosystem initiative level and not coded at the program level.
This gap limits the cost-efficiency analysis conducted for the evaluation.
The GLSF program database is out-of-date with respect to entries after
2006–2007, and key measures such as financial expenditures and leveraged
funding from partners and program activities are either not captured or not
captured systematically year-to-year. This was due to a shift from the use of
MOUs to a grants and contributions (G&C) funding mechanism (with different
reporting templates and a centralized information management system), as
well as the internal re-organization of the unit.
There is no centralized database for performance measurement purposes for
the program.
As a result, it was not possible to provide a complete picture of program activities and
outputs. As well, the cost-efficiency analysis for the program was limited by the lack of
clear financial information, though this analysis was supplemented by qualitative
assessments of cost-efficiency obtained in the key informant interviews.
Finally, it is important to note that the determination of progress in AOCs and the
achievement of program outcomes, particularly related to restoration and delisting, are
only realizable in the longer term. The five-year term of GLAP IV and, indeed, the 20
years of GLAP funding are widely perceived to be insufficient to address the complex
ecological issues in the AOCs. In addition, the achievement of longer-term GLAP IV
program outcomes (e.g., delisting) requires the significant contribution of other partners
and stakeholders. External factors, such as the availability of funding for infrastructure
projects, are critical to the achievement of program outcomes. Thus, evaluation findings
related to longer-term outcomes of GLAP IV must be considered in this context.
2.4
Reporting Note
In the next chapter, findings are presented for the following five evaluation issues
explored through the evaluation: relevance, design and delivery, achievement of
18
The process of transformation was designed to enable the Department to plan, manage and
report by results. Transformation involved re-defining the results structure (Program Activity
Architecture) and new management structures and processes to promote integrated
management and decision making in the context of a clearer view of departmental results and
strategic direction (Environment Canada, Report on Plans and Priorities, 2006–2007).
Environment Canada
17
program outcomes, efficiency, and economy.19 A rating is also provided for each
evaluation question. The ratings are based on a judgement of whether the findings
indicate that:
›
›
›
›
the intended outcomes or goals have been achieved or met (labelled as
Achieved);
considerable progress has been made to meet the intended outcomes or
goals, but attention is still needed (labelled as Progress Made, Attention
Needed);
little progress has been made to meet the intended outcome and attention is
needed on a priority basis (labelled as Little Progress, Priority for Attention);
and
a rating is not applicable (identified by the N/A symbol).
In addition, a tilde symbol (~) is used to denote instances where outcome achievement
ratings are based solely on subjective evidence. A summary of ratings for the evaluation
issues and questions is presented in Annex H (under separate cover).
19
Treasury Board policy requires that all evaluations of federal programs and initiatives address
the core issues of relevance and performance. Performance is a blended construct that
includes a program’s effectiveness (achievement of outcomes), efficiency (achievement of
outputs/activities at the lowest cost) and economy (achievement of outcomes at the lowest
cost).
Environment Canada
18
Audit and Evaluation Branch
Evaluation of the Great Lakes Action Plan IV
3.0 Findings
3.1
Relevance
This section presents the evaluation findings on the continued need for GLAP IV, the
alignment of the program with federal and departmental priorities, and the extent to
which GLAP IV is consistent with federal roles and responsibilities.
Overall Findings
The GLAP remains a relevant program that enables the federal government to address
the continued need for restoration and maintenance of the Great Lakes AOCs. While
recent trends in Great Lakes’ ecosystem conditions are variable, historical sources of
stress and new challenges are generating negative impacts in many areas of the lakes.
Environmental need, together with the social and economic benefits of the Great Lakes,
supports the continued relevance of the program. Public opinion is consistent with
scientific information and key informant views on the environmental and societal
importance of the Great Lakes and the continued need for the GLAP.
There is little redundancy risk associated with GLAP IV. The program uniquely
addresses federal commitments in the GLWQA and the COA by providing a framework
to address BUIs linked to AOC RAP priority areas. Unlike other public funding (federal,
provincial and municipal) and non-public funding (industry, foundations,
non-governmental organizations, etc.), GLAP IV is targeted exclusively to AOCs. Key
informants express few concerns about the possibility of duplication or overlap between
GLAP IV and other programming. Rather, GLAP IV is seen as highly complementary
and a catalyst in orienting other funding sources to provide support in the AOCs.
GLAP IV is consistent with federal and departmental priorities. The program is part of the
GLBEI, one of six ecosystem initiatives at Environment Canada under the Priority
Ecosystem Initiatives Sub-Activity. The GLBEI supports the Department’s strategic
outcome that “Canada’s natural capital is restored, conserved and enhanced.” GLAP IV
outcomes and priority investments also support a number of other sub-activities within
Environment Canada’s program activity architecture, including Aquatic Ecosystems,
Wildlife, Priority Ecosystems, Ecosystems Sustainability, and Assessment and
Ecological Monitoring.
Environment Canada’s GLAP IV is aligned to support federal government commitments
and obligations under the GLWQA and the COA, as well as federal government priorities
outlined in the 2007 Speech from the Throne and two recent federal budgets. Other
legislative authorities include the Canada Water Act, the International Boundary Waters
Treaty Act, and the Canadian Environmental Protection Act, 1999.
Environment Canada
19
3.1.1
Continued Need
Evaluation Issue
Is there a continued need
for a GLAP?
Indicator(s)
›
›
›
Demonstration of societal/
environmental need
Presence/absence of other
programs that complement or
duplicate the objectives of
the program
Methods
›
›
›
Document review
Rating
Achieved
File review
Key informant
interviews
Reach and activities are
connected to
societal/environmental needs
Connection to Environmental/Societal Need
For over four decades, the GLWQA and COA have served as important and effective
mechanisms for protecting and restoring the Great Lakes. Canadian and United States
actions implemented under the GLWQA have resulted in the lowering of phosphorus
loads to the Great Lakes, and improvements in several water quality indicators, in
particular in the more heavily nutrient-impacted lower lakes, as indicated in the State of
the Lakes Ecosystem Conference’s State of the Great Lakes 2009 report. At the same
time, a 2007 review of the GLWQA by the IJC found that stronger linkages are needed
between the Agreement’s stated overall purpose—to restore and maintain the chemical,
physical and biological integrity of the waters of the Great Lakes Basin ecosystem—and
the measures outlined in the Agreement’s articles and annexes. The review further
emphasized that the Agreement should be revised to address today’s pressing issues,
including the impacts of climate change, aquatic invasive species and urbanization.
There was also recognition that these issues affect biodiversity. Accordingly, in
June 2009 the federal governments of Canada and the United States announced a
commitment to strengthen and modernize the Agreement so as to better address
pollution, invasive species and climate change.20
Recent trends of Great Lakes ecosystem conditions vary: some conditions are improving
and some are deteriorating.21 Efforts under the GLWQA directed at degradation and
contamination in AOCs through the implementation of RAPs and lake-wide management
plans have coordinated activities directed at addressing numerous BUIs. Yet, despite the
progress made to date, the Great Lakes are exhibiting symptoms of extreme stress from
a combination of sources, including toxic contaminants, invasive species, nutrient
loading, shoreline and upland land use changes, and hydrologic modifications. In large
areas of the lakes, historical sources of stress have combined with new ones to reach
tipping points, at which ecosystem-level changes occur rapidly and unexpectedly,
20
See Budget 2010, p.106 (www.budget.gc.ca/2010/pdf/budget-planbudgetaire-eng.pdf).
21
Environment Canada and U.S. Environmental Protection Agency. 2009. State of the Great
Lakes 2009, Highlights.
National Wildlife Federation. 2005. Prescription for Great Lakes Ecosystem Protection and
Restoration – Avoiding the Tipping Point of Irreversible Changes.
Environment Canada
20
confounding traditional relationships between sources of stress and expected ecosystem
responses. In a 2008 TB MAF review of the GLBEI, an examination of the research
concluded that, despite significant progress and many successes in addressing past
challenges, the Great Lakes continue to be at risk of unprecedented changes due to a
range of stresses acting in combination, including: degradation of the nearshore zone;
continued introduction and spread of aquatic invasive species; significant changes to the
lower food web; wetland and other natural habitat loss; chemical contamination; threats
to drinking water; and impacts of climate change.
Among key informants, there is consensus across all respondent groups on the
continued need for GLAP IV. Almost all interviewees agree that there is a need for
federal funding to support the restoration and maintenance of Great Lakes AOCs. The
Great Lakes are a priority ecosystem with strong economic and social importance for all
Canadians. For most interviewees, the restoration and maintenance of the Great Lakes
represents not only a shared responsibility among all jurisdictions (federal, provincial and
municipal), but one for which federal leadership and funding is critical for a number of
reasons (among them, the federal mandate for transboundary waters, in accordance
with the Canada Water Act and the International Boundary Waters Treaty Act).
A majority of interviewed key informants indicate that there is a continued need for
federal funding to ensure the coordination and leveraging of funding with other
jurisdictions and stakeholders to restore and maintain the Great Lakes. A minority of
interviewees (though a majority of committee members and external experts) refer to the
federal government’s jurisdictional responsibilities for international waters as well as
those outlined in international and national agreements, as the rationale for the
continued need for GLAP IV. A number of interviewees (including a majority of
committee members and of federal managers and partners) cite the magnitude of the
remaining issues and problems facing the Great Lakes in general, and the AOCs in
particular, as evidence for the continued need for federal leadership and funding. For a
few GLSF project proponents, the issue of “legacy sites” and non-point source
contaminants, and the broader economic benefits of the lakes as a resource, represent
additional reasons for the continued need for federal funding in support of the Great
Lakes AOCs.
While the AOC designation is useful in focusing attention and resources on areas of
stress, some key informant interviewees indicate there is also a need for a more holistic
lake- or ecosystem-wide approach to adequately address environmental issues affecting
the health of the Great Lakes ecosystems. Some respondents believe there is a need to
expand the focus of GLAP IV beyond the designated AOCs. A few respondents indicate
that the GLAP should be broadened to include activities identified in the lake-wide
management plans developed for each of the Great Lakes. For a small number of
interviewees, there is a perceived need to broaden the focus to include shoreline and
watershed remediation activities considered to be important for the long-term integrity
and sustainability of the Great Lakes.
Environment Canada
21
Most key informants believe all GLAP IV priority investment areas are important and that
there is an ongoing need to continue supporting all targeted priority areas.22 A majority of
interviewees underscore the need for Environment Canada to be flexible in maintaining
the GLAP’s responsiveness to the different needs of the individual AOCs (i.e., that
priorities need to be assessed on an AOC by AOC basis).
Public opinion is consistent with scientific and key informant views on the importance of
the Great Lakes and the continued need for support. Recent public opinion research
reveals a growing concern among Canadians about Canada’s freshwater resources and
the need for governments to ensure the availability of freshwater supplies. A 2009
Nanos poll found that six in ten Canadians (62 per cent) consider freshwater to be our
most important natural resource, compared to two out of ten (22 per cent) who feel oil
and gas are the most important. Also, four in ten Canadians (40 per cent) indicate that
water pollution from industry, agriculture and/or households is their “greatest concern”
regarding freshwater in Canada.
Canadians’ confidence in the supply and safety of our freshwater resources has declined
significantly in recent years. The 2009 Water Attitudes Study found that, while nearly all
Canadians believe it is important to conserve freshwater on an ongoing basis
(95 percent), confidence that Canada has enough freshwater for the long term has
declined by 11 percentage points, from 81 per cent in 2008 to 70 per cent in 2009.
Similarly, Canadians’ confidence in the safety of Canada’s freshwater supply has
declined, from 81 per cent in 2008 to 72 per cent in 2009. Seven in ten Canadians
(70 per cent) believe that government is the “most responsible” for ensuring the
availability of clean water for their communities.23
The Great Lakes United Poll (2007) found that Ontarians are “very concerned” about
untreated sewage entering the Great Lakes (82 per cent), contamination of the food
chain by toxic pollution such as mercury (78 per cent), and the loss of wetlands and
animal habitat (67 per cent). The poll also revealed that eight in ten Ontarians (78 per
cent) favour spending $2 billion per year for ten years to restore the health of the Great
Lakes and St. Lawrence River. Majorities of Ontarians agree that the following constitute
“very good” reasons for spending Canadian tax dollars to clean up the Great Lakes: 10
million Canadians get their drinking water from the Great Lakes (83 per cent of
Ontarians consider this a “very good” reason); we should act now because problems in
the lakes will cost more to fix in the future (77 per cent); the Great Lakes represent one
fifth of the world’s freshwater supply (77 per cent); and contamination of the food chain
(72 per cent). Ontarians also perceive a number of benefits to protecting the integrity of
the Great Lakes, as evidenced by majorities of Ontarians who “agree” that: we all have a
personal responsibility to leave the Great Lakes in good health for our children (99 per
cent); government spending to upgrade sewage and water systems in the province will
22
These priority areas include: contaminated sediment assessment; habitat restoration;
municipal infrastructure improvements; assessments of ecosystem health; monitoring of water
quality and ecosystem health improvements; and management of federal programs and
coordination with other partners.
23
Options included: government, water companies, large companies, citizens, farmers or
non-governmental organizations.
Environment Canada
22
create jobs (89 per cent); and the Great Lakes will help shield us from the impacts of
global warming like heat and drought (72 per cent).
Potential for Duplication and Overlap
Program documents and key informants point to the availability of several other funding
programs that support remediation or science and monitoring in the Great Lakes AOCs.
A key source is provincial funding for implementation of Ontario commitments under the
COA. Relevant provincial ministries, the MNR and MOE, use these funds to fulfill
provincial commitments in the COA, including those in Annex 1 pertaining to AOCs. In
addition to provincial COA funds, funding sources include:
›
›
›
›
›
›
›
Environment Canada A-base funds (salary dollars for permanent staff
involved in science and monitoring in the AOCs and elsewhere);
other Environment Canada funding programs (e.g., Habitat Stewardship
Program, EcoAction Community Funding Program, Environmental Damages
Fund) that may fund projects in the Great Lakes AOCs;
the Federal Action Plan on Clean Water, which provided $48.9 million for
sediment remediation in the Great Lakes AOCs;
municipal funding;
foundations (e.g., the Trillium Foundation), associations (e.g., Ontario
Federation of Anglers and Hunters), NGOs (e.g., Ducks Unlimited),
universities;
Infrastructure Canada funding for infrastructure projects such as wastewater
treatment improvements; and
industry-provided funding.
Unlike other funding sources, however, GLAP IV is uniquely targeted to remediation
work in the Great Lakes AOCs. Also distinguishing GLAP IV from other initiatives is the
program’s framework for funding that is closely tied to Canada’s stated AOC
commitments under the GLWQA, priorities identified in the RAP reports in each AOC,
and its science-based focus. Funding and work planning is thus linked to BUIs identified
in each AOC, with the objective of completing federal actions to address BUIs in order to
delist.
Key informants across all respondent groupings expressed few concerns about
duplication or overlap between GLAP IV and other funding sources. Mitigating the risk of
duplication between GLSF-funded projects and other funding sources is informal
information-sharing at the officer level—for example, sharing of GLSF project proposal
and funding lists among officers with responsibilities for the various funding programs.
Also, more formally, the technical review of project proposals may also bring to light
potential areas of funding overlap or areas where efficiencies among projects can be
achieved. While GLSF project proponents also perceived GLSF and other funding
sources to be complementary rather than overlapping, they noted that there was
Environment Canada
23
duplication of effort for project proponents in terms of their efforts to access funds from
multiple sources (i.e., preparing funding applications for various funding programs).
The risk of duplication with respect to funding of science and monitoring activities is
mitigated because work in the AOCs is conducted by a relatively small science
community, with effective collaboration and awareness of each other’s activities between
federal researchers and their provincial and U.S. counterparts. As well, each AOC has
an assigned federal or provincial lead, which further ensures that duplication and overlap
is avoided.24
Summary: Recent trends of Great Lakes ecosystem conditions vary; some
conditions are improving and some are deteriorating. In large areas of the lakes,
historical sources of stress have combined with new ones to reach a tipping
point, at which radical ecosystem-level changes can occur rapidly and
unexpectedly. Public opinion is consistent with scientific and key informant views
on the environmental and societal importance of the Great Lakes and the
continued need for the GLAP. GLAP IV is targeted exclusively to AOCs, unlike
other public (federal, provincial and municipal) and non-public (industry,
foundations, NGOs, etc.) initiatives. Key informants express few concerns about
the possibility of duplication or overlap between GLAP IV and other programming.
Rather, GLAP IV is seen as complementary in that it is a catalyst for orienting
other funding sources to support activities in the AOCs.
3.1.2
Alignment with Federal and Departmental Priorities
Evaluation Issue
Is GLAP IV aligned with
federal government
priorities?
Indicator(s)
›
›
›
24
Program’s objectives
correspond to
recent/current federal
government priorities
Methods
›
›
Document review
Rating
Achieved
Key informant
interviews
Program’s objectives are
aligned with current
departmental strategic
outcomes
Views on the alignment of
program objectives with
current federal government
and departmental priorities
According to the COA Annex, a) Canada and Ontario will co-lead the RAP process in the
Toronto and Region, St. Mary’s River, St. Clair River and Detroit River AOCs; b) Canada will
lead the RAP process in the Thunder Bay, Hamilton Harbour, Port Hope and St. Lawrence
River AOCs; and c) Ontario will lead the RAP process in the Nipigon Bay, Jackfish Bay,
Peninsula Harbour, Spanish Harbour, Wheatley Harbour, Niagara River and Bay of Quinte
AOCs.
Environment Canada
24
Federal Priorities
Environment Canada’s GLAP is aligned to support federal government priorities
established under the 2007 Speech from the Throne and two recent federal budgets.
Notably, one of the five priorities set out in the October 2007 Speech from the Throne is
“A Healthy Environment for Canadians.” The Throne Speech commits the Government
of Canada to implementing a new “water strategy” to help clean up major lakes and
oceans. Budget 2007 created the Action Plan for Clean Water, which allocated funds to
improve the quality of water in Canada’s rivers, lakes and oceans, including $48.9 million
over eight years to accelerate existing actions for the remediation of contaminated
sediment in eight Great Lakes AOCs.
The Great Lakes are among the high-priority ecosystems identified nationally and
recommended for coordinated action.25 This recommendation was identified through the
2007–2008 Priority Ecosystem Initiatives Management Framework initiative, which was
intended to improve the Priority Ecosystems Initiatives program, optimize integration of
Environment Canada programs and activities, and strengthen accountability and
reporting. Ongoing federal support is further evident in Canada’s signing in 2007–2008
of a new three-year COA to work toward a healthy and clean Great Lakes Basin
Ecosystem.26
Almost all key informants concurred that the objectives of GLAP IV are well-aligned with
current federal government priorities, pointing most often to the GLWQA. By signing this
Agreement, Canada has an obligation to work toward improving the Great Lakes. Some
interviewees further point to the federal interest in water quality (relating to the 2007
Throne Speech, and Budget 2007 establishment of an Action Plan for Clean Water and
a priority on providing “A Healthy Environment for Canadians”27) as evidence of GLAP IV
alignment with federal priorities. In addition, some key informants (both internal
departmental managers and partners, and external experts and RAP representatives)
note that GLAP IV is an important vehicle to fulfill international commitments and to
contribute to positive Canada–U.S. relations.
25
Environment Canada. 2009. Status Report on the Implementation of an Ecosystem Approach
in Environment Canada.
26
Environment Canada. 2008. 2007–2008 Departmental Performance Report. Available at:
www.tbs-sct.gc.ca/dpr-rmr/2007-2008/inst/doe/doe00-eng.asp
27
Government of Canada. “Speech from the Throne”. October 16, 2007.
http://www.pm.gc.ca/eng/media.asp?id=1859
Environment Canada
25
Departmental Priorities
The GLBEI is one of six ecosystem initiatives at Environment Canada under the Priority
Ecosystem Initiatives Sub-Activity, and aligns with one of the department’s seven
program priorities (Ensuring Water Quality and Quantity).28 The Priority Ecosystem
Initiatives Sub-Activity adopts an ecosystem approach to environmental management for
the benefit of governments, industry and individuals, by aligning science, monitoring,
on-the-ground action and policy expertise, as well as enhancing collaborative
governance and decision-making mechanisms. The GLBEI supports the Department’s
strategic outcome that “Canada’s natural capital is restored, conserved, and enhanced.”
GLAP IV outcomes and priority investments were designed to complement a number of
other sub-activities within Environment Canada’s program activity architecture, including
Aquatic Ecosystems, Wildlife, and Assessment and Ecological Monitoring.
GLAP IV is considered by most key informants to be well-aligned with Environment
Canada departmental priorities as a priority ecosystem. A number of federal
respondents also note that activities within the Great Lakes, such as science and
monitoring, have national applicability and that this research can be used to benefit
areas beyond the Great Lakes. Furthermore, GLAP IV is viewed by key informants as a
consistent fit with Environment Canada’s mandate to preserve and enhance the quality
of the natural environment (habitat, biodiversity, species at risk) and conserve and
protect Canada’s water resources.
Summary: GLAP IV aligns well with federal and departmental priorities. The
program uniquely addresses federal commitments in AOCs under the GLWQA and
COA, by providing a framework to address BUIs linked to AOC RAP priority areas.
The GLBEI, of which GLAP IV is a part, is one of six ecosystem initiatives at
Environment Canada under the Priority Ecosystem Initiatives Sub-Activity. The
GLBEI supports the Department’s strategic outcome that “Canada’s natural
capital is restored, conserved, and enhanced.” GLAP IV outcomes and priority
investments support a number of sub-activities within Environment Canada’s
program activity architect, including Aquatic Ecosystems, Wildlife, and
Assessment and Ecological Monitoring.
28
Environment Canada. 2009. 2008–2009 Report on Plans and Priorities. Available at: www.tbssct.gc.ca/rpp/2008-2009/inst/doe/doe00-eng.asp
Environment Canada
26
Consistency with Federal Roles and Responsibilities
Evaluation Issue
Is GLAP IV consistent with
federal roles and
responsibilities?
Indicator(s)
›
›
Program mandate aligned
with federal government
jurisdiction
Methods
›
›
Document review
Rating
Achieved
Key informant
interviews
Views on the
appropriateness of federal
involvement
The Government of Canada, through the GLWQA, has committed to working with other
levels of government to assess, restore and protect beneficial uses in AOCs. Legislative
authorities include:
›
›
›
The Canada Water Act, which establishes and reinforces Environment
Canada’s mandate in the management and protection of water quality in
Canada. This Act authorizes the Minister of the Environment to enter into
agreements with provincial governments, subject to Governor in Council
approval, where there is a significant national interest in the management of a
water resource, and to work with provinces in designing and implementing
projects for the efficient conservation, development and utilization of those
waters. Authority is also granted to establish joint commissions, boards or
other bodies empowered to direct, supervise and coordinate those programs.
The International Boundary Waters Treaty Act, which provides the principles
and mechanisms to help resolve disputes and to prevent future ones,
primarily those concerning water quantity and quality along the boundary
between Canada and the United States. This Act supports the establishment
of the IJC under the boundary waters treaty signed by Canada and the United
States in 1909.
The Canadian Environmental Protection Act, 1999, which is the federal
legislation respecting pollution prevention and the protection of the
environment and human health, in order to contribute to sustainable
development.
Key informants across all respondent groups concur that it is appropriate for the federal
government to play a role in the restoration and maintenance of the Great Lakes AOCs.
Collectively, key informants point to several factors that illustrate the need for a federal
role in the Great Lakes. First, the federal government has a responsibility to support
work to fulfill international agreements (such as the Great Lakes Water Quality
Agreement). Further, as the Great Lakes are the largest freshwater body in North
America, it is considered by the majority of key informants to be a critical national
resource. For example, one key informant stated that “they are clearly important to water
quality, industry and transportation, and affect a significant proportion of the Canadian
population.”
The complexity of the boundaries of the waters—four of five lakes form part of the
Canada-U.S. border and five AOCs are binational—underscores the appropriateness of
Environment Canada
27
a federal lead in this area. An external expert summarized this point by noting that
“because we are dealing with binational issues, federal funding that enables place-based
remediation and protection to happen is paramount to the success of the government’s
promise to the [Great Lakes Water Quality] Agreement.”
Some key informants feel that fulfilling this commitment is especially important now that
the Government of the United States has committed $475 million in new funding to the
Great Lakes under the U.S. Great Lakes Restoration Initiative.29 This increase in funding
is perceived by some key informants to raise expectations for Canada to provide a
commensurate investment in the Great Lakes. Through GLAP IV efforts, some key
informants point out that Canada has taken a leadership role relative to the United
States on Great Lakes AOC restoration and maintenance. They urged that these
activities, along with the financial commitment, continue.
Although key informants agree that there is a need for federal involvement, including a
leadership role, it was also noted that the federal government should not and cannot
play the exclusive role, and that provincial and municipal governments, as well as
community-driven organizations, have a responsibility and critical role to play as well. An
example of this is that the jurisdiction for wastewater infrastructure improvements rests
with municipalities and the provinces.
Summary: The federal role in GLAP IV is appropriate. GLAP IV supports federal
government commitments and obligations under the GLWQA and COA, as well as
federal government priorities outlined in the 2007 Speech from the Throne and
two recent budgets. Other legislative authorities include the Canada Water Act,
International Boundary Waters Treaty Act, and Canadian Environmental
Protection Act, 1999.
29
http://greatlakesrestoration.us/. Accessed on November 25, 2010.
Environment Canada
28
3.2
Program Performance: Design and Delivery
This section presents the evaluation findings related to the adequacy of the program
design and delivery from a number of aspects outlined in the evaluation matrix, including
the engagement of partners, work planning activities, roles and responsibilities, resource
allocations, monitoring and reporting activities, and GLSF program delivery.
Overall Findings
Overall, GLAP IV was viewed as a sensible model to achieve intended outcomes.
Engagement of partners represents a critical and beneficial aspect of the GLAP IV
program delivery. Partnerships occur at many levels and in many ways. Formal
agreements between Canada and the United States (GLWQA) and Canada and Ontario
(COA) represent the foundation for commitments of the jurisdictional parties with respect
to the AOCs. Implementation involves horizontal partnerships within Environment
Canada and across the federal government. RAP committees provide a forum for
governments (federal, provincial, municipal and First Nations) and key stakeholder
groups (conservation authorities, community groups and NGOs) to coordinate their
efforts through to the delisting process. The funding structure of GLSF projects promotes
partner engagement at the local and regional level with partner contributions, including
cash and in-kind funding, assistance with delivery, and participation in advisory or
research capacities.
The implementation of GLAP IV diverged from the original design of the program in
several areas. The five-year work plans required by federal partner departments to
access GLAP IV funds were considered to be a positive and worthwhile exercise. The
intended work-planning annual reporting, review and updating process did not occur as
intended, however, which resulted in a loss of flexibility, responsiveness and
accountability.
Due to the horizontal nature of the initiative and broad changes at Environment Canada,
GLAP IV governance and management activities do not reflect those outlined in the
management framework developed for the program. Areas identified as a potential
source of confusion or concern by federal key informants include an inadequate
instrument to ensure that partner contributions to program goals are fulfilled, and clarity
of roles and responsibilities with respect to delisting AOCs.
For federal partners who received their recommended funding, the amount of GLAP IV
funding was seen to be adequate, with the caveat that they would like future funding to
be indexed for inflation. However, some federal partners report receiving significantly
fewer resources than their original approved funding levels, which they feel negatively
affected their ability to implement planned activities. For federal partners that received
GLAP IV funding allocations in full, a more pressing concern was the high ratio of
operations and maintenance (O&M) to salary dollars and inability to convert these
dollars (i.e., from O&M to salary), which left many groups with insufficient salary dollars
to hire scientific and technical staff. GLSF project proponent and committee key
Environment Canada
29
informants were more apt to suggest that the GLAP focus and resources be expanded to
include priority areas and activities that extend beyond the designated AOCs.
GLAP IV monitoring and reporting activities are undertaken to meet annual reporting
obligations associated with both the GLWQA and the COA. At the level of funded
projects, GLSF project reporting is quite complete and the majority of reviewed GLSF
files provided clear evidence of intended output achievements. However, limited
evidence was provided on the outcomes of remedial actions taken in the AOCs. While
federal proponents were supportive of increasing reporting requirements for GLAP
projects to enhance performance measurement and accountability, regular reporting of
federal science and monitoring projects is not a formal requirement and reporting to the
GLAP IV program is ad hoc.
Project proponent key informants indicate a high level of satisfaction with the GLSF
program. Program priorities are considered to be clear and appropriate. The application
process is regarded as clear and transparent by most funding recipients and selection
criteria are viewed to be explicit and well-communicated. Overall, the GLSF proposal
review process is perceived to be logical and GLSF funding decisions are viewed as
strategic and fair. Program staff are seen by project proponents to be accessible and
supportive. Modifications to the GLSF program (including the shift from the use of MOUs
to contribution agreements), implemented by Environment Canada in 2007, have
challenged the timely delivery of the program, reduced flexibility and increased
uncertainty among partners. Project proponents indicate a preference for streamlining
the application and approval process, for multi-year agreements, and for more
straightforward funding agreements.
Environment Canada
30
Evaluation Issue
Is the GLAP IV design
appropriate for achieving
expected program results?
Indicator(s)
›
›
›
›
3.2.1
Plausible link between
program activities, outputs
and intended outcomes
Clearly defined and
understood governance
structure, including program
processes, roles,
responsibilities and
accountabilities
Program resources/capacity
commensurate with
expected program results
Methods
›
›
›
Document review
Rating
Achieved
Key informant
interviews
File review
Little progress,
priority for
attention
Progress made,
attention needed
Views on the
appropriateness of program
activities, processes and
governance structures
Engagement of Partners
Key informants across the respondent groups had a favourable view of GLAP IV as a
program model: the program was generally viewed as a sensible approach to achieve
the intended outcomes. No major redundancies or notable gaps were identified within
the program.
An important feature of the program, consistently highlighted by key informants across
various respondent groups, is the engagement of partners and stakeholders. This occurs
in many ways, perhaps most importantly in the formal agreements between Canada and
the United States (GLWQA) and Canada and Ontario (COA). These agreements are the
foundation for commitments of the parties with respect to the AOCs and are a critical
framework in organizing work in the AOCs. Other examples of partner engagement
include:
›
›
›
Multiple groups within Environment Canada, including the Canadian Wildlife
Service and the NWRI. In addition, the program formally includes eight other
federal government departments, which are represented on the GLEC.
RAP committees, which play an important role in driving work in the AOCs.
The RAP three-stage reporting process is the basis for updating the GLWQA
on progress in the AOCs and the delisting process, as well as for engaging
local partners “on the ground.” While RAP committees vary in their size and
structure, key member representatives can include: federal representatives
(e.g., DFO, Environment Canada), other government representatives (Ontario
MOE or MNR, municipal/regional, First Nations), Conservation Authorities,
community groups and NGOs.
GLSF projects, which require partnerships as a condition of funding approval,
particularly (though not exclusively) funding partners. The review of GLSF
project files confirmed that all GLSF projects include partners—six on
Environment Canada
31
average, with organizations having a local/regional scope of operations being
engaged most often (85 per cent of projects), followed by organizations
operating at the provincial level (72 per cent) (Table 3.1). Types of partners
included: government (federal, provincial, municipal/regional, First Nations)
(83 per cent), community-based organization (69 per cent) and the private
sector (industry, landowners) and educational institutions (both at
29 per cent). The nature of partner contributions most often included cash
and in-kind contributions (e.g., equipment, land, office space or supplies)
(97 per cent); followed distantly by assistance with delivery (e.g., planting) or
technical advice/assistance (18 per cent each), and participation in an
advisory or research capacity (13 and 11 per cent, respectively). “Other”
contributions include such things as publicity/promotion and training.
Partnerships for federal partner projects were not analyzed in the same way,
however, publication listings indicate the involvement of partners as
investigators. As well, NWRI project-level documentation for the NWRI shows
collaborations with municipalities, the province and other researchers.
Contributions include: data sharing; sampling/provision of samples, facilities
and field support; and constructing facilities. Leveraged resources are also
indicated ($550,000, for two projects out of five in total).
Table 3.1: Characteristics of GLSF Agreements: Partnerships*,**
Scope of partners’ operations
Per cent of files
Local/regional
85%
Provincial
72%
National
51%
International
5%
Sectors represented
Governments
83%
Community-based
69%
Private sector
29%
Educational institutions
29%
Professional associations
14%
Other countries
3%
Other
14%
Partners’ contribution
Funding
97%
Service delivery
18%
Technical advice/assistance
18%
Advisory committee
13%
Research
11%
Not described
5%
Other
13%
* Based on a sample of 39 GLSF project files
** The average number of partners per project is 6
›
Although federal projects do not have the same requirement for partnering,
16 of 21 federal project files that were reviewed identified partners. These
Environment Canada
32
partners typically included Environment Canada or federal OGDs, the
Province of Ontario, municipalities, a RAP committee, or university. The
nature of partner contributions for federal projects focused most often on
collection or sharing of samples or data and information analysis or
exchange.
Summary: Engagement of partners represents a critical and beneficial aspect of
GLAP IV. Partnerships occur at many levels and in many ways. Formal
agreements between Canada and the United States (GLWQA) and Canada and
Ontario (COA) represent the foundation for commitments of the jurisdictional
parties with respect to the AOCs. Horizontal partnerships within Environment
Canada and across the federal government are necessary for GLAP
implementation. RAP committees provide a forum for governments (federal,
provincial, municipal and First Nations) and key stakeholder groups (conservation
authorities, community groups and NGOs) to coordinate their efforts through the
delisting process and engaging local partners “on the ground.” The funding
structure of GLSF projects promotes partner engagement at the local and regional
level. Partner contributions include cash and in-kind funding, assistance with
delivery, and participation in an advisory or research capacity.
3.2.2
Work Planning
To access GLAP IV funds, federal partner departments, including Environment Canada,
each submitted five-year work plans that outlined the projects they proposed to
implement using GLAP IV funds and described how each aligned with program priorities.
A review team (with representatives from all federal departments that submitted
work plans) assessed each of the work plans. In evaluating departmental work plans,
priority was given to completing federal actions in revised group 1 AOCs, while revised
group 2 AOCs were accorded second priority.30 As well, the review team did not
recommend projects:
 where a need was not clearly identified or where it was uncertain;
 that were considered purely research;
 that were not an essential part of a coordinated package of projects; or
 that were not considered essential for completing federal actions for BUI
restoration or AOC delisting.
The work plans were intended to steer work so that it would be responsive to the needs
of the AOCs and be a tool for accountability. In this regard, recipient departments were
to report on progress and update their work plans annually, at which time work plan
priorities and activities could be adjusted as needed. With the endorsement of the
GLEC, the Environment Canada Program Director would sign off on the allocation. While
funds for the full five-year work plan period were approved in principle to provide
30
The review committee assigned first priority to those AOCs with the greatest potential for
delisting in the short to medium term (Group 1 AOCs). These Group 1 AOCs include: Thunder
Bay, Nipigon Bay, Peninsula Harbour, Wheatley Harbour, St. Lawrence River, Bay of Quinte
and St. Clair River. In the COA, Group 1 AOCs are Nipigon Bay, Jackfish Bay, Wheatley
Harbour and St. Lawrence River (Cornwall)
Environment Canada
33
resource stability to organizations, it was also stated that, if necessary, departments
would be able to reallocate their total resources based on the reviewed and
adjusted/updated work plans. Due to a Department-wide transformation beginning in
2005/06, the GLBEI (of which GLAP IV was a part) was co-led by the RDG–Ontario and
RDG–Quebec. The intended work planning annual reporting, and review and updating
process, never occurred.
Departmental managers and federal partners who were interviewed for the evaluation
approved, in general, of the work planning process as a worthwhile exercise to guide
activities in AOCs. However, the process as it was intended was abandoned and
work plans were not reviewed annually. There was therefore no mechanism available for
changing work priorities or reallocating funds on an annual basis (though this latter issue
was not universally perceived to be problematic)—the approved projects became
“hard-wired” for the five-year term of the program. As well, the role of the work planning
process as a tool for accountability (i.e., to ensure that activities were being conducted
by the various partners according to their work plans and to address GLAP IV
objectives) did not materialize.
Summary: The five-year work plans required by federal partner departments to
access GLAP IV funds and proposed projects were recommended by the review
team and approved by the GLEC. Federal key informants consider work planning
a positive and worthwhile exercise to guide remedial and monitoring activities in
the AOCs. However, the intended work-planning annual reporting, review and
updating process did not occur, which resulted in a loss of flexibility,
responsiveness and accountability.
3.2.3
Roles and Responsibilities
GLAP IV is a horizontal and bilateral initiative, with an AOC-based RAP committee
structure (described in subsection 1.1). As a result, contributions to the delivery of
federal commitments in AOCs are highly distributed. Commitments of the federal and
provincial governments are outlined in the COA, and the GLWQA articulates
Canada–U.S. commitments in the AOCs.
Program documentation with respect to the governance and management of GLAP IV
(Management Framework: Great Lakes Action Plan for Areas of Concern (2005–2010))
does not reflect the current operation of the program in at least one respect. As a result
of the departmental transformation that occurred in 2005–2006, the original structures
and authorities for program management were not established in their entirety. The
original documentation on the roles and responsibilities for the program identified a
Program Management Committee that would have responsibility to support the GLEC,
help facilitate the flow of information among the departments, and allow
directors/managers the opportunity to brief their senior management on key issues. The
Program Management Committee has not been an active committee during the past
Environment Canada
34
several years.31 The communication and coordination function has fallen to the COA
Annex Implementation Committee, which has a similar membership. This was not widely
noted by key informants, and some respondents perceived a gap in coordination and
communication across federal partners.
Despite the horizontal and bilateral nature of the initiative and broad changes in the
Department, most federal key informants felt they had a good understanding of their own
roles and responsibilities with respect to the program and, at a general level, those of the
major partners in the program. GLSF proponents also believed they had a clear
understanding of their roles and responsibilities.
However, the roles and responsibilities of federal partners was an aspect of GLAP IV
that came under criticism in the CESD’s reviews of the program in 2001 and 2008 and
the TB MAF exercise with respect to the GLBEI in 2008. The 2008 CESD report noted
that “while the department has recently clarified some responsibilities, it has still not
clearly specified who is responsible for carrying out all the required remedial actions,
who will pay for those efforts, and within what timelines the actions will be taken.”
Similarly, the Round V TB MAF exercise carried out in 2007–08 urged the Department to
clarify partner roles/contributions and to “review the change to a distributed responsibility
model to deliver Great Lakes Water Quality Agreement/Canada-Ontario Agreement
commitments to ensure sufficient resource allocation and the ability to meet changing
priorities.”
While generally aware of GLAP IV roles and responsibilities, a number of key informants
echoed some of the CESD and Treasury Board concerns with the “distributed
responsibility” model of this horizontal program. For example, there was a feeling among
some, particularly those at the working level, that there is a disconnect (or at least the
connection is not readily apparent) between the needs of the individual AOCs (as they
are articulated in the RAPs) and the remedial as well as science and monitoring projects
being carried out by the federal partners. Other key informants noted that, in general,
communication and coordination among federal partners is an area that merits further
attention from the program. It is difficult to say what type of internal management
mechanism or instrument might have most benefit for the program (and key informants,
when pressed, did not specify), though there is generally little enthusiasm for additional
structures and meetings unless the benefits appear evident.
In response to the CESD report, the program has taken steps in 2009–2010 to clarify the
actions remaining in each AOC, the responsibility of each partner in the AOCs and the
proposed timelines. This information, including an update on the status,
accomplishments and the responsible partners and timeframes for remaining actions in
each AOC, was presented in January 2010 to RAP committees and federal and
provincial partners during one of a series of program workshops,32 and garnered positive
reviews (as heard in the evaluation interviews).
31
With the dissolution of the planning section in the former Great Lakes and Corporate Affairs
Branch (concurrent with and as a result of Environment Canada transformation), there was no
single focal point to lead the planning function.
32
These included, in 2004 and 2006, Sharing experiences and a Habitat workshop in 2008.
Environment Canada
35
Another issue raised by some federal key informants is a grey area surrounding roles
and responsibilities with respect to delisting AOCs. The GLWQA states that “The Parties
shall cooperate with State and Provincial Governments to classify Areas of Concern by
their stage of restoration progressing from the definition of the problems and causes,
through the selection of remedial measures, to the implementation of remedial
programs, the monitoring of recovery, and, when identified beneficial uses are no longer
impaired and the area restored, the removal of its designation as an Area of Concern”
and that “The Parties, in cooperation with State and Provincial Governments, shall
ensure that the public is consulted in all actions undertaken pursuant to this Annex.” In
practice, the definition of delisting criteria has proven challenging because of difficulty in
determining a target or end point for actions (i.e., defining “when the job is done”). The
absence of clear delisting criteria for AOCs came under criticism in the 2008 CESD
report as well. Several AOCs are now approaching a state of restoration, but, for some
respondents, the authority and process for decision making about delisting are unclear.
A small number of respondents noted the potential for a built-in disincentive to delist,
due to the resulting termination of GLAP dollars for the AOC and the lack of alternative
funding to monitor these areas to ensure that they remain restored. Environment
Canada is currently working with the IJC to reach agreement on a consistent set of
metrics for recommending restoration of BUIs and delisting of AOCs.
Summary: GLAP IV is a horizontal and bilateral initiative governed by
commitments outlined in the GLWQA and COA, with an AOC-based RAP
committee structure. Due to the horizontal nature of the initiative and broad
changes at Environment Canada, GLAP IV governance and management activities
do not reflect those outlined in the management framework developed for the
GLAP for AOCs (2005–2010). Areas identified as a potential source of confusion or
concern by federal key informants include: instruments for ensuring that partner
contributions occur and are coordinated and consistent with the goals of GLAP
IV; and clarity of roles and responsibilities with respect to delisting AOCs.
3.2.4
Resources
As indicated, the original allocation for GLAP IV was $40 million over five years. The
work planning process identified and recommended funding to federal partners with the
intention that amounts would be reviewed and determined in a flexible manner annually,
based on evolving needs and identified priorities. In the first year of GLAP IV, allocations
were consistent with recommended funding levels (albeit rolled out late in the fiscal
year). However, a number of federal partners within Environment Canada noted that the
remaining years of GLAP IV saw a significant divergence in the allocation, with some
groups such as the Canadian Wildlife Service reportedly receiving substantially less
funding (though this could not be confirmed due to the absence of financial tracking
information). As well, the annual review (and possible adjustment) of the funding
allocation did not occur; funds for subsequent years were allocated to federal partners
for the remainder of GLAP IV based on the amount of the first fiscal year allocation.
For those federal partners that received GLAP IV funds, a serious concern was the mix
of salary and O&M dollars that were received. DFO and groups within Environment
Environment Canada
36
Canada were allocated funds that, in the first year of program implementation, had the
flexibility to change O&M for salary dollars and vice versa. A Department-wide initiative
subsequently froze these dollars in their respective categories. Because the original
allocation was heavily weighted toward O&M, many groups with responsibility for
delivering GLAP IV–funded initiatives had the challenge of insufficient salary dollars to
hire the research or technical staff required to carry out the labour-intensive science and
monitoring project activities on a long-term basis.
For federal departments and groups within Environment Canada that received their
original allocation, the funding amount was generally seen to be “about right,” with the
caveat that yearly funding amounts for future work in the AOCs would benefit from being
increased to account for inflation. In contrast, GLSF project proponents were more apt to
argue strongly for an increase in funding to this program component to permit a more
aggressive and comprehensive approach to addressing issues in the AOCs. Funding to
the GLSF was curtailed after 2005–06.
A small number of key informants, particularly among the committee members, strongly
favoured an increase in GLAP resources. However, this view also included a preference
for expanding the terms and conditions of the program beyond AOCs (e.g., expanded
AOC boundaries to include watersheds, heavily degraded areas around the lakes that
are not AOCs, or the entire nearshore). In other words, while the current level of
resourcing for GLAP IV was deemed to be sufficient for the program’s narrow focus on
AOCs, these respondents would like to see both the resources and programming focus
expanded.
A critical issue with respect to resources and financial accountability is the very limited
use of financial codes to track GLAP IV program resources. In the June 2006
departmental transformation, Environment Canada determined that priority ecosystem
initiatives, including the GLBEI (of which GLAP IV is a part), would be delivered through
an unbundling33 of activities and resources. GLAP IV resources were allocated to
Outcome Project Groups34 within Environment Canada that were responsible for
achieving GLAP IV objectives. However, these monies were not coded separately from
Great Lakes A-Base resources, nor were they coded separately from resources
dedicated to the GLBEI within the Department, thus severely limiting the financial
accountability of the program.
Summary: For federal partners and groups within Environment Canada for whom
resource allocations were consistent with their original recommended funding
levels, allocated GLAP IV funding was viewed to be adequate, with the caveat that
future funding should be indexed for inflation. However, some federal groups
33
The department receives a share of its annual budget from temporary (as opposed to core) funding
allocations from the Treasury Board in order to deliver programs for various specific purposes.
While typically used in the intended manner, the Deputy Minister has the authority to realign this
temporary funding to address departmental priorities. Just such a realignment occurred in 2005-06
and this process has come to be known within Environment Canada as "unbundling".
34 OPGs were the basis of the new management structure in the department to promote
integrated management and decision-making in the context of a clearer view of departmental
results and strategic direction.
Environment Canada
37
received significantly less resources than their original approved funding levels,
thus negatively affecting their ability to implement planned activities. For federal
partners and groups within Environment Canada that received GLAP IV funds, a
more pressing concern than the level of funding was the high ratio of O&M to
salary dollars, which left many groups with insufficient salary dollars to hire the
staff required to undertake science and monitoring project activities. With direct
transfer of funds to partners, but no program-specific coding of expenditures,
financial accountability is a significant weakness of the program. GLSF project
proponent and committee key informants were more apt to suggest that the GLAP
focus and resources be expanded to include priority areas and activities that
extend beyond the designated AOCs.
3.2.5
Monitoring and Reporting
There are reporting obligations associated with the GLWQA and COA to which GLAP IV
must respond. The GLWQA requires the parties (Canada and the United States) to
report progress on AOCs to the IJC. Comprehensive reporting on AOCs was provided in
2003 to the IJC (resulting in the April 2003 IJC report Status of Restoration Activities in
Great Lakes Areas of Concern: A Special Report). An informal briefing on the status of
AOCs was also provided to the Canadian Section of the IJC (the Canadian secretary
and IJC staff members) in 2009. The Report urged Canada and the United States to:
fulfill their commitments to report on “…progress in developing and implementing
Remedial Action Plans and in restoring beneficial uses…”; “…document their
considerable investment and achievements to date in order to provide the public with a
true reflection of their accomplishments.”; and “…ensure that monitoring, data support
and information management systems are in place...”
Reporting of GLAP IV also occurs annually to update delivery of federal commitments in
the COA. Progress in relation to achievement of COA commitments is assessed
annually and reported to the COA Management Committee, based on an extensive COA
database, where federal and provincial parties input all their projects against each COA
result number as per the COA agreement.
At the level of funded projects, GLSF projects follow a reporting template that includes
progress reporting (where required) and a final report (or annual report for multi-year
projects). Final reports include background/project description, a listing of
activities/outputs against intended deliverables, key products (e.g., published papers,
research documents), highlights of the project, as well as final financial reporting on
project expenditures (GLSF funding and leveraged contributions). The file review
confirmed that GLSF project reporting is quite complete: the majority of files contained
documentation relevant to the purpose of the project (proposal, technical review), MOU
or contribution agreement, as well as a final report and associated products. The file
review further confirmed that in the vast majority of files (80 per cent), evidence of
intended output achievement was provided to a very or moderately clear degree in the
documentation. Three quarters of projects were rated to have a very close or moderate
alignment between the project’s design and implementation
Environment Canada
38
Several key informants in various respondent groups pointed out that a challenge in
reporting on GLSF-funded projects is measuring the outcomes of the remedial actions
taken in the AOCs. While habitat or remediation initiatives may be considered a success
in terms of outputs, it is much more difficult to determine the level of success in terms of
outcomes (e.g., BUI and AOC restoration). Indeed, the review of files indicated that
projects were more likely to have output-related objectives than outcome-related
objectives: 38 per cent of files contained output-related objectives only; 15 per cent
contained outcome-related objectives only; and 46 per cent contained both types of
statements.35
As well, the files were far more likely to provide evidence of achievement of intended
outputs (80 per cent clear or moderately clear evidence of outputs) than outcomes
(54 per cent clear or moderately clear evidence of outcomes). Material on outcomes was
more frequently available for projects that had a monitoring component (e.g., water
quality or population data) or that were conducted as part of an ongoing multi-year effort.
External evaluations of project results were rare.
Reporting requirements for federal science and monitoring projects under GLAP IV are
not specified to the same extent as GLSF projects. As a result and as noted previously,
the review of federal projects was addressed in a qualitative and descriptive fashion.
Reporting on federal projects may include reporting / annual reports to the RAP
committees (for whom the science or monitoring was conducted), GLWQA/COA
reporting, and contributions to the scientific literature (papers and presentations). For
projects conducted by DFO, for example, almost 70 conference presentations were
prepared and over 25 reports and publications were prepared for the sample of nine
projects funded under GLAP IV.
Regular and structured reporting to the program on project activities has not been made
a formal requirement of the funding, and therefore reporting has been sporadic. Federal
partners noted that periodic workshop presentations or summaries of activities were
requested and were provided to the program on an ad hoc basis.
Federal proponents were generally supportive of increasing reporting requirements for
GLAP IV projects, a trend they viewed as being consistent with overall increased
attention in government to performance measurement and accountability. While there
are some projects, such as basic science, for which performance measures may be
difficult to develop for outcomes, respondents were, for the most part, agreeable to
enhanced reporting on deliverables outlined in their work plan.
An aspect of reporting that was raised by a small number of key informants was the
absence of a mechanism to share the results of science and monitoring activities carried
out in the Great Lakes AOCs in order to facilitate technical transfer and share lessons
learned. This was a commitment that was set out in the COA Annex 4 pertaining to the
35
Examples of outcome-related objectives include: rehabilitation of aquatic and riparian habitat
resulting in re-establishment of fish and wildlife; and monitoring and reducing rural non-point
source pollution.
Environment Canada
39
AOCs, but was never fulfilled.36 The development of web-based information-sharing
would fall under the Internet Content Renovation Initiative (formerly the One Department,
One Website initiative), which moves the Department to a more centralized approval
process for publishing of web content. With this, the Ecosystem Sustainability Board is
responsible for approving all new web content, and the COA commitment has not been
prioritized and approved by the Board. However, GLAP IV has held workshops to
facilitate technical transfer. Finally, the lack of inclusion of members of the public or lay
audiences in plans for dissemination of activities and results of GLAP IV work in the
AOCs was also noted by a small number of key informants as a deficit in program
communications.
Summary: GLAP IV monitoring and reporting activities are undertaken to meet
annual reporting obligations associated with the GLWQA and COA. At the level of
funded projects, GLSF project reporting is quite complete. The vast majority of
reviewed GLSF files provided clear evidence of intended output achievements,
however, only limited evidence was provided on the outcomes of remedial actions
taken in the AOCs. Regular reporting of federal science and monitoring projects is
not a formal requirement of the funding program, and reporting to the GLAP IV
program over the period reviewed has been ad hoc. Federal proponents were
generally supportive of increasing reporting requirements for GLAP projects in
order to enhance performance measurement and accountability. There was a
weakness identified in information sharing among GLAP IV stakeholders and the
general public.
3.2.6
GLSF Program Delivery
Feedback from GLSF funding recipients indicated a high level of satisfaction with the
design and delivery of the program. Almost all project proponent interviewees
considered GLSF priorities to be clear and appropriate. GLSF projects are perceived to
be aligned with activities outlined in the RAPs developed for the respective AOCs, and
are seen by interviewed project proponents as contributing to the overall health of the
Great Lakes.
Nearly all project proponents considered the GLSF application process to be clear and
transparent. Selection criteria were viewed by most funding recipients to be explicit and
well-communicated, with direct links to GLSF priority areas and to delisting criteria
indicated in their AOC’s RAPs. The GLSF proposal review process was perceived to be
logical and GLSF funding decisions are considered to be strategic and fair. According to
one project proponent, GLSF is “one of the better-run programs—very focused.”
For the majority of funding recipients, the proposal review process was seen as a
collaborative and consultative exercise that provides for two-way conversations on the
alignment of proposed activities with GLAP IV and RAP priorities and needs. While
36
Goal 2 in Annex 4 is to “Continue to improve the discovery and sharing of data, information
and trends in the Great Lakes Basin Ecosystem” with the expected result of “Increased sharing
of data and information among governments, organizations and Basin residents”, including
through web-based means.
Environment Canada
40
almost all interviewees were satisfied with the clarity and transparency of the application
process, a few project proponents found the process to be overly complex and,
depending on the scope and technical nature of the project, noted that it could represent
a disincentive for organizations without appropriate in-house technical support, or for
smaller projects where the value of the funding sought may not justify the level of effort
expended to apply for funding.
The support provided by program staff during the proposal development and review
process was seen by nearly all GLSF proponent interviewees as being constructive, with
a good level of interaction and support provided. As one project proponent indicated, it is
“a very interactive and beneficial process.” Most interviewed GLSF recipients indicated
that program staff are accessible during the proposal development process, and the
feedback and support provided by project officers is helpful. For some project
proponents, Environment Canada support enabled them to improve their proposal by
enhancing the alignment of the project with the goals and objectives of GLAP IV or by
providing technical support for high-profile projects in binational AOCs. A few project
proponent interviewees experienced a decline in the level of support provided by
program staff, which was attributed to high turnover rates of GLSF staff that occurred
during a 2007–08 program-restructuring initiative. However, these interviewees also
noted recent improvements in the level of support provided by program staff.
While there is a high level of satisfaction in the overall design and delivery of the GLSF
program, there is also a high level of consistency among project proponents in the
perceived weaknesses of the current design and delivery of the GLSF program. A
number of changes to the GLSF approval and funding processes were implemented by
Environment Canada in 2007. Notably, the MOU mechanism traditionally used to fund
GLSF projects using O&M resources was replaced by contribution agreements to
support projects using G&C funds, and the approval process was amended to require
senior management and ministerial authorization. The contribution agreement funding
mechanism (and associated approval requirements) has compromised the timely
start-up of some time-sensitive on-the-ground projects. Other concerns noted by project
proponents included unexpected decreases in GLSF funding, which has led some to
consider cancelling or limiting the scope of their projects, and some lack of clarity in the
GLSF agreement’s numerous clauses.
A majority of funding recipients believe Environment Canada should implement a more
streamlined application and approval process to enhance the efficient delivery of the
GLSF program by reducing the level of effort expended to access funding (e.g., simplify
the application form for non-technical or low-dollar-value projects, and enable “umbrella”
annual proposal submissions by organization) and by shortening the amount of time
needed to approve funding proposals and agreements (i.e., simplify the internal review
and approval process). A minority of funded recipients indicate a preference for
multi-year agreements that would increase the effectiveness of on-the-ground projects
by enabling multi-year project planning, thus ensuring timely start-up of field work by
avoiding yearly approval processes.
The issues raised by GLSF project proponents about G&C approval processes are
consistent with those identified by the Blue Ribbon Panel on Grants and Contributions in
Environment Canada
41
2006.37 Since that time, federal departments (including Environment Canada) have been
undertaking grants and contributions reform initiatives, which include action plans to, for
example, become more client-focused, such as through making the funding process and
programs more transparent, reducing the administrative burden on funding recipients,
and providing funding to clients on a more timely basis.38
Summary: Project proponent key informants indicate a high level of satisfaction
with the GLSF program. Program priorities are considered to be clear and
appropriate. The application process is regarded as clear and transparent by most
funding recipients and selection criteria are viewed to be explicit and
well-communicated. Overall, the GLSF proposal review process is perceived to be
logical and GLSF funding decisions are viewed as strategic and fair. Program staff
are seen by project proponents to be accessible and supportive. Modifications to
the GLSF program, implemented in 2007 by Environment Canada, have challenged
the timely delivery of the program, reduced flexibility and increased uncertainty
among partners. Project proponents indicate a preference for streamlining the
application and approval process, for multi-year agreements, and for more
straightforward funding agreements.
3.3
Program Performance: Achievement of Program Outcomes
This section presents the evaluation findings related to the achievement of the program’s
intended outcomes, and the identification of any unintended impacts of GLAP IV.
Information is also provided on the overall importance of the contributions made by
GLAP IV to the restoration and maintenance of AOCs, as well as the identification of
external factors that may be affecting, either positively or negatively, the program’s
activities and outcomes.
37
Treasury Board of Canada Secretariat. 2006. From Red Tape to Clear Results: The Report of
the Independent Blue Ribbon Panel on Grant and Contribution Programs. Ottawa.
38
Environment Canada. May 2009. Grants and Contributions Reform–Presentation to Executive
Management Committee.
Environment Canada
42
Overall Findings
In general, key informants had favourable impressions about the performance of the
program with respect to the achievement of immediate and intermediate outcomes, a
finding that is supported by the program documentation and review of GLSF and federal
project files. As indicated, engagement of partners at the local level and across
jurisdictions and scientific communities is perceived to be a strength of the program. This
is supported by GLAP IV funds allocated to federal partners to facilitate coordination and
management of GLAP IV (e.g., with the province and First Nations). Engagement of
partners and participation at the local level is facilitated through committees, informal
networks and the RAP structure (which also receive GLSF funding support). With
respect to remedial actions, addressing pollution (through sediment remediation, support
to municipal infrastructure improvements, and reducing non-point sources of pollution) is
a significant priority for the program and is allocated almost one half of the GLAP IV
dollars. While the success of federal partners’ efforts is difficult to discern due to gaps in
the federal project files, GLSF projects that supported this objective were found to be
well-documented in terms of outputs and experienced few challenges in implementation.
BUI assessment and monitoring work is occurring on many fronts to assess the status of
BUIs and the effectiveness of restoration activities, and to define BUI goals/targets.
Fewer federal projects were funded for habitat restoration (though this is a particular
focus of GLSF). Habitat projects are more easily documented in terms of outputs (e.g.,
plantings, wetland acreage), but these projects are also more subject to implementation
challenges. Overall, program investments advance activity in each AOC to some degree
(though some coordination projects target all AOCs generically). Note that while group 1
AOCs (those closest to delisting) were intended to be assigned a higher priority for
GLAP IV investments, the number of projects funded in these group 1 AOCs is lower
than in the group 2 AOCs.
Achievement of the program’s longer-term outcomes is much less evident and the
majority of BUIs that were originally identified in AOCs still exist. Of the over 100 BUIs
identified, only 20 per cent have been restored. In consideration of the status of the
BUIs, the program has identified outstanding priority actions for each AOC and projected
timelines for delisting. Only two AOCs have been delisted and one AOC designated an
area in recovery, and these occurred prior to GLAP IV. The original program goal of
completing federal actions in seven group 1 AOCs has not been achieved. Three AOCs
are expected to be delisted or designated as Areas in Recovery within the next 1–2
years, and a fourth, Wheatley Harbour, has just been delisted (announced on April 16,
2010). With respect to the remaining AOCs, the picture is much more complex and the
time frames for delisting are longer-term, with most of these AOCs estimating delisting
as occurring between 2015 and 2020. For committee and expert key informants, the
overall importance of the contribution made by GLAP IV to the restoration and
maintenance of AOCs is viewed as being very significant, if not critical.
For key informants, positive external factors that support program success include:
leveraged funding, particularly infrastructure funding; U.S. investments; and
community/political engagement (for example, the Great Lakes St. Lawrence Cities
Initiative). External factors that interviewees identified as having the potential to
negatively affect the success of GLAP IV include: new and changing ecosystem issues,
Environment Canada
43
and the economic downturn. Unintended outcomes of GLAP IV cited by key informants
tend to be positive, and focus on unexpected interest and engagement of non-targeted
groups (e.g., landowners) and the general public as well as on unanticipated
opportunities for collaboration and knowledge transfer.
3.3.1
Program Outcomes
Evaluation Issue
To what extent have
intended outcomes been
achieved as a result of the
GLAP IV?
Indicator(s)
›
›
›
Evidence of / views on
intended output and
outcome achievement
Evidence of / views on
factors outside the
program that have
influenced the
achievement of intended
outcomes
Methods
›
›
›
Document review
Key informant
interviews
Rating
Progress made,
attention needed
File review
Views on the extent to
which intended outcomes
have been achieved as a
result of the program
Immediate and Intermediate Outcomes
The extent to which GLAP IV has contributed to intended outcomes was explored in key
informant interviews, as well as the review of program documents and the file review.39
In general, key informants who were asked about program-level outcomes had
favourable impressions of the performance of the program regarding shorter-term
outcomes. Building on the achievements of previous GLAP programs, much work has
already been accomplished in identification and characterization of problems in AOCs.
The assessment of progress toward achievement of intended shorter-term outcomes
identified in the program logic model is based on the perceptions of key informant
interviewees, as well as the program documents and files. While descriptions of and
allocations to federal projects have been reviewed (i.e., from the original approved
work plans), there is limited evidence on the outputs and success of these efforts.
Evidence related to the achievement of each of the program’s immediate and
intermediate outcomes is as follows:
›
More effective and better-integrated remedial actions in AOCs. GLAP IV has
supported effective and integrated remedial actions in AOCs in a number of
ways:
◊ RAP committees, as the basis for links between the federal level and
community/provincial stakeholders, ensure that environmental needs at
the AOC level are addressed. The committees work within the RAP
39
Interviews with federal partners and RAP committee members focused on success relating to
specific program outcomes, while interviews with GLSF proponents, experts and committee
members focused on success in more general terms.
Environment Canada
44
reporting framework as outlined in the GLWQA. While the RAP
committees predate GLAP IV, the program continues to support the
committees in AOCs where there is interest and capacity. According to
key informants, the structure of RAP committees varies across the AOCs,
as does their level of activity and engagement. In some AOCs (e.g.,
northern Lake Superior locations), some RAP committees were inactive
for quite some time. In other locations such as the St. Lawrence and Bay
of Quinte, the RAP committees are very active and integrally involved in
remedial actions.
◊ GLSF program officers and other federal partners (depending on the type
of expertise required) participate on RAP committees and play a
leadership role in those AOCs where the federal level is identified as the
lead.
◊ Each year, a portion of GLSF funds is used to support RAP committees.
For example, in 2008–09, $180,500 in contribution dollars (nine per cent
of GLSF funds in that fiscal year) was allocated to AOC RAP–related
coordination and governance in four AOCs. In 2007–08, $470,500 (18 per
cent of GLSF funds in that fiscal year) was allocated to governance
activities in six AOCs.
◊ Nearly one in ten GLSF-funded projects reviewed focused on
management/coordination, and of these, two thirds provided evidence of
output and outcomes achievement.
◊ Among federal projects, ECB was allocated $318,000 annually for RAP
coordination. This activity included “collaborative action among
government, organizations and basin residents …[leading to] functioning
implementation frameworks.”
◊ In addition to RAP committees and as noted by key informants, effective
and integrated remedial actions are facilitated through the work of formal
committees (e.g., the COA Annex Implementation Committee) and
informal networks of technical and scientific professionals working in the
Great Lakes.
›
Improved identification of environmental problems and progress in AOCs.
Identification of environmental problems in the AOCs has largely been
accomplished in previous GLAP iterations, through the development of the
Stage 1 RAP reports. However, additional work has been carried out in
GLAP IV to update the status of environmental problems in selected AOCs
and further characterize their nature through research (e.g., characterization
of sediment in Randle Reef). A noted benefit of GLAP IV funds is the ability to
expand problem identification and monitoring beyond narrow municipal
boundaries in order to undertake more comprehensive, system-wide
assessments. Ongoing monitoring of the status of BUIs is an important
component of work under GLAP IV:
◊ This is an area where federal partners contribute greatly. For example,
among the projects recommended for funding, approximately $11 million
was allocated to BUI assessment and monitoring.
Environment Canada
45
◊ In the sample of GLSF files that was reviewed, almost six in ten files had
objectives related to assessment and monitoring of water quality (26
per cent), contaminated sediment (23 per cent), or ecosystem health
based on BUIs (10 per cent). Outputs are moderately well-documented in
the files.
›
Improved management and coordination of efforts to restore and maintain the
Great Lakes Basin ecosystem. As indicated, engagement of partners was
identified as a strength of the GLAP model. A committee structure predates
GLAP IV, but has continued under this iteration of the program, including
binational (GLWQA and the Great Lakes Binational Executive Committee,
coordination in binational AOCs) and federal/provincial coordination through
the COA and associated committees. Other federal-partner efforts in the area
of management and coordination included the following:
◊ ECB was funded for management and coordination of GLAP IV, which
involved activities and reporting to ensure that federal actions are
implemented in coordination with initiatives of other organizations and to
facilitate collaboration with stakeholders (e.g., $207,000 was allocated
annually to activity coordination and annual reporting on the
implementation of COA annexes, $3,000 annually to collaborate with First
Nations communities, and $57,000 annually for meetings between federal
and provincial agencies on water initiatives).
◊ Federal efforts (led by ECB) were dedicated to the development of tools
and methods that could be used across AOCs to facilitate consistent
monitoring. Just under $1 million was allocated over GLAP IV’s five-year
term to “coordinated and efficient federal/provincial scientific monitoring,
including leading collaborative efforts and technology transfer.” This
included development and implementation of monitoring plans using
consistent delisting criteria, collection of inter-agency–compatible
environmental quality information, and adoption of common protocols
among agencies for water quality assessments.
›
Pollution from identified sources is minimized or eliminated in AOCs.
Sediment remediation is an important focus of GLAP IV, with $13.7 million
initially allocated to this priority. This includes federal projects dedicated to
sediment remediation (e.g., contaminated sediment risk-management
assessment, and studies and plans for harbour sediment remediation). A
small number of GLSF projects in the file review addressed non-point
sources of pollution. With respect to industrial pollution and wastewater
management, GLAP IV plays a supportive, though important, role:
◊ Almost $6 million over the five-year GLAP IV term was dedicated to
“support to municipalities to implement RAP infrastructure
recommendations for sewage treatment, combined sewer overflows and
stormwater management.” The nature of support includes scientific and
technical studies to position municipalities for infrastructure funding.
◊ In the review of the sample of GLSF files, municipal infrastructure support
projects represented 26 per cent of reviewed files. The GLSF funds
Environment Canada
46
projects to support municipal applications for infrastructure improvements
to wastewater treatment plants, sediment identification, and evaluation of
innovative, cost-effective technologies. Outputs for these projects were
found to be well-identified in the file review, with few challenges in
implementation.
›
Habitats in AOC ecosystems are restored. Just over $7 million was dedicated
in the original work plan to “… rehabilitation of fish and wildlife habitat through
the development and implementation of rehabilitation strategies and fish
management plans. Activities include wetland creation/enhancement, coastal
and stream rehabilitation, fish barrier removal, colonial water birds,
stewardship, riparian and upland plantings, project management and
monitoring, and protection through incorporation of strategies into municipal
operating procedures.” (From the GLAP IV Workplan.)
◊ A large portion of these funds are allocated through the GLSF.
Approximately half of the projects included in the review of the sample of
GLSF files had objectives related to habitat restoration (e.g., information
for municipalities on how best to protect and restore their habitats, and on
shrub and tree planting, wetland creation, and shoreline stabilization).
The vast majority of these projects (over 80 per cent) were able to
demonstrate project outputs in the file documentation, though habitat
projects were also more apt to suffer challenges in implementation (e.g.,
due to weather-related issues or the voluntary nature of landowner
commitments).
◊ Funds were also made available to federal projects to support the
development and demonstration of new methods and technology transfer
across AOCs and regions, and to support collaboration with local
implementation agencies and OGDs.
›
Activities of federal partners and stakeholders advance remedial actions,
monitoring, outreach and engagement in each AOC. Activities are occuring
on many fronts, and the program tries to ensure some progress is made in all
AOCs. This is confirmed by the GLSF file review: in only one AOC, Jackfish
Bay, was no project funded. Group 1 AOCs accounted for 21 per cent of
projects that were reviewed using the COA definition and 43 per cent of
reviewed projects when using the work plan review team definition. Similarly,
for federal projects, many cut across several or even all AOCs and, as a
result, federal efforts support all AOCs to some extent. In addition, GLSF
projects have engaged numerous partners, with each project having an
average of six partners (including First Nations, community and academic
groups, and the private sector, among others). Federal partners have also
engaged other stakeholders such as provincial and U.S. counterparts and
other government departments, to steer their efforts toward program
priorities.
Long-Term Outcomes
The long-term outcomes for the GLAP IV are that:
Environment Canada
47
›
›
beneficial uses are determined to be unimpaired and AOCs are delisted; and
Canada’s international commitments related to Great Lakes AOCs are met.
Restoration of BUIs and Delisting of AOCs
Achievement of the program’s longer-term outcomes was an area where the CESD
(2008) was critical of the program: “After more than 20 years, only 2 of Canada’s original
17 areas of concern have been delisted [Collingwood Harbour and Severn Sound]—the
latest in 2003. Priority actions have been completed for one other area, which is now
recognized by the government as an ‘area in recovery [Spanish Harbour].’ The majority
of impairments to beneficial uses that were originally identified in areas of concern still
exist today.”
Current program data indicate that while many BUIs (21) are assessed as being
restored, many more remain impaired (over 80 BUIs across the 15 AOCs) (see Annex I,
under separate cover).40 However, a simple binary restored/impaired measure is not a
highly sensitive measure of progress toward restoration and, in some cases, BUIs in
various areas may be in the process of being monitored to assess impairment (i.e., to
update information on extent/nature of impairment).
The status of BUIs is indicative of overall progress toward delisting of AOCs. Recall that
the goal of GLAP IV was to complete federal actions in group 1 AOCs (closest to
delisting). Of these seven originally identified AOCs, only three are expected to be
delisted or designated an area in recovery by the end of GLAP IV.
Using the work plan Review Committee group 1 priority grouping, three of the seven
group 1 priority AOCs were expected to be delisted or designated areas in recovery
within the next 1–2 years: Wheatley Harbour has just been delisted (announced on
April 16, 2010); Nipigon Bay is scheduled for delisting in 2011 or 2012, pending
infrastructure upgrades; and the St. Lawrence River (Cornwall) AOC has reached the
Stage 3 RAP reporting phase and is also scheduled for delisting in 2010 (see Annex I,
under separate cover).
With respect to priority group 2 AOCs, the picture is more complex and the timeframes
for delisting are longer-term. Several AOCs have now prepared Stage 2 RAP reports or
updates (e.g., Detroit River, Niagara River). There are a cluster of AOCs such as
Hamilton Harbour and Toronto and Region that require large infrastructure projects to
address BUIs in the area. Finally, areas such as St. Mary’s River continue with
assessment, monitoring and community engagement (Annex I, under separate cover).
While restoration of BUIs and delisting of AOCs has been slower than anticipated, key
informants consistently noted the significant challenges of ecological restoration work:
heavily degraded areas such as city harbours require expensive infrastructure solutions;
ecosystems such as the Great Lakes are complex and dynamic, requiring significant
time to diagnose problems, identify solutions and test their effectiveness; and as
40
In 2005, there were 98 BUIs.
Environment Canada
48
scientific knowledge evolves, new problems or the need for new solutions are often
revealed.
Key informants also highlight the notable successes in several of the AOCs (e.g.,
Wheatley Harbour and the St. Lawrence River), and the overall importance of the
contribution made by GLAP IV to the restoration and maintenance of AOCs. For
example, experts and committee member respondents note that GLAP IV funds “focus
attention on AOCs,” and they generally underscore that federal involvement through
GLAP IV funding and scientific expertise is a critical catalyst, stating that “there wouldn’t
be any movement without it”, or that “any holdback now would kill momentum” and that
“there would have been [continued] degradation if not for GLAP.”
Respondents also noted that despite the critical role played by GLAP IV in the
restoration of AOCs, it is not an exclusive one. In monetary terms, GLAP IV funds
represent a relatively modest contribution in comparison to larger-scale infrastructure
investments. In 2003, the Government of Canada estimated that wastewater and
infrastructure improvements across the AOCs would cost approximately $2 billion.
Meeting Canada’s International Commitments Related to Great Lakes AOCs
The IJC assists with implementation of the GLWQA, which sets out a series of
commitments for Canada and the United States—including a requirement that both
countries take remedial action in the heavily degraded AOCs. Using the RAP reporting
process, Canada and the United States, as parties to the Agreement, are directed to
classify AOCs by their stage of restoration, progressing from
›
›
›
›
the definition of the problems and causes (Stage 1 RAP),
the selection and implementation of remedial measures (Stage 2 RAP),
the monitoring of recovery, and
when, identified beneficial uses are no longer impaired and the area is
restored, the removal of its designation as an Area of Concern (Stage 3
RAP).
RAPs were developed for all AOCs in the late 1980s when the GLWQA was revised.
RAPs are submitted to the IJC for review and comment at three stages. The IJC also
adopted an initiative involving status assessments to examine progress toward
restoration of beneficial uses in individual AOCs, in an effort to enhance the restoration
process. To date, five AOCs remain at the Stage 1 RAP level, though status
assessments have been completed for the Detroit River and St. Mary’s River AOCs.
Stage 2 RAPs have been submitted to the IJC for eight Canadian AOCs and a Stage 3
RAP has been submitted for review for one AOC, Wheatley Harbour (Table 3.2).
Environment Canada
49
Table 3.2: Summary of Remedial Action Plan Status
Stage 1
Jackfish Bay
Detroit River
St. Mary’s River
Peninsula Harbour
Thunder Bay
Stage 2
Bay of Quinte
Toronto and Region
St. Clair River
Spanish Harbour
St. Lawrence River (Ontario)
Niagara River (Ontario)
Hamilton Harbour
Nipigon Bay
Stage 3
Wheatley Harbour (now delisted,
April 2010)
Source: www.ijc.org/rel/boards/annex2/rap_process.htm
External Factors
Interview respondents across all categories were asked to identify any external factors
that might have a positive or negative impact on the restoration and/or maintenance of
AOCs. Positive external factors that were seen to have supported program successes
include:
›
›
›
External funding sources: Several interview respondents note that the
availability of infrastructure funding (e.g., through Infrastructure Canada and
municipalities) is critical to improvements in many AOCs, addressing costly
upgrades such as improvements to sewage treatment plants (e.g., Nipigon
Bay, Cornwall). Conversely, in locations where municipalities are unable to
secure sufficient funding or cannot meet the matching funding requirements
of infrastructure programs (e.g., Hamilton), capital projects cannot proceed.
Also, leveraging of funds from external partners through GLSF projects was
noted as a positive factor that enhances the scope of activities accomplished
in AOCs. Partner relationships can also leverage technical expertise as well
as public focus and engagement.
Community/political engagement: A number of interview respondents noted
that community support for restoration efforts is a factor that has the potential
to affect success. As an example, restoration efforts in Hamilton Harbour
have led to a rejuvenation and renewed use of the waterfront by residents,
which has in turn increased public interest in and support for restoration
efforts. Another example of the positive impact of community engagement
identified by several committee members is the Great Lakes–St. Lawrence
Cities Initiative. This initiative, which involves Great Lakes and Seaway city
representatives, was identified as a positive external factor that has resulted
in discussions between this group and the provincial government on water
conditions and issues of particular interest to the group (e.g., algae growth,
beach foulings, sewer discharge and naturalization of the shoreline).
U.S. investments: A number of interview respondents note that recent funding
announcements by the U.S. government for restoration and maintenance in
Environment Canada
50
the Great Lakes may influence Canada to similarly continue and/or increase
its own investments.41
External factors that may have had a negative impact on the success of GLAP IV
include:
›
›
›
New challenges: Several interview respondents note that the Great Lakes are
a dynamic ecosystem that is constantly changing, with new issues cropping
up unexpectedly—such as new invasive species (e.g., Goby fish, zebra
mussels) and the impact of climate change. These changes may alter targets
or introduce new challenges, requiring flexibility in the approach to restoration
and/or maintenance.
Lack of public engagement: While public support is identified as a positive
factor influencing success, limited communications and weak public support
or engagement is similarly identified as a negative factor by several interview
respondents. These interview respondents identify a need for greater
communications to the public on the remediation that needs to be done,
efforts currently under way, and value/impacts of this work. These
respondents believe that increased public support may also translate into
enhanced political will and action.
Economic downturn: The economic downturn was cited by a few interview
respondents as having a negative impact on GLAP IV efforts, by making it
more difficult to obtain partners and financial support for projects.
Other challenges or negative external factors identified by respondents include the
challenge of having a number of policy renewal activities (i.e., for the GLWQA and
COA), which diverts attention from implementation work; and the challenge of restoring
and delisting binational sites, due to the jurisdictional complexity and to the generally
slower pace of work to date on the U.S. side of the Detroit River.
The GLSF file review also revealed that, on a small number of projects (six), challenges
to implementation were identified. These challenges primarily involved project delays,
due to issues such as staff turnover, site conditions, and reliance on volunteer effort.
Summary: In general, key informants had favourable impressions on the
performance of the program with respect to shorter-term outcomes, and this is
confirmed by the program documentation and files. Engagement of partners and
participation at the local level, a perceived strength of the program, is advanced
through various projects to support overall management and coordination, as well
as the RAP committee structure. Addressing pollution (through sediment
remediation and municipal infrastructure support) is a high priority for the
program, and there have been some achievements in this area. BUI assessment
and monitoring are ongoing activities that provide scientific support. Habitat
41
In fact, Canada has renewed its commitment to GLAP in Budget 2010 for $8 million per year
(http://www.budget.gc.ca/2010/pdf/budget-planbudgetaire-eng.pdf p106)
Environment Canada
51
restoration is a somewhat lower priority. Priority funding for group 1 AOCs is not
currently reflected in the number of projects funded in these AOCs.
Achievement of the program’s longer-term outcomes is much less evident, as the
majority of impairments to beneficial uses that were originally identified in AOCs
still exist. In consideration of the status of the BUIs, the program has identified
outstanding priority actions for each AOC and projected timelines for delisting.
Three AOCs are expected to be delisted or designated as Areas in Recovery
within the next 1–2 years, and a fourth, Wheatley Harbour, has just been delisted
(announced on April 16, 2010). With respect to the remaining AOCs, the picture is
much more complex and the time frames for delisting are longer-term, with most
of these AOCs estimating delisting as occurring between 2015 and 2020. For
committee and expert key informants, the overall importance of the contribution
made by GLAP IV to the restoration and maintenance of AOCs is viewed as being
very significant, if not critical.
For key informants, positive external factors that support program success
include: leveraged funding; U.S. investments; and community/political
engagement. Identified external factors that may negatively affect the success of
GLAP IV include: new and changing ecosystem issues; and the economic
downturn.
Unintended Impacts
Evaluation Issue
Have there been any
unintended (positive or
negative) outcomes?
Indicator(s)
›
›
Presence/absence of
unintended outcomes
Opinions of key informants
on whether unintended
outcomes occurred
Methods
›
›
Document review
Rating
N/A
Key-informant
Interviews
Few unintended impacts were identified, either in key-informant interviews or through the
GLSF file review. Five percent of the projects reviewed in the file review identified
unintended positive impacts (e.g., school-based partnerships that yielded interest in field
courses and the potential to adopt material into the curriculum, as well as opportunities
for additional water sampling due to above-average precipitation).
Most unintended impacts identified by interview respondents are positive and involve
unexpected public interest in AOC projects or interest among the non-targeted
community or groups (e.g., interest and engagement of non-participating landowners in
habitat restoration initiatives), and unanticipated opportunities for collaboration among
partners or stakeholders.
A second unintended impact noted by a small number of key informants was technical or
knowledge transfer to other aquatic ecosystem or other research areas (e.g., tall-grass
seed production and sales arising from habitat work, as well as site habitation data used
to identify potential sites for fish nurseries).
Environment Canada
52
Summary: Unintended outcomes of GLAP IV identified by key informants tend to
focus on unexpected interest and engagement of non-targeted groups (e.g.,
landowners) and the general public, and on unanticipated opportunities for
collaboration and knowledge transfer.
3.4
Program Performance: Efficiency
This section examines the efficiency of GLAP IV activities and delivery, i.e., whether the
program is undertaking activities and delivering products in an efficient manner. These
evaluation findings include observations on resources leveraged through GLAP IV and
administrative costs, as well as perceptions of the efficiency of project implementation
and program delivery mechanisms.
Overall Findings
The cost-efficiency of GLAP IV is difficult to determine with existing information, due to
the unbundling of GLBEI funds and activities and the introduction of a shared,
results-based accountability approach to environmental initiatives. The impact on
financial data availability resulting from changes implemented to Environment Canada’s
resource allocation and accountability structure is that GLAP IV funds are not specifically
coded at the program level and the A-base support allocated to GLAP IV is not known.
Financial analysis of the GLSF program component indicates that for every contribution
dollar, $0.24 is spent on program administration (salary and O&M). This amount
compares favourably to the EcoAction Community Funding Program, which has a
cost-efficiency ratio of $0.39, but unfavourably with the Habitat Stewardship Program
($0.08) or the Invasive Alien Species Partnership Program ($0.13). Like EcoAction, the
delivery of the GLSF involves additional staff responsibilities such as participation in
RAP committees.
For key informants, there is an impression of efficient program delivery both overall and
at the project level, with a number of factors identified that support cost-efficiency at the
program and project levels. Some suggestions to improve efficiency were nonetheless
received.
Environment Canada
53
Evaluation Issue
Is GLAP IV undertaking
activities and delivering
products in the most
efficient manner?

How could the
efficiency of the
program’s activities be
improved?
 Are there alternative,
more efficient ways of
delivering the
program?
Indicator(s)
›
›
›
›
›
›
Comparison of program
activities and products
delivered by other similar
programs
Resources leveraged from
GLAP contributions and
their perceived impact on
the program / funded
projects
Methods
›
›
›
›
Document review
Key informant
interviews
Rating
Progress made,
attention needed
File review
Financial analysis
Analysis of actual program
operational costs in relation
to the production of outputs
Views on whether the cost
of producing program
outputs is as low as
possible
Views on how the
efficiency of program
activities could be
improved
Views on whether there are
alternative, more efficient
ways of delivering program
activities and outputs
This section presents evidence related to two indicators of cost-efficiency examined in
this report — leveraging and administrative efficiency — and discusses challenges
associated with each of these analyses.
›
Leveraging
Leveraging refers to the total value of non-federal contributions to GLAP IV projects. A
key challenge related to the calculation of leveraging in the context of GLAP IV concerns
the fact that total project costs and partner contributions have not been updated in the
electronic program database files since 2006–2007.
The file review, using a sample of GLSF files, does provide some information on
leveraged funding for GLSF projects (Table 3.3). There is a wide range in the value of
contributions—from a low of $6,500 to a high of $267,000. Consistent with program
guidelines, Environment Canada contribution funding represents approximately one third
of the total program cost. A portion of files, 29 per cent, was funded at a higher level, but
about half of these are within a few percentage points of the one-third funding ratio and
most of the remainder are projects funding RAP implementation.
Environment Canada
54
Table 3.3: Environment Canada Funding and Leveraged Funding*
Amount of Environment Canada Contribution Agreement
Funding
< $50,000
$50,000–$100,000
> $100,000
Average contribution
Total Amount of Funding / Total Cost of Project
< $100,000
$100,000–$250,000
>$250,000
Average Total Cost of Project
Ratio of Funding to Cost
< 33%
33%
> 33%
Average
* Based on a sample of 39 GLSF project files
Per cent of Files
36%
38%
26%
$76,678
28%
31%
42%
$225,940
56%
15%
29%
32%
Project proponents also noted the importance of GLSF resources as a catalyst for
leveraged funding. Consistent with the importance of collaboration in Great Lakes
restoration and maintenance issues, one key informant confirms that the “federal level
has been particularly effective in providing seed money, for example, GLSF provides 1/3
[funding], as well as a stamp of approval, enabling leveraging from other sources.” RAP
committee key informants identified other ways in which GLAP IV funding served as a
catalyst by noting that “GLAP money has been essential to support the Remedial Action
Planning which involves many stakeholders” and federal involvement is “important in
terms of guidance, experience, coordination, and focusing the efforts.”
Unfunded GLSF applicants were asked to indicate whether their project proceeded
despite the lack of success in obtaining funding. One of three unfunded applicants
interviewed indicates that their project went ahead as planned with several smaller
sources of funding. Two of three unfunded applicants interviewed note that their lack of
success in obtaining funding has set their projects behind; the projects continue but have
been delayed and are proceeding more slowly as they try to secure other funding
sources.
Federal projects were far less likely to have leveraged resources from partners. Of the
21 sampled federal project files, only two indicated that the project involved leveraged
resources from partners. Leveraged resources therefore represent a small fraction of
total project costs (approximately five percent).
›
Administrative Efficiency
Another indicator of program efficiency is derived by examining administrative efficiency
(the ratio of operational costs to program dollars). Again, there are limitations in the
extent to which this analysis can be conducted, due to departmental changes that
Environment Canada
55
occurred during GLAP IV. As indicated, beginning in 2005–2006 the Department
transitioned to a shared, results-based accountability approach to ecosystems initiatives
and unbundled GLBEI funds and activities. The aggregate effect of these developments
was that GLAP IV expenditures were no longer tagged or coded at the program level by
Environment Canada partners. Accordingly, total expenditures and salary, as compared
to project or contribution dollars, are not available for GLAP IV overall.
This particular cost-efficiency analysis can therefore be conducted only for the GLSF.
However, within this program there have been changes as well, including a merger
between the GLSF and the Restoration Program divisions and a reduction in overall
contribution dollars, thus making expenditure analysis a challenge.
Based on currently available information (Table 3.4), GLSF project funding over four
years totalled $12.66 million. A significant reduction in the annual funding amounts and
number of projects funded annually after 2005–2006 is noted. In total, 223 projects were
funded, with the average cost per project being $57,478. With regards to cost-efficiency,
for every contribution dollar an average of $0.24 is spent on salaries and O&M for the
program. This is lower than the EcoAction Community Funding Program, which has a
cost-efficiency ratio of $0.39 but is an intensive community-based delivery model. The
GLSF ratio is higher than the Habitat Stewardship Program ($0.08) and the Invasive
Alien Species Partnership Program ($0.13). The comparison with EcoAction is likely
more appropriate given that GLAP IV staff also undertake significant community-based
work (e.g., participation in RAP committees, partnership development).
Table 3.4: GLSF Cost-Efficiency Analysis*
2005
$196,000
$653,038
$5,423,936
95
$59,094
16%
2006
$120,000
$842,000
$2,566,283
50
$51,326
37%
2007
$120,000
$587,821
$2,624,950
40
$65,623
27%
2008**
$162,500
$398,000*
$2,047,015
38
$53,869
27%
O&M
Salaries
Project Funding Total
No. of Projects
Avg. Funding/Project
Avg. Admin+Salaries as %
of Project Funding
* Final expenditure figures for 2009–2010 were not available at the time this report was being prepared.
** Estimate only, as the Great Lakes AOC Office and Remediation Program Division were merged in 2008–
2009.
Program managers and federal partners had favourable opinions about the efficiency of
the program. When asked to elaborate on factors supporting program efficiency,
interviewees noted the following:
›
›
GLSF leveraging. The file review indicates that, on average, GLSF funding
constitutes 32 percent of overall project costs, consistent with program
guidelines. Leveraging of volunteer efforts is also notable.
Continued directed call for GLSF proposals and sharpened focus on
“essential-to-do” projects.
Environment Canada
56
›
Science and monitoring work that is coordinated among provincial, state and
federal counterparts.
On the other hand, respondents were also asked to discuss factors that undermine
program efficiency. The following issues were identified:
›
›
›
The cumbersome nature of the initiative requires coordination efforts, due to
the multiple jurisdictions, many players (within Environment Canada, and the
number of OGDs), many AOCs and RAP committees, and a complex
ecosystem.
Departmental transformation that occurred during the program created some
disarray in the implementation of GLAP IV and consequent difficulties in
monitoring the activities of federal partners and program financial
expenditures.
A lack of a clear management mechanism for ongoing work planning and
reporting.
Key informants’ suggested means of improving cost-efficiency focused on increasing
communication (external and internal) and coordination to increase partner and public
engagement in the restoration of AOCs, and to ensure that priority needs in the AOCs
(remediation, science, monitoring and engagement) are cohesively linked to work
planning. In addition, the ability to hire a stable core of staff was noted as a way to
improve program efficiency by enhancing continuity of scientific/technical teams and
reducing management time spent on human resources issues.
At the project level, federal and GLSF project proponents were asked about the extent to
which their projects’ activities and outputs were delivered in the most efficient manner.
Respondents generally agreed that their projects were efficient, citing lean operations,
collaborations, leveraging of funds from external partners and volunteer efforts. Any
impediments to project efficiency were either described as events and constraints
beyond the control of the proponents (e.g., staff turnover, weather interfering with
fieldwork) or issues pertaining to the GLSF application and funding process. In the case
of the latter, a few proponents stressed that constraints, such as the absence of
multi-year funding and the delays in funding approval, present challenges to carrying out
projects at peak efficiency.
With respect to GLSF project implementation, the most frequent suggestion to improve
cost-efficiency concerned the GLSF application process and approval times. This
process was described as “lengthy” and, in the words of one respondent, involves “a
great deal of red tape for both government and the proponent” and would require
“streamlining the bureaucratic hurdles” in order to improve efficiency. Related to this, a
few respondents pointed out that more could be accomplished within the same project
(and thus greater efficiency could be achieved) if project approvals were better aligned
with the seasonal nature of many projects and/or if agreements were multi-year. Other
suggestions included: adopting a highly directive approach to funding GLSF projects to
ensure that projects are tightly linked to delisting; and continuing to seek avenues to
leverage funding (e.g., private donations, graduate student participation in projects).
Environment Canada
57
Summary: The cost-efficiency of GLAP IV is difficult to determine with existing
information, due to the departmental transformation that led to the unbundling of
GLBEI funds and activities and a loss of program control over activities and
funding. Program financial data is limited, as GLAP IV funds were not specifically
coded at the program level.
Cost-efficiency analyses were conducted for the GLSF program component only.
This analysis indicates that for every contribution dollar, $0.24 is spent on
program administration (salary and O&M). This amount compares favourably to
the EcoAction Community Funding Program, which has a cost-efficiency ratio of
$0.39, but unfavourably with the Habitat Stewardship Program ($0.08) and Invasive
Alien Species Partnership Program ($0.13). The GLSF is more comparable to
EcoAction given both programs’ more intensive involvement with project delivery
(i.e., with EcoAction project proponents or with RAP committees for GLSF).
For key informants there is an impression of efficient program delivery both
overall and at the project level. Factors identified as supporting program
efficiency include: GLSF leveraging, continued directed calls for GLSF proposals,
and coordination of science and monitoring activities with provincial and U.S.
counterparts. Factors cited as undermining program efficiency refer to the
complexity of the initiative (cross-jurisdictional nature, number of partners and
AOCs involved, and complexity of the lakes’ ecosystems); the departmental
transformation; and the lack of a clear management mechanism for ongoing work
planning and reporting. At the project level, lean operations, collaborations, and
leveraging of funds and volunteer efforts were viewed as contributing to
efficiency, while impediments to efficiency arose from unanticipated events (staff
turnover, weather) and inefficiencies in the GLSF application and funding process.
3.5
Performance: Program Economy
This section explores the extent to which GLAP IV is achieving its intended outcomes in
an economical manner. The evaluation findings presented include the perceived
cost-effectiveness of GLAP IV and key informant opinions on alternative, more
cost-effective approaches to program delivery.
Environment Canada
58
Overall Findings
There is consensus among key informants that GLAP IV is a good investment of public
funds and that results are achieved in an economical way. Design aspects that are seen
as contributing to the program’s cost-effectiveness include: a focused effort on AOCs;
cost-sharing among jurisdictions and key stakeholders; local involvement and delivery;
and continuity of intervention and knowledge acquisition over successive GLAP
programs. A few key informants also noted the economic benefits of remediation, as well
as the cost of not intervening.
No viable alternative approaches to achieve the same or better outcomes at a lower cost
were identified. The prevailing sentiment among key informants is to “stay the course” in
that GLAP provides a solid foundation for work in the AOCs. Fundamental changes to
the program were discouraged by key informants as they may jeopardize the impact of
work completed to date.
Evaluation Issue
Is GLAP IV achieving its
intended outcomes in the
most economical manner?

Are there alternative
program models that
would achieve the
same expected
outcomes at a lower
cost?
Indicator(s)
›
›
›
3.5.1
Extent to which program’s
intended outcomes have
been achieved at the
lowest possible program
cost
Views on whether good
value is being obtained
with respect to the use of
public funds
Methods
›
›
›
›
Document review
Rating
Achieved (~)
Performance data
analysis
File review
Key informant
interviews
Evidence of / views on
whether there are
alternative program
models that would achieve
the same expected
outcomes at a lower cost
Cost-Effectiveness
The issue of economy concerns whether the achievement of program outcomes occurs
in an economical manner and whether there are alternatives to the current program that
could achieve the same outcomes at a lower cost. For GLAP IV, it is not possible to
quantify total program expenditures due to the absence of financial tracking information.
In addition, there is no program point of comparison to assess the relative effectiveness
of the program in comparison to other approaches. The analysis is therefore based on
key informants’ subjective views on the cost-effectiveness of the program.
There was a consensus among key informants across all respondent groups that
GLAP IV is a good investment of public funds. While there were some challenges in the
implementation of the program (described previously), the majority opinion among key
Environment Canada
59
informants was that these challenges did not detract from the overall effectiveness and
cost-effectiveness of the program.
Aspects of GLAP IV that were highlighted by key informants as contributing to
cost-effectiveness include the following:
›
›
›
›
›
Focused effort: investments in AOCs were perceived to maintain the focus of
all levels of government on areas of joint priority, and the framework and
science-based approach for selection and implementation of remediation
work supports this targeted approach. This is particularly important when
there are finite resources, so that available funds can be focused for
maximum impact. GLAP IV’s renewed emphasis on “essential-to-do”
remedial as well as scientific and monitoring work to delist AOCs was also
noted by several key informants as a positive development to increase
effectiveness.
Sharing of costs: shared jurisdiction and provincial COA funds, as well as
leveraged funding from external partners through the GLSF (e.g., municipal,
industry, foundations), increased the positive impact of the federal
investment.
Local involvement: GLAP IV’s value further lies in fostering partnerships and
collaborations among many partners, which leverages community resources
(e.g., volunteers). As well, the RAP committee structure encourages local
involvement and the tapping of resources “on the ground” which was
perceived to increase effectiveness and efficiency.
Longevity: the program draws on scientific and technical expertise, from
federal and provincial jurisdictions, that has been built over the course of
successive GLAP programs. This level of continuity and knowledge was
perceived to contribute to the overall effectiveness of current efforts.
Return on investment: noted by a small number of interviewees are the
broader benefits of Great Lakes cleanup, such as for tourism and industry,
which contributes to the overall value of the GLAP IV investment. An example
of this is a benefits assessment of Randle Reef sediment remediation in
Hamilton Harbour conducted for Environment Canada in 2006. The
assessment identified a number of environmental, economic and social
benefits of remediation and assigned quantified accumulated benefits to each
beneficiary. The result was an estimated cumulative total benefit for the local
area of $126 million during the period 2007 to 2032 with implementation of
the Randle Reef project only, and $914 million with full implementation of all
remediation projects (totals are approximate, to provide an indication of how
results might be used in a social cost-benefit analysis).
Summary: There is consensus among key informants that GLAP IV is a good
investment of public funds and that results are achieved in an economical way.
Design aspects that are seen as contributing to the program’s cost-effectiveness
include: a focused effort on AOCs; cost-sharing among jurisdictions and key
stakeholders; local involvement and delivery; and continuity of intervention and
Environment Canada
60
knowledge acquisition over successive GLAP programs. A few key informants
also noted the economic benefits as well as the cost of not intervening.
3.5.2
Alternative Approaches
Key informants proposed few alternative approaches to the GLAP IV model that would
achieve program outcomes at a lower cost. One potential alternative to GLAP IV is the
program delivery model used in U.S. AOCs. In the United States, the federal jurisdiction
transfers funds to the state level for program implementation and there is a greater
reliance on contracted research conducted by universities. This model, while feasible in
the Canadian context, was not viewed as desirable by the small number of key
informants who discussed the option, including provincial representatives. The transfer
of federal funds to other jurisdictions or external organizations was perceived to
undermine control over the funding, as well as federal visibility. The continuity of
scientific expertise and leadership of the federal government were also viewed as
contributing greatly to the success of Canada’s efforts in the AOCs. As well, a model that
involves contracting-out aspects of work was perceived to increase, not reduce, costs.
Another view, not typically raised as an alternative per se but worth noting here, is the
preference expressed by several key informants in various respondent groups to take a
broader approach to addressing degradation in the Great Lakes beyond the narrow
boundaries of the designated AOCs. These respondents believe there is a need to
expand the scope of GLAP priorities (and resources) in order to increase
responsiveness to nearshore or lake-wide issues and to promote a more holistic
systems-wide approach to addressing remediation of the Great Lakes.
With finite resources, however, key informants were far more likely to suggest a stay the
course approach, and generally approved of the GLAP model to restoring AOCs and
meeting Canada’s commitments under the GLWQA and COA. (Some design and
delivery improvements, discussed above, were preferred over potential alternative
approaches.) In defence of the current model, key informants noted that there is 20
years of experience with GLAP IV and its predecessors, which has built a foundation of
expertise, partnerships and supporting mechanisms (e.g., the GLSF funding program
and agreements with other jurisdictions) to move AOCs toward delisting. Moreover,
many key informants believe that GLAP investments are now beginning to reach fruition,
with several AOCs poised to be delisted or designated an area in recovery.
Summary: No viable options for alternative approaches to achieve the same
outcomes at a lower cost were identified. The prevailing sentiment among key
informants is to “stay the course” in that GLAP IV provides a solid foundation for
work in the AOCs. Fundamental changes to the program are discouraged by key
informants as they may jeopardize work completed to date.
Environment Canada
61
Audit and Evaluation Branch
Evaluation of the Great Lakes Action Plan IV
4.0 Conclusions
The evaluation of Environment Canada’s GLAP IV focused on the five-year span of the
program, from fiscal years 2005–2006 to 2009–2010. The findings of the evaluation lead
to the following broad conclusions about the relevance, design and delivery, and
performance (effectiveness, efficiency and economy) of GLAP IV.
4.1
4.1.1
Relevance
Continued Need for the Program
1) The evaluation evidence indicates that GLAP IV is an important program to
address federal commitments under the GLWQA and COA with respect to
Great Lakes AOCs. The program is needed to continue to address the
complex and evolving Great Lakes ecosystem. Public opinion supports the
continued need for the program, as Canadians indicate water resources to be
a high priority. The work toward delisting of AOCs is not yet complete and, in
light of finite resources, the use of the AOC designation is a way to focus
resources and attention (with a science-based framework and including local
involvement) on these highly degraded areas that impact the overall quality of
the Great Lakes. The GLAP IV program priority areas continue to resonate
and do not require adjustment.
2) GLAP IV is the only program that focuses on Great Lakes AOCs, acting as a
catalyst for leveraged funding from other sources. While there are other
sources of funding that support remediation as well as science and
monitoring in the AOCs, there was no evidence of risk of duplication or
overlap. This risk is mitigated by coordination at the program-officer level and
through collaborative networks of provincial, federal and U.S. scientists and
technical experts.
4.1.2
Alignment with Federal and Departmental Priorities
3) GLAP IV aligns well with federal and departmental priorities. At the federal
level, a “healthy environment for Canadians,” including clean water, is a
stated priority, and at the departmental level, GLAP IV is consistent with
Environment Canada’s mandate to preserve and enhance the quality of the
natural environment. Moreover, the Great Lakes are identified as a
high-priority ecosystem nationally and for the Department.
4.1.3
Consistency with Federal Roles and Responsibilities
4) The shared boundary of the Great Lakes waters with the United States and
Canada’s commitments under the GLWQA provide a strong rationale for
federal jurisdiction in the AOCs. As well, a number of legislative authorities
Environment Canada
62
(the International Boundary Waters Treaty Act, Clean Water Act and
Canadian Environmental Protection Act, 1999) support federal jurisdiction in
this area.
4.2
Program Performance: Design and Delivery
5) The GLAP IV model is a sensible approach to achieve program outcomes.
Engagement of partners and involvement at the local AOC level are
distinguishing and positive features of the model that leverage expertise and
resources for remediation. The overarching framework for guiding GLAP IV
and the science-based approach are also notable strengths that organize
efforts in the AOCs and maintain the focus on delisting.
6) Delivery of some aspects of GLAP IV is inconsistent with the original design.
The annual work planning process for the program and aspects of the
program’s management framework were not implemented. Therefore, few
levers were available to the program to monitor and ensure that the activities
to achieve the goals of GLAP IV, and the activities for which partners were
funded, were undertaken in a coherent way. The distributed responsibility
model was an identified weakness in other reviews of the program, with a
recommendation to ensure that appropriate resource allocation, capacity and
priority are assigned to meet program goals. Evidence suggests that many or
all of these issues stemmed from the departmental transformation that
occurred in 2005–2006.
7) Resources for funding federal and GLSF partners to undertake priority
activities in AOCs are largely sufficient, with increases needed to address
inflation as well as the declines in funding to the GLSF that followed the first
year of GLAP IV. The effectiveness and efficiency of departmental partners’
efforts was attenuated by a limitation on the ability of Environment Canada
programs to convert O&M to salary dollars, which created challenges in hiring
the human resources required to achieve project objectives.
8) Financial accountability of the program was weak. Following transformation,
program resources were no longer tracked by program but rather by
departmental result. Furthermore, all B-base funding (i.e., temporary funding
received for a specific purpose) was unbundled at this time, and so financial
tracking of GLAP IV funding at the program level by internal Environment
Canada partners was very limited. Some federal partners may not have
uniformly received the GLAP IV funds that were allocated for their approved
work plans.
9) Existing processes for reporting of GLSF-funded projects are quite complete
(though only entered electronically to 2006–07) and are well-understood by
project proponents and managers. Reporting is focused to some extent on
achievement of outputs (as opposed to outcomes, which are far more
challenging to measure). For projects conducted by federal partners,
Environment Canada
63
processes for reporting to the program are not clearly articulated and have
been sporadic over the life of the program.
10) Conducting monitoring and measuring results for program performance
purposes is weak. Program reporting is not currently guided by a
performance measurement framework and no systematic process exists to
aggregate or summarize project-based reporting (costs [including partner
contributions], activities and outputs).
11) The design and delivery of the GLSF component of GLAP IV is generally
satisfactory. The program is longstanding and many proponents are involved
in ongoing projects under this program. GLSF funds are an important catalyst
in leveraging funding from other sources. The deficiencies in delivery of the
program are the delays between proposal submission and funding approval
(due in part to the program transition from the use of MOUs to contribution
agreements as part of a Department-wide initiative to increase
accountability), and the complexity of the application process. As several
AOCs are moving closer to delisting, developing delisting criteria and
clarifying responsibilities and the process for delisting are also emerging
issues.
4.3
Program Performance: Achievement of Program Outcomes
12) GLAP IV has made solid progress in achieving shorter-term outcomes, but
only modest progress toward long-term outcomes that build on the
achievements of predecessor GLAP programs. Only three AOCs have been
delisted and one AOC designated an area in recovery, and these occurred
prior to GLAP IV. Moreover, a majority of the BUIs identified in 2003 remain
impaired. The original program goal of completing federal actions in seven
group 1 AOCs has not been achieved. Still, momentum appears to be
increasing as the program expects that three group 1 AOCs will be delisted
within two years. As well, a host of external factors influence program
effectiveness, including the dynamic and complex nature of the ecosystem,
the program’s dependence on contributions of other partners to achieve its
outcomes, and significant challenges in developing delisting criteria for BUIs
that must be addressed.
13) Unintended impacts of the program are few in number but positive in nature,
such as unanticipated public interest, engagement of partners and technical
transfer.
4.4
Program Performance: Cost-Efficiency
14) Although data related to administrative expenses for the first four years of the
program were available for the GLSF component only (with administrative
Environment Canada
64
costs for 2007–2008 based only on program estimates), these limited data
indicate that efficiency of this component compares reasonably well with
other contribution programs at Environment Canada that require more
intensive staff involvement in program delivery. There are, however, a
number of avenues to improve efficiency at the program level (e.g., improve
strategic management, and coordination among federal partners) and at the
GLSF project level (e.g., streamlined application process).
4.5
Program Performance: Cost-Effectiveness
15) GLAP IV is an appropriate investment of public funds. Results are beginning
to be achieved in the AOCs, though time frames for delisting have been long.
As well, there are some broader economic benefits to restoration of AOCs.
There were no alternative approaches that would result in a more economical
achievement of program objectives.
Environment Canada
65
Audit and Evaluation Branch
Evaluation of the Great Lakes Action Plan IV
5.0 Recommendations
Recommendations for the current or future iterations of the GLAP are based on the
findings and conclusions of the evaluation. Overall, GLAP IV continued to be a relevant
program to address ongoing needs for ecological restoration in the AOCs and is
well-aligned with departmental and federal priorities. At the end of its five-year term,
GLAP IV has not fully achieved its intended longer-term outcomes in the areas of
addressing BUIs and delisting AOCs, due in large part to the complexity and long-term
nature of the task. The following recommendations focus on improvements to the
management of the program in three areas: delivery and oversight; financial
management and accountability; and performance measurement and reporting.
Delivery and Oversight
Oversight of GLAP IV was challenged by several changes that occurred within
Environment Canada during the program’s five-year term. The changes affected both the
overall management and the coordination of the program, and the GLSF. One
recommendation pertains to improving coordination and accountability of the program,
and one pertains to ensuring the timeliness and responsiveness of the GLSF.
1) The Regional Director General (RDG)–Ontario should develop a suitable
instrument to enhance coordination of federal partners involved in
GLAP and the accountability of their efforts toward achieving GLAP
goals. GLAP IV is a complex initiative that requires the contribution of many
federal partners to achieve program goals. With the demise of the work plan
review teams and process, GLAP IV lacked the instruments to ensure that
these efforts were coordinated and that federal partners were implementing
work plans to achieve program goals. The work planning process, convened
annually as it was envisioned, could be re-instituted to enhance coordination
and accountability.
2) The RDG–Ontario should explore means of streamlining both the GLSF
funding approvals process and the GLSF application process. While
mostly satisfied with the GLSF program, proponents identified two issues with
respect to administration: timeliness of approvals and an overly complex
application process. Consideration should be given to scrutinizing the timing
of the funding cycle to ensure that funding approvals coincide with seasonal
activities, and that the application process is streamlined to promote
efficiency (perhaps by exploring adoption/applicability of common application
forms/guidelines being developed as part of the wider federal G&C reform
exercise). Although most contribution agreements are single-year only, the
majority of projects are of an ongoing nature. The use of multi-year
agreements may be appropriate in some of these instances.
Environment Canada
66
Financial Management and Accountability
There is limited information on GLAP IV program expenditures. Financial tracking of
GLAP IV funds was weak during this iteration of the program due to limited use of
codes for expenditures at the program level. One recommendation pertains to financial
monitoring of GLAP funds at Environment Canada.
3) The RDG–Ontario should explore ways to enhance the financial
accountability of the program. The direct transfer of funds to internal
partners and absence of program-specific coding of expenditures within the
Department has weakened financial accountability of the program. Diligent
coding of expenditures at the program level is a key element to ensure
transferred funds are received and utilized within Environment Canada by
departmental partners to meet program goals and priorities. The merit and
potential disadvantages of direct transfers of program funds within
Environment Canada to departmental partners for financial accountability
should be examined.
Performance Measurement and Reporting
GLAP IV monitoring and reporting was not guided by a performance measurement
framework. As a result, a broad spectrum of performance measures for federal partners
and the GLSF were not identified early in the program and reporting requirements,
particularly for federal partners, were not clearly articulated. Five recommendations
pertain to improving performance measurement and reporting.
4) The RDG–Ontario should develop a more regular and robust reporting
approach for GLAP projects conducted by federal partners. Performance
reporting for projects conducted by federal partners was unspecified and
sporadic. A more robust reporting approach should be based on clear terms
and conditions for reporting on funds allocated to federal partners. Reporting
should link activities/deliverables to approved work plans, recognizing that, in
some cases, federal projects are undertaken as part of an integrated science
program.
5) The RDG–Ontario should develop a performance measurement
framework for the program, and include a spectrum of more sensitive
performance measures (shorter- and longer-term). The program does not
have a performance measurement framework and is not currently
well-positioned to demonstrate progress and interim results because
measures such as delisting AOCs are long-term. A more nuanced approach
must balance the benefit of performance measurement with the cost of
monitoring and assessing BUIs in the AOCs. Recent efforts by the program to
document accomplishments and assign roles, responsibilities and timelines
for outstanding actions and priorities are acknowledged as a strong
foundation for this ongoing work.
Environment Canada
67
6) The RDG–Ontario should support the development of information
management tools to enhance implementation and monitoring of GLSF
and federal projects. No systematic process exists for monitoring and
reporting the activities, outputs and performance for GLAP-funded projects or
the program overall. A Department-wide Management Information System for
Environment Canada’s community funding programs, including the GLSF, is
in development, although this system is not expected to be implemented until
2012. Opportunities for interim tracking of activities and project and partner
contributions should be identified.
7) The RDG–Ontario should support the development of informationsharing tools to facilitate broader access to the research generated by
GLAP IV by the scientific community and the public. There is no ongoing
mechanism available for technical transfer of research and results across the
AOCs. This is an outstanding commitment in Annex 4 of the COA.
8) The RDG–Ontario should continue efforts to define delisting criteria for
AOCs and clarify responsibilities with respect to delisting
decision making. Roles and responsibilities for aspects of delisting AOCs
are a grey area and may be variable across the AOCs. As more AOCs
approach restoration, clarifying the steps to delisting will increase in
importance.
Environment Canada
68
Audit and Evaluation Branch
Evaluation of the Great Lakes Action Plan IV
6.0 Management Response
This section outlines the management response to the evaluation recommendations.
The RDG–Ontario accepts the evaluation and all of its recommendations, and the Great
Lakes Division has provided a plan to implement the following management actions in
response to the evaluation recommendations within the context of the program’s
renewal.
1. The RDG–Ontario should develop a suitable instrument to enhance
coordination of federal partners involved in the GLAP and the accountability of
their efforts toward achieving GLAP goals. GLAP IV is a complex initiative that
requires the contribution of many federal partners to achieve program goals. With the
demise of the work plan review teams and process, GLAP IV lacked the instruments
to ensure that these efforts were coordinated and that federal partners were
implementing work plans to achieve program goals. The work planning process,
convened annually as it was envisioned, could be re-instituted to enhance
coordination and accountability.
The RDG–Ontario agrees with this recommendation.
A GLAP five-year work plan will be developed by Environment Canada, Great Lakes
Division, with input from all federal departments and Responsibility Centres within
Environment Canada contributing to the restoration of Great Lakes AOCs and accessing
GLAP funds. The GLAP Work Plan Review Team will be re-established and will annually
conduct a review of progress achieved, identify actions required to complete the
restoration of AOCs, determine priorities for the coming year, and propose adjustment of
the work plan and funding allocations as required. The GLAP Work Plan Review Team
will be led by the Great Lakes Division of Environment Canada and will comprise
representatives of Environment Canada, the Department of Fisheries and Oceans,
Public Works and Government Services Canada and other federal government
departments as necessary. Recommendations of the GLAP Work Plan Review Team
will be presented to the Director, Great Lakes Division, for approval. A GLAP work
planning template will be established specifying the project rationale, (i.e., how the
proposed project contributes to AOC delisting), scope of work, deliverables, schedule
and resource requirements.
Timeline
August 2010
Deliverable(s)
GLAP Work Plan Review Team established
October 2010
Establish GLAP work planning and
reporting template
Five-year work plans for all federal
departments and groups within
Environment Canada accessing GLAP
funds contributing to the remediation of
AOCs
November 2010
Environment Canada
Responsible Party
Director, Great Lakes
Division
Director, Great Lakes
Division
Director, Great Lakes
Division
69
March–April
2011, 2012,
2013, 2015
Review and revision of GLAP Work Plan,
Annual Reports on Results, funding
allocation decision
Director, Great Lakes
Division; and
GLAP-funded groups
2. The RDG–Ontario should explore means of streamlining both the GLSF
funding approvals process and the GLSF application process. While mostly
satisfied with the GLSF program, proponents identified two issues with respect to
administration: timeliness of approvals and an overly complex application process.
Consideration should be given to scrutinizing the timing of the funding cycle to
ensure that funding approvals coincide with seasonal activities, and that the
application process is streamlined to promote efficiency (perhaps by exploring
adoption/applicability of common application forms/guidelines being developed as
part of the wider federal G&C reform exercise). Although most contribution
agreements are single-year only, the majority of projects are of an ongoing nature.
The use of multi-year agreements may be appropriate in some of these instances.
The RDG–Ontario agrees with this recommendation.
Environment Canada’s Action Plan to Reform the Administration of Grants and
Contributions will improve the efficiency of the GLSF application and approval process.
In addition, improvements have been made to the GLSF application and proposal
evaluation processes to simplify and improve the efficiency and consistency of the GLSF
application and approval process. Multi-year agreements will be used where appropriate.
Timeline
June 2010
Deliverable(s)
Adopt application and reporting processes
from the Departmental Action Plan to
Reform the Administration of Grants and
Contributions
Responsible Party
Director, Great Lakes
Division
3. The RDG–Ontario should explore ways to enhance the financial accountability
of the program. The direct transfer of funds to internal partners and absence of
program-specific coding of expenditures within the Department has weakened
financial accountability of the program. Diligent coding of expenditures at the
program level is a key element to ensure transferred funds are received and utilized
within Environment Canada by departmental partners to meet program goals and
priorities. The merit and potential disadvantages of direct transfers of program funds
within Environment Canada to departmental partners for financial accountability
should be examined.
The RDG–Ontario agrees with this recommendation.
The Great Lakes Division will take steps to ensure that departmental resources
expended in the delivery of the GLAP are adequately documented. To this end, the
Great Lakes Division will take the necessary steps to adopt program-specific authority
codes for all Environment Canada internal salary, O&M and G&C expenditures of GLAP
funds. Furthermore, all GLAP funds will be managed by the Great Lakes Division and
Environment Canada
70
distributed to other federal departments and Responsibility Centres within Environment
Canada in accordance with the GLAP five-year work plan annual updates.
Timeline
May 2010
(done)
May 2010
(done)
Deliverable(s)
Request for the establishment of a
GLAP-specific authority code in the
Financial Information System
Establish program-specific authority codes
for GLAP salary, O&M and G&C funds
Responsible Party
Director, Great Lakes
Division
Environment Canada
Finance Directorate
4. The RDG–Ontario should develop a more regular and robust reporting
approach for GLAP projects conducted by federal partners. Performance
reporting for projects conducted by federal partners was unspecified and sporadic. A
more robust reporting approach should be based on clear terms and conditions for
reporting on funds allocated to federal partners. Reporting should link
activities/deliverables to approved work plans, recognizing that, in some cases,
federal projects are undertaken as part of an integrated science program.
The RDG–Ontario agrees with this recommendation.
Environment Canada, Great Lakes Division, will require annual reports on all work plan
activities from Environment Canada and other federal partners participating in the
restoration of Great Lakes AOCs. Reporting information will be considered in the annual
review and revision of work plans. Clear guidelines will be established for monitoring and
reporting of GLAP funds by Environment Canada and federal partners. This will include
the establishment of a short reporting template, including financial reporting, and
pertinent performance information.
Timeline
October 2010
March–April
2011, 2012,
2013, 2015
Deliverable(s)
Establish guidelines for monitoring and
reporting of GLAP funds; GLAP work
reporting template established
An annual report on the status of GLAP
work plan commitments will be prepared
and reviewed through the annual GLAP
work planning process
Responsible Party
Director, Great Lakes
Division
Director, Great Lakes
Division
5. The RDG–Ontario should develop a performance measurement framework for
the program, and include a spectrum of more sensitive performance measures
(shorter- and longer-term). The program does not have a performance
measurement framework and is not currently well-positioned to demonstrate
progress and interim results because measures such as delisting AOCs are longterm. A more nuanced approach must balance the benefit of performance
measurement with the cost of monitoring and assessing BUIs in the AOCs. Recent
efforts by the program to document accomplishments and assign roles,
responsibilities and timelines for outstanding actions and priorities are acknowledged
as a strong foundation for this ongoing work.
Environment Canada
71
The RDG–Ontario agrees with this recommendation.
A performance measurement framework will be developed that will consider feasible
measures to provide a more detailed and incremental assessment of progress toward
restoration of BUIs and delisting of AOCs. An Assessment of the Status of Remaining
Actions to Delist AOCs has been completed and will serve as an activity-based,
short-term performance measurement framework. A Beneficial Use Impairment Status
and Progress Report has also been completed and provides a longer-term, result-based,
performance measurement framework. The Assessment and Report will be reviewed,
revised and reported on biennially in alternating years.
Timeline
March 2011
February 2012,
February 2014
February 2011,
February 2013,
February 2015
Deliverable
Performance measurement framework
developed for the program
Updated Beneficial Use Impairment Status
and Progress Report
Updated Assessment of the Status of
Remaining Actions to Delist AOCs
Responsible
Party
Director, Great
Lakes Division
Director, Great
Lakes Division
Director, Great
Lakes Division
6. The RDG–Ontario should support the development of information management
tools to enhance implementation and monitoring of GLSF and federal projects.
No systematic process exists for monitoring and reporting the activities, outputs and
performance for GLAP-funded projects or the program overall. A Department-wide
Management Information System for Environment Canada’s community funding
programs, including the GLSF, is in development, although this system is not
expected to be implemented until 2012. Opportunities for interim tracking of
activities and project and partner contributions should be identified.
The RDG–Ontario agrees with this recommendation.
Environment Canada’s Action Plan to Reform the Administration of Grants and
Contributions (the Department’s G&C reform initiative) is developing an online
application and information management system for G&C programs. This system will
improve program efficiency, enhance alignment with departmental priorities and improve
the ability to report collectively on the results of departmental funding programs. The
system is scheduled to be in operation by 2011–2012. In the interim, the Great Lakes
Division has established an Excel-based system to track GLSF project proposals,
proposal reviews and selection, project recipients and contribution agreements, and
project financials, products and outcomes.
Timeline
2010
Deliverable
Adopt information management system from
the Departmental Action Plan to Reform the
Administration of Grants and Contributions.
Environment Canada
Responsible
Party
Director, Great
Lakes Division
72
7. The RDG–Ontario should support the development of information-sharing
tools to facilitate broader access to the research generated by GLAP IV by the
scientific community and the public. There is no ongoing mechanism available for
technical transfer of research and results across the AOCs. This is an outstanding
commitment in Annex 4 of the COA.
The RDG–Ontario agrees with this recommendation.
Information on AOCs is provided through the Environment Canada website and other
communications products. Research findings are regularly published in scientific journals
and presented at scientific forums, including the Great Lakes State of the Lakes
Ecosystem Conference and the International Association for Great Lakes Research
Conference.
Additionally, AOC Progress Reports have been developed jointly with the Province of
Ontario for release in 2010. This tool will be built upon and modified to publicly report on
progress regarding remediation of Great Lakes AOCs, and to enhance sharing of
research results and other information. Reporting on AOCs will be implemented on a
three-year cycle, consistent with other Great Lakes reporting.
Timeline
Deliverable
2013
Area of Concern Progress Reports will be
issued every three years and will communicate
research results to the scientific community and
general public
Responsible Party
Director, Great Lakes
Division
8. The RDG–Ontario should continue efforts to define delisting criteria for AOCs
and clarify responsibilities with respect to delisting decision making. Roles and
responsibilities for aspects of delisting AOCs are a grey area and may be variable
across the AOCs. As more AOCs approach restoration, clarifying the steps to
delisting will increase in importance.
The RDG–Ontario agrees with this recommendation.
The GLWQA clearly states that the parties (Canada and the United States) are
responsible for leading all aspects of the AOC process “in cooperation with State and
Provincial Governments.”
Environment Canada’s January 2010 Great Lakes RAP Workshop, which included
federal and provincial representatives involved in Great Lakes AOCs as well as the local
RAP coordinators, addressed the delisting process issue as an agenda item. The
principle outcome was a commitment to develop a Canada–Ontario Guide to the
Designation of Beneficial Use Impairments, Preparing Stage 3 Reports and Delisting
Areas of Concern. The guide, which is being developed collaboratively with the Ontario
Ministry of the Environment, will clarify the responsibilities with respect to delisting
decision making.
Environment Canada
73
Timeline
Deliverable
December Canada–Ontario Guide to the Designation of
2010
Beneficial Use Impairments, Preparing Stage 3
Reports and Delisting Areas of Concern
Environment Canada
Responsible Party
Director, Great Lakes
Division
74