Provincial Government Programs Cabinet Secretariat, Executive Council

Guide to Developing a Risk-Based Departmental Evaluation Plan
PURPOSE OF THIS GUIDE ........................................................................................... 2 INTRODUCTION ............................................................................................................. 3 OVERVIEW ..................................................................................................................... 4 Why do Risk-Based Planning? ..........................................................................................4 Transitional Considerations ...............................................................................................5 When to Start the Planning Process ................................................................................5 GETTING STARTED ...................................................................................................... 8 WHAT GOES IN A PLAN? ............................................................................................. 9 Policy Requirements ............................................................................................................9 Five Step Process ...............................................................................................................10 GLOSSARY .................................................................................................................. 18 DOCUMENTS GLOSSARY .......................................................................................... 21 APPENDIX A: PRE-SCREENING ASSESSMENT TEMPLATE .................................. 26 APPENDIX B: EVALUABILITY ASSESSMENT TEMPLATE ...................................... 27 APPENDIX C: EVALUATION PRODUCT PROFILE ................................................... 28 APPENDIX D: DEPARTMENTAL EVALUATION PLAN .............................................. 29 APPENDIX E: INITIATIVE INVENTORY UPDATE....................................................... 30 APPENDIX F: GOVERNMENT PROGRAM RISK ASSESSMENT TOOL ................... 31
Cabinet Secretariat │ December 2011 1

Guide to Developing a Risk-Based Departmental Evaluation Plan This document guides departments in developing a risk-based Departmental Evaluation Plan using a five step process and provides templates to assist with the formatting of the Plan. The five steps are: 1. Compile List of Departmental Initiatives 2. Pre-screen for Exclusion / Inclusion 3. Assess Risk 4. Assess Readiness for Evaluation 5. Submit Plan - Evaluation Product Profiles The Policy on Evaluation requires that all new and existing government initiatives be assessed for evaluation priority and that those priorities are documented in a Departmental Evaluation Plan. New initiatives are assessed for evaluation priority through the completion of an Accountability Framework
1
as part of the Cabinet submission process. Existing initiatives are assessed for evaluation priority using a risk-based approach as defined in this document. A web Risk-Based Assessment tool developed by the Economic and Statistics Branch (ESB) of the Department of Finance to assist departments in ranking their initiatives from highest to lowest risk. A Guide to using this tool is contained in Annex F. Words or terms defined in the Glossary are bolded in their initial appearance in this document. 1 Developing an Accountability Framework: a Resource and Reference Guide is available through the Provincial Government Programs Office: 729-2590. Cabinet Secretariat │ December 2011 2

Guide to Developing a Risk-Based Departmental Evaluation Plan
The Policy on Evaluation requires Deputy Ministers/equivalents 2 to develop a Departmental Evaluation Plan for approval by the Minister and submission to the Clerk of the Executive Council. On behalf of the Clerk, the Provincial Government Programs Office will review departmental evaluation plans and provide comments to the Clerk on the appropriateness of evaluation coverage including the risk-based approach used to identify evaluation priorities. The Clerk may refer some or all plans to Treasury Board or Cabinet for approval.
POLICY PROCESS
 Deputy Ministers are accountable for
developing and implementing the Plan.
 Ministers approve the Plan.  Clerk may send the Plan to Treasury
Board or Cabinet for approval.
Plans are to be strategically focused and founded on an assessment of risk that the department ’s initiatives (programs and services and major policies) may no longer be relevant, effective or efficient. A department has the choice of developing an annual evaluation plan or a multi-year plan. Multi-year plans are subject to annual review by the department and all plans are to be monitored and updated as appropriate. 2 For purposes of this Guide, the term department refers to any organization identified in Annex A to the Policy on Evaluation and the term Deputy Minister refers to any head of these organizations. Cabinet Secretariat │ December 2011 3

Guide to Developing a Risk-Based Departmental Evaluation Plan
Why do Risk-Based Planning?
The ultimate goal of evaluation is improving client outcomes and ensuring government initiatives provide value for money (i.e. demonstrating effectiveness, efficiency, and economy). However not all initiatives need to be evaluated or can be evaluated in a cost effective manner. The purpose of a risk-based evaluation plan is to ensure evaluation-related activities are strategically integrated into the department's existing planning and performance measurement processes and reflect the broader need to support Cabinet decision-making and budget deliberations. Such planning ensures resources are allocated to evaluation products (includes evaluations as well as evaluation-related activities) that can provide the greatest value-added to a department and Government generally. As the Deputy Minister is accountable for the development and implementation of the Plan, it should inform the Minister and Deputy Minister of the scope of the department's evaluation activities and the limitations of that scope. There should be sufficient information presented in the Plan to help the Deputy Minister determine if the evaluation/evaluation-related activities proposed are sufficient and support the priorities of the Department and of Government. The best evaluation plan is one that directly supports the information or decision-making needs of the Minister, Deputy Minister, Cabinet and Treasury Board. It is more than a list of evaluation-related activities. A good evaluation plan is a tool to:  Engage the Minister and senior executives in identifying the priority evaluation products required to provide them with information necessary to make informed decisions;  Communicate the importance of reliable performance data to support monitoring and evaluation of departmental and government initiatives; and  Identify and secure the resources for completion of the evaluation activities. Cabinet Secretariat │ December 2011 4

Guide to Developing a Risk-Based Departmental Evaluation Plan
Transitional Considerations
The Policy on Evaluation recognizes that departments are not on a level playing field when it comes to the complexity of their programs and their current evaluation capacity. The Policy therefore accommodates the capacity discrepancy by allowing for a transition period to full implementation by March 31, 2014. The composition of a plan is flexible: plans may be annual or multi-year; there is no expectation that all initiatives be evaluated, nor is there an established minimum number of initiatives to be evaluated annually. Deputy Ministers must identify their evaluation priorities and identify what evaluation products will be completed or commenced during the fiscal year, beginning with 2012-13. Evaluation products may include evaluation-related training, terms of reference (TOR) or Requests for Proposals (RFP) for a future program evaluation, accountability frameworks, research and data analysis, evaluation reports and reviews as part of a department’s Regulatory Improvement Plan. The identification of evaluation priorities and the associated resources are the critical elements to be reflected in the Departmental Evaluation Plan. Some departments may not be in a position to conduct any evaluations during 2012-13 and may instead engage in capacity building activities or develop performance measurement plans for priority programs. As part of building capacity, departments are encouraged to pilot at least one internal evaluation in 2012-13.
When to Start the Planning Process
To avoid duplication of effort, the planning process can be integrated with the departmental strategic planning, budgeting and reporting processes. In most cases, the evaluation planning process should start in the Fall and be completed by early Spring for the subsequent year's evaluation-related activities (See Table 1). Initial Departmental Evaluation Plans are to be forwarded to the Clerk no later than April 30, 2012. Departments should identify which evaluation-related activities can be completed using existing resources and what assistance is needed to complete other product(s). Ministerial approval is required prior to seeking incremental resources through the Cabinet Secretariat │ December 2011 5

Guide to Developing a Risk-Based Departmental Evaluation Plan budget process. If such approval is extended, a draft Departmental Evaluation Plan must be included with the annual budget submission.
3 Department’s choosing multi-year evaluation planning shall submit their Plan to the Clerk at the end of April during the year in which the existing plan expires. All plans shall be reviewed annually. Where substantive changes to a Plan are required, the Clerk shall be notified as part of the Deputy Minister’s performance contract process. 3 Refer to the Department of Finance’s annual budget guidelines for process considerations. Cabinet Secretariat │ December 2011 6

Guide to Developing a Risk-Based Departmental Evaluation Plan
Table 1 Central Agency Reporting Schedule Documentation Jun Jul Aug Sept Oct Nov Dec Jan Feb Mar Apr May Transparency and Accountability Plans (every three years) Transparency and Accountability Annual Report * Budget Submissions DM Contract (submission and reporting) ** HR Accountability Report Evaluation Plan (Department)
* If requesting additional resources to implement the proposed Departmental Evaluation Plan, a draft of the plan must be attached to the request and submitted with the budget briefing note. Refer to the Department of Finance’s annual budget guidelines for process considerations. ** Process and timing under review by the Public Service Secretariat Cabinet Secretariat │ December 2011 7

Guide to Developing a Risk-Based Departmental Evaluation Plan
As stated earlier, the best evaluation plan is one that directly supports the information or decision-making needs of the
Potential Information Sources
Minister, Deputy Minister, Cabinet and Treasury Board. For this reason it is advisable to Budget Decisions Budget Monitoring Reports Auditor General Reports Client Surveys or Public Consultations meet with the Minister and Deputy Minister to identify their information needs, priorities, and departmental challenges for the upcoming year. To complete an evaluation plan there are multiple documents that can be helpful in attaining information. Not all documents are necessary as the required information may be accessible in Cabinet or Treasury Board Directions Deputy Minister Performance Contract Department Cabinet Submission Plan Fiscal Framework to identify sun setting funds Ministerial Mandate Letters Operational Plan Prior Year’s Departmental Evaluation Plan Regulatory Improvement Plan Work Plan Departmental Plan (Strategic, Business, Activity) and Annual Report Performance Monitoring and Evaluation Reports Public Strategy Documents Strategic Directions Speech From the Throne and Budget Speeches more than one document. A short description of documents and how they can be accessed is provided in the glossary. Cabinet Secretariat │ December 2011 8

Guide to Developing a Risk-Based Departmental Evaluation Plan
Policy Requirements
In accordance with the Policy on Evaluation, the plan should include:  Specific evaluations or evaluation –related activities (e.g. developing Accountability
WHAT GOES IN A PLAN?
 Evaluations directed by the
Clerk, Cabinet, Federal Provincial Territorial or other Funding Agreements
 Evaluations required to
support horizontal initiatives
 Evaluations required to inform
departmental decision making
 Programs with sun setting
funds
Frameworks) as directed by Cabinet or requested by the Clerk of the Executive Council following consultation with the affected Deputy Minister (e.g. evaluations outlined in the evaluation plan 4 ).  Evaluations required government-wide by inter governmental funding agreements or legislative requirements;  Specific evaluation activities required to support the department’s responsibilities in moving identified horizontal initiatives forward;  Programs for which program funding is set to terminate unless it has already been determined that an evaluation is not necessary; and  Other direct spending of the department as appropriate to the information needs of the Minister, Deputy Minister and Senior Management Team. 4 The Clerk will submit to the Premier by March 31, 2012, the initial government-wide evaluation plan . Cabinet Secretariat │ December 2011 9

Guide to Developing a Risk-Based Departmental Evaluation Plan
Five Step Process
This Guide describes a five s tep process for developing a Department’s Evaluation Plan. 1. Compile List of Departmental Initiatives 2. Pre-screening for Exclusion / Inclusion 3. Assess Risk 4. Assess Readiness 5 Submit Plan – Evaluation Product Profiles Steps 2-4 have associated templates (Appendix A, Risk Assessment Report and Appendix B) on which assessment decisions are recorded. Each of these completed templates should be attached to the final Departmental Evaluation Plan.
1. Compile List of Departmental Initiatives
Developing a plan begins with identifying all departmental initiatives. The PGP Office requests that departments review with their executive, at the beginning of a department ’s evaluation planning cycle (either annual or multiyear), their inventory of programs and services to ensure initiatives are accurately reflected, that is organizational initiatives are listed not organizational structures or functions. Inventories requiring updating should be submitted to the PGP office using the template in Appendix E 5 . Each initiative listed in the inventory should include the associated budget/cost. Some initiatives do not have a specified budget; in such cases, the cost of resources to deliver that initiative should be estimated. Initiatives are defined as a group of departmental activities designed to achieve specific outcomes. They may be programs or services designed to benefit or influence external clients, services to internal clients or activities required to monitor policy or program delivery by third parties (e.g. HCS is responsible for provincial policy with RHAs delivering the programs) . Where a department’s initiative is associated with a 5 Initiatives identified by departments in 2011 have been pre-populated into the risk assessment database by the Economic and Statistics Branch of the Department of Finance. Please contact your assigned Program Officer, Provincial Government Programs Office for information on how to access the web-based tool. Cabinet Secretariat │ December 2011 10

Guide to Developing a Risk-Based Departmental Evaluation Plan Horizontal Initiative, the initiative should be itemized separately as per Template 6 of Appendix E. Corporate Shared Services such as printing services, payroll, purchasing, pensions and group insurance administration, corporate financial services, various OCIO support services, etc. should be included in the inventory of the department providing the service. Departmental support functions/divisions such as Executive Support, Finance Units, Strategic Human Resource Management Units, Policy and Planning Divisions, and Communications Divisions should not be included.
2. Pre-Screen for Exclusion / Inclusion
From your inventory there may be initiatives that can be immediately excluded or immediately prioritized for evaluation (included). The rationale for including or excluding an initiative at the prescreening stage should be recorded in the standard template found in Appendix A which will be attached to your Departmental Evaluation Plan submission. Exclusion Evaluations can be time and resource intensive. While departments should have a performance measurement system in place to regularly monitor and report on program performance, evaluations should only be undertaken on a periodic basis and only if the benefits of the evaluation are expected to exceed the costs of completing an evaluation. Accordingly, initiatives that have been evaluated in the last two years can be excluded from the next year’s departmental evaluation planning. Other initiatives may become excluded after completing the Evaluablility Assessment in Section 5.2.4
of this document. Cabinet Secretariat │ December 2011 11

Guide to Developing a Risk-Based Departmental Evaluation Plan
Inclusion
For some initiatives, there may already be a mandatory requirement for an evaluation. The source of the requirement may have been through Cabinet or Treasury Board direction, initiatives identified under Accountability Frameworks (for new initiatives or as part of an Horizontal Initiative), initiatives requiring funding renewal (as specified in budget decisions or MCs), legislative requirements 6 , requests by Ministers or Deputy Ministers, or where other funders are involved (e.g. the federal government) as a requirement of the Contribution Agreement. Those initiatives with mandatory evaluation requirements do not require a Risk-Based Assessment but do require an Evaluability Assessment under Section 5.2.4
of this document. Reviews or evaluations identified in a department’s Regulatory Improvement Plan should be documented in the Departmental Evaluation Plan but do not require assessment for risk or evaluability.
3. Assess Risk
All department initiatives remaining after the pre-screening process must be assessed for risk. The probability that a program is no longer relevant, effective or efficient is the overall risk that is being measured. The Economic and Statistics Branch (ESB) of the Department of Finance has developed a web Risk-Based Assessment tool to assist departments in ranking their initiatives from highest to lowest risk.
7 The tool’s ranking is based on the following seven risk variables with predefined weighted sub criteria. The attached Risk Assessment Tool Guide will provide step by step instructions for completing the on line tool. Budget: This risk factor considers materiality of the base budget. Initiatives utilizing a significant percentage of a department’s resources may be evaluated periodically to 6 to reviewing the legislation itself. 7 Legislative requirements refer to any legislation requiring periodic reviews of initiatives. It does not refer Please contact your Program Officer to determine how to access this web-based resource. Cabinet Secretariat │ December 2011 12

Guide to Developing a Risk-Based Departmental Evaluation Plan confirm value for money (effectiveness, efficiency and economy).
8 Helpful documents may include: Budget Decisions, Estimates. Gross Expenditure Variance: This risk factor considers the significance of over or under spending trends. Initiatives experiencing significant over or under spending should be evaluated periodically to confirm adequacy and relevance (i.e. Are existing resources adequate to address the identified population ’s need; is the initiative over resourced based on existing needs). Helpful documents may include: Budget Monitoring Reports, Report on the Program Expenditures and Revenues of the Consolidated Revenue Fund. Priorities: This risk factor considers how important the initiative is to achieving a government-wide priority. Helpful documents may include: Prior year ’s Departmental Evaluation Plan, Ministerial Mandate Letters, Speech from the Throne, Public Strategy Documents, Deputy Minister Performance Contract, Departmental Plan (Strategic, Business, or Activity), Annual Report, Operational Plan, and Work Plan. Funding Renewal Commitments: This risk factor applies where evaluation may support a request to continue funding that was approved for a fixed period or for which there is no continuing provision in the fiscal framework. Helpful documents may include: Fiscal Framework, Budget Decisions. Public Opinion: This risk factor considers client and other stakeholder feedback about an initiative. There may be current sensitivities, interests or pressures from clients and other stakeholders that would warrant an evaluation (e.g. feedback from the Auditor General, individuals, groups or MHAs). Helpful documents may include client surveys, public consultations, Auditor General Reports, public strategy documents, departmental correspondence such as letters to Ministers, etc. 8 ESB led a group of Policy Directors and PGP staff through a multiple criterion evaluation process to assign weights to the 7 factors. As a result of this process, the Budget factor is not statistically significant. Departments may choose to include initiatives which use significant resources in their Plan even if assigned a low risk assessment using this tool. Cabinet Secretariat │ December 2011 13

Guide to Developing a Risk-Based Departmental Evaluation Plan Relevance: This risk factor consider s the results of a department’s environmental scan and SWOT (strengths, weaknesses, opportunities, threats) analysis. Departmental strategic planning and/or horizontal strategy development may have identified changes in the demographics and/or needs of the target population that requires a revised needs assessment, process review and/or effectiveness evaluation (e.g. growing wait lists for services, lower than expected take-up). Helpful documents may include: Auditor General Reports, Performance Monitoring and Evaluation Reports, Departmental Plan (Strategic, Business, or Activity) and Annual Report. Performance Findings: This risk factor considers the findings from the performance monitoring and or past evaluation reports. Helpful documents may include: Performance Monitoring and Evaluation Reports, Auditor General Reports. Upon completing the web Risk-Based Assessment Tool for all initiatives, a final report can be automatically generated ranking initiatives from 1 (highest risk) to 0 (lowest risk). The final Risk Assessment Report is to be reviewed and approved by executive and attached to the Departmental Evaluation Plan submission. There is no identified “high risk” cut-off point. Initiatives will cluster differently along the continuum across departments. For example some departments ’ initiates may cluster primarily in the orange to red zone while others may cluster closer towards the blue spectrum. In either case departmental executive must choose a Risk Level Cut-Off point (between 1-0). All departments will be expected to identify Evaluation Products within the planning cycle. Those initiatives close to or above the cut off point must be assessed for evaluability (i.e. which priority initiatives are able to have an evaluation product addressed in this planning cycle). Cabinet Secretariat │ December 2011 14

Guide to Developing a Risk-Based Departmental Evaluation Plan
4. Assess Readiness
The risk assessment process identifies which initiatives are potential priorities for evaluation: the evaluability process identifies which priorities are feasible to evaluate in the current planning cycle. Initiatives to be assessed for evaluability include those under Section 5.2.2 assessed as mandatory and those initiatives that have been identified as priority due to their proximity to the Department’s Risk Cut-Off point (Section 5.2.3).
All Evaluability Assessment outcomes should be recorded in the standard template found in Appendix B. Each Evaluation or Evaluation Activity to be completed in this planning cycle must have a corresponding Evaluation Product Profile (Appendix C) completed to describe what is expected to be completed by when and at what cost. The evaluability process is defined by three key questions as illustrated in Chart 1.
Chart 1 Process Flow Chart: Priority Evaluation Initiatives
Cabinet Secretariat │ December 2011 15

Guide to Developing a Risk-Based Departmental Evaluation Plan
Question 1: Will the initiative likely be modified based on evaluation results?
Department leads should consult with their Minister and Deputy Minister to determine if there is an ability to modify the initiative based on findings. If there are timing issues which make evaluation in the current planning cycle not feasible, the initiative should be kept in view and reassessed in future planning cycles.
Questions 2: Is the initiative ready to be evaluated?
Preconditions for evaluating the effectiveness and efficiency of initiatives include:  a clearly articulated initiative  clearly specified goals and outcomes  a rationale linking the initiative’s activities to the desired outcomes and  appropriate performance measurement data If an initiative cannot be evaluated, the departmental evaluation plan should identify what Evaluation Activity is required to position the initiative for evaluation at a future date (e.g. developing a logic model, implementing a performance measurement plan, gathering baseline data, and so forth). These Evaluation Activities must be detailed in an Evaluation Product Profile, Appendix C.
Question 3: Are resources available to complete the evaluation?
Evaluation activity planning will require assessing internal/external resource requirements. Where existing departmental resources are inadequate to cover all priority evaluations, those that can be deferred should be kept in view for consideration in subsequent evaluation cycles. Those that cannot be deferred and would require incremental funding should be brought to the Deputy Minister’s attention. The Minister and Deputy Minister must determine if existing resources can be reallocated or consider the budget guidelines to determine if they will seek additional resources through the budget process 9 . If incremental resources are being requested, a draft of the Departmental Evaluation Plan should be attached to the budget submission. 9 Departments need to consider the Department of Finance budget guidelines when determining if additional resources should be sought through the budget process. Cabinet Secretariat │ December 2011 16

Guide to Developing a Risk-Based Departmental Evaluation Plan The Economic and Statistics Branch (ESB) is a resource available to all departments. The ESB may be able to assist in gathering baseline data, developing data from administrative or other sources, ensuring datasets are suitable for an evaluation and/or developing or assessing potential measurement tools for an evaluation. They can assist in making sure the data and tools needed for an evaluation are available when needed by considering requirements early in the process. If there will be incremental cost to the ESB to provide the support, there may be a charge to the department which should be identified in the budget submission along with any other incremental costs associated with the evaluation.
5. Submit Plan – Evaluation Product Profiles
Once the Minister and Deputy Minister have been briefed and their input obtained, a final Departmental Evaluation Plan should be prepared. A standard template is presented in Appendix D. For each proposed evaluation product, an overview should be included reflecting: critical timelines (at a minimum expected start and end dates); resource requirements including whether these would be internal or external; and `the department lead overseeing the activity. A standard template to be completed for each evaluation product profile is presented in Appendix C. The final plan should be forwarded to the Minister for approval. As per the Policy on Evaluation, the approved plans must be forwarded to the Clerk of Executive Council by April 30, 2012 for the initial plan, and in subsequent years as plans are scheduled for renewal. The Clerk may submit the Departmental Evaluation Plan to Treasury Board or Cabinet for further consideration. The Clerk will be advised of the department’s progress towards implementation of the plan as part of the Deputy Minister s’ performance reporting process. Cabinet Secretariat │ December 2011 17

Guide to Developing a Risk-Based Departmental Evaluation Plan
Accountability Framework:
The Policy on Evaluation requires Deputy Ministers/equivalents to ensure all Cabinet submissions for new policies and programs identify the client, other benefits, and where feasible, establish specific, measurable, achievable, realistic, and time bound (SMART) targets before decisions on introducing/funding are made by Cabinet. It also requires them to “ensure Cabinet submissions include an accountability framework documenting how performance will be monitored and determining if and when a more comprehensive evaluation of the program/p olicy will occur”. An Accountability Framework makes roles, responsibilities, and expectations clear, supporting the availability of reliable and timely reports about intended and actual results. The Accountability Framework structure includes performance-based planning, monitoring, performance-based reporting and feedback. Accountability Frameworks are attached to applicable Cabinet papers submitted since July 1, 2011. Consolidated Revenue Fund: is the account into which public money (taxes and revenues) is deposited, and from which funds are withdrawn in order to pay the costs of public services. The Legislature authorizes the use of the money in the Consolidated Revenue Fund. Departmental Evaluation Plan: is a clear and concise summary that establishes the department's evaluation activities / products to be undertaken over a specified period, in accordance with the Policy on Evaluation. Economy: minimizing the use of resources. Economy is achieved when the cost of resources used approximates the minimum amount of resources needed to achieve expected outcomes (i.e. attaining the right program inputs at the lowest possible costs). Effective: the extent to which the observed outcomes are consistent with the intended outcomes. Cabinet Secretariat │ December 2011 18

Guide to Developing a Risk-Based Departmental Evaluation Plan Efficient: the extent to which resources are used such that a greater level of output is produced with the same level of input or a lower level of input is used to produce the same level of output (i.e. attaining the most program outputs possible for each program input). The level of input and output could be increases or decreases in quantity, quality or both. Evaluability Assessment: is used to determine if an initiative is able to be evaluated in the current planning cycle. An initiative is evaluable if: it is likely be modified based on evaluation results; the initiative can be clearly and logically articulated, it has performance data or can be developed using evaluation methods and there are resources available to complete the evaluation. Evaluation: the systematic collection and analysis of information on the performance of a policy, program or initiative to make judgments about relevance, progress or success and cost-effectiveness and / or to inform future programming decisions about design and implementation. Types of program evaluations include: Evaluability Assessment, Needs Assessment, Monitoring / Review compliance with performance standards, Implementation / Process Evaluation (Formative Evaluation), Impact/Outcome Evaluation (Summative Evaluation), Program Review and Efficiency Assessment (Cost/Benefit Analysis or Cost Effectiveness Analysis). Evaluation Related Activity: any output of the departmental evaluation function which may include, but is not limited to, the following: training, terms of reference (TOR) or Requests for Proposals (RFP) for a future program evaluation, accountability frameworks, research and data analysis, and reviews monitored under a department’s Regulatory Improvement Plan. Evaluation Product: Evaluations and evaluation-related activities. Horizontal Initiative: An initiative is considered horizontal if multiple provincial ministers are accountable for the results of the initiative at this point in time. An initiative’s accountability structure may change over the course of its life cycle. Cabinet Secretariat │ December 2011 19

Guide to Developing a Risk-Based Departmental Evaluation Plan Committees of multi-departmental initiatives should conduct periodic reviews over the initia tive’s life cycle to determine the appropriate level of accountability. Performance Measure: a quantitative or qualitative standard that characterizes and defines initiative’s results including outputs and outcomes (e.g. the “success standard” by which you know there has been a change in knowledge, skill, behavior, or attitude). Performance Measurement: the process of selection, development and ongoing use of performance measures for program management or decision-making. Generally performance measurement occurs in the form of quarterly or annual monitoring of performance measures to determine if the desired change is occurring. Performance measurement can identify the direction of change but does not attribute the cause of the change to the initiative. Performance: the actual results measured against defined standards (i.e. the extent to which effectiveness, efficiency and economy are achieved by a program). Policy: A policy enables or restricts actions as a means of specifying outputs, outcomes, or parameters. Policy consists of official guidelines or operating principles that influence behaviour towards a stated outcome. Program: a group of related purposeful activities that is intended to achieve one or several related objectives. A program is often treated as a budget unit, that is designed and managed to meet a specific public need either directly or indirectly (that is, supports the operations of government). Relevant: the extent to which a program responds to a demonstrable need of Newfoundlanders and Labradorians and is appropriate for delivery by the provincial government. Risk-Based Approach: is a method for considering risk when planning the extent of evaluation coverage of direct program spending / policy initiatives. The risk being measured is that an initiative may not be, or is no longer relevant, effective or efficient, Cabinet Secretariat │ December 2011 20

Guide to Developing a Risk-Based Departmental Evaluation Plan Value for Money: the extent to which a program demonstrates relevance and performance (effectiveness, efficiency and economy). It is an important goal for public officials and other stakeholders who are concerned with whether taxpayers and citizens are receiving effective and efficient programs and services for their tax dollars.
Auditor General
’s Reports: As a legislative auditor, the Auditor General audits financial statements and other accountability documents, evaluates management practices and control systems, and determines compliance with legislative and other authorities. The Auditor General reports at least annually to the House of Assembly on significant matters which result from his/her examinations. Budget Speeches: An itemized summary of estimated or intended expenditures for a given period along with proposals for financing them. Annual budget speeches are accessible on the House of the Assembly publications website. The Budget combines the projected accrual revenues and expenses of the Consolidated Revenue Fund with the budge of various Crown Corporations, Boards and Authorities which are controlled by the Government of NL. Budget Decisions: Subsequent to Cabinet decisions on department’s budget requests, the Budget Division of the Department of Finance will issue documents recording the decisions. From time to time, there may be a requirement for a department to submit additional information or report back to Treasury Board or Cabinet on specific issues. These documents are filed with Deputy Ministers and with Department Comptrollers. Budget Monitoring Reports: On a monthly basis, Departmental Controllers submit to the Budget Division of Finance, a report highlighting any variances between projected expenditures to March 31st of each year as compared with the original budget. Departmental Controllers will have copies on file. Cabinet or Treasury Board Directions: This category includes minutes and other correspondence issued by Cabinet or Treasury Board to a department directing a Cabinet Secretariat │ December 2011 21

Guide to Developing a Risk-Based Departmental Evaluation Plan specific action. These documents are generally in the form of a Minute of Council (MC); Order in Council (OC) or a Treasury Board Minute (TBM). These documents are filed in the minister ’s office. Client Surveys or Public Consultations: Many initiatives consult with citizens in the development stage or report progress at incremental stages. Feedback from consultations is generally summarized and used to improve on program performance. This feedback will be valuable in answering the risk-based assessment public opinion question and may be relevant in answering question 1 of the evaluability assessment. Oftentimes public feedback is included in public reports ( e.g. “What We Heard”. Program Managers will be able to identify where this information can be found). Departmental Plan (Strategic, Business, Activity) and Annual Reports: The Transparency and Accountability Act requires all government entities to submit to Cabinet a planning document. Depending on the entity ’s categorization (category 1, 2, or 3) these planning documents are referred to as Strategic, Business or Activity Plans respectively. Departments are required to report annually to Cabinet on their plan’s progress (i.e. Annual Reports. All plans and annual reports are public documents and are posted on the department’s website). Department Cabinet Submission Plan: To facilitate Cabinet scheduling, departments are required to identify what issues they intend to bring before Cabinet, with a brief description and an anticipated submission date. As the Policy on Evaluation requires all Cabinet submissions requesting major new policies or new initiatives to have an accountability framework, the Cabinet submission is a priority document when developing an evaluation plan. The plans are highly confidential and are filed in the departm ent’s ministerial office. Cabinet Submission Plan information may be accessed through department Policy Directors.
D
eputy Minister’s Performance Contract: The Deputy Minister reports to the Minister and the Clerk of Executive Council in ensuring that all over-arching policies of government are communicated and coordinated, and that matters and sensitive issues Cabinet Secretariat │ December 2011 22

Guide to Developing a Risk-Based Departmental Evaluation Plan requiring strategic direction on the part of government are monitored to ensure their expeditious resolution. Each Deputy Minister has a performance contract which is reviewed periodically with the Clerk in consultation with the Minister. These Contracts are highly confidential and are filed in the Deputy Minister’s Office and with the Clerk of Executive Council. Departmental policy divisions draft these contracts and can ensure all pertinent information needs are met by the proposed plan. Estimates of the Program Expenditure and Revenue Fund: is a supplementary document to the Budget. It outlines the estimated expenditures and revenues of the Consolidated Revenue Fund for the year and reflects the policies, programs and priorities of government. This document is available on the Internet: www.fin.gov.nl.ca/fin/public_accounts/index.html
. Fiscal Framework: While Estimates are prepared for a single fiscal year, the Budget Division of the Department of Finance maintains a multi-year forecast of departmental expenditures and revenues by budget categories (e.g. salary and operating envelopes, major programs). A quick review of this Fiscal Framework can identify budget adjustments such as the removal of funds for one-time initiatives or those being piloted. Ministerial Mandate Letters: A letter oftentimes issued by the Premier to his/her Ministers outlining the government priorities that fall within a ministry’s mandate. Such letters would be filed in the Minister ’s Office. Operational Plan: The operational plan focuses on the key priorities of the Deputy Minister which relate to the internal administration of the organization (i.e. monitoring and evaluating organizational performance, organizational structure, development and implementation of organizational policies and procedures). Operational plans are encouraged by the Transparency and Accountability Office and, as such, department Accountability Coordinators will be of assistance in accessing operational plans. Performance Monitoring and Evaluation Reports: Many programs have undergone monitoring and/or evaluation. Understanding the monitoring and evaluation history, Cabinet Secretariat │ December 2011 23

Guide to Developing a Risk-Based Departmental Evaluation Plan including recommendations for program changes, will facilitate risk-based assessments. For example, programs that have been evaluated in the last two years are excluded for consideration under the current year’s risk-based assessment. However, these monitoring or evaluation reports may make recommendations regarding future monitoring and evaluation needs. These reports come in many forms and may be generated internally or externally. Program Managers and Directors may be of assistance in tracking such reports. Public Strategy Documents: Strategic documents outline a plan of action designed to achieve a particular goal. Such documents are used by departments to clarify an approach to an important broad-based issue (e.g. wellness, climate change, innovation, and so forth). These reports are helpful in identifying the various initiatives that are covered under the strategy. Most departments post their strategy documents on their website. Policy Directors can facilitate access to such documents if they are not readily available. Prior Year Departmental Evaluation Plans: Evaluation plans illustrate the decision making process used to define priority initiatives for evaluation. An understanding of these processes will facilitate identifying current priority initiatives. Evaluation Plans are not public documents. They can be accessed through Policy Directors. Regulatory Improvement Plan: All Departments are required under the Regulatory Reform Initiative to submit an annual or multi-year plan to Service Newfoundland and Labrador. Quarterly progress reports by departments demonstrate qualitative and quantitative improvements in regard to developing and implementing regulation. Review-related activities monitored under this plan are considered to be an “evaluation product” and should be credited as such in the Departmental Evaluation Plan. Regulatory Improvement Plan initiatives do not require Risk or Evaluability Assessments under the Policy on Evaluation. All departments have a Regulatory Reform Representative who can advise on, as applicable, the initiatives included within the department ’s Regulatory Improvement Plan.
Cabinet Secretariat │ December 2011 24

Guide to Developing a Risk-Based Departmental Evaluation Plan
Speech from the Throne:
The Speech from the Throne occurs at the opening of a session of the House of Assembly. The Speech is read by the Lieutenant Governor-in Council and prepared by the Premier's Office. Speeches are publicly accessible on the House of the Assembly publications website. Strategic Directions: Strategic direction means the articulation of a desired physical, social or economic outcome that would normally require action or involvement of more than one government entity. They are a summary of government’s commitments to the people of the province. They are communicated by Government through Platform Documents, the Speech from the Throne and Budget Speeches, press releases, and policy documents. Each minister has his/her own strategic directions. Accountability Coordinators play an essential role in the development of the Strategic Directions for their respective ministers. The strategic directions of a department are contained in the Department’s Plan (strategic, business, activity). Work Plan: Work plan issues may emanate from the department plan, the operational plan, or reflect priorities at a branch, division or position level. The work plan contains the work direction of a specific branch, division or position. Divisional Directors will be of assistance in accessing work plans. Cabinet Secretariat │ December 2011 25

Guide to Developing a Risk-Based Departmental Evaluation Plan
From your inventory there may be initiatives that can be immediately excluded or immediately prioritized for evaluation (included). The rationale for including or excluding an initiative at the prescreening stage should be recorded in template 1. This template is to be attached to your Departmental Evaluation Plan submission.
Template 1 Pre-Screen Inventory PRE-SCREENING OUTCOME Initiative
Initiative W Initiative X Initiative Y Initiative Z
Mandatory Inclusion
Commitment MC2009-xxxx Ministerial Priority Poverty Reduction Strategy Evaluation
Exclusion
Evaluated in the last two years  2010 Cabinet Secretariat │ December 2011 26

Guide to Developing a Risk-Based Departmental Evaluation Plan
The Template 2 below records evaluability decisions for all priority initiatives. Priority initiatives include those identified through the Pre-Screening: Mandatory Inclusion (Section 5.2.2) and those initiatives that have been identified as priority relative to the Department’s Risk Cut-Off point. Using Template 2, list the initiatives that have been assessed for “Evaluability” and check the box identifying the assessment outcome. Where “Other” has been checked, please add a footnote to provide an explanation for the choice. Template 2 Evaluability Assessment Template
Initiative
Evaluation To Be Completed In This Planning Cycle W X Y Z

EVALUABILITY ASSESSMENT OUTCOMES
Evaluation Activity To Be Completed In This Planning Cycle: Not Ready For Evaluation Insufficient Capacity (e.g. need staff training, data bases developed etc) No Evaluation Or Evaluation Activity To Be Completed In This Planning Cycle: Other Initiative Unlikely To Be Modified Based On Evaluation Results Insufficient Resources Other
   * An Evaluation Product Profile (Appendix C) is to be completed for each initiative that indicates a check under the categories of Evaluation or Evaluation Activity. Cabinet Secretariat │ December 2011 27

Guide to Developing a Risk-Based Departmental Evaluation Plan
The Evaluablity Assessment Outcomes (template 2) identify all Evaluations and Evaluation Activities that are to be completed within this evaluation planning cycle. For each planned Evaluation and Evaluation Activity, an Evaluation Product Profile (template 3) is to be completed to describe what is to be completed, by when and at what cost.
Template 3 Evaluation Product Profile INITIATIVE TITLE: Brief description of the initiative and the scope of the evaluation product to be
undertaken. begin date: end date:
Proposed Timeframe Who is responsible to oversee this work? Resource Requirements Internal External Economic Statistics Branch Total
$ Within Base
*Budget Request $ Reallocated $ New
* Please refer to the Department of Finance’s annual Budget Guidelines to determine the parameters under which new funding can be requested. Cabinet Secretariat │ December 2011 28

Guide to Developing a Risk-Based Departmental Evaluation Plan
Template 4 Departmental Evaluation Plan
1. INTRODUCTION 1.1 Department’s Strategic Directions and Goals: Provide a brief discussion of how evaluation is being used to support the department’s mandate. 2. ASSESSMENT DOCUMENTS 2.1 Pre-Screening Outcomes Attach the completed Appendix A Template 1 2.2 Risk Matrix Outcomes Attach the web-based Risk Assessment Report 2.3 Evaluability Assessment Outcomes Attach the completed Appendix B Template 2 3. YEAR IN REVIEW 3.1 Evaluation Products Conducted in Year XXXX-XXXX : This section should identify evaluations and evaluation products that your department has completed in the previous evaluation planning cycle. The Provincial Government Programs Office has developed an evaluation inventory for each department for the period 2009-2011. Please contact your PGP Officer for a copy. 3.2 Evaluation Activities to Be Carried Forward: This section should identify evaluations and evaluation products that your department had anticipated completing in the previous planning cycle but are required to carry over into the next planning cycle. 4. CURRENT YEAR’S WORK PLAN 4.1 Proposed Evaluation Product Profiles: Complete Template 3 in Appendix C for each initiative for which there is an evaluation product to be addressed in the current year’s evaluation cycle. 4.2 Proposed Timeline for Evaluation Activities for Current Year: Summarize the evaluation products to be delivered in this evaluation cycle through a time line. 4.3 Resource Requirements: Summarize the resource requirements identified in the individual product profiles 4.4 Regulatory Improvement Plans (RIP): Evaluations/Product commitments associated with RIP should be noted in this section and cross referenced to the RIP. Cabinet Secretariat │ December 2011 29

Guide to Developing a Risk-Based Departmental Evaluation Plan
Departmental initiative inventories and their budgets should be reviewed, with executive, at the beginning of each evaluation planning cycle to ensure initiatives are updated and accurately reflected (i.e. new initiatives, initiatives removed, and ensure organizational initiatives are listed not organizational structures or functions). Templates 5 and 6 should be completed and submitted to the PGP Officer for population into the web-database. The templates below provide examples.
Template 5 Departmental Initiative Inventory Update Department Initiative: Component * Estimated 2011 Budget Health and Community Services Finance
NL Provincial Drug Program: Access Program Tax Expenditures: PIT - Child Benefit
% of Gross Dept Budget
Where a department’s initiative is associated with a Horizontal Initiative, this association should be identified in the inventory as illustrated in template 6.
Template 6 Horizontal Initiative Inventory Update Horizontal Initiative (HI) Provincial Waste Management HI Lead Department
Environment and Conservation
Sub-Initiative (SI)
Policy Implementation Enforcement
SI Department
EC MA SNL
* Estimated Budget for Year 20XX
$ 40,000 $50M $100,000 * If the initiative does not have its own Cost Centre in ORACLE GL, please provide an order of magnitude estimate of the budget resources it consumes (can be rounded to nearest $100,000 or 1/2 million) Estimated 20xx Budget / Total Gross Current Account Expenditures per Estimates 20xx Cabinet Secretariat │ December 2011 30

Guide to Developing a Risk-Based Departmental Evaluation Plan
USING THE RISK-BASED ASSESSMENT TOOL
The Guide Developing a Risk-Based Departmental Evaluation Plan, Step 3, discusses the process for completing a risk-based assessment. All department initiatives remaining after the pre-screening process must be assessed for risk. The probability that a program is no longer relevant, effective or efficient is the overall risk that is being measured. Step 3 of the Guide references an online web Risk-Based Assessment tool developed by the Economic and Statistics Branch (ESB) of the Department of Finance to assist departments in ranking their initiatives from highest to lowest risk. The tool’s ranking is based on seven risk variables with predefined weighted sub criteria. The information contained in this appendix will aid employees in navigating through the web Tool. The tool can be accessed using the following address. Please open your internet browser and copy and paste the following - http://inform.stats.gov.nl.ca/riskassessment . A single user ID and password has been assigned to each Department but more than one person can access the database at one time. Once logged in you will see the following screen. Cabinet Secretariat │ December 2011 31

Screen 1
Guide to Developing a Risk-Based Departmental Evaluation Plan Screen 1 provides three tab options. The “Add Risk Assessment” button is used each time you go to complete the initial risk assessment. The “Revised Risk Assessment” button is used to ma ke changes to an already completed risk assessment. The “View My Projects” button allows you to view and export to Excel or Word all risk assessments completed by Project. To begin a new risk assessment, click on Tab “Add Risk Assessment” and you will get the following screen: Cabinet Secretariat │ December 2011 32

Screen 2
Guide to Developing a Risk-Based Departmental Evaluation Plan A “Select Project” and “Program Names” appears on Screen 2. The “Select Project” tab has been pre-populated using the Departmental abbreviation and 2012 and the associated program names have been similarly pre-populated based on information provided to the Provincial Government Programs Office (PGP). The PGP can add new Projects (for example if a lead for a Horizontal Initiative decides to use this tool to assist in developing an evaluation framework for their strategy) and or new program names. In this illustration it presents the list of programs for CS2012 (Cabinet Secretariat) Program Description as well as a tab for each of the seven risk factors. When printing out reports it is the “Select Project” identifier that will be used. All risk assessments completed under your department’s Project name will be available for exporting and printing. Cabinet Secretariat │ December 2011 33

Guide to Developing a Risk-Based Departmental Evaluation Plan To complete a risk assessment, scroll through the list of program names and select the one you want to assess. If it does not exist, please contact the PGP. Once you select a program and click NEXT, you will advance to the following screen.
Screen 3
Budget: This risk factor considers materiality of the base budget. Initiatives utilizing a significant percentage of a department’s resources may be evaluated periodically to confirm value for money (effectiveness, efficiency and economy).
10 Helpful documents may include: Budget Decisions, Estimates. The budget criterion is unique as there are two screens to be completed: Budget Value and Gross Expenditure Budget. The screen with the highest risk (1-5) must be the only screen with the “top box” checked. 10 ESB led a group of Policy Directors and PGP staff through a multiple criterion evaluation process to assign weights to the 7 factors. As a result of this process, the Budget factor is not statistically significant. Departments may choose to include initiatives which use significant resources in their Plan even if assigned a low risk assessment using this tool. Cabinet Secretariat │ December 2011 34

Guide to Developing a Risk-Based Departmental Evaluation Plan In this example, the budget value was 1. Before hitting Next, click on gross
expenditure budget and complete the following screen. Screen 4
In this example the Gross Expenditure Budget is a 3; while the Budget Value was a 1. As the Gross Expenditure response is higher, we leave the check mark for Gross Expenditure Budget and now hit Next (Had the Budget Value been 4 or 5, then we would have checked the Budget Value box and hit Next). If this is lower and you have to go back to the Budget Value screen, the previous check is removed and you will need to reenter it. Cabinet Secretariat │ December 2011 35

Guide to Developing a Risk-Based Departmental Evaluation Plan Once you have complete the applicable Budget screen, hit Next and this will bring you to the following screen:
Screen 5
Gross Expenditure Variance: This risk factor considers the significance of over or under spending trends. Initiatives experiencing significant over or under spending should be evaluated periodically to confirm adequacy and relevance (i.e. Are existing resources adequate to address the identified population’s need). Is the initiative over resourced based on existing needs? Helpful documents may include: Budget Monitoring Reports, Report on the Program Expenditures and Revenues of the Consolidated Revenue Fund. When your selection has been entered, click Next to bring you to the next screen. Cabinet Secretariat │ December 2011 36

Screen 6
Guide to Developing a Risk-Based Departmental Evaluation Plan Priorities: This risk factor considers how important the initiative is to achieving a government-wide priority. Helpful documents may include: Prior year ’s Departmental Evaluation Plan, Ministerial Mandate Letters, Speech from the Throne, Public Strategy Documents, Deputy Minister Performance Contract, Departmental Plan (Strategic, Business, Activity), Annual Report, Operational Plan, and Work Plan. When your selection has been entered, click Next to bring you to the next screen. Cabinet Secretariat │ December 2011 37

Screen 7
Guide to Developing a Risk-Based Departmental Evaluation Plan Funding Renewal Commitments: This risk factor applies where evaluation may support a request to continue funding that was approved for a fixed period or for which there is no continuing provision in the fiscal framework. Helpful documents may include: Fiscal Framework, Budget Decisions. When your selection has been entered, click Next to bring you to the next screen. Cabinet Secretariat │ December 2011 38

Screen 8
Guide to Developing a Risk-Based Departmental Evaluation Plan Public Opinion: This risk factor considers client and other stakeholder feedback about an initiative. There may be current sensitivities, interests or pressures from clients and other stakeholders that would warrant an evaluation (e.g. feedback from the Auditor General, individuals, groups or MHAs). Helpful documents may include client surveys, public consultations, Auditor General Reports, public strategy documents, departmental correspondence such as letters to Ministers etc. If after such a review you remain
unaware of client or stakeholder feedback then click on risk level 5 as there is an increased risk that there may be issues/concerns of which you are not aware.
When your selection has been entered, click Next to bring you to the next screen. Cabinet Secretariat │ December 2011 39

Screen 9
Guide to Developing a Risk-Based Departmental Evaluation Plan
Relevance:
This risk factor considers the results of a department’s environmental scan and SWOT (strengths, weaknesses, opportunities, threats) analysis. Departmental strategic planning and or horizontal strategy development may have identified changes in the demographics and or needs of the target population that requires a revised needs assessment, process review and/or an effectiveness evaluation (e.g. growing wait lists for services, lower than expected take-up). Helpful documents may include: Auditor General Reports, Performance Monitoring and Evaluation Reports, Departmental Plan (Strategic, Business, Activity) and Annual Report. When your selection has been entered, click Next to bring you to the next screen. Cabinet Secretariat │ December 2011 40

Screen 10
Guide to Developing a Risk-Based Departmental Evaluation Plan Performance Findings: This risk factor considers the findings from the performance monitoring and or past evaluation reports. Helpful documents may include: Performance Monitoring and Evaluation Reports, Auditor General Reports. Please
note that level 5 should be selected for initiatives where there are no performance
findings available. If there are no performance findings, there is increased risk that an initiative is not performing as well as one may think. Screen 10, performance findings, is the final of the seven risk criteria. Upon completing Screen 10, click on the Review Selections tab. Note: On this or any other criteria screen, you can click Back to go to an earlier screen or if you want to go a specific criterion to review or change you can click on the applicable top tab. Cabinet Secretariat │ December 2011 41

Guide to Developing a Risk-Based Departmental Evaluation Plan Risk Overview: Screen 11 provides an overview of the selections you have chosen for all criteria for a particular initiative. This screen provides the options of editing your selections or calculating your risk assessment level.
Screen 11
If you are satisfied with your risk choices, click Calculate Risk Assessment Level. Please note you can change your calculations at a later date, by using the Revise Risk Assessment on Screen1. Once you select the Calculate Risk Assessment Level you will see the following screen. Cabinet Secretariat │ December 2011 42

Screen 12
Guide to Developing a Risk-Based Departmental Evaluation Plan As each additional initiative is assessed, its summary is added to your Risk Assessment Report in accordance with its risk level, thus providing a final report of all initiatives ranked by risk. A Risk Assessment Report, illustrated below, can be generated anytime by clicking the View My Risk Assessment Projects tab on Screen 1 (Home Page) and converting to Excel or Word. Cabinet Secretariat │ December 2011 43

Guide to Developing a Risk-Based Departmental Evaluation Plan Risk levels are colour coded, as illustrated below, with blue representing lowest risk and red representing highest risk. The black markers represent the calculated risk level of each initiative listed in your Risk Assessment Report. Screen 12 will require you to click the EXIT button. Doing so will take you back to Screen 1 (Home Page) where you can begin a new risk assessment on another initiative, revise any of your previous initiatives’ risk assessments, or view all your risk assessments. Cabinet Secretariat │ December 2011 44

Screen 1 Guide to Developing a Risk-Based Departmental Evaluation Plan To view all your completed risk assessments and produce an Excel or Word report, click the View My Projects button. The following is an example of a screen you will see. Cabinet Secretariat │ December 2011 45

Guide to Developing a Risk-Based Departmental Evaluation Plan If you do not want to print a particular program, you can mark the box to the left of the applicable program name and hit Remove Selected Program. Once you are satisfied with the report you have the option to export to Excel or Word by clicking the applicable button. Once you are finished running reports or completing risk assessments, you can exit by closing your browser. Cabinet Secretariat │ December 2011 46