Appendix E
Performance Measurement Strategy
for the
Strategy for Patient-Oriented Research
October 2014
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
2
Table of Contents
Page #
1.0
Introduction ....................................................................................................3
2.0
Program Profile ..............................................................................................4
2.1
Need for SPOR ................................................................................................................. 4
2.2
Alignment with Government Priorities ............................................................................ 5
2.3
Target Populations............................................................................................................ 6
2.4
Stakeholders ..................................................................................................................... 7
2.5
Governance....................................................................................................................... 8
2.6
Resources ......................................................................................................................... 9
3.0
Logic Model ...................................................................................................10
3.1
Logic Model Overview .................................................................................................. 10
3.2
Logic Model Narrative ................................................................................................... 11
3.2.1
Activities ................................................................................................................. 12
3.2.2
Outputs .................................................................................................................... 13
3.2.3
Immediate Outcomes .............................................................................................. 14
3.2.4
Intermediate Outcomes ........................................................................................... 15
3.2.5
Ultimate Outcomes ................................................................................................. 16
3.3
4.0
5.0
5.1
Risk Factors that may Influence SPOR’s Ability to Achieve Results ........................... 17
Performance Measurement Strategy Framework ....................................18
Evaluation Strategy ......................................................................................30
SPOR Evaluation Framework ........................................................................................ 30
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
1.0
3
Introduction
The Strategy for Patient-Oriented Research (SPOR) is a ten year strategy aiming to improve
patient health outcomes through evidence-informed care. While Canadian Institutes of Health
Research (CIHR) plays a leadership role within this national Strategy, it engages a broad
coalition of stakeholders from across Canada that collectively defines the Strategy and its
implementation.
The Strategy was announced by the Minister of Health in August 2011. The announcement noted
that by putting patients first and integrating research evidence into clinical practice, research will
have a greater impact on treatments and services provided in clinics, hospitals and doctor’s
offices across Canada.
In light of Treasury Board requirements, a Performance Measurement (PM) Strategy for SPOR
has been developed. The PM Strategy is a results-based management tool that is used to guide
the selection, development and ongoing use of performance measures. Treasury Board
guidelines require that PM Strategies include a program profile, a logic model, a performance
measurement strategy framework and an evaluation strategy.
This PM Strategy for SPOR is structured as per these requirements:

Section 2.0 provides a program profile outlining its context, alignment with federal
priorities, target groups, governance structure and resources.

Section 3.0 presents the program logic model and describes the logical sequence from
activities and outputs to outcomes. Risk factors that could affect the program’s
performance are also outlined.

Section 4.0 outlines the performance measurement requirements, including performance
indicators, data sources, and targets.

Section 5.0 describes the evaluation strategy including the proposed approach, scope,
timing, questions, indicators and data collection methods for the planned evaluation of
SPOR.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
2.0
4
Program Profile
SPOR is comprised of five core elements, each structured to address the specific challenges in
Canada that have delayed, and in some instances prevented, the translation of high quality
funded research into direct improvements in patient outcomes:
1)
2)
3)
4)
5)
Support for People and Patient-Oriented Research and Trials (SUPPORT) Units are
specialized, multidisciplinary research service centres located in jurisdictions across
Canada structured to provide the necessary expertise to those engaged in patientoriented research.
SPOR Networks represent a collaboration of patients1, health service providers,
policy/decision makers, and health researchers across Canada that produces research
and information responding to the needs of patients and health decision-makers.
Capacity Development within SPOR requires those participating in patient-oriented
research are trained, aligned, prepared, and working collaboratively towards the
common goal of increased access to high quality health care and improved patient
outcomes.
Improvements in Canada’s competitiveness in conducting clinical trials will be made
primarily through the development of the Canadian Clinical Trials Coordinating
Centre (CCTCC), which has been created in order to operationalize the nine
recommendations coming from the 2011 Clinical Trials Summit Action Plan..
Patient Engagement is an integral component in the development and implementation
of all elements of SPOR. It ensures that patients meaningfully and actively
collaborate in the governance, priority setting, and conduct of research, as well as in
summarizing, distributing, sharing, and applying its resulting knowledge.
SPOR is also unique in that cost-sharing is an integral and mandatory element. Specifically, all
CIHR investments under the SPOR umbrella must be matched at a minimum of 1:1 ratio by nonfederal government funding partners.
2.1
Need for SPOR
Patient-oriented research, the cornerstone of evidence-informed health care, refers to a
continuum of research that engages patients as partners, focusses on patient-identified priorities
and improves patient outcomes. This research, conducted by multidisciplinary teams in
partnership with relevant stakeholders, aims to apply the knowledge generated to improve
healthcare systems and practices.
The objective of SPOR is to foster evidence-informed health care by bringing innovative
diagnostic and therapeutic approaches to the point of care, so as to ensure greater quality,
accountability, and accessibility of care.. It aims to ensure that the right patient receives the right
clinical intervention at the right time, ultimately leading to better health outcomes.
1
Patients, in the context of SPOR, refer to individuals with personal experience of a health issue and informal
caregivers, including family and friends.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
5
Canada already has a strong foundation on which to build excellence in the domain of patientoriented research, including:
 a strong biomedical research community;
 a clinical workforce that is distributed regionally and across disciplines;
 a highly qualified, internationally respected clinical and health services research
workforce; and,
 world-leading expertise in systematic reviews.
In addition, large provincial and national administrative and clinical databases have enabled
researchers to assess existing care patterns, make important associations between interventions
and outcomes, and observe practice variations. Canada also has extensive experience in
multidisciplinary and cross-disciplinary clinical research networks.
Despite these undisputable strengths, Canada faces a dual challenge in the research-to-practice
continuum, often referred to as the two "Valleys" in the health care landscape. Valley 1 refers to
the decreased capacity to translate the results of discoveries generated by basic biomedical
research in the laboratory to the bedside or careside as well as to successfully commercialize
health discoveries. Valley 2 refers to the limited capacity to synthesize, disseminate and integrate
research results more broadly into clinical practice and health care decision-making. These two
valleys must be bridged if Canada is to bring evidence to bear to enhance health outcomes and
ensure a sustainable health care system.
2.2
Alignment with Government Priorities
CIHR was created in 2000 under the authority of the Canadian Institutes of Health Research Act.
CIHR’s mandate is to excel, according to internationally accepted standards of scientific
excellence, in the creation of new knowledge and its translation into improved health for
Canadians, more effective health services and products and a strengthened Canadian health care
system.
In Budget 2011, the Government of Canada pledged to support advanced health-related research,
such as the Strategy on Patient-Oriented Research. Subsequently, in Budgets 2012, 2013 and
2014, the Government further strengthened its commitment to SPOR in its efforts to support.
These investments in high quality research will help create and apply new knowledge that can
improve health outcomes for Canadians, lead to innovative products and services that improve
Canada’s health care system and create high quality employment and commercial opportunities.
The Government of Canada’s Science & Technology Strategy identified four areas of research
that are in the national interest from a social and economic perspective, with one of them being
“health and related life sciences and technologies”. In addition, the Science, Technology, and
Innovation Council has articulated the importance of health research as a national priority and
the need to aggressively pursue strategic international science, technology and innovation
partnerships to advance Canadian interests.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
6
CIHR’s Strategic Plan, Health Research Roadmap, incorporates key elements of the Government
of Canada’s Science & Technology Strategy and the research priorities identified by the Science,
Technology, and Innovation Council2. The Roadmap identifies five health research priorities
including:
1. enhancing patient-oriented care and clinical results;
2. supporting a high-quality, accessible and sustainable healthcare system
3. reducing health inequities of Aboriginal people and other vulnerable populations
including those in the North;
4. preparing for and responding to emerging threats to health; and
5. reducing the burden of chronic disease and mental illness.
SPOR, which aims to improve health outcomes and enhance patients' health care experience
through integration of evidence in the health care system, addresses the issues identified in four
of the five priorities of CIHR. More specifically, SPOR sets out to achieve the following goals:





2.3
to create a collaborative, pan-Canadian process for identifying, establishing and
addressing patient-oriented research priorities;
to establish an integrated, leading-edge pan-Canadian clinical research infrastructure
along the full continuum of patient-oriented research;
to grow Canada's capacity to attract, train and mentor health care professionals and health
researchers, as well as to create sustainable career paths in patient-oriented research;
to strengthen organizational, regulatory and financial support for clinical studies in
Canada and enhance patient and clinician engagement in these studies; and
to improve processes for the early identification of best practices, expedite their
development and harmonization into guidelines for patient care and support their
adoption by clinicians, caregivers and patients.
Target Populations
All Canadians will benefit from SPOR, as it will lead to:




2
improved health for Canadians by ensuring that the best research evidence moves into
practice, enhancing the health care experience for patients and improving health
outcomes for Canadians;
economic benefits by optimizing spending on health care systems, reinvesting resources
where the evidence shows greatest impact, and attracting private investments in
evaluative research;
innovation in patient-centred care in areas such as e-health, implementation science and
clinical practice;
more clinical research by improving the environment for clinical research in Canada; and
The Roadmap is currently being refreshed. Actions will be taken to ensure that SPOR remains aligned with CIHR
priorities.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)

7
collaboration among provinces and territories by providing jurisdictions with
opportunities to learn from each other, translating best practices in patient-centred care
across Canada, to the benefit of all Canadians.
The patient will benefit from SPOR through receiving the right care in the right place at the right
time. Patient-oriented researchers will benefit from training, research support services and
improved environment for clinical research. Health care professionals and policy makers will
benefit from the timely and efficient translation of research innovations from the research setting
to patient care settings and the evaluation and synthesis of existing knowledge and its proper
transfer to the clinical setting. Finally, the provincial governments and health care administrators
will benefit from a cost-effective, efficient, affordable health care system that meets the needs of
patients and continuously improves the health of Canadians.
2.4
Stakeholders
Federal departments
Other federal departments with an interest in issues relevant to SPOR may be consulted where
appropriate. This includes:








Health Canada
Industry Canada
Foreign Affairs, Trade and Development Canada
Public Health Agency of Canada
Statistics Canada
Treasury Board Secretariat
Ministry of Finance
Privy Council Office
Provincial and territorial governments and funding agencies
SPOR partners maintain linkages with provincial and territorial governments, primarily through
the following mechanisms which have responsibilities to address health issues:



Federal/Provincial/Territorial Ministers Responsible for Health
Federal/Provincial/Territorial governments and agencies
National Alliance of Provincial Health Research Organizations (NAPHRO)
National Partners and Stakeholders
SPOR maintains linkages with many players concerned with patient-oriented research, including
patients and caregivers, health charities and other not-for-profit organizations, researchers,
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
8
academic institutions, health practitioners, health organizations, and pharmaceutical sector. Some
examples are provided in the following:









Health Charities Coalition of Canada (HCCC)
Graham Boeckh Foundation (GBF)
Canadian Institute for Health Information (CIHI)
Association of Universities and Colleges of Canada (AUCC)
Canadian Association of University Research Administrators (CAURA)
Canadian Medical Association (CMA)
Canadian College of Health Leaders (CCHL)
Canada's Research-Based Pharmaceutical Companies (Rx&D)
HealthCareCAN
International Stakeholders
SPOR maintains linkages with several international organizations, which includes:







2.5
National Institutes of Health (NIH, US)
Patient-Centered Outcomes Research Institute (PCORI, US)
Medical Research Council (MRC, UK),
National Institute for Health & Clinical Excellence (NICE, UK),
National Coordinating Centre for Public Engagement (NCCPE, UK)
James Lind Alliance (UK)
National Health and Medical Research Council (NHMRC, Australia)
Governance
The governance structure of SPOR consists of a National Steering Committee and a CIHR
Working Group. The governance structure is supported by the Strategic Initiatives Branch of
CIHR. In addition, each SUPPORT Unit and SPOR Network is required to have a governance
structure, which includes appropriate mechanisms for patient engagement.
SPOR National Steering Committee
The National Steering Committee oversees the development and implementation the Strategy for
Patient-Oriented Research.
The committee is co-chaired by the the Deputy Minister of the Ontario Ministry of Health and
Long Term Care and the President of CIHR. Members include patients,
federal/provincial/territorial governments, provincial health research funders, academic
institutions, health care organizations, researchers, health charities and industry. The national
steering committee’s composition also respects the nationwide geographical representation.
CIHR provides secretariat services for the national steering committee.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
9
SPOR Working Group
The SPOR Working Group provides scientific leadership within CIHR for the development,
implementation and co-ordination of CIHR’s activities and initiatives related to SPOR. The
working group also provides on-going monitoring and identifies refinement of activities and
initiatives, as needed. Ad-hoc external advisory committees are engaged for advices on various
specific issues.
The working group is chaired by the Chief Scientific Officer, and consists of five Scientific
Directors, each being a champion of one of the five SPOR components. It also includes the
Director of Strategic Initiative Branch, Director of Partnerships and Business Development,
Manager of Major Initiatives and two Assistant Directors.
2.6
Resources
An overview of existing funding resources specifically dedicated to SPOR for fiscal year 201314 is provided in Table 2.1 below:
Table 2.1 Funding for SPOR
Activities
Personnel costs
EBP
Operations and Maintenance (O&M)
Grants
2013-14
1,730,594
346,119
1,316,683
47,395,646
Accommodations
224,977
Total Allocation
51,014,019
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
3.0
Logic Model
3.1
Logic Model Overview
10
SPOR’s logic model provides a graphic illustration of how its activities will contribute to the
achievement of the expected outcomes.
Since SPOR aims to change the health research culture in Canada, it is estimated that it will take
at least ten years to get evidence-based patient-oriented research integrated in the health care
system, at the clinical practice and decision-making levels. Over its ten-year span, the Strategy
will make patient-oriented research a major focus of health research at CIHR and in Canada.
Due to its complexity and the extensive stakeholder engagement that is required, there have been
delays in its implementation. This delay was necessary in order to allow key partners, including
provincial governments, time to mobilize resources, identify matching funding and develop
strong business cases. The ten year life cycle of SPOR and the delay in its implementation are
important factors to consider in performance measurement and evaluation.
The following figure depicts the Strategy’s logic model, which focuses on the strategic level
outcomes. It is followed by a narrative description of its activities, outputs and outcomes.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
11
The Strategy for Patient-Oriented Research (SPOR) will demonstrably improve health outcomes and enhance patients' health care experience
through integration of evidence at all levels of health care.
SPOR will deliver on this objective through the following elements: SUPPORT Units, SPOR Networks, Capacity Development, Improving the Clinical Trials Environment, and
Patient Engagement
Activities
Outputs
(Ongoing)
(Ongoing)
• Patient & stakeholder
engagement in governance,
decision-making processes,
research & knowledge
translation
• Design and manage SPOR
funding tools and platforms
• Provide support to, and facilitate
links between, SPOR elements
• Overssee the implementation of
the National Clinical Trials
Summit Action Plan
• Develop, implement and
maintain the governance
structure
• Multi-disciplinary and crosssectorial partnerships
• Funding issued for:
• SUPPORT units
• Networks
• Canadian Clinical Trials
Coordinating Centre (CCTCC)
• Patient engagement
• Patient Engagement and
Capacity Development
frameworks
• Knowledge translation products
& events
• Performance measurement &
evaluation reports
• Ongoing monitoring of
performance of SPOR overall
and its elements
• Communication & promotion
products
• Communications and promotion
• Meetings & workshops
Immediate
Outcomes
Intermediate
Outcomes
(3 - 5 years)
(5+ years)
• Key stakeholders, including
patients, are supported to
engage in research & decisionmaking processes
• Patients, health care providers,
and decision-makers are active
partners in both research &
implementation of evidencebased improvements
• Cultural change in patientoriented research that values
the involvement of, and
interaction between, all
stakeholders
• Infrastructure & support
services are responsive to
stakeholder & health systems
needs
• Improved patient health
outcomes
• Multi-disciplinary, integrated &
accessible research platforms
are established
• Pan-Canadian networks are
mobilized to address health
challenges
• SPOR elements provide
capacity development
opportunities that align with the
SPOR Capacity Development
framework
• CCTCC is established
• Methods in patient-oriented
research are advanced
• Research evidence is
generated, published,
disseminated and integrated
into health care practice &
policy
• Strengthened Canadian
capacity in patient-oriented
research
• Improved clinical trial
environment in Canada through
CCTCC, specialized services &
methodological expertise
Ultimate Outcomes
(10+ years)
• Enhanced patient health care
experience
• Improved Canadian health care
system through evidence-based
practice.
• Canada is recognized
internationally as a favoured
environment for the conduct of
clinical trials
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
3.2
12
Logic Model Narrative
3.2.1 Activities
The main SPOR activities are as follows:
Patient & stakeholder engagement in governance, decision-making processes, research &
knowledge translation: Patients and stakeholders are engaged at all stages of SPOR, from the
development of the strategy through to the implementation of its components. SPOR was
developed after a series of consultations were held with relevant stakeholders, including officials
from the federal, provincial and territorial governments and provincial health research funding
agencies. To implement SPOR, CIHR is actively working with these stakeholders on an ongoing
basis. Most importantly, patients are involved in key governance structures and decision-making
processes in all SPOR elements and are supported by a variety of tools and resources to
participate meaningfully in all stages of research reprocess. .
Design and manage SPOR funding tools and platforms: The core components of SPOR include
complex funding tools and platforms. It is important to collaborate with stakeholders in their
design and management so that SPOR outcomes are fully achieved. In addition, innovative
approaches will be used to facilitate research across the full spectrum of patient-oriented
research.
Provide support to, and facilitate links between, SPOR elements: CIHR organizes meetings and
workshops to bring together participants and stakeholders of SPOR to provide opportunities for
knowledge sharing and to facilitate greater collaborations. CIHR also co-chairs and provides
secretariat support to a national SPOR SUPPORT Unit Council which has been established to
coordinate activities across the various SUPPORT Units and ensure a sharing of best practices
and lessons learned.
Oversee the implementation of the National Clinical Trials Summit Action Plan: The National
Clinical Trials Summit in 2011 resulted in “An Action Plan to Help Attract More Clinical Trials
to Canada.” Together with the Association of Canadian Academic Healthcare Organizations
(ACAHO) and Canada's Research-Based Pharmaceutical Companies (Rx&D), CIHR has
established a coordination centre to implement the action plan.
Develop, implement and maintain the governance structure: A National Steering Committee was
established to oversee the development and implementation of SPOR. A Working Group was
also struck to provide leadership within CIHR for the development, implementation and
oversight of SPOR. Both the National Steering Committee and the Working Group meet
regularly to address emerging issues and to make decisions on the direction of SPOR activities.
Ongoing monitoring, performance of SPOR overall and its elements: CIHR collects on-going
data for performance measurement and reporting to Parliament, such as in the annual
Departmental Performance Report. In addition, CIHR is working with the SUPPORT Units and
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
13
the SPOR networks to collectively develop relevant performance measurement frameworks for
the ongoing collection and analysis of performance information. CIHR will lead the evaluation
of the SPOR, including the SUPPORT Units and the SPOR networks, to assess the Strategy’s
design, delivery and performance to date and meeting federal government evaluation coverage
requirements.
Communications and promotion: CIHR works closely with the office of the Minister of Health to
promote the SPOR program to Canadians, stakeholders, and researchers. Communications
efforts are focused on explaining patient-oriented research and its benefits to Canadians,
engaging stakeholders, and promoting funding opportunities to researchers and partners.
3.2.2 Outputs
The primary SPOR outputs are as follows:
Multi-disciplinary and cross-sectorial partnerships: The core components of SPOR require
matching funding from non-federal government partners. This funding can come from a variety
of public, private for-profit and/or private not-for-profit sources, including ministries of health,
social services, and/or education, provincial health research funding organizations, health
charities, and private foundations.
Funding issued for SUPPORT units, Networks, CCTCC and Patient Engagement: At capacity,
SPOR will provide funding to up to nine SUPPORT Units across Canada. Up to ten networks
will also be funded to link the best researchers across Canada to undertake patient-oriented
research in designated health areas. Funding has been issued to the Canadian Clinical Trials
Coordinating Centre to address the recommendations outlined in the Summit Action Plan. A
variety of funding opportunities will build capacity for more meaningful patient engagement in
research process.
Meetings and Workshops: As part of SPOR’s governance structure, the National Steering
Committee meets up to two times per year. The SPOR Working Group also meets regularly. In
addition to these governance meetings, workshops will provide substantial opportunities for
exploring potential synergies, linkages, and partnerships for those involved in different SPOR
elements, including SPOR Networks and SUPPORT Units.
Patient Engagement and Capacity Development frameworks: A SPOR Patient Engagement
Framework has been developed to guide active and fruitful collaborations between patients, researchers,
health care professionals, decision-makers and other stakeholders throughout the entire health research
continuum. A SPOR Capacity Development Framework will be developed after consultations with
relevant stakeholders.
Knowledge translation products & events: CIHR will undertake knowledge translation activities to
ensure the results of SPOR-related research are disseminated to a wide audience. Knowledge translation
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
14
products may include: plain language publications, scientific publications, educational material, mass
media products or social media products.
Performance measurement & evaluation report: CIHR requires annual performance reporting related
to progress towards, or achievement of, the SPOR outputs and outcomes from each of the SUPPORT Unit
and Network elements. These elements are also required to submit mid-term and final evaluation reports.
In alignment with TBS requirements, a program evaluation report for SPOR will be produced.
Communication & promotion products: Apart from maintaining a website to provide accurate
and up-to-date information regarding SPOR, products are developed and maintained to support
SPOR communication activities, including roll-up banners, brochures, fact sheets and
newsletters.
3.2.3 Immediate Outcomes
The immediate outcomes that are expected to result from the SPOR activities and outputs are as
follows:
Key stakeholders, including patients, are supported to engage in research & decision-making
processes: In accordance with the SPOR Patient Engagement Framework CIHR will collaborate
with relevant stakeholders to identify mechanisms to support patients, researchers, health care
professionals and decision-makers to work, CIHR will provide funding opportunities, facilitate
seminars, workshops and other events, and share knowledge and best practices. CIHR will also
work closely with SPOR Networks and SUPPORT units to support them in embedding patient
engagement in their work.
Multidisciplinary, integrated and accessible research platforms are established: As the funding
is released, SUPPORT units will be established across Canada to provide the necessary expertise
to pursue patient-oriented research and help lead reforms in response to locally-driven health
care needs. Each of these Units will have its own governance structure, will be staffed with
multi-disciplinary professionals to deliver highly specialized data, to provide integrated and
accessible methodological and research services, and be equipped with both a performance
measurement plan and a sustainability plan.
Pan-Canadian networks are mobilized to address health challenges: The SPOR Networks will
be mobilized as collaborative alliances that incorporate the perspectives of all stakeholders,
including patients, health care professionals and decision makers, in research and knowledge
translation activities that lead to integration of evidence into practice. These networks will be
national in scope and implement a coordinated nationwide patient-oriented research agenda in
collaboration with relevant stakeholders, particularly patients. Each network will include a
governance and management structure with appropriate mechanisms to ensure patient
engagement, and will be equipped with both a performance measurement plan and a
sustainability plan.
SPOR elements provide capacity development opportunities that align with the SPOR Capacity
Development framework: In accordance with the SPOR Capacity Development Framework,
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
15
indirect training, mentoring and career support will be integrated into the SPOR Networks and
SUPPORT Units. The unique research and research support environments created will be
leveraged to provide unique cross-disciplinary training environments for the next generation of
patient-oriented researchers.
Canadian Clinical Trials Coordinating Centre is established: The established CCTCC will
complete the summit action plan’s recommendations, serve as a coordinating function for
relevant provincial and national initiatives, and determine next steps upon completion of the
action plan. The CCTCC will be guided by an Executive Committee comprised of the Summit
partners.
Methods in patient-oriented research are advanced: The core functions of SPOR SUPPORT Units
include providing specialized services and methodological expertise to stakeholders within the clinical
trial environment. The responsiveness of SUPPORT Unit services and expertise to the needs of
stakeholders will result the advancement of knowledge in methods in patient-oriented research. Further
advancement of knowledge and methods will occur as a result of patient-oriented research conducted by
SPOR Networks.
3.2.4 Intermediate Outcomes
The intermediate outcomes that are expected to result from the SPOR activities, outputs and
immediate outcomes are as follows:
Patients, health care providers and decision-makers are active partners in both research &
implementation of evidence-based improvements: As new resources and mechanisms are
developed, and linkages to the existing ones fully utilized, patients, health care providers and
decision-makers will be informed about the broad spectrum of research-related activities and
actively involved in those. With the infrastructures in place, and strong commitments from the
stakeholders, patients, health care providers and decision-makers will be involved in both the
research and decision-making processes.
Infrastructure and support services are responsive to health systems needs: As a result of the
establishment of the SUPPORT Units, timely access to data and specialized and
multidisciplinary methodological expertise will be provided. These units will also respond to the
needs of health systems by identifying and addressing the needs of patients and knowledge users,
facilitating research; and assisting decision makers to apply the knowledge generated by
research.
Research evidence is generated, published, disseminated and integrated into health care practice
and policy: SPOR Networks are designed to be national research collaborations involving
patients, researchers, health care professionals and policy makers across Canada. Together, they
conduct research and help bridge the gap between research evidence and health care practice.
Networks aim to generate evidence and innovations that will lead to transformative and
measurable improvements in patient health, health care, and the efficiency and effectiveness of
service delivery.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
16
Strengthened Canadian capacity in patient-oriented research: With enhanced training and career
support provided to a full spectrum of patient-oriented researchers including clinical
investigators, methodologists, epidemiologists, biostatisticians, health economists and healthrelated social scientists, Canada will be more able to attract, train and mentor health care
professionals and patient-oriented researchers, as well as to create sustainable career paths in
patient-oriented research.
Improved clinical trial environment in Canada through CCTCC, specialized services &
methodological expertise: As the Summit Action Plan is implemented, Canada will achieve trust
and profitability through operational efficiencies in those areas that impact clinical trial start up
times and costs. The clinical trial environment will also be improved because the SUPPORT
Units will provide expert assistance in areas such as innovative trial design, data management,
statistical analysis, ethics approvals and multi-site management for clinical trials.
3.2.5 Ultimate Outcomes
Ultimately, SPOR aims to improve health outcomes and enhance patients' health care experience
through integration of evidence at all levels in the health care system.
Cultural change in patient-oriented research that values the involvement of, and interaction
between, all stakeholders: When all participants in patient-oriented research – patients,
researchers, health care professionals and decision makers – are trained and supported to work
together, and when it happens routinely, the value of collaboration will become apparent and a
culture shift will occur.
Improved patient health outcomes: Patient health outcomes will be improved when the patient
receives the right care, in the right place at the right time. This will happen when SPOR enables
health care professionals and policy makers integrate the latest evidence in their practices and
decision making processes.
Enhanced patient health care experience: When patients participate in all aspects of the research
and decision making processes, from setting research priorities to implementing evidence into
practice, policy makers and health practitioners will be better able to understand their needs and
deliver high-quality services responding to those needs. Ultimately, the patient health care
experience will be enhanced when health services and products are truly patient-oriented.
Improved Canadian health care system through evidence-based practice: SPOR will enable the
timely and efficient translation of research innovations from the research setting to patient care
settings, the evaluation and synthesis of existing knowledge, and the proper transfer of existing
knowledge to the clinical setting. This will ultimately lead to evidence-informed improvements
to the system.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
17
Canada is recognized internationally as a favoured environment for the conduct of clinical
trials: By overcoming the barriers identified at the Summit, Canada’s competitiveness in
conducting clinical trials will be increased, and it will once again be recognized as an
international player in conducting clinical trials.
3.3
Risk Factors that may Influence SPOR’s Ability to Achieve Results
The factors that could influence SPOR's ability to achieve the results as described in this Strategy
include: the ability to secure matching funds, the capacity to ensure the required level of
oversight and monitoring of SPOR activities; and the ability to coordinate a large number of
different SPOR components and create synergy.
To the extent possible, mitigation strategies have been developed by to address the balance of
risks outlined above. Actions have been taken to engage partners for matching funds through
ongoing communication efforts, as well as hosting workshops to provide opportunities for
exploring potential partnerships for those involved in different SPOR components. CIHR will
ensure adequate internal resources are in place with specialized skill sets relating to analysis,
governance and performance measurement. Each funded components will be required to report
on its progress and conform to a performance measurement framework established as part of the
funding agreement. Progress and results of SPOR as a whole will be assessed through the
performance measurement and evaluation strategies established in this Strategy. Systems and
protocols are in place to facilitate performance measurement and evaluation. Finally, several
governance structures have been created to coordinate different SPOR components, which are
also guided by the SPOR Patient Engagement Framework and the SPOR Capacity Development
Framework. An annual SPOR summit will also be created so that SPOR participants, partners
and stakeholders gather together to establish links with and learn from each other.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
4.0
18
Performance Measurement Strategy Framework
This is intended to be a high-level measurement framework for the overall performance of SPOR
in order to report to Treasury Board. It is not intended to capture all outcomes of each SPOR
component. Rather, it shows how the components collectively lead to the outcomes at the
strategy level. Some of the indicators listed are a roll-up of performance measurement data from
individual SPOR components. More specificity for these indicators will come when the
milestones and performance indicators for the individual components are finalized. In addition,
some of the indicators related to patient engagement and training are listed as “to be developed”
awaiting the completion of the respective strategies. Performance measures will be reviewed
annually to ensure proper alignment with the implementation of SPOR.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
19
Table 4.1: SPOR Performance Measurement Strategy Framework
SPOR outputs &
outcomes
OUTPUTS
Multi-disciplinary &
cross-sectional
partnerships
Indicator
Data source
Frequency
of data
collection
Baseline
Target
Date to
achieve
target
Unit
Responsible for
data collection
Number of formal agreement (i.e.,
number of SPOR funding partners)
Administrative
data / EIS3;
Annual reports
Administrative
data / EIS;
Annual reports
Administrative
data / EIS;
Annual reports
SPOR documents
Administrative
data / EIS
Administrative
data / EIS
Annual
0
Increase
TBD
Finance; Major
Initiatives
Annual
0
Increase
TBD
Finance; Major
Initiatives
Annual
0
1:1
May 2013
Finance; Major
Initiatives
Annual
Annual
0
0
Increase
9
2014
April 2014
Annual
0
8
TBD
Major Initiatives
Performance &
Accountability
Performance &
Accountability
Administrative
data / EIS
SPOR documents
Annual
$0
$250k/yr
2014
Finance
N/A
N/A
N/A
TBD
Major Initiatives
SPOR documents
N/A
N/A
N/A
TBD
Major Initiatives
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
Percent of formal agreements with
‘non-traditional’ partners
Ratio of leveraged funds
(SPOR $: Partner $)
Funding issued
Number of non-funding partners
Number of SUPPORT units grants
issued
Number of Networks grants issued
CIHR contribution to CCTCC
Patient Engagement
and Capacity
Development
frameworks and other
tools
Capacity Development framework
finalized
Patient Engagement framework
finalized
Knowledge translation
products & events
Number of CIHR-produced SPOR KT
products by type4
3
CIHR’s electronic information system (EIS)
4
SU report template collects by the following KT product types: Plain language publications; Scientific publications; Educational material; Mass media
products; Social media products; Other (further categories may be developed after review of the first round of data collection).
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
SPOR outputs &
outcomes
Performance
measurement &
evaluation reports
Communication &
promotion products
Meetings & workshops
IMMEDIATE OUTCOMES
Key stakeholders,
including patients, are
supported to engage
in research &
5
20
Indicator
Data source
Number of CIHR-held SPOR events
by type5
% of SPOR elements completing a
mid-term evaluation as planned
% of SPOR elements completing a
final evaluation as planned
Number of CIHR PM and evaluation
reports
Number of CIHR-held SPOR
communication and promotion
events6
Number of CIHR-produced SPOR
communication and promotion
products by type7
Number of National Steering
Committee meetings held
Number of SPOR working group
meetings held
Number of workshops held
Number of orientation and
engagement tools
Number of KT products (produced
by SPOR elements) by type8
Baseline
Target
SPOR documents
Frequency
of data
collection
Annual
N/A
Date to
achieve
target
N/A
Unit
Responsible for
data collection
Major Initiatives
N/A
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
SPOR & CIHR
documents
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
Annual
N/A
N/A
N/A
Major Initiatives
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
SU report template collects by the following KT event types: Workshop; Conference; Webinar; Presentations; Other (further categories may be developed after
review of the first round of data collection).
6
SU report template collects by the following communications event types: Media releases; Interview; Other (further categories may be developed after review
of the first round of data collection).
7
SU report template collects by the following communication product types: Plain language publications; Mass media products; Social media products; Other
(further categories may be developed after review of the first round of data collection).
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
21
SPOR outputs &
outcomes
Indicator
Data source
decision-making
processes
Number of KT products (produced
by SPOR elements) by primary
knowledge user9
Number of KT events (held by SPOR
elements) by type10
Number of KT events (held by SPOR
elements) by primary knowledge
user11
Number of training opportunities
(delivered by SPOR elements)
directed towards nonresearcher/academic audiences by
focus of training12
Number of patients on CIHR SPOR
decision-making committees
Percent of SUPPORT units meeting
developmental milestones
regarding:
 Governance structure
 Staffing
 Infrastructure
 Performance measurement
 Stakeholder engagement
 Sustainability plan
Multi-disciplinary,
integrated &
accessible research
platforms are
established
8
Baseline
Target
SPOR documents
Frequency
of data
collection
Annual
N/A
Date to
achieve
target
N/A
Unit
Responsible for
data collection
Major Initiatives
N/A
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
SPOR documents
Annual
N/A
N/A
N/A
Major Initiatives
Annual reports
Annual
N/A
100%
2 years
after
funding
released
to
SUPPORT
unit
Major Initiatives
Same as footnote #2.
SU report template collects by the following primary user types: Patients & informal caregivers/ consumers of health care; Consumer group/charitable
organization; Researchers / academics; Health system/care practitioners; Health system/care administrators or managers; Health system/care professional
organization; Policy makers; Media; Other.
10
Same as footnote #3.
11
Same as footnote #7..
12
SU report template collects by the following training focus categories: Research methods and key competencies; Knowledge translation; Leadership in
research; Leadership in health systems; Patient partnerships; Other (further categories may be developed after review of the first round of data collection).
9
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
SPOR outputs &
outcomes
Pan-Canadian
networks are
mobilized to address
health challenges
SPOR elements
provide capacity
development
opportunities that
align with the CIHR
SPOR Capacity
Development
framework14
13
22
Indicator
Data source
Total number of integrated
datasets by type13
Total number of requests
completed by SUPPORT units
Percent of Networks meeting
developmental milestones
regarding:
 Governance structure
 Staffing
 Stakeholder engagement
 Performance measurement
 Sustainability plan
Number of people provided with
training, mentoring or career
support through SPOR elements by
type of trainee15
Number of courses or training
programs offered by SPOR
elements by focus of the training16
Baseline
Target
Annual reports
Frequency
of data
collection
Annual
N/A
Date to
achieve
target
N/A
Unit
Responsible for
data collection
Major Initiatives
0
Annual reports
Annual
0
N/A
N/A
Major Initiatives
Annual reports
Annual
N/A
100%
2 years
after
funding
released
to
Network
Major Initiatives
TBD
TBD
TBD
TBD
TBD
TBD
Annual reports
Annual
0
TBD
TBD
Major Initiatives
SU report template collects the following dataset types: Health services administrative; People and geography; Special collections; Cohort; Survey; Clinical
data; Chart abstraction data; Provider characteristic data; ‘Omic’; Other (further categories may be developed after review of the first round of data collection).
Definitions of each type can be found in the SU annual report appendix.
14
This section may need further development (Training Capacity framework still in draft form).
15
Same types as footnote #7 as well as Masters students, Postdoctoral students, and Postdoctoral fellows.
16
Same as footnote #10.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
23
SPOR outputs &
outcomes
Indicator
Data source
Canadian Clinical Trials
Coordinating Centre is
established
Percent of developmental
milestones met regarding:
 Governance structure
 Staffing
 Performance measurement
Annual reports
INTERMEDIATE OUTCOMES
Patients, health care
% of SPOR grants reporting
providers, and
participation of stakeholders in the
decision-makers are
research process
active partners in both
research &
implementation of
Composition of SUPPORT Unit core
evidence-based
teams and Networks by member
improvements
type17
Role of non-research/academic
SUPPORT Unit team and Networks
members by key function18
Infrastructure &
Percent of SUPPORT units meeting
support services are
mid-term milestones for relevant
responsive to
core functions19:
stakeholder and
 Data platforms & services
health system needs
 Methods support &
development
 Health systems, knowledge
translation & implementation
Frequency
of data
collection
Annual
Baseline
Target
N/A
100%
End-of-grant
report
Once
N/A
TBD
Annual reports
Annual
N/A
TBD
Mid-term report
Once
N/A
100%
17
Date to
achieve
target
1 year
after
funding
released
to CCTCC
Unit
Responsible for
data collection
Major Initiatives
18
months
post
expiry to
use funds
TBD
Major Initiatives
/ Performance &
Accountability
3 years
after
funding
released
Major Initiatives
Major Initiatives
SU report template collects the following team member types: Researchers/academics; Trainees; Health system/care practitioners; Health system
care/administrators or managers; Patients/consumers of health care; Family members of patients; Health system/care professional organization rep;
Federal/provincial rep; Community/municipal organization rep; Consumer group/charitable organization rep; Policy makers; Industry rep; Partner rep; Media
rep; other.
18
SU report template collects the team key functions in alignment with the SU core components.
19
Performance metrics for mid-term and final deliverables/milestones are under development for each core function of SUPPORT units.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
SPOR outputs &
outcomes
Indicator
24
Data source
Frequency
of data
collection
Baseline
Target
Date to
achieve
target
Unit
Responsible for
data collection
Annual reports
Annual
N/A
N/A
N/A
Major Initiatives
Annual reports
Annual
N/A
N/A
N/A
Major Initiatives
Mid-term report
Once
N/A
100%
3 years
after
funding
released
Major Initiatives


Research evidence is
generated, published,
disseminated and
integrated into health
care practice & policy
Real world clinical trials
Consultation & research
services
Number of relevant service or
advice requests to SUPPORT Units
completed by type of requestor20
Number of completed service or
advice requests to SUPPORT Units
completed by core function
Percent of networks meeting midterm milestones for Networks
program objectives21:
 Generate evidence
 Nation-wide research agenda
 Develop & implement
evidence-informed practice,
policy, service, products,
programs
 Translate knowledge into
clinical applications, practice,
policy
20
SU report template collects the following type of requestor: SPOR Networks; Other SPOR SUPPORT Units; Other non-SPOR researchers/academics; Heatlh
care/system practitioners; Health care/system administrators or managers; Patients and informal caregivers/consumers of health care; Policy makers; Other.
21
Performance metrics for mid-term and final deliverables/milestones are under development for each objective of the Networks program. Objectives for the Networks program are to: 1) to generate
evidence & innovations that will lead to transformative & measureable improvements in health outcome & health care; 2) establish priorities & develop a nation-wide agenda of outcomes-oriented
research; 3) develop & implement evidence-informed practices, policies, services, products, and programs to improve the quality, efficiency , and effectiveness of care; 4) Accelerate the translation
of new knowledge into clinical applications, practice, & policy; 5) strengthen the capacity to conduct patient-oriented research through training & mentoring initiatives.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
SPOR outputs &
outcomes
Strengthened
Canadian expertise &
capacity in patientoriented research
22
25
Indicator
Data source
Frequency
of data
collection
Annual
Baseline
Target
Number of peer-reviewed
publications from SPOR elements
Annual reports
N/A
N/A
N/A
End-of-grant
report
Once
N/A
% of SPOR grants reporting use of
findings in health care practice &
policy
End-of-grant
report
Once
N/A
TBD
Share of top 5% cited SPOR-related
publications acknowledging CIHR
Bibliometric
contract
Bi-annually
N/A
N/A
18
months
post
expiry to
use funds
18
months
post
expiry to
use funds
N/A
Number of citations of findings
from SPOR grants in non-scientific
(e.g., policy) publications
Number of SPOR trainees earning
formal qualifications by type of
degree
Bibliometric
contract
Bi-annually
N/A
N/A
N/A
End-of-grant
report
Once
N/A
TBD
% of SPOR trainees employed in the
Canadian research enterprise
End-of-award
survey22
Once
TBD
TBD
18
months
post
expiry to
use funds
18
months
post
expiry to
use funds
Currently under development.
Date to
achieve
target
N/A
Unit
Responsible for
data collection
Major Initiatives
/ Performance &
Accountability
Major Initiatives
/ Performance &
Accountability
Major Initiatives
/ Performance &
Accountability
Major Initiatives
/ Performance &
Accountability
Major Initiatives
/ Performance &
Accountability
Major
Initiatives /
Performance &
Accountability
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
SPOR outputs &
outcomes
Improved clinical trial
environment in
Canada through
CCTCC, specialized
services and
methodological
expertise
ULTIMATE OUTCOMES
Cultural change in
patient-oriented
research that values
the involvement of,
and interaction
23
26
Indicator
Data source
Frequency
of data
collection
Biannual
Baseline
Target
N/A
Increase
Date to
achieve
target
TBD
Share of top 5% of cited POR
publications authored by Canadians
Bibliometrics
contract
Average relative citations
Bibliometrics
contract
Biannual
N/A
Increase
TBD
Percent of SUPPORT units meeting
mid-term milestones for ‘career
development’ core function2
Mid-term report
Once
N/A
100%
Percent of Networks meeting midterm milestones for ‘training &
mentoring’ objective3
Mid-term report
Once
N/A
100%
3 years
after
funding
released
3 years
after
funding
released
Percent of CCTCC mid-term
milestones achieved for each
Action Plan recommendation23
CCTCC mid-term
report
Once
N/A
100%
Number of clinical trials in Canada
Health Canada
TBD
TBD
Stakeholder perception of the
extent to which their involvement
in POR is valued
Researcher perceptions of the
Survey /
interviews?
TBD
Survey /
TBD
Unit
Responsible for
data collection
Major Initiatives
/ Performance &
Accountability
Major Initiatives
/ Performance &
Accountability
Major Initiatives
Major Initiatives
Major Initiatives
Increase
3 years
after
funding
issued
TBD
N/A
N/A
N/A
N/A
N/A
N/A
Major Initiatives
/ Performance &
Accountability
Major Initiatives
Performance metrics for mid-term and final deliverables/milestones are under development for each Action Plan recommendation. The nine recommendations are found at
http://www.acaho.org/?document&id=345.
Major Initiatives
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
27
SPOR outputs &
outcomes
Indicator
Data source
between, all
stakeholders
Improved patient
health outcomes
involvement of stakeholders in POR
interviews?
% of SPOR grants reporting
contribution to improved health of
Canadians
Enhanced patient
health care experience
Improved Canadian
health care system
through evidencebased practice
24
Frequency
of data
collection
Baseline
Target
Date to
achieve
target
End-of-grant
report
Once
N/A
TBD
To be developed (specific health
outcome measures will align with
the research area of Networks and
SPOR grants)
TBD
TBD
TBD
TBD
18
months
post
expiry to
use funds
TBD
Percent of patients CCHS24
respondents reporting difficulties
accessing routine or on-going care
Percent of CCHS respondents
reporting difficulties accessing
health information or advice
Percent of CCHS respondents
reporting difficulties accessing
immediate care for a minor health
problem
% of SPOR grants reporting
contribution to more effective
health services and products
CCHS / Statistics
Canada CANSIM
Biennial
16.8%
Decrease
2023
Major Initiatives
CCHS / Statistics
Canada CANSIM
Biennial
16.4%
Decrease
2023
Major Initiatives
CCHS / Statistics
Canada CANSIM
Biennial
21.7%
Decrease
2023
Major Initiatives
End-of-grant
report
Once
N/A
TBD
18
months
post
expiry to
use funds
Major Initiatives
/ Performance &
Accountability
Canadian Community Health Survey
Unit
Responsible for
data collection
/ Performance &
Accountability
Major Initiatives
/ Performance &
Accountability
Major Initiatives
/ Performance &
Accountability
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
SPOR outputs &
outcomes
Indicator
Data source
Potentially avoidable mortality rate
(age-standardized per 100,000
population)
Canadian Vital
Statistics,
Statistics Canada
CANSIM
Canadian Vital
Statistics,
Statistics Canada
CANSIM
Canadian Vital
Statistics,
Statistics Canada
CANSIM
Canadian Vital
Statistics,
Statistics Canada
CANSIM
End-of-grant
report
Mortality rate from preventable
causes (age-standardized per
100,000 population)
Mortality rate from treatable
causes (age-standardized per
100,000 population)
Number of deaths due to
complications of medical and
surgical care
Canada is recognized
internationally as a
favoured environment
for the conduct of
clinical trials
28
Percent of networks meeting final
milestones for Networks program
objectives3:
 Develop & implement
evidence-informed practice,
policy, service, products,
programs
 Translate knowledge into
clinical applications, practice,
policy
Number of international clinical
trials conducted in Canada
Number of citations of Canadianconducted clinical trials
Percent of CCTCC final milestones
achieved for each Action Plan
Frequency
of data
collection
Annual
Baseline
Target
Decrease
Date to
achieve
target
2023
Unit
Responsible for
data collection
Major Initiatives
178.8
Annual
116.0
Decrease
2023
Major Initiatives
Annual
62.8
Decrease
2023
Major Initiatives
Annual
213
Decrease
2023
Major Initiatives
Once
N/A
100%
5 years
after
funding
released
Major Initiatives
TBD
TBD
TBD
Increase
TBD
Major Initiatives
Bibliometrics
database
CCTCC end-ofgrant report
TBD
TBD
Increase
2023
Major Initiatives
Once
N/A
100%
End-ofgrant
Major Initiatives
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
SPOR outputs &
outcomes
Indicator
recommendation4
29
Data source
Frequency
of data
collection
Baseline
Target
Date to
achieve
target
Unit
Responsible for
data collection
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
5.0
30
Evaluation Strategy
The evaluation strategy provides a high-level overview of the evaluation plan for SPOR. The
proposed approach, scope and timing for the evaluation respond to the following key drivers:
 commitments made in the 2012 and 2011 Treasury Board submissions;
 requirements under the Financial Administration Act and the Policy on Evaluation; and
 the need to provide CIHR and SPOR management with valid, insightful and useful
findings regarding the ongoing relevance, design and delivery, and performance of the
Strategy since inception.
In particular, the evaluation strategy enables CIHR senior management, SPOR management and
the Chief Audit & Evaluation Executive (in capacity as head of evaluation) to:
 ensure that the data generated through this performance measurement strategy will
effectively support evaluation and will be available in a timely manner;
 identify the rationale, need and timing for the evaluation to inform and align with the
CIHR’s Evaluation Plan to ensure it meets legislated and/or policy-driven evaluation
deadlines; and
 engage in early planning for the SPOR evaluation and develop rigorous cost-effective
evaluation approaches and designs.
The evaluation of SPOR will be led by CIHR’s Evaluation Unit and is scheduled to begin in
2014-15 with expected completion in 2015-16. The evaluation will enable a mid-term
assessment of the Strategy’s relevance, design and delivery, and performance to report on the
results achieved to date and ensure that CIHR complies with Treasury Board policy requirements
and commitments in previous Treasury Board submissions. Given the ten-year lifecycle of
SPOR, the evaluation will focus on assessing the progress toward immediate outcomes within
the five core elements of the Strategy. The evaluation will utilize primary and secondary data
collection methods and draw on available performance information collected through the
performance measurement strategies of SPOR and its components (e.g., SUPPORT Units and
Networks).
A set of preliminary evaluation questions are outlined in the evaluation framework (see Table
5.1), which address the key information needs of CIHR senior management and Treasury
Board’s core evaluation issues.
5.1
SPOR Evaluation Framework
The draft evaluation framework presented below includes the following:
 initial evaluation questions covering the five core evaluation issues and key issues
identified by the CIHR and SPOR management;
 indicators, data sources and methods of data collection; and, where applicable,
 information on whether or not baseline data needs to be collected.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
31
The evaluation framework is revisited annually as SPOR is implemented based on the available
information collected and reported with respect to the implementation of SPOR’s components
(e.g., networks, SUPPORT units, action plans), activities (e.g., funding tools/platforms,
communication/engagement strategies) and performance to ensure that the evaluation's data
requirements are met. Specifically, the updated framework proposed an in-depth assessment of
the core elements that were implemented earlier in the SPOR lifecycle (e.g., case studies of
SUPPORT Units and Networks) and broader assessment of more recently implemented core
elements (e.g., CCTCC, Patient Engagement Framework, Capacity Development Framework).
The current draft framework was updated in preparation for the planning of the evaluation
scheduled to begin in October 2014.The evaluation framework is an important preparatory tool
in the evaluation process because it will serve as a means through which to consult CIHR and
SPOR management on the evaluation issues, questions, scope, data requirements, methods,
resources and schedule.
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
32
Table 5.1: SPOR Evaluation Framework
Evaluation Questions
Relevance
1. To what extent does
SPOR addresses the
need for patientoriented research to
foster evidenceinformed health care?
2. To what extent is
SPOR aligned with
federal roles and
responsibilities?
Indicators
Data sources
Data collection methods
Baseline data required
1.1. Assessment of the nature and
extent the ongoing need in
Canada
 Existing documents and data
 Academic and professional
 Document review
 Literature review
 Environmental scan
 Key informant interviews
 No
 Document review
 Literature review
 Environmental scan
 Key informant interviews
 No
 Document review
 Key informant interviews
 Survey
 No
 Document review
 Key informant interviews
 No
 Document review
 Key informant interviews
 Survey
 No
literature
 CIHR management
 Funded researchers
 SPOR partner and
1.2. Comparative assessment of
SPOR approach to other
patient-oriented health
research programs,
approaches and/or delivery
models


2.1. Identified role for the federal
government in supporting
patient-oriented research








3. To what extent is
SPOR aligned with
federal government
and CIHR priorities?
3.1. Alignment of SPOR
objectives/expected outcomes
with federal government and
CIHR plans and priorities
3.2. Extent of alignment/overlap
of SPOR objectives with
other federal government
initiatives/funding programs









stakeholders
Existing documents and data
Academic and professional
literature
CIHR management
Funded researchers
SPOR partner and
stakeholders
SPOR documents
CIHR documents
Federal government policy
documents
CIHR management
SPOR partners and
stakeholders
SPOR documents
CIHR documents
Federal government policy
documents
SPOR staff
CIHR management
SPOR documents
CIHR documents
Federal government policy
documents
SPOR staff
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
Evaluation Questions
Indicators
33
Data sources
Data collection methods
Baseline data required
 Document review
 Key informant interviews
 Survey
 No
 Administrative data / EIS
 No
 CIHR management
 Funded researchers
 SPOR partners and
stakeholders
Design and Delivery
4. To what extent has the
SPOR been
implemented as
planned?
4.1. Assessment of effectiveness
of SPOR’s structure and
processes to implement
governance structure, engage
stakeholders, manage
funding tools/platforms,
coordinate communications,
measure performance and
manage risk
4.2. Views of researchers,
partners and stakeholders of
delivery of SPOR (e.g., peer
review, funding tools,
matching funds, knowledge
translation, and conflict of
interest)
4.3. Best practices/lessons
learned/ potential
improvements from SPOR
components implemented to
date
 SPOR documents
 SPOR staff
 CIHR management
 SPOR researchers, partners
4.4. Factors that have facilitated
or inhibited the
implementation of SPOR
 SPOR documents
 SPOR staff
 CIHR management
 Funded researchers
 SPOR partners and
and stakeholders
 SPOR documents
 SPOR researchers, partners
and stakeholders
 SPOR documents
 SPOR staff
 CIHR management
 Funded researchers
 SPOR partners and


analysis
Key informant interviews
Survey
 Document review
 Key informant interviews
 Survey
 No
 Document review
 Key informant interviews
 Survey
 No
stakeholders
stakeholders
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
Evaluation Questions
Performance
5. To what extent has
SPOR made progress
toward the
achievement of
expected immediate
outcomes?



Multi-disciplinary,
integrated and
accessible research
platforms are
established
Key stakeholders,
including patients,
are supported to
engage in research
and decisionmaking processes
Pan-Canadian
networks are
mobilized to
34
Indicators
Data sources
Data collection methods
Baseline data required
5.1. Evidence of the achievement
of immediate outcomes in
the five core elements
 Administrative data / EIS
 SUPPORT Unit documents
 SPOR documents
 SUPPORT Unit staff
 SPOR stakeholders and
 Document review
 Survey
 Focus groups
 Case studies
 No
5.2. Number of multi-disciplinary
and cross-sectorial
partnerships established (by
type, discipline, sector)
 Administrative data / EIS
 SPOR Network documents
 SPOR documents
 SPOR stakeholders and
 Administrative data / EIS
 Yes
partners
partners







analysis
Document review
Survey
Case studies of SUPPORT
Units
Document review
Survey
Focus groups
Case studies of SUPPORT
Units
5.3. Number and type of supports
and services (e.g., access to
data, provision of
methodological expertise,
advice) provided/facilitated
by SUPPORT units
 Administrative data / EIS
 SUPPORT Unit documents
 SPOR documents
 SUPPORT Unit staff
 SPOR stakeholders and
 No
5.4. Assessment of the
implementation and
performance of the SPOR
engagement strategy
5.5. Awareness among key
stakeholders of the
importance of patient
engagement in research and
decision-making processes
 SPOR documents
 SPOR management
 Document review
 Key informant interviews
 No
 SPOR documents
 SPOR Network documents
 SUPPORT Unit documents
 SPOR stakeholders and
 Document review
 Survey
 Yes
5.6. Number, type and nature of
stakeholders engaged in
SPOR research and decisionmaking
5.7. Extent to which networks
have established a national
research agenda and


 Document review
 Survey
 Yes
 Case studies of SPOR
 No
partners
partners
SPOR documents
SPOR stakeholders and
partners
 Administrative data / EIS
 SPOR Network documents
Networks
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
Evaluation Questions
address health
challenges


SPOR elements
provide capacity
development
opportunities that
align with the
SPOR Capacity
Development
framework
CCTCC is
established
Indicators
priorities in outcomesoriented research
35
Data sources
5.8. Extent to which networks
have developed and
implemented evidenceinformed practices, policies,
services, products and
programs





5.9. Number of SPOR trainees
(direct and indirect)
receiving patient-oriented
research training (by type of
support, trainee, researcher)
5.10. Assessment of quality of
patient-oriented training
(direct and indirect) provided
to researchers and health
professionals




stakeholders and partners
Administrative data / EIS
SPOR Network documents
SPOR documents
SUPPORT Network staff
SPOR Network researchers,
stakeholders and partners
Administrative data / EIS
SPOR documents
SPOR Network documents
SUPPORT Unit documents
 Direct SPOR trainees
 Indirect SPOR trainees
 SPOR Training
 SPOR staff
 SPOR Network staff
 SUPPORT Unit staff
5.11. Assessment of the
 Administrative data / EIS
implementation of the Career  SPOR documents
Development Framework to
 SPOR management
address recommendations of
SPOR External Advisory
Committee on Training and
Career Development
5.12. Assessment of the
implementation and
operation of the Canadian
Clinical Trials Coordinating
Centre (CCTCC)
5.13. Actions taken to implement
recommendations of the
Data collection methods
Baseline data required
 Case studies of SPOR
 No
 SPOR documents
 SUPPORT Network staff
 SPOR Network researchers,
Networks
 Administrative data analysis
 Document review
 Yes
 Key informant interviews
 Survey
 Focus groups
 No
 Administrative data / EIS
 No


analysis
Document review
Key informant interviews
 SPOR documents
 Document review
 CCTCC documents
 Key informant interviews
 CCTCC staff and stakeholders
 SPOR stakeholders and
 No
partners
 SPOR documents
 CCTCC documents
 Document review
 Key informant interviews
 No
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
Evaluation Questions

Methods in patientoriented research
are advanced
6. To what extent has
SPOR made progress
toward the
achievement of
expected intermediate
outcomes?
Indicators
2011 Clinical Trials Summit
Action Plan
36
Data sources
5.14. Number and type of
improvements to the clinical
trial environment (e.g.,
model templates developed,
efficiencies achieved)


5.15. Assessment of research
output of SPOR network
supported researchers (e.g.,
journal articles, books/book
chapters, reports/technical
reports)
5.16. Assessment of research
impact of SPOR network
supported publications
(ARC)


6.1. Evidence of early progress
toward the achievement of
intermediate outcomes in the
five core elements
Data collection methods
Baseline data required
 Document review
 Literature review
 Key informant interviews
 Survey
 No
 Administrative data / EIS
 Yes
 CCTCC staff and stakeholders
 SPOR stakeholders and



partners
SPOR documents
Academic and professional
literature
SPOR stakeholders and
partners
Administrative data / EIS
Academic and professional
literature
Bibliometric data
SPOR funded researchers
 Administrative data / EIS
 Academic and professional
literature
 Bibliometric data
 Administrative data / EIS
 SUPPORT Unit documents
 SPOR documents
 SUPPORT Unit staff
 SPOR stakeholders and



analysis
Literature review
Bibliometric analysis
Survey
 Administrative data / EIS






analysis
Literature review
Bibliometric analysis
Document review
Survey
Focus groups
Case studies
 Yes
 No
partners
6.2. Evidence of integrated , panCanadian, stakeholder
engaged process for the
patient-oriented research and
evidence-informed health
care
 Administrative data / EIS
 Academic and professional
6.3. Number of clinical
applications, practices,
procedures and policies
resulting from SPOR
 SPOR documents
 SPOR Network documents
 SUPPORT Unit documents
literature
 SPOR funded researchers
 SPOR stakeholders and
partners
 Administrative data / EIS




analysis
Literature review (citation
analysis)
Survey
Case studies of SUPPORT
Units
Case studies of SPOR
Networks
 No
 Yes
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
Evaluation Questions
Indicators
Network grants
37
Data sources
Data collection methods
Baseline data required
 Administrative data / EIS
 No
 SPOR stakeholders and
partners
6.4. Evidence that SPOR research
evidence has informed and
been integrated into clinical
applications, practices,
procedures and policies
 Administrative data / EIS
 Academic and professional


literature
SPOR funded researchers
SPOR stakeholders and
partners




6.5. Evidence that SPOR research
platforms and networks are
meeting needs of the health
care and systems
stakeholders and decisionmakers
 SPOR documents

 Relevant Canadian health data 
(Statistics Canada, CIHI)

 Academic and professional

literature

 SPOR stakeholders and
partners

6.6. Evidence of increased
number of health researchers
and health care professionals
involved in patient-oriented
research
 SPOR documents

 Relevant Canadian health data 
(Statistics Canada, CIHI)

 Academic and professional

literature

 SPOR stakeholders and

partners

6.7. Number and type (e.g.,
multi-center trials) of clinical
trial applications submitted
to Health Canada
 Administrative data / EIS

 Relevant Canadian health data
(Statistics Canada, CIHI)

 Health Canada documents

 Academic and professional
literature
analysis
Literature review (citation
analysis)
Survey
Case studies of SPOR
Networks
Case studies of SUPPORT
Units
Document review
Literature review
Key informant interviews
Survey
Case studies of SPOR
Networks
Case studies of SUPPORT
Units
Document review
Literature review
Key informant interviews
Survey
Focus groups
Case studies of SPOR
Networks
Case studies of SUPPORT
Units
Administrative data / EIS
analysis
Document review
Literature review
 No
 Yes
 Yes
Performance Measurement Strategy for the
Strategy for Patient-Oriented Research (SPOR)
38
Evaluation Questions
Indicators
Data sources
Data collection methods
Baseline data required
7. To what extent is
SPOR being delivered
in a cost-efficient
manner?
7.1. Ratio of SPOR operating
expenditures to grant funds
awarded (planned vs. actual)
 Administrative data / EIS
 SPOR Network documents
 SUPPORT Unit documents
 Administrative data / EIS
 Yes
7.2. Ratio of SPOR grant funds
awarded to partner funds
leveraged (planned vs.
actual)
7.3. Assessment of resource
utilization by SPOR and its
components
 Administrative data / EIS
 SPOR Network documents
 SUPPORT Unit documents
 Administrative data / EIS
 Administrative data / EIS
 SPOR Network documents
 SUPPORT Unit documents
 SPOR Network staff
 SUPPORT Unit staff
 Stakeholders and partners
 Administrative data / EIS







7.4. Evidence of the need for
changes to improve the costefficiency of SPOR
 SPOR Network documents
 SUPPORT Unit documents
 SPOR Network staff
 SUPPORT Unit staff
 Stakeholders and partners





analysis
Survey
 Yes
analysis
Survey
analysis
Document review
Key informant interviews
Surveys
Case studies of SUPPORT
Units
Case studies of SPOR
Networks
Document review
Key informant interviews
Surveys
Case studies of SUPPORT
Units
Case studies of SPOR
Networks
 No
 No