ACADEMIC QUALITY
ASSURANCE
SUBJECT REVIEW
HANDBOOK
May 2013
1
A. INTRODUCTION
The University of Calgary Eyes High Vision, Academic Plan and Strategic Research Plan lay out
ambitious goals for our University and the strategies by which to achieve those goals. Our
foundational commitment to “sharpen our focus on research and scholarship” embodies an
increased emphasis on excellence and a focusing of effort on the specific areas. Focus produces a
virtuous cycle within those areas that drives excellence. Importantly, the focusing of the efforts
of multiple investigators across disciplinary boundaries is increasingly recognized as key to
solving large-scale problems of societal importance. For these reasons, the Quality Assurance
Process includes formal reviews that target the research subject areas that are of strategic
importance to the University as identified in the Institutional Research Plan.
Subject Reviews focus on research; educational programs are only relevant in terms of their
integration to research. Subject Reviews will concentrate on the number and distribution of
faculty members in that subject area, the total productivity and impact of the University in the
subject compared to other institutions, research funds obtained by faculty members or subject
areas to support the subject, core facilities and other research infrastructure that supports the
subject area, and the organization of academic leaderships for the subject area. All of these
elements are interconnected in ways that drive research excellence and impact within that subject
area. Subject reviews must be benchmarked using evidence-based metrics. Relevant
benchmarks are to other institutions both nationally and internationally.
The Office of the Vice-President Research and the Office of the Provost and Vice-President
(Academic) have joint administrative responsibility for the Subject Review Process. The reason
for this dual responsibility is that these reviews are part of the overall Quality Assurance Process
which resides within the Office of the Provost and Vice-President (Academic).
B.
DEFINITIONS
Office of Institutional Analysis (OIA): The OIA generates and houses all data used for
benchmarking and analysis of Units, Programs and Subject Areas.
Quality Assessment Review Report: the report written by the Review Team based on a unit’s
or subject area’s Self-Appraisal report and the Review Team’s assessment based on information
obtained during the site visit.
Review Coordinator: The Review Coordinator provides support to the Lead and to the units or
subject areas under review. He or she coordinates the resources necessary to have the reviews
completed in a timely and accurate fashion.
Review Lead: The Review Lead for Subject Reviews will be the Deputy Provost, who is
charged with the responsibility of administering Quality Assessment Reviews.
Review Team: The Review Team is normally composed of four reviewers: three external
(National and International) scholars and a scholar from the University of Calgary who is
external to the Subject Area being reviewed. The external reviewers will do a site visit to the
subject area at a mutually agreeable time.
Self-Appraisal Report: This is the primary document prepared by the Subject Lead and Subject
Committee and provided to the Review Team. See guidelines below.
Subject Committee: A group of research leaders in the subject area selected to support the
writing of the Self-Appraisal Report.
Subject Lead: The Subject Lead has the primary responsibility for the generation of the Subject
Area Self-Appraisal document and for organizing the leaders of the subject area for all activities
related to the review. The Subject Lead can be an Institute Director or head of a Unit with
primary responsibility for the subject area or simply selected from the group of investigators who
are research leaders in the subject within the University.
Subject Review: A review of an interdisciplinary subject area of strategic importance to the
University. These reviews will document how academic resources are distributed and aligned
with the University in this subject area and how the performance of the University in the subject
area compares nationally and internationally.
Unit: Stands for a Faculty, Department, Institute or Program as is appropriate.
C. SUBJECT REVIEW PROCESS
1. Beginning the Process
A schedule of Subject Reviews will be established by the Vice-President Research in
conjunction with the Provost and Vice-President Academic. A review is initiated by the VicePresident Research with communication to the Deputy Provost who administers the Quality
Assurance processes. A review normally begins with an open call to investigators to attend an
open house unless a subject area already has an administrative organization that captures its
entirety within the University.
2. Timelines and Steps
The reviews will be conducted in the following steps:
STEP
WHEN?
1. Open house led by Vice-President 9-12 months in advance
Research and call for expression
of the site visit
of interest across the University
Community. Selection of subject
area leaders and formation of
Subject Committee.
2. Meeting with subject area leaders 8-10 months in advance
to initiate formal Review Process
of the site visit
and to discuss development of the
Self-Appraisal document;
Designation of Subject Lead.
3. Generation of comparative data
package by OIA.
4. Determination of Review Team
7-9 months prior to site
and scheduling of the review
visit
5. Submission of Self-appraisal
Document to Review Lead for
internal review
6. Submission of Final Self-appraisal
Document to Review Lead
7. Site Visit
8. Submission of Review Report
9. Response to Review Report
10. Final Meeting with Subject Lead
and provisional approval of
response to review.
11. Presentation to RSC and APPC
1. Interim Progress report submitted
to Office of Provost and presented
to RSC and APPC as well as to
VPR
3 months prior to site
visit
6 weeks in advance of
site visit
1 month after the site
visit
1 month after receipt of
the Report
Within 1 month
following receipt of
Review Report
This should be done at
the first RSC/APPC
meeting immediately
after successful
conclusion of step 10.
Midpoint of review
cycle.
WHO?
Vice-President Research
or designate and
investigator community
Vice-President Research
or designate, Deputy
Provost.
Review Lead in
conjunction with
Subject Lead, VPR and
Provost
Subject Lead to Review
Lead
Subject Lead to Review
Lead, who distributes to
Review Team.
All
External Reviewers
Subject Lead to Review
Lead and VPR
VPR, Provost, Review
Lead, and Subject Lead
Subject Lead, Review
Lead
Subject Lead, Review
Lead, and VPR
2. Support for the Subject Leaders :
 Appropriately vetted institutional data relevant to the subject area will be provided prior
to the start of the review process.
 The Review Lead and the Review Coordinator meet with the Subject Lead to discuss the
components of the review - the Self-appraisal Report, site visit, etc. - to clarify any
issues or concerns regarding the review process. There is also a subsequent meeting at
which the Review Lead provides data tables to the Subject Lead and they identify any
issues that exist or can be seen in the data.
 For Faculties or Institutes, the Dean or Director can expect support and advice from the
Review Lead throughout the process.
 The Review Coordinator handles all logistics of the Review Team on behalf of the
Review Lead, and makes travel, accommodation and honorarium arrangements.
 The Review Coordinator is available to deal with any questions or concerns regarding
procedures.
3. Costs:
The University will provide funds through the Office of the Provost to cover honoraria and
expenses for the external reviewers including hotel accommodation, meals, economy class
airfare and ground transportation. Costs for alcohol and purchases of a personal nature will
not be covered.
4. What is done with the Quality Assurance Review?
At the conclusion of the Quality Assurance Review process a summary report along with the
response from the Institution will be submitted to to RSC and APPC for information. The report
will be used:
 To help the University build or maintain research excellence in the Subject Area.
 To document research quality and impact relative to other institutions or subject areas
nationally and internationally
 To provide information to Senior University Administrators and Deans to guide the
allocation (or re-allocation) of resources
 To meet public accountability expectations through a credible, transparent and actionoriented review process including publication of assessment outcomes
 To augment Comprehensive and Strategic Plans with expert assessments of existing
and planned activities in research and teaching on a regular basis.

D.
SUBJECT AREA REVIEWERS
1. Selection of Reviewer:
The Review Team will normally be composed of four reviewers.
Three External Reviewers, one national and two international scholars, who are as outstanding
scholars and experts in the field. Experience in administration, curriculum or program reviews is
an asset.
One Internal Reviewer, who is a University of Calgary scholar not associated with the subject
area under review who is knowledgeable about the subject and associated undergraduate and
graduate programs at the University of Calgary. The Subject Lead will provide the names,
affiliations, and email addresses of five potential reviewers in each category (international and
national), a brief rationale for their nominations, and a brief CV for each . Any known
significant relationship between proposed reviewers and the subject area should be disclosed.
These names should be provided by the Subject Lead to the Vice-President (Researech) who may
add to or modify the lists and then provide them to the Review Lead. The Subject Lead should
NOT have any contact with the potential reviewers.
The Review Lead, in conjunction with the VPR and the Provost, may add to the list of reviewers
suggested by Subject Lead. The VPR and Provost, in conjunction with the President, will select
the reviewers from the overall list of names in each category.
2. Contacting the Review Team:
The Review Lead will work with the Subject Lead to confirm a date for the review, and will
contact the reviewers to invite them to participate on the selected dates. For all site visits, the
Provost will meet with the Review Team at the beginning and the end of the review.
The Review Coordinator organizes the Review Team’s travel, accommodation and honorarium.
3. Communication with Reviewers:
There should be no communication between the Subject Lead or Subject Committee and
reviewers prior to the review process other than that coordinated through the Provost’s Office.
During the review process, members of the subject area should make every effort to ensure the
objectivity of the review. Members of the Review Team must not be asked to make
presentations or performances during their visit.
4. Information for the Review Team:
Members of the Review Team will receive a copy of the Self-Appraisal Report, any other
University-wide plan that relates to the subject area, a Site Visit Schedule, a copy of Eyes High,
the Academic Plan and the Research Plan. Additional materials may be provided by the Subject
Lead with approval of the Review Lead.
E.
SELF-APPRAISAL
The purpose of the Subject Review is to assess the current strength and disposition of a subject
area that is of strategic importance to the University and that is distributed across subject areas
within the University. Subject Reviews will focus on the number and distribution of faculty
members in that subject area, the total productivity and impact of the University in the subject
compared to other institutions, research funds obtained by faculty members or subject areas to
support the subject, core facilities and other research infrastructure that supports the subject area,
and the organization of academic leaderships for the subject area.
F.
SUBJECT REVIEW SITE VISIT
The Subject Lead and the Subject Committee members as well as any key administrators with
responsibility for the subject (e.g., Directors of relevant Institutes) must be available during
the site visit. Site visits will normally be comprised of two full days preceded by an evening
meeting the night before the site visit. The last half of the second day will be unscheduled, so
that the Review Team can develop preliminary recommendations that can be shared with a
small group (see below) prior to their departure.
1. The Site Visit Schedule
Because the reviewers are external to the University of Calgary, it is important to schedule as
much time together as possible while they are here. The team will be provided with an
opportunity to meet by itself the evening before the first day. The opening meeting of the site
visit will be with the Provost and Review Lead.
The closing meeting with the Review Team should include the Provost, Review Lead, VPR or
designate, the Subject Lead, the Subject Committe and the Dean of FGS. The meeting will
provide everyone with an opportunity to hear and comment on the initial recommendations of the
Review Team.
The Review Team should tour all facilities in which research or education related to the Subject
Area occurs.
The Review Team should meet with faculty members and students working in the Subject Area
as well as with relevant admininistrative leads and support staff. To facilitate discussion,
meetings with faculty members can be grouped by junior faculty, intermediate faculty, and
senior faculty, or by subdisciplines within the Subject. Meetings with students can likewise be
grouped by undergraduate, graduate or by subdisciplines.
The Appendix to this Handbook provides a sample schedule for a site visit. A site visit will
normally be two full, working days but, the duration may vary as warranted by the review. The
site visit schedule will be approved by the Review Lead two weeks in advance of the Site Visit.
The reviewers should operate independently during the visit, other than at scheduled meetings.
Lunches should be working lunches, for example, meetings with students, or discussion time for
the Review Team. The Review Coordinator will ensure that food and snacks are available.
Evening meetings with subject area faculty and students should be avoided to allow the
reviewers to meet privately over dinner which will be arranged by the Review Coordinator.
G. QUALITY ASSURANCE REVIEW REPORT
Reviewers will be required to submit a final Review Report to the Provost. The Provost will
share the document with the VPR who will discuss the document with the Subject Lead and
decide how best to share information in the document with the scholars and students who
comprise the subject area.
H.
POST-REVIEW REPORT STEPS
1. Subject Area Response to the Quality Assurance Review Report
The Subject Lead, with the help of the Subject Committee, is required to provide the Provost
with a written response to the Review Report within one month of receipt. The response should
be clear and concise. The response should indicate with which recommendations the Subject
Committee agrees and provide a rationale for those with which they do not agree. In both cases,
the Subject Committee must suggest strategies, including timelines and resource implications,
for addressing each concern identified in the report.
The Review Coordinator will arrange a meeting with VPR, Provost, Subject Lead and Review
Lead to discuss the Review Report and the Subject Committee’s Response. If concerns are
identified at this stage, they must be addressed to the extent possible before the public document
is submitted to RSC and APPC.
The Subject Lead will be required to attend the meetings of RSC and APPC to discuss the report
and responses to key recommendations, as contained within the public document. After the
successful conclusion of this step, the public document will be placed on the University’s
website and reported to GFC. The Subject Review documentation will remain confidential in the
Provost’s Office, unless the VPR releases it to the community. Any agreed-upon
recommendations from the Review Report must be referred to in subsequent plans produced by
the University for this subject area including future iterations of the Research Plan.
The Review Lead will create a public document based on the the Review Report and response
from the Subject Committee and create a public document will be approved by the VPR and
Provost and shared with RSC and APPC. The report will then be submitted to GFC for approval,
after which the final Review Report with responses will be given to the VPR and Provost for
implementation. The VPR/Provost may delegate this to the Subject Lead, relevant Institute
Directors, or an Associate Vice President Research. The delegate will have the responsibility to
follow up on the recommendations of the report and to produce the Interim Progress Report.
2. Interim Progress Report
The Interim Progress Rreport must be submitted to the Provost at the mid-pont of the review
cycle. It will address the recommendations in the Review Report and the strategies identified in
the Subject Committee Response and must indicate progress made on these reccomendations
using the same metrics and benchmarks as in the original review. Reasons for failure to improve
the metrics and additional strategies to address these limitations should be identified along with
timelines and resource plans. The Interim Progress Report will be reviewed by the VPR and
Review Lead and then submitted to RSC and APPC.
3. Feedback about the Review process:
The subject area may provide any comments on the process to the Office of the Provost.
Each member of the Review Team will be asked to complete an evaluation questionnaire about
the review process.
APPENDIX I: INSTRUCTIONS FOR THE SUBJECT REVIEW SELFAPPRAISAL
1. Structure of the Self-Appraisal Document
The document should be concise and clear with a maximum of 25 pages, not including
appendices. The tables to be provided in the appendices are specified and will be populated
centrally to the extent possible. The Subject Committe is free to review and vet the data
provided. The document is free form, but must address the key elements described below.
II. Key Elements:
A. History and Overview: The Self-Appraisal document should begin with a one page
comprehensive overview of history of the subject within the University of Calgary. This
overview should be impactful and written as a stand-alone document. The salient points
covered in the document as a whole should be addressed in this one page summary. The
overview should highlight what is distinctive about the University’s contribution to the
subject at a national or international level. It should also cover salient characteristics
such as complement, organization, partnerships, programs and key research
infrastructure. The one-page summary is a very important component of the SelfAppraisal document because it sets the stage for external reviewers, who should be able
to make ready comparisons to groups of similar size and scope.
B. Structure and Governance: This section addresses the administrative organization of the
subject area. The key questions to be addressed:
a. What is the organizational structure that supports research and education in the
subject area (illustrated with an organizational chart). Is there a rationale for this
structure and how effectively does this structure support the subject?
b. How are decisions actually made and communicated? Is there transparency in
process?
C. Complement and Personnel Resources: This section addresses the staffing complement
and student enrollment that supports work in the subject area. The key questions to be
addressed in this section are:
a. What is the number, distribution and disciplinary mix of faculty members
working in the subject area.
b. Is the demographic mix of faculty and support staff appropriate?
c. Is the content expertise mix of the faculty appropriate for the aspirations of the
University for this subject area?
d. Is the mix of full-time appointments (ie, tenure and tenure track professoriate and
instructors) relative to sessional and term appointments appropriate?.
e. Is the number of graduate students and the ratio of graduate students to faculty
members appropriate.
f. How many students are enrolled in the educational programs that relate to the
subject area?
g. Is the support staff complement appropriate given research and program
infrastructure needs of the subject area? A standard table is provided in the
appendix to identify the staff to provide this complement information.
D. Educational Programs: In this section, describe the graduate and undergraduate
programs in the subject area at the University of Calgary. This topic is more fully
addressed through reviews of Faculties. Here, focus on the relationships between these
programs and the subject area under review.
a. What educational programs in the University relate to the subject area? How are
they distributed across Faculties and what is their organizational and conceptual
relationship to the subject area.
b. How do these programs compare to others across Canada? Here centrally
provided data can be used to benchmark.
c. How are graduate students funded in the subject area?
d. What activities are fostered in the subject area to build community among
scholars in this here. Here, visiting speaker or lecture series, brown bag lunches,
seminar series, research symposia, social events and journal clubs would
examples of such activities. For each type of activity discussed, provide data on
frequency and estimates of average attendance if available.
E. Research Funding: This section should provide an overview of the research funding
within the Subject area. This section is supported by centrally provided data in the
appendices. The specific issues to be addressed are:
a. How do you assess research quality of programs within the subject area and of
individual faculty members in the FPC process?
b. How does your subject area compare, using objective benchmarks, to similar
subject areas across Canada. International benchmarks may apply as well if valid
comparative data are available.
F. Research Strengths, Productivity and Impact: This section should provide an overview of
the research strengths, strategic priorities for research, and metrics for benchmarking
performance. This section is supported by centrally provided data in the appendices. The
specific issues to be addressed are:
a. What are the research strengths within the general subject area (measured using
objective criteria) and research priorities? If these do not align, what steps are
being taken to align strengths to priorities?
b. How do the research priorities of other subject areas that intersect with the subject
area under review align with the Strategic Research Plan of the University and
with the Comprehensive Institutional Plan?
c. What is the publication productivity and impact of scholars working in this
subject on a consolidated and per/faculty member basis. How does this compare
to other institutions nationally and internationally using data provided by the OIA.
d. What major national and international research prizes and honours have been
awarded to faculty members working in this subject. Here, provide major awards
only. The OIA will provide a list of such awards.
e. What translational research impacts have been produced through work in this
subject area at the University of Calgary. Here, provide summary data on
commercialization as well as implemented policy and practice innovation. Also
provide brief descriptions of key translational impacts.
f. What other evidence of research impact can be used to assess institutional
performance in this subject area? Here provide any subject-specific evidence of
impact not covered under item e.
G. Research Infrastructure: This section provides a high-level summary of the physical
infrastructure that supports the academic enterprise of the subject area. A centrally
provided table in the Appendix will list the space assigned to the subject area. The key
questions to be addressed in this section are:
a. What are the major facilities and infrastructure elements support work in this
subject? Within which areas are these infrastructure elements contained?
b. How is access and efficient use of these facilities administered? Does this
effectively serve the needs of the scholarly community in this subject?
H. Research Partnerships: This section should identify and describe the key partnerships
external and internal to the University that are contributed to by scholars within the
subject area. In particular, focus on large-scale consortia and research partnerships that
are likely have major scholarly and societal impact.
III. Do’s and Don’ts
Do provide evidence to support all claims in your document. Unsubstantiated claims will be
viewed negatively. Be concise and clear. Use the document to articulate the current state of
affairs within the subject area.
Do not use the document to justify requests for additional resources. Those issues must be
separated from the Quality Assurance Process.
IV. Appendices
The appendices consist of tables of data provided by the Office of Institutional Analysis. In some
cases, key data points may be missing in which the subject area should endeavour to provide
them. The subject area should also review the data for accuracy and inform the Office of
Institutional Analysis about any discrepancies. The data included are all those used in the
standard set of Faculty performance metrics along with additional metrics related to research
impact and productivity as well as others that may be specific to the subject area.