Cyclical Program Review
2014-2015
John Shepherd
Vice-Provost and Associate Vice-President (Academic)
Office of Quality Assurance
The Context – QA in Ontario
 Ontario Universities’ Council on Quality Assurance
(Quality Council) was established July 1, 2010
 No more OCGS appraisals; No more UPR
 The new regime covers the approval and review of
all new and existing undergraduate and graduate
programs.
The Context – QA in Ontario
 Each university in Ontario has been required to
develop its own Institutional Quality Assurance
Process (IQAP) for ratification by the Quality
Council.
 Carleton’s Institutional Quality Assurance Process
(IQAP):
│
│
│
│
│
Approved by Senate June, 2010
Ratified by Quality Council March, 2011
Revised to include DUC – Approved by Senate February, 2012
Ratified by Quality Council May, 2012
Joint Carleton-University of Ottawa IQAP approved by Senate
January, 2012
│ Ratified by Quality Council May, 2012
QA@Carleton – What’s the point?
• To constantly improve programs that are already of good
quality
• To have units feel the exercise was worthwhile and
beneficial
• To constantly strengthen Carleton’s academic planning
processes
• To help position Carleton advantageously in the changing
Provincial context
The Carleton Process:
Who Administers It?
 Carleton’s Office of Quality Assurance (Academic Programs)
(OQA):
|
|
|
|
Vice-Provost
Manager
Quality Assurance Officer
Program Review Co-ordinator
 The Carleton University Committee on Quality Assurance
(CUCQA):
|
|
|
|
|
|
Vice-Provost (Chair)
Provost (ex officio)
Associate Dean (Programs and Awards), FGPA (ex officio)
Associate University Librarian
7 senior faculty members
1 Dean
OQA and CUCQA – What’s Our
Objective?
 Quality assurance can have two purposes:
| Program improvement
| Accountability
 Our objective is program improvement
 We work with programs to support them
 We are not the QA police!!
Not just an exam to be passed and forgotten!
Cyclical Program Review
Cyclical Program Review should not be
approached as a hurdle to be overcome.
Rather, it should be seen as an opportunity to
strengthen programs that are currently of good
quality or can achieve good quality as part of an
overall process of continuous improvement.
Cyclical Program Review
 Occurs on an 8 year cycle
 Simultaneous review of graduate and undergraduate
programs
 Self-assessment of a program’s strengths and areas for
improvement
 Consultative – informed by input from faculty, staff,
students, associate dean(s), dean(s), and external
reviewers
 20-month process
Cyclical Program Review - Steps
 Self-Study (3 volumes)
 Site Visit
 Reviewers’ Report
 Final Assessment Report
 Action Plan
OQA – Here to Help
Our objective is program improvement – we are here to help
Customized template for program review
Coordination of data, reports and other materials to inform
program review
Compilation of data into the required tables. Customized
data and reports may be available upon request.
Electronic document sharing site - cuCollab
Meetings with Review Team members
Review full and partial drafts of the self-study
Require other assistance? Just ask!
Volume I: The Self-Study
Critical, self-reflective, and program-centric
Program history – response from the last review
Program structure and delivery
Degree Level Expectations and Learning Outcomes
and Learning Objectives
Students: Enrolment, retention, satisfaction
Faculty, staff resources
Program improvements
Volume I: Template
 Each review team will have an electronic workspace on
cuCollab for document sharing
| OQA will post the template, data, reports, etc. on this site.
| Review teams will also be able to post documents
 Each Review Team will be provided with a customized
template for Cyclical Program Review.
 Model tables are included in the template
| Data from OIRP and CURO will be provided. Tables will be
populated by OQA.
| Customized tables can also be generated.
How to Write a Bad Self-Study
 Ineffective self-studies are:
| Descriptive rather than reflective, analytical, self-critical, and
evaluative
| Defensive or aimed at justifying the status quo
| Focused on the academic unit rather than the program(s)
| Does not address or only superficially addresses Degree Level
Expectations and Learning Outcomes
| Raw data are attached as appendices, or only used in a descriptive
manner
| Written by a single faculty member without evidence of buy-in of
faculty and students
Adapted from: Quality Assurance Framework: Creating an Effective Self-Study for
Program Reviews
Writing an Effective Self-Study
 Effective Self-Studies are:
| Reflective, analytical, self-critical, and evaluative
| Aimed at quality improvement. Self-appraisal includes an analysis of
strengths and weaknesses, and outlines how improvements can be
made
| Focused on the programs under review
| Curriculum is fully examined, with an eye to Degree Level
Expectations, Learning Outcomes, and to change and improvement.
| Expresses Degree Level Expectations and learning objectives that
operationally drive admission requirements, curriculum content,
modes of delivery, bases of evaluation of student performance and
commitment of resources.
| Data are analyzed and contribute to the evaluation of strengths and
weaknesses of the program.
Adapted from: Quality Assurance Framework: Creating an Effective Self-Study for
Program Reviews
Participant Experiences
Dr. Stephen Godfrey
Department of Physics
Degree Level Expectations and
Learning Outcomes
Program Learning Outcomes
• Departmental vision, characteristics of ideal
graduate
Course Learning Objectives
• Course design
Degree Level Expectations
➊ Depth and breadth of knowledge
➋ Undergraduate: Knowledge of methodologies
Graduate: Research and scholarship
➌ Application of knowledge
➍ Communications skills
➎ Awareness of the limits of knowledge
➏ Autonomy and professional capacity
COU Report: Ensuring the Value of University Degrees in Ontario November 2011
Learning Objectives and Outcomes
 Learning Objectives
| Learning Objectives are the instructional goals of the program.
They outline the program inputs or learning opportunities
students will have over the course of the program
 Learning Outcomes
│ Learning Outcomes are the measurable skills, knowledge,
competencies and/or behaviours that students will have as a
result of successfully completing the program
Learning Objectives and Outcomes
 For QA, program-level learning outcomes apply to
programs not to individual courses.
 We need to look at the individual courses to get a
“picture” of how the learning outcomes are addressed
within the program.
 Model Tables enable an account of how program level
learning outcomes are achieved and Degree Level
Expectations are met.
 A summative narrative is required of how courses as a
collective achieve the intended learning outcomes.
 Support will be provided in evaluating learning
outcomes.
Volume II: Faculty CVs
 The CVs of all faculty associated with the
program need to be included.
 Flexibility in format: No more OCGS template!
 All CVs must be in same format.
| Format to be approved by OQA.
Volume III: External Reviewers
 10 external academic reviewers.
 (4 external professional reviewers.)
 An internal reviewer will be nominated by OQA,
in consultation with the unit and Dean(s)
Timeline & Milestones
September 2013
 Review team is established
October 2013
 Attend OQA workshop
 Review team begins developing student survey/focus group
questions (template provided)
 Review team confirms faculty/instructor information with OQA
 Review team should begin work on Sections A-D of self-study, with
particular focus on the development of program learning
objectives/outcomes
 OQA conducts student surveys/focus groups as applicable
November/December 2013
 Review team receives library reports, research funding tables, space
management report, and survey/focus group reports
Timeline & Milestones
January 2014
 Review team receives data tables from OIRP
February/March 2014
 Review team should be finalizing the self-study
 Faculty, students, associate dean(s), dean(s), should be consulted on
drafts
It is highly recommended that OQA review drafts of the brief
and provide feedback before it is submitted.
April 1, 2014
 Volumes I, II, and III submitted to OQA
Timeline & Milestones
 Once OQA confirms that the brief is ready, it is
submitted to the Carleton University Committee
on Quality Assurance (CUCQA):
| Assigned a discussant, who prepares a report to CUCQA
| Program Review Lead is invited to attend the CUCQA meeting at
which the program is discussed
| Supplementary questions (dean consulted)
| Review Committee selected (dean consulted)
Timeline & Milestones
Summer 2014
In consultation with the unit, OQA will arrange the site visit for fall
2014.
Fall 2014
Two day site visit: external reviewers meet with dean(s), faculty, staff,
and students
Within one month of the site visit, the external reviewers submit a
report to OQA
The Review team prepares a response to the report, in consultation
with the dean(s)
CUCQA receives the external reviewers report and the Review team ’s
response.
Timeline & Milestones
 CUCQA considers the brief, report and response:
| Discussant recommendation report
| CUCQA recommends a categorization:
•
•
•
•
•
Good Quality with international or national presence
Good Quality
Good Quality with report
Conditional approval
Not approved to continue
│ Categorization sent to unit and dean(s)
 Action plan requested
Timeline & Milestones
Winter 2015
 Action plan submitted to CUCQA
 CUCQA: Final Assessment Report and Executive Summary
│ Submitted to the Provost
 Provost considers Report and Summary
 Documentation to SAPC/Senate for approval
 academic unit;
 dean(s);
 Board of Governors;
 Quality Council;
 Carleton website
Participant Experiences
Dr. Mitchell Frank
Institute of Comparative Studies in Art and
Culture
Help!?
 Office of Quality Assurance
|
|
|
|
Office of Institutional Research and Planning
Carleton University Research Office
Office of Space Management and Capital Planning
University Library
 Undergraduate Programs: Faculty undergraduate
associate deans
 Graduate Programs: Associate Dean (Programs and
Awards), FGPA
Contact
Office of Quality Assurance
421 Tory Building
www.carleton.ca/oqa
Jessica DeVries, Manager (Interim) & Program Review
Coordinator
jessica.devries@carleton.ca x 3231
Ann Clarke-Okah (OQA Consultant)
ann.clarke-okah@carleton.ca