Program Quality Assurance at Algonquin College of Applied Arts
Program Quality Assurance at
Algonquin College of Applied Arts and
Technology
Program Quality Review
Ontario College Credentials
2014-2015 Issue 1.0
A Guide for Team Leaders
PREFACE
This Program Quality Review (PQR) Team Leader guide is written to aid Program Quality
Reviews for Ontario College Certificate, Diploma, Advanced Diploma and Graduate Certificate programs offered at Algonquin College. It serves as the master Team Leader Guide for
Program Quality Reviews.
Adaptations are provided separately for Apprenticeship programs,
Algonquin College Certificates and Bachelor’s degrees going through Quality Review.
2
Contents
PROGRAM QUALITY ASSURANCE AT ALGONQUIN COLLEGE ...........................................11
3
Student, Graduate and Employer Satisfaction Data (Tab: Program Performance) .................26
KPI Graduate & Employer Satisfaction – Capstone Questions (KPI0002) ..........................27
KPI Graduate & Employer Satisfaction – Detailed Responses ( pending ) ...........................28
4
– with help text .............................................................................40
Faculty Education, Experience, and Professional Development Survey .............................56
5
Faculty Education, Experience, and Professional Development Survey .............................92
Analysis and Review of General Education Maps and Courses for PQR ......................... 109
6
APPENDIX E – COURSE ASSESSMENT SURVEY QUESTIONS ........................................ 122
7
PROGRAM QUALITY ASSURANCE – AN OVERVIEW
Program Quality Assurance Process Audit (PQAPA)
One of the prerequisites for granting greater autonomy to the Colleges of Applied Arts and
Technology in Ontario to govern their own programs is the implementation of a process to assure program quality. The Ontario Colleges of Applied Arts and Technology Act, 2002 outlined two elements of this expectation:
quality assurance at the individual college level, and
a self-regulating process at the system level.
The Program Quality Review process forms a major part of adherence to providing quality assurance at the College level, and is therefore essential to College operations.
At the system level, that is, at the level which encompasses all Colleges of Applied Arts and
Technology in Ontario, a separate quality assurance process takes place, the Program Quality
Assurance Process Audit (PQAPA). This is run through the Ontario College Quality Assurance
Service (OCQAS 1 ). OCQAS was established to provide effective and efficient mechanisms that ensure specific program quality and consistency standards are met by the Colleges of Applied
Arts and Technology (CAAT) in Ontario.
The following six 2 criteria define institutional policies and practices that OCQAS expects a college to have developed and implemented to ensure the quality of their programs:
Criterion 1.
Program-level learning outcomes for all programs of instruction are set, are consistent with the college mission and the programs’ intended purpose, and are appropriate for the credential offered upon successful completion of the program.
Criterion 2.
Admission, credit for prior learning, promotion, graduation, and other related academic policies support program development and student achievement of program learning outcomes.
Criterion 3.
Programs conform to the Framework for Programs of Instruction 3 and the
Credentials Framework, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance.
Criterion 4.
Methods of program delivery and student evaluation are consistent with the program learning outcomes.
Criterion 5.
Human, physical, financial, and support resources to support student achievement of program learning outcomes are available and accessible.
1
http://www.ocqas.org/index-en.html
2 A sixth criterion, focusing on program learning outcomes, was introduced in Summer 2011.
3 The Minister’s Binding Policy Directive Framework for Programs of Instruction is one of the policy documents issued by the
Ministry of Training, Colleges and Universities in April 2003 to guide the development and delivery of programs of instruction offered by the colleges of applied arts and technology in Ontario..
8
Criterion 6.
Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.
Transition to Accreditation Process
Building on the PQAPA , Ontario’s Colleges are moving to an accreditation process effective
September 2015. In preparation for this transition, six Accreditation Quality Standards have been developed from the current PQAPA quality criteria, and approved in March 2015 by the
Management Board of OCQAS. Of particular note is that while the essence of the criteria has been retained, there has been a reordering and renumbering of the criteria. These changes are for ease of reference, and do not imply any prioritization or weighting of significance.
Quality Standard 1 (Program Quality Management System, maps to Criterion 6 above)
Quality assurance processes ensure program development, program review, and on-going program quality assessment involve faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement.
Quality Standard 2 ( Existence and Communication of Policies and Practices, maps to
Criterion 2 above) Quality assurance processes ensure the existence and communication of effective policies and practices related to academic issues that support program development and student achievement of program learning outcomes. This would include, for example, admission, credit for prior learning, promotion, graduation, and other related academic policies.
Quality Standard 3 ( Program Design, maps to Criterion 1 above) Quality assurance processes ensure that: program-level learning outcomes are established for all program of instruction; they are consistent with the programs’ intended purposes; and, they are appropriate for the credentials offered upon successful completion of the programs.
Quality Standard 4 ( Program Delivery and Student Assessment, maps to Criterion 4 above)
Quality assurance processes ensure that methods of program delivery, including the design, development and execution of teaching and learning activities and student evaluation strategies, are consistent with the program learning outcomes.
Quality Standard 5 ( Conformity with Government Requirements, maps to Criterion 3 above)
Quality assurance processes ensure that all programs of instruction conform to current government policy related to the design and delivery of programs of instruction, are consistent with accepted college system nomenclature / titling principles, and maintain relevance.
Quality Standard 6 ( Availability and Allocation of College-wide Resources, maps to Criterion
5 above) Quality assurance processes ensure the existence and availability of human resources, physical resources, financial resources, student support resources, and
9
technological infrastructure to support student achievement of program learning outcomes wherever and however they are delivered.
For up to date information on the transition to the Accreditation model, please visit the
OCQAS web page at http://www.ocqas.org/en/?p=5014 .
In 2014-2015 Program Quality Review evaluation criteria are being updated to reflect the incoming Accreditation Quality Standards. Programs in review in Fall 2014 may use the 2013-2014 quality criteria elements that are based on the PQAPA quality criteria as outlined in this guide. Effective Winter 2015, programs commencing PQR are to implement the revised Quality Criteria Elements. A second issue of the PQR
Team Leader guide will be published to support these reviews
10
PROGRAM QUALITY ASSURANCE AT ALGONQUIN COLLEGE
Program review is annual event at Algonquin College, and currently consists of four processes:
Annual Curriculum Review
Annual Program Mix Review
Cyclical Program Quality Review
Cyclical Review of Online Elective General Education courses
Annual Curriculum Review
Each year program faculty members revise and update curriculum to maintain its currency and to ensure that graduates will have the skills necessary to be successful in the workplace.
Feedback from faculty, students, and advisory committee members is taken into consideration before updates are made to keep the program current. Revisions to the curriculum and the program narrative information are made in GeneSIS, the student information system, and are subsequently reviewed in the Office of the Vice President Academic. Once the changes are approved, a new program version is created. The information in GeneSIS becomes the foundation for course loading, fees loading, scheduling, registration and College publications.
Program Mix Review
Staff in the Academic Operations and Planning department compile data that reflect the Quality
Measures for a program. Admissions and registration data, employment rates, capstone questions from Key Performance Indicators (KPI) and course assessment surveys, and program costing data are reviewed. Programs not meeting college benchmarks are reviewed and strategies are developed by the Chair and Dean to improve the quality of the program and/or the financial contribution.
Program Quality Review
Program Quality Review (PQR) ensures that each program goes through a comprehensive review every five years. The PQR process builds on annual review activities by including analysis of three to five years of data compiled for the program mix review, and soliciting direct feedback from stakeholders and students into the program review. Each PQR results in an implementation plan containing resourced actions to maintain and improve the program over the next five years.
This manual guides Team Leaders when conducting a Program Quality Review.
Cyclical Review of Online Elective General Education courses
Online General Education courses form an integral part of many programs offered at the
College. In 2009, the review process for these courses was formalized. Each of the online
General Elective courses in the College pool is reviewed according to a published cycle, first by the General Education committee for overall adherence of the course to General Education objectives, and then in detail by individual faculty and student reviewers.
11
PROGRAM QUALITY REVIEW
PQR Goals
To assess the program against established quality criteria in order to determine whether the program meets or exceeds College and Ontario College Quality Assurance Service
(OCQAS) standards (for Ontario College Credentials) or Postsecondary Education Quality
Assessment Board (PEQAB) standards (for Bachelors’ Degrees).
To establish and implement a quality improvement plan 4 for the program, in response to the findings of the review.
PQR Process
Program Quality Review follows a systematic process, and is outlined in Figure 1. After the
Team Leader has been identified to conduct the review, information pertinent to the program is collected and analyzed, allowing an initial evaluation of the program against established quality criteria. The program information reviewed includes published and internal documentation, program data, current curriculum maps, and input collected from key players in the program – faculty, students, and external stakeholders.
Once the review is complete, the evaluation against the criteria is finalized, and all findings and recommendations for improvement are captured in a report. An implementation plan for the recommendations is drawn up, with an update on the actions provided to the PQAA at least annually until completion.
The PQR Team Leader conducts the PQR, with assistance from the Chair.
Note that this guide is written to provide assistance with each of the main components of PQR:
Evaluation Against the Quality Criteria
Completing the Final PQR Report
Schedule
Programs are reviewed according to a defined schedule, normally every five years. The schedule is revised annually, in consultation with the Academic Departments, and aligned with other Quality Assurance requirements, for example, accreditations or consent renewals for bachelor ’s degrees.
The PQR schedule is published at http://www.algonquincollege.com/academicdevelopment/our-services/program-quality-assurance/
4 Also referred to as “the implementation plan”
12
Cyclical Program Quality Review
(Figure 1)
Confirm Programs
(20% per Year)
Assign Team Leader
Team Leader and Chair
Review PQR data reports and Conduct SWOT
Analysis
Provide Team Leader With: o 5 Years of Central Data
(Quality Index, Program
Costing, KPI Results) o Orientation to Process,
Resources and
Templates
Faculty Review of Curriculum Data
Student Focus Group
External Stakeholder and/or
Advisory Committee Focus Groups
Review of Findings
Formulation of Recommendations
Development of Implementation
Plan
Team Leader Completes Final
Report
Implementation of
Recommendations
Annual Follow-up of Implementation
Plan
13
Roles
Program Quality Assurance Administrator
Program Quality Reviews are coordinated by the Program Quality Assurance Administrator
(PQAA), who reports to the Dean, Academic Development, who in turn reports to the Vice
President Academic.
The responsibilities of the Program Quality Assurance Administrator include
ensuring the College is compliant with Ministry requirements regarding Program Quality
Assurance.
consulting with major academic constituencies College-wide regarding Program Quality
Review.
preparing documents related to Program Quality Review.
planning and organizing orientation/training workshops for Team Leaders.
supporting the Program Quality Review process in programs.
meeting regularly with PQR Team Leaders during the review process to monitor progress and provide guidance.
facilitating evaluation of the Program Quality Review process.
ensuring that the final report is signed off fully by the PQAA, Chair and Dean
arranging archiving of the final report; forward e-copies to the TL, Chair, Dean, and providing access to all reports to the VPA.
Program Quality Review Team Leader
The PQR Team Leader is normally a faculty member who is assigned to conduct Program
Quality Review for the program in which s/he teaches.
The responsibilities of the Team Leader include
attending meetings scheduled by the Program Quality Assurance Administrator.
attending orientation/training workshops related to the Team Leader role.
reporting to the Chair weekly to monthly on the progress of the Program Quality Review.
analyzing data and incorporating the results of the analysis in the final PQR report.
convening meetings of the program faculty team.
convening a Student Focus Group.
convening an External Stakeholders / Advisory Committee Focus Group.
facilitating Student and External Stakeholder Focus Groups for a program from another
School and submitting reports for these meetings.
evaluating the Program Quality Review process.
submitting the final report to the Chair and PQAA, who will review and sign-off. (The
Chair then forwards the report to the Dean for final review/sign-off, and notifies the
PQAA who will advise the Vice President Academic of its completion.)
14
Dean
The responsibilities of the Dean include
identifying programs for Program Quality Review.
supporting the Program Quality Review process.
reviewing and signing-off on the Program Quality Review report.
following up regarding results and recommendations included in the report.
reporting to the Vice President Academic on the progress of Program Quality Review and regarding the follow-up on the recommendations.
Chair
The responsibilities of the Chair include
identifying Team Leaders (TL) each spring for programs identified to be in the next cycle.
It is preferred that a full-time professor or coordinator is assigned.
acting as a facilitator for Focus Groups in the event of illness of the TL.
meeting with each TL in the Spring to discuss the upcoming process, and to encourage and facilitate attendance at the Aligning and Building Curriculum Conference if the TL has not already attended.
attending information meetings for Chairs.
providing release time for the TL as outlined in the PQR process
meeting with the TL early in the process to review the program data and initial assessment of the quality criteria. Assisting with agenda setting for focus groups and helping to create a list of those to invite.
providing the relevant budget code for all expenses.
sending an email to all program faculty inviting them to participate in the PQR process and to fill out the credentials and experience sheet.
providing a summary report of faculty credentials and experience to the Team Leader, to be included in the final report.
arranging secretarial support as outlined in the PQR process
meeting regularly with the TL and monitoring progress.
attending focus group meetings and introducing the group members to the process.
ensuring feedback is provided to all those who participated in the focus groups.
providing feedback regarding the PQR process.
collaborating in the development of the final recommendations and implementation plan, ensuring adequate budgetary support for those recommendations that are to be implemented immediately (i.e. within one year) and consideration of those recommendations for which funds are not currently available.
including PQR recommendations in their Performance Contract.
facilitating the implementation of the final recommendations.
tracking progress on the implementation plan, and preparing the submission of the PQR
Annual Follow Up Report until all recommendations have been implemented.
15
Orientation
Orientation for Team Leaders is scheduled where possible outside of classroom contact weeks to minimize the negative impact of training within teaching weeks.
The orientation process for Team Leaders involves a series of workshops. Upon completing the
Team Leader orientation process, Team Leaders are able to
evaluate a program against identified criteria; identify quality in a program when measured against identified standards and determine how to reinforce/maintain this quality.
lead focus groups for internal and external stakeholders.
map curriculum to validate that the program of study meets current Ministry requirements.
analyze program data and make inferences based on trends.
analyze curriculum maps to identify gaps and redundancies.
formulate realistic and attainable recommendations to improve quality where needed.
Release Time
Team Leaders are entitled to receive the following release time for the duties and responsibilities associated with Program Quality Review:
Fall 2014 or Winter 2015:
Release from teaching hours for the equivalent of one three-hour course including evaluation and preparation time which, for consistency purposes, has been equated to six hours of assigned workload. Faculty members who are reviewing a three-year program, a four-year program, or the equivalent of three or more years of curriculum, will be assigned a nine hour workload. All authorized travel expenses incurred by Team
Leaders to facilitate Focus Groups at another campus will be reimbursed.
Where an internal Team Leader is not available to conduct the PQR, an external Team Leader may be assigned to the role. Before the PQR commences, the Chair, in consultation with the
PQAA, identifies a Team Leader and negotiates a contract with remuneration within the PQR budget allocation for the program under review. Note that the funds allotted may be less for smaller programs, for example for short College Approved programs. The PQAA provides details on the funds available for review of each program.
In addition to the Team Leader s’ release time, additional financial support is provided to departments as follows:
Support Staff: up to 48 hours per PQR at $15 per hour
Part-time Faculty : up to 16 hours per PQR at $30 per hour
Hospitality 5 : up to $100 per PQR to contribute towards light refreshments at focus groups.
5 Expenditure must follow guidelines as per College Policy AD12 “
Travel, Meals and Hospitality
”
16
PQR expenses are charged to the program budget code with the exception of travel:
Departments are compensated annually by Academic Development for the specified release time.
Travel expense forms are submitted directly to the PQAA.
Timeline
Fall Course Release
Timing
May 2014
Event(s)
Confirmation of Programs to undergo PQR, Team
Leaders assigned:
Team Leader orientation June or August 2014
September 2014, or as required Refresher orientation (for those who have been TLs before)
September to November, 2014 Touch base meetings with PQAA, as required
November 27, 2014 Draft report due
December 18, 2014
(submit to Chair and PQAA)
Submit final report
Winter Course Release
Timing
May 2014
June or August 2014
January 2015, or as required
January to March, 2015
March 26, 2015
April 16, 2015
Event(s)
Confirmation of Programs to undergo PQR, Team
Leaders assigned:
Team Leader orientation
Refresher orientation (for those who have been TLs before)
Touch base meetings with PQAA, as required
Draft report due
(submit to Chair and PQAA)
Submit final report
Detailed individual workplans are maintained by the PQR Team Leader with assistance from the
Program Quality Assurance Administrator (PQAA) as required.
Need Assistance?
Contact the PQAA at any time with questions about Program Quality Review:
Wilma McCormack, Room C-532, 613-727-4723 ext. 7681 mccormw@algonquincollege.com
17
THE FINAL REPORT – AN OVERVIEW
For more details on the contents of each section in the main document, see Appendix A .
The final output of the PQR is a report which is submitted to the Office of the Vice President
Academic. This report follows a standard structure:
The Main Document
Executive Summary
Written last, the Executive Summary prefaces the entire report. It summarizes the highlights of the quality review of the program, and includes the overall status of the program, its strengths, weaknesses, as well as challenges and recommendations.
Introduction
Written first, the Introduction paints a picture of the program at the outset of the review – it does not contain any information gained as part of the review process.
Evaluation of the Program against the Criteria
This section is the working part of the document.
Information is gathered as each stage of PQR - document review, data analysis, and the outcomes of the focus groups – is carried out, and notes are captured against each of the elements of the quality criteria as the review continues. By the end of the review process, each element of the quality criteria will have been responded to (yes/no) and a justification of the response will have been provided.
Conclusions
The conclusions wrap up the findings of the review. The outcomes of the document and data review are provided, along with the highlights from each of the focus groups. The recommendations are stated, and any observations or non-implementable recommendations are noted.
Recommendations and Implementation Plan
The recommendations that are to be implemented are captured in a table, and are linked, as far as possible, to the pillars of the College Strategic Plan. Each recommendation will have an associated list of action items that must be completed in order to meet the goal of the recommendation. Each action item must be resourced, and have a realistic completion date.
The implementation plan is a living document – it is carried forward year over year, and updated at least annually until action items are completed or closed.
The Appendices
The appendices carry the information collected during the review process. This information will be referenced within the main document.
Appendix A
This contains program and curriculum information. It consists of the
Program Monograph and Review of Promotional Materials
Program Vocational Learning Outcomes 6
Program Delivery Information
Curriculum Mapping Information:
Vocational Learning Outcomes curriculum map (Overall and by Level)
Analysis of the Vocational Learning Outcomes curriculum map
Essential Employability Skills curriculum map (Overall and by Level)
Analysis of the Essential Employability Skills curriculum map
General Education curriculum map
Analysis of the General Education offering
Course Outline Review Summary
Appendix B
This contains the information collected from the program faculty, namely
Faculty focus group report(s)
Summary of the Faculty Program Self-Audit
Summary of Faculty Credentials and Experience
Appendix C
This contains the information collected from the External Stakeholders, normally the
External Stakeholders focus group report
Appendix D
This contains the information collected from the current students, normally the
Students’ focus group report
Appendix E
This contains summary information only from the review of the statistical and survey data for the program. Typically, observations from analysis of registration, graduation,
OCAS program choice, program withdrawal information and KPI surveys are included.
There is no need to enclose all program data available through the Enterprise Reporting
Portal. However, if department/program specific surveys have been utilized in the data analysis, these should be included in Appendix E. Please contact the PQAA for guidance if required.
6 In the case of Bachelor’s Degrees, Program Outcomes. The curriculum mapping section is adjusted accordingly.
19
DOCUMENT REVIEW
The following documents must be reviewed by the Team Leader, in conjunction with the Chair and department faculty, to ensure that the information contained therein is complete, accurate, unambiguous, current, that it conforms to Ministry and College Standards, and that pertinent information is readily available for applicants and students to allow them to make informed decisions about program choice.
Program Monograph
Program Promotional Materials
Program Council Meeting Minutes
Program Advisory Committee Meeting Minutes
Further Study Pathways
Reports from External Accreditation or Licensing bodies
Program Learning Outcomes
Program of Study
Course Outlines and Course Section Information
These documents are referred to when evaluating the program against the quality criteria.
Program Monograph
The program monograph contains several key pieces of information used by applicants when determining which program to choose. It is reviewed annually as part of Annual Curriculum
Review. Although the whole monograph should be reviewed during PQR, an in-depth review of the following sections may be necessary in order to respond to some elements of the quality criteria.
Admission Requirements should ensure that students are adequately prepared to enter the program (i.e. that they have the foundational knowledge needed on which to base their learning) without unnecessarily limiting access to a program.
College Eligibility is Ministry mandated and is usually standard for postsecondary programs - Ontario Secondary School Diploma, or at least 19 years of age.
Program Eligibility is determined by the department and should reflect any necessary knowledge and skills that the students must have prior to entering the program, for example a specific level of mathematics or science. Mature students may be required to meet the program eligibility even though they are 19 or older and do not have a high school diploma. Proven competency in English is a College-wide requirement.
Note that the current monograph publishes admission requirements for the subsequent academic year.
20
Fees and Expenses should list all relevant program expenses to allow students to plan their financial commitment. Fees and Application Information are supplied by the
Registrar’s Office.
The Program of Study is published in the monograph and is the foundation of the
College’s contract with an applicant. It outlines the learning opportunities that student can expect in return for paying tuition.
The full program monograph (not just a weblink) is included in the final PQR report within
Appendix A.
Program Promotional Material
Other promotional materials used by the program (in addition to the Program Monograph) are also to be reviewed. The list of promotional materials should be provided in the final PQR report within Appendix A, with URLs if appropriate. Full documents are not required to be included.
Program Council Meeting Minutes
Minutes of Program Council meetings for the past 5 years are reviewed to ensure that
Program Council meetings occur on a regular basis, and in accordance with Policy AA03
– Program Council . (Program Council meetings must happen once a term at a minimum.)
issues raised at Program Council meetings are captured, and have an action plan to address.
there is a feedback mechanism in place to ensure that the students are informed of what action, if any, will take place and the proposed timeline for any resolution.
The review of the minutes may also help identify agenda items for the upcoming focus groups.
The Program Council meeting minutes are not included in the final PQR report.
Program Advisory Committee Meeting Minutes
Minutes of Advisory Committee meetings for the past 5 years are reviewed to ensure that
Advisory Committee meetings occur on a regular basis, and in accordance with Policy
AA01 – Advisory Committees . (Advisory Committees must meet at least once a term)
issues raised at Advisory Committee meetings are captured, and have an action plan to address these issues.
there is a feedback mechanism in place to ensure that students are informed of what action, if any, will take place and the proposed timeline for any resolution.
The review of the minutes may also help identify agenda items for the upcoming focus groups.
The Advisory Committee meeting minutes are not included in the final PQR report.
21
Further Study Pathways
Established agreements for further study currently in place for the program, for example, articulation agreements, are reviewed to ensure that curriculum changes made since the last issue have been reflected within the agreement.
The final PQR report must include a list and brief description of any such agreements that exist with other postsecondary institutions. If no such agreements currently exist and graduates from the program regularly seek further education, it may be prudent to recommend in the PQR final report that an articulation agreement be considered.
Also review any pre-College laddering opportunities, for example School to College to Work
Initiative (SCWI) projects occurring in partnership with the program, (e.g. program specific
Specialist High Skills Majors (SHSM) or Dual Credit initiatives).
Information on pathways is found at: http://www.algonquincollege.com/prospective/programs_degrees.htm
.
Reports from External Accreditation or Licensing Bodies
If applicable to the program, information on the success or otherwise of graduates obtaining licenses or external accreditation should be collected and noted against quality criteria element
#6.8. This information may also be summarized in the Executive Summary and Conclusions sections within the PQR final report.
Program (Vocational) Learning Outcomes
The current ministry approved program Vocational Learning Outcomes (VLOs) are stored in
GeneSIS, with a copy available in the Course Outline Mapping and Management System
(COMMS). VLOs are reviewed during Curriculum Mapping .
A listing of the program Vocational Learning Outcomes is included in the final PQR report within
Appendix A. Note here if there are Provincial Program Standards for the program, and include any pertinent differences between the Program Standard and the VLOs used at Algonquin (e.g., inclusion of College-specific VLOs).
Program of Study
The Program of Study is stored in GeneSIS, with copies available on the Program Monograph
and in COMMS. The Program of Study is reviewed during Curriculum Mapping .
The Program of Study forms part of the Program Monograph, which is included in the final PQR report within Appendix A.
Course Outlines and Course Section Information
Course outlines and Course Section Information are reviewed by the course owners each year as part of Annual Curriculum Review, with the output of this review utilized within PQR. Course
outline review will be discussed further in the Curriculum Mapping section of this document. To
note:
22
a course outline must be developed for each and every course offered in the program.
all sections of the course outline must be reviewed.
Policy AA26 – Course Outlines and Course Section Information provides additional information regarding the expected content of course outlines and course section information.
Course outlines and course section information for full-time Ontario College Credentials and
Bachelor’s degrees are maintained in COMMS. Course outlines and course section information for programs delivered through CCOL are being integrated into the COMMS system.
23
DATA REVIEW
Program specific data must be reviewed by the Team Leader, with support of the Chair, to gain an understanding of the program trends before focus group meetings are held. This aids with evaluation against the criteria and setting the agenda for focus group meetings.
The following data are provided, when available, for Program Quality Reviews, and are currently accessed through the Data Reporting Portal at https://reporting.algonquincollege.com
. Note that data repository is being updated and improved during 2014-2015 and information below pertains to layout and data availability as of August 2014. The PQAA will inform Team Leaders and Chairs to changes in access and content.
Registration Data (Tab: Program Performance)
Up to five years of registration related data are provided, where available, in order to review trends in admissions and enrolment. These data should be reviewed in the context of specific constraints of the individual program, for example, the maximum registration allowed, or any physical (space) constraints.
Registration Milestones by Program (PQR1000) : Actual registrations are provided for
Day 1, Day 10 and the audit date. Items of concern for a program include a high dropout rate in level 01, significant withdrawals either from Day 1 to Day 10 or between Day
10 and the audit date, and an academic level that appears to have a low returning rate.
Application / Registration Ratio (pending, to be deployed in PQR1000): This ratio is the number of applications vs. number of registered students (both full-time and parttime) on Day 1 in the intake level.
It is important to look at the trends in this ratio: although each program is unique, established programs tend to have a fairly constant ratio over the years. If the ratio is trending upwards, one thing to consider is whether additional spaces should be opened in the program. Conversely, if the ratio is decreasing year by year, it may be time to consolidate. If no trend is evident – the ratio is erratic – consider why the applications and registrations vary, and whether modifications should be made to the program to introduce stability.
Intake / Projected Enrolment (PQR1000) : The actual enrolment (full-time only) over the projected enrolment (full-time only) on the audit date in the intake level expressed as a percentage.
Projected enrolment is used for planning financial, human and space resources. Ideally, the intake / projected enrolment should run close to 100% - significant deviation either way results in recalculation of the budget and related items during the academic year.
Program Withdrawals (PQR1020) : This report may help identify why students are not continuing in the program. It is important to review the reasons given to see if any
24
patterns arise. Note that students do not always fill out the standard withdrawal form and for those that do, the reason given may not be the real reason for withdrawing.
OCAS Program Choice (PQR1040) : The OCAS program choice report shows the distribution of registered students by their program choice. It may assist with identifying why students have withdrawn from level 01. For example, a program with a significant number of 4th and 5th choice applicants may have higher withdrawal rates.
Occasionally, values of 0 or 6 appear in the program choice report. Where a 0 is present, this means that the student did not apply via OCAS. Where a 6 is present, this means the student is enrolled in a program that was not on their OCAS submission.
Student Success Data (Tab: Program Performance)
Where available, five years of Student Success Data are provided in order to see if trends or patterns are emerging in the program.
Course Failure Rate (PQR1030) : This report is produced when 20% or more of the students with a registered status (on the audit date) in the course receive an F grade.
This report is only produced when five or more students are registered in the course for the term.
The Failure Rate report helps identify issues and raise questions for consideration regarding student success, for example:
are admission requirements adequate to allow the students to be successful
(especially for level 01 courses)?
are the courses sequenced properly to ensure the students are taught foundational information on which to build their learning?
are students allowed to progress in the program after failing a course when the failed course should be considered a prerequisite for the next level?
are the learning materials, the learning activities, and the evaluation methodologies appropriate to allow students with a variety of learning styles to learn the material and demonstrate the course learning requirements?
is the content or the assessment more difficult than it need be?
Many teachers have noted a link between attendance and student success on exams. If this is the experience in the program under review, consider
how to communicate this link to students
how students with involuntary absences are assisted to be successful while respecting that, with a few exceptions, attendance is not mandatory and that adult learners are accountable for their own learning. Formally monitoring attendance in class on an ongoing basis has been shown to have a positive influence on attendance and success.
25
In some cases, the report will indicate high failure rates as a result of a course being offered off-cycle for a few students who had previously dropped or failed a course, and did not successfully complete the course in the off-cycle term.
Course Assessment Roll Up (not currently published) : This report is a roll up of all course section course assessments in a program on a term-by-term basis. (The roll up preserves the anonymity of the individual course section professors.) It provides student feedback at the program level and will identify program strengths and areas of concern.
Course Assessment surveys have been updated over the years. The questions for each survey (pre-2009F, 2009-2011, 2011-2012, 2012-2013 and 2013-2014) are provided in
Appendix E . The Course Assessment capstone question
is ‘ Overall, please rate the quality of this course ’ and was used in computing the Program Mix Measure Instructional
Quality .
Student, Graduate and Employer Satisfaction Data (Tab: Program
Performance)
Provincial surveys are conducted to provide Key Performance Indicators (KPIs) for
College programs, namely:
Student Satisfaction Survey
Graduate Outcomes/Graduate Satisfaction Survey
Employer Satisfaction
Surveys are posted to the program data repository. Additionally, Student, Graduate and
Employer Satisfaction measures are directly calculated from these KPI survey data, and are provided in the Program Mix Report.
For information on how KPI surveys are conducted, see Appendix D .
KPI Student Satisfaction – Capstone Questions (KPI0001)
Percentage of students who rated Very Satisfied or Satisfied for the overall program/college experience. This measure is the average of responses from the four capstone questions of the Student Satisfaction survey:
Overall, your program is giving you the knowledge and skills that will be useful in your future career
The overall quality of the learning experiences in this program
The overall quality of the facilities/resources in the college
The overall quality of the services in the college.
Reviewing the trend of the individual questions provides insight into the (1) program content, (2) its delivery, (3) quality of services in the College and (4) quality of facilities/resources in the College. It is important to review the trends for each question
26
as well as the aggregate value to determine where the strengths and challenges of the program lie.
The four KPI Student Satisfaction Survey capstone questions have remained constant for the past 6+ years; the numbering changed in 2014:
KPI Question
Overall, your program is giving you the knowledge and skills that will be useful in your future career
The overall quality of the learning experiences in this program
The overall quality of the services in the college
The overall quality of the facilities/resources in the college
Numbering 2009-2013 Numbering 2014
Q14S Q13S
Q26S
Q45S
Q44S
Q24S
Q39S
Q49S
KPI Student Satisfaction
– All Questions (KPI0004)
Student satisfaction rate for all KPI Student Satisfaction survey questions.
KPI Graduate & Employer Satisfaction – Capstone Questions (KPI0002)
This report tabulates the graduate satisfaction and employer satisfaction rates, and also presents key items such as employment rate. Review the past five years data to garner trends. Sample sizes are indicated.
Graduate Satisfaction Percentage : Percentage of students who rated Very
Satisfied or Satisfied for the usefulness of the college education in achieving goals after graduation.
Total Employment : Percentage of students who are employed six months after graduation from their college education.
Related Employment : Percentage of students who responded Yes or Yes, partially to related employment after graduation. This question is only asked of students who are currently employed.
Employer Satisfaction: Percentage of employers who answered Very Satisfied or Satisfied for the graduate’s overall college preparation for current employment.
When considering the information from Employer and Graduate surveys, do take into account that the trends may not be as valid as those from the Student
Survey - there are lower numbers of respondents and students who are working successfully are more willing to have employers contacted than students who may not be performing well.
For detailed information on graduates, and comparable statistics for programs at other colleges, see the MTCU Employment Profile report at http://www.tcu.gov.on.ca/eng/labourmarket/employmentprofiles/ .
27
KPI Graduate & Employer Satisfaction – Detailed Responses (pending)
Similar to the detailed responses for the Student Satisfaction surveys, detailed graduate and employer satisfaction responses are planned to be deployed to the reporting portal in Fall 2014.
Report on hold (information retained for reference purposes only) :
Instructional Quality Percentage of students who rated the overall quality of the course as Excellent or Very Good . This program measure is aggregated from the Course
Assessment survey capstone question:
Overall, please rate the quality of this course.
Not every course within the program may be assessed each term, and thus this measure only gives an indication of the student’s view on the quality of the program as a whole.
Net Contribution Data (Tab: Program Costing)
Costing Year Detail by Program (ACA1000)
This report provides a multi-year program contributions trend chart, along with costing figures for the program for a given fiscal year. The report allows drill-through to further data, including space cost, depreciation and enrolment. Details for other fiscal years is accessed via drop-down menu.
28
CURRICULUM MAPPING
Analyzing the mapping of a program’s curriculum to its Program Standards is an essential part of PQR. Building on the course outline reviews conducted as part of annual curriculum review, it provides the opportunity to review cumulative effects of changes made to the program’s curriculum over the last five years, determine whether it meets the requirements of the Program
Standards, and plan for updates to the program. The exercise of performing this analysis is called Curriculum Mapping.
Program Standards
Since 1993 the Government of Ontario has been developing system-wide program standards for postsecondary programs in colleges across Ontario. Program standards apply to all similar programs of instruction offered by colleges across the province. Each program standard contains the following elements:
Vocational Standard (Vocational Learning Outcomes) – VLOs
Essential Employability Skills - EES
General Education requirement
Learning outcomes represent culminating demonstrations of learning and achievement. They describe performances that demonstrate significant integrated learning by graduates of the program has been achieved and verified. Learning outcomes are interrelated and cannot be viewed in isolation of one another.
Vocational Standard
The Vocational Standard captures the vocationally specific learning outcomes which apply to each diploma and certificate program. Vocational standards apply to all similar programs offered by colleges across the province.
Not all programs have a vocational standard. In this case, the College will have established internal Vocational Learning Outcomes, and these are reviewed and used for mapping purposes.
Note that, effective Fall 2013, all Ontario College Credentials offered at Algonquin
College were to formally embed the Sustainability and Internationalization VLO “ The graduate has reliably demonstrated the ability to identify and apply discipline-specific practices that contribute to the local and global community through social responsibility, economic commitment and environmental stewardship” within the program. Curriculum mapping will help assess the extent to which this has occurred.
Essential Employability Skills
Essential Employability Skills relate to the essential employability skills (previously known as Generic Skills) needed for both career and personal success (communication skills, numeracy skills, thinking and problem-solving skills, information management skills, interpersonal skills, and personal skills). They are expressed as learning outcomes. There are eleven Essential Employability Skills learning outcomes which apply to each program:
29
1. communicate clearly, concisely and correctly in the written, spoken, and visual form that fulfills the purpose and meets the needs of the audience.
2. respond to written, spoken, or visual messages in a manner that ensures effective communication.
3. execute mathematical operations accurately.
4. apply a systematic approach to solve problems.
5. use a variety of thinking skills to anticipate and solve problems.
6. locate, select, organize, and document information using appropriate technology and information systems.
7. analyze, evaluate, and apply relevant information from a variety of sources.
8. show respect for the diverse opinions, values, belief systems, and contributions of others.
9. interact with others in groups or teams in ways that contribute to effective working relationships and the achievement of goals.
10. manage the use of time and other resources to complete projects.
11. take responsibility for one’s own actions, decisions, and consequences.
See the EES web page http://www.algonquincollege.com/ees/ or contact the EES committee or PQAA for more information.
General Education Requirement
The General Education Requirement relates to the requirements for general education courses that provide all learners with choice and breadth of experience beyond the vocational areas. General education courses cover areas of general interest and are divided into five themes:
Theme 1 - Arts in Society
Theme 2 - Civic Life
Theme 3 - Social and Cultural Understanding
Theme 4 - Personal Understanding
Theme 5 - Science and Technology
See the General Education web page for more information http://www.algonquincollege.com/gened/gened.html
.
The Essential Employability Skills and General Education Requirement apply only to Ontario
College Certificate, Diploma and Advanced Diploma programs. Apprenticeship and Graduate
Certificate programs are not required to meet these standards. Degree programs also have different standards to meet with respect to breadth courses. For details on each credential, refer to the Credentials Framework
Figure 2 illustrates how Ontario College Credentials fit within the Credentials Framework, and the interdependency between the Program Standards and the entirety of the courses within that program. A short explanatory video on Provincial Program Standards is also available at http://www.youtube.com/watch?v=awWQhUeAgdc
30
Curriculum Services
In addition to the EES and General Education committee representatives, Algonquin College has a dedicated team, Curriculum Services , which supports curriculum development and review. A Curriculum Consultant is assigned to each area of the College, and may assist in
Curriculum Review and Remapping activities for PQR.
Figure 2: Aligning Curriculum - A Roadmap
31
Purposes of Curriculum Mapping
A curriculum map can be used to
provide an overview of the curriculum for the total program
document curriculum and the inclusion of standards for accrediting/credentialing groups
identify opportunities in the program for learners to demonstrate learning outcomes at the required level
serve as a planning tool to ensure that all program standards are developed within the program 7
identify areas of redundancy where content is more than adequately covered
identify paths that learners can follow to meet graduation requirements
help faculty, learners and others “situate” courses and learning experiences within the program curriculum
Performing Curriculum Mapping
At Algonquin College, course outlines for courses which form part of an Ontario College
Credential are housed within COMMS , the Course Outline Mapping and Management System.
This system facilitates the review of various curriculum maps by generating reports at the program and course level. Currently, available maps/reports include
Course Essential Employability Skills by Level and Program
Course Vocational Learning Outcomes by Level and Program
EES Program Summary
Program Overview
Vocational Learning Outcome Program Summary
When course owners enter the course outline into COMMS, they provide the information on which the maps are based. They also identify whether the course is core or elective, and vocational or general education. Additionally, whether each learning outcome is taught, assessed and if there is opportunity to demonstrate a culminating performance of the outcome is captured in the outline prior to approval.
Table 1 provides the guidelines used to categorize teach (T), assess (A) and culminating performance (CP) for each activity within the course outline.
7 Some programs were established and approved prior to Program Standards being introduced into the
College system. If this is noted as part of PQR, a plan needs to be initiated to develop the VLOs.
32
T
A each ssess
Do you provide instruction/learning opportunities in this skill in your course?
Is this skill identified in one or more course learning requirements?
Do you devote a significant amount of time to facilitating student development of the skills and knowledge embedded in the outcome?
Are there assignments, tests or projects which are designed to allow you to evaluate or assess student performance of this outcome or some of its elements?
Do you, in your evaluation of student performance, verify that this particular outcome (or a significant component of it) has been achieved?
Is this outcome reflected in your course outline in the course learning requirements and/or embedded knowledge and skills, and in the evaluation of the course outline?
C ulminating P erformance Culminating performances are tasks or activities designed to assess a learner’s ability to demonstrate one or more learning outcomes in their totality. While they do not necessarily occur at the end of a program of study, they do evaluate whether a learner is able to integrate and apply their learning to demonstrate the performance described in the learning outcome(s) at the exit level.
Is there an opportunity for you to evaluate the outcome in its totality?
Does the evaluation result in a final product or performance which allows you to determine whether the learner has integrated the knowledge and skills identified in the elements of performance?
Can you determine from this performance if the learner has demonstrated the outcome?
Program culminating performances should require learners to demonstrate learning similar to what would be expected of new graduates in as close t o a “real world” context as possible.
If a culminating performance is assigned to a course, the professor teaching the course will be asked to evaluate whether or not the students successfully demonstrated the outcome(s) the culminating performance addresses
Table 1: Guidelines for Teach, Assess, Culminating Performance
33
Course Outline Review
Each year, as part of Annual Curriculum Review, course owners are expected to formally review their course outlines and course section information and update courses as necessary to reflect changes made in the course content. Course outline updates may include changes to course learning requirements, learning activities, or assessment methodologies.
In the PQR year, each course teacher submits their course outline check-up worksheets or
equivalent for each course taught to the Team Leader ahead of the faculty focus group meeting.
The course teacher uses this review to reflect on the course outline, identify areas which may require strengthening, and to determine whether the course is meeting College requirements for course documentation (see for example, the Data Entry Guidelines and Policy AA26 Course
Outlines and Course Section Information ). The Team Leader b uilds on the course teachers’ observations, using the check-up sheets to formulate discussion points for the faculty focus
group, and prepare a Course Outline Review summary sheet for the final PQR report.
For the summary sheet, the course teacher and Team Leader need to assess each course against the five evaluation elements pertaining to course learning requirements, program outcomes, learning activities and resources, evaluation methods, and prior learning assessment and recognition (PLAR) to determine whether:
There is congruency between the course learning requirements and the program learning outcomes (Quality Criteria Element #3.8). (Note this is for VLOs and EES outcomes.)
There is a match between course learning requirements, course learning activities, and learning resources (Quality Criteria Element #3.9).
Learning methods are published and are matched to the learning requirements (Quality
Criteria Element #4.3).
Evaluation methods allow students to demonstrate the course learning requirements
(Quality Criteria Element #4.7).
PLAR opportunities exist and are based on course learning requirements (Quality
Criteria Element #1.4).
The program overview report, available via COMMS, is particularly helpful when reviewing the
Curriculum Map, as it provides in one report, a side-by-side view of all courses per level. For any curriculum gaps noted, an action plan is developed and included in the final recommendations.
Vocational Learning Outcomes – Reviewing the Map
At the faculty focus group, the Team Leader facilitates the review of the Vocational Learning
Outcomes maps (by Program and by Level) using the checklist Analysis and Review of
This checklist is included in the final PQR report.
In addition to analyzing the VLO maps for coverage of each VLO and flow of the T, A and CP for each outcome, the Program of Study is reviewed. Due consideration must be given to the sequencing of the courses – is it appropriate? – and to the prerequisites and corequisites – are they appropriate to ensure students have an adequate foundation on which to build their learning without unnecessarily impeding progression in the program?
34
Any actions arising are noted, and included in the recommendations in the final PQR report.
Essential Employability Skills – Reviewing the Map
Algonquin College has an Essential Employability Skills committee whose membership includes faculty members and the PQAA. This committee reviews the current approved EES map for all mandated courses in the program under review, and submits a report to the PQR Team Leader.
Items that are considered in the review include whether
all EES learning outcomes are included at a program level
there are at least two culminating performances for each skill
there is an opportunity for learners to be taught and assessed in each skill prior to demonstrating the culminating performance
there is congruency between what is reflected on the map and in the course outlines
there are any gaps or redundancies within the map.
The EES report is used by the Team Leader to aid in Curriculum Mapping. It is also included in
Appendix A of the final PQR report.
Details regarding EES at Algonquin College are found at: http://www.algonquincollege.com/ees and within Policy AA28 -- Essential Employability Skills . Both sites contain information regarding the leveling expected from EES outcomes for each Ontario College Credential.
General Education Requirement – Reviewing the Map
Algonquin College has a General Education committee, whose membership includes faculty members and the PQAA. This committee reviews the current approved General Education map for all mandated courses in the program, and submits a report back to the PQR Team Leader.
In addition, the committee reviews the mandated General Education courses to determine whether they meet General Education requirements. This information is also forwarded to the
Team Leader.
The General Education report is used by the Team Leader to aid in Curriculum Mapping. It is also included in Appendix A of the final PQR report.
Details regarding General Education at Algonquin College are found at: http://www.algonquincollege.com/gened and within Policy AA27
–
General Education .
35
EVALUATION AGAINST THE QUALITY CRITERIA
The criteria against which the program is evaluated are provided below. First, a listing of the
elements are based on the PQAPA quality criteria, and at times there is necessary repetition.
Evaluation Criteria Listing
Criterion 1. Program learning outcomes are set, are consistent with the college mission and the program’s intended purpose, and are appropriate for the credential offered upon completion of the program.
1.1 Program learning outcomes are consistent with the College mission, appropriate to the level at which the qualification is offered, consistent with the requirements of the
Credentials Framework, and appropriate to the occupational requirements of the program graduates.
1.2 Program learning outcomes are consistent with MTCU Provincial Program Standards where they exist.
1.3 Program learning outcomes are reflected in course outlines.
1.4 Program learning outcomes are used in prior learning assessment and recognition
(PLAR).
1.5 Changes to courses and program learning outcomes are introduced on a timely basis and are designed to maintain the relevance of the program.
1.6 The capabilities of program graduates, including knowledge, understanding, skills, and attitudes, are consistent with the intended program learning outcomes.
Criterion 2. Admission, credit for prior learning, promotion, graduation, and other related academic policies support program development and student achievement of program learning outcomes.
2.1 The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program.
2.2 Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them.
2.3 Students know how to get internal and external transfer of academic credits and recognition for prior learning.
2.4 Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.
2.5 Students know how they will be evaluated.
2.6 Students indicate the learning requirements are relevant and meaningful.
2.7 Students indicate that assessment methods relate to the learning requirements.
Criterion 3. Programs conform to the Framework for Programs of Instruction and the
Credentials Framework, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance.
3.1 The duration and structure of the program are consistent with the program learning outcomes and the credential offered.
36
3.2 Appropriate credits are allocated for each component of the program, and transfer and laddering options are stated.
3.3 Prerequisites do not unnecessarily hinder progress in the program.
3.4 The program title is consistent with college system nomenclature / titling principles.
3.5 The program has established articulation agreements.
3.6 The program conforms to the College policy for the number of English courses.
3.7 All curriculum documentation is up-to-date including course outlines and the program monograph information.
3.8 There is congruency between the course learning requirements and the program learning outcomes.
3.9 There is a match between course learning requirements, course learning activities, and learning resources.
3.10 Concepts of social, economic, and environmental sustainability are embedded in the program curriculum.
3.11 Work Integrated Learning, such as co-operative education work placement, clinical/field placement, service learning, and/or participating in applied research projects, is embedded in the program curriculum.
3.12 Students have opportunities to develop the skills, knowledge, and attitudes necessary to succeed in a global economy.
Criterion 4. Methods of program delivery and student evaluation are consistent with the program learning outcomes.
4.1 Program delivery, including that which takes place off-site, is consistent with the nature of the program, its learning outcomes, and the needs of the students.
4.2 There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities.
4.3 Learning methods are published and are matched to the learning outcomes.
4.4 College designated targets regarding hybrid courses are met.
4.5 Learners are provided the skills necessary to be successful with the learning strategies selected.
4.6 Evaluation criteria are published and students are aware of how and when they are going to be evaluated.
4.7 There is a match between course learning requirements and evaluation methods, i.e., evaluation methods allow students to demonstrate the course learning requirements.
4.8 Evaluation methods are valid and reliable.
4.9 Students indicate that feedback is timely and allows them to build on their learning.
4.10 Students perceive evaluation to be fair.
4.11 Practices for resubmissions, supplementals, and appeals are published, and are appropriate, fair, valid, and implemented consistently.
4.12 Student workload and assessment is balanced across the term at both the course and program level.
4.13 There is a range of evaluation methods used consistent with a variety of learning styles.
4.14 Learners can earn credit for up to 75% of the program hours using the PLAR process.
37
4.15 Academic policies and practices that provide for the development and continuous improvement of teaching and learning methods are valued, documented, and supported.
Criterion 5. Human, physical, financial, and support resources to support student achievement of program learning outcomes are available and accessible.
5.1 Program faculty members, as a whole have adequate academic preparation and workplace experience to deliver a quality program.
5.2 Program faculty members are formally evaluated every three years.
5.3 Program faculty members engage in professional development activities that ensure they are current in their field and developing teaching expertise.
5.4 Program faculty members work within clear and well-structured instructional plans.
5.5 Program faculty members participate in reflective practice.
5.6 Students consider faculty to be available.
5.7 Students consider faculty to be adequately prepared for class.
5.8 Students consider faculty to promote a positive attitude to learning.
5.9 All students are assigned an academic advisor.
5.10 Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.
5.11 Labs, clinical facilities and placement facilities are complementary to, and integrated into, the program and allow the learner to demonstrate the program learning outcomes.
5.12 Students indicate that there are adequate and accessible learning resource materials to allow them to be successful, including: textbooks in the bookstore; online materials; print resources; equipment; and student support services.
5.13 The program is financially viable.
5.14 The demand for the program has been sustained for the last five years.
5.15 There is a future demand for graduates of this program.
5.16 The learning environment is safe.
5.17 The students are provided with the information they need to know to function safely in both the College and workplace learning environments.
5.18 Program materials, including the program monograph, course outlines and course materials, are provided in an accessible format.
Criterion 6. Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.
6.1 Students indicate that they are satisfied with the program.
6.2 Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the council.
6.3 Learners progress through the program, achieve program outcomes and graduate in a timely fashion.
6.4 Learners with a wide range of abilities demonstrate the expected learning outcomes.
6.5 Items raised by the Program Advisory Committee are considered in a timely fashion and feedback is provided to the committee.
6.6 Graduates are satisfied with the overall program experience.
38
6.7 Graduates are obtaining employment in their fields.
6.8 Graduates are successful in obtaining external licenses or credentials where relevant.
6.9 Employers are satisfied with graduate performance.
Responding to the Evaluation Criteria
Each evaluation criterion is designed to examine a specific area or areas of quality in the program. In addition to responding Yes, No, or Not Applicable (NA) to each element of each quality criterion, the reason or reasons for the response must be provided. It is preferable to have each response supported by more than one piece of evidence, for example, a response to element 5.16 could be
5.16 The learning environment is safe (Yes)
Course assessment roll up data indicate that students feel safe (range is 84 to 91% of students over past 5 years).
This was confirmed by current students at the student focus group, although it was noted that in some instances level 01 students were reluctant to ask questions in large classroom environments.
39
Evaluation Criteria Listing – with help text
The following text will help to evaluate the program against the quality criteria. In some situations there will be adequate survey data and additional information to state that an element is met without further inquiry. In other areas, there may be residual questions after reviewing the available documentation. These questions then need to be explored with any or all of the
Chair, the faculty, the students or external stakeholders.
Criterion 1. Program learning outcomes are set, are consistent with the college mission and the program’s intended purpose, and are appropriate for the credential offered upon completion of the program.
1.1 Program learning outcomes are consistent with the College mission, appropriate to the level at which the qualification is offered, consistent with the requirements of
PEQAB, and are appropriate to the occupational requirements of the program graduates.
Algonquin College’s mission is
‘To transform hopes and dreams into skills and knowledge, leading to lifelong career success.’
Program learning outcomes form the foundation of any program offered at Algonquin College.
They provide the structure from which programs are developed and renewed, and as such, a review of the program’s learning outcomes is a fundamental component of PQR.
Review the Provincial Program Standard for the program, or, if not published, the PQAA can assist with locating the program description.
Vocational Learning Outcomes are routinely reviewed by Faculty and with the Program Advisory
Committee. During PQR they are examined to ensure that the outcomes remain valid and are at the level appropriate for the credential, and appropriate for the occupational requirements of
program graduates ( Curriculum Mapping 8
and with the External Stakeholders).
The Credentials Framework is included in Appendix B of this guide for reference. Care should
be taken to review all the expectations for the credential offered – duration, structure, General
Education requirements etc. This information will be useful for responding to other quality criteria elements, for example #3.1, #3.2.
In a situation where curriculum does not meet the College or Ministry Standard in any area, a plan to ensure the curriculum will conform in the subsequent academic year must
8 Note that Curriculum Mapping ensures that the review of the outcomes extends to the General Education component of the program, and the Essential Employability Skills outcomes. Information on EES leveling can be obtained from the EES committee, or through the EES Policy AA28 Essential Employability Skills . (Reports on General Education and EES are provided by the appropriate committees to the PQR Team Leader.)
40
be clearly identified in the recommendations and implementation plan . Changes will be finalized during the annual curriculum review period.
1.2 Program learning outcomes are consistent with MTCU Provincial Program Standards where they exist.
Program learning outcomes must meet or exceed the MTCU Provincial Program Standards if they exist for the program. The programs for which provincial program standards exist are found at http://www.tcu.gov.on.ca/pepg/audiences/colleges/progstan/ .
1.3 Program learning outcomes are reflected in course outlines.
This will be examined during Curriculum Mapping . All programs with course outlines loaded into
COMMS should have program outcomes reflected in course outlines. It is an important part of
PQR to review to ensure that program outcomes are easily identifiable, valid and mapped appropriately at the course level.
1.4 Program learning outcomes are used in prior learning assessment and recognition
(PLAR).
NB: how a student can challenge the course using PLAR needs to be identified on each course outline.
As each course is reviewed, it needs to be determined how faculty can evaluate the outcomes for the purposes of PLAR. Usually a department has two weeks to set up an appropriate challenge. Therefore, it is prudent that the challenge be determined in advance. The two weeks lead time allows for updating the challenge.
1.5 Changes to courses and program learning outcomes are introduced on a timely basis and are designed to maintain the relevance of the program.
Against this element, describe how the department determines what updates are necessary and how they are incorporated into the curriculum. For example, state as a result of feedback from student program councils, agency staff, and advisory committee members, changes are made annually as part of the annual curriculum review process. Refer to relevant meeting minutes to indicate how this information was gathered and used.
1.6 The capabilities of program graduates, including knowledge, understanding, skills, and attitudes, are consistent with the intended program learning outcomes.
Describe how the program provides experiential learning opportunities that allow the learners to perform, with support, as a practitioner in their field. Discuss planned learning activities that allow the student to perform as a practitioner. In addition, review and discuss the student feedback as identified in student surveys or focus groups.
41
Review the evaluation strategies for courses in the final level of the program. Are the students ready to function in the workplace at the entry level expected? This is an important question to pose at the external stakeholders focus group meeting
Criterion 2. Admission, credit for prior learning, promotion, graduation, and other related academic policies support program development and student achievement of program learning outcomes.
2.1 The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program.
In order to assess this element, each Team Leader needs to review the current admission requirements with faculty, students, and the external stakeholder groups. Faculty teaching in level 01 of the program will be able to provide valuable input about the degree of preparedness of the students entering the program. In addition, review the failure rate report. If there are courses in level 01 with consistently high failure rates, admission criteria may need to be reviewed or the course content may need to be spread out across more levels.
Students in the student focus group will give their perspective about the admission criteria and how well they feel they were prepared to be successful in level 01 without encountering difficult barriers when applying to the program.
Some programs offer remedial opportunities to assist students with specific deficiencies when the department does not want to inhibit access to the program, but recognizes that students need assistance to be successful. These opportunities should be highlighted in the report indicating the contribution they make to student success.
The response to this element may start with the wording:
Application information is published approximately 15 months in advance of the student starting in the program. This information is entered into the student information system
(GeneSIS) and approved during the curriculum review period each Winter. The admissions information is published on the OCAS website, in the Monograph, on the
College website, and the College Calendar each year in September for the next year.
and be supported by additional evidence as outlined above.
2.2 Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them.
This information is published in the Program Narrative section of the Program Monograph.
Also, one of the four capstone questions on the KPI Student Satisfaction Survey asks students
to comment on whether the program provides the skills and abilities specific to their chosen career. It is essential that the information given to applicants allows them to understand what skills they will learn, and how the program will prepare them for the workplace, as well as for which jobs in the workplace.
42
The items within this element should form key discussion points for the Student Focus Group.
The program narrative can be reviewed with students. For example, ask Did students find the program description useful?
How useful are the success factors and the employment information in helping the student with decisions regarding selecting the right program?
In addition, review the Reason for Withdrawal report to identify further issues, for example, if a large number of students withdraw citing #10, Program unrelated to career goals , as the reason for leaving the program, the published materials may need to be revised.
The members of the external stakeholders focus group will be able to provide feedback regarding the Employment Opportunities section of the Program Monograph.
2.3 Students know how to get internal and external transfer of academic credits and recognition for prior learning.
Several College publications identify the process for students to follow to obtain credit for previous education and experience. A key item for discussion at student focus group meetings is whether students knew where to find this information and if they found this information useful.
It is important to ask this question to students who did apply for credit to determine if they received the assistance they needed in a timely fashion, as students who did not need to apply for exemptions may not be knowledgeable on the process.
2.4 Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.
Students need to know what the expectations and obligations are for them to advance in the program. This might include the necessity to submit health and immunization forms, proof of a driver’s licence, or a police records check. Identify any special requirements the program might have, ensuring that these are in line with College and MTCU guidelines, and note how the students are informed.
The program may also have special progression requirements, for example, students need to obtain a higher grade than just a pass to proceed on to a course for which it is a prerequisite.
Identify and review these unique situations to ensure they are legitimate requirements. Note this information, and how it is communicated to students.
2.5 Students know how they will be evaluated.
Students need to understand what to expect regarding evaluation methods, how to receive special accommodation, how to appeal a grade, and how progression is determined. Describe how the department informs students regarding assessment and progression. Examples of unique situations where students have received special accommodation can be briefly discussed. Cite information included in letters to students/applicants, within course descriptions, and course outlines. This question should be posed at the Student Focus Group.
43
2.6 Students indicate the learning requirements are relevant and meaningful.
Student surveys and the Faculty program self-audit responses may provide some insight into students perceptions. Current students can be asked this question at the Student Focus Group.
2.7 Students indicate that assessment methods relate to the learning requirements.
Student surveys and the Faculty program self-audit responses may provide some insight into students perceptions. Current students can be asked this question at the Student Focus Group
Criterion 3. Programs conform to the Framework for Programs of Instruction and the
Credentials Framework, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance.
3.1 The duration and structure of the program are consistent with the program learning outcomes and the credential offered.
The Credentials Framework document is included in Appendix B of this guide. Compare the
program duration and outcomes to the framework to ensure the program is in the appropriate credential level.
3.2 Appropriate credits are allocated for each component of the program, and transfer and laddering options are stated.
It is important for faculty to ensure that courses are organized and sequenced to support student learning, are allocated appropriate hours of study, and are consistent with published requisites. (Note that this review is formally part of annual curriculum review.)
A review of the courses which have a failure rate of more than 25% might help identify areas where students may need more foundational knowledge or where content in a particular academic level needs to be reviewed.
Opportunities for graduates to move into the next credential either at the College or at university, opportunities for receiving external certifications etc, should be noted within the final
PQR report. Also opportunities to specialize or select majors should be discussed if applicable.
3.3 Prerequisites do not unnecessarily hinder progress in the program.
Prerequisites may not be an issue for some programs. If there are situations where students, as a result of failing a prerequisite course, cannot continue in the program without waiting a whole year, it is valuable to review the prerequisite to ensure that the knowledge is indeed foundational and the student is not likely to be successful if allowed to progress without the prerequisite. When programs have identified strategies to help students stay in the program even though they may have to repeat a course, it should be noted here.
3.4 The program title is consistent with college system nomenclature / titling principles.
Please check that the program title is consistent with that expected for the credential offered, for example, ‘technician’ is normally applied to a 2-year diploma program, ‘technologist’ to a 3-year
44
advanced diploma. Note if an alternative name is attributed to the program, for example, for marketing purposes.
3.5 The program has established articulation agreements.
Articulation agreements currently in place for the program should be reviewed to ensure that any major curriculum changes made since the last review have been reflected within the agreement.
The final PQR report should include a list and brief description of any articulation agreements that exist with other postsecondary institutions. If no articulation agreements currently exist and graduates from the program regularly seek university education, the implementation plan may include a recommendation to consider establishing an articulation agreement.
Information on degree pathways is found at: http://www3.algonquincollege.com/degreepathways/ .
Note: Include in this section a list of any pre-College laddering opportunities, for example
School to College to Work Initiative (SCWI) projects occurring in partnership with the program,
(e.g. program specific Specialist High Skills Majors (SHSM) or Dual Credit initiatives).
3.6 The program conforms to the College policy for the number of English courses.
The minimum English hours of instruction for two-year and three-year programs is 90 hours, normally delivered in two 45-hour courses, and in one-year programs it is 45 hours.
Criterion 3.6 is not applicable for locally approved College Certificate, Ontario College
Graduate Certificates or Bachelor’s Degrees.
3.7 All curriculum documentation is up-to-date including course outlines and the program monograph information.
updated. Any work required to ensure course outlines are complete will be discussed with the
Chair and captured in the Implementation Plan. This includes refinements based on EES and
General Education course reviews.
Program Monograph information is updated each year. If any areas have been designated as requiring modification as a result of this PQR, state what changes are being implemented.
Otherwise, indicate the changes that are being made as a part of the annual update process.
3.8 There is congruency between the course learning requirements and the program learning outcomes.
Faculty who teach in the program will be able to provide valuable insight into the sequencing of the courses and the culmination of the various course learning requirements to the program learning outcomes. Curriculum maps will demonstrate the distribution of the program outcomes
45
across all levels and indicate if any areas need consideration, for example learning outcomes that have more than adequate coverage.
The analysis of the curriculum maps is discussed in the Curriculum Mapping section of this
manual.
3.9 There is a match between course learning requirements, course learning activities, and learning resources.
This is an important element for faculty to review as part of the Course Outline Review process.
It is important to ensure that the match is obvious. This information is captured in the Course
Outline Review summary table.
3.10 Concepts of social, economic, and environmental sustainability are embedded in the program curriculum.
Describe how concepts of sustainability are, or are planned to be, introduced into the program curriculum. If embedding of sustainability into the curriculum is a new area for consideration, faculty need to discuss plans with the Chair. The resulting implementation plan may recommend discussions at advisory committee meetings for the upcoming year to gauge required changes, with updates to the curriculum to be implemented in the following year.
Note that, effective Fall 2013, all Ontario College Credentials offered at Algonquin College are expected to formally embed the Sustainability and Internationalization VLO “ The graduate has reliably demonstrated the ability to identify and apply discipline-specific practices that contribute to the local and global community through social responsibility, economic commitment and environmental stewardship” within the program. Going forward, there must be teaching, assessment and demonstration of culminating performances for this outcome. Course outline review will facilitate the areas where courses can include this learning.
As per the Algonquin Experience and the College Strategic Plan, in addition to embedding sustainability within the curriculum, faculty also need to identify how they will role model sustainability concepts by embedding them into their day-to-day teaching. A three year faculty plan can be included in the PQR implementation plan.
3.11 Work Integrated Learning, such as co-operative work placement, clinical/field placement, service learning and/or participating in applied research projects, is embedded in the program curriculum.
As per the Algonquin Experience , programs should include an experiential component, characterized by practical, hands-on learning such as co-operative education, clinical/field placement, service learning and/or participating in applied research projects.
Describe how Work Integrated Learning is, or is planned to be, embedded into the program, and carry forward into the PQR implementation plan as appropriate.
46
3.12 Students have opportunities to develop the skills, knowledge, and attitudes necessary to succeed in a global economy.
Describe how concepts of cultural diversity, team work, flexibility, transfer of skills, interpersonal skills, problem-solving and decision-making ability, adaptability, creative thinking, selfmotivation, the capacity for reflection, etc. are incorporated within the program curriculum.
Some of these skills are currently embedded through the incorporation of the Essential
Employability Skills and the College-Wide Sustainability and Internationalization outcome (see
3.10 above).
Course outline review may highlight areas in the program where there are opportunities to include specific learning outcomes or course learning requirements that directly reflect this learning. These should be noted and the plans to update the specific course outlines should be included in the implementation plan.
Again, this element aligns with the expectations of the Algonquin Experience .
Criterion 4. Methods of program delivery and student evaluation are consistent with the program learning outcomes.
4.1 Program delivery, including that which takes place off-site, is consistent with the nature of the program, the learning outcomes, and the needs of the students.
This element should be discussed with students and faculty. With the broad mix of demographics in the classroom, curriculum design needs to be flexible. As well, students are not equally prepared to learn online, in groups, or using independent studies. It is important to note successes as well as challenges faced in the classroom.
If the program is delivered on more than one campus, discuss how regional differences are accommodated. Outline how communication between faculty of different campuses is facilitated.
(Aside: as much as possible, programs delivered at multiple campuses are reviewed in the same PQR cycle, ideally as a joint review.)
If a program is delivered off-site, describe accommodations made to ensure there is consistency in the ability of the students to demonstrate the program outcomes. This could include specifications as outlined in the Memorandum of Understanding and the process for ensuring that off campus graduates truly meet the learning outcomes.
4.2 There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities.
The Team Leader and faculty review the course outlines and the information in the course outline review form to evaluate this element for each course. Faculty need to discuss the current offering of learning activities, and determine whether the variety of activities is sufficient for the learners in each course.
State any anecdotal information about special accommodations that the program implements to aid student success. For example, discuss strategies used to engage learners, support special needs, or provide valuable learning opportunities.
47
4.3 Learning activities are published and are matched to the learning outcomes.
Learning activities and learning outcomes are published on each course outline. It is important to review the stated learning activities to ensure that the learning activities chosen do indeed map to the stated outcomes.
4.4 College designated targets regarding hybrid courses are met.
Online learning is an integral component of programs offered at Algonquin College. With some exceptions, all programs are to have a minimum of 20% of program hours conducted through online learning. Examine how online learning is integrated in the program, whether the 20% expectation is being met.
Related to this element is the examination of the positioning of online learning – is it appropriately sequenced through the Program of Study, with sufficient introduction and orientation? This is a topic to discuss at the faculty focus group.
4.5 Learners are provided the skills necessary to be successful with the learning strategies selected.
As part of annual curriculum review activities, faculty review course learning activities and ensure that relevant skills are introduced at appropriate points in the Program of Study to ensure students have the necessary skills to be successful. Document how, for example, faculty provide orientation to online learning to ensure success in later courses, or, if group projects are required, how orientation to group work is provided. Strategies used for any other skills needed by students to be successful learners may also be captured.
It is important to review the Failure Rate report for courses where special skills are required and identify whether students are successful.
Note that the second English course is expected to provide students with presentation skills.
This course should be reviewed to ensure the content includes the skills needed to meet the expectations of the workplace.
4.6 Evaluation criteria are published and students are aware of how and when they are going to be evaluated.
The course outline review process will indicate areas where evaluation criteria, including type and timing of evaluation, and marking schemes, such as rubrics, are not available to the student. Penalties for late assignments need to be reviewed to ensure that penalties are consistent across the program or that any differences in specific courses or course sections are justifiable, published, and implemented equitably (see also element #4.11). Briefly describe policies and how they impact student success.
4.7 There is a match between course learning requirements and evaluation methods, i.e., evaluation methods allow students to demonstrate the course learning requirements.
This match is assessed during course outline review. It is important to include examples about the evaluation methods used and how they allow students to demonstrate the outcomes.
Highlight special accommodations made or innovative methods used by the program faculty.
48
There are many resources available regarding learning assessments. One useful website is http://gototheexchange.ca/index.php/curriculum-at-course-level/assessing-the-achievements-oflearning maintained by the ABC program.
4.8 Evaluation methods are valid and reliable.
Describe processes that faculty use to ensure that evaluation methods are reliable and valid, for example, using rubrics, examining negatively discriminating questions on multiple choice examinations prior to submitting final grades, or reviewing exam outcomes to determine questions that students answer poorly.
Pay particular attention to courses which have a failure rate of 25% or higher. It may be that the evaluation strategies are not aligned with the outcomes, or the course needs to be better placed. Actions for curriculum update should be noted and carried forward into the implementation plan.
4.9 Students indicate that feedback is timely and allows them to build on their learning.
Student surveys may provide insight into past students’ experience. This element can be further explored at the student focus group. If the students indicate that feedback is not timely or helpful, note this in the PQR report being careful to maintain confidentiality. Recall that PQR is examining the program as a whole. Should matters arise that pertain to individuals, please work with the Chair/PQAA to ensure confidentiality is preserved.
4.10 Students perceive evaluation to be fair.
Student surveys may provide insight into past students’ experience. This element can be further explored at the student focus group.
4.11 Practices for resubmissions, supplementals, and appeals are published, and are appropriate, fair, valid, and implemented consistently.
Describe the program policy and how it is published and implemented. In particular, department practices can be compared to Policy AA13 - Evaluation of Student Learning and Policy AA39 -
Program Progression and Graduation Requirements .
4.12 Student workload and assessment is balanced across the term at both the course and program level.
This should be examined by the Team Leader and at the faculty focus group meeting. Discuss here the efforts faculty members make to ensure a balance of workload and assessment of all courses. The COMMS Program Overview report is a useful tool to visualize the student experience in each level, and to help plan future workload balancing.
Further feedback may be requested at the student focus group meeting.
4.13 There is a range of evaluation methods used consistent with a variety of learning styles.
This element will need to be discussed with both faculty and students. The COMMS Program
Overview report is a useful tool to visualize the student experience in each level. In the PQR
49
report, highlight methods used that demonstrate meeting the needs of various learning styles, and highlight any innovative evaluation strategies used.
4.14 Learners can earn credit for up to 75% of the program hours using the PLAR process.
Whether a course is eligible for PLAR is noted on each course outline, along with the method(s) students may use to provide evidence of learning achievement. The team leader notes whether the course is eligible on the Course Outline Review Summary Sheet. To determine whether the
75% threshold is being met, tally the number of hours in each PLAR-eligible course and compare this to the total number of program hours.
Where a course is not eligible for PLAR, capture the reason(s) why in the Course Outline
Review Summary Sheet. This should be reflected also in the course outline.
If the program has less than the required number of hours with PLAR challenges available, note this against the element, and include a recommendation to address the shortfall in the PQR implementation plan.
Note that PLAR challenges need to be established using the existing College and Ministry policies and fees structures. Assistance to define the challenges is available from the Centre for
Organizational Learning . Recall that, on the course outline itself, it is only necessary to identify whether the course is challengeable, and to provide contact information. The actual challenge format should be identified and planned, although time will be needed and allowed to ensure the challenge tool is updated.
Aside: Once a challenge process is in place for a course, the PLAR flag needs to be set within
GeneSIS for the course.
4.15 Academic policies and practices that provide for the development and continuous improvement of teaching and learning methods are valued, documented, and supported.
Describe how program faculty have experimented with new teaching methods and how they evaluate the effect on student success.
Identify any relevant Professional Development that faculty have attended and found useful.
This information will be found in the summary of faculty credentials and experience summarized by the Chair.
It is helpful to identify areas the faculty feel could be improved to help them develop their teaching skills. For example, if students identify online courses as challenging, a follow up action may be to assign PD for faculty to improve online course delivery.
50
Criterion 5. Human, physical, financial, and support resources to support student achievement of program learning outcomes are available and accessible.
5.1 Program faculty members, as a whole, have adequate academic preparation and workplace experience to deliver a quality program.
Information regarding the preparation and experience of the faculty team is gathered by the
Chair, who submits a summary report to the Team Leader. Student surveys will provide past student perceptions of faculty knowledge and skills.
Any concerns that faculty have should be identified and considered in the recommendations.
Similarly, celebrate the experience and expertise that is available to support student learning.
5.2 Program faculty members are formally evaluated every three years.
Data to support this element is within the summary report that the Chair will submit to the Team
Leader. Summarize the data in the final report.
5.3 Program faculty members engage in professional development activities that ensure they are current in their field and developing teaching expertise.
Data to support this element is within the summary report that the Chair will submit to the Team
Leader. Summarize the data in the final report.
Feedback should also be solicited during the Faculty focus group meeting regarding the availability and usefulness of professional development.
5.4 Program faculty members work within clear and well-structured instructional plans.
All courses are expected to have documented course outlines and course section information within which faculty work. This will be verified during Curriculum Mapping. Should there be any issues at a program level, a plan to address should be made with the Chair, and captured within the PQR implementation plan.
5.5 Program faculty members participate in reflective practice.
Data to support this element is within the summary report that the Chair will submit to the Team
Leader. Summarize the data in the final report.
Feedback could also be solicited during the Faculty focus group meeting regarding the reflective practice methods used, successfully or otherwise, by faculty.
5.6 Students consider faculty to be available.
Students views are captured at the focus group, and garnered from student surveys, including
KPIs. Additionally, the Chair can learn if there is a generic or individual issue as part of the
Student Course Feedback process. The Chair should not release confidential information; however, if survey data reflects that student perception is that faculty are not available, assurance needs to be provided that any issues regarding faculty availability will be addressed in the Implementation Plan. Ask the PQAA for assistance with the writing of this response if necessary.
51
5.7 Students consider faculty to be adequately prepared for class.
Students views are captured at the focus group, and garnered from student surveys, including
KPIs.. It is necessary here, as above, that any concerns raised by students are discussed in a confidential manner and are addressed in the Implementation Plan similarly. The PQAA may be of assistance when determining how to document areas which need to be addressed.
5.8 Students consider faculty to promote a positive attitude to learning.
A positive attitude to learning is considered to be an essential aspect of a quality learning environment. Please capture, if overall, the students consider program faculty to promote a positive learning environment. This question can be posed at the student focus group. Again, if concerns arise, these should be discussed in a confidential and respectful manner. The PQAA can assist in documenting appropriately.
5.9 All students are assigned an academic advisor.
Describe how academic advisors are allocated for the program.
5.10 Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.
Describe how students are informed about their academic advisor and in general, how the program implements Policy AA40 – Academic Advising . This is a good topic for the Student
Focus Group. Remember that not all departments use the same terminology, therefore a more generic question can be posed, like “ Do you know where to get help if you have a problem with your program?
”.
5.11 Labs, clinical facilities and placement facilities are complementary to, and integrated into, the program and allow the learner to demonstrate the learning outcomes.
Discuss how classrooms are used, the types of labs available, and placement activities used to provide learning opportunities for students, and allow them to demonstrate program outcomes.
Celebrate quality facilities and identify changing or future needs that may have been identified.
5.12 Students indicate that there are adequate and accessible learning resource materials to allow them to be successful, including: textbooks in the bookstore; online materials; print resources; equipment; and student support services.
Student surveys will provide insight into past students’ views. This quality criteria element can also be discussed at the Student Focus Group meeting.
5.13 The program is financially viable.
5.14 The demand for the program has been sustained for the last five years.
5.15 There is a future demand for graduates of this program.
Conclusions for these three elements can be drawn from the program data which summarizes the last 5 years of program contribution, and application and registration data.
52
Programs are expected to make a 25% net contribution. Given program budgets are developed using student enrolment projections, programs may face financial challenges if registration numbers are below projections.
The Chair and the Team Leader review the net contribution data and application/registration ratio to see the trend over the last five years. Combined with a review of the employment stats trend, issues about demand for the program, and/or demand for graduates can be identified for the focus group meetings. Registration and withdrawal statistics from the last 5 years will help identify any significant areas that may require follow up in the focus groups.
5.16 The learning environment is safe.
This question must be asked at the Student Focus Group meeting. If students identify areas where they feel unsafe, strategies to resolve issues should be discussed and feedback given. It is important that students feel safe asking questions, i.e., that the classroom environment is conducive to students being comfortable sharing their learning needs. Student surveys will provide insight into past students’ views.
5.17 The students are provided with the information they need to know to function safely in both the College and workplace learning environments.
Describe any special orientation processes provided for learners to help them function in the classroom, lab, or workplace environments. Any special precautions taken in lab settings, for example WHMIS protocols, and use of safety equipment can be discussed here.
5.18 Program materials, including the program monograph, course outlines and course materials, are provided in an accessible format.
Program materials are required to be available in an acces sible format as per the AODA’s
Integrated Accessibility Standard Regulations . With the Chair and faculty, review the current status of program materials, and gauge faculty awareness of the AODA. PD requirements and required updates to program materials (in order to comply with requirements of AODA) must be captured in the PQR Implementation plan.
Criterion 6. Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.
6.1 Students indicate that they are satisfied with the program.
Several questions on the KPI Student Satisfaction Survey speak to student satisfaction:
overall quality of knowledge and skills given by the program
overall quality of the learning experiences in the program
overall college facilities
overall quality of services
The five year trend for the average student satisfaction, as well as for each capstone, is in the program data . Outcomes on these questions may be discussed at the student focus group.
53
If available, Course Assessment Roll Up Report may also be used to identify student satisfaction with course quality, especially the capstone question ‘Overall how do you rate the quality of this course?’.
Analysis of student survey data student may identify issues that need to be discussed at the student focus group meeting.
6.2 Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the Council.
Discuss Program Council activities here, and how that process works in the program. A review of Program Council minutes can provide supporting data that indicates that issues from one term are followed up until resolved or identified as no longer needing action. The discussion of the feedback mechanism that is in place for the academic staff to communicate to students across the program can also be discussed.
6.3 Learners progress through the program, achieve program outcomes and graduate in a timely fashion.
When considering this element, examine retention, progression and graduation rate for the program.
Registration reports for the last five years help identify any terms in particular that appear to inhibit a student’s progress through the program.
Identification of courses which have high failure rates may need to be reviewed and strategies recommended that allow the students to be successful while maintaining the academic integrity of the program. Faculty and students can provide valuable feedback for this element.
Concerns here also include core courses that are only offered once a year. Consider whether there are strategies available to allow students to continue without having to lose a whole academic year if they fail the course.
When courses with high failure rates are offered off-cycle to help students remain on cycle in the program, this should be discussed together with strategies developed to help the students be successful, for example specialized learning tools for courses that require a lot of rote learning, such as anatomy or geography.
In cases where students are not successfully progressing through the program, endeavor to establish the cause – curriculum, financial, other inhibitor – in order that consideration may be made to provide additional support and/or adapt the curriculum.
6.4 Learners with a wide range of abilities demonstrate the expected learning outcomes.
While no statistics are available for the number of students with differing abilities and their success in the program, faculty and students may provide some anecdotal information relevant to this element. Highlight special accommodation that program staff members have made to assist students with special needs to meet the program outcomes and be successful in graduating from the program.
54
6.5 Items raised by the Program Advisory Committee are considered in a timely fashion and feedback is provided to the Committee
Discuss Program Advisory Committee activities here, and include specifically how the relevance of the program to the field of practice it serves is assessed, and the program modified if required. A review of Program Advisory Committee minutes can provide supporting data to indicates whether program items are followed up until resolved or identified as no longer needing action.
6.6 Graduates are satisfied with the overall program experience.
Five years of Graduate Satisfaction data are provided within the program data (KPI graduate satisfaction report). Low satisfaction and/or declining trends in this area can be further investigated at the external stakeholders focus group meeting. It is critical to ensure that several graduates attend this meeting.
6.7 Graduates are obtaining employment in their fields.
The program data provides five years of data about employment rates, which is calculated directly from the KPI Employer Satisfaction Survey. Trends within these data can be discussed at the external stakeholder focus group meeting.
It is important to identify factors affecting employment rates at the time of the KPI Employer
Satisfaction Survey. For example, graduates of a building trade program who finish their program in August may not find employment until the following Spring, however the survey may be carried out in February.
6.8 Graduates are successful in obtaining external licenses or credentials where relevant.
Include this information and its relevance as applicable. For example, if the students write a licencing examination to obtain provincial or national certification and the results of the success of your graduates on those examinations are available, include them here.
6.9 Employers are satisfied with graduate performance.
Review the five year trend for employer satisfaction within the program data. Note that the response rate for the employer survey is usually low, as prior to contacting the employer, the graduate ’s permission must be obtained (along with employer contact details). Due to the lower response rates, the data may not be as meaningful as data from other surveys.
Any issues that arise from the survey data can be added to the agenda of the external stakeholders focus group meeting.
55
CONDUCTING FOCUS GROUPS
The memos and templates mentioned in this section are found in Appendix C .
Focus groups are held to gather feedback from the participants about their perception of the program’s effectiveness. Once the Team Leader and the Chair have reviewed the available survey data and program documentation, they determine which elements of the evaluation criteria require additional input from members of the focus groups and prepare the agendas for the specific focus group meetings.
Each program Team Leader plans the focus group meetings and prepares the agendas for their own program meeting, and, excepting the faculty focus group, facilitates the meetings for a
Team Leader from another School. The report of the focus group meetings will be written by the facilitator, with an opportunity provided to the program Team Leader and Chair to review the report for accuracy before it is finalized.
Note : Facilitator partners need to work together to set meeting dates and room logistics for shared focus group meetings,
The following pages contain detailed directions to assist in planning and facilitating a focus group meeting and writing a report on the feedback obtained. It is understood that facilitators will dress and act professionally for these meetings. In the event of illness, the Chair acts as a facilitator, so please ensure that each Chair is informed of upcoming focus groups.
For some programs it is not feasible to convene a focus group. In these instances, an alternate means of collecting the information may be considered, for example, through an online survey.
The survey would be composed by the program Team Leader, but conducted, collated and written into the report by the partner facilitator.
Memos from the Chair and Team Leader
At the outset of the PQR, the Chair sends a notice to the faculty to introduce the Team Leader,
and to ask each staff member to fill out a faculty credentials and experience sheet.
the program self audit form , and to bring the most recent review of each program course that
they teach.
The Chair will also send out an invitation to the External Stakeholders (normally the advisory
committee plus some recent graduates and relevant employers) to attend a focus group meeting for PQR. The invitation to the student focus group meeting can be sent by the Team
Leader.
Faculty Education, Experience, and Professional Development Survey
Ahead of the faculty focus group, each member of faculty fills and sends to the Chair a credentials and experience sheet. The Chair then summarizes this and sends it to the
Team Leader, who uses it to prepare for the upcoming focus groups, and to respond to the elements of the evaluation criteria.
56
Program Self-Audit
Ahead of the faculty focus group, each member of faculty assesses the program against the quality criteria, and provides their assessment to the Team Leader to summarize for inclusion into Appendix B of the final PQR report, and use to prepare for the upcoming focus groups.
Course Outlines
Each course that contributes to the program needs to be self-assessed ahead of the faculty focus group meeting 9 . A tool for course writers (the
tool, available in Appendix C) can be used to review each course, within which there is an option for peer review. The review of course outlines forms part of curriculum mapping. Course section information should also be self-assessed in order to respond to element #5.4 and possibly #5.5.
Setting the Focus Group Agendas
An example agenda for each focus group is provided in Appendix C . Agendas are adjusted
based on the findings from the document review and data analysis performed ahead of the focus group meetings. Discussion points are defined or expanded to address gaps or ambiguities. Where there is adequate evidence that an element of a criterion is met, it does not need to be brought forward to the focus group.
Tips for Planning and Facilitating Focus Groups
Preparing and Planning for the Meeting
Objectives - identify the major objectives of the meeting
Questions - develop lead questions o Use five or six questions from the list of possible items for discussion, provided for each focus group o Focus on areas that assist in projecting the future of the program o Identify areas of improvement within the program
Date and Time - set date and time (two hour meeting plus social/networking time)
Location - reserve a conference room and order light refreshments 10
Invitees - generate guest list and send invitations (four weeks prior to the meeting)
Agenda - set the agenda based on the information gathered to that point (begin with an simple topic)
Materials - determine the meeting recorder and prepare/order necessary materials (flip chart paper or laptop/projector, markers, etc.)
Facilitating the Meeting
Thank you – thank guests for their participation
Introductions – establish positive rapport and comfortable environment by taking the time for personal introductions
Overview – review the meeting agenda and purpose and set the ground rules
9 Course outline review happens annually as part of Annual Curriculum Review
10 Expenditure must follow guidelines as per College Policy AD12 “ Travel, Meals and Hospitality ”
57
Questions – ask each question and record the responses. Listen attentively and request clarification as necessary then reflect back to ensure the notation is accurate.
Participation – encourage participation by all attendees
Timing – be mindful of the meeting duration, allowing time for review and recommendations.
Closing – thanks guests once again for their valuable feedback and close the meeting
SMILE – be positive and face the group as much as possible
Post Meeting Tasks
Organize – gather notes, number pages, and comment on any important observations
Report - write the report within two weeks
Ensure that focus group participants are provided feedback on the meeting.
58
Faculty Focus Group
The Team Leader will convene a meeting of as many faculty members who teach in the program as possible, including part-time faculty and professors of service courses. Part-time
faculty members will be reimbursed for attending this meeting as per guidelines earlier in this
document.
Faculty Focus Group sessions should be standardized as much as possible for all programs.
The agenda for each session should closely resemble that in Appendix C .
Ahead of the faculty focus group meeting,
all faculty must provide a completed Program Self-Audit, which the Team Leader will have summarized in order to set discussion items for the agenda.
course teachers are to provide a review of their course, using the course outline summary sheet or equivalents
The main agenda items for the faculty focus group meeting are
reviewing the mapping of Vocational Learning Outcomes to courses
review of Course Outlines against the 5 course specific elements of the quality criteria
(as per section Curriculum Mapping )
review the interaction of course material in support of achievement of learning outcomes
discussing issues (arising from Program Self-Audit)
A second faculty meeting may be planned after all the focus groups have been completed, depending on the number of faculty on the team. At this time, the Team Leader and the Chair can share with faculty recommendations made at the external stakeholders focus group or the student focus group meeting. The input from this meeting, should it be held, would be documented in a separate report and included in Appendix B, and conclusions and final recommendations formulated for the final report
Facilitating the focus group
The Team Leader opens the meeting by welcoming everyone and providing an opportunity for introductions as necessary. After a brief overview of the Program Quality Review process, the
Team Leader then reviews the Agenda, and explains the process and ground rules.
The ground rules for the discussion are:
Ask that only one person speak at a time.
Members are asked to be as concise and focused as possible.
All comments/perceptions should be freely stated and will be recorded by the Team
Leader/recorder
All contributions are regarded as valid and no consensus is required with respect to the perceptions and feedback stated and recorded.
The Team Leader will attempt to achieve consensus only with respect to final recommendations to be made to the program.
59
All elements of the discussion should be recorded during the meeting. The Team Leader may enlist the help of an external recorder, if one is available in the department. The meeting notes will serve as a permanent record of the discussion and will provide the basis for the focus group report.
Preparing the focus group report
The Team Leader will prepare the report for the meeting which s/he has facilitated, using the notes from the meeting as the basis for the report. The Chair and the Team Leader must meet to review the report before it is finalized.
The faculty focus group report includes
the agenda, date, and time of the meeting
the names of the faculty members who attended the meeting
a summary of the meeting – the highlights, key findings, and recommendations.
The final report does not contain the detailed discussion notes taken by the Team
Leader/recorder.
60
External Stakeholders Focus Group
Planning the External Stakeholders Focus Group
The program Team Leader and the Chair meet to determine who to invite to the external stakeholders focus group. Participants include relevant Advisory Committee Members, additional employers, recent graduates and other guests as determined by the chair and Team
Leader. Faculty may be invited to the meeting. The Chair sends the invitation. A draft invitation
The program Team Leader prepares the agenda, including discussion questions, for the external stakeholders focus group meeting, however Team Leaders facilitate the external focus group for a program other than their own. Conduct of the external stakeholders focus group meetings should be standardized as much as possible for all programs. The agenda for each
meeting will closely resemble the sample included in Appendix C .
Note that the focus group leader will prepare the report for the meeting which s/he has facilitated, using the meeting notes as the basis for the report.
After the focus group meeting has been held, a letter is sent to each participant by the Chair to thank participants. This letter can include
a brief summary of the recommendations
a note that these recommendations may be included in the final implementation plan
how the department will inform the participants of the availability of the focus group report.
Time Frame for the Meeting
The working part of the meeting should be about two hours in length. Refreshments should be available at the beginning of the meeting to minimize interruptions once the meeting commences.
Leadership of the Meeting
The Chair of the department states the purpose of the focus group meeting, introduces the focus group leader, and invites the participants to introduce themselves. The focus group facilitator then reviews the agenda, explains the PQR process and the ground rules for the focus group meeting. The Chair of the department may then leave the meeting.
Ground Rules for the Meeting
Ask that only one person speak at a time.
Participants are asked to be as concise and focused as possible.
All comments/perceptions should be freely stated and are recorded by the facilitator / recorder.
All contributions are regarded as valid and no consensus is required with respect to the perceptions and feedback stated and recorded.
The facilitator will attempt to achieve consensus only with respect to final recommendations to be made to the program.
61
If consensus cannot be reached with respect to a recommendation, this should be noted.
The final decision rests with the Chair.
All elements of the discussion should be recorded during the meeting. The focus group facilitator may enlist the help of an external recorder, if one is available in the department. The meeting notes will serve as a permanent record of the discussion and will provide the basis for the focus group report. Note that the actual meeting notes are not included in the report.
Writing the external stakeholders focus group report
The focus group leader will prepare the report for the meeting which s/he has facilitated, using the discussion notes from the meeting as the basis for the report. The final report will include
the agenda, date, and time of the meeting.
the names and capacity (e.g., the company they represent) of the attendees at the external stakeholders focus group
a summary of the meeting - highlights, key findings, and recommendations.
The intent is to give the reader a clear understanding of the input and viewpoints of the attendees of the meeting, without the heavy overhead of verbatim minutes.
A draft of the external stakeholders focus group report should be verified by the program Team
Leader and the Chair before it is finalized. A copy of the meeting notes should be available for consultation during this review.
The draft report is to be provided by the focus group facilitator to the program Team Leader within two weeks of the meeting and copied to the PQAA. If the program Team Leader wishes the final report earlier, a date can be negotiated.
Possible Items for Discussion
Below are elements of the quality criteria that may be relevant for discussion at the external stakeholders focus group meeting. Some of these elements may not need to be addressed, as earlier review of the survey data and program documents may have concluded that no further information is required.
Criterion 1
Program Learning Outcomes are appropriate for the program’s intended purpose, and the credential offered upon completion.
Does the group feel that the current vocational learning outcomes are appropriate for the program and credential?
Are changes in the field anticipated such that the vocational learning outcomes should be changed in the next 5-10 years?
Does the group have any feedback on the relevancy of any of the provincial standards?
Are graduates you receive ready to function at the entry level expected?
Criterion 2
62
Processes for granting of internal and external transfer of academic credits are clearly defined and available to students.
Does the group feel that current employees have opportunities to upgrade/receive credit
for experience/have access to the courses needed to obtain a college credential?
Has anyone at the meeting used the advanced standing, external credit or Prior
Learning Assessment and Recognition process?
Do people know where to look for transfer credit/PLAR infomation?
Do they have suggestions about where they would expect to find it?
Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them.
Do employers have suggestions regarding the employment opportunities (or other program narrative information) as outlined in the program monograph?
Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.
Do employers have any suggestions or changes to current job requirements like police records checks, special driver’s licences and/or immunizations?
Criterion 3
The duration and structure of the program are consistent with the program learning outcomes and the credential offered.
Does the group feel that the program curriculum needs to change in the next 5-10 years in anticipation of a change in the knowledge base needed to work in the field?
Does the group have any feedback on the relevancy of any of these standards?
The curriculum conforms to the College Policy for the number of English Courses.
What English skills would employers look for in our graduates? Any suggestions for applications in the workplace that could be adapted to learning activities in the classroom?
Criterion 4
Learners are provided the skills necessary to be successful with the learning strategies selected.
Are there specific skills you require graduates to demonstrate? (Examples may be related to computer competencies, specific communications skills)
Criterion 5
Labs, clinical facilities, and placement facilities are complementary to and integrated into the program to allow the learner to demonstrate the learning outcomes.
Does the group have any recommendations about the labs, equipment, and placement facilities?
63
The students are provided with the information they need to know, to function safely in both the
College and workplace learning environments.
Is there any new information the College should know about, for example, WHMIS changes, new safety regulations?
Criterion 6
Students are obtaining employment in their field.
What is the perspective of the group with respect to future employment opportunities for the next 5-10 years?
Graduates are successful in obtaining external licenses or credentials where relevant.
Do the graduates understand the legal responsibilities of their job? Are they aware of the accreditations required?
’
Employers are satisfied with graduate performance.
Does the group feel that the graduates have adequate practical experience? Do the graduates understand the reality of the workforce? Are they able to relate the skills they have learned to the workforce
64
Student Focus Group
Planning the Student Focus Group
The program Team Leader invites students to participate in the focus group meeting. It is recommended that 20-30 students be invited, depending on the size of the program. There should be fairly equal representation from all years and all sections/groups. Students should respond to the invitation, allowing time to find an alternate if the student cannot attend.
Students can be selected randomly from a class list – e.g., every fourth or fifth student, depending on the size of the class. Faculty is not invited to the meeting. A sample invitation is
The program Team Leader prepares the agenda, including discussion questions, for the student focus group meeting; however Team Leaders facilitate the student focus group for a program other than their own. Student focus group meetings should be standardized as much as possible for all programs. The agenda for each meeting will closely resemble the sample
Note that the focus group leader will prepare the report for the meeting which s/he has facilitated, using the meeting notes as the basis for the report. Note that the actual meeting notes are not included in the report.
After the focus group meeting has been held, a letter is sent to each participant by the Chair to thank participants. This letter can include
a brief summary of the recommendations
a note that these recommendations may be included in the final implementation plan
how the department will inform the participant of the availability of the focus group report.
Time Frame for the Meeting
The working part of the meeting should be about two hours in length. Refreshments should be available at the beginning of the meeting to minimize interruptions once the meeting commences.
Leadership of the Meeting
The Chair of the department states the purpose of the focus group meeting, introduces the focus group leader, and invites the participants to introduce themselves. The Chair then leaves, allowing uninhibited discussion to take place. Thereafter, the focus group leader reviews the agenda, and explains the PQR process and the ground rules for the focus group meeting.
Ground Rules for the Meeting
Ask that only one person speak at a time.
Participants are asked to be as concise and focused as possible.
All comments/perceptions should be freely stated and are recorded by the facilitator/recorder.
All contributions are regarded as valid and no consensus is required with respect to the perceptions and feedback stated and recorded.
65
The focus group leader will attempt to achieve consensus only with respect to final recommendations to be made to the program.
This is an opportunity to discuss global issues regarding the program. Any students who have individual issues regarding faculty are invited to make an appointment to discuss them with the Chair.
All elements of the discussion should be recorded during the meeting. The focus group facilitator may enlist the help of an external recorder, if one is available in the department. The meeting notes will serve as a permanent record of the discussion and will provide the basis for the focus group report. Note that the actual meeting notes are not included in the report.
Writing the Student Focus Group Report
The focus group leader will prepare the report for the meeting which s/he has facilitated, using the transcriptions of the discussion notes as the basis for the report. The final report will include
the agenda, date and time of the meeting.
the number of students, and their current academic level (it is not necessary to list the actual names or the participants, level is important as it provides context for the responses) 11 .
a summary of the meeting - highlights, key findings, and recommendations.
The intent is to give the reader a clear understanding of the input and viewpoints of the attendees of the meeting, without the heavy overhead of verbatim minutes.
A draft of the student focus group report should be verified by the program Team Leader and the Chair before it is finalized. A copy of the meeting notes should be available for consultation during this review.
The draft report is to be provided by the focus group facilitator to the program Team Leader within two weeks of the meeting and copied to the PQAA. If the program Team Leader wishes the final report earlier, a date can be negotiated.
Possible Items for Discussion
Below are elements of the quality criteria that may be relevant for discussion at the student focus group meeting. Some of these elements may not need to be addressed, as earlier review of the survey data and program documents may have indicated that no further information is required.
Criterion 1
Program Learning Outcomes are appropriate for the program’s intended purpose, and the credential offered upon completion.
Did you/ do you know the expected outcomes for your program?
Do you know where to locate them?
If so, do you think they are appropriate for the program?
11
Note that the facilitator will provide the Chair the list of participants, in order to send the thank you letter indicated earlier
66
Criterion 2
The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program.
Was the program description provided useful? And the Success Factors?
Did you know what you needed to know to get into the program?
Did you have enough of a Mathematics foundation to succeed in level 01?
Did you know anyone who wanted to get into the same program as you but didn’t – and why?
Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them.
Does the course content match what you expected to learn?
Do you expect these studies will lead to the type of career you are seeking?
Were you prepared for the financial and workload commitments needed?
Students know how to get internal and external transfer of academic credits and recognition for prior learning.
Did you know how to get any exemptions for courses taken at another postsecondary institution or here at Algonquin College in another program?
Were you eligible for exemptions or PLAR?
Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.
Does your program have special requirements you must have before you can be able to demonstrate learning outcomes/graduate?
Was the information you needed readily available?
Students know how they will be evaluated.
Is the published information adequately available and meaningful to ensure you know how you will be evaluated?
Students indicate that assessment methods relate to the learning requirements.
Are the assessment methods related to the learning requirements?
How do you know?
Criterion 3
Prerequisites do not unnecessarily hinder progress in the program.
Are there any courses that seem to obstruct progression in the program?
If so, do you have any suggestions about how this could be improved?
Courses are organized and sequenced to support student learning, are allocated appropriate hours of study, and are consistent with published prerequisites.
Does the sequencing of the courses make sense?
Do you have the foundational knowledge needed as you progress in the program?
67
There is congruency between the course learning requirements and the program learning outcomes.
Do you think that each of the courses contribute to the overall learning outcomes?
There is a match between the course learning requirements, course learning activities and learning resources.
Do the learning activities and learning resources support your success in demonstrating the course learning requirements?
Curriculum design maximizes flexibility of student learning.
Do you feel that you have adequate flexibility to accommodate your learning?
Do you have any suggestions regarding how the program delivery could be more flexible to support your learning? (What do you consider to be needed for flexibility – courses available at different times to allow them flexible schedules; different learning offerings, e.g., online as well as in class; multiple resources available, a variety of learning activities; opportunities for PLAR challenges; laddering of curriculum?)
Criterion 4
The program provides experiential learning opportunities that allow the learners to perform, with support, as a practitioner in their field.
Do you feel you are provided with adequate opportunities to perform as a practitioner in your field?
There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities.
Do the instruction methods support your learning style and your needs and abilities? Do you have any suggestions for improving this aspect of the program delivery?
Learning methods are published and are matched to the learning outcomes.
Did you know what learning methods to expect and do you feel they match the learning outcomes?
Learners are provided the skills necessary to be successful with the learning strategies selected.
Do you receive adequate preparation to be successful with the learning strategies being used, for example, online learning or group presentations?
Evaluation criteria are published and students are aware of how and when they are going to be evaluated.
Were you aware of when and how you were going to be assessed?
There is a match between course learning requirements and evaluation methods.
Do you feel that the evaluation methods allow you to demonstrate the outcomes?
Students indicate that feedback is timely and allows them to build on their learning.
68
Is feedback timely? Is it given is a way to aid future learning?
Students perceive evaluation to be fair.
Are your evaluations fair? Are there any improvements/adjustments that could be considered?
Evaluation methods allow students to demonstrate the course learning outcomes.
Do the ways the professors evaluate you allow you to demonstrate the course learning requirements?
Are there other ways that may better allow you to demonstrate your learning?
Student workload and assessment is balanced across the term at both the course and program level.
Do you find the workload to be balanced across the term for all your courses?
There is a range of evaluation methods used consistent with a variety of learning styles.
Do the evaluation methods used match your learning style?
Can you offer suggestions about other evaluation methods that would suit you better?
Criterion 5
Students consider faculty to be available.
In your experience, has faculty been available?
Students consider faculty to be adequately prepared.
In your experience, is faculty prepared for class?
Students consider faculty to promote a positive attitude to learning.
Overall, has this been your experience with your program, and at the College?
Students are assigned an academic advisor.
Do you know who your academic advisor is?
Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.
Have you met with your academic advisor this term or connected in another way? Do you have any suggestions about how your academic advisor can better support your learning?
Labs, clinical facilities and placement facilities are complementary to and integrated into the program and allow the learner to demonstrate the learning outcomes.
Do the labs, clinical, and placement facilities allow you to demonstrate the learning outcomes?
69
Students indicate that there are adequate and accessible learning resource materials including: textbooks in the bookstore; online materials; print resources; equipment; and student support services, to allow them to be successful.
Are the learning resource materials accessible? Should anything be improved?
Removed?
Do you have adequate access to student support services?
The learning environment is safe.
Do you feel safe in the College?
Is the classroom environment conducive to asking questions?
Do you feel safe in all learning environments?
Do you have adequate supervision when demonstrating new skills in a placement environment?
If there is an area where students do not feel safe, do they have suggestions about how this could be resolved?
The students are provided with the information they need to know how to function safely both in the College and workplace learning environments.
Do you feel that you are well orientated to practice lab and workplace learning environments?
Are you offered specialized safety training?
Can you identify any situations where you felt you were not able to practice your skills in a safe manner?
Criterion 6
Students indicate that they are satisfied with the program.
What parts of the program provide you with the most satisfaction?
What parts of the program provide the least satisfaction?
Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the council.
Are issues from Program Council meetings addressed and progress communicated back to students? ( Aside: Note that if an issue cannot be resolved, this must be communicated back to the Program Council.)
Learners progress through the program, achieve program outcomes, and graduate in a timely fashion.
Do you feel that the learning outcomes are achievable in the designated time frame for the program?
Have you been successful in all your courses so far? If not, why not?
70
COMPLETING THE FINAL REPORT – AN OVERVIEW
As the Program Quality review is being conducted, the Team Leader should
write the introduction section of the final report
review the available documentation and data
respond iteratively to the elements of the evaluation criteria (i.e., after document and data review and after each focus group meeting was held)
complete curriculum mapping
identify program Strengths, Weaknesses, Opportunities, and Threats
start to identify recommendations.
To close the report, the Team Leader needs to
complete responses to the elements of the evaluation criteria
write the conclusions and prepare a draft of the recommendations and implementation plan
review the report with the Chair
with the Chair, develop the Implementation Plan 12
with the Chair, write the Executive Summary
send the draft report to the PQAA for review
make updates as required, and
resubmit to the PQAA for approval and archiving.
The draft report should be submitted to the PQAA and the Program Chair as a .docx file.
All items, such as the Program Monograph and COMMS curriculum maps, should be included electronically into this final report. If administrative assistance is required in compiling the final report, please contact the PQAA.
On receipt of the draft report, the PQAA and the Program Chair will review and advise the Team
Leader of any updates required. Once the report has been finalized, the formal review and signoff procedure is followed, where the report is reviewed by the PQAA, the Chair and finally the
Dean. Once approved by the Dean, the timing for the annual follow up report is provided. A
formal sign-off sheet is used, and is provided in Appendix C . The master copy of the PQR is
stored electronically by Academic Development. A hard-copy can be provided on request.
12 The Chair is responsible for actioning the recommendations
71
ANNUAL FOLLOW UP
Each spring, the PQAA sends out a notice requesting follow up from each Chair regarding current PQR actions. Chairs summarize the progress made, and update the program implementation plan, adjusting actions based on changes such as College expectations, program budget and availability of human resources.
In addition
the Chair includes PQR recommendations in his/her annual performance contract.
the PQAA presents a summary report which includes the highlights and recommendations for each program involved in PQR for that cycle. This report is submitted to the Vice President Academic.
The PQAA presents a report which summarizes the progress on open action items from all previous PQRs. This report is also submitted to the Vice President Academic.
72
GLOSSARY AND LIST OF ACRONYMS
AODA
CAAT
COMMS
Accessibility for Ontarians with Disabilities Act
Colleges of Applied Arts and Technology
Course Outline Mapping and Management System
COL Centre for Organizational Learning
GeneSIS The College Student Information System
KPI
MTCU
OCAS
OCQAS
PLAR
PQAA
PQR
TL
Key Performance Indicators
Ministry of Training, Colleges and University
Ontario College Application Service
Ontario College Quality Assurance Service
Prior Learning Assessment and Recognition
Program Quality Assurance Administrator
Program Quality Review
Team Leader
73
APPENDIX A – THE FINAL REPORT
The template for the final report is found online at http://www.algonquincollege.com/acad_dev/MinistryApprove.htm
.
Help text for the main sections of the report – the executive summary, the introduction, the conclusions, and the implementation plan – are provided on the following pages of this
Appendix and in the final report template for ease of reference.
74
Executive Summary
The Executive Summary is a one-page, single-spaced review of the highlights of the Program
Quality Review. The focus of this summary is to highlight the overall status of the program, its special strengths and achievements, and any challenges the program is facing and the strategies recommended to address these challenges. Four to five paragraphs can cover the information needed:
the purpose of the report,
background - a brief history of the program and delivery information,
key findings - strengths and highlights, challenges (SWOT analysis),
a brief summary of the conclusions and recommendations.
Example Executive Summary
(This is fictitious but uses some real situations from recent reports.)
This report is a summary of the findings of the Program Quality Review Process for the 2008-200 academic year for the Housemaking program offered at Algonquin College.
The Housemaking Program was first offered at Algonquin College in 1981 as a one-year certificate. In
1992, the program was converted to a two year diploma program to meet the demands for more highly skilled and knowledgeable workers, and in 1997, the delivery mode was compressed into a non-semester diploma program format. The program prepares graduates to enter the housebuilding industry in the
Ottawa and surrounding area. The program is also offered at the Perth Campus.
There has been a consistently high demand for the program and its graduates except for a minor slump in the housing depression in the early 90s. There are a number of local builders who have supported the program over the last 30 years and our graduate employment rate is consistently above 90%. The KPI and Course Assessment Surveys note high satisfaction rates with respect to the dedication and expertise of the program faculty. This commitment from both full-time and part-time faculty members has ensured the program has maintained a high standard of delivery.
Two major concerns will be addressed in the implementation plan. The first is the need for the students to be ready to enter the job market in the spring rather than the fall. The faculty will review the delivery mode over the next year and determine ways to deliver the curriculum to meet the demands of the students to move quickly through the program yet be available at the start of the construction season.
The other most significant concern is the expectation to lose the coordinator and the other most senior full-time professor in the next year as both of these faculty members are eligible to retire. Succession planning will be a high priority to ensure we have excellent teaching and curriculum development skills in the department along with our current practical expertise. A review of course outlines indicates that some outlines are not as complete as would be desired. A plan is in place to ensure this will be resolved in the
May-June 2014 planning session.
This program has enjoyed a high demand, and high employment rates. Industry projections in the Ottawa area suggest that this demand will continue. Advice from our industry partners on an annual basis will assist us in keeping this program strong over the next few years.
75
Introduction
The introduction ‘sets the tone’ for the entire PQR report. It introduces the program as it stands at the start of the PQR review period, and includes
the purpose of the program – why does it exist, who does it serve in terms of the student body and the future employers,
the programs’ evolution from its first inception to today,
relevant external accreditation organizations linked to the program, if applicable,
number of faculty, both full-time and part-time,
the nature of the student body,
special contributions that the program offers in the community,
variations in delivery like co-op options, campus options, Winter intakes, multiple intakes etc.,
a discussion of the use of classroom, labs, computer labs, placement opportunities, field trips, etc.
If the program has previously gone through PQR, re-use material as appropriate within the introduction, and note the key recommendations and progress against them as applicable. If PQR actions remain open, they should be considered for current applicability, and carried forward as appropriate.
This section does not include any information accumulated during the current PQR.
76
Findings
EVALUATION OF THE PROGRAM AGAINST THE CRITERIA
Program Quality Review Criteria
The findings section is where each of the elements of the evaluation criteria are responded to by Yes/No or occasionally N/A.
Each element of the criteria requires a justification, i.e. explain why the response is
Yes, or No or N/A.
Previous PQR reports within the department can be used for reference when writing.
Also, the PQAA can provide guidance on how to respond to each of the elements of the criteria.
To avoid repetition, the evaluation criteria and their elements are not listed here.
77
Conclusions
This part of the report addresses the analysis of the data, documents, and input from the focus groups.
Discuss the review of the program data and the identification of agenda items for the focus groups.
A brief paragraph can be dedicated to highlights of each of the three focus groups.
The last half or third of this section speaks to the evaluation of the various recommendations from each of the focus groups and the decisions taken for the formulation of the implementation plan. For example, students may suggest that the computer applications course is irrelevant and they do not like the hybrid courses. The external stakeholders provide feedback that the graduates are not computer literate. The final recommendation would probably indicate the need to make the computer applications course relevant to the industry by adding inventory control and invoicing software exercises and communication to the students that computer skills are desirable in the workplace. To address the students’ dislike of hybrid courses, a recommendation could identify the development of better orientation to Blackboard for both faculty and students. (Reminder: it is important to provide feedback to focus group participants regarding resolutions of issues raised at the focus group.) Program strengths, weaknesses, opportunities and threat can be noted.
Some recommendations may arise that will not be feasible to implement and therefore will not be actioned in the final recommendations. These need to be discussed here. For example, one group suggested that students have a university level course as a prerequisite for admission to the program, as the students need a strong science background. Given that this is not allowable by Ministry standards, and is not feasible to implement, the recommendation might be modified to ensure that highest Grade 12
College course is listed as a required course, or if this is already the case, the recommendation may not warrant any action. If the students have the high school subject at the grade twelve level, there may need to be a college level course in level 01 to help the students with the material. This should be explained here, i.e., recommendations from the Focus Groups should not just disappear without explanation and feedback.
Some recommendations will be desirable to implement but may have budget constraints.
These recommendations can be maintained but identified as having fiscal constraints.
Sometimes recommendations may be carried forward until such time that the funds are available or it is determined that a less costly solution is needed and implemented.
Note: Recommendations are identified under the relevant pillar of the College’s Strategic
Plan .
78
Recommendations
Recommendations resulting from the PQR are listed, even if they are not able to be implemented, and the pillar(s) of the Strategic Plan under which they fall is noted.
The four pillars of the Strategic Plan and their associated goals are:
Applied Education and Training o Goal 1: Deliver an exemplary applied education and training experience. o Goal 2: Create a unique suite of programs, products and services geared to meet the needs and expectations of our clients and students. o Goal 3: Leverage technology to enhance the educational experience. o Goal 4: Provide opportunities for every full-time student to have a work experience outside of the classroom.
Student and Client Success o Goal 5: Deliver exceptional service to our diverse student and client populations. o Goal 6: Leverage technology to automate and modernize our business processes, fostering an environment of continuous improvement.
Empowered People o Goal 7: Attract, develop and retain employees who have the knowledge and skills to be fully contributing members of the College. o Goal 8: Create and foster an environment in which the College’s model of leadership competencies and behaviours is supported.
Financial Sustainability o Goal 9: Align our funded operational expenditures with provincial funding.
o Goal 10: Expand non-funded opportunities to increase revenue.
o Goal 11: Leverage strategic business partnerships to meet the capital needs of the College.
o Goal 12: Create the technological foundation to align with the digital direction.
79
Implementation Plan
Actions arising from the stated recommendations are captured, and must be resourced and measureable. Any recommendations that are not able to be implemented at this time should be noted.
The PQR implementation plan is carried forward as a ‘living document’ to be used as part of the ongoing operations of the program, and is updated at least annually.
Recommendation 1: <State the overall recommendation here>
Links to Strategic Plan Pillar(s): (one or more references here…)
Action 1.1
Resource(s)
<List an action which contributes to the fulfillment of the recommendation here>
<State who will prime the activity, for example, “Chair with coorindator”, “coordinator with faculty”>
<State when the action will be completed> Timing
Action 1.2
Resource(s)
Timing
Recommendation 2:
Links to Strategic Plan Pillar(s): (one or more references here…)
Action 2.1
Resource(s)
Timing
Action 2.2
Resource(s)
Timing
Action 2.3
Resource(s)
Timing
Recommendation 3:
Links to Strategic Plan Pillar(s): (one or more references here…)
(E.g.) Not able to be actioned currently.
Recommendation 4:
Links to Strategic Plan Pillar(s): (one or more references here…)
Action 4.1
Resource(s)
Timing
< …extend table as applicable…>
80
APPENDIX B – THE CREDENTIALS FRAMEWORK
Extracted from the “Minister’s Binding Policy Directive “ Framework for Programs of Instruction ” found at: http://www3.algonquincollege.com/program-development/files/2013/03/Framework-
ProgramsOfInstructionJuly2009.pdf
81
83
84
85
86
87
88
APPENDIX C – TEMPLATES
The following templates and review aids are available to download from the PQR website at: http://www.algonquincollege.com/academic-development/program-quality-review-ontariocollege-credentials/ and are included in the following pages:
Introductory Memo from Team Leader
Faculty Education, Experience and PD Survey
Faculty Education, Experience and PD Summary
Course Outline Check Up Review Tool
Course Outline Review Summary Sheet
Curriculum Mapping – Terms and Concepts
Analysis of VLO maps (feedback form)
Analysis of EES maps (feedback form)
General Education Feedback Form
Sample Faculty Focus Group Agenda
Sample Invitation to External Stakeholders
Sample External Stakeholders Focus Group Agenda
Sample Student Focus Group Invite
Sample Student Focus Group Agenda
Student Focus Group Purpose and Objectives
Sample Thank You Letter to Focus Group participants
PQR Approval (Submission Sign-off) form
The blank PQR final report template is also available to download at the above URL.
Introductory Memo from the Chair
Memo
To:
Faculty members of <program title here>
From: <Chair’s name here>, Chair, <department name here>,
CC:
Date:
<Team Leader ’s name here>, Team Leader for 2014-2015 Program Quality
Review
Re: Program Quality Review
The <program title here> program is part of the College’s cyclical Program Quality Review
(PQR) process for 2014-2015, and <Team Leader name here> has been assigned the task of reviewing the program against Ministry and College quality criteria.
Part of the process is to determine whether the program is appropriately staffed to ensure the delivery of a quality program. To that end, please complete the attached Faculty
Education, Experience, and Professional Development Survey and submit it to me by <date>.
I will summarize the information and submit cumulative totals for the final PQR report. If anyone has any concerns about how this information will be conveyed, please feel free to discuss them with me.
You will also be receiving documents from <Team Leader name here> that will assist in reviewing the curriculum. Curriculum review is an important part of your work each and every year, and this is an opportunity for us to review the cumulative effects of changes made to the program’s curriculum over the last five years. We need to ensure that the program is compliant with the three parts of the Ministry Program Standards: Vocational
Outcomes, Essential Employability Skills, and General Education. As well, we need to ensure the congruency between the identification of course learning outcomes, learning activities, and assessment activities.
You will also receive a copy of the Program Self Audit document that lists the evaluation criteria used to assess the program. Please do consider the Program Self Audit document carefully, and identify any areas of concern that you might have. Take the time to acknowledge the areas of quality in our program, so we can devise strategies to ensure that the quality is enhanced.
Thank you for your support for these PQR activities.
Attachment: Faculty Education, Experience, and Professional Development Survey
90
Introductory Memo from the Team Leader
Memo
To:
Faculty members of <program title here> program
From:
<TL name here>, Team Leader for 2014-2015 Program Quality Review
CC:
Date:
<Chair’s name here>, Chair, <department name>,
Re:
Faculty Focus Group
The <program title here> program is part of the College’s cyclical Program Quality Review (PQR) for
2014-2015. You are invited to a Faculty Meeting on <date here> in room <location here>. Please review and complete the following documents as outlined below. These documents will be discussed at the meeting and used by the me to compile summary reports.
Program Self Audit: Please provide to the Team Leader one week ahead of Faculty Focus group meeting
This document includes Ministry and College criteria against which programs at the College are evaluated. Please read the document carefully and indicate your opinion regarding whether the program meets each of the elements of the criteria. Put a check mark in the appropriate column and include any comments you have. These may be discussed further at the meeting. Issues raised by the faculty members may be brought forward to the student and external stakeholder focus groups for further exploration if necessary.
.
Course outline review document documents : Please review your course(s) and bring reports to the
Faculty Focus group meeting
Each Faculty member is asked to complete the course outline review document for each course that they teach. This will help identify that the course outline meets Ministry criteria and also will provide valuable feedback that will be used the next time the course outline is updated.
If you require any assistance with these documents, please contact me at <email. phone, other
contact details>.
Attachments :
1. Curriculum Mapping Terms and Concepts
2. Course Outline Check-up for Course Writers
3. Program Self Audit Template
91
Faculty Education, Experience, and Professional Development Survey
This survey is to be submitted directly to the Chair of the program who will summarize the data for the final PQR report. This survey is considered confidential information once completed.
Name:
Date:
Program:
Education
Response text here
Number of years relevant experience in the workplace
Response text here
Number of years teaching in this program
1. as a full-time professor Response text here
2. as a part-time professor Response text here
Date of hire: Response text here
Other relevant Credentials
Response text here
Other relevant Experience
Response text here
List any Professional Development activities attended in the last 5 years. Include formal courses, workshops, conferences, industry renewal opportunities, sabbatical leaves, etc.
Response text here
Describe any reflective practices used for Professional Development in the last 5 years. (For example, review of Course Assessments each term, peer-to-peer feedback, informal feedback from students, and the associated outcome(s)).
Indicate if not regularly engaged in reflective practice.
Response text here
When was your last performance review? Response text here
Confidential once completed
92
Summary of Faculty Member s’ Credentials and Experience
Program:
Date:
Program Summary Information Count/Notes
Total number of students in the program
Total number of faculty teaching in the program FT:
PT:
Count/Notes Program Faculty Education Summary
Number of Faculty with PhD
Number of Faculty with EdD
Number of Faculty with Master’s Degree
Number of Faculty with Bachelor’s Degree
Number of Faculty with Diploma
Number of Faculty with special credential
(please list the relevant credentials and the number of Faculty with the credential)
Number of Faculty with less than Diploma preparation
Teaching Experience Count/Notes
Total number of years of full-time teaching experience
Total number of years of part-time teaching experience
Special areas of strength of Faculty
Any areas of concern regarding Faculty education and/or experience
Workplace Experience
Number of years of relevant workplace experience combined.
Count/Notes
Number of Faculty with less than Diploma preparation
Professional Development:
Number of Faculty members who have attended professional development within the last two years.
Number of Faculty members participating in reflective practice.
(please list examples of the reflective practices being used )
Performance Reviews
Count/Notes
Number of Faculty with a written performance review done in the last year.
Number of Faculty with written performance review done more than a year ago but within the last two years.
Number of Faculty with written performance review done more than two years ago but within the last three years.
Number of Faculty with no written performance reviews within the last three years.
Count/Notes
FT:
PT:
FT:
PT:
FT:
PT:
FT:
PT:
93
Program Quality Review - Program Self-Audit
Each faculty member completes this program self-audit form and submits it to the Team Leader who summarizes the input.
The self-audit is intended as a quick reference check to capture faculty input regarding the program in order to guide discussions at the focus groups and to continue to evaluate against the quality criteria. Comments can be placed against any quality criteria element, but some are indicated as being covered elsewhere.
Prior to filling this document, it is advised to review the Vocational Learning Outcomes (VLOs) and the Essential Employability Skills (EES) outcomes for the program. These will be supplied by the Team Leader, and are available via COMMS.
Criterion 1. Program learning outcomes are set, are consistent with the College mission and the program’s intended purpose, and are appropriate for the credential offered upon completion of the program.
1.1 Program learning outcomes are consistent with the College mission, appropriate to the level at which the qualification is offered, consistent with the requirements of the
Credentials Framework, and appropriate to the occupational requirements of the program graduates.
Yes No Comments
1.2 Program learning outcomes are consistent with MTCU
Provincial Program Standards where they exist.
Team Leader will assess.
1.3 Program learning outcomes are reflected in course outlines.
Assessed during course review and curriculum mapping.
1.4 Program learning outcomes are used in prior learning and assessment (PLAR).
1.5 Changes to courses and program learning outcomes are introduced on a timely basis and are designed to maintain the relevance of the program.
Assessed during course review and
curriculum mapping.
1.6 The capabilities of program graduates, including knowledge, understanding, skills, and attitudes are consistent with the intended program learning outcomes.
Criterion 2. Admission, credit for prior learning, promotion, graduation, and other related academic policies support program development and student achievement of program learning outcomes.
94
In this section, review the monograph and other program information, and provide your insight into how information is received by the students.
2.1 The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program.
2.2 Students have adequate information to allow them to make informed choices about:
selecting the correct program to meet their career aspirations;
the financial commitment needed;
the workload commitment needed; and
the study options available to them.
2.3 Students know how to get internal and external transfer of academic credits and recognition for prior learning.
Yes No Comments
2.4 Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.
2.5 Students know how they will be evaluated.
2.6 Students indicate the learning requirements are relevant and meaningful.
2.7 Students indicate that assessment methods relate to the learning requirements.
Criterion 3. Programs conform to the Framework for Programs of Instruction and the
Credentials Framework, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance.
Yes No Comments
3.1 The duration and structure of the program are consistent with the program learning outcomes and the credential offered.
3.2 Appropriate credits are allocated for each component of the program, and transfer and laddering options are stated.
Assessed by Team Leader ahead of
Curriculum Review and Mapping.
3.3 Prerequisites do not unnecessarily hinder progress in the
95
program.
3.4 The program title is consistent with college system nomenclature / titling principles.
3.5 The program has established articulation agreements.
3.6 The program conforms to the College policy for the number of English courses
3.7 All curriculum documentation is up-to-date including course outlines and the program monograph information.
Team Leader verifies
Team Leader verifies
Team Leader verifies
3.8 There is congruency between the course learning requirements and the program learning outcomes.
Initial view here. Will draw together at faculty meeting and when reviewing curriculum.
Initial view here. Will draw together at faculty meeting and when reviewing curriculum.
3.9 There is a match between course learning requirements, course learning activities and learning resources.
Initial view here. Will draw together at faculty meeting and when reviewing curriculum.
3.10 Concepts of social, economic and environmental sustainability are embedded in the program curriculum.
Please give examples.
3.11 Work Integrated Learning, such as co-operative work placement, clinical/field placement, service learning and/or participating in applied research projects, is embedded in the program curriculum.
3.12 Students have opportunities to develop the skills, knowledge and attitudes necessary to succeed in a global economy.
Please give examples.
Please give examples.
Criterion 4. Methods of program delivery and student evaluation are consistent with the program learning outcomes.
Yes No Comments
96
4.1 Program delivery, including that which takes place offsite, is consistent with the nature of the program, the learning outcomes, and the needs of the students.
4.2 There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities.
4.3 Learning methods are published learning outcomes.
and are matched to the
Please give examples.
4.4 College designated targets regarding hybrid courses are met.
4.5 Learners are provided the skills necessary to be successful with the learning strategies selected.
Team Leader to verify.
Initial view here. Will draw together at faculty meeting and when reviewing curriculum.
4.6 Evaluation criteria are published and students are aware of how and when they are going to be evaluated.
Initial view here. Will draw together at faculty meeting and when reviewing curriculum.
4.7 There is a match between course learning requirements and evaluation methods, i.e., evaluation methods allow students to demonstrate the course learning requirements
4.8 Evaluation methods are valid and reliable.
4.9 Students indicate that feedback is timely and allows them
Please give examples.
Initial view here. Will draw together at faculty meeting and when reviewing curriculum.
Initial view here. Will draw together at faculty meeting and when reviewing curriculum.
Initial view here. Will draw together at faculty meeting and when reviewing curriculum.
97
to build on their learning.
4.10 Students perceive evaluation to be fair.
4.11 Practices for resubmissions, supplementals, and appeals are published, and are appropriate, fair, valid, and implemented consistently.
4.12 Student workload and assessment is balanced across the term at both the course and program level.
4.13 There is a range of evaluation methods used consistent with a variety of learning styles.
4.14 Learners can earn credit for up to 75% of the program hours using the PLAR process.
4.15 Academic policies and practices that provide for the development and continuous improvement of teaching and learning methods are valued, documented, and supported.
Criterion 5. Human, physical, financial, and support resources to support student achievement of program learning outcomes are available and accessible.
Team Leader to verify.
Yes No Comments
5.1 Program faculty members, as a whole have adequate academic preparation and workplace experience to deliver a quality program.
Chair will provide Team Leader with the information.
5.2 Program faculty members are evaluated every three years. Chair will provide Team Leader with the information.
5.3 Program faculty members engage in professional development activities that ensure they are current in their field and developing teaching expertise.
Chair will provide Team Leader with the information.
5.4 Program faculty members work within clear and wellstructured instructional plans.
98
5.5 Program faculty members participate in reflective practice. Chair will provide Team Leader with the information.
5.6 Students consider faculty to be available. Give your impression of student’s view.
5.7 Students consider faculty to be adequately prepared for class.
5.8 Students consider faculty to promote a positive attitude to learning.
5.9 All students are assigned an academic advisor.
5.10 Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.
5.11 Labs, clinical facilities and placement facilities are complementary to and integrated into the program and allow the learner to demonstrate the learning outcomes.
5.12 Students indicate that there are adequate and accessible learning resource materials including:
textbooks in the bookstore; online materials;
print resources;
equipment and student support services, to allow them to be successful.
5.13 The program is financially viable.
Give your impression of student’s view.
Give your impression of student’s view.
Give your impression of student’s view.
Chair and Team Leader verify.
5.14 The demand for the program has been sustained for the last five years.
5.15 There is a future demand for graduates of this program.
5.16 The learning environment is safe.
Chair and Team Leader verify.
5.17 The students are provided with the information they need to know to function safely in both the College and workplace learning environments.
99
5.18
Program materials, including the program monograph, course outlines and course materials, are provided in an accessible format.
Criterion 6. Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.
6.1 Students indicate that they are satisfied with the program.
Yes No Comments
6.2 Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the Council.
6.3 Learners progress through the program , achieve program outcomes and graduate in a timely fashion.
Chair and Team Leader verify.
Chair and Team Leader verify.
6.4 Learners with a wide range of abilities demonstrate the expected learning outcomes.
6.5 Items raised by the Program Advisory Committee are considered in a timely fashion and feedback is provided to the committee.
Chair and Team Leader verify.
6.6 Graduates are satisfied with the overall program experience.
6.7 Graduates are obtaining employment in their fields.
6.8 Graduates are successful in obtaining external licenses or credentials where relevant.
6.9 Employers are satisfied with graduate performance.
100
Course Outline Check-Up Review Tool
Course outlines document the essence of a given course for an academic year. All parts of a course outline contribute to the definition of a course. This Course Outline Check-up provides a guide for reviewing an outline with a view to strengthening and improving the outline because it is the foundation for the design, modification, and updating of delivery plans.
Once the S ELF -A SSESSMENT portion is complete, the outline and the Check-up can be given to a peer for a second look. The P EER R EVIEW element emerges from the realization that our outlines regularly find themselves in the hands of third parties who need to be able to answer questions about curriculum coverage and equivalency.
Self-Assessment
1. The Course Description is written from the perspective of the student/learner. 3 2 1
2. The Course Description refers to the content, and the activities and evaluations.
3. The VLOs and EES apply to this course in terms of both content and evaluations.
3 2 1
3 2 1
4. All CLRs complete the phrase "… you will have demonstrated the ability to".
5. All CLRs are written with active verbs that lead to clear assessments of performance.
3 2 1
3 2 1
6. All EKS provide clarification and elaboration of the associated CLRs.
7. All EKS are a continuation of the phrase "… you will have demonstrated the ability to".
8. The Learning Resources apply to all sections of the course that are offered.
9. The Learning Activities are related to the CLRs and EKS and independent of delivery mode.
3 2 1
3 2 1
3 2 1
3 2 1
10. The evidence for Evaluation/Earning Credit lists evaluation components individually. 3 2 1
11. The evidence for Evaluation/Earning Credit outlines the nature of the evaluation component. 3 2 1
12. The evidence for Evaluation/Earning Credit is independent of delivery mode.
13. The validation section in Evaluation/Earning Credit has a manageable number of choices.
Evaluation/Earning Credit lists performances that can be assessed
3 2 1
3 2 1
3 2 1
15. The PLAR section is complete and provides a clear indication of the requirements.
Peer Review
16. Reading the Course Description , a clear picture of the course comes to mind.
17. Given the Course Hours, the mapped VLOs and EES seem reasonable.
18. Reading the CLRs and EKS , a picture of the teaching/learning comes to mind.
19. The evidence for Evaluation/Earning Credit aligns with the performances in the CLRs.
20. The validation section in Evaluation/Earning Credit is reasonable given the weighting.
3 2 1
3 2 1
3 2 1
3 2 1
3 2 1
3 2 1
A) Calculate the total for each column.
B) Add the totals together.
This is your result.
Check-up Results
Over 40 — Keeping up with the annual curriculum updates should ensure this outline stays in good order.
25 – 40 — Reviewing Lifesaver #3: Developing Course Outlines should help the areas that "Need Work".
Under 25 — An appointment with a Curriculum Consultant, or a PD session on Course Outlines might be in order.
101
102
Course Outline Review Summary Sheet
Program Name: Date:
1.
There is congruency between the course learning requirements, and the program learning outcomes.
<Relates to Quality Criteria element #3.8>
2.
There is a match between course learning requirements, course learning activities and learning resources.
<Relates to Quality Criteria element #3.9>
3.
Learning activities are published and are matched to the learning requirements.
<Relates to
Quality Criteria element #4.3>
4.
Evaluation methods allow students to demonstrate the course learning requirements.
<Relates to Quality Criteria element #4.7>
5.
PLAR opportunities exist and are based on course learning requirements.
<Relates to
Quality Criteria element #1.4>
Instructions
:
Please enter each course name and number in the Program of Study as presented in the
Program Monograph (level by level). Add table rows as required. Do not include General
Education electives – these are reviewed separately.
The numbers in the table refer to the items in the numbered list above. Answer Y (yes) or N
(no) or N/A (Not Applicable) against each numbered item.
Comments are collated from the VLO, EES and General Education reviews, and general faculty review of the curriculum. For item 5, if PLAR is N/A for the course, please state why.
This summary may be used when next updating course outlines.
Course Name and Number
1 2 3 4 5
Comments – list strengths and recommendations for change/update, and if
PLAR is N/A, please indicate why.
103
Curriculum Mapping - Terms and Concepts
T
A
C each ssess ulminating P erformance
Do you provide instruction/learning opportunities in this skill in your course?
Is this skill identified in one or more course learning requirements?
Do you devote a significant amount of time to facilitating student development of the skills and knowledge embedded in the outcome?
Are there assignments, tests or projects which are designed to allow you to evaluate or assess student performance of this outcome or some of its elements?
Do you, in your evaluation of student performance, verify that this particular outcome (or a significant component of it) has been achieved?
Is this outcome reflected in your course outline in the course learning requirements and/or embedded knowledge and skills, and in the evaluation of the course outline?
Culminating performances are tasks or activities designed to assess a learner’s ability to demonstrate one or more learning outcomes in their totality. While they do not necessarily occur at the end of a program of study, they do evaluate whether a learner is able to integrate and apply their learning to demonstrate the performance described in the learning outcome(s) at the exit level.
Is there an opportunity for you to evaluate the outcome in its totality?
Does the evaluation result in a final product or performance which allows you to determine whether the learner has integrated the knowledge and skills identified in the elements of performance?
Can you determine from this performance if the learner has demonstrated the outcome?
Program culminating performances should require learners to demonstrate learning similar to what would be expected of new graduates in as close to a “real world” context as possible.
If a culminating performance is assigned to a course, the professor teaching the course will be asked to evaluate whether or not the students successfully demonstrated the outcome(s) the culminating performance addresses
104
Analysis and Review of VLO Maps for PQR
Program Name – Program Number
This sheet has been prepared to facilitate the analysis and review of the VLO maps used in the curriculum review process. The items mentioned in the C OMMENTS section should be reviewed and considered for inclusion in the recommendations and action plan that are part of the PQR report.
There are four steps in this review process:
Step 1: Review the program maps, and answer questions A through E.
Step 2: Review the course outlines, provide recommendations for consideration related to specific course outlines, and answer questions F and G.
Step 3: Complete summary section based on review findings.
Step 4: Attach a copy of VLO program maps that were used for the review.
SUMMARY
Are the VLOs as mapped generally supported by the information in the course outlines?
YES NO
Unable to
Answer
What key changes are required as a result of this review?
A.
Are there instances of teaching (T), assessment (A), and culminating performances (CP) for each of the VLOs?
YES NO
Unable to
Answer
Comments:
B.
Does the program meet the minimum requirement of two culminating performances (CP) for each of the VLOs?
YES NO
Unable to
Answer
Comments:
C.
Does the program map show any instances of assessment (A) with no previous or associated teaching (T) for any of the VLOs?
YES NO
Unable to
Answer
Comments:
D.
Are culminating performances (CP) found in courses that are likely to support the demonstration of the complete Vocational Learning outcome at the level expected of a graduate?
Comments:
E.
Does the program map show a reasonable balance of teaching (T) and assessment (A) of VLOs in the program?
YES NO
YES NO
Unable to
Answer
Unable to
Answer
Comments:
105
General Recommendations for program consideration:
F.
Do the course learning requirements, learning activities and/or assessment strategies in the course outlines reflect the VLOs that are attached to the course? (i.e., Is it possible to visualize the performances based on the content of the course outlines?)
YES NO
Unable to
Answer
Comments:
G.
What trends or general concerns are evident in the course outlines for the program as a whole?
Recommendations for consideration related to specific course outlines:
LEVEL 1
LEVEL 2
<add levels as required>
Program Maps
The program maps used for this review were generated on DD-MM-YYYY, and are appended below.
106
Analysis and Review of EES Maps for PQR
Program Name – Program Number
This sheet has been prepared to facilitate the analysis and review of the EES maps used in the curriculum review process. The items mentioned in the C OMMENTS section should be reviewed and considered for inclusion in the recommendations and action plan that are part of the PQR report.
SUMMARY
Are the EESs as mapped generally supported by the information in the course outlines?
YES NO
Unable to
Answer
What key changes are required as a result of this review?
A.
Are there instances of teaching (T), assessment (A), and culminating performances (CP) for each of the EES outcomes?
YES NO
Unable to
Answer
Comments:
B.
Does the program meet the minimum requirement of two culminating performances (CP) for each of the EES outcomes?
YES NO
Unable to
Answer
Comments:
C.
Does the program map show any instances of assessment (A) with no previous or associated teaching (T) for any of the EES outcomes?
YES NO
Unable to
Answer
Comments:
D.
Are culminating performances (CP) found in courses that are likely to support the demonstration of the complete EES outcome at the level expected of a graduate?
Comments:
E.
Does the program map show a reasonable balance of teaching (T) and assessment (A) of EES outcomes in the program?
YES NO
YES NO
Unable to
Answer
Unable to
Answer
Comments:
General Recommendations for program consideration:
F.
Do the course learning requirements, learning activities and/or assessment strategies in the course outlines reflect the EES outcomes that are attached to the course? (i.e., Is it possible to visualize the performances based on the content of the course outlines?)
YES NO
Unable to
Answer
107
Comments:
G.
What trends or general concerns are evident in the course outlines for the program as a whole?
Recommendations for consideration related to specific course outlines:
LEVEL 1
LEVEL 2
<add levels as required>
Program Maps
The program maps used for this review were generated on DD-MM-YYYY, and are appended below.
108
Analysis and Review of General Education Maps and Courses for PQR 13
13 This is the standard form for General Education reporting. The Committee may choose to provide an alternate format.
109
110
Faculty Focus Group – Sample Agenda
SAMPLE AGENDA
Program Name:
Date:
Location:
Introductions
Objectives of the session
Process/ground rules/groups
Tasks/activities o Review content of Program Monograph/Program Promotional Materials o Review program outcomes o Review program of study – including sequencing, orientation to skills o Review Mapping of Vocational Learning Outcomes and Essential Employability
Skills report (if available) o Review of Course Outlines and Syllabus’ o Identification of issues arising from Program Self-Audit o Identification of agenda items for student and external stakeholder focus groups o Identification of recommendations for consideration
Adjournment
111
Sample Invitation to External Stakeholders
Address
Date
Dear
<invitee name here>
As part of Quality Assurance at Algonquin College all programs participate in a cyclical
Program Quality Review process. This year the
<program title here>
program is undergoing review. The College is seeking your participation in the External
Stakeholder Focus Group session which will be held on
<date, time, and location here
>.
Other participants will include members of the Program Advisory Committee, recent graduates, and employers.
During the review process the College solicits perceptions of the program held by various stakeholders who interact regularly with program faculty, students and graduates. A list of issues and topics which may be addressed is also included
< or will be sent at a later date
,
adjust as applicable>.
The Program Quality Review process seeks input from as broad a base as possible and from a wide range of perspectives held by various sectors linked to the program. The data collected will contribute valuable information and guidance to ensure that the program remains current and meets the changing needs of our students and the workplace.
I have enclosed a draft agenda for the focus group session. I anticipate that the focus group session will require two hours of your time, not including travel. <
You may wish to include directions about parking.>
To ensure full participation for this special meeting, we ask that you respond to this invitation on or before
<RSVP date here>,
to allow us to invite an alternate in the event you are unable to attend.
The format for the Focus Group will be discussion-based and the session will be facilitated by one of the College’s Program Quality Review Team Leaders. Your perceptions are important to the program and we hope that you will be able to attend.
Thank you for your ongoing support of our program.
Sincerely,
Chair
112
External Stakeholder Focus Group
SAMPLE AGENDA
Program Name:
Date:
Location:
Introductions
Objectives of the session
Process/ground rules
Program Overview (Chair) o Events/Highlights/Issues o Environmental Influences
Discussion Items o Review of Program Monograph o Perceived strengths and weaknesses of the program (based on the responses to quality criteria) o Present and anticipated trends in the vocation/workplace o Review of vocational learning outcomes o Actions needed to address future trends.
Recommendations arising
113
Student Focus Group – Sample Invite
SAMPLE INVITE
Dear <Program Name> Students;
Algonquin College has several quality assurance processes to ensure that programs remain current and continuously improve. This coming year the <***> Program is engaging in the comprehensive Program Quality Review process. The College is seeking your active participation in the Student Focus Group session which will be held on <**date**> from <**start time**> – <**end time**>, in room <**Room number**>.
The Student Focus Group Session is intended to generate feedback from students about their experience and their perception about the education they are receiving, and to identify factors students feel impact their learning environment. An agenda with the topics to be discussed will be distributed prior to the session for your consideration. The data collected will contribute valuable information and guidance to ensure that the Program remains current and meets the changing needs of our students and the workplace.
I anticipate that the Focus Group session will require two hours of your time. Light refreshments will be provided at the start of the session.
To ensure full participation for this special meeting, we ask that you respond to this invitation on or before <reply date> , to allow us to properly prepare for the session.
Please notify me at <**e-mail**> , of your desire to attend. Space is limited to the first 20 respondents.
The format for the Focus Group will be discussion-based and the session will be facilitated by one of the College’s Program Quality Review Team Leaders. Your perceptions are important to the Program and we hope that you will be able to attend.
Thank you for your ongoing support of the <**Program Name**> Program.
Sincerely,
Program Quality Review Team Leader
114
Student Focus Group - Sample Agenda
SAMPLE AGENDA
Program Name:
Date:
Location:
Facilitator:
Introductions
Program Quality Review, and Objectives of the Student Focus Group
Ground rules of meeting
Round table – student feedback on program quality (questions to be distributed)
Recommendations arising
Adjournment
115
Student Focus Groups – Sample Purpose and Objectives
Purpose and Objectives
Purpose of the Student Focus Group
Focus group sessions have been convened to gather feedback from program clients. The sessions are an important part of the College’s Program Quality Review process.
Objective of the Focus Group Meeting
Focus group sessions are intended to
generate feedback from students about their experience with the College and their perception about the education they are receiving
identify factors students feel impact their learning environment
Format of the Meeting
An active process is used to generate as much feedback about a range of program related issues as possible within the time allotted. A full two hours is assigned to the meeting, which is facilitated by a program quality review Team Leader or designate.
Anticipated Outcomes of the Session
The focus group is expected to provide
feedback about the program as it is currently delivered
a record of the focus group members’ perceptions of the present status of the program, and future trends that may affect the program
recommendations for program updates/revision
116
Thank You Note to Focus Group Participants - Sample
Address
Date
Dear
<participant name here>
Thank you for your recent participation in our focus group meeting, as part of the
Program Quality review for the <program title here> program. Your contribution to our process for improving this program is greatly appreciated.
I have included a copy of the focus group report for your information. This will be included in the final PQR report, and used to help form overall program improvement recommendations.
Thank you for your ongoing support of the program.
Sincerely,
Chair
117
PQR Submission Sign Off Form
Program Quality Review 2014-2015
PQR Report Submission Sign-off Form
Program Name(s)/Code(s):
1.
The Program Quality Assurance Administrator (PQAA) has reviewed the PQR report, and advises the Team Leader and Chair that the reviewed PQR Report is considered final.
______________________ _______________________ ______________________
PQAA (Print Name) PQAA Signature Date
2.
Program Chair forwards the PQR Report electronically to the Dean for review/approval, along with a scanned copy of the PQR Report Submission Sign-off form:
______________________ _______________________ ______________________
Chair (Print Name) Chair Signature Date
3.
Program Dean approves the PQR Report and forwards electronically to the PQAA for printing and distribution along with a scanned copy of the PQR Report Submission Sign-off form:
______________________ _______________________ ______________________
Dean (Print Name) Dean Signature Date
PQR Follow Up Report Due Date
Once the PQAA receives notification from the Program Dean that the PQR report has been approved, the schedule for follow up reports is confirmed with the Chair and Dean and captured below.
Follow up report due: _____________________________________________________
118
APPENDIX D – Program Data
Key Performance Indicators (KPIs)
Student Satisfaction Survey
This survey is administered by an appointed group of administrators to all students in class during a “Reference Week” identified in October or February. All information gathered is handled confidentially. Students are asked to comment on experiences with their Program,
Courses, Teachers and the Facilities/Resources and Services available.
The four KPI Student Satisfaction Survey capstone questions have remained constant for the past 6+ years; the numbering changed in 2014:
KPI Question
Overall, your program is giving you the knowledge and skills that will be useful in your future career
The overall quality of the learning experiences in this program
The overall quality of the services in the college
The overall quality of the facilities/resources in the college
Numbering 2009-2013 Numbering 2014
Q14S Q13S
Q26S
Q45S
Q44S
Q24S
Q39S
Q49S
Graduate Outcomes/Graduate Satisfaction Survey
Approximately six months following graduation an independent consulting firm - Forum
Research – contacts graduates by telephone to conduct this system-wide survey. Graduates are asked to comment on their experiences at the college as well as information on employment and salary levels to determine the success of a program, program development, and provide information to potential students. All information gathered is handled confidentially.
At that time, the graduate is asked for permission to speak to their employer (their front line supervisor to be exact). If the graduate grants the Forum Research interviewer permission to contact their employer then the employer will receive a call within a month or less of contacting the graduate. If the graduate declines permission to the Forum Research interviewer to contact their employer then the employer will not receive a call.
Employer Satisfaction Survey
Employers are asked to comment on the graduate's workplace abilities (based on knowledge and skills acquired from their college program) as well as their perspective on how the workplace is changing. The employer is also given the opportunity to suggest what new skills are expected from today's college graduates. All information gathered is handled confidentially.
119
Algonquin College Performance Measures
Application/Registration Ratio: Number of applications vs. number of registered students (both full-time and part-time) on Day 1 in the intake level.
The formula used to calculate this ratio is: number of applications/number of registered students
The data is taken from the Ontario College Application Services (OCAS) report from
GeneSIS (SRR058) for each semester: Spring, Fall, Winter.
Registered students are students with registration status of 0 or 700 (Registered or
Withdrawn After Registration).
Intake/Projected Enrolment: Percentage of the actual enrolment (full-time only) over the projected enrolment (full-time only) in the intake level.
The formula used to calculate this percentage is: actual enrolment/projected enrolment
The data is taken from the Download Registration Summary Data report from GeneSIS
(SRR025) on audit days (Spring: June 30, Fall: November 01, Winter: March 01)
KPI Graduation Satisfaction: Percentage of students who rated Very Satisfied or
Satisfied for the usefulness of the college education in achieving goals after graduation.
Responses are aggregated from Q34 of the Graduate Satisfaction Survey: How would you rate your satisfaction with the usefulness of your college education in achieving your goals after graduation?
KPI Total Employment: Percentage of students who are employed after graduation from the college education. Responses are aggregated from question Q6 from the KPI
Graduation Satisfaction survey: During the week of **the survey** were you employed or self-employed; employed or self-employed, but looking for another job; not employed but have accepted a job to start shortly; not employed but looking for a job; not employed, but not looking for a job.
KPI Related Employment Rank: Percentage of students who responded Yes or Yes, partially to related employment after graduation. Responses are aggregated from question Q20 from the KPI Graduation Satisfaction survey, only for students who are currently employed: Was this job related to the program that you graduated from?
KPI Employer Satisfaction: Percentage of employers who answered Very Satisfied or
Satisfied for the graduate’s overall college preparation. Responses are aggregated from question Q74 from the KPI Employer Satisfaction survey: In general, how would you rate your satisfaction with this employee’s overall college preparation for the type of work he/she was doing.
120
Algonquin College Quality Measures
KPI Student Satisfaction : Percentage of students who rated Very Satisfied or Satisfied for the overall program/college experience. This measure is the average of responses from the four capstone questions of the Student Satisfaction Survey: o Overall, your program is giving you the knowledge and skills that will be useful in your future career o The overall quality of the learning experiences in this program o The overall quality of the facilities/resources in the college o The overall quality of the services in the college.
Instructional Quality (This measure was used in Program Mix Review, and is not currently provided for PQR purposes. The definition is retained for reference):
Students who rated the overall quality of the course Excellent or Very Good. Aggregated from the capstone question in the Course Assessment surveys conducted during the fiscal year (i.e. Spring, Fall, and Winter terms in that order):
Overall, please rate the quality of this course.
Note that not every course within the program may be assessed each term, therefore this measure shows the summative rate for the courses in the program, extrapolated to the program as a whole.
Quality Average: Average of Student Satisfaction and Instructional Quality.
Additional data:
School Average: Average of all the values explained previously, rolled-up by school.
Benchmark: based on the values for the 8th position college in Ontario from the overall KPI survey.
121
APPENDIX E – COURSE ASSESSMENT SURVEY QUESTIONS
The course assessment survey, also known as Student Course Feedback survey, is conducted at a course section level. For PQR purposes, the survey information is aggregated to a program level. Prior to summer 2009, the survey was paper-based and consisted of 19 questions. An electronic survey of 23 questions was used from 2009-2011, and shorter survey of just 9 questions implemented in 2011-2012 with further adjustments annually. The questions used in each survey are presented below for reference.
Paper-based Survey – 2009 and previous
Questions:
# Description
1 The professor demonstrates a good knowledge of the subject area
2 The professor helps me to understand and apply information
3 The professor provides feedback that helps me to improve my performance
4 The professor encourages me to actively participate in this course
5 The professor communicates clearly
6 The professor relates to students in ways which promote mutual respect
7 The professor evaluates my performance fairly
8 The professor is available for consultation.
9 The learning requirements for this course are clearly stated so that I know what is expected of me
10 This course contributes to the program in which I am enrolled
11 This course is well organized
12 The learning experiences are related to the learning requirements of this course
13 The required support materials are useful
14 The methods used to evaluate my performance are clearly outlined in writing
15
Tests, assignments, projects and/or other expectations relate well to the learning requirements of this course
16 The workload in this course is appropriate
17 Overall, please rate the quality of this course
18 There is adequate access to appropriate equipment in good working order for this course
19 The physical environment in which this course is conducted supports my learning
Response scale:
# Description
1 Strongly Agree / Excellent
2 Agree / Very Good
3 Undecided / Good
4 Disagree / Fair
5 Strongly Disagree / Not Satisfactory
6 Does Not Apply / No Opinion
122
Electronic Survey – 2009-2011
Questions:
# Description
1
This course integrates educational technologies (e.g. online learning tools e-classroom audio-visual equipment etc) in support of my learning.
2 Course learning activities (e.g. lectures discussions practical work group work etc) are varied.
3
The course learning requirements for this course are clearly stated so that I know what to do to be successful in this course.
4 The required course materials (e.g. textbooks manuals software etc) are used in the course.
5 Opportunities exist to link the course material to the real world or workplace setting.
6 The methods used to evaluate my performance are clearly outlined in writing.
7 Course learning activities are linked to the course learning requirements.
8 All of the course learning requirements are covered in the course.
9
The professor’ s expectations for this course are clearly stated so that I know what to do to be successful in this course.
10 The methods used to evaluate my performance are linked to the course learning requirements.
11 Overall please rate the quality of this course
12
(The professor…)
Covers all elements of the course outline.
13 Demonstrates a good knowledge of the subject area.
14 Relates to students in ways which promote mutual respect supports student learning and success.
15 Communicates clearly.
16 Helps me understand and apply information.
17 Provides opportunities for me to participate in the course.
18 Provides timely feedback that helps me to improve my performance.
19 Evaluates my performance fairly.
20 Uses class time effectively including starting and ending classes on time.
21 Is prepared and organized.
22 Is available for consultation (e.g. email office hours appointments phone etc).
23 Overall please rate the effectiveness of your course professor .
Response scale:
# Description
1 Strongly Agree / Excellent
2 Agree / Very Good
3 Undecided / Good
4 Disagree / Fair
5 Strongly Disagree / Not Satisfactory
6 Does Not Apply / No Opinion
123
Electronic Survey – 2011-2012
Questions:
#
1 What did you like most about this course?
2 How could the course be improved to be of benefit to future students?
3 Overall, please rate the quality of this course.
4 Course learning activities (e.g., lecture, discussion, practical work, group work etc) are varied.
5 Course learning activities are linked to the course learning requirements.
6 All of the course learning requirements are covered in the course.
7 What did you like the most about the course professor?
8 What, if anything, could the professor do differently to be of benefit to future students?
9 Overall, please rate the effectiveness of your course professor.
Response Type:
# Description
1 Long Answer
2 Long Answer
3 Excellent to Not Satisfactory N/A
4 5-point Likert Scale wN/A
5 5-point Likert Scale wN/A
6 5-point Likert Scale wN/A
7 Long Answer
8 Long Answer
9 Excellent to Not Satisfactory N/A
Scale responses:
1 Strongly Agree / Excellent
2 Agree / Very Good
3 Undecided / Good
4 Disagree / Fair
5 Strongly Disagree / Not Satisfactory
6 Does Not Apply / No Opinion
124
Electronic Survey – 2012-13
Questions:
#
1 What did you like most about this course?
2 How could the course be improved to be of benefit to future students?
3 Overall, please rate the quality of this course.
4 What did you like the most about the professor?
5 What, if anything, could the professor do differently to be of benefit to future students?
6 Overall, please rate the effectiveness of your course professor.
7 What do you feel would be most beneficial to future students of this course if it were improved.
8 Course learning activities (e.g., lectures, discussions, practical work, group work etc) are varied.
9 Course learning activities are linked to the course learning requirements.
10 All of the course learning requirements are covered in the course.
Response Type:
# Description
1 Long Answer
2 Long Answer
3 Excellent to Not Satisfactory N/A
4 Long Answer
5 Long Answer
6 Excellent to Not Satisfactory N/A
7 Long Answer
8 5-point Likert Scale wN/A
9 5-point Likert Scale wN/A
10 5-point Likert Scale wN/A
Scale Responses:
# Description
1 Strongly Agree / Excellent
2 Agree / Very Good
3 Undecided / Good
4 Disagree / Fair
5 Strongly Disagree / Not Satisfactory
6 Does Not Apply / No Opinion
125
Electronic Survey – 2013-14
Questions:
#
1 What did you like most about this course?
2 How could the course be improved to be of benefit to future students?
3 Overall, please rate the quality of this course.
4 Course learning activities (e.g., lectures, discussions, practical work, group work etc) are varied.
5 Course learning activities are linked to the course learning requirements.
6 All of the course learning requirements are covered in the course.
7 What did you like most about the course professor?
8 What, if anything, could the professor do differently to be of benefit to future students?
9 Overall, please rate the effectiveness of your course professor.
Response Type:
# Description
1 Long Answer
2 Long Answer
3 Excellent to Not Satisfactory N/A
4 5-point Likert Scale wN/A
5 5-point Likert Scale wN/A
6 5-point Likert Scale wN/A
7 Long Answer
8 Long Answer
9 Excellent to Not Satisfactory N/A
Scale responses:
1 Strongly Agree / Excellent
2 Agree / Very Good
3 Undecided / Good
4 Disagree / Fair
5 Strongly Disagree / Not Satisfactory
6 Does Not Apply / No Opinion
126
