New Web-Based Course
Evaluation Services Available to
Schools and Departments
Presentation to Faculty Council
November 6, 2009
Course Evaluation at Carolina
• UNC Board of Governors policy:
– Requires only that student evaluations of instruction must be conducted
“…at regular intervals” (at least one semester each year) and “on an
ongoing basis” for each instructional faculty member.
• Carolina does not have a policy mandating the use
of a common method or instrument for student
evaluation of instruction.
– Deans are authorized to determine how their courses will be
evaluated.
– The Provost is committed to providing centralized services to
support the basic course evaluation needs of schools and
departments. However, use of these services – including the
new product to be implemented this year -- is optional.
Methods Currently Used By
Professional Schools
• Some have developed their own course evaluation
processes to meet curriculum, accreditation, or other
local needs. Examples:
– Medicine: Comprehensive system of web-based student
evaluations tailored for each component of the MD curriculum.
– Business: Web-based system developed in-house to
administer common evaluation form.
– Law: Paper survey with open-ended questions distributed in
class.
– Others: Customized solution developed by outside vendor.
Current Services Offered Centrally:
Carolina Course Evaluations
• Standard instrument designed by 1999 Provost’s Task Force on
Student Evaluation of Teaching in response to concerns about the
validity of the old Carolina Course Review for use in tenure and
promotion decisions.
• Instrument contained sets of standard items to provide:
– Instructors with feedback to help improve teaching skills,
– Administrators with data use in personnel decisions,
– Students with information to aid course selection.
• Also intended to include:
– Flexibility to allow departments, schools, and instructors to choose items that would
be diagnostic for improving teaching and/or applicable for certain course settings.
– Separate distribution of results to intended audiences: administrators, instructors,
and students.
– Ongoing evaluation of psychometric properties.
Development of Current Services
• Task Force report endorsed by Faculty Council and Student
Government in 1999
• Additional Faculty Council resolution in 2001 to provide funding for
complete implementation
• Center for Teaching and Learning charged with implementation
• Two Formats:
– Traditional paper (bubble sheet) version administered in class and taken to
Information Technology Services (ITS) for scanning.
– Web-based version developed by ITS in 2006 and piloted with a few schools.
• Feedback collected from instructors, administrators, and students by
Office of Institutional Research & Assessment in 2007-2008
Limitations to Current Paper and
Web-Based Processes
• Many of the flexible features originally intended to allow instructors
and departments to add items for instructional improvement
purposes were not fully implemented in either the paper or webbased system.
– Example: Item bank was available in web-version but it was not possible for
instructors to create their own items.
• One-size-fits-all design – Could not adjust format for multiple
instructors or schedule different evaluation periods to meet needs of
courses with non-traditional session lengths
• Required core questions did not appear to be relevant to distance
education courses
• Specialized reports for instructors, administrators, and students
were difficult to deliver to intended audiences
• Very resource-intensive
• Would require reprogramming to work with PeopleSoft
Search for a New System or
Services
• Course Evaluation Advisory Committee appointed and charged by
Provost to work with Institutional Research & Assessment (OIRA) to
identify a new web-based evaluation system or services from an
outside vendor.
• Reviewed feedback from pilot studies, surveys, and interviews with
campus users to establish requirements.
• Researched commercial products and services, surveyed peer
institutions, investigated open-source solutions (e.g., Sakai) and
future capacity of Blackboard and PeopleSoft to evaluate courses.
• Course Response™ by Digital Measures offered the best
combination of functionality, ease of administration, and cost.
• Very positive feedback from advisory committee and academic unit
representatives attending demonstrations this summer.
Features of Course Response™
• Flexible functions that can be set centrally or controlled at school,
department, or course section level to:
•
•
•
•
•
Customize contents of evaluation instruments; add items supplied by Provost,
dean, chair, or instructor.
Schedule evaluations when needed
Evaluate multiple instructors, TA’s, guest lecturers, etc.
Define who will receive what results (within limits set by University Counsel)
Create custom reports and do ad hoc analysis of results
• A hosted, turn-key solution: Vendor provides all services including
software, helpdesk, and security in IBM-owned and managed data
center.
• Students, faculty, and administrators enter through existing campus
portals and authenticate using Onyen
• Anonymity of student responses; FERPA-compliant
• Quick implementation; can be ready to use with fall 2009 courses
Costs
• Total annual cost to Carolina is based on number of
schools participating and faculty FTE
• Cost per school ranges from $2,500 for < 25 FTE to
$4,500 for 100+ FTE
• Discounts for additional schools and three-year
agreement
• Estimated total annual cost for campus: $35,000 $40,000, to be funded by Provost’s Office
Implementation
• Participants for Fall 2009 include:
–
–
–
–
School of Nursing
School of Education (selected courses)
School of Journalism & Mass Communication
College of Arts & Sciences – Departments of Romance Languages,
Music, Sociology, and others
• Training and support provided for faculty and staff with
eventual goal of self-management by academic units
• January 2010 – OIRA will collect user feedback and work with
schools to plan for spring 2010 term.
For more information, contact Larry Mayes (ldmayes@email.unc.edu) or Lynn
Williford (lynn_williford@unc.edu) in the Office of Institutional Research &
Assessment or call 962-1500.