Reviewing Course
Evaluation Data:
Perception vs reality
Peter D Munn and
Sheila D Scutter
Introduction
• Graduate Course Evaluation
Questionnaire (GCEQ) provided to all
graduates Australia wide
– Comprises 25 Likert scale statements and
two free response items
– Six key areas of graduates’ experience
•
•
•
•
•
•
Clear goals and standards
Appropriate workload
Appropriate assessment
Good teaching
Generic skills
Overall satisfaction
Introduction
• University of South Australia uses good
teaching, generic skills and overall
satisfaction as key performance
indicators of perceived teaching quality
• Areas of concern with Nursing program
GCEQ scores.
– Assumed to be due to course content
– A comprehensive review process to
understand specific areas of concern in
nursing.
Nursing program
•
•
•
•
•
•
Three year undergraduate program
Offered in 2 city and 1 regional campus
Offered in internal and external mode
Approximately 1000 students
Many part time students
Access via Year 12, STAT, bridging
programs
• Special entry test for ATSI students
• Very high participation of equity groups.
Method
• Graduate Course Evaluation
Questionnaire(GCEQ)
– GCEQ data for the nursing programs
across the city and regional campuses
were analysed by external and internal
mode of study
– Thematic analysis on comments on the
‘best aspects’ and ‘areas most in need of
improvement’ responses analysed by
mode of study
– Individual item scores reviewed
• Student experience questionnaire
(SEQ)
– Distributed online to all currently enrolled
students
– Feedback on aspects of academic life,
resources and services
– Sixteen Likert scale items and two open
response items concerning course and
program quality
– Responses compared by mode of study
– Thematic analysis of comments by mode
of study
• Course Evaluation Instrument
– Each course in the University is evaluated
every time it is offered
– Course Evaluation Questionaire (CEI),
online instrument developed by the
University
– CEI contains 10 core Likert-scale
questions concerning course quality
– Additional items may be added, this
analysis concentrated on the 10 core items
• Focus Groups with Students
Three focus groups were conducted:
– Two with interstate, rural and metropolitan
South Australian students
– One with indigenous students and their
learning support coordinator
• Focus Groups with Staff
– Eight staff representing both campuses
Results
GCEQ Scores, Good Teaching
Results
GCEQ data
• Concerns over good teaching, overall
satisfaction and generic skills;
• Low response rates may be a source of bias
Results
Student Evaluation Questionnaire
• Responses to most statements were positive
with a mean score over 4 (maximum 5);
• Lower scores received for items relating to:
– Support provided by teaching staff;
– Timeframe for return of assignments;
– Knowing what is expected in assignments;
– Consistency in marking.
Results
CEI data
• Limited response by students made
interpretation of data difficult;
• Feedback on courses ranged from very good
to very poor;
• Main areas of concern:
– I felt there was a genuine interest in my learning
needs and progress;
– The workload for this course was reasonable given
my other study commitments;
– I have received feedback that is constructive and
helpful.
Focus Groups: GCEQ/SEQ analysis
• Course Materials and Content
–
–
–
–
–
Little comment
Valued flexibility
Often arrived late
External students felt “second best
Revised courses in 2004 to update content
• Communication between students and
academic staff
– A major concern of external students
– Interpreted as lack of interest by academics
– External students felt disadvantaged compared
to internal students.
– Staff concerns about students accessing email
– Geographical support groups disestablished
• Learning Support Service Availability
–
–
–
–
–
Requirements for assignment writing
Particular concern for indigenous students
Available online but prefer alternative delivery
Need for support/study groups
Difficulty accessing learning advisers.
• Teaching and Learning Issues
–
–
–
–
–
–
Time not spent productively in workshops
Clinical experiences very positive
Staff/student ratio low
Lack of training for staff in external delivery
Student preparation for the online environment
High percentage for exams in science based courses
• Assignments and Assessment
–
–
–
–
Consistency of requirements within courses
Turnaround times major concern
Clarity of expectations.
Large number of students affecting
turnaround
Discussion
• GCEQ scores indicated concerns about overall
satisfaction, good teaching and generic skills
–
–
–
–
The instrument
Response rates
Timing
Interpreting the data
• Students are not given the opportunity to
provide many areas of this feedback in GCEQ.
• Where is was possible to provide comment in
the GCEQ, it was not reflected in item scores.
Discussion
Key concerns of students identified from
thematic analysis and focus groups:
•
•
•
•
•
Timely, consistent and useful feedback;
Communication with academic staff;
Learning support;
Practical workshops/professional placements
Lack of preparation for study
Recommendations
Agreed practice model to be implemented
– Provide up to date study guide
– Early arrival of course materials
– Fortnightly email communication
– Respond to emails and calls in 24 hours
– Contribute to discussion board weekly
– Harvard reference material
– Geographical location maps
– Information packages for Maths and
English support
Agreed Practice model…
Key points sheet for assignments and marking
• Moderation of assignment marking
• Staff development for external mode
• Two week turnaournd for first assignment
• Feedback before next assignment due
• Promote course evaluation
• Respond to student feedback
• Initiatives around GCEQ response rates and
distribution.
• Extension of preparatory courses (Sciences)
• Review nursing workshops
• Review assessment
Conclusion
• GCEQ scores alone do not provide
direction for program improvement.
• Thorough analysis of course and
program feedback is necessary to
understand problems and to develop
appropriate changes to programs.