Sheila D Scutter and Peter D Munn
Division of Health Sciences
University of South Australia
Sheila.scutter@unisa.edu.au
Peter.munn@unisa.edu.au
Abstract: All University graduates receive the Graduate Course Evaluation Questionnaire (GCEQ) in the period following completion of their University studies. The GCEQ collects information on six key areas of a graduate’s experience, these being clear goals and standards, appropriate workload, appropriate assessment, good teaching, generic skills and overall satisfaction. These later GCEQ scores are seen as key performance indicators of the quality of programs provided at Universities. The University of South Australia was concerned about the GCEQ scores of the
Nursing program in the Division of Health Sciences. This paper describes the processes used to review the external nursing program across two campuses, and the changes implemented in response. What became clear from this process was that the perceived views of management for the low GCEQ data may not always be the same as the views expressed by the graduates of the program. This difference in perceptions results in the changes required being different from those initially intended by management.
Keywords: evaluation, graduate course experience questionnaire, teaching,quality
The undergraduate (pre-registration) nursing program at the University of South Australia is offered in internal, external and mixed mode across three campuses of the University; two city based campuses and a regional campus based in Whyalla. Students enter the program via a variety of pathways, including from year 12, STAT, by completing bridging programs and by a special entry selection test for Aboriginal and
Torres Strait Islander students. In addition, enrolled nurses or those with a TAFE certificate IV gain automatic entry to the program, with one semester of recognition of prior learning (RPL). Many students, especially those studying externally, complete the program on a part time basis.
Each year, graduates from all University programs in Australia receive a Graduate Course Evaluation
Questionnaire that seeks feedback about their program of study. Using 25 Likert scale statements, the GCEQ reports on six key areas of a graduate’s experience, these being clear goals and standards, appropriate workload, appropriate assessment, good teaching, generic skills and overall satisfaction. At the University of
South Australia, scores on the latter three skills scales are used as key performance indicators (KPIs) of perceived teaching quality. The University was concerned about the GCEQ scores of the undergraduate
Nursing program. It was apparent that changes were needed to aspects of the program to improve the graduates’ perception of their experience. Although the GCEQ provides information from graduates about the quality of the program they have completed, the data provided was not sufficient to allow the development of strategies to address areas of concern. In order to make changes to the program to improve the students’ experience, it is necessary to more thoroughly understand the specific areas of concern. Thus a comprehensive review process was undertaken that incorporated a detailed analysis of the GCEQ data and comments, data and comments from an online Student Evaluation Questionnaire, online course evaluation data and focus groups with students and staff. In response to the findings of the evaluative process, recommended strategies to address these were developed.

GCEQ data for the nursing programs across the three campuses were analysed according to whether students studied in external or internal mode. Responses from students studying in mixed mode were limited and were not included in the analysis. All six GCEQ scales were reviewed, although only three of these are used as
KPIs of teaching quality. Students are given the opportunity to comment on the “best aspects” and “areas most in need of improvement” from their course experience in the GCEQ. A thematic analysis of student responses to these questions was undertaken, with responses analysed separately by mode of study.
The University distributes an online Student Evaluation Questionnaire (SEQ) to all currently enrolled students. The SEQ seeks feedback about many aspects of academic life, resources and services. Sixteen
Likert scale items and two open response items concerning course and program quality are included. A similar process of analysis was undertaken with the SEQ data as for the GCEQ data. Thus student responses to the SEQ were compared by mode of study and a thematic analysis of comments in the last two items was undertaken.
Program and course quality assurance processes at the University require that every course is evaluated every time it is offered. Although a variety of evaluation processes can be used, many courses use an online instrument developed by the University, the Course Evaluation Questionnaire (CEI). The CEI contains 10 core Likert-scale questions concerning course quality, as shown in Table 1. Many additional optional items can also be included in the CEI but this analysis concentrated on the 10 items that are used for every course.
I have a clear idea of what is expected of me in this course.
The ways in which I was taught provided me with opportunities to pursue my own learning.
The course enabled me to develop and/or strengthen a number of the Qualities of a University of South Australia
Graduate
I felt there was a genuine interest in my learning needs and progress.
The course developed my understanding of concepts and principles.
The workload for this course was reasonable given my other study commitments.
I have received feedback that is constructive and helpful.
The assessment tasks were related to the Qualities of a
University of South Australia Graduate
The staff teaching in this course showed a genuine interest in their teaching.
Overall I was satisfied with the quality of this course.
Three focus groups were held with students. The first two focus groups included students from interstate, rural and metropolitan South Australia. The third involved two indigenous students and their learning support coordinator.
Eight staff at the City and Regional Campuses took part in the focus group. Summaries of student feedback were provided and discussed during the focus group.

Comparison of the GCEQ mean scores on the three scales used as KPI indicators (Table 2) identifies concerns with the Good Teaching and Overall satisfaction scales in the area of Nursing-initial compared to the Australian average. For calculation of mean scores, agreement with each GCEQ item scores 50, while strong agreement scores 100. Disagreement scores - 50 and strong disagreement scores -100, with a neutral response scoring zero. Thus the average score for all nursing programs is quite low, with UniSA scores even lower. The response rate of 229 is small compared to the total number of students enrolled in the programs and may be a source of bias.
GCEQ percentage agreement scores are available for each item, as shown in Table 3, which also shows
No Good Teaching Overall satisfaction Generic skills
UniSA Aust diff UniSA Aust diff UniSA Aust diff
Nursing
229
(UniSA) 2 12 -10 which items are used for the development of each scale.
15 28 -13 28 32 -4

% agreement
Q3 The teaching staff of this program motivated me to do my best work 27%
32% Q7 The staff put a lot of time into commenting on my work
Q15 The staff made a real effort to understand difficulties I might be having with my work
Q17 The teaching staff normally gave me helpful feedback on how I was going
Q18 My lecturers were extremely good at explaining things
Q20 The teaching staff worked hard to make their subjects interesting
Mean for Good Teaching scale
23%
27%
30%
20%
26%
Q25 Overall I was satisfied with the quality of this program 39%
Q2 The program developed my problem solving skills 45%
Q5 The program sharpened my analytic skills
Q9 The program helped me develop my ability to work as a team member
Q10 As a result of the program, I feel confident about tackling unfamiliar problems
54%
54%
Q11 The program improved my skills in written communication
Q22 The program helped me to develop the ability to plan my own work
Mean for Generic Skills scale
Q1 It was always easy to know the standard of work expected
Q6 I usually had a clear idea of where I was going and what was expected of me
Q13 It was often hard to discover what was expected of me in this program (rev)
Q24 The staff made it clear right from the start what was expected of us
Mean for Clear Goals and Standards scale
Q4 The workload was too heavy (reversed)
Q14 I was generally given enough time to understand the things I had to learn
Q21 There was a lot of pressure on me to do well in this program
(reversed)
Q23 The sheer volume of work in this course meant that it couldn’t all be thoroughly comprehended (reversed)
Mean for Appropriate Workload scale
Q8 To do well in this program all you really needed was a good memory (reversed)
Q12 The staff seemed more interested in testing what I had memorised than what I had understood (rev)
70%
48%
59%
55%
30%
42%
27%
25%
31%
43%
28%
38%
27%
34%
55%
39%
Q19 Too many staff asked me questions just about facts (reversed) 41%
Mean for Appropriate Assessment scale 45%

Student Evaluation Questionnaire
The SEQ is scored so that a strong agreement with an item scores 5, agreement 4, neutral 3, disagreement 2 and strong disagreement 1. As can be seen in Table 4, responses to most statements were quite positive, with a mean score of over 4. Lower scores were received for items relating to the support provided by teaching staff, timeframe for return of assignments, knowing what is expected in assignments and consistency in marking.
Table 4. Responses to SEQ by internal and external mode.
SEQ results
Overall, I am satisfied with my experiences as a student at the 4.0
University of South Australia.
I would recommend my program at the University to a friend, 4.2 colleague, or family member.
Overall, the knowledge and skills gained in my program will 4.0 prepare me well for my future career.
I have found most of my courses to be intellectually 4.2 challenging.
I am provided with the necessary support materials to do my 3.9 study.
The support materials for my program are professionally 4.1 prepared.
The on-line delivery of course components contributes to my 4.0 learning.
I find the teaching staff approachable
When I need to contact teaching staff, they are helpful and 3.5 responsive.
I feel the teaching staff are doing their best to help me 3.1 progress in my program.
Assignments are returned within a reasonable time frame. 3.3
I understand what is expected from me in assignments.
I find there is consistency in marking across all my courses. 2.9
The feedback I receive helps me understand how I can 3.4 improve my work.
Mean
N=123
3.7
3.4
My program director provides useful information and advice. 3.4
Program information is appropriate for helping me decide 3.7 which courses I will take.
Changes to program organisation or administration are 3.5 communicated adequately to me.
Overall I am enjoying my studies. 4.1
Limited student responses to the CEI made reporting and interpretation of the data difficult. Feedback ranged from being very good for some courses to very poor for others. Main areas of concern for these courses related to the questions:

I felt there was a genuine interest in my learning needs and progress.

The workload for this course was reasonable given my other study commitments.

I have received feedback that is constructive and helpful.

As similar themes emerged from the focus groups and from the thematic analysis of the GCEQ and SEQ, this data has been combined to give an overall perspective.
Of interest was the fact that little comment was made by students and staff on the quality of the course materials. Students saw their studies as challenging and quite well organised. Both internal and external students commented favourably on the flexibility of their studies.
Regional students did, however, indicate that they felt ‘second best’ as the materials were developed for those studying through the city campuses and not specifically for them. External students consistently identified themselves as disadvantaged compared with internal students. They considered that they received less value for their HECS fees, and were resentful of this. Regional students also often received course materials late, which made preparation for study and timely completion of assignments difficult.
Students believed that they should receive course outlines already printed out rather than being referred to websites to download their material. The main issue with this was difficulty with internet access; however, students also resented having to print out materials at their own cost.
Indigenous students felt that there was an inadequate explanation of how the material in the packages was to be used.
Staff felt that there were some issues with course material but that this would be addressed in the revised course being implemented in 2004.
Communication between students and academic staff
External students were very concerned about lack of communication with academic staff. Students interpreted this as a lack of interest by the lecturer. External students felt most disadvantaged by this lack of communication compared with internal students, who they perceived could have regular contact with nursing academic staff.
One important area for indigenous students was the need for lecturers to be more aware of how cultural issues may impact on an indigenous student.
Some students were unaware of the organisational arrangements associated with the School. Most students did not know who the course coordinators were, and most had no idea who the Program Director or Head of
School were, or how to contact these people.
Communication between staff and students was an issue of concern for academic staff. Staff were concerned that some students did not access their e-mail and that this caused problems that were often not identified for some time
Staff were supportive of the geographical support groups that had previously been established by the
University but in recent years had been dropped.
One aspect that students were keen to understand early in their studies was the requirements for assignment writing. This was of particular concern for indigenous students. Whilst they recognised that this information was available online most students were keen to have it presented in other mediums such as a preparation video. Students who did not have much knowledge of the internet felt further disadvantaged compared to internal students as most of the material was expected to be accessed via web pages.
The need for support/study groups amongst external students was identified, who often study in isolation.

It was acknowledged by staff that external students pay the same fees as internal students but often consider they do not receive the same level of service. A Learning Advisor is much harder for them to interact with than for internal student when more in-depth work is required than simply identifying learning resources.
Similarly, students who do not have a good understanding of mathematics have difficulty gaining access to people with the knowledge to provide them with the assistance required.
Often students only saw the need for learning assistance after poor results in their first assignments in the various courses. Gaining help from a distance was seen as harder to do than for internal students with easier access to the Learning Advisors.
The external program requires students to attend practical workshops on several occasions during the year.
There was a general feeling that the time at workshops was not always spent in the most productive manner.
This was of concern to those students who had to travel large distances the workshops, sometimes at considerable cost.
In general, clinical placements had been a very rewarding experience for students. There were no negative comments about their clinical placements and across all programs there was a desire for more placement experience.
One aspect that was identified in the staff discussion was the lack of training provided for those involved in external delivery. Participants saw it as the forgotten area of professional staff development in teaching.
Staff considered that students need to understand the online environment and have to be prepared to embrace this learning medium. At the same time (as indicated previously) it was recognised this had costs involved and could disadvantage students in areas of poor connection to the internet.
Nursing staff were conscious of the large percentage placed on the exam mark in the science based courses and were concerned how this impacted on indigenous students.
The length of time allocated to external workshops was discussed. A view held by several participants was that workshops should be longer as at present there was often not enough time to allow for effective interaction with students.
Assignments and Assessment
Concern was expressed about consistency of requirements between lecturers for assignments.
Turnaround times of assignments were problematic and students felt it was critical that they received feedback before submitting their next assignment.
There were also concerns raised about the clarity of assignment instructions. Students suggested that the due dates for assignments need to recognise other demands such as placement requirements.
Academic staff were aware of the importance of having a short turnaround time for assignments, particularly the first assignment in each course. It was also recognised that students needed quality feedback. However, as the assignments become more involved through the Semester it becomes more difficult to staff to turnaround a large number of assignments within a two week time-frame.
Analysis of the GCEQ data of the undergraduate nursing programs offered through UniSA reveal areas of significant concern, particularly with the three scales identified as key performance indicators of good teaching and learning: overall satisfaction, good teaching and generic skills.

When reflecting on the GCEQ scores, it becomes apparent that there are several main issues which need to be considered:
.
The items in the GCEQ are most appropriate for programs which are offered in a “traditional” or
“transmission” mode. The relevance of these items for external programs is not directly apparent. For example, items such as “My lecturers are good at explaining things to me” may be difficult to interpret for students who have undertaken their studies externally. The GCEQ does not include any opportunity for students to provide feedback about professional placements or workshops during their program, and does not allow for them to distinguish between teaching provided on professional placements or in theoretical components of their programs. Whilst acknowledging the limitations of the instrument, the reality is that it is likely to be used for the foreseeable future. Therefore a mechanism is needed that assists students to interpret the items in terms of their educational experience. Possible ways of achieving this are to discuss with students the concepts which the GCEQ is measuring, and how the items may be interpreted, before the students complete the program. As many items on the CEI reflect GCEQ items, this could be achieved through increased use of the CEI and discussion with students about their response to the CEI items and how these might be addressed.
.
The response rate from nursing students GCEQ is poor. This can make interpretation of the data difficult, as it is not possible to know how an increased response rate would affect the scores. Strategies are needed to increase response rate. Suggestions include: a.
A letter from a course coordinator/program director with whom they are familiar advising them that they will receive the GCEQ, explaining the importance of their feedback and requesting their response. b.
Alerting students before they complete their studies that they will receive the GCEQ. c.
Following up students who have not responded to the GCEQ.
.
The GCEQ is sent out to students in May following completion of their studies in November of the previous year. In nursing, most students have commenced working (GDS reveals that 98% of nursing graduates are employed). The generally positive SEQ results in comparison to the GCEQ scores suggest that timing or response rates may influence the GCEQ responses of students in a negative way.
The question arises as to whether student’s perceptions of their program of study may have been influenced by their work experience. The GCEQ for the students completing in 2003 will be distributed in December, which may positively impact on response rates and on responses.
.
There are obvious concerns about the GCEQ scores for the nursing programs. However the data provided gives little direction as to how the program should be improved and what it is that is of most concern to students. The thematic analysis of GCEQ and SEQ scores, and the focus groups with students, revealed that it was not the course materials that were of concern to external students. The main areas that students were concerned about were: a.
Lack of communication between academic staff and students. b.
Consistency of assessment between lecturers c.
Time for return of assignments. d.
External students feel “disadvantaged” compared to internal students e.
Lack of preparation for study

f.
Dissatisfaction with workshops
The Agreed Model for Working with External Students (Appendix), which arose from the focus group with staff and was further worked on at the retreat, addresses and sets guidelines for the first three of these concerns, and to some extent will address the fourth concern, which mainly revolved around access to academic staff to ask for feedback or clarification of study guide content.
Aboriginal students commented in their focus group on the need for better preparation for study, in particular in relation to computer skills and assignment writing. Students in the other focus groups also commented on the need for better preparation for study, including study skills, computer skills and background knowledge.
A Preparation for Biosciences course was run this year and attended by 150 students. This course was aimed at students who had not previously studied biology or who had not studied for several years. It may be necessary to provide support for external students to attend the course, or to offer the course through
Whyalla campus or the South-East, to cater for external students. Alternatively, the course could be developed online for some students.
External students commented that they felt “disconnected” from the campus and were generally not aware of the staff, resources or processes within the School of Nursing and Midwifery or the University generally.
One way of addressing this is to provide students with a CD which contains information about academic and general staff, their role and how to contact them.
Online discussion boards were mentioned by students as being a useful and effective way of communicating with each other and with academic staff. However, discussion boards are currently not used in many courses.
External students were concerned that workshops did not make the best use of their time, and that given the cost of attending the workshops, a longer workshop would be more valuable. There are obvious cost implications for the School in running longer workshops but some review of content and teaching and learning approaches may be warranted.
The wide range of pathways used by students to enter nursing, the high representation of equity groups, the different study modes and loads indicate that the teaching and learning needs of a wide range of students needs to be taken into account when preparing course materials, methods of presentation and assessment tasks. A variety of ways of presenting course material and different assessment tasks to suit the styles of different students is warranted.
As mentioned during the staff focus group, teaching externally required different skills than teaching internally. There is a wide range of experience in the School of Nursing in teaching in the external mode, including online. For some, professional development in external/online teaching would be of assistance.
The assessment in some areas, particularly the science subjects, was of concern to the Aboriginal students.
Assessment in these courses was also perceived as a problem by academic staff and other students and warrants review.
The limited amount of CEI data from previous years, and the limited response rates, makes interpretation of the CEI data difficult. However, it is clear that there are considerable problems with some courses.
Analysis of student comments in the focus groups and in the SEQ and CEQ indicate that students often comment on areas that are not covered in the GCEQ items. For example, most student comments related to timely receipt of external materials, workshops, clinical placements and communication with staff. The
GCEQ does not give graduates the opportunity to provide feedback on these areas, and they are likely to be

reflected most in overall satisfaction scores. Specific feedback about these areas of concern needs to be sought from students, by use of focus groups, discussion boards and additional questions in the CEI.

Appendix.
AGREED MODEL-TEACHING FOR EXTERNAL NURSING STUDENTS
BACHELOR OF NURSING
Course Content

Provide up-to-date study
Communication

Fortnightly communication
Learning Support Availability

Provide students with Harvard guides;

Ensure that course material sent before course commences; with students via e-mail
(follow-up non – respondents to initial email)

Response rate to e-mails and phone calls is 24 hours
(arrangements made if absent)

Contributions made on at least a weekly basis to
Discussion Board reference material;

Develop geographical location maps;

Referral information package for contacts for mathematics, English support
Teaching and Learning

Establish a key points sheet for assignments and standard
 marking proforma;
Establish an external discussion board for shared interaction of students/staff;

Moderation undertaken on assignment marking;

Provision of staff development for external lecturers;

Workload formula –needs to recognise the needs of external students
Assignments and Assessment

Our goal is a 2 week turnaround of the first assignment;

3 weeks for ordinary assignments;

3-4 weeks for greater than
3,000 words

Support for marking assignments with large
 numbers;
Change date of assignments due from Friday to Monday


Feedback on assignments needs to constructive and consistent;
Feedback must be received before next assignment is due.
Administration Support

Assignments need to be logged in
–establish barcode system;

Quality IT support required to assist staff and students;

Succession plan developed
Evaluation

Needs to be considered from early in the course;

CEI needs to be open for a sufficient time period allowing students opportunity to respond;

Staff need to identify how they will respond to feedback of students on course information