STUDENT EVALUATION OF TEACHING AND UNITS (SETU) SUMMARY REPORT OF UNIT EVALUATION RESULTS SEMESTER 2, 2011 Mary Tam & Nicola Powell 09 January 2012 INTRODUCTION The Student Evaluation of Teaching and Units (SETU) replaces the previously separated unit evaluation and MonQueST surveys into one instrument. The original unit evaluation questions were included plus four new teaching questions.1 Semester 2 2011 is the second time the survey has been administered across Monash. This report provides a summary of the results for the unit evaluation ‘overall satisfaction’ question. Students were asked to respond to the statement ‘Overall I was satisfied with the quality of this unit’. The response options were ‘Strongly Agree’ (5), ‘Agree’ (4), ‘Neutral’ (3), ‘Disagree’ (1)’ and ‘Strongly Disagree’ (1). From these data a ‘median’2 score has been calculated for each unit offering3. Unit offerings are then classified into one of four groups depending upon this median score: • Outstanding: ≥4.70 • Meeting aspirations: 3.60 – 4.69 • Needing improvement: 3.01 – 3.59 • Needing critical attention: ≤3.00 FINDINGS Unit evaluation overall satisfaction data was collected for 2,904 degree level4 unit offerings in Semester 2 2011. Note that more unit offerings were included in the SETU but only those with data for the overall satisfaction question have been included here. Unit offerings without overall satisfaction data are usually those that were included but had zero or very small enrolment numbers. 1. FACULTY VARIATION As shown in Table 1 and Figure 1, the vast majority of unit offerings (82.5%) were categorised as ‘meeting aspirations’ or ‘outstanding’, with 14.4% in the latter category. The results for the Arts and Law faculties were particularly strong with students rating 23% of the Arts unit offerings and 21.7% of those in Law with a median score of 4.7 or above. While overall satisfaction levels are generally high, 7.6% of unit offerings are in ‘need of critical attention’ after obtaining a score of 3 or less. Education and Pharm & Pharm Sci all had more than 10% of their unit offerings in this category. Table 2 and Figure 2 show the data for those unit offerings with 5 or more responses. The data for unit offerings with less than 5 responses can be seen in Table 3 and Figure 3. 1
Further detail on the SETU can be found at: http://www.opq.monash.edu.au/us/surveys/setu/index.html Further detail on how the median is calculated can be found at: http://www.opq.monash.edu.au/us/surveys/setu/setu_median_calculation.pdf 3
A unit offering is defined here as a unit at a specific location, in a specific mode (eg on campus/off campus) and offered for a specific calendar type. 4
Evaluations were also undertaken for Monash College diploma units and Sth Africa Foundation Program units but these are not included in this report. 2
Office of Planning and Quality – Statistics Unit
Table 1: No. of unit offerings falling into each "traffic light" category by faculty, Semester 2 2011 Owning Faculty Art Des & Architecture Arts Business & Economics Education Engineering Information Technology Law Med Nursing & Health Sci Pharmacy & Pharm Sci Science Grand Total Median Response Band for Overall Satisfaction Item Needing Critical Attention (≤3.0) 11 70 34 21 11 12 7 36 8 11 Needing Improvement (3.01‐3.59) 27
49
60
27
29
20
10
35
9
21
221 287
Meeting Aspirations (3.6‐4.69) 87
570
428
126
134
137
77
256
47
115
Outstanding (≥4.7) 24 206 50 31 12 16 26 41 4 9 Total 149
895
572
205
186
185
120
368
68
156
419 2,904
1,977
Table 2: No. of unit offerings with 5 or more responses falling into each "traffic Light" Category by Faculty, Semester 2 2011 Owning Faculty Art Des & Architecture Arts Business & Economics Education Engineering Information Technology Law Med Nursing & Health Sci Pharmacy & Pharm Sci Science Grand Total Median Response Band for Overall Satisfaction Item Needing Critical Attention (≤3.0) 6 21 17 18 10 5 3 15 6 5 Needing Improvement (3.01‐3.59) 16
34
51
21
29
17
7
24
8
19
106 226
Meeting Aspirations (3.6‐4.69) 68
412
384
112
131
120
59
193
39
103
Outstanding (≥4.7) 17 67 26 18 10 2 16 17 9 Total 107
534
478
169
180
144
85
249
53
136
182 2,135
1,621
Table 3: No. of unit offerings with less than 5 responses falling into each "traffic Light" Category by faculty, Semester 2 2011 Owning Faculty Art Des & Architecture Arts Business & Economics Education Engineering Information Technology Law Med Nursing & Health Sci Pharmacy & Pharm Sci Science Grand Total Median Response Band for Overall Satisfaction Item Needing Critical Attention (≤3.0) 5 49 17 3 1 7 4 21 2 6 Needing Improvement (3.01‐3.59) 11
15
9
6
115 3
3
11
1
2
Meeting Aspirations (3.6‐
4.69) 19
158
44
14
3
17
18
63
8
12
Outstanding (≥4.7) 7 139 24 13 2 14 10 24 4 Total 42
361
94
36
6
41
35
119
15
20
61
356
237 769
Figure 1: Proportion of unit offerings falling into each ‘traffic light category’ by faculty, semester 2 2011 100.0%
90.0%
80.0%
70.0%
% of unit offerings
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Art Des & Business & Arts Architecture Economics (n = 895)
(n = 149)
(n = 572)
Needing Critical Attention
7.4%
Information Education Engineering Technology (n = 205)
(n = 186)
(n = 185)
7.8%
5.9%
10.2%
5.9%
6.5%
Med Pharmacy & Law Nursing & Science Grand Total Pharm Sci Health Sci (n = 120)
(n = 156)
(n = 2904)
(n = 68)
(n = 368)
5.8%
9.8%
11.8%
7.1%
7.6%
Needing Improvement
18.1%
5.5%
10.5%
13.2%
15.6%
10.8%
8.3%
9.5%
13.2%
13.5%
9.9%
Meeting Aspirations
58.4%
63.7%
74.8%
61.5%
72.0%
74.1%
64.2%
69.6%
69.1%
73.7%
68.1%
Outstanding
16.1%
23.0%
8.7%
15.1%
6.5%
8.6%
21.7%
11.1%
5.9%
5.8%
14.4%
Figure 2: Proportion of unit offerings with 5 or more responses falling into each ‘traffic light category’ by faculty, semester 2 2011 100.0%
90.0%
80.0%
70.0%
% of unit offerings
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Art Des & Business & Arts Architecture Economics (n = 534)
(n = 107)
(n = 478)
Needing Critical Attention
5.6%
3.9%
3.6%
10.7%
5.6%
3.5%
Med Pharmacy & Law Nursing & Science Grand Total Pharm Sci Health Sci (n = 85)
(n = 136)
(n = 2135)
(n = 53)
(n = 249)
3.5%
6.0%
11.3%
3.7%
5.0%
Needing Improvement
15.0%
6.4%
10.7%
12.4%
16.1%
11.8%
8.2%
9.6%
15.1%
14.0%
10.6%
Meeting Aspirations
63.6%
77.2%
80.3%
66.3%
72.8%
83.3%
69.4%
77.5%
73.6%
75.7%
75.9%
Outstanding
15.9%
12.5%
5.4%
10.7%
5.6%
1.4%
18.8%
6.8%
0.0%
6.6%
8.5%
Information Education Engineering Technology (n = 169)
(n = 180)
(n = 144)
Figure 3: Proportion of unit offerings with less than 5 responses falling into each ‘traffic light category’ by faculty, semester 2 2011 100.0%
90.0%
80.0%
70.0%
% of unit offerings
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Art Des & Business & Arts Architecture Economics (n = 361)
(n = 42)
(n = 94)
Information Education Engineering Technology (n = 36)
(n = 6)
(n = 41)
Pharmacy & Med Science Grand Total Law Nursing & Pharm Science (n = 20)
Health Sci (n = 769)
(n = 35)
(n = 15)
(n = 119)
Needing Critical Attention
11.9%
13.6%
18.1%
8.3%
16.7%
17.1%
11.4%
17.6%
13.3%
30.0%
15.0%
Needing Improvement
26.2%
4.2%
9.6%
16.7%
0.0%
7.3%
8.6%
9.2%
6.7%
10.0%
7.9%
Meeting Aspirations
45.2%
43.8%
46.8%
38.9%
50.0%
41.5%
51.4%
52.9%
53.3%
60.0%
46.3%
Outstanding
16.7%
38.5%
25.5%
36.1%
33.3%
34.1%
28.6%
20.2%
26.7%
0.0%
30.8%
2. CAMPUS VARIATION Table 4 shows the average of the median overall satisfaction scores obtained by the unit offerings. The data are disaggregated by location and faculty. On average, the unit offerings within each of the faculties fell into the ‘meeting aspirations’ category with the faculty average medians ranging from 3.85 for Pharm & Pharm Sci to 4.17 for Law. The same can be said for the locations ranging from 3.72 at the ‘Other Australian Locations’ to 4.27 at ‘Other Offshore Locations’. There were only two areas in which the unit offerings, on average ‘needed improvement’: • Arts at Peninsula • MNHS at the ‘Other Australian Locations’. There were also two ‘Other Australian Location’ Bus Eco unit offerings that fell into the ‘needing critical attention’ category with a median score of only 2.75. Further investigation showed that these unit offerings only had 5 enrolments, one unit fell into the ‘needing improvement’ category and the other ‘needing critical attention’. The locations with more than 10% of evaluated unit offerings needing critical attention are highlighted in red. These were Gippsland, Parkville, Peninsula and Prato. 3. UNIT MODE VARIATION Table 5 shows the average of the median overall satisfaction scores obtained by the unit offerings disaggregated by faculty and mode. There was only one area in which the unit offering on average ‘needed improvement’ off campus in Science (3.37) and one ‘needing critical attention’ off campus in Art Des & Architecture. Further Investigation of the ‘needing critical attention’ category showed a total of 10 enrolments within the 3 unit offerings, two units fell into the ‘needing improvement’ category and the other ‘needing critical attention’. Units offered off campus (14.5%) were much more likely than those offered on campus (6.4%) to be obtain a score of 3 or below and therefore be classified as ‘needing critical attention’. 4. UNIT LEVEL VARIATION Table 6 shows the average of the median overall satisfaction scores obtained by the unit offerings disaggregated by faculty and unit level. There was only one area in which the unit offerings on average ‘needed improvement’: • Level 4 in Pharmacy and Pharmaceutical Science. There was also one level 5 unit offering in Science which had an average median score in the ‘needing critical attention’. While there does not appear to be a strong relationship between level and the likelihood of a unit ‘needing critical attention’. 10% of the units at level 5 and just over 22% at level L5 were in this category. 5
Level L units = unit level at later year in an UG program. Table 4: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and location, semester 2 2011 Location
Berwi ck
Ca ul fi el d
Art Des & Arch
Arts
4.02
23
Measure
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Cl a yton
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Gi pps l a nd
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
Pa rkvi l l e
4.02
131
7
3.60
18
4
4.03
105
4
4.04
114
11
3.91
116
10
3.88
31
2
4.05
113
14
3.87
67
9
3.84
31
5
3.92
14
3.84
27
3
4.30
61
1
4.14
30
2
Other Aus tra l i a n No. of uni t offeri ngs
Loca ti on
No. needi ng cri ti ca l a ttenti on Other Av. Medi a n
Offs hore No. of uni t offeri ngs
Loca ti on
No. needi ng cri ti ca l a ttenti on
3.94
10
3
4.57
7 2
4.02
677
49
3.76
52
8
4.08
42
2
3.77
28
3.87
47
3
4.22
32
5
3.97
14
1
4.01
63
1
3.80
56
1
3.73
20
3
4.36
6
2.75
2 1 4.34
13 Total
4.06
77
4.06
139
6
4.01
94
5
4.07
1111
73
3.63
31
8
3.72
34
5
3.87
335
48
3.88
3.84
65
56
7
3.83
141
7
3.87
7
1
15
4.21
114
3
3.79
3.94
232
9
3.79
11
3
4.29
3.55
11
3
3.72
51
6
3.84
6 124
12
4.27
26
61
2
4.03
17
3.75
12
1
3.93
28
1
% needing critical attention
2.6%
7.2%
6.6%
14.3%
12.5%
10.6%
2.6%
3.9%
27.3%
9.7%
0.0%
3.97
149
11
Sci
56
7
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
Overa l l Av. Medi a n
Tota l no. of uni t offeri ngs
Tota l no. needi ng cri ti ca l a ttenti on
4.08
81
3
Pharm & Pharm MNHS
Sci
3.88
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
4.16
470
33
No. needi ng cri ti ca l a ttenti on
South Afri ca Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Pra to
Eng
3.99
233
16
3.19
1
Sunwa y
Info Tech
Law
4.26
12
2
4.10
180
15
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
Peni ns ul a
Bus Eco
Educ
4.03
42
4.14
895
70
3.99
572
34
3.99
205
21
3.88
186
11
3.98
185
12
4.17
120
7
3.92
368
36
3.85
68
8
3.93
156
11
4.02
2904
221
7.6% Table 5: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and Mode, semester 2 2011 Owning Faculty
Art Des & Architecture
Arts
Business & Economics
Education
Engineering
Information Technology
Law
Med Nursing & Health Sci
Pharmacy & Pharm Science
Science
Measure
Av. Median
No. of unit offerings
No. needing critical attention
Av. Median
No. of unit offerings
No. needing critical attention
Av. Median
No. of unit offerings
No. needing critical attention
Av. Median
No. of unit offerings
No. needing critical attention
Av. Median
No. of unit offerings
No. needing critical attention
Av. Median
No. of unit offerings
No. needing critical attention
Av. Median
No. of unit offerings
No. needing critical attention
Av. Median
No. of unit offerings
No. needing critical attention
Av. Median
No. of unit offerings
No. needing critical attention
Av. Median
No. of unit offerings
No. needing critical attention
Overall Av. Median
Total No. of unit offerings
No. needing critical attention
% needing critical attention
Off Campus On Campus
3.00
3.99
3
146
1
10
4.10
4.14
120
763
14
55
3.60
4.03
58
514
11
23
4.01
3.97
67
133
8
13
3.91
3.88
16
170
2
9
3.75
4.01
22
163
3
9
4.17
120
7
3.80
3.98
122
228
18
17
4.14
3.72
20
47
2
6
3.37
3.98
14
142
5
6
3.88
4.04
442
2426
64
155
14.5%
6.4%
On/Off Campus
4.12
12
1
4.09
5
4.00
18
1
4.57
1
4.07
36
2
5.6%
Grand Total
3.97
149
11
4.14
895
70
3.99
572
34
3.99
205
21
3.88
186
11
3.98
185
12
4.17
120
7
3.92
368
36
3.85
68
8
3.93
156
11
4.02
2904
221
7.6% Table 6: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and unit level, semester 2 2011 Grand Owning Faculty Measure
1
2
3
4
5
6
9
L
Total
Art Des & Av. Median
4.08 3.99 4.01 3.65
4.19
3.97
Architecture
No. of unit offerings
41
47
32
25
4
149
No. needing critical attention
3
2
2
4
11
Arts
Av. Median
4.08 4.13 4.14 4.22
4.15
4.14
No. of unit offerings
163 254 270
117
91
895
No. needing critical attention
7
15
24
10
14
70
Business & Av. Median
3.93 3.93 4.00 4.42
4.05 4.50 3.93
3.99
Economics
No. of unit offerings
80 136 157
17
95
4
83
572
No. needing critical attention
2
8
8
8
8
34
Education
Av. Median
3.92 4.00 3.93 3.99
4.02
3.76
3.99
No. of unit offerings
13
14
14
101
58
5
205
No. needing critical attention
2
1
13
5
21
Engineering
Av. Median
3.83 3.71 3.92 3.99
3.88
3.88
No. of unit offerings
23
43
45
62
13
186
No. needing critical attention
2
4
2
1
2
11
Information Av. Median
3.89 3.95 3.92 3.97
4.11
4.11
3.98
Technology
No. of unit offerings
28
43
53
10
38
13
185
No. needing critical attention
2
4
4
1
1
12
Law
Av. Median
3.82 4.28 4.00 4.11
4.25
4.17
No. of unit offerings
2
3
16
32
67
120
No. needing critical attention
2
3
2
7
Med Nursing & Av. Median
3.96 3.76 3.90 3.97
3.86 4.51 4.00 3.69
3.92
Health Sci
No. of unit offerings
34
57
55
103
91
18
1
9
368
No. needing critical attention
1
7
5
8
13
2
36
Pharmacy & Av. Median
3.74 3.79 3.75 3.30
3.92
4.20
3.85
Pharm Science No. of unit offerings
12
14
17
4
3
18
68
No. needing critical attention
1
1
3
1
2
8
Science
Av. Median
3.90 3.90 4.01
1.00
3.93
No. of unit offerings
34
55
66
1
156
No. needing critical attention
2
5
3
1
11
Overall Av. Median
3.99 3.98 4.03 4.04
4.05 4.51 3.99 3.69
4.02
Total No. of unit offerings
430 666 725 471 461 22 120 9 2,904
No. needing critical attention
20 48 54 41 46
10 2 221
% needing critical attention
4.7% 7.2% 7.4% 8.7% 10.0% 0.0% 8.3% 22.2%
7.6% 5. CHANGES OVER TIME Figures 4 and 5 compare the results of the Semester 1 and 2 2011 unit evaluations with those obtained in 2009 and 2010 (full year) to see how the proportion of unit offerings classified as ‘outstanding’ or ‘needing critical attention’ has changed over that time. For the most part, the proportion of unit offerings classified as ‘outstanding’ has remained fairly stable over that time. However, in Pharmacy and Pharmaceutical Sciences the proportion has jumped markedly from just under 3% to over 8%. At the other end of the spectrum though, this faculty has also experienced a rise in the proportion of unit offerings classified as ‘needing critical attention’ (6.6% in 2010 to 11.2% in 2011). Overall, the proportion of unit offerings classified as ‘needing critical attention’ (see Figure 5) has also remained fairly stable (9.9% in 2009, 8.3% in 2010 and 8.1% in 2011). Over the past three years there has been a steady decline in the unit offerings in Arts (11.5% in 2009 down to 8.1% 2011). Also the proportion has nearly halved within Art Des & Architecture (10.1% in 2010 to 6.2% 2011). However, in addition to Pharmacy and Pharmaceutical Sciences mentioned above, Education and Medicine, Nursing and Health Sciences have more than 10% of the unit offerings classified as ‘needing critical attention’. Figure 4: proportion of evaluated unit offerings classified as ‘outstanding’ by faculty, 2009‐2010 (Full year) and 2011 (semesters 1 & 2) 25.0%
% of unit offerings 'outstanding'
20.0%
15.0%
10.0%
5.0%
0.0%
Art Des & Architecture
Arts
Business & Economics
Education
Engineering
Information Technology
Law
Med Nursing & Health Sci
Pharmacy & Pharm Sci
Science
Total
2009 ‐ Full Year
9.3%
18.5%
7.0%
11.2%
4.8%
7.7%
15.5%
6.8%
2.7%
6.9%
11.2%
2010 ‐ Full Year
11.1%
19.3%
7.2%
12.0%
3.4%
5.1%
13.5%
8.2%
2.8%
6.2%
11.5%
2011 ‐ (S1 & S2)
12.1%
19.1%
8.4%
12.8%
6.6%
6.9%
18.4%
8.8%
8.2%
5.6%
12.3%
Figure 5: Proportion of evaluated unit offerings classified as ‘needing critical attention’ by faculty, 2009‐2010 (Full year) and 2011 (semesters 1 & 2) 25.0%
% of unit offerings 'needing critical attention'
20.0%
15.0%
10.0%
5.0%
0.0%
Art Des & Architecture
Arts
Business & Economics
Education
Engineering
Information Technology
Law
Med Nursing & Health Sci
Pharmacy & Pharm Sci
Science
Total
2009 ‐ Full Year
9.3%
11.5%
4.5%
13.8%
12.0%
16.6%
1.5%
10.3%
7.3%
5.6%
9.9%
2010 ‐ Full Year
10.1%
9.6%
4.5%
10.4%
6.4%
10.1%
1.4%
11.7%
6.6%
4.3%
8.3%
2011 ‐ (S1 & S2)
6.2%
8.1%
5.2%
10.8%
8.8%
8.8%
4.6%
11.4%
11.2%
6.9%
8.1%