STUDENT EVALUATION OF TEACHING AND UNITS (SETU) SUMMARY REPORT OF UNIT EVALUATION RESULTS SEMESTER 1, 2012 Nikki Powell 13 July 2012 INTRODUCTION The Student Evaluation of Teaching and Units (SETU) replaces the previously separated unit evaluation and MonQueST surveys into one instrument. The original unit evaluation questions were included plus four new teaching questions.1 This report provides a summary of the semester 1 2012 results for the unit evaluation ‘overall satisfaction’ question. Students were asked to respond to the statement ‘Overall I was satisfied with the quality of this unit’. The response options were ‘Strongly Agree’ (5), ‘Agree’ (4), ‘Neutral’ (3), ‘Disagree’ (1)’ and ‘Strongly Disagree’ (1). From these data a ‘median’2 score has been calculated for each unit offering3. Unit offerings are then classified into one of four groups depending upon this median score: • Outstanding: ≥4.70 • Meeting aspirations: 3.60 – 4.69 • Needing improvement: 3.01 – 3.59 • Needing critical attention: ≤3.00 FINDINGS Unit evaluation overall satisfaction data was collected for 2,671 degree level4 unit offerings in Semester 1 2012. Note that more unit offerings were included in the SETU but only those with data for the overall satisfaction question have been included here. Unit offerings without overall satisfaction data are usually those that were included but had zero or very small enrolment numbers. 1. FACULTY VARIATION As shown in Table 1 and Figure 1, the vast majority of unit offerings (84.3%) were categorised as ‘meeting aspirations’ or ‘outstanding’, with 10.6% in the latter category. The results for the Arts and Law faculties were particularly strong with students rating 17.2% of the unit offerings with a median score of 4.7 or above. While overall satisfaction levels are generally high, 7.4% of unit offerings are in ‘need of critical attention’ after obtaining a score of 3 or less. Education and Medicine Nursing & Health Sciences both had more than 10% of their unit offerings in this category. Table 2 and Figure 2 show the data for those unit offerings with 5 or more responses. The data for unit offerings with less than 5 responses can be seen in Table 3 and Figure 3. 1
Further detail on the SETU can be found at: http://www.opq.monash.edu.au/us/surveys/setu/index.html Further detail on how the median is calculated can be found at: http://www.opq.monash.edu.au/us/surveys/setu/setu_median_calculation.pdf 3
A unit offering is defined here as a unit at a specific location, in a specific mode (e.g. on campus/off campus) and offered for a specific calendar type. 4
Evaluations were also undertaken for Monash College diploma units and Sth Africa Foundation Program units but these are not included in this report. 2
1
Office of Planning and Quality – University Planning and Statistics
Table 1: No. of unit offerings falling into each "traffic light" category by faculty, Semester 1 2012 Median Response Band for Overall Satisfaction Item
Needing Critical Attention
(≤3.0)
3
Needing Improvement
(3.01‐3.59)
11
Meeting Aspirations (3.6‐4.69)
71
Outstanding (≥4.7)
9
Total
94
Arts
60
53
559
140
812
Business & Economics
33
42
421
39
535
Education
21
14
145
15
195
Owning Faculty
Art Des & Architecture
Engineering
Information Technology
Law
Med Nursing & Health Sci
Pharmacy & Pharm Science
Science
Grand Total
8
25
136
7
176
14
11
134
11
170
4
2
71
16
93
47
41
266
32
386
3
8
50
9
70
4
16
115
5
140
197
223
1,968
283
2,671
Table 2: No. of unit offerings with 5 or more responses falling into each "traffic Light" Category by Faculty, Semester 1 2012 Median Response Band for Overall Satisfaction Item
Needing Critical Attention
(≤3.0)
3
Needing Improvement
(3.01‐3.59)
10
Meeting Aspirations (3.6‐4.69)
69
Outstanding (≥4.7)
9
Total
91
Arts
23
40
432
60
555
Business & Economics
26
36
383
22
467
Education
Owning Faculty
Art Des & Architecture
11
13
133
12
169
Engineering
8
24
129
3
164
Information Technology
9
9
117
7
142
Law
Med Nursing & Health Sci
3
1
61
10
75
24
32
216
16
288
Pharmacy & Pharm Science
2
7
41
2
52
Science
2
15
110
3
130
111
187
1,691
144
2,133
Grand Total
Table 3: No. of unit offerings with less than 5 responses falling into each "traffic Light" Category by faculty, Semester 1 2012 Median Response Band for Overall Satisfaction Item
Owning Faculty
Art Des & Architecture
Arts
Business & Economics
Needing Critical Attention
(≤3.0)
0
Needing Improvement
(3.01‐3.59)
1
Meeting Aspirations (3.6‐4.69)
2
Outstanding (≥4.7)
0
Total
3
37
13
127
80
257
7
6
38
17
68
10
1
12
3
26
Engineering
0
1
7
4
12
Information Technology
5
2
17
4
28
Law
1
1
10
6
18
23
9
50
16
98
Pharmacy & Pharm Science
1
1
9
7
18
Science
2
1
5
2
10
86
36
277
139
538
Education
Med Nursing & Health Sci
Grand Total
2
100%
90%
80%
% of unit offerings
70%
60%
50%
40%
30%
20%
10%
0%
Pharmacy & Art Des & Business & Information Med Nursing Arts Education Engineering Law Pharm Architecture Economics Technology & Health Sci (n = 812)
(n = 195)
(n = 176)
(n = 93)
Science (n = 94)
(n = 535)
(n = 170)
(n = 386)
(n = 70)
Science Grand Total (n = 140)
(n = 2671)
Needing Critical Attention
3.2%
7.4%
6.2%
10.8%
4.5%
8.2%
4.3%
12.2%
4.3%
2.9%
7.4%
Needing Improvement
11.7%
6.5%
7.9%
7.2%
14.2%
6.5%
2.2%
10.6%
11.4%
11.4%
8.3%
Meeting Aspirations
75.5%
68.8%
78.7%
74.4%
77.3%
78.8%
76.3%
68.9%
71.4%
82.1%
73.7%
Outstanding
9.6%
17.2%
7.3%
7.7%
4.0%
6.5%
17.2%
8.3%
12.9%
3.6%
10.6%
Figure 1: Proportion of unit offerings falling into each ‘traffic light category’ by faculty, semester 1 2012 3
100%
90%
80%
% of unit offerings
70%
60%
50%
40%
30%
20%
10%
0%
Pharmacy & Art Des & Business & Information Med Nursing Arts Education Engineering Law Pharm Architecture Economics Technology & Health Sci (n = 555)
(n = 169)
(n = 164)
(n = 75)
Science (n = 91)
(n = 467)
(n = 142)
(n = 288)
(n = 52)
Science Grand Total (n = 130)
(n = 2133)
Needing Critical Attention
3.3%
4.1%
5.6%
6.5%
4.9%
6.3%
4.0%
8.3%
3.8%
1.5%
5.2%
Needing Improvement
11.0%
7.2%
7.7%
7.7%
14.6%
6.3%
1.3%
11.1%
13.5%
11.5%
8.8%
Meeting Aspirations
75.8%
77.8%
82.0%
78.7%
78.7%
82.4%
81.3%
75.0%
78.8%
84.6%
79.3%
Outstanding
9.9%
10.8%
4.7%
7.1%
1.8%
4.9%
13.3%
5.6%
3.8%
2.3%
6.8%
Figure 2: Proportion of unit offerings with 5 or more responses falling into each ‘traffic light category’ by faculty, semester 1 2012 4
100%
90%
80%
% of unit offerings
70%
60%
50%
40%
30%
20%
10%
0%
Pharmacy & Art Des & Business & Information Med Nursing Arts Education Engineering Law Pharm Architecture Economics Technology & Health Sci (n = 257)
(n = 26)
(n = 12)
(n = 18)
Science (n = 3)
(n = 68)
(n = 28)
(n = 98)
(n = 18)
Science Grand Total (n = 10)
(n = 538)
Needing Critical Attention
0.0%
14.4%
10.3%
38.5%
0.0%
17.9%
5.6%
23.5%
5.6%
20.0%
16.0%
Needing Improvement
33.3%
5.1%
8.8%
3.8%
8.3%
7.1%
5.6%
9.2%
5.6%
10.0%
6.7%
Meeting Aspirations
66.7%
49.4%
55.9%
46.2%
58.3%
60.7%
55.6%
51.0%
50.0%
50.0%
51.5%
Outstanding
0.0%
31.1%
25.0%
11.5%
33.3%
14.3%
33.3%
16.3%
38.9%
20.0%
25.8%
Figure 3: Proportion of unit offerings with less than 5 responses falling into each ‘traffic light category’ by faculty, semester 1 2012 5
2. CAMPUS VARIATION Table 4 shows the average of the median overall satisfaction scores obtained by the unit offerings. The data are disaggregated by location and faculty. On average, the unit offerings within each of the faculties fell into the ‘meeting aspirations’ category with the faculty average medians ranging from 3.89 for Medicine Nursing & Health Sciences to 4.19 for Law. The same can be said for the locations ranging from 3.8 at Peninsula campus to 4.12 at Prato. Only one unit offerings ‘needed critical attention’ within the Faculty of Arts at ‘other offshore locations’ obtaining a median score of 3. Further investigation showed that this unit offering only had 1 enrolment. The locations with more than 10% of evaluated unit offerings needing critical attention are highlighted in red. These were Gippsland, Parkville, Peninsula and Prato. 3. UNIT MODE VARIATION Table 5 shows the average of the median overall satisfaction scores obtained by the unit offerings disaggregated by faculty and mode. On average, the unit offerings within each of the modes of study fell into the ‘meeting aspirations’ category with the average medians ranging from 3.88 for off campus to 4.05 for on campus. Units offered off campus (17.3%) were much more likely than those offered on campus (5.6%) to be obtain a score of 3 or below and therefore be classified as ‘needing critical attention’. 4. UNIT LEVEL VARIATION Table 6 shows the average of the median overall satisfaction scores obtained by the unit offerings disaggregated by faculty and unit level. There was only one area in which the unit offerings on average ‘needing critical attention’ Level 6 in the Faculty of Arts. While there does not appear to be a strong relationship between level and the likelihood of a unit ‘needing critical attention’ 10.9% of the units at level 5 were in this category. 6
Table 4: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and location, semester 1 2012 Location
Berwi ck
Measure
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Ca ul fi el d
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Cl a yton
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Gi pps l a nd Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Pa rkvi l l e
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Peni ns ul a Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
South Afri ca Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Sunwa y
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Pra to
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Other Av. Medi a n
Aus tra l i a n No. of uni t offeri ngs
Locati ons
No. needi ng cri ti cal a ttenti on
Other Av. Medi a n
Offs hore No. of uni t offeri ngs
Locati ons
No. needi ng cri ti cal a ttenti on
Overa l l Av. Medi a n
Total no. of uni t offeri ngs
Total no. needi ng cri ti ca l a ttention
Art, Des Pharm & & Arch Arts Bus Eco Educ Eng Info Tech Law MNHS Pharm Sci
3.99
3.86
4.02
4.69
4.33
24
34
6
6
12
3
2
1
0
0
4.07
4.14
4.05
4.15
3.92
91
183
218
76
60
3
16
9
3
9
4.38
4.18
3.98
3.95 3.95
3.74 4.11
3.96
1
401
87
102
109
29
32
120
0
18
3
12
6
3
2
8
4.08
3.98
3.85
3.99 3.80
3.80
3.71
2
113
69
31
13
24
34
0
13
14
3
1
5
6
4.07
54
3
3.87
3.88
4.00
3.64
1
28
44
71
0
3
2
16
4.05
4.23
3.95
4.02
53
25
14
11
3
1
1
4.08
4.03
4.00 3.87
3.79
4.10
3.74
29
72
1
54
21
14
16
3
2
0
1
2
0
0
3.88
4.17
4
20
2
1
4.17
4.50
4.26
3.92
3
2
41
53
1
0
1
7
3.00
3.65
4.09
1
11
11
1
3
0
4.08
4.12
4.00
3.96 3.91
3.99 4.19
3.89
4.00
94
812
535
195
176
170
93
386
70
3
60
33
21
8
14
4
47
3
Sci
4.05
81
2
4.00
31
2
3.88
28
0
4.01
140
4
% needing critical Total attention
4.04
82
6
7.3%
4.08
628
40
6.4%
4.06
962
54
5.6%
3.91
317
44
13.9%
4.07
54
3
5.6%
3.80
144
21
14.6%
4.08
103
5
4.9%
3.95
235
8
3.4%
4.12
24
3
12.5%
4.08
99
9
9.1%
3.83
23
4
17.4%
4.02
2671
197
7.4%
7
Table 5: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and Mode, semester 1 2012 Owning Faculty
Art, Des i gn & Archi tecture
Arts
Bus i nes s & Economi cs
Educa ti on
Engi neeri ng
Informa ti on Technol ogy
La w
Med Nurs i ng & Hea l th Sci
Pha rma cy & Pha rm Sci ence
Sci ence
Overa l l Av. Medi a n
Tota l No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
% needi ng cri ti ca l a ttenti on
Measure
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Off Campus On Campus
3.75
4.08
2
92
0
3
3.97
4.15
110
695
17
42
3.81
4.03
66
469
13
20
3.79
4.00
51
135
9
12
3.74
3.93
13
163
2
6
3.69
4.03
20
150
6
8
4.19
93
4
3.82
3.91
121
242
23
23
4.26
3.88
21
48
1
2
4.15
3.99
13
127
1
3
3.88
4.05
417
2214
72
17.3%
123
5.6%
On/Off Campus
4.04
40
Grand Total
4.08
94
3
4.12
812
60
4.00
535
33
3.96
195
21
3.91
176
8
3.99
170
14
4.19
93
4
3.89
386
47
4.00
70
3
4.01
140
4
4.02
2671
2
5.0%
197
7.4%
3.77
7
1
4.28
9
0
4.02
23
1
4.17
1
8
Table 6: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and unit level, semester 1 2012 Owning Faculty
Art, Des i gn & Archi tecture
Arts
Bus i nes s & Economi cs
Educati on
Engi neeri ng
Informa ti on Technol ogy
Law
Med Nurs i ng & Heal th Sci
Pha rma cy & Pha rm Sci ence
Sci ence
Overal l Av. Media n
Tota l No. of uni t offeri ngs
No. needi ng cri ti ca l a ttention
% needi ng cri ti ca l attenti on
Measure
Av. Medi an
No. of unit offeri ngs
No. needing cri ti ca l attenti on
Av. Medi an
No. of unit offeri ngs
No. needing cri ti ca l attenti on
Av. Medi an
No. of unit offeri ngs
No. needing cri ti ca l attenti on
Av. Medi an
No. of unit offeri ngs
No. needing cri ti ca l attenti on
Av. Medi an
No. of unit offeri ngs
No. needing cri ti ca l attenti on
Av. Medi an
No. of unit offeri ngs
No. needing cri ti ca l attenti on
Av. Medi an
No. of unit offeri ngs
No. needing cri ti ca l attenti on
Av. Medi an
No. of unit offeri ngs
No. needing cri ti ca l attenti on
Av. Medi an
No. of unit offeri ngs
No. needing cri ti ca l attenti on
Av. Medi an
No. of unit offeri ngs
No. needing cri ti ca l attenti on
1
4.14
21
0
4.09
123
4
3.88
75
5
4.13
18
0
3.87
23
0
3.99
25
0
3.92
1
0
3.92
51
3
3.80
13
1
3.94
33
1
3.99
383
14
3.7%
2
4.11
32
1
4.06
236
17
3.89
126
10
4.07
16
1
3.80
40
3
3.86
41
4
4.10
2
0
3.77
56
8
4.06
14
4.01
48
0
3.97
611
44
7.2%
3
3.99
22
2
4.17
251
16
3.98
135
7
3.96
14
2
3.89
47
2
3.99
46
4
4.08
13
1
3.90
67
7
3.75
13
1
4.03
59
3
4.04
667
45
6.7%
4
4.14
11
0
4.19
123
11
4.21
19
1
3.94
80
7
4.08
54
1
3.68
6
2
4.12
29
2
3.74
85
16
3.88
8
0
5
3.92
7
0
4.09
78
11
4.12
93
7
3.90
66
11
3.65
11
2
4.09
36
3
4.27
48
1
3.98
106
13
3.60
5
1
4.02
4.04
415
450
40
49
9.6% 10.9%
6
4.00
1
0
3.00
1
1
4.13
4
1
9
L
4.12
83
2
4.00
1
0
4.50
1
0
4.19
16
1
4.33
17
0
3.86
1
0
4.45
17
0
3.91
3
0
4.22
23
2
8.7%
4.18
119
3
2.5%
3.91
3
0
0.0%
Grand Total
4.08
94
3
4.12
812
60
4.00
535
33
3.96
195
21
3.91
176
8
3.99
170
14
4.19
93
4
3.89
386
47
4.00
70
3
4.01
140
4
4.02
2671
197
7.4%
9
5.
CHANGES OVER TIME Figures 4 and 5 compare the results of the Semester 1 2012 unit evaluations with those obtained in 2009 to 2011 (full year) to see how the proportion of unit offerings classified as ‘outstanding’ or ‘needing critical attention’ has changed over that time. For the most part, the proportion of unit offerings classified as ‘outstanding’ has remained fairly stable over that time, with a slight decrease noted for semester 1 2012 results (10.6%, 12.4% in 2011). A trend reflected across the faculties with the exception of Pharmacy and Pharmaceutical Sciences for whom the proportion of unit offerings classified as ‘outstanding’ has jumped markedly from just under 3% to nearly 13%. Whilst also experiencing a rise in the proportion of unit offerings classified as ‘needing critical attention’ between 2010 and 2010 (6.6% in 2010 to 11.2% in 2011), this has now decreased markedly in 2012 semester 1 (4.3%). Overall, the proportion of unit offerings classified as ‘needing critical attention’ (see Figure 5) has remained fairly stable (8% in 2011, 7.4% in 2012 semester 1). Over the past three years there has been a steady decline in unit offerings ‘needing critical attention’ across the faculties of Arts (11.5% in 2009 down to 7.4% in 2012 semester 1); Art Des & Architecture (10.1% in 2010 to 3.2% in 2012 semester 1); and Information Technology (16.6% in 2009 to 8.2% in 2012 semester 1). Education and Medicine, Nursing and Health Sciences continue to have more than 10% of the unit offerings classified as ‘needing critical attention’. 10
25%
% of unit offerings 'outstanding'
20%
15%
10%
5%
0%
Art Des & Architecture
Arts
Business & Economics
Education
Engineering
Information Technology
Law
Med Nursing Pharmacy & & Health Sci Pharm Sci
2009 ‐ Full Year
9.3%
18.6%
7.0%
11.2%
4.8%
7.7%
15.5%
6.8%
2010 ‐ Full Year
11.1%
19.3%
7.2%
12.0%
3.4%
5.1%
13.5%
2011 ‐ Full Year
12.1%
19.1%
8.2%
13.1%
6.6%
7.3%
2012 (S1)
9.6%
17.2%
7.3%
7.7%
4.0%
6.5%
Science
Grand Total
2.7%
6.9%
11.2%
8.2%
2.8%
6.2%
11.5%
19.2%
9.1%
8.2%
5.6%
12.4%
17.2%
8.3%
12.9%
3.6%
10.6%
Figure 4: Proportion of evaluated unit offerings classified as 'outstanding' by faculty, 2009‐2011 (full year) and 2012 (semester 1) 11
25%
% of unit offerings 'needing critical attention'
20%
15%
10%
5%
0%
Art Des & Architecture
Arts
Business & Economics
Education
Engineering
Information Technology
Law
Med Nursing Pharmacy & & Health Sci Pharm Sci
2009 ‐ Full Year
9.3%
11.4%
4.5%
13.8%
12.0%
16.6%
1.5%
2010 ‐ Full Year
10.1%
9.6%
4.5%
10.4%
6.4%
10.1%
1.4%
11.7%
6.6%
4.3%
8.3%
2011 ‐ Full Year
6.2%
8.0%
5.4%
10.3%
8.8%
8.7%
4.2%
11.4%
11.2%
6.9%
8.0%
2012 (S1)
3.2%
7.4%
6.2%
10.8%
4.5%
8.2%
4.3%
12.2%
4.3%
2.9%
7.4%
10.3%
7.3%
Science
Grand Total
5.6%
9.8%
Figure 5: Proportion of evaluated unit offerings classified as 'needing critical attention' by faculty, 2009‐2011 (full year) and 2012 (semester 1) 12