STUDENT EVALUATION OF TEACHING AND UNITS (SETU) SUMMARY REPORT OF UNIT EVALUATION RESULTS SEMESTER 1, 2013 18 July 2013 INTRODUCTION The Student Evaluation of Teaching and Units (SETU) is an integrated version of two previous surveys; the unit evaluation and MonQueST. SETU includes the original unit evaluation questions and four new teaching questions.1 This report summarises the semester 1 2013 responses to SETU’s ‘overall satisfaction’ item. The overall item required students to rate the statement; ‘Overall I was satisfied with the quality of this unit’, with either ‘Strongly Agree’ (5), ‘Agree’ (4), ‘Neutral’ (3), ‘Disagree’ (2)’ or ‘Strongly Disagree’ (1). From these data, a ‘median’2 score was calculated for each unit offering3. Unit offerings were then classified into one of four groups using the following median score cut‐offs: • Outstanding: ≥4.70 • Meeting aspirations: 3.60 – 4.69 • Needing improvement: 3.01 – 3.59 • Needing critical attention: ≤3.00 FINDINGS Overall satisfaction data was collected for 2,608 degree level4 unit offerings in Semester 1 2013. Note that this figure only includes offerings with overall satisfaction responses. 1. FACULTY VARIATION Table 1 and Figure 1 both show that the vast majority of unit offerings were categorised as either ‘meeting aspirations’ (71.5%) or ‘outstanding’ (12.8%). High performing faculties were Law (22.6% of units ‘outstanding’) and Art Des & Architecture (19.2% of units ‘outstanding’). Notwithstanding high overall satisfaction levels, 7.2% of unit offerings were classified as ‘needing critical attention’. A relatively high percentage of units (>10%) offered by Med Nursing & Health Science (13.1%) fell into this category. Table 2 and Figure 2 show the results for unit offerings with 5 or more responses. Results for unit offerings with less than 5 responses are in Table 3 and Figure 3. 1
Further detail on the SETU can be found at: http://www.opq.monash.edu.au/us/surveys/setu/index.html Further detail on how the median is calculated can be found at: http://www.opq.monash.edu.au/us/surveys/setu/setu_median_calculation.pdf 3
A unit offering is defined here as a unit at a specific location, in a specific mode (e.g. on campus/off campus) and offered for a specific calendar type. 4
Evaluations were also undertaken for Monash College diploma units and Sth Africa Foundation Program units but these are not included in this report. 2
1
Office of Planning and Quality – University Planning and Statistics
Table 1: No. of unit offerings falling into each "traffic light" category by faculty, Semester 1 2013 Median Response Band for Overall Satisfaction Item
Needing Critical Attention
(≤3.0)
6
Needing Improvement
(3.01‐3.59)
11
Meeting Aspirations (3.6‐4.69)
67
Outstanding (≥4.7)
20
Total
104
Arts
52
32
511
126
721
Business & Economics
25
52
404
56
537
Education
17
20
160
37
234
Engineering
17
26
131
14
188
Information Technology
12
20
127
9
168
3
3
66
21
93
358
Owning Faculty
Art Des & Architecture
Law
Med Nursing & Health Sci
47
39
235
37
Pharmacy & Pharm Science
3
4
51
5
63
Science
5
15
113
9
142
187
222
1,865
334
2,608
Grand Total
Table 2: No. of unit offerings with 5 or more responses falling into each "traffic Light" Category by Faculty, Semester 1 2013 Median Response Band for Overall Satisfaction Item
Needing Critical Attention
(≤3.0)
5
Needing Improvement
(3.01‐3.59)
7
Meeting Aspirations (3.6‐4.69)
62
Outstanding (≥4.7)
14
Total
88
Arts
26
21
400
64
511
Business & Economics
11
45
355
29
440
Education
13
16
131
22
182
Engineering
14
23
120
7
164
Information Technology
7
16
109
4
136
Law
1
2
50
12
65
31
25
181
16
253
Owning Faculty
Art Des & Architecture
Med Nursing & Health Sci
Pharmacy & Pharm Science
2
4
43
2
51
Science
2
14
102
1
119
112
173
1,553
171
2,009
Grand Total
Table 3: No. of unit offerings with less than 5 responses falling into each "traffic Light" Category by faculty, Semester 1 2013 Median Response Band for Overall Satisfaction Item
Needing Critical Attention
(≤3.0)
1
Needing Improvement
(3.01‐3.59)
4
Meeting Aspirations (3.6‐4.69)
5
Outstanding (≥4.7)
6
Total
16
Arts
26
11
111
62
210
Business & Economics
Owning Faculty
Art Des & Architecture
14
7
49
27
97
Education
4
4
29
15
52
Engineering
3
3
11
7
24
Information Technology
5
4
18
5
32
Law
2
1
16
9
28
105
Med Nursing & Health Sci
16
14
54
21
Pharmacy & Pharm Science
1
0
8
3
12
Science
3
1
11
8
23
75
49
312
163
599
Grand Total
2
100%
90%
80%
% of unit offerings
70%
60%
50%
40%
30%
20%
10%
0%
Pharmacy & Art Des & Business & Information Med Nursing Arts Education Engineering Law Pharm Architecture Economics Technology & Health Sci (n = 721)
(n = 234)
(n = 188)
(n = 93)
Science (n = 104)
(n = 537)
(n = 168)
(n = 358)
(n = 63)
Science Grand Total (n = 142)
(n = 2608)
Needing Critical Attention
5.8%
7.2%
4.7%
7.3%
9.0%
7.1%
3.2%
13.1%
4.8%
3.5%
7.2%
Needing Improvement
10.6%
4.4%
9.7%
8.5%
13.8%
11.9%
3.2%
10.9%
6.3%
10.6%
8.5%
Meeting Aspirations
64.4%
70.9%
75.2%
68.4%
69.7%
75.6%
71.0%
65.6%
81.0%
79.6%
71.5%
Outstanding
19.2%
17.5%
10.4%
15.8%
7.4%
5.4%
22.6%
10.3%
7.9%
6.3%
12.8%
Figure 1: Proportion of unit offerings falling into each ‘traffic light category’ by faculty, Semester 1 2013 3
100%
90%
80%
% of unit offerings
70%
60%
50%
40%
30%
20%
10%
0%
Needing Critical Attention
Art Des & Arts Architecture (n = 511)
(n = 88)
5.7%
Information Business & Education Engineering Technology Economics (n = 182)
(n = 164)
(n = 136)
(n = 440)
5.1%
2.5%
7.1%
8.5%
5.1%
Law (n = 65)
1.5%
Med Nursing & Pharmacy & Science Health Sci Pharm Science (n = 119)
(n = 253)
(n = 51)
12.3%
3.9%
1.7%
Grand Total (n = 2009)
5.6%
Needing Improvement
8.0%
4.1%
10.2%
8.8%
14.0%
11.8%
3.1%
9.9%
7.8%
11.8%
8.6%
Meeting Aspirations
70.5%
78.3%
80.7%
72.0%
73.2%
80.1%
76.9%
71.5%
84.3%
85.7%
77.3%
Outstanding
15.9%
12.5%
6.6%
12.1%
4.3%
2.9%
18.5%
6.3%
3.9%
0.8%
8.5%
Figure 2: Proportion of unit offerings with 5 or more responses falling into each ‘traffic light category’ by faculty, Semester 1 2013 4
100%
90%
80%
% of unit offerings
70%
60%
50%
40%
30%
20%
10%
0%
Art Des & Arts Architecture (n = 210)
(n = 16)
Information Business & Education Engineering Technology Economics (n = 52)
(n = 24)
(n = 32)
(n = 97)
Needing Critical Attention
0.0%
Needing Improvement
25.0%
5.2%
Meeting Aspirations
31.3%
52.9%
Outstanding
37.5%
29.5%
27.8%
12.4%
14.4%
7.7%
12.5%
15.6%
7.2%
7.7%
12.5%
50.5%
55.8%
45.8%
28.8%
29.2%
Law (n = 28)
Med Nursing & Pharmacy & Science Health Sci Pharm Science (n = 23)
(n = 105)
(n = 12)
8.3%
7.1%
15.2%
12.5%
3.6%
13.3%
0.0%
4.3%
8.2%
56.3%
57.1%
51.4%
66.7%
47.8%
52.1%
15.6%
32.1%
20.0%
25.0%
34.8%
27.2%
Figure 3: Proportion of unit offerings with less than 5 responses falling into each ‘traffic light category’ by faculty, Semester 1 2013 5
13.0%
Grand Total (n = 599)
12.5%
2. CAMPUS/FACULTY VARIATION Table 4 displays average median satisfaction (overall) scores across unit offerings, with data separated by campus and faculty. There were no campus/faculty groups who had average median scores in the ‘needing critical attention’ range. Three campus/faculty groups reached ‘Outstanding’ criteria for their average median score (Information Technology at Berwick, Art Design & Architecture at China‐South East University, and Arts at ‘Other Australian Locations’), although all had small unit and response counts. 3 responses formed the result for Information Technology and five responses across three unit offerings formed the result for Arts. Art Design & Architecture at China‐South East University had a slightly higher number of responses (although still small) with 13 responses for each of the three unit offerings. The total average median score of China‐South East University also obtained an ‘Outstanding’ classification (4.72), on the basis of 141 responses across 10 unit offerings. Examining Campus alone (far right column) indicates a relatively high percentage of units with the critical attention flag at Berwick (10.3%), Gippsland (12.0%), Peninsula (14.0%), and Prato (20.0%). For all other campuses, less than 10% of units fell into the critical range. Examining Faculty alone (third bottom row of table 4), shows the average median for all faculties is in the ‘meeting aspirations’ range. 3. UNIT MODE VARIATION Table 5 displays average median satisfaction broken down by faculty and mode (on‐campus, off‐campus, or on/off campus). The majority of faculty/mode groups fell into the ‘meeting aspirations’ category. The on/off campus Medicine, Nursing and Health Science group (3.48) was the exception, falling into the ‘needing improvement’ category. Collapsing across faculty groups (bottom 4 rows of table 5) shows that average median scores from all three modes (on, off, on/off‐campus) are in the ‘meeting aspirations’ range. However, it also shows that units offered on campus received less negative attention (5.7% in the critical range) than either on/off campus units (21.2% in the critical range) or off‐campus units (14.5% in the critical range). 4. UNIT LEVEL VARIATION Table 6 shows average median satisfaction split by faculty and unit level. Level 6 in the Faculty of Arts, Design and Architecture appears concerning (average median = 3.00), although only two responses comprised this result. In contrast, the average median satisfaction with level 6 units offered by the faculty of Business & Economics was well into the ‘outstanding’ range (4.95). Eight responses across the two units formed this result. Collapsing across faculty (bottom 4 rows of table 6), shows the average median score from each unit level is in the ‘Meeting Aspirations’ range, and all levels obtained a percentage of units in ‘needing critical attention’ at or below 10%. 6
Table 4: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and location, Semester 1 2013 Location
Berwi ck
Measure
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Ca ul fi el d
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Cl a yton
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Gi pps l a nd Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Pa rkvi l l e
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Peni ns ul a Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
South Afri ca Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Sunwa y
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Pra to
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Os Chi na
Av. Medi a n
South Ea s t No. of uni t offeri ngs
Univers i ty No. needi ng cri ti cal a ttenti on
Other Av. Medi a n
Aus tra l i a n No. of uni t offeri ngs
Locati ons
No. needi ng cri ti cal a ttenti on
Other Av. Medi a n
Offs hore No. of uni t offeri ngs
Locati ons
No. needi ng cri ti cal a ttenti on
Overa l l Av. Medi a n
Total no. of uni t offeri ngs
Total no. needi ng cri ti ca l a ttention
Art, Des Pharm & & Arch Arts Bus Eco Educ Eng Info Tech Law MNHS Pharm Sci
3.96
3.95
3.95
4.75
3.70
26
40
14
1
16
3
1
2
4
4.09
4.12
4.08
4.05
3.69
96
156
207
75
54
6
8
8
3
9
4.23
3.94
4.10 3.93
3.86 4.32
3.99
347
92
116
112
28
35
107
17
4
9
10
2
10
4.23
4.00
4.14
4.13 3.86
3.97
3.74
5
95
67
41
15
22
30
14
6
2
3
3
6
4.01
47
3
4.36
3.77
3.99
3.82
1
24
51
60
5
4
10
3.97
4.32
3.76
4.04
58
30
14
11
4
1
1
3.91
4.11
4.13 3.91
3.67
3.75
3.82
29
77
2
58
24
14
16
3
1
4
3
1
3.17
4.28
6
14
3
1
4.87
4.67
4.66
3
3
4
5.00
3
4.25
44
2
4.03
65
6
4.00
1
4.28
93
3
3.88
358
47
4.13
10
4.12
104
6
4.13
721
52
4.06
537
25
4.07
234
17
3.93
188
17
3.95
168
12
3.96
63
3
Sci
4.07
81
2
4.04
34
3
3.88
27
4.03
142
5
% needing critical Total attention
3.92
97
10
10.3%
4.05
588
34
5.8%
4.10
918
54
5.9%
4.02
309
37
12.0%
4.01
47
3
6.4%
3.88
136
19
14.0%
4.04
113
6
5.3%
3.93
247
12
4.9%
3.94
20
4
20.0%
4.72
10
0.0%
4.14
112
8
7.1%
4.12
11
0.0%
4.04
2608
187
7.2%
7
Table 5: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and Mode, Semester 1 2013 Owning Faculty
Art, Des i gn & Archi tecture
Arts
Bus i nes s & Economi cs
Educa ti on
Engi neeri ng
Informa ti on Technol ogy
La w
Med Nurs i ng & Hea l th Sci
Pha rma cy & Pha rm Sci ence
Sci ence
Overa l l Av. Medi a n
Tota l No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
% needi ng cri ti ca l a ttenti on
Measure
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Off Campus On Campus
3.74
4.13
3
101
6
3.91
4.16
88
624
15
35
4.06
4.06
52
485
6
19
3.95
4.12
62
164
6
10
4.01
3.92
13
175
2
15
3.83
3.96
20
148
3
9
4.28
93
3
3.85
3.92
104
238
17
26
4.13
3.90
17
46
2
1
3.95
4.04
14
128
3
2
3.93
4.07
373
2202
54
14.5%
126
5.7%
On/Off Campus
3.72
33
Grand Total
4.12
104
6
4.13
721
52
4.06
537
25
4.07
234
17
3.93
188
17
3.95
168
12
4.28
93
3
3.88
358
47
3.96
63
3
4.03
142
5
4.04
2608
7
21.2%
187
7.2%
3.93
9
2
3.98
8
1
3.48
16
4
8
Table 6: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and unit level, Semester 1 2013 Owning Faculty
Art, Des i gn & Archi tecture
Arts
Bus i nes s & Economi cs
Educati on
Engi neeri ng
Informati on Technol ogy
La w
Med Nurs i ng & Hea l th Sci
Pha rma cy & Pha rm Sci ence
Sci ence
Overa l l Av. Medi a n
Total No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
% needi ng cri ti ca l attenti on
Measure
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
1
4.09
22
2
4.13
125
3
4.01
73
3
4.18
19
3.94
23
1
3.91
25
2
3.76
1
3.83
50
4
3.83
13
4.04
33
1
4.02
384
16
4.2%
2
4.24
32
4.03
220
24
3.94
133
6
4.09
17
2
3.73
43
5
3.84
36
3
4.20
2
3.71
53
9
3.89
12
1
4.01
45
1
3.96
593
51
8.6%
3
3.96
22
1
4.18
237
14
4.07
144
7
4.00
16
2
3.96
48
5
3.99
40
3
4.25
10
4
4.21
20
2
4.20
91
6
4.03
29
2
4.04
111
9
4.00
59
4
3.74
7
1
4.43
28
3.78
63
8
3.95
13
3.88
72
13
3.93
8
4.03
64
3
4.06
657
43
6.5%
4.07
425
37
8.7%
5
3.95
6
4.12
46
5
4.17
81
5
4.11
71
4
4.09
15
2
3.99
44
2
4.22
52
3
3.99
98
12
4.17
3
1
6
3.00
1
1
4.50
2
9
4.29
1
4.95
2
4.15
75
2
L
4.12
16
1
4.41
15
1
4.09 4.41
416
20
34
2
8.2% 10.0%
3.98
4
3.80
3
4.12
14
1
4.13
110
4
3.6%
3.80
3
Grand Total
4.12
104
6
4.13
721
52
4.06
537
25
4.07
234
17
3.93
188
17
3.95
168
12
4.28
93
3
3.88
358
47
3.96
63
3
4.03
142
5
4.04
2608
187
7.2%
9
5.
CHANGES OVER TIME Figures 4 and 5 display the faculty‐level trends in unit evaluations between 2010 and 20135, with regard to the percentage of units classified as ‘outstanding’ (figure 4) or as ‘needing critical attention’ (figure 5). The percentage of unit offerings classified as ‘outstanding’ (all faculties combined) has increased gradually since 2010 (11.5% in 2010, 12.3% in both 2011 and 2012, 12.8% in 2013). At the faculty level the Art Des & Architecture show a sharp improvement (19.2% in Semester 1 2013 following 13% in 2012). In addition, improvement is also seen in the faculties of Business and Economics, Education, Engineering, Medicine, and Science. Decreases are noted for semester 1 2013 results in Arts (19.3% in 2010 dropping to 17.5% in 2013), Information Technology (9.3% in 2012 decreasing to 5.4% in 2013) and Pharmacy (9.8% in 2012 decreasing to 7.9% in 2013). The percentage of unit offerings classified as ‘needing critical attention’ (all faculties combined) has also gradually decreased (8.3% in 2010, 8.1% in 2011, 7.8% in 2012, 7.2% in semester 1 of 2013). At the faculty level, the past four years have seen a regular decline (i.e., improvement) in the percentage of units in the ‘needing critical attention’ range at Arts (9.6% in 2010 to 7.2% in 2013) and Information Technology (10.1% in 2010 to 7.1 in 2013). Signs of improvement can also be seen at Business & Economics, Education, and Law where the percentage of units in the ‘needing critical attention’ range has decreased in 2013. The percentage of units in the critical range at the Medicine, Nursing and Health Sciences faculties has exceeded 10% every year since 2010. 5
2013 figures include only Semester 1 data. 10
25%
% of unit offerings 'outstanding'
20%
15%
10%
5%
0%
Art Des & Architecture
Arts
Business & Economics
Education
Engineering
Information Technology
Law
Med Nursing & Health Sci
Pharmacy & Pharm Sci
Science
Grand Total
2010 ‐ Full Year
11.1%
19.3%
7.2%
12.0%
3.4%
5.1%
13.5%
8.2%
2.8%
6.2%
11.5%
2011 ‐ Full Year
12.1%
19.1%
8.5%
12.6%
6.6%
6.9%
18.4%
8.8%
8.7%
5.6%
12.3%
2012 ‐ Full Year
13.0%
18.1%
9.1%
11.8%
4.4%
9.3%
23.0%
9.8%
9.8%
5.5%
12.3%
2013 ‐S1
19.2%
17.5%
10.4%
15.8%
7.4%
5.4%
22.6%
10.3%
7.9%
6.3%
12.8%
Figure 4: Percentage of evaluated unit offerings classified as 'outstanding' by faculty, 2010‐2013. 11
25%
% of unit offerings 'needing critical attention'
20%
15%
10%
5%
0%
Art Des & Architecture
Arts
Business & Economics
Education
Engineering
Information Technology
Law
Med Nursing & Health Sci
Pharmacy & Pharm Sci
Science
Grand Total
2010 ‐ Full Year
10.1%
9.6%
4.5%
10.4%
6.4%
10.1%
1.4%
11.7%
6.6%
4.3%
8.3%
2011 ‐ Full Year
6.2%
8.2%
5.2%
10.8%
8.8%
8.8%
4.6%
11.4%
11.0%
6.9%
8.1%
2012 ‐ Full Year
8.3%
8.0%
6.0%
11.3%
5.7%
7.4%
4.7%
10.7%
8.4%
5.2%
7.8%
2013 ‐S1
5.8%
7.2%
4.7%
7.3%
9.0%
7.1%
3.2%
13.1%
4.8%
3.5%
7.2%
Figure 5: Proportion of evaluated unit offerings classified as 'needing critical attention' by faculty, 2010‐2013 12