STUDENT EVALUATION OF TEACHING AND UNITS (SETU) SUMMARY REPORT OF UNIT EVALUATION RESULTS SEMESTER 2, 2012 Leo Roberts 10 December 2012 INTRODUCTION The Student Evaluation of Teaching and Units (SETU) is an integrated version of two previous surveys; the unit evaluation and MonQueST. SETU includes the original unit evaluation questions and four new teaching questions.1 This report summarises the semester 2 2012 responses to SETU’s ‘overall satisfaction’ item. The overall item required students to rate the statement; ‘Overall I was satisfied with the quality of this unit’, with either ‘Strongly Agree’ (5), ‘Agree’ (4), ‘Neutral’ (3), ‘Disagree’ (2)’ or ‘Strongly Disagree’ (1). From these data, a ‘median’2 score was calculated for each unit offering3. Unit offerings were then classified into one of four groups using the following median score cut‐offs: • Outstanding: ≥4.70 • Meeting aspirations: 3.60 – 4.69 • Needing improvement: 3.01 – 3.59 • Needing critical attention: ≤3.00 FINDINGS Overall satisfaction data was collected for 2,699 degree level4 unit offerings in Semester 2 2012. Note that this figure only includes offerings with overall satisfaction responses. 1. FACULTY VARIATION Table 1 and Figure 1 both show that the vast majority of unit offerings were categorised as either ‘meeting aspirations’ (69.8%) or ‘outstanding’ (13.7%). High performing faculties were Law (26.9% of units ‘outstanding’) and Arts (19.1% of units ‘outstanding’). Notwithstanding high overall satisfaction levels, 8.1% of unit offerings were classified as ‘needing critical attention’. A relatively high percentage of units (>10%) offered by Art Design & Architecture (13.1%), Pharmacy and Pharmaceutical Science (12.3%) and Education (12.0%) fell into this category. Table 2 and Figure 2 show the results for unit offerings with 5 or more responses. Results for unit offerings with less than 5 responses are in Table 3 and Figure 3. 1
Further detail on the SETU can be found at: http://www.opq.monash.edu.au/us/surveys/setu/index.html Further detail on how the median is calculated can be found at: http://www.opq.monash.edu.au/us/surveys/setu/setu_median_calculation.pdf 3
A unit offering is defined here as a unit at a specific location, in a specific mode (e.g. on campus/off campus) and offered for a specific calendar type. 4
Evaluations were also undertaken for Monash College diploma units and Sth Africa Foundation Program units but these are not included in this report. 2
1
Office of Planning and Quality – University Planning and Statistics
Table 1: No. of unit offerings falling into each "traffic light" category by faculty, Semester 2 2012 Median Response Band for Overall Satisfaction Item
Needing Critical Attention
(≤3.0)
13
Needing Improvement
(3.01‐3.59)
10
Meeting Aspirations (3.6‐4.69)
60
Outstanding (≥4.7)
16
Total
99
Arts
67
55
508
149
779
Business & Economics
31
46
435
62
574
Education
25
11
140
32
208
Engineering
13
25
144
9
191
Information Technology
12
17
128
20
177
6
7
63
28
104
31
35
242
38
346
9
9
50
5
73
Science
11
12
113
12
148
Grand Total
218
227
1,883
371
2,699
Owning Faculty
Art Des & Architecture
Law
Med Nursing & Health Sci
Pharmacy & Pharm Science
Table 2: No. of unit offerings with 5 or more responses falling into each "traffic Light" Category by Faculty, Semester 2 2012 Median Response Band for Overall Satisfaction Item
Needing Critical Attention
(≤3.0)
8
Needing Improvement
(3.01‐3.59)
10
Meeting Aspirations (3.6‐4.69)
52
Outstanding (≥4.7)
12
Total
82
Arts
21
34
390
70
515
Business & Economics
14
39
372
27
452
Education
15
10
120
21
166
Engineering
11
23
134
7
175
4
11
111
8
134
Owning Faculty
Art Des & Architecture
Information Technology
Law
Med Nursing & Health Sci
Pharmacy & Pharm Science
Science
Grand Total
4
4
52
10
70
14
27
195
14
250
4
6
40
2
52
8
12
101
7
128
103
176
1,567
178
2,024
Table 3: No. of unit offerings with less than 5 responses falling into each "traffic Light" Category by faculty, Semester 2 2012 Median Response Band for Overall Satisfaction Item
Needing Critical Attention
(≤3.0)
5
Needing Improvement
(3.01‐3.59)
0
Meeting Aspirations (3.6‐4.69)
8
Outstanding (≥4.7)
4
Total
17
Arts
46
21
118
79
264
Business & Economics
17
7
63
35
122
Education
10
1
20
11
42
Engineering
2
2
10
2
16
Information Technology
8
6
17
12
43
Law
2
3
11
18
34
17
8
47
24
96
Pharmacy & Pharm Science
5
3
10
3
21
Science
3
0
12
5
20
115
51
316
193
675
Owning Faculty
Art Des & Architecture
Med Nursing & Health Sci
Grand Total
2
Figurre 1: Proportion o
of unit offerings faalling into each ‘trraffic light category’ by faculty, sem
mester 2 2012 3
Figurre 2: Proportion o
of unit offerings w
with 5 or more ressponses falling intto each ‘traffic ligh
ht category’ by faculty, semester 2 2012 4
Figurre 3: Proportion o
of unit offerings w
with less than 5 responses falling into each ‘traffic ligght category’ by faaculty, semester 2
2 2012 5
2. CAMPUS/FACULTY VARIATION Table 4 displays average median satisfaction (overall) scores across unit offerings, with data separated by campus and faculty. Two campus/faculty groups had average median scores in the ‘needing critical attention’ range (Bus & Eco at ‘Other Australian Location’ and Education at Sunway), although both had small unit and response counts. Six responses across two unit offerings formed the result for Bus & Eco at ‘Other Australian Location’, whereas two responses from a single unit produced the Education at Sunway result. Satisfaction was much higher for Information technology students at China‐ South East University, where the average median reached the ‘Outstanding’ criteria (4.82), on the basis of 60 responses across 4 offerings. Examining Campus alone (far right column) indicates a relatively high percentage of units with the critical attention flag at Parkville (14.3%) and Gippsland (12.7%). For all other campuses, less than 10% of units fell into the critical range. Examining Faculty alone (third bottom row of table 4), shows the average median for all faculties is in the ‘meeting aspirations’ range. 3. UNIT MODE VARIATION Table 5 displays average median satisfaction broken down by faculty and mode (on‐campus, off‐campus, or on/off campus). The majority of faculty/mode groups fell into the ‘meeting aspirations’ category. The off‐
campus Art Design and Architecture group (3.03), the on/off campus Arts group (3.43) and the on/off campus Medicine, Nursing and Health Science group (3.51) were exceptions, falling into the ‘needing improvement’ category. Collapsing across faculty groups (bottom 4 rows of table 5) shows that both on and off‐campus averages are in the ‘meeting aspirations’ range, while the on/off campus average is in the ‘needing improvement’ range. It also shows that units offered on campus received less negative attention (6.2% in the critical range) than either on/off campus units (25.9% in the critical range) or off‐campus units (17.9% in the critical range). 4. UNIT LEVEL VARIATION Table 6 shows average median satisfaction split by faculty and unit level. Level 6 in the Faculty of Arts, Design and Architecture appears concerning (average median = 1.59), although only 5 responses across two units comprised this result. In contrast, the average median satisfaction with level 6 units offered by the faculty of Business & Economics was well into the ‘outstanding’ range (4.92). Seven responses across three units formed this result. Collapsing across faculty (bottom 4 rows of table 6), shows little relationship between unit level and the average median, however the likelihood of a unit ‘needing critical attention’ appears correlated. For example, 12.1% of level 5 units met the critical attention criteria, compared with 4.6% of level 1 units. 6
Table 4: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and location, semester 2 2012 Location
Berwi ck
Measure
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Ca ul fi el d
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Cl a yton
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Gi pps l a nd Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Pa rkvi l l e
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Peni ns ul a Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
South Afri ca Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Sunwa y
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Pra to
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti cal a ttenti on
Os Chi na
Av. Medi a n
South Ea s t No. of uni t offeri ngs
Univers i ty No. needi ng cri ti cal a ttenti on
Other Av. Medi a n
Aus tra l i a n No. of uni t offeri ngs
Locati ons
No. needi ng cri ti cal a ttenti on
Other Av. Medi a n
Offs hore No. of uni t offeri ngs
Locati ons
No. needi ng cri ti cal a ttenti on
Overa l l Av. Medi an
Total no. of uni t offeri ngs
Total no. needi ng cri ti ca l a ttenti on
Art, Des Pharm & & Arch Arts Bus Eco Educ Eng Info Tech Law MNHS Pharm Sci
4.05
4.03
3.55
4.06
3.90
21
42
6
5
9
2
2
1
3.96
4.11
4.08
4.11
3.84
94
176
222
77
42
13
15
9
2
4
4.19
3.90
4.00 3.93
3.81 3.92
4.05
381
103
113
118
29
36
125
25
9
16
8
4
3
7
3.63
3.97
4.01
4.04 3.79
3.95
3.74
2
98
71
34
16
25
29
13
7
3
3
4
5
3.87
56
8
3.71
4.04
3.99
3.79
1
29
45
68
2
3
9
3.97
4.28
4.01
4.15
57
32
14
11
9
3.89
4.07
1.00 3.89
3.78 4.50
3.96
3.80
30
72
1
57
23
2
19
17
5
1
1
2
2
1
4.67
4.50
3.50
3
15
1
Sci
3.93
87
6
4.00
33
4
3.85
28
1
4.82
4
3.00
2
1
4.33
66
3
4.32
9
3.97
99
13
4.12
779
67
4.04
574
31
3.99
208
25
3.91
191
13
4.00
177
12
4.19
104
6
4.02
40
4
3.67
3
1
3.94
346
31
3.85
73
9
3.93
148
11
% needing critical Total attention
3.99
83
5
6.0%
4.06
611
43
7.0%
4.04
992
78
7.9%
3.96
308
39
12.7%
3.87
56
8
14.3%
3.90
143
14
9.8%
4.08
114
9
7.9%
3.92
249
13
5.2%
4.47
19
0.0%
4.82
4
0.0%
4.19
108
8
7.4%
4.16
12
1
8.3%
4.02
2699
218
8.1% 7
Table 5: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and Mode, semester 2 2012 Owning Faculty
Art, Des i gn & Archi tecture
Arts
Bus i nes s & Economi cs
Educa ti on
Engi neeri ng
Informa ti on Technol ogy
La w
Med Nurs i ng & Hea l th Sci
Pha rma cy & Pha rm Sci ence
Sci ence
Overa l l Av. Medi a n
Tota l No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
% needi ng cri ti ca l a ttenti on
Measure
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Off Campus On Campus
3.03
4.00
3
96
1
12
3.89
4.16
106
663
19
45
3.82
4.06
60
514
10
21
3.90
4.04
60
141
11
13
3.90
3.91
17
174
1
12
3.67
4.04
19
158
5
7
4.19
104
6
3.86
3.98
92
244
16
12
3.91
3.83
21
52
4
5
3.74
3.95
12
136
3
8
3.85
4.06
390
2282
70
17.9%
141
6.2%
On/Off Campus
3.53
27
Grand Total
3.97
99
13
4.12
779
67
4.04
574
31
3.99
208
25
3.91
191
13
4.00
177
12
4.19
104
6
3.94
346
31
3.85
73
9
3.93
148
11
4.02
2699
7
25.9%
218
8.1%
3.43
10
3
3.72
7
1
3.51
10
3
8
Table 6: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings needing critical attention by faculty and unit level, semester 2 2012 Owning Faculty
Art, Des i gn & Archi tecture
Arts
Bus i nes s & Economi cs
Educati on
Engi neeri ng
Informa ti on Technol ogy
La w
Med Nurs i ng & Heal th Sci
Pha rma cy & Pha rm Sci ence
Sci ence
Overa l l Av. Medi an
Total No. of uni t offeri ngs
No. needi ng cri ti ca l attenti on
% needi ng cri ti ca l a ttenti on
Measure
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
1
4.02
28
2
4.03
133
7
3.95
75
3
3.94
15
3.94
23
1
4.12
25
3.66
2
3.93
46
2
3.90
12
1
3.86
33
2
3.98
392
18
4.6%
2
3.88
30
6
4.11
222
18
3.96
134
5
3.88
17
3
3.68
43
5
3.97
41
1
4.03
3
3.79
56
5
3.92
14
3.94
53
3
3.97
613
46
7.5%
3
4.28
21
4.12
256
20
4.11
173
8
4.08
13
3.96
50
3
3.92
47
5
3.92
5
4
4.22
12
1
4.26
110
11
3.93
23
4
4.07
104
11
4.01
61
4
3.88
9
2
3.97
27
3
3.99
78
7
3.65
8
2
5
3.46
6
2
4.01
58
11
4.14
88
3
3.86
57
11
3.96
14
4.03
40
4
4.32
67
3
3.90
78
13
3.25
6
3
6
1.59
2
2
9
4.92
3
3.97
78
8
4.46
2
3.91
4.33
59
20
3
1
3.79
18
2
3.96
62
6
4.06 4.07
4.03 4.18
704
432
414
25
47
45
50
3
6.7% 10.4% 12.1% 12.0%
L
4.14
15
3.90
1
3.96
8
4.17
15
1
4.03
111
9
8.1%
3.96
8
0.0%
Grand Total
3.97
99
13
4.12
779
67
4.04
574
31
3.99
208
25
3.91
191
13
4.00
177
12
4.19
104
6
3.94
346
31
3.85
73
9
3.93
148
11
4.02
2699
218
8.1%
9
5.
CHANGES OVER TIME Figures 4 and 5 display the faculty‐level trends in unit evaluations between 2009 and 20125, with regard to the percentage of units classified as ‘outstanding’ (figure 4) or as ‘needing critical attention’ (figure 5). The percentage of unit offerings classified as ‘outstanding’ (all faculties combined) has remained fairly stable over the four years, with a slight decrease since 2011 (12.2% in 2012; 12.4% in 2011). At the faculty level, Law improved on already elevated figures (up to 22.3% ‘outstanding’ in 2012 from 19.2% in 2011). Additionally, improvement at Pharmacy and Pharmaceutical Sciences in 2012 (9.8% in 2012 following 8.2% in 2011) backed up sharp improvement in 2011 (8.2% in 2011 following 2.8% in 2010). The percentage of unit offerings classified as ‘needing critical attention’ (all faculties combined) has also remained stable (8% in 2011; 7.7% in 2012). At the faculty level, the past four years have seen a regular decline (i.e., improvement) in the percentage of units in the ‘needing critical attention’ range at Information Technology (16.6% in 2009 to 7.5% in 2012). Noticeable four year improvement has also occurred at the Arts faculty (11.4% in 2009 to 8% in 2012), although 2011 and 2012 results were the same. The reverse trend is evident at Law and Bus & Eco, where for both faculties the percentage of units meeting the ‘critical attention’ criteria has increased since 2010. The percentage of units in the critical range at the Education and the Medicine, Nursing and Health Sciences faculties has exceeded 10% every year since 2009. 5
2012 figures are not entirely complete; the 2012 summer semester data is not included. 10
Figurre 4: Percentage o
of evaluated unit offerings classifie
ed as 'outstandingg' by faculty, 2009
9‐2012. 1
11
Figurre 5: Proportion o
of evaluated unit o
offerings classified as 'needing critical attention' by faculty, 2009‐201
12 1
12