STUDENT EVALUATION OF TEACHING AND UNITS (SETU) SUMMARY REPORT OF UNIT EVALUATION RESULTS SEMESTER 1, 2011 Dr Clare Hourigan 27 July 2011 INTRODUCTION After being piloted over the 2010‐2011 Summer period, the Student Evaluation of Teaching and Units (SETU) was rolled out across Monash for the first time in Semester 1 2011. The SETU replaces the previously separated unit evaluation and MonQueST surveys into one instrument. The original unit evaluation questions were included plus four new teaching questions.1 This report provides a summary of the results for the unit evaluation ‘overall satisfaction’ question. Students were asked to respond to the statement ‘Overall I was satisfied with the quality of this unit’. The response options were ‘Strongly Agree’ (5), ‘Agree’ (4), ‘Neutral’ (3), ‘Disagree’ (1)’ and ‘Strongly Disagree’ (1). From these data a ‘median’2 score has been calculated for each unit offering3. Unit offerings are then classified into one of four groups depending upon this median score: • Outstanding: ≥4.70 • Meeting aspirations: 3.60 – 4.69 • Needing improvement: 3.01 – 3.59 • Needing critical attention: ≤3.00 FINDINGS Unit evaluation overall satisfaction data was collected for 2,782 degree level4 unit offerings in Semester 1 2011. Note that more unit offerings were included in the SETU but only those with data for the overall satisfaction question have been included here. Unit offerings without overall satisfaction data are usually those that were included but had zero or very small enrolment numbers. 1. FACULTY VARIATION As shown in Table 1 and Figure 1, the vast majority of unit offerings (82.1%) were categorised as ‘meeting aspirations’ or ‘outstanding’, with 10% in the latter category. The results for the Arts and Law faculties were particularly strong with students rating 15.2% of the Arts unit offerings and 13.2% of those in Law with a median score of 4.7 or above. While overall satisfaction levels are generally high, 8.5% of unit offerings are in ‘need of critical attention’ after obtaining a score of 3 or less. Education, Engineering, Info Tech, MNHS and Pharm & Pharm Sci all had more than 10% of their unit offerings in this category. Table 2 and Figure 2 show the data for those unit offerings with 5 or more responses. The data for unit offerings with less than 5 responses can be seen in Table 3 and Figure 3. 1
Further detail on the SETU can be found at: http://www.opq.monash.edu.au/us/surveys/setu/index.html Further detail on how the median is calculated can be found at: http://www.opq.monash.edu.au/us/surveys/setu/setu_median_calculation.pdf 3
A unit offering is defined here as a unit at a specific location, in a specific mode (eg on campus/off campus) and offered for a specific calendar type. 4
Evaluations were also undertaken for Monash College diploma units and Sth Africa Foundation Program units but these are not included in this report. 2
1
Office of Planning and Quality – Statistics Unit
TABLE 1: NO. OF UNIT OFFERINGS FALLING INTO EACH "TRAFFIC LIGHT" CATEGORY BY FACULTY, SEMESTER 1 2011 Owning Faculty Median Response Band for Overall Satisfaction Item Needing Critical Attention (≤3.0) Art & Design Arts Business & Economics Education Engineering Information Technology Law Med Nursing & Health Sci Pharmacy & Pharm Science Science Grand Total Needing Improvement (3.01‐3.59) Meeting Aspirations (3.6‐4.69) Outstanding (≥4.7) Total 7 73 23 22 21 20 2 52 7 10 13 63 60 17 24 21 3 36 4 18 109 603 407 133 119 127 61 288 48 112 11 132 43 20 12 9 10 27 7 8 140 871 533 192 176 177 76 403 66 148 237 259 2007 279 2782 TABLE 2: NO. OF UNIT OFFERINGS WITH 5 OR MORE RESPONSES FALLING INTO EACH "TRAFFIC LIGHT" CATEGORY BY FACULTY, SEMESTER 1 2011 Owning Faculty Median Response Band for Overall Satisfaction Item Needing Critical Attention (≤3.0) Art & Design Arts Business & Economics Education Engineering Information Technology Law Med Nursing & Health Sci Pharmacy & Pharm Science Science Grand Total Needing Improvement (3.01‐3.59) Meeting Aspirations (3.6‐4.69) Outstanding (≥4.7) Total 4 31 10 16 19 13 1 23 4 5 12 46 51 14 24 16 3 25 3 15 94 425 373 124 111 111 60 215 41 108 9 59 25 16 6 2 7 12 3 6 119 561 459 170 160 142 71 275 51 134 126 209 1662 145 2142 2
TABLE 3: NO. OF UNIT OFFERINGS WITH LESS THAN 5 RESPONSES FALLING INTO EACH "TRAFFIC LIGHT" CATEGORY BY FACULTY, SEMESTER 1 2011 Owning Faculty Art & Design Arts Business & Economics Education Engineering Information Technology Law Med Nursing & Health Sci Pharmacy & Pharm Science Science Grand Total Median Response Band for Overall Satisfaction Item Needing Critical Attention (≤3.0) 3 42 13 6 2 7 1 29 3 5 111 Needing Improvement (3.01‐3.59) 1 17 9 3 Meeting Aspirations (3.6‐4.69) Outstanding (≥4.7) Total 11 1 3 15 178 34 9 8 16 1 73 7 4 2 73 18 4 6 7 3 15 4 2 21 310 74 22 16 35 5 128 15 14 50 345 134 640 5 3
100.0%
90.0%
80.0%
70.0%
% of unit offerings
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Art & Business & Arts Design Economics (n = 871)
(n = 140)
(n = 533)
Needing Critical Attention
5.0%
8.4%
4.3%
Information Education Engineering Technology Law (n = 76)
(n = 192)
(n = 176)
(n = 177)
11.5%
11.9%
11.3%
2.6%
Med Pharmacy & Science Grand Total Nursing & Pharm Health Sci Science (n = 148)
(n = 2782)
(n = 403)
(n = 66)
12.9%
10.6%
6.8%
8.5%
Needing Improvement
9.3%
7.2%
11.3%
8.9%
13.6%
11.9%
3.9%
8.9%
6.1%
12.2%
9.3%
Meeting Aspirations
77.9%
69.2%
76.4%
69.3%
67.6%
71.8%
80.3%
71.5%
72.7%
75.7%
72.1%
Outstanding
7.9%
15.2%
8.1%
10.4%
6.8%
5.1%
13.2%
6.7%
10.6%
5.4%
10.0%
FIGURE 1: PROPORTION OF UNIT OFFERINGS FALLING INTO EACH ‘TRAFFIC LIGHT CATEGORY’ BY FACULTY, SEMESTER 1 2011 4
Office of Planning and Quality – Statistics Unit
100.0%
90.0%
80.0%
70.0%
% of unit offerings
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Art & Business & Arts Design Economics (n = 561)
(n = 119)
(n = 459)
Needing Critical Attention
3.4%
5.5%
2.2%
Information Education Engineering Technology Law (n = 71)
(n = 170)
(n = 160)
(n = 142)
9.4%
11.9%
9.2%
1.4%
Med Pharmacy & Science Grand Total Nursing & Pharm Health Sci Science (n = 134)
(n = 2142)
(n = 275)
(n = 51)
8.4%
7.8%
3.7%
5.9%
Needing Improvement
10.1%
8.2%
11.1%
8.2%
15.0%
11.3%
4.2%
9.1%
5.9%
11.2%
9.8%
Meeting Aspirations
79.0%
75.8%
81.3%
72.9%
69.4%
78.2%
84.5%
78.2%
80.4%
80.6%
77.6%
Outstanding
7.6%
10.5%
5.4%
9.4%
3.8%
1.4%
9.9%
4.4%
5.9%
4.5%
6.8%
FIGURE 2: PROPORTION OF UNIT OFFERINGS WITH 5 OR MORE RESPONSES FALLING INTO EACH ‘TRAFFIC LIGHT CATEGORY’ BY FACULTY, SEMESTER 1 2011 5
100.0%
90.0%
80.0%
70.0%
% of unit offerings
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Art & Business & Arts Design Economics (n = 310)
(n = 21)
(n = 74)
Needing Critical Attention
14.3%
Information Education Engineering Technology Law (n = 5)
(n = 22)
(n = 16)
(n = 35)
13.5%
17.6%
27.3%
12.5%
20.0%
20.0%
Med Pharmacy & Science Grand Total Nursing & Pharm Health Sci Science (n = 14)
(n = 640)
(n = 128)
(n = 15)
22.7%
20.0%
35.7%
17.3%
Needing Improvement
4.8%
5.5%
12.2%
13.6%
0.0%
14.3%
0.0%
8.6%
6.7%
21.4%
7.8%
Meeting Aspirations
71.4%
57.4%
45.9%
40.9%
50.0%
45.7%
20.0%
57.0%
46.7%
28.6%
53.9%
Outstanding
9.5%
23.5%
24.3%
18.2%
37.5%
20.0%
60.0%
11.7%
26.7%
14.3%
20.9%
FIGURE 3: PROPORTION OF UNIT OFFERINGS WITH LESS THAN 5 RESPONSES FALLING INTO EACH ‘TRAFFIC LIGHT CATEGORY’ BY FACULTY, SEMESTER 1 2011
6
2. CAMPUS VARIATION Table 4 shows the average of the median overall satisfaction scores obtained by the unit offerings. The data are disaggregated by location and faculty. On average, the unit offerings within each of the faculties fell into the ‘meeting aspirations’ category with the faculty average medians ranging from 3.83 for MNHS to 4.21 for Law. The same can be said for the locations ranging from 3.7 at the ‘Other Offshore Locations’ to 4.29 at Prato. There were only three areas in which the unit offerings, on average ‘needed improvement’: • Engineering at Gippsland • MNHS at Gippsland • MNHS at the Other Offshore Locations. There was also one offshore Information Technology unit offering that fell into the ‘needing critical attention’ category with a median score of only 2. Further investigation showed that this unit only had one enrolment. The locations with more than 10% of evaluated unit offerings needing critical attention are highlighted in red. These were Berwick, Gippsland, Parkville, Peninsula and those classified as ‘Other Offshore Location’. 3. UNIT MODE VARIATION Table 5 shows the average of the median overall satisfaction scores obtained by the unit offerings disaggregated by faculty and mode. There were only three areas in which the unit offerings on average ‘needed improvement’: • Off campus in Art & Design • Off Campus in Information Technology • Combined units in MNHS. However units offered off campus (17%) or through a combined on/off mode (13.3%) were much more likely that those offered on campus to be obtain a score of 3 or below and therefore be classified as ‘needing critical attention’. UNIT LEVEL VARIATION Table 6 shows the average of the median overall satisfaction scores obtained by the unit offerings disaggregated by faculty and unit level. There were only two areas in which the unit offerings on average ‘needed improvement’: • Level 1 in Information Technology • Level 4 in Pharmacy and Pharmaceutical Science. Also the three level 5 unit offerings in Pharmacy and Pharmaceutical Science had an average median score in the ‘needing critical attention’ category along with the one level 5 unit offering in Science. While there does not appear to be a strong relationship between level and the likelihood of a unit ‘needing critical attention’ just over 10% of the units at level 4 and 5 (10.7% and 11.1%) were in this category. 7
Office of Planning and Quality – Statistics Unit
TABLE 4: AVERAGE ‘OVERALL SATISFACTION’ MEDIAN SCORE, AND NUMBER AND % OF UNIT OFFERINGS NEEDING CRITICAL ATTENTION BY FACULTY AND LOCATION, SEMESTER 1 2011 Art & Design
Bus Eco
Arts
Educ
Eng
Info Tech
Law
Measure
Av. Medi a n
No. of uni t offeri ngs
4.00
1
3.86
30
3.84
37
3.69
11
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
4.05
4
4.10
2
4.02
3
3.97
3.83
9
4.02
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
125
6
159
13
216
7
77
6
66
6
643
38
Av. Medi a n
No. of uni t offeri ngs
4.46
1
4.12
479
3.95
95
3.95
107
3.93
106
3.76
27
4.07
35
3.96
131
3.97
93
4.03
1074
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
34
3.92
5
3.80
15
3.88
12
3.38
3
3.74
2
3.94
13
3.59
6
3.86
90
3.82
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
13
1
114
13
69
7
30
4
14
4
24
3
35
6
30
4
329
42
Cl a yton
Gi pps l a nd
Pa rkvi l l e
4.25
1
3.88
54
3.89
55
7
3.85
3.97
3.71
7
3.82
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
3
24
1
45
3
79
15
151
19
Other Av. Medi a n
Aus tra l i a n No. of uni t offeri ngs
Loca ti on
No. needi ng cri ti ca l a ttenti on
Sunwa y
Av. Medi a n
4.10
7
4.36
2
3.84
58
4.04
102
3.84
4.00
3.80
3.71
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
27
4
63
1
56
5
3.98
52
4.28
27
South Afri ca Av. Medi a n
No. of uni t offeri ngs
Pra to
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
4.35
35
5
9
3.87
3.75
3.79
19
2
13
12
25
3.71
18
3.87
10
4.00
107
1
4.29
8
4.29
6
6
2
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Other Offs hore Loca ti on
9
3.85
215
12
11.4%
5.9%
8.4%
12.8%
12.7%
12.6%
8.8%
5.6%
7.5%
0.0%
Av. Medi a n
No. of uni t offeri ngs
4.13
10
2.00
1
3.43
10
3.70
21
No. needi ng cri ti ca l a ttenti on
Overa l l Av. Medi a n
4.04
4.06
3.97
3.96
3.85
1
3.82
4.21
2
3.83
3.86
3.92
3
3.96
Tota l no. of uni t offeri ngs
Tota l no. needi ng cri ti ca l a ttenti on
140
7
871
73
533
23
192
22
176
21
177
20
76
2
403
52
66
7
148
10
2782
237
% needing critical attention
3.83
79
Av. Medi a n
No. of uni t offeri ngs
No. needi ng cri ti ca l a ttenti on
Av. Medi a n
Total
4.13
Peni ns ul a
Sci
Location
Berwi ck
Ca ul fi el d
Pharm & Pharm MNHS
Sci
14.3%
8.5% 8
TABLE 5: AVERAGE ‘OVERALL SATISFACTION’ MEDIAN SCORE, AND NUMBER AND % OF UNIT OFFERINGS NEEDING CRITICAL ATTENTION BY FACULTY AND MODE, SEMESTER 1 2011 On/Off Campus Grand Total Art & Design Av. Median No. of unit offerings No. needing critical attention 3.47 6 1 4.06 134 6 Arts Av. Median No. of unit offerings No. needing critical attention 3.85 108 19 4.09 750 54 Business & Economics Av. Median No. of unit offerings No. needing critical attention 3.68 62 10 4.01 471 13 Education Av. Median No. of unit offerings No. needing critical attention 3.87 60 9 4.00 121 10 Engineering Av. Median No. of unit offerings No. needing critical attention 3.83 5 3.85 171 21 3.85 176 21 Information Technology Av. Median No. of unit offerings No. needing critical attention 3.37 24 7 3.90 153 13 3.82 177 20 Law Av. Median No. of unit offerings No. needing critical attention 4.21 76 2 4.21 76 2 Med Nursing & Health Sci Av. Median No. of unit offerings No. needing critical attention 3.83 138 22 3.85 245 27 3.51 20 3 3.83 403 52 Pharmacy & Pharm Science Av. Median No. of unit offerings No. needing critical attention 3.93 19 3 3.82 46 4 4.13 1 3.86 66 7 Science Av. Median No. of unit offerings No. needing critical attention 3.63 13 3 3.95 135 7 Overall Av. Median 3.79 4.00 3.86 3.96 Total No. of unit offerings 435 2302 45 2782 74 157 6 237 17.0%
6.8%
13.3% 8.5%
% needing critical attention On Campus Measure No. needing critical attention Off Campus Owning Faculty 4.04 140 7 4.24 13 4.06 871 73 3.97 533 23 3.99 11 3 3.96 192 22 3.92 148 10 9
TABLE 6: AVERAGE ‘OVERALL SATISFACTION’ MEDIAN SCORE, AND NUMBER AND % OF UNIT OFFERINGS NEEDING CRITICAL ATTENTION BY FACULTY AND UNIT LEVEL, SEMESTER 1 2011 Owning Faculty Measure 1 2 3 4 5 4.07 4.07 6 4.13 3.86 3.80 44 30 2 23 4 8 1 Art & Design Av. Median No. of unit offerings No. needing critical attention Arts Business & Economics Av. Median No. of unit offerings No. needing critical attention 4.01 3.97 4.08 4.31 148 7 267 26 273 25 124 6 Av. Median 3.92 3.88 3.99 4.25 4.07 71 122 145 21 84 No. needing critical attention 2 4 5 1 Education Av. Median No. of unit offerings No. needing critical attention 4.09 4.20 3.78 15 12 1 14 2 Engineering Information Technology Av. Median No. of unit offerings No. needing critical attention 3.64 3.74 3.82 23 5 40 4 50 7 Av. Median 3.53 3.75 3.91 3.88 28 41 50 No. needing critical attention 7 3 Av. Median No. of unit offerings No. needing critical attention 3.82 4.13 Law Med Nursing & Health Sci No. of unit offerings Av. Median No. of unit offerings Av. Median No. of unit offerings No. needing critical attention Science No. of unit offerings No. needing critical attention Pharmacy & Pharm Science 33 Av. Median No. of unit offerings No. needing critical attention 1 4.04 1 140 7 4.02 4.06 59 9 871 73 3.96 3.94 3.97 2 88 533 4 7 23 3.91 3.95 4.23 3.96 78 8 69 11 4 192 22 3.99 4.16 3.85 7 176 21 3.88 4.13 3.82 7 37 14 177 4 1 4 1 20 4.05 4.11 5.00 4.21 1 76 2 7 56 5 26 2 4.31 4.75 Grand Total L 4.00 2 9 39 1 3.92 3.64 3.84 3.74 3.85 4.35 3.98 3.88 3.83 43 53 64 99 118 19 4 3 403 2 11 3 19 15 3.75 4.00 3.81 3.56 13 14 13 2 1 2 52 2.58 4.18 3.86 6 3 17 66 1 2 1 7 3.84 4.00 3.95 1.00 3.92 36 3 46 1 65 5 1 1 148 10 Overall Av. Median 3.91 3.91 3.99 4.02 3.97 4.36 4.00 3.88 3.96 Total No. of unit offerings 411 641 711 440 425 23 128 3 2782 28 51 53 47 47 2 No. needing critical attention % needing critical attention 9 6.8% 8.0% 7.5% 10.7% 11.1% 8.7% 7.0% 0.0% 237 8.5% 10
4. CHANGES OVER TIME Figures 4 and 5 compare the results of the Semester 1 2011 unit evaluations with those obtained in 2009 and 2010 (full year) to see how the proportion of unit offerings classified as ‘outstanding’ or ‘needing critical attention’ has changed over that time. For the most part, the proportion of unit offerings classified as ‘outstanding’ has remained fairly stable over that time. However, in Pharmacy and Pharmaceutical Sciences the proportion has jumped markedly from just under 3% to over 10%. At the other end of the spectrum though, this faculty has also experienced a rise in the proportion of unit offerings classified as ‘needing critical attention’ (6.6% in 2010 to 10.6% in Semester 1 2011). Overall, the proportion of unit offerings classified as ‘needing critical attention’ has also remained fairly stable after a slight drop from 9.9% to 8.3% between 2009 and 2010. Over the past three years there has been a steady decline in these unit offerings in Arts (11.5% in 2009 down to 8.4% in Sem 1 2011) and within Art & Design the proportion has halved from 10.1% in 2010 to 5% this year. 11
25.0%
% of unit offerings 'outstanding'
20.0%
15.0%
10.0%
5.0%
0.0%
Art & Design
Arts
Business & Economics
Education
Engineering
Information Technology
Law
Med Nursing & Health Sci
Pharmacy & Pharm Science
Science
Total
2009 ‐ Full Year
9.3%
18.5%
7.0%
11.2%
4.8%
7.7%
15.5%
6.8%
2.7%
6.9%
11.2%
2010 ‐ Full Year
11.1%
19.3%
7.2%
12.0%
3.4%
5.1%
13.5%
8.2%
2.8%
6.2%
11.5%
2011 ‐ Sem 1
7.9%
15.2%
8.1%
10.4%
6.8%
5.1%
13.2%
6.7%
10.6%
5.4%
10.0%
FIGURE 4: PROPORTION OF EVALUATED UNIT OFFERINGS CLASSIFIED AS ‘OUTSTANDING’ BY FACULTY, 2009‐2011 12
Office of Planning and Quality – Statistics Unit
25.0%
% of unit offerings 'needing critical attention'
20.0%
15.0%
10.0%
5.0%
0.0%
Art & Design
Arts
Business & Economics
Education
Engineering
Information Technology
Law
Med Nursing & Health Sci
Pharmacy & Pharm Science
Science
Total
2009 ‐ Full Year
9.3%
11.5%
4.5%
13.8%
12.0%
16.6%
1.5%
10.3%
7.3%
5.6%
9.9%
2010 ‐ Full Year
10.1%
9.6%
4.5%
10.4%
6.4%
10.1%
1.4%
11.7%
6.6%
4.3%
8.3%
2011 ‐ Sem 1
5.0%
8.4%
4.3%
11.5%
11.9%
11.3%
2.6%
12.9%
10.6%
6.8%
8.5%
FIGURE 5: PROPORTION OF EVALUATED UNIT OFFERINGS CLASSIFIED AS ‘NEEDING CRITICAL ATTENTION’ BY FACULTY, 2009‐2011 13