STUDENT EVALUATION OF TEACHING AND UNITS (SETU) SUMMARY REPORT OF UNIT EVALUATION RESULTS SEMESTER 2, 2013 19 December 2013 INTRODUCTION The Student Evaluation of Teaching and Units (SETU) is an integrated version of two previous surveys; the unit evaluation and MonQueST. SETU includes the original unit evaluation questions and four new teaching questions.1 This report summarises the semester 2 2013 responses to SETU’s ‘overall satisfaction’ item. The overall item required students to rate the statement; ‘Overall I was satisfied with the quality of this unit’, with either ‘Strongly Agree’ (5), ‘Agree’ (4), ‘Neutral’ (3), ‘Disagree’ (2)’ or ‘Strongly Disagree’ (1). From these data, a ‘median’2 score was calculated for each unit offering3. Unit offerings were then classified into one of the four groups using the following median score cut-offs: Outstanding: ≥4.70 Meeting aspirations: 3.60 – 4.69 Needing improvement: 3.01 – 3.59 Needing critical attention: ≤3.00 FINDINGS Overall satisfaction data was collected for 2,654 degree level4 unit offerings in Semester 2, 2013. Note that this figure only includes offerings with overall satisfaction responses. 1. FACULTY VARIATION Table 1 and Figure 1 both show that the vast majority of unit offerings were categorised as either ‘meeting aspirations’ (72.0%) or ‘outstanding’ (13.4%) and the highest performing faculty was Law (23.9% of units ‘outstanding’). Notwithstanding high overall satisfaction levels, 6.9% of unit offerings were classified as ‘needing critical attention’. Among all faculties, Information Technology (9.7%) and Medicine Nursing & Health Sciences (9.3%) had a relatively high percentage of units falling into this category. Table 2 and Figure 2 show the results for unit offerings with 5 or more responses. Results for unit offerings with less than 5 responses are in Table 3 and Figure 3. 1 Further detail on the SETU can be found at: http://www.opq.monash.edu.au/us/surveys/setu/index.html Further detail on how the median is calculated can be found at: http://www.opq.monash.edu.au/us/surveys/setu/setu_median_calculation.pdf 3 A unit offering is defined here as a unit at a specific location, in a specific mode (e.g. on campus/off campus). It does not take into account instances where the unit offering may run in different calendar types within the same survey period or different unit classes within the same mode ie. DAY, EVENING (both ON campus mode). Also on-shore non-campus locations are grouped into ‘Other Australian Locations’ and off-shore non-campus locations are grouped into ‘Other Overseas locations’. 4 Evaluations were also undertaken for Monash College diploma, Malaysia National Subject units and Sth Africa Foundation Program units but these are not included in this report. 2 1 Office of Planning and Quality – University Planning and Statistics Table 1: No. of unit offerings falling into each "traffic light" category by faculty, Semester 2 2013 Median Response Band for Overall Satisfaction Item Owning Faculty Art Des & Architecture Arts Business & Economics Education Engineering Information Technology Law Med Nursing & Health Sci Pharmacy & Pharm Science Science Grand Total Needing Critical Attention (≤3.0) Needing Improvement (3.01-3.59) Meeting Aspirations (3.64.69) Outstanding (≥4.7) Total 7 43 27 19 11 17 5 35 6 13 183 8 42 42 13 24 14 2 38 16 6 205 70 497 408 158 147 122 82 259 48 119 1,910 18 120 60 29 15 22 28 43 5 16 356 103 702 537 219 197 175 117 375 75 154 2,654 Table 2: No. of unit offerings with 5 or more responses falling into each "traffic Light" Category by Faculty, Semester 2 2013 Median Response Band for Overall Satisfaction Item Owning Faculty Art Des & Architecture Arts Business & Economics Education Engineering Information Technology Law Med Nursing & Health Sci Pharmacy & Pharm Science Science Grand Total Needing Critical Attention (≤3.0) Needing Improvement (3.01-3.59) Meeting Aspirations (3.64.69) Outstanding (≥4.7) Total 6 12 10 13 9 8 2 22 3 10 95 7 28 32 11 22 12 2 24 13 6 157 64 409 365 141 140 101 68 213 41 111 1,653 10 60 35 23 8 11 15 20 3 10 195 87 509 442 188 179 132 87 279 60 137 2,100 Table 3: No. of unit offerings with less than 5 responses falling into each "traffic Light" Category by faculty, Semester 2 2013 Median Response Band for Overall Satisfaction Item Owning Faculty Art Des & Architecture Arts Business & Economics Education Engineering Information Technology Law Med Nursing & Health Sci Pharmacy & Pharm Science Science Grand Total Needing Critical Attention (≤3.0) Needing Improvement (3.01-3.59) Meeting Aspirations (3.64.69) Outstanding (≥4.7) Total 1 31 17 6 2 9 3 13 3 3 88 1 14 10 2 2 2 0 14 3 0 48 6 88 43 17 7 21 14 46 7 8 257 8 60 25 6 7 11 13 23 2 6 161 16 193 95 31 18 43 30 96 15 17 554 2 Figure 1: Proportion of unit offerings falling into each ‘traffic light category’ by faculty, Semester 2 2013 3 Figure 2: Proportion of unit offerings with 5 or more responses falling into each ‘traffic light category’ by faculty, Semester 2 2013 4 Figure 3: Proportion of unit offerings with less than 5 responses falling into each ‘traffic light category’ by faculty, Semester 2 2013 5 2. CAMPUS/FACULTY VARIATION Table 4 displays average median satisfaction (overall) scores across unit offerings, with data separated by campus and faculty. There were no campus/faculty groups which had average median scores in the ‘needing critical attention’ range. The only campus/faculty group reached ‘Outstanding’ criteria for the average median score was Arts at ‘Other Australian Locations’ (5.00), although it had only one unit with one response count. It also noted that the average median score for Information Technology at Malaysia (3.58) fell into ‘needing improvement’ range, with 5 of the 24 units reached ‘needing critical attention’ range. Examining Campus alone (far right column) indicates a relatively high percentage of units with the critical attention flag at Parkville (10.2%) and Gippsland (15.0%). For all other campuses, less than 10% of units fell into the critical range. Examining Faculty alone (third bottom row of table 4), shows the average median for all faculties is in the ‘meeting aspirations’ range. 3. UNIT MODE VARIATION Table 5 displays average median satisfaction broken down by faculty and mode (on-campus, off-campus, or on/off campus). The majority of faculty/mode groups fell into the ‘meeting aspirations’ category. The ‘off campus’ for Information Technology group (3.56) was the exception, falling into the ‘needing improvement’ category. Collapsing across faculty groups (bottom 4 rows of table 5) shows that average median scores from all three modes (on, off, on/off-campus) are in the ‘meeting aspirations’ range. However, ‘off campus’ had a relatively higher percentage of units falling into ‘needing critical attention’ range (15.6%) than ‘on campus’ (5.4%) and ‘on/off campus’ (8.3%). 4. UNIT LEVEL VARIATION Table 6 shows average median satisfaction split by faculty and unit level. The average median satisfaction with level 6 units offered by the faculty of Business & Economics was well into the ‘outstanding’ range (5.00), with eight responses across the two units forming the result. In addition, level 4 in Science may need attention for the falling of average median score (3.55) into ‘needing improvement’ category. Collapsing across faculty (bottom 4 rows of table 6), it shows the average median score from each unit level is in the ‘Meeting Aspirations’ range, and all levels obtained a percentage of units in ‘needing critical attention’ at or below 10%. 6 Table 4: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings ‘needing critical attention’ by faculty and location, Semester 2 2013 Location Berwick Measure Av. Median No. of unit offerings No. needing critical attention Caulfield Av. Median No. of unit offerings No. needing critical attention Clayton Av. Median No. of unit offerings No. needing critical attention Gippsland Av. Median No. of unit offerings No. needing critical attention Parkville Av. Median No. of unit offerings No. needing critical attention Peninsula Av. Median No. of unit offerings No. needing critical attention South Africa Av. Median No. of unit offerings No. needing critical attention Malaysia Av. Median No. of unit offerings No. needing critical attention Prato Av. Median No. of unit offerings No. needing critical attention Os China Av. Median South East No. of unit offerings University No. needing critical attention Other Av. Median Australian No. of unit offerings Locations No. needing critical attention Other Av. Median Offshore No. of unit offerings Locations No. needing critical attention Overall Av. Median Total no. of unit offerings Total no. needing critical attention Art, Des & Arch Arts 4.19 25 1 4.09 4.18 97 150 7 10 4.29 327 6 3.86 96 15 4.33 1 4.61 6 3.98 62 7 3.84 31 4 4.46 9 Bus Eco Educ Eng 3.99 4.02 37 14 3 2 4.10 203 6 4.05 4.01 4.00 95 106 124 2 11 6 3.96 4.00 3.74 69 32 16 9 3 3 4.06 28 4 4.17 32 1 4.00 73 2 Info Tech 4.18 76 3 3.93 35 3 3.83 22 6 4.08 47 2 4.03 16 4.13 3.91 3 57 2 Pharm & MNH Pharm Law S Sci 3.84 17 1 3.94 53 7 4.11 4.03 42 117 4 5 3.99 26 3 3.90 59 6 3.96 65 8 4.13 12 3.58 4.64 24 8 5 3.97 22 3.70 16 Sci 4.06 93 7 3.96 32 5 3.92 28 1 4.62 2 5.00 1 4.12 103 7 4.16 702 43 4.05 537 27 4.23 17 1 4.04 3.95 219 197 19 11 4.31 67 1 3.72 63 11 4.00 4.26 175 117 17 5 3.94 375 35 4.50 1 3.86 75 6 4.02 154 13 % needing critical Total attention 4.02 93 7 7.5% 4.11 579 33 5.7% 4.12 939 44 4.7% 3.91 293 44 15.0% 3.90 59 6 10.2% 4.02 141 14 9.9% 4.05 122 8 6.6% 3.92 262 14 5.3% 4.52 15 0.0% 4.62 2 0.0% 4.04 132 12 9.1% 4.23 17 1 5.9% 4.06 2654 183 6.9% 7 Table 5: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings ‘needing critical attention’ by faculty and Mode, Semester 2 2013 Owning Faculty Art, Design & Architecture Arts Business & Economics Education Engineering Information Technology Law Med Nursing & Health Sci Pharmacy & Pharm Science Science Overall Av. Median Total No. of unit offerings No. needing critical attention % needing critical attention Measure Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Off Campus On Campus 4.12 103 7 3.95 4.19 93 598 14 29 3.73 4.09 54 483 10 17 3.95 4.09 59 149 6 11 3.87 3.96 17 180 1 10 3.56 4.05 18 157 7 10 4.26 117 5 3.88 3.95 105 256 15 19 3.99 3.81 22 53 3 3 3.81 4.03 10 144 3 10 3.88 4.09 378 2240 59 15.6% 121 5.4% On/Off Campus 4.13 36 Grand Total 4.12 103 7 4.16 702 43 4.05 537 27 4.04 219 19 3.95 197 11 4.00 175 17 4.26 117 5 3.94 375 35 3.86 75 6 4.02 154 13 4.06 2654 3 8.3% 183 6.9% 4.24 11 3.92 11 2 4.22 14 1 8 Table 6: Average ‘Overall Satisfaction’ median score, and number and % of unit offerings ‘needing critical attention’ by faculty and unit level, Semester 2 2013 Owning Faculty Art, Design & Architecture Arts Business & Economics Education Engineering Information Technology Law Med Nursing & Health Sci Pharmacy & Pharm Science Science Overall Av. Median Total No. of unit offerings No. needing critical attention % needing critical attention Measure Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention Av. Median No. of unit offerings No. needing critical attention 1 4.08 26 2 4.10 131 3 3.97 75 4 4.08 17 3.85 23 2 3.98 24 2 3.82 2 4.03 46 3.79 12 4.10 34 1 4.03 390 14 3.6% 2 4.03 29 4 4.16 223 14 3.94 129 8 4.05 18 1 3.85 46 3 3.92 40 3 3.99 3 3.73 58 9 3.80 16 1 3.94 53 6 4.00 615 49 8.0% 3 4.27 25 1 4.18 238 15 4.08 160 5 4.15 17 1 3.97 52 4 4.01 44 4 4.11 10 1 3.90 70 7 3.86 18 1 4.07 62 4 4.08 696 43 6.2% 4 5 4.14 4.21 18 3 0 4.23 4.11 73 35 6 5 4.40 4.13 25 77 2 4 4.06 3.98 108 59 9 8 4.05 3.97 62 14 2 3.82 4.04 9 41 2 4 4.22 4.32 30 72 2 2 4.11 3.87 62 111 3 16 3.71 3.81 7 7 1 2 3.55 5 2 4.12 4.05 399 419 29 41 7.3% 9.8% 6 3.50 1 0 3.75 2 9 4.14 1 5.00 2 4.07 69 4 4.31 2 4.11 15 2 4.35 15 0 4.26 5 L 4.11 8 4.08 15 1 4.31 22 4.09 105 7 6.7% 4.11 8 Grand Total 4.12 103 7 4.16 702 43 4.05 537 27 4.04 219 19 3.95 197 11 4.00 175 17 4.26 117 5 3.94 375 35 3.86 75 6 4.02 154 13 4.06 2654 183 6.9% 9 5. CHANGES OVER TIME Figures 4 and 5 display the faculty-level trends in unit evaluations between 2010 and 2013, with regard to the percentage of units classified as ‘outstanding’ (Figure 4) or as ‘needing critical attention’ (Figure 5). It is worth noting that the average median satisfaction scores in Figure 4 and 5 have been adjusted with regard to the following changes: Adding of Summer Semester data in 2010 and 2011 to reflect the true full year results Removing non-standard unit offerings originated from Malaysia National Subjects and Enhancement Studies Program. The percentage of unit offerings classified as ‘outstanding’ (all faculties combined) has increased gradually since 2010 (Grand Total column: 11.4% in 2010; 12.4% in 2011; 12.3% in 2012; 13.4% in 2013). At the faculty level the Art Deigns & Architecture shows a sharp improvement (18.4% in 2013 following 13% in 2012). Decreases are noted in 2013 results in Arts (19.3% in 2010 dropping to 17.3% in 2013) and Pharmacy & Pharmaceutical Sciences (9.8% in 2012 decreasing to 7.2% in 2013). In addition, improvement is also seen in all other faculties in 2013. The percentage of unit offerings classified as ‘needing critical attention’ (all faculties combined) has also gradually decreased (8.3% in 2010, 8.2% in 2011; 7.8% in 2012, 6.9% in 2013). At faculty level, Arts has shown a graduate decline in the percentage of units in the ‘needing critical attention’ range since 2010 (9.4% in 2010 to 6.6% in 2013). In addition, signs of improvement are observed at Art Design & Architecture, Business & Economics, Education, Law and Pharmacy & Pharmaceutical Sciences, where the percentage of units in the ‘needing critical attention’ range has decreased in 2013. On the other hand, the percentage of units in critical range has increased in four faculties: Engineering, Information Technology and Medicine, Nursing and Health Sciences. The percentage of units in Medicine, Nursing and Health Sciences faculties has exceeded 10% every year since 2010. 10 Figure 4: Percentage of evaluated unit offerings classified as 'outstanding' by faculty, 2010-20135, 6 5 Adjustment 1: 2010 to 2011 data has been adjusted to include Summer Semester data. Adjustment 2: Non-standard unit offerings originated from ‘Malaysia National Subjects’ and ‘Enhancement Studies Program’ were excluded. 6 2012 and 2013 figures include full year data, i.e. Semester 1, Semester 2 and Summer Semester data. 11 Figure 5: Proportion of evaluated unit offerings classified as 'needing critical attention' by faculty, 2010-20135, 6 12