CENTRAL WASHINGTON UNIVERSITY WINTER 2015 STUDENT EVALUATIONS OF INSTRUCTION (SEOIs) A COMPARISON OF RESPONSES FROM FACE-TO-FACE, INTERACTIVE VIDEO, AND ONLINE COURSES I. II. III. IV. V. Abstract ……………………………………………………………………………………………….. Executive Summary …………………………………………………………………………..…. Background…………………………………………………………………………………………… Overall SEOI Results …………………………………….…………………….…………….…… A. Ratings of Learning …………………………………..……………………………….. B. General Information ………………………………………………….……………… Summary …………………………………………………………………………………………….. Appendix 1. Terms, Acronyms, and Abbreviations ……………………………….. Appendix 2. SEOI Response Rates by Time and Form by Department …… Tom Henderson, Director of Institutional Assessment March 25, 2015 2. 2. 2. 4. 4. 6. 7. 8. 9. I. ABSTRACT All CWU course evaluations were administered online during the winter 2015 quarter. This report compares the evaluations of students enrolled in courses taught face-to-face in traditional classrooms (F2F), via synchronous interactive video (ITV), and “online” or to students enrolled in courses taught 100% online. The main research questions this report addresses are: How well do CWU students rate their instruction? Do course evaluations indicate any major differences in learning outcomes between courses taught 100% online, via interactive video, or lecture courses taught in person? II. EXECUTIVE SUMMARY CWU students’ ratings of instruction during the winter 2015 quarter were, on average, very positive. Student evaluations of online courses and interactive video courses are similar to the ratings of students in traditional face-to-face courses. ITV courses had several medians lower than F2F or online courses. However, all differences in averages were substantively small. III. BACKGROUND During the spring 2012 quarter Central Washington University administered all course evaluations online for the first time. New forms developed by an SEOI Task Force were also used institution-wide for the first time. The course evaluation forms for online and face-to-face courses ask many questions that have similar wording. Mock ups of the course evaluation forms can be found at: Form A - Lecture Form C - Skills Acquisition Form D - Laboratory Form E - Visual and Performing Arts Form F - Field Experience / Student Teaching Form W - Online An SEOI web site has information about the online course evaluation process and links to the final SEOI Task Force report: http://www.cwu.edu/seoi. Many of the questions on CWU’s course evaluations use a five point Likert scale arranged in a table format (a visual analog scale). For this analysis a response of “Strongly agree” is assigned a value of 5. A response of “Strongly disagree” is assigned a value of 1 (see “Likert scale” in appendix 1). See Table 5 on page 6 for all Likert values. The CWU-wide response rates decreased from the fall 2012 term. The spring 2013 term was the first quarter that SEOIs were administered for just one week before finals (as opposed to two weeks before finals in prior quarters). Another major difference was that faculty could no longer download a list of Winter 2015 Course Evaluation Summary p. 2 of 14 students who did or did not respond to SEOIs. The decrease may also be due to CWU’s switch from Blackboard to Canvas. Canvas originally did not allow pop-up windows asking students to complete course evaluations. Pop-up windows were implemented in Canvas during the fall 2014 term. This may account for the increase in response rates during the fall, 2014 and winter, 2015 terms. Table 1 - Response Rates to SEOIs by Form by Term with Length of Administration Spring Fall Winter Spring Fall Winter Spring Fall Winter 2012 2012 2013 2013 2013 2014 2014 2014 2015 2 weeks 2 weeks 2 weeks 1 week 1 week 2 weeks 2 weeks 2 weeks 2 weeks Form A - Lecture 56% 57% 51% 41% 44% 44% 33% 39% 40% Form C - Skills Acquisition 48% 48% 41% 33% 39% 36% 31% 37% 33% Form D - Lab 55% 51% 45% 37% 46% 38% 32% 37% 30% Form E - Visual & Perform. Arts 49% 45% 37% 27% 38% 32% 27% 28% 24% Form F - Field Experience N/A N/A N/A 53% 51% 56% 53% 36% 36% Form W - Online 62% 64% 62% 53% 46% 49% 43% 47% 53% Overall Average 56% 56% 50% 41% 44% 43% 34% 39% 40% The average SEOI ratings don’t seem to be affected by administering them one week before finals or two weeks prior to finals. Table 2 shows the weighted average ratings for the first two question banks on the SEOIs: “Student Learning Environment” and “Teaching for Student Learning.” Note, this report compares averages between Form A – Lecture (face-to-face and interactive video) to Form W – Online. Table 2 – CWU Institution-wide Averages to First Two Question Banks on SEOI Forms Form A - Lecture Student Learning Environment Teaching for Student Learning Form C - Skills Acquisition Student Learning Environment Teaching for Student Learning Form D - Laboratory Student Learning Environment Teaching for Student Learning Form E - Visual & Performing Arts Student Learning Environment Teaching for Student Learning Form F - Field Experience Student Learning Environment Teaching for Student Learning Form W - Online Student Learning Environment Teaching for Student Learning S2012 F2012 W2013 S2013 F2013 W2014 S2014 F2014 W2015 2 weeks 2 weeks 2 weeks 1 week 1 week 2 weeks 2 weeks 2 weeks 2 weeks 4.5 4.5 4.5 4.4 4.5 4.5 4.4 4.5 4.5 4.3 4.3 4.3 4.2 4.3 4.3 4.3 4.3 4.3 4.5 4.4 4.5 4.3 4.5 4.4 4.5 4.3 4.6 4.4 4.5 4.3 4,5 4.4 4.5 4.4 4.5 4.4 4.5 4.4 4.5 4.4 4.5 4.3 4.5 4.3 4.5 4.3 4.6 4.4 4.5 4.3 4.5 4.3 4.5 4.4 4.6 4.5 4.6 4.5 4.5 4.4 4.5 4.4 4.5 4.4 4.5 4.4 4.6 4.4 4.6 4.5 4.6 4.5 NA NA NA NA NA NA 4.7 4.5 4.5 4.4 4.6 4.6 4.6 4.5 4.3 4.2 4.5 4.5 4.3 4.3 4.4 4.3 4.4 4.3 4.3 4.3 4.3 4.3 4.3 4.3 4.3 4.3 4.4 4.4 4.4 4.3 Winter 2015 Course Evaluation Summary p. 3 of 14 IV. OVERALL RESULTS A. RATINGS OF LEARNING Table 3 summarizes the average ratings by college to the first two Likert question banks on Form A – Lecture. The average ratings across colleges for these two question banks are very similar. The percent of responses by college to different delivery methods varies quite a bit. For example, the College of Business has the lowest enrollment of the colleges but by far the highest percent of responses to Interactive Video courses. The college does not appear to have much of an effect on the average ratings of the first two question banks. Table 3 – Form A Lecture Average Ratings by College to the First Two Likert Question Banks Fall Winter Spring Collegge of Arts & Humanities 2013 2014 2014 Student Learning Environment 4.5 4.5 4.5 Teaching for Student Learning 4.3 4.3 4.3 College of Business Student Learning Environment 4.4 4.4 4.4 Teaching for Student Learning 4.2 4.2 4.4 College of Education & Professional Studies Student Learning Environment 4.5 4.5 4.4 Teaching for Student Learning 4.3 4.5 4.3 College of the Sciences Student Learning Environment 4.4 4.4 4.4 Teaching for Student Learning 4.2 4.2 4.3 CWU-wide Average College of the Sciences 4,5 4,5 4,4 Student Learning Environment 4.3 4.3 4.3 Fall Winter 2014 2015 4.4 4.5 4.3 4.3 4.4 4.2 4.5 4.3 4.4 4.2 4.5 4.3 4.5 4.3 4.4 4.2 4,5 4.3 4.5 4.3 Table 4 points out that the “mix” of responses from the colleges varies quite a bit by delivery mode. However, the average ratings for questions relating to “student learning environment” and “teaching for student learning” are substantively the same. The different ratios could have an effect on General Question averages in Table 7. Winter 2015 Course Evaluation Summary p. 4 of 14 Table 4 – Winter 2015 Percent of Responses by Colleges for Face-to-Face (F2F), Interactive Video (ITV), and Online Students F2F CAH CB CEPS COTS Other CAH CB CEPS COTS Other Replies 21% 8% 25% 42% 4% 10,181 ITV 4% 59% 11% 27% 0% 335 Online 8% 9% 54% 26% 2% 3,034 Combined 18% 10% 31% 38% 4% 13,550 CWU student evaluations of instruction (SEOI) ratings continue to be very positive. All of the median ratings for the first “learning” questions are 5.0 for both face-to-face and online courses, indicating that at least 50% of the students gave the highest possible ratings. Table 6 highlights all medians to questions 1, 3, and 4 that are less the 5.0. All medians below 5.0 are from ITV courses. Table 6 also presents a comparison of average ratings for face-to-face, interactive-video, and online courses. Only the questions that have the same or similar text on both SEOI forms (Form A – Lecture and Form W – Online) are listed and compared. All of the differences in average ratings were substantively small. The differences in the average ratings between face-to-face and interactive video courses are all small as measured by Hedge’s g (a measure of “distance” between averages). The differences in the average ratings between face-to-face and online courses are all small as measured by Hedge’s g (a measure of “distance” between averages). Note: Hedge’s g “effect size” is a method of quantifying the substantive difference or “distance” between the means of two groups. Effect sizes with an absolute value of 0.3 or less are often considered “small,” 0.5 are “medium” and 0.8 or higher are large. Hedge’s g was used because of the large differences in sample sizes between F2F, ITV, and online courses. Table 5 – Values Assigned to Likert Scales on questions 1 through 5 for this Analysis A Likert scale was used on questions one through five, values were assigned as follows. -5Strongly agree -4Agree -3Neutral -2Disagree Winter 2015 Course Evaluation Summary -1Strongly disagree p. 5 of 14 IT V F2 F M ed i Winter 2015 SEOI Results Face-to-Face (F2F) vs. Interactive Video (ITV) vs. 100% Online an s M ed i a O ns nl in e M ed ia F2 ns F Av er x ag IT es V Av er ag es O nl in e Av er ag H ed es ge s' s x g: H F2 ed F ge vs 's .I g: TV F2 D F if f v er s. en O ce nl in in e F2 D F if f vs er en .I TV ce in F2 F vs .O nl in e Table 6 - Medians, Averages, and Comparison of Average Ratings of students enrolled in Face-toFace, Interactive Video, and Online Courses 1 - STUDENT LEARNING ENVIRONMENT To what extent do you instructor fostered a fair and respectful 1.a./1.a. 5.0 5.0 learning environment? instructor seemed genuinely concerned with 1.b./1.b. 5.0 5.0 whether students learned? standards of online behavior were clearly 1.c./1.c. 5.0 5.0 communicated and enforced? agree or disagree that the… x 5.0 4.5 4.4 4.5 0.15 5.0 4.4 4.2 4.4 5.0 4.4 4.3 4.4 small small 0.16 (0.01) small small 0.14 (0.05) small small If YES, did the instructor provide help? 5.0 5.0 5.0 4.4 4.2 4.3 0.19 3 2. TEACHING FOR STUDENT LEARNING To what extent do you agree or disagree that the…. Medians less than 5.0 are highlighted 4.a./4.a. course objectives were clearly communicated? 5.0 5.0 5.0 4.3 4.2 4.4 overall course content was presented in an 4.b./4.b. 5.0 4.0 5.0 4.3 4.1 4.3 understandable sequence? instructor used a variety of methods, as 4.c./4.c. 5.0 4.0 5.0 4.2 4.1 4.2 needed, to make content clear? assignments and tests were connected to 4.d./4.e. 5.0 5.0 5.0 4.4 4.4 4.5 course content? evaluation and grading techniques were 4.e./4.f. 5.0 4.0 5.0 4.2 4.2 4.3 clearly explained? instructions for class activities were clearly 4.f./4.g. 5.0 4.0 5.0 4.3 4.1 4.3 communicated? instructor provided useful feedback on student 4.g./4.h. 5.0 4.0 5.0 4.1 4.0 4.2 work? instructor provided timely feedback on student 4.h./4.i. 5.0 4.0 5.0 4.2 4.0 4.3 progress? class sessions (online activities) were well 4.i./4.j. 5.0 4.0 5.0 4.3 4.1 4.3 organized? out-of-class (online) work was useful in 4.j./4.k. 5.0 4.5 5.0 4.3 4.1 4.3 understanding course content? instructor encouraged students to connect 4.k./4.l. 5.0 5.0 5.0 4.3 4.3 4.3 course content to issues beyond classroom? course activities challenged students to think 4.l./4.m. 5.0 5.0 5.0 4.4 4.3 4.4 critically? GENERAL INFORMATION How would you compare this course with other courses of similar credits at this level (i.e., amount of work OUTSIDE of class / online 5.a./8.a. 3.0 3.0 3.0 3.4 3.4 3.5 environment level of engagement/active learning IN class / 5.b./8.b. 3.0 3.0 3.0 3.5 3.4 3.4 online environment 5.c./8.c. intellectual challenge presented to you 3.0 4.0 3.0 3.5 3.6 3.5 0.03 small small 0.09 (0.08) small small 0.12 (0.05) small small 0.12 (0.04) small small 0.08 (0.06) small small 0.04 (0.09) small small 0.19 0.05 small small 0.15 (0.07) small small 0.18 (0.08) small small 0.17 (0.03) small small 0.12 (0.05) small small 0.08 0.10 0.01 0.01 small small (0.04) small small 100, 200, etc.) taken at CWU? (0.07) (0.09) small small 0.09 0.12 small small (0.06) 0.04 small small B. GENERAL INFORMATION On average, students enrolled in face-to-face courses have lower class standing, study fewer hours per week, and take more courses required for General Education credits. Students’ grades expectations are similar whether they take courses face-to-face, via interactive video, or online. See Table 7. Winter 2015 Course Evaluation Summary p. 6 of 14 Table 7 - General Information Averages and Percentage Responses by Students in Face-to-Face, Interactive Video, and Online Courses. Major differences are highlighted in gray. For this class, about how many hours outside of class did you spend in a typical 7-day week studying, reading, conducting research, writing, doing homework or lab work, analyzing data, rehearsing, and other academic activities? Responses 0 1-3 4-6 7-10 11-15 16-20 21+ Est. Average* Face-to-Face 9,441 2% 33% 36% 18% 8% 2% 1% 5.7 hrs. per week Interactive Video 333 1% 20% 34% 24% 12% 5% 3% 7.4 hrs. per week 100% Online 3,015 0% 16% 33% 25% 14% 8% 3% 8.2 hrs. per week * averages were estimated from the mid points of each category Why did you take this course? Please select all that apply. Responses Major Minor Cert. GenEd FacRep Time/Online Interest Other Face-to-Face 13,317 60% 9% 4% 27% 9% 9% 18% 4% Interactive Video 469 73% 15% 2% 5% 9% 12% 19% 6% 100% Online 4,508 69% 11% 4% 12% 5% 25% 18% 5% What is your class standing? Responses 1st Yr = 1 2nd Yr = 2 3rd Yr = 3 4th Yr = 4 Grad = 5 Other Average Face-to-Face 9,384 20% 17% 29% 28% 4% 2% 2.7 between 2nd and 3rd year Interactive Video 329 1% 1% 38% 47% 9% 5% 3.5 between 3rd and 4th year 100% Online 2,993 3% 5% 34% 45% 11% 2% 3.5 between 3rd and 4th year What grade do you expect to earn in this course? Responses A=4 B=3 C=2 D=1 F=0 Other Average Face-to-Face 9,413 49% 37% 11% 1% 0% 1% 3.3 Interactive Video 331 50% 36% 11% 2% 1% 1% 3.3 100% Online 2,999 59% 31% 7% 1% 0% 1% 3.5 It is interesting that the three instructional methods have different average class standings, differences in the ratio of courses from colleges, and difference in course delivery methods but still have strong and similar results in course ratings. V. SUMMARY A. How did CWU students evaluate their instruction during the winter 2015 term? On average, student evaluations at CWU are very positive on all questions relating to “Student Learning Environment” and “Teaching for Student Learning.” All median ratings for online and students in tradition face-to-face courses were 5.0, the highest possible rating. ITV courses had, in general, lower median scores. However, the differences in average scores when compared to online or face-to-face courses were substantively small. B. Do course evaluations indicate any major differences in learning outcomes between courses taught face-to-face, via interactive video, or online? No – not when students were asked questions about “Student Learning Environment” or “Teaching for Student Learning.” C. CWU should focus on improving response rates to online student evaluations of instruction. Winter 2015 Course Evaluation Summary p. 7 of 14 APPENDIX 1 Terms, Acronyms, and Abbreviations CAH CB CEPS COTS Hedge’s g College of Arts and Humanities College of Business College of Education and Professional Studies College of the Sciences A statistical measure of “effect size.” “Effect size is a method of quantifying the difference between two groups that has some advantages over the use of tests of statistical significance alone.” A pooled standard deviation is used as the denominator in this analysis. A pooled standard deviation is called for when the sample sizes of the variables studied are significantly different. See this paper on effect size: http://www.leeds.ac.uk/educol/documents/00002182.htm F2F “Face-to-Face” refers to courses taught in traditional classrooms with students face-to-face with an instructor Frequently asked questions Form A is the SEOI form developed for students to evaluate lecture courses that meet face-to-face. Form W is the SEOI (or course evaluation) form developed for students in courses taught 100% online using Blackboard. Courses taught via synchronous, interactive video. Many of the questions on CWU’s course evaluations use a five point Likert scale arranged in a table format (a visual analog scale). For this analysis a response of “Strongly agree” is assigned a value of 5. A response of “Strongly disagree” is assigned a value of 1. Whether individual Likert items can be considered as interval-level data, or whether they should be considered merely orderedcategorical data is the subject of disagreement. This analysis assumes that the responses can be considered as interval data, especially since the Likert scale items are arranged in a visual analog format, have “balanced” levels, and a five point scale. The average or mean is a parametric statistic that is robust to ordinal data. Hedge’s g may tend to overstate distances for ordinal data. Mid-point of ranked data. It is a statistical measure of central tendency. For example if the median is 5 on a five point scale, at least 50% of students answered 5. Courses taught 100% online via Blackboard or Canvas, usually asynchronously. “Student Evaluation of Instruction” CWU’s term for course evaluations An abbreviation for sample “standard deviation.” FAQ Form A Form W ITV Likert scale Median Online SEOI St Dev Winter 2015 Course Evaluation Summary p. 8 of 14 APPENDIX 2 WINTER 2015 SEOI RESPONSE RATES BY TIME-OF-DAY, FORM, AND BY DEPARTMENT All Data Bars use a Scale of 0% to 100% Winter 2015 Course Evaluation Summary p. 9 of 14 WINTER 2015 SEOI SUBMISSION TIME-OF-DAY FOR F2F, ITV, AND ONLINE (W) STUDENTS The X-axis scale uses a 24 hour clock 10% 9% 8% 7% 6% 5% 4% 3% 2% 1% 0% F2F% 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 24:00 1% 0% 0% 0% 0% 0% 1% 3% 5% 7% 6% 9% 8% 7% 8% 7% 6% 6% 5% 6% 5% 4% 3% 2% Online% 1% 0% 0% 0% 0% 1% 2% 4% 5% 6% 5% 8% 8% 7% 7% 6% 5% 7% 6% 6% 6% 4% 3% 2% ITV% 1% 1% 0% 1% 1% 1% 3% 7% 7% 4% 7% 10% 5% 6% 6% 6% 7% 4% 6% 5% 4% 4% 2% 2% Winter 2015 Course Evaluation Summary p. 10 of 14 WINTER 2015 FORM A – LECTURE Department MLS COM ECTL SHP CDS MUS NEHS TH ABS LC TEAC CWU CS ART GEOG CHEM MATH CNED PHYS PE AIS LAJ MANA WS HIST ANTH Sent Replied 119 21 568 146 11 3 11 3 32 9 472 134 1,165 334 689 206 47 15 336 108 543 175 135 44 578 193 110 39 636 236 911 340 1,803 673 37 14 338 128 612 233 54 21 834 326 559 219 51 20 445 175 652 260 Response % Data Rate Bars 18% 26% 27% 27% 28% 28% 29% 30% 32% 32% 32% 33% 33% 35% 37% 37% 37% 38% 38% 38% 39% 39% 39% 39% 39% 40% SEOI RESPONSE RATES BY DEPARTMENT Department STEP FCS SOC ACSK BIOL FVS IET EDFC ENG LLSE PHIL ACCT GEOL ECON PSY FNLA SCED POSC FINO AVIA AST PRIM DHC ITAM AFRO REM Sent 25 806 731 228 988 26 972 612 1,521 779 640 734 437 458 1,489 501 102 260 584 337 25 23 143 240 82 27 Winter 2015 Course Evaluation Summary Replied 10 331 304 95 413 11 412 261 661 344 283 327 195 205 668 229 47 121 280 163 14 13 91 181 70 25 Response % Data Rate Bars 40% 41% 42% 42% 42% 42% 42% 43% 43% 44% 44% 45% 45% 45% 45% 46% 46% 47% 48% 48% 56% 57% 64% 75% 85% 93% p. 11 of 14 WINTER 2015 FORM C - SKILLS ACQUISITION Department NEHS FVS COM PE TH IET CHEM ENG CS SCED ANTH ITAM Sent 18 39 313 1,310 225 105 6 18 35 13 5 322 WINTER 2015 FORM D - LAB Department MLS COM ANTH TH NEHS PHYS CHEM BIOL GEOL EDFC GEOG AFRO SEOI RESPONSE RATES BY DEPARTMENT Replied 2 5 76 361 63 30 2 6 12 6 3 236 Response % Data Rate Bars 11% 13% 24% 28% 28% 29% 33% 33% 34% 46% 60% 73% SEOI RESPONSE RATES BY DEPARTMENT Sent Replied 22 1 26 3 25 5 15 3 560 139 258 66 654 193 288 86 189 61 150 57 22 12 73 68 Response % Data Rate Bars 5% 12% 20% 20% 25% 26% 30% 30% 32% 38% 55% 93% Winter 2015 Course Evaluation Summary p. 12 of 14 WINTER 2015 FORM E – VISUAL & PERFORMING ARTS Department MUS TH PE ART Sent 1,126 136 91 359 SEOI RESPONSE RATES BY DEPARTMENT Response % Data Replied Rate Bars 219 19% 34 25% 28 31% 126 35% WINTER 2015 FORM F – FIELD EXPERIENCE SEOI RESPONSE RATES BY DEPARTMENT Department Sent Replied TEACH 38 9 EDFC 83 33 LLSE 38 16 Response % Data Rate Bars 24% 40% 42% Winter 2015 Course Evaluation Summary p. 13 of 14 WINTER 2015 FORM W – ONLINE Department NEHS GEOG GEOL FINO COM HIST LAJ AVIA ECON TH FVS CS MANA CDS EDFC PSY SOC FCS AST IET MATH LLSE ENG ACCT AIS CWU CNED PE ITAM POSC PHIL ADPR ANTH SEOI RESPONSE RATES BY DEPARTMENT Sent Replied 94 22 117 31 28 8 39 14 161 60 24 9 342 133 28 11 106 42 50 21 47 20 21 9 428 196 28 13 142 67 494 243 407 201 119 59 24 12 45 23 59 31 51 27 164 89 51 28 14 8 170 98 80 47 255 150 1,952 1,206 42 26 36 23 128 88 22 19 Response % Data Rate Bars 23% 27% 29% 36% 37% 38% 39% 39% 40% 42% 43% 43% 46% 46% 47% 49% 49% 50% 50% 51% 53% 53% 54% 55% 57% 58% 59% 59% 62% 62% 64% 69% 86% Winter 2015 Course Evaluation Summary p. 14 of 14