CALIFORNIA LUTHERAN UNIVERSITY ONLINE COURSE EVALUATION TEAM FACULTY MEMBERS Date and Location: Thursday, January 22, 2009 SBET 127 Time: 10:00am – noon; lunch provided Attendees: Julius Bianchi, Carol Coman, Bruce Gillies, Herb Gooch, Veronica Guerrero, Ed Julius, Halyna Kornuta, Henri Mondschein, Karissa Oien, Melinda Wright Regrets: Jim Bond, Deb Erickson, Paul Hanson, Myungsook Klassen, Paul Williams, Paul Witman MINUTES It is important to collect data. It is more important to review data. The ultimate is to use data to make data-informed decisions. A. Members were welcomed and introductions were offered a. Suggestions were made to also include Kris Butcher, Mike McCambridge, and Dru Pagliassotti to the next meeting. B. Reviewed Online Course Evaluations to date by offering link to Assessment Website a. http://www.callutheran.edu/assessment/resources/CourseEvaluations.php C. Discussed and responded to questions and issues raised by faculty (see below) a. Some information came from Sam Thomas’ minutes from the December Faculty meeting D. Report to faculty at the February Faculty meeting QUESTIONS AND ISSUES 1. Response Rate a. What are the response rates? b. What is a valid response rate? c. Investigate response strategies used by faculty d. What is the minimum number of responses needed for the data to be useable by ART? e. What is the profile of low RR classes? RESPONSES / ACTIONS a. See Chart – Suggested changes made in blue 1. To keep people informed of the current response rates, place the thermometer on the CLU home page and the CLU Portal – Matt Ward owns the web page b. Review of Literature Handout – Henri Mondschein 1. Melinda will add the handout links to the Assessment Website c. January 2009 Survey to 11 week fall session and 15 week fall semester faculty to learn about their strategies to inform / encourage students to complete forms 1. The survey was passed out for members to comment on and offer suggestions. 2. Melinda will make the suggested changes to the survey. d. This is a judgment call that ART has to make. 1|Page QUESTIONS AND ISSUES 2. 3. Administration Time a. When is the final two weeks of classes? The week of finals or the week before? Faculty Handbook states, "Student course evaluations are conducted during the final two weeks of classes" (Section Two, p. 26). Question: Does the “final two weeks of classes” include finals week or not? The Registrar’s office defines the last week of classes to include finals. Concern: If CoursEvals are distributed to include finals week, the final evaluation may influence students’ comments on CoursEval, for example, comments / scores will reflect how difficult the final may have been, etc. Some faculty return finals before evaluations are completed. b. When students drop, are they included in CoursEval? Course Evaluation items a. How does ART use the data received? b. Review of items? Example: 1) “uses highly effective teaching methods” 2) “The syllabus was…” [Disagree (1) Agree (5)] 3) “Compared to other courses, this course was…” [Excellent (1) Very Poor (5)] 4) Concern: the current procedure has too many variables and may not be statistically valid. 5) Rewrite items to better suit courses c. Currently, classes with < 5 students to not have a voice, for example, music instruction. Should they be included in some way? RESPONSES / ACTIONS a. Based on the Memo that was distributed with the paper and pencil evaluations, the majority of the people in attendance were under the impression that the final two weeks of classes does not include finals week. 1. Possible recommendation to take to the Provost: As a pilot study, distribute the Winter 09 evaluations during the two weeks before finals (weeks 9 & 10). b. Class lists are checked up until the day prior to administration. b. Bruce demonstrated a PowerPoint Presentation chart that showed the ADEP means for the first 16 items on the course evaluations. 1. “Uses highly effective teaching methods” is consistently low and is the only question to use the word “highly” 2. It is the assumption that students do not understand the meaning of the question 3. Possible actions to take to the T&L Committee: a. Remove the word “highly” from the question b. Remove the whole question 2|Page QUESTIONS AND ISSUES 4. Student Reminders a. How many email reminders are sent? b. Can there be a change in the bookmark text to indicate responses are used for promotion and tenure? What about adding the purpose of the evaluations? RESPONSES / ACTIONS a. Announcement/Invitation/4-5 Reminders b. Suggestions for changes to bookmark: 1. Simple, easy steps on the front. Add important, detailed information to the back 2. Instead of adding that the evaluations help with promotion, instead say that they are used to evaluate faculty. 3. Instead of the picture of the syllabus being “changed,” have it say “syllabus improved” 4. Karissa is going to type a draft of the new bookmark and send to the members to review c. The key to get the students to buy in to the new system lays with the instructors 5. Student Comments a. What does a text analysis of responses indicate? a. Karissa is going to organize a focus group with undergraduate students. Herb Example: Ratio of positive to negative mentioned doing the same for graduate students. b. Is there a difference between UG and G responses? If b. Idea Ed plans to do with his students: have students anonymously answer the question of whether or not they completed their Fall 08 course evaluations. If not, why? so, what is it? 6. CoursEval comparative data a. How do the results received during the pilot compare a. Communication 231 to results received from paper and pencil administration (review comparative classes with faculty permission)? b. Is there a relationship between grades received and evaluations given? c. http://www.callutheran.edu/assessment/resources/sept dec2008.php d. Should the charts continue to be part of the public d. Idea for another comparison chart: Compare adjunct responses to faculty. Bruce gave CLU website? permission to use ADEP as pilot. 7. CoursEval Faculty Report Data a. Legend defining abbreviations needed a. Exists – highlight in faculty video b. Terms not clearly defined (e.g. 1=Disagree, 5=Agree, 2, 3, 4 not defined) c. What does “E” (excellent) really mean? How is it different from “good”? 3|Page Augsburg Online Administration Dates Summer 2007 Fall 2007-UG Fall 2007-GR Spring 2008 Summer 2008 Student Response Rate 50% 48% 54% 41% 50% CLU Administration Date Courses Paper & Pencil Paper & Pencil Summer 2007 Fall 2007 Online Online Summer 2008 Fall 2008 11 week 11 week 15 week 11 week 11 week 15 week Student Response Rate 62.9% 77.7% 70% 50% 54% 62% # Students Surveyed 1416 8322 3255 7350 2152 # Students Surveyed 1072 1496 10434 1182 1426 10266 # Students Responded 674 1163 7353 587 764 6348 # Students Responded 715 3966 1771 3013 1084 Class Response Rate 73.7% 83.2% 83.2% 100% 100% 98.9% # Classes Surveyed 76 95 643 85 97 590 # Classes Responded 56 79 535 85 97 584 15 Week Course Evaluation completions by day: Prior to Finals Week 9 912 381 286 576 708 559 374 531 181 126 4643 Finals Week 188 444 307 361 352 53 1705 4|Page 11 Week Course Evaluation completions by day: Prior to Finals Week 24 49 46 55 53 49 38 78 25 22 Finals Week 40 31 53 46 54 20 14 61 6 5|Page 439 325 6|Page