Learning Outcomes Assessment
advertisement
North Texas Library Assessment Group From Micro to Macro: Correlating Instruction with Student Learning & Success through Local and National Initiatives Zoltán Szentkirályi, Director of Assessment Hollie Gardner, User Experience Librarian Central University Libraries Southern Methodist University Spring Meeting 2015 March 20, 2015 Student Outcomes Research GWLA Learning Outcomes Conference & Workshop November 13-15, 2013 at UNLV, Las Vegas 33 GWLA member universities, each encouraged to send a team of three people to discuss SLO’s. SMU Team: “Manager in Charge of Library Instruction Program” Elizabeth Killingsworth “Assessment Representative” Zoltán Szentkirályi “Instruction Librarian” Hollie Gardner Student Outcomes Research GWLA Greater Western Library Alliance The Greater Western Library Alliance (GWLA) is a consortium of 33 research libraries located in the central and western United States. Arizona State University Baylor University Brigham Young University Colorado State University Iowa State University Library Kansas State University Oklahoma State University Oregon State University Rice University Southern Illinois University Southern Methodist University Texas A&M University Texas Tech University University of Arizona University of Arkansas University of Colorado at Boulder University of Hawai'i at Mānoa University of Houston University of Illinois at Chicago University of Kansas University of Missouri University of Nevada, Las Vegas University of New Mexico University of Oklahoma University of Oregon University of Southern California The University of Texas at Austin University of Utah University of Washington University of Wyoming Utah State University Washington State University Washington University in St. Louis Student Outcomes Research GWLA Student Learning Task Force Initial Charge Established Spring 2011 Goals for the GWLA Student Learning Task Force: To investigate and share local evaluation/assessment practices and evidence obtained, across GWLA institutions To create a research question and identify ways to assess students, using all GWLA universities as a possible sample population Student Outcomes Research GWLA Student Learning Task Force First Phase of the Charge To investigate and share local evaluation/assessment practices and evidence obtained, across GWLA institutions 2011 – 2014 Report Compiled and Submitted Student Outcomes Research GWLA Student Learning Task Force Second Phase of the Charge To investigate and share local evaluation/assessment practices and evidence obtained, across GWLA institutions To create a research question and identify ways to assess students, using all GWLA universities as a possible sample population Spring 2014 - Call For Participants for Research Project Student Outcomes Research GWLA Libraries Impact on Student Learning Task Force Research Project started in 2014 Melissa Bowles-Terry Head of Educational Initiatives, UNLV Libraries Chair of Student Learning Outcomes Task Force Patty Iannuzzi, Dean of UNLV Libraries, Dean Liaison Joni Blake, Executive Director GWLA, GWLA Liaison 32 Task Force Members from 20 Participating Institutions GWLA Student Learning Taskforce Student Outcomes Research GWLA Student Learning Research Project Roster of Participating Institutions & Representatives GWLA Student Learning Research Project & Task Force Institution Name Arizona State University Lisa Kammerlocher Baylor University John Bales Brigham Young University Suzanne Julian Brigham Young University Holt Zaugg GWLA Anne McKee GWLA Joni Blake Kansas State University Sara Kearns Kansas State University Melia Fritch Oregon State University Cheryl Middleton Oregon State University Laurie Bridges Southern Methodist University Zoltan Szentkiralyi Texas A&M University Stephanie Graves Texas Tech University Laura Heinz University of Arkansas Norma Johnson University of Hawaii Vicky Lebbin University of Houston Christina Gola University of Illinois-Chicago Paula Dempsey University of Illinois-Chicago Annie Armstrong University of Missouri Goodie Bhullar University of Missouri Navadeep Khanal University of Nevada, Las Vegas Melissa Bowles-Terry University of Nevada, Las Vegas Patty Iannuzzi University of Nevada, Las Vegas Greg Voelker University of New Mexico Mark Emmons University of Southern California Jade Winn University of Southern California Carolyn Caffrey Gardner University of Utah Lorelei Rutledge University of Utah Alfred Mowdood University of Washington Jackie Belanger University of Washington Amanda Hornby Utah State University Britt Fagerheim Washington State University Steve Borrelli Position Social Sciences Librarian Liaison Librarian Library Instruction Coordinator Assessment Librarian Program Officer for Resource Sharing Executive Director Head of Undergraduate and Community Services Instruction Team Lead Associate University Librarian for Learning & Engagement Instruction and Emerging Technologies Librarian Director of Assessment Coordinator of Learning & Outreach Head of Research, Instruction & Outreach Reference Librarian Social Sciences Librarian Head of Liaison Services for Instruction & Outreach Research Services & Resources Librarian Coordinator of Bibilographic Instruction Head of Instructional Services E-Learning Librarian Head of Educational Initiatives Dean Resources Workflow Manager Director of Instruction & Assessment Assistant Dean for Instruction, Assessment and Engagement Info Literacy & Educational Technology Librarian Research and Information Services Librarian Head of Research and Information Services Assessment Coordinator Teaching and Learning Program Librarian Head of Reference Services Strategic Assessment Librarian GWLA Student Learning Taskforce Student Outcomes Research Timeline for Research Project May 2014 - Task Force starts meeting every 2 weeks Research Design subgroup – prepare research design Grants subgroup – seek & prepare grant applications Fall 2014 – Institutions begin tracking Library Instruction activity, per the specifications of the Research Design Spring 2015 – each participating institution work with their Institutional Research office to make sure they can get student data, and submit IRB application for the study GWLA Student Learning Taskforce Student Outcomes Research Timeline for Research Project Fall 2015 – Request Institutional Data, e.g. student grades & retention, per the specifications of the Research Design Fall 2015 – Spring 2016 Merge Library Instruction Data w/ Institutional Data for all 20 participating institutions, data cleaning, analysis and report of first year results Leverage first year results to secure grant and continue research project as longitudinal study over the next 4-6 years GWLA Student Learning Taskforce Student Outcomes Research Research Design Two-Tiered Approach Library Instruction Variables - (teaching methods) Logs of instruction sessions Librarian contribution to design Methods and characteristics Institutional Variables - (student success) Student Demographics Courses, Credit Hours, Grades GPA and Retention GWLA Student Learning Taskforce Student Outcomes Research Library Instruction Variables INDIVIDUAL SESSIONS (data collection tool): Session by session level data collection for library instruction activity, entered by librarian/library-instructor after a session. ● ● ● ● ● ● ● ● ● ● Course number (#) associated with this library instruction session Course section (#) associated with this library instruction session Instructor (last, first) – In most cases, a non-librarian. Session Librarian/Library-Instructor (last, first) – Name of person providing the library instruction for this session. The person listed as Session Librarian/Library-Instructor is: ○ A faculty or staff librarian with a library degree ○ A faculty or staff without a library degree ○ A student or graduate assistant who has not yet completed a library degree Date of this library instruction session Duration of this library instruction session in minutes Number of students present during this library instruction session The class meeting format for this library instruction session was: ○ Online only ○ In-person only Teaching Method for this library instruction session: ○ Lecture only – a presentation and/or demonstration, with or without the help of projection of the active website, powerpoint slides, handouts, etc., with students listening/watching but not actively practicing the methods. ○ Flipped – Students were assigned material to complete in advance (modules, videos, tutorials, assignments, etc.), then followed by library instruction which covers the material in greater depth or covers other additional material ○ Directed practice – Students followed along on their computers and performed GWLA ○ A faculty or staff without a library degree ○ A student or graduate assistant who has not yet completed a library degree Student Outcomes Research ● Date of this library instruction session Student Learning Taskforce ● Duration of this library instruction session in minutes ● Number of students present during this library instruction session ● The class meeting format for this library instruction session was: ○ Online only ○ In-person only ● Teaching Method for this library instruction session: ○ Lecture only – a presentation and/or demonstration, with or without the help of projection of the active website, powerpoint slides, handouts, etc., with students listening/watching but not actively practicing the methods. ○ Flipped – Students were assigned material to complete in advance (modules, videos, tutorials, assignments, etc.), then followed by library instruction which covers the material in greater depth or covers other additional material ○ Directed practice – Students followed along on their computers and performed tasks, e.g. using certain search terms and strategies suggested by the librarian. ○ Active Learning/Assignments – Students worked in groups or individually to complete in-depth activities and tasks assigned by librarian instructions or to capture learning. ○ Other_________________________ ○ None – there was no instruction for this session, only orientation or tour. ● One-shot instruction session (yes/no) – Was this session a one-time library instruction for this class, with no other meeting planned? ● Course-integrated (yes/no) – Was the library instruction for this session integrated with, or directly related to, the course content? ● Session Librarian helped design content or assignment (yes/no) – Session Librarian collaborated with instructor to develop the content or assignment related to this instruction session. Library Instruction Variables GWLA Student Learning Taskforce Student Outcomes Research Library Instruction Variables 2 SYSTEM-WIDE CHARACTERISTICS: Library instruction program-level data that may apply broadly across courses or sections at the institution, which can be added to the data afterthe-fact by someone knowledgeable at each institution, e.g. instruction coordinator and/or GWLA task force member librarian, prior to submitting. Confirm or fill in Course/Section level data collection (compiled and cross-checked by library representative): ● Campus or university ○ Sub level Campus (for institutions that wish to report on more than one campus) ● Course number (#) ● Course section (#) ● Instructor (last, first) – In most cases, a non-librarian. ● Librarian (last, first) – If a single librarian is associated with this course/section. If the librarian is the only instructor of record, put librarian’s name as both instructor and librarian. ● Librarian helped design course (yes/no) – A librarian helped develop the overall course assignments and/or syllabus through collaboration with the instructor of record for the course. ● Librarian helped design course/section only (yes/no) – A Librarian collaborated with the instructor of record for this section to develop the assignments for the course/section. ● Librarian helped design assignment for course/section (yes/no) – Librarian collaborated with instructor to develop at least one major credit-bearing assignment for the course/section. ● Online Library Tutorial or Learning Objects (yes/no) – Did this course/section have an online library tutorial or learning objects that were integrated as assignment(s) or GWLA ● Librarian helped design course (yes/no) – A librarian helped develop the overall course assignments and/or syllabus through collaboration with the instructor of record for the Student Outcomes Research course. Student Learning Taskforce ● Librarian helped design course/section only (yes/no) – A Librarian collaborated with the instructor of record for this section to develop the assignments for the course/section. ● Librarian helped design assignment for course/section (yes/no) – Librarian collaborated with instructor to develop at least one major credit-bearing assignment for the course/section. ● Online Library Tutorial or Learning Objects (yes/no) – Did this course/section have an online library tutorial or learning objects that were integrated as assignment(s) or assessment(s)? ● Course/section is co-taught by librarian/instructor team (yes/no) – A librarian and at least one non-librarian are both instructors of record for the course/section. ● Credit-bearing library class, taught by librarian(s) (yes/no) – Librarian(s) is/are the only instructor(s) of record. ○ Number of units or credit hours (#) ○ Full-credit course or partial-credit course ○ Quarter, semester, or other Library Instruction Variables 2 ANALYST: course/section level variables created from the individual library instruction session data logged by the librarians. Counts of sessions per, librarians per, combinations of teaching methods used, confirming one-shot status for a course/section (i.e. not more than one session was logged), general data groupings and data management based on data that was logged. ● Count of number of class meetings with a librarian (#) – The number of times a librarian met with the course/section as a class. ● Count of number of individual librarians/library-instructors that interacted with this course/section (#) ● Confirm: One-shot instruction for this course/section (yes/no) – Did only one (1) library instruction session occur for this course/section? ● Confirm: The class meeting format for this course/section was: ○ In-person only sessions (yes/no) – The IL instruction session(s) happened inperson only, and never online. ○ Online only sessions (yes/no) – The IL instruction session(s) happened online only, and never in-person. GWLA Student Learning Taskforce Student Outcomes Research Institutional Variables Student's unique de-identified code (permanently assigned for longitudinal purposes) Student's Gender, Ethnicity, Class Level (fr, so, jr, sr), Age, ESL required Student's total # of credits earned (cumulative at 1st year, 2nd year, 3rd year, etc.) Student's Fall 2014 to Fall 2015 Retention designation (Yes/No) Student's Fall 2014 term GPA, # credits attempted & completed Student's Winter 2015 term GPA (if applicable), # credits attempted & completed Student's Spring 2015 term GPA, # credits attempted & completed Student's Summer 2015 term GPA (if applicable) , # credits attempted & completed Student's First-Year GPA (2014-2015 Academic Year), # credits attempted & completed All Courses Taken by the Student during the 2014-2015 Academic Year: Course Number & Department, Course Section, Course Name, Term taken, Course Grade GWLA Student Learning Taskforce Student Outcomes Research Data Merged by Course-Section Library Instruction: Course-Sections taught Teaching Method Used First Year, First Time Students Student Grades by Course-Section Student Retention after the first year All student & library data de-identified in order to obtain IRB approval GWLA Student Learning Taskforce Student Outcomes Research Sample Library Instruction Data Collection Form GWLA Student Learning Taskforce Student Outcomes Research Sample Library Instruction Data Library Instructor 1 Session Date Session Duration Participants (in minutes) Professor / Subject Instructor (Lastname, Course Subject Course Firstname) Number Course Section libr_nam sessdate sessdura students inst_nam crs_subj crs_numb crs_sect SMU01 10/9/2014 90 18 Harvell DISC - Discernment 1313 & Discourse 003 SMU01 7/14/2014 60 15 Grumbles (Blackman) DISC - Discernment 1313 & Discourse 0012 SMU01 6/17/2014 60 8 Lange DISC - Discernment 1313 & Discourse 0011 SMU02 9/16/2014 80 17 Hall, Alane DISC - Discernment 1313 & Discourse 022 SMU02 9/16/2014 80 17 Hall, Alane DISC - Discernment 1313 & Discourse 023 SMU02 9/29/2014 60 17 Kolberg, Stephanie DISC - Discernment 1313 & Discourse 028 SMU02 9/29/2014 60 17 Kolberg, Stephanie DISC - Discernment 1313 & Discourse 032 SMU02 9/29/2014 60 17 Kolberg, Stephanie DISC - Discernment 1313 & Discourse 033 SMU02 10/2/2014 80 17 Mueller, Mary Katherine DISC - Discernment 1312 & Discourse 055 SMU02 10/2/2014 80 15 Mueller, Mary Katherine DISC - Discernment 1312 & Discourse 056 SMU02 10/2/2014 80 16 Mueller, Mary Katherine DISC - Discernment 1312 & Discourse 057 SMU02 10/9/2014 30 20 Umobong, Rebecca HDEV - Human 1110 Development 006Studies SMU02 10/9/2014 80 16 Newton, Pauline DISC - Discernment 1313 & Discourse 017 SMU02 10/9/2014 80 15 Newton, Pauline DISC - Discernment 1313 & Discourse 027 SMU03 9/16/2014 80 17 Newton, Pauline DISC - Discernment 1313 & Discourse 011 SMU03 9/16/2014 80 17 Newton, Pauline DISC - Discernment 1313 & Discourse 012 SMU04 11/4/2014 75 17 Whitehead, Barbara DISC - Discernment 1313 & Discourse 006 SMU01 11/4/2014 90 16 Levy DISC - Discernment 1313 & Discourse 007 SMU01 11/12/2014 60 10 Lange DISC - Discernment 1313 & Discourse 008 SMU01 11/12/2014 60 16 Lange DISC - Discernment 1313 & Discourse 009 SMU01 11/12/2014 60 15 Lange DISC - Discernment 1313 & Discourse 010 SMU02 11/6/2014 80 16 Wood, Angela DISC - Discernment 1313 & Discourse 036 SMU02 11/7/2014 50 7 Maitland, Annie ENGL - English 1400 001 SMU02 11/10/2014 50 9 Maitland, Annie ENGL - English 1400 001 SMU02 11/11/2014 80 16 Wood, Angela DISC - Discernment 1313 & Discourse 036 SMU02 11/14/2014 50 17 Seaney, Ona DISC - Discernment 1312 & Discourse 010 SMU02 11/14/2014 50 15 Seaney, Ona DISC - Discernment 1312 & Discourse 011 SMU02 11/12/2014 50 17 Diaconoff, Cara DISC - Discernment 1312 & Discourse 023 SMU02 11/17/2014 50 9 Maitland, Annie ENGL - English 1400 001 SMU02 11/18/2014 80 17 Polster, Kristen DISC - Discernment 1313 & Discourse 004 SMU02 11/18/2014 90 17 Polster, Kristen DISC - Discernment 1313 & Discourse 005 SMU04 11/21/2014 60 23 Cloward, Karisa PLSC - Political 1340 Science 001 SMU02 11/21/2014 50 7 Maitland, Annie ENGL - English 1400 001 SMU02 11/24/2014 50 9 Maitland, Annie ENGL - English 1400 001 SMU05 7/29/2014 90 7 Pocklington DISC - Discernment 1312 & Discourse 2545 SMU05 7/22/2014 90 7 Pocklington DISC - Discernment 1312 & Discourse 2545 SMU05 7/15/2014 90 7 Pocklington DISC - Discernment 1312 & Discourse 2545 Session: Lecture Session: Flipped Session: Instruction Directed Session: Practice Active Session: Learning One Session: Shot Session: Online Instruction Tutorial Research Session: Used Guide Helped Session: Used Design LibraryAssign Tour tm_lect tm_flipd tm_pract tm_activ oneshot tutorial resguide libassgn libtour Yes No No No Yes No No No No Yes No No No Yes No Yes No No Yes No No No Yes No Yes No No No Yes No No No Yes Yes No No No Yes No No No Yes Yes No No No Yes No No No Yes Yes No No No Yes No No No Yes Yes No No No Yes No No No Yes Yes No No Yes No Yes No No No No No Yes Yes No Yes No No No No No Yes Yes No Yes No No No No No Yes Yes No No No Yes No Yes No No Yes No No Yes Yes No Yes No No Yes No No Yes Yes No Yes No No No Yes No No Yes Yes No Yes No No Yes No No Yes Yes No Yes No Yes Yes Yes Yes Yes Yes Yes No No Yes No No No Yes No Yes No No Yes No No No Yes No Yes No No Yes No No No Yes No Yes No No Yes No No No Yes No Yes No No Yes Yes No Yes No Yes No No No Yes No Yes Yes No No No Yes No Yes No Yes Yes No No No Yes No No No Yes No No No No No No Yes No No No No No No No Yes Yes No No No No No No No Yes Yes No Yes No No No No No Yes Yes No Yes Yes No No No Yes No Yes No Yes Yes Yes No No No No Yes No Yes Yes Yes No No No No Yes No Yes Yes Yes No Yes No No No No No No No No No No No No No No No No No No No No Yes No Yes Yes No No No No No Yes No Yes Yes No No No No No Yes No Yes Yes No No No No No GWLA Student Learning Taskforce Student Outcomes Research Massive Dataset of all first year students across 20 institutions Planned Analysis Stratified Control Samples Instruction vs. No Instruction: Effect? Instruction Methods: Effects? Likely analysis will consist of hierarchical model, taking into account groupings of similar institutions, similar student groups GWLA Student Learning Taskforce Student Outcomes Research First Year Students With Instruction vs. No Instruction Control Group matched on key demographics: Students with Library Instruction Students without Library Instruction GWLA Student Learning Taskforce Student Outcomes Research Effect of Specific Teaching Methods Students with Instruction Method C Students with Instruction Method A Students with Instruction Method B Student Outcomes Research GWLA Student Learning Task Force Research Project – progress made in 2015 Primary Investigator has been named: Dr. Jade Winn Asst. Dean of Instruction, Assessment & Engagement University of Southern California IRB approval obtained at USC, UNLV, ASU GWLA member university deans & directors recently approved funding to hire data analyst Southern Methodist University will play a key role in data analysis and secure storage of data for all 20 participating institutions GWLA Student Learning Taskforce Student Outcomes Research Questions? Thank you!
