Using Assessment of Undergraduate Research to Improve Programming and Student Outcomes Presentation made to SoTE Conference March 28, 2014 Susan Larson What is undergraduate research, scholarship & creative activity (UR)? Council on Undergraduate Research: An inquiry or investigation conducted by an undergraduate student that makes an original intellectual or creative contribution to the discipline. www.cur.org Concordia College: Undergraduate research, encompassing all forms of scholarship and creative activity, is an investigation or inquiry conducted by a student, under the mentorship of a faculty member, which contributes to a high-level intellectual or creative outcome. Defining UR Student, process centered Outcome, product centered Student initiated Faculty initiated All students Honors students Curriculum based Co-curricular fellowships Collaborative Individual Original to the student Original to the discipline Multi- or interdisciplinary Disciplinary based Campus community audience Professional audience - Beckman & Hensel, CUR Quarterly, 2009 Value of UR Cognitive and intellectual growth Gains in knowledge and skills Academic achievement and educational attainment Professional growth and advancement Personal growth and development Assessing the Benefits of UR Survey of Undergraduate Research Experiences (SURE) Includes demographic variables, learning gains, and evaluation of aspects of summer programs. Students report the greatest learning gains on items related to the research process, scientific problems, and lab techniques Students report learning gains on personal development measures (e.g., tolerance for obstacles and working independently) Summer research sustains an interest in pursuing science Lopatto (2004), Cell Biol Educ, 270-277. http://www.grinnell.edu/users/lopatto Assessing the Benefits of UR Assessment of undergraduate research opportunity program (UROP) at Univ. of Michigan Longitudinal assessment evaluating student retention, academic performance, engagement, pursuit of graduate and professional education Evaluated UROP students and matched control group: individuals who applied, but did not participate Significant positive impact of UROP on degree completion rates (most notably for African American students) and likelihood of pursuing post-graduate education Nagda, Gregerman, et al. (1998), The Review of Higher Education, 55-72 http://www.lsa.umich.edu/urop/aboutus/evaluationassessment Assessing the Benefits of UR Ethnographic study of summer undergraduate research (UR) experiences Initial work done at four liberal arts colleges Focus groups with students and faculty Both groups described gains related to process of "becoming a scientist" Faculty focused on “professional socialization” Students more focused on “personal and intellectual development” Hunter, Laursen, Seymour (2007). Science Education, 36-74. http://www.colorado.edu/eer/index.html Why Should YOU Assess UR? Evaluate the effectiveness of your program Determine if your programming is meeting your objectives Develop your programming For donors, your administrators, grant writing Understand the benefits of UR for your students To improve student learning in your context To improve faculty mentoring To broaden participation in UR But, we don’t have to re-invent the wheel What Should You Assess? Student (learning) outcomes What did you learn in this experience? How did this affect your educational experience, career, etc.? Strategies for developing an assessment program Example of assessment program at my home institution Program effectiveness Make use of COEUR Characteristics of Excellence in Undergraduate Research Questions to Ask What do you want to know? Should align with the objectives of your program Student researchers more likely to go to grad school? Self-confidence and self-efficacy? Impact on departments/advisors scholarly activity? Who is the target of your assessment? Who does the assessing? What is the quality of the instrument? How will you use the results? Benchmarking Program improvement Understanding student learning outcomes Assessment of UR and SoTL There are great opportunities for SoTL projects related to undergraduate research So, consider framing your UR assessment to answer the questions you need to ask, but also the questions you want to ask. E.g., benefits to faculty: often discussed, less often assessed E.g., how is UR different/similar to other forms of engaged learning at UVU? Designing a Sustainable Assessment Program State expected outcomes, clear and measurable Research skills, student development goals Program goals Determine methods and criteria to assess outcomes Direct assessment and/or indirect assessment Qualitative and/or quantitative assessment Identify and collect information Start small, start with successes, minimize the burden Set priorities, conduct periodic and timely assessments Use the data! Share results, use assessment results in planning and progam evaluation The Assessment Cycle Assessment of UR is an on-going process of Establishing clear, measurable expected outcomes for UR programs Providing participants (students, faculty, institutions, system) with opportunities to achieve those outcomes Systematically gathering, analyzing, and interpreting evidence to determine how well UR activities match our expectations Using the resulting information to improve UR programs and initiatives Maki, Journal of Academic Leadership, 2002 Example of an Assessment Plan for UR: Concordia College The assessment plan for the Office of Undergraduate Research seeks to accomplish the following goals Provide a count of undergraduate research, scholarship and creative activities at Concordia, including demographic data Assess learning achieved in undergraduate research experiences Evaluate the effectiveness of the Office of Undergraduate Research. Counting UR Faculty surveys administered by the Office of Institutional Research Faculty report on # of students they mentored # of projects they mentored Funding for the projects Outcomes of the projects, e.g., presentations, conferences Survey also distinguishes classroom-based and individually-mentored research Other Ways to Count UR Enrollment in directed research courses # of students participating in funded travel opportunities # of students presenting at on-campus symposia or related events Course syllabi which note an extensive research experience Count of students who engage in off campus research Make use of department annual reports OTHERS? The Challenge and Value of the Count Do you need to count everyone? Can we even do it if we tried? Is counting enough? Do you need to know what happens to these students? Do you need to know their demographic characteristics? Assessment of Student Learning Students are expected to achieve some of the following learning goals/objectives. I. II. III. IV. V. VI. formulate questions in their field demonstrate ability to understand literature explain methodology demonstrate proficiency in the use of the tools and instruments of the area of study analyze and interpret literature and/or results as appropriate to their field of study demonstrate ability to communicate (writing, presenting) within the area of study, and Assessment of Summer Research Program Indirect assessment -- student surveys In-house self-report assessment of learning gains and confidence Responses to open-ended question about learning outcomes Evaluation of the degree to which students met the goals in the learning agreement they sign with their mentor at the start of the summer. Science students complete Lapatto’s SURE survey (http://www.grinnell.edu/academic/csla/assessment/sure). Assessment of Summer Research Program Faculty mentor survey Faculty rate their student’s learning gains and respond to open-ended questions about learning gains Faculty and students rate their satisfaction with the summer program activities. How Much Did You Gain as a Result of Your Summer Research Experience? 0= No gain No A little Moder Good Great 5=Great gain gain ate Formulating research questions All students: mean=3.77, 19% 13% 39% 29% SD=1.09 1st year: mean=3.90, SD=.99 10% 20% 40% 30% 1st yr mentors: mean= 3.78, 11% 22% 44% 22% SD=.97 Proficiency in the use of the tools and instruments in your area of study All students: mean=4.13, 13% 29% 58% SD=.88 1st year: mean=4.80, SD=.42 20% 80% 1st yr mentors: mean=3.78, 44% 33% 22% SD=.83 Analyzing data & interpreting results All students: mean=4.45, 10% 19% 35% 32% SD=.72 1st year: mean=4.00, SD=1.05 10% 20% 30% 40% 1st yr mentors: mean=4.56, 33% 33% 33% SD=.53 Meeting Learning Agreement Goals Considering your learning agreement, rate your learning gains for the goals specified in the agreement. All Concordia summer research students 1st year research students 1st year research student mentors Mean S.D. 3.96 .96 4.40 .52 4.11 .86 Open Ended Question “What were one or two of the most important things you learned from your research experience?” Research skills specific to specific projects Understanding of the research process Understanding of science E.gs., interpreting data; reading scientific literature; the importance of a detailed lab notebook. E.g., understanding how scientists work together; the importance of previously published findings; appreciation of the differing perspectives individuals from different disciplines bring to a problem Personal gains Patience; honesty; flexibility; independence; the importance of team work. Summer Program Activities Impression of summer activities Mean S.D. 3.65 .63 3.88 .71 (very useful/enjoyable = 5) How useful were the summer program activities? How enjoyable were the summer program activities? Allowed them to develop some practical skills Appreciated the chance to interact with other student researchers Some felt that the workshops targeted first-time researchers and took too much time away from the lab Other Ways to Assess SLOs Institutional assessment data Departmental assessment programs Nationally available survey’s E.g., NSSE and critical thinking measures employed by the institution E.g., SURE and others by Lapotto E.g., Undergraduate Research Student Self-Assessment (URSSA), on salgsite.org Rubrics for scoring posters/talks given by research students Reflective essays for students who travel and present UR Focus groups Other Ways to Assess Student Outcomes Post graduate activities Alumni survey Career center survey Publications and presentations generated by involvement in UR Review CUR’s assessment database for ideas https://www.zotero.org/groups/curassessmentbibliography/items/ Using the Assessment Data Closing the assessment loop Lower scores on communicating research Lower rating on learning gains related to ethics Added an oral presentation expectation to the summer program Will enhance summer program workshop on ethics Information about mentor availability and its impact on the summer experience Conversations with mentors about their expectations Program Evaluation May want to go beyond assessing the experience of students to understand if the UR program you offer is robust and effective Could be accomplished with the assistance of COEUR http://www.cur.org/publications/publication_listings/coeur/ What is COEUR? A summary of best practices that support and sustain highly effective undergraduate research environments The purpose of COEUR is to provide a guide for those who wish to build, evaluate, and maintain robust, productive, meaningful and sustainable undergraduate research programs Function as a guide for strategic planning Provide programmatic and institutional benchmarking against national standards Outline a structure for UR self-study Provide a framework for assessment and evaluation of UR The Characteristics Campus mission and culture Administrative support e.g., UR program office Research infrastructure e.g., institutional commitment e.g., space, instrumentation and equipment Professional development opportunities e.g., research leaves, mentorship training The Characteristics Recognition External funding e.g., internal and external funding for research Dissemination e.g., UR in promotion and tenure guidelines e.g., peer-reviewed publication, exhibition, or performance; student research conferences Student-centered issues e.g., community of student scholars The Characteristics Curriculum Summer research program e.g., faculty & student compensation Assessment activities e.g., integration of teaching and research; student course credit for research e.g., assessment of student learning Strategic Planning Simple Evaluation Rubrics CUR’s Characteristic of Excellence Campus mission and culture Institutional commitment Scholarly faculty Faculty commitment Broad disciplinary participation Accessible opportunities for undergraduates Integration with other engaging and high-impact opportunities Thinking about your own context successes at your institution challenges at your institution What is your marker of excellence? EXAMPLE Administrative support Student travel student funding EXAMPLE X# of students traveling to professional meetings How do you measure this? EXAMPLE Student receiving travel funds fill out pre-post survey; reported annually; count %age of students doing UR who travel to meetings What evidence do you have that this component is a priority at your institution? EXAMPLE Increased travel funds from X$ per year to X$$ per year, student travel award recipients published in alumni magazine http://www.cur.org/publications/publication_listings/coeur/ Conclusions There are many published benefits of UR But, local, contextualized assessment of UR are important When undertaking an assessment program Know what you are assessing and why Keep it simple and sustainable Don’t assess everything all the time Keep your focus on enhancing student learning Use assessment data and program evaluation to improve your UR programming COEUR can be a useful tool for improving the experience you offer students