Evaluating Undergraduate Research:
Measuring Student Outcomes
Jill Singer, Buffalo State College
Bridget Zimmerman, Buffalo State College
Dan Weiler, Daniel Weiler Associates
UNDERGRADUATE RESEARCH AS TRANSFORMATIVE PRACTICE
13th National Conference June 19-22, 2010
Weber State University, Ogden, Utah
Outline
• Motivation for evaluating Buffalo State’s summer
research program
• Process we followed
• Findings
• Modifications to procedure and instruments
• Available resources
• Questions
Motivation: Why Evaluate the Undergraduate
Summer Research Program (USRF)
• All programs on our campus are required to have assessment plans
• The USRF is the largest single program administered by Buffalo
State’s Office of Undergraduate Research
• Program supports research and creative endeavors
• Each award includes a student stipend of $2500 (in 2010 increased
to $2750), faculty stipend of $1000, and $500 for project-related
travel and supplies
• Program length is 8 weeks
• USRF currently in its 12th year
• Program enjoys a high level of prestige and participants are very
supportive of our ongoing efforts to evaluate the impact of
undergraduate research
• Made sense to develop an assessment process that would apply
both to science and non-science disciplines
• Multi-year effort aimed at development and field-testing a
methodology for measuring student learning and related outcomes
(evaluation findings reported are for summers 2008 and 2009)
Purposes of the Evaluation
• Obtain reliable assessment of the USRF’s impact on
participating students that goes beyond student survey
data
• Obtain information from faculty mentors on how the USRF
program has influenced their teaching and view of student
capabilities
• Provide information to participating students that helps
them assess their academic strengths and weaknesses
• Begin the creation of a longitudinal database that can
provide impact data on a range of outcomes for different
types of students and a variety of academic disciplines
Process: Retreat  Pilot

Full Scale
• A two day retreat in summer 2006 attended by faculty
representing a range of disciplines (arts, humanities, social
sciences, and physical sciences) was held to reach consensus on
student learning outcomes to be measured
• In advance of the retreat faculty were asked to read relevant
papers
• First day of the retreat devoted to identifying and confirming
11 outcome categories
• Second day of retreat devoted to refining outcome categories
and identifying outcome components
• The retreat resulted in 11 outcome categories to be measured
and a detailed rubric that describes the specific components
for each outcome
Outcome Categories
Communication
Creativity
Autonomy
Ability to deal with obstacles
Practice and process of inquiry
Nature of disciplinary knowledge
Critical thinking and problem solving
Understanding ethical conduct
Intellectual development
Culture of scholarship
Content knowledge skills/methodology
…. and option for mentor and their student to identify and add more
outcomes
Outcome Categories
• Each outcome category includes list of specific outcome components.
• For “creativity” components are: brings new insights to the problem at
hand; shows ability to approach problems from different perspectives;
combines information in new ways and/or demonstrates intellectual
resourcefulness; and effectively connects multiple ideas/approaches.
• For “critical thinking and problem solving” components are: trouble-shoots
problems, searches for ways to do things more effectively and generates,
evaluates and selects between alternatives; recognizes discipline-specific
problems and challenges established thinking when appropriate;
recognizes flaws, assumptions and missing elements in arguments.
• For “ability to deal with obstacles” components are: learns from and is not
discouraged by set-backs and unforeseen events; shows flexibility and a
willingness to take risks and try again.
• This pattern is followed for all 11 outcome categories.
Process: Retreat  Pilot

Full Scale
• Six student-mentor pairs participated in pilot implementation
during summer 2007 of the evaluation effort
• Focus groups composed of student researchers and faculty
mentors led by Dan Weiler with Bridget Zimmerman taking
notes
• Based on findings, minor modifications made to instruments
and clarification of overall evaluation process
Process: Retreat  Pilot

Full Scale
• Full scale implementation of the evaluation effort for two
years (2008 and 2009) involving 37 students and 37
faculty mentors
• Mentors represented both first time and experienced (‘repeat’)
mentors familiar with the USRF program (some having mentored
students 5 or more times over the program’s 12 years)
• Evaluation data (qualitative and quantitative) compiled and
analyzed by Bridget Zimmerman
• Qualitative parts of the evaluation took advantage of the
already existing mid-summer and final report forms and the
quantitative portions used the newly developed assessment
instruments
Implementation and Data (1)
• Student completes a survey answering general and specific
questions designed to define a ‘starting point’. Questions cover
such areas as: motivation, knowledge and expectations, and their
understanding of their academic strengths and weaknesses.
• Survey has two purposes: (1) provide mentors with insights into student
knowledge and thought processes, as an aid to the mentors in completing
preliminary student assessments; and (2) provide students with structured
opportunity to explore goals, knowledge, and readiness for the program.
• Mentor reviews responses and meets with student to help them
learn enough to formulate preliminary assessment (intended to
overcome challenge of completing pre-research assessment with
limited knowledge about the student).
• Mentors provided with list of candidate questions for this purpose,
showing relationship between each candidate question and relevant
student outcomes.
Implementation and Data (2)
• Both student and mentor complete a pre-research assessment
and meet to share their responses.
• Instrument has 11 outcome categories
• Five-point scale (the student always, usually, often, seldom, or never
displays the particular outcome, or not applicable).
• Given that mentors may not have equally rich information about
their students across every outcome listed, asked to indicate for
each score their level of confidence (very, fairly, somewhat, not
terribly, or not at all confident).
• Student and mentor meet to compare assessments and discuss
reasons for any differences and help students obtain a more
realistic assessment of their strengths and weaknesses.
• Research begins.
Implementation and Data (3)
• Students keep a journal (electronic or paper) and mentors are
encouraged to keep notes about the student’s progress.
• Student and mentors complete mid-summer reports that include
several questions about progress, changes from the original
projects, and plans for the second half of the summer. It also
contains all the questions asked in the pre-research assessment
questionnaire.
• Both student and mentor meet to share their responses.
• Research continues.
• Student and mentors complete final reports at the end of the
summer that include questions about research findings, places
where students might present their work, mentoring process (for
faculty), and also includes the same assessment questions.
Findings (1)
• Evaluation focused on student and mentor responses to
assessments, mid-summer and final project reports, and journals
provided anecdotal information.
• Assessment survey repeated three times (pre-, mid-, and post-) and
contains a base of 68 questions (34 questions and 34 confidence
scores).
• Cronbach’s alpha used to determine how strongly associated the items
of the scale relate and how well items hold together. The mentor
survey (N= 70) had an overall coefficient alpha of .98. The student
survey (N=68 ) had an overall coefficient of .96. Both coefficients
suggest that there is high level of internal reliability.
• Factor Analysis/Principal Components Analysis (PCA) used to
determine if item structure could be characterized by smaller set of
underlying components and make sure all questions add value to
defining program’s impact on participants.
Findings (2)
• Thirty-four of the sixty-eight base questions are related to
confidence in score’s accuracy.
• Students did not seem to fully understand why they were being
asked to rate themselves on their ability to rate themselves. Based on
this finding, the confidence questions were dropped from the student
version of the surveys and replaced with a single question about
their certainty in responding at the end of the questionnaire.
• Mentors understood the rationale for indicating confidence. Their
degree of confidence increased with each survey and increasing
knowledge of their student. To reduce survey fatigue, the confidence
interval was retained for the faculty pre- and post-research
assessments, but dropped from the mid-summer report.
• To determine if program had impact on participants, pre-, midand post-responses were analyzed for significant differences
with Repeated Measures ANOVA.
Findings (3)
• Overall surveys were well received by students and mentors
and compliance was very high (reflects a number of reasons
related to the value of the USRF program).
• Open-ended responses (both students and mentors) are
thoughtful, thorough, and insightful
• Student responses in particular demonstrated significant growth in
disciplinary expertise from the pre- to the post-assessments.
Findings (4)
• Student findings:
• Tended to initially rate themselves slightly higher than their mentor
rated them.
• Reported growth on every survey item.
• Because students often ‘inflated’ their pre-assessment scores and later
scored themselves more realistically, the actual differences between
pre- and post-scores was less frequently significant.
• Mentor findings:
• Initially rated the students lower than the students rated themselves but
by the end of the program rated the students higher than the students’
self-reported scores (higher rate of significant differences).
• A few items with decreased scores indicated that the mentor initially
overrated their student and later learned that the student was not as
strong as originally scored (e.g., writing skills, contribution to discipline).
Findings (5)
• Student comments included remarks that highlighted:
• Improvements in knowledge of discipline, how scholarly activities take
place in the discipline, new methods and tools for conducting research,
and ability to interact and communicate on a scholarly level
• Contributions to the discipline and that they were making discoveries
that had value as well as contributing resources to future students
• Value of their participation in the USRF program and how it would
contribute to future endeavors (going to grad school, competing in job
market, networking, etc)
• Gained knowledge beyond the classroom, their peers that were not
conducting research, and exceeded their own expectations for what
they could accomplish
Findings (6)
• Mentor* comments included remarks that highlighted:
• Balance between ‘teacher’ and ‘mentor’ allowing student to explore,
make mistakes, and grow without constant presence (reflects stages of
mentoring)
• Value of working in a collaborative/collegial way with their student
• Program’s influence on teaching practice. In particular, spoke of desire
to increase opportunities for students to engage in research within a
course and also raised expectations about what students can
accomplish.
* While some mentors initially commented on the redundancy of completing a pre-,
mid-, and post-assessment, at the end of the process many recognized the value of
capturing the growth of their particular student. The reason for this in part stems
from seeing how the pre-, mid-, and post-assessments and subsequent conversations
with their student was invaluable to the student’s self-awareness of their growth,
strengths, and weaknesses and that the student found the process very instructive.
Modifications Made Based on Findings
• Clearer instructions about evaluation process provided to
students and mentors along with summary of rationale for
evaluation methodology and modifications to protocol
• Elimination of ‘confidence’ score from all three student
questionnaires and replaced by a single question at the end of
the survey that asked students how certain they were about
their skill level and responses and why
• Elimination of ‘confidence’ scores from mid-summer report for
mentor (retained in pre- and post-research assessments)
• Simplification of questionnaires so that optional outcomes
identified at end rather than after each outcome category
Resources
• http://www.buffalostate.edu/undergraduateresearch/x561.xml
Links to static versions of the survey instruments are
provided from this url
• CUR Quarterly article (Spring 2009) describes the approach
Download article from above url
Concluding Remarks
• Evaluation process is at a point where it is recognized by
students and mentors as a valuable part of participating in the
USRF program.
• Unanticipated benefit to the evaluation process is improved
mentoring (provided a mechanism for facilitating regular and
semi-structured interactions between student and mentor)
• Recognized by administration and has helped make a case for
continued growth in the program funding.
• Evaluation is ongoing (for summer 2010 twenty-four USRF
awards were made).
• Additional year of data will inform future modifications and
refinements to our instruments and process as well as yield data
documenting the program’s impact on students and mentors.
Questions and Comments?