Assessment in service learning:
Student learning outcomes
Julie A. Hatcher, Exec. Dir.
Assoc. Prof. Philanthropic Studies
jhatcher@iupui.edu
Kristin Norris, ABD
norriske@iupui.edu
Assessment Director
Center for Civic & Social Responsibility
University of Kansas
May 21, 2014
Outcomes for work/play shop
• Basic intro to assessment (of x, y, z)
• Implications of a “public work” approach at KU
• IUPUI “case study”
– CMG
– NSSE
• Identify 1-2 AAC&U learning outcomes for either
– Course
– Program
– Campus
• Identify next steps you will take on your
assessment matrix
Current Support for “Civic”
• Membership Organizations
–
–
–
–
–
–
AAC&U
AASC&U
American Academy of Arts and Sciences
Campus Compact
Imagining America
NASPA
• Foundations
–
–
–
–
–
–
–
–
Bonner Foundation
Carnegie Foundation: Elective Classification for C.E
Kettering Foundation
Lang Foundation
Lilly Endowment
Lumina Foundation
Teagle Foundation
John Templeton Foundation
• Government
–
–
State Commissions of Higher Education
Massachusetts Board of Higher Education
Rigor in Research
(Assessment)
•
•
•
•
•
•
•
•
Rigorous thinking
Hypothesis exploration
Systematic approach
Builds upon prior research
Contributes to the field
Triangulation across inquiry
Information synthesis
Creates useful, relevant knowledge
(Patton, 2012)
Schoen’s Reflective Practice
Trait of a good professional
Reflection-in-action
Attentive to new knowledge
Test, adjust, reframe their models of
practice
• Revise plans
•
•
•
•
Kolb’s Experiential Learning Cycle
Concrete Experience
Active Experimentation
Reflective Observation
Abstract Conceptualization
Reflective SL Practitioner
Practice
SL Course/Program/
Curriculum Design
Formative Assessment
Implications for you/others
Reflection Activities
Program/Course Redesign
Classroom Assessment
Theory
Research/SoTL
My Reflective Observations
• Wine or peanut M&Ms
• Important to identify a “north star”
• Takes a significant amount of time
• Year of Assessment in 1997 ish
• Highly connected to scholarship
• We find our way by walking…
•
•
•
•
3 semesters until “in the saddle”
Refine, improve reflection strategies
Ongoing improvement of processes
Tweaks
Boyte’s concept of Public Work
• Places public work (not deliberation or
social justice) at the center of democracy
• Common pool resources (Ostrom School)
• Collective action – “we” not “they”
• Lay participation is essential
• Trait of a democratic professional (Dzur)
• Conflict is intrinsic and healthy aspect
• Co-generators of knowledge -- partners
Assessment as Public Work
• Portland State case study -- resources
– Assessing service-learning and civic
engagement: Principles and techniques
• AAC&U – VALUE rubric case study
– 16 rubrics; all available free
– No need to reinvent the wheel
• IUPUI case study
Current Context at KU
• Reflect – pair & share – wagon wheel
• What are the key “pressures” or levers
that are motivating you personally to be
here today?
• What do you hope to do, improve,
advance as a result of being here today?
• Who do you need to bring to the table
so this is done collectively?
Terms of Scholarship -- data
• Assessment (What occurred?)
– Formative – during the course/program
– Summative – end of the course/program
• Program Evaluation (What aspect?)
– Program Evaluation Research
• Research (Why? What conditions?)
– Qualitative
– Quantitative
– Mixed-methods
Definition
Service learning is a course-based, creditbearing educational experience* in which
students
a) participate in an organized service activity
that meets identified community needs*, and
b) reflect on the service activity in such a
way as to gain
– further understanding of course content,
– a broader appreciation of the discipline, and
– an enhanced sense of personal values* and civic
responsibility.
(Bringle & Hatcher, 1995)
Formative Assessment in SL
• Classroom Assessment Techniques
(Angelo & Cross, 1995)
– Exit Cards
– Minute Papers
– Mid-semester evaluation
• Reflection activities
– Pair and Share
– Journals - various types (DEAL model)
– Value of informal/unstructured
• Community partner feedback
– Coffee……and conversation
Summative Assessment -- Data
• Course-based
– Grades
– End-of-Course survey; pre/post
– Products
• Curriculum-based
– E-portfolio
– Digital stories
– Exit survey for majors
• University-based
– Counting; classifying
– Longitudinal impact on students
– Evidence of community impact
Assessment in SL Course
• Are grades sufficient? (perhaps)
• What literature review could help to “ground”
the assessment? (ex. Nutrition, Architecture)
• How can you use/analyze reflection activities
as a source of data to document student
learning?
• What pre/post learning could you assess?
• What other products could you gather?
• Multiple sources of data -- triangulation
Designing and Assessing SLClass/Program
Learning Objectives of
Course - Curriculum - Campus
Assessment of Student
Learning and Products
Selection of Community
Partner and Service
Experience
Scholarship of
Teaching/Learning SOTL
Structured Reflection Activities Framed by
Learning Objectives
Products are Created
Service Learning Outcomes
• Academic Development
– Persistence and retention
– Achievement and aspirations
• Life Skills
– Racial tolerance
– Cultural understanding
• Civic Responsibility
– Commitment to community
– Aspirations to volunteer
(Sax & Astin, 1997; UCLA/HERI)
(www.compact.org/resource/aag.pdf)
Meta-Analyses of SL
• Requires a body of research
• Psychology, business, communications
•
•
•
•
Academic
Personal
Social
Citizenship
Case Study from IUPUI
• Institutional culture and commitment
• Assessment & Civic Engagement
• Assessment Institute (Oct 21-23., 2014)
• Ethos in our work
• Bob Bringle’s leadership
• Scholarship and research
• Signature Center designation
• IUPUI Book Series – Research on SL Vol 1, Vol 2
• Boyer Scholars Faculty Learning Community
• IUPUI Research Academy (each May)
Types of Citizenship Across Disciplines
•
•
•
•
•
•
•
Civic professionalism
Social responsibility
Social justice
Connected knowing: Ethic of care
Public leadership
Public intellectual
Engaged/public scholarship
(Battistoni, 2002)
Civic-Minded Graduates
• “to what end? ”
• Civic-Minded Professionals
• John Dewey
• Democracy and Education
• William Sullivan
• Habits of the Heart
• Work and Integrity
• Ernest Boyer
• “public good” argument
CMG as a “North Star”
A civic-minded graduate is one who
a) is formally educated and
b) has the capacity and orientation to
work with others
c) in a democratic way
d) to improve the community.
(Hatcher, 2011)
Civic-Mindedness
“a person’s inclination or disposition to
be knowledgeable of and involved in the
community, and to have a commitment to
act upon a sense of responsibility as a
member of that community”
(Steinberg, Hatcher, Bringle, 2011)
Civic-Minded Graduate Model
Civic and Workforce Development
• Developing civic-minded graduates
and professionals
• Developing workforce “soft” skills
• Recognizing the importance of place
(Battistoni & Longo, 2012)
Domains of CMG: Civic Knowledge
• More than purely academic knowledge (dates,
places, important civic or political events)
• Knowledge of volunteer opportunities (ways to
contribute to society and of nonprofit organizations)
• Knowledge of contemporary social issues (current
events and the complexity of issues in modern
society)
(Steinberg, Bringle, & Hatcher 2011)
Domains of CMG: Civic Skills
• Communication and Listening (ability to communicate
with others and listen to divergent points of view)
• Diversity (understanding the importance of, and the
ability to work with others from diverse backgrounds)
• Consensus-building (ability to work across difference to
come to an agreement or solve a problem)
Domains of CMG: Civic Dispositions
– Valuing community engagement (understanding the
importance of service to others, and being actively
involved in the community)
– Self-efficacy (have the desire to take personal action,
with a realistic view that the action will produce the
desired result)
– Social trustee of knowledge (feeling a sense of
responsibility and commitment to use the knowledge
gained in college to serve others)
CMG Survey
• 30 Likert-type items (student self-report)
• Knowledge, skills, dispositions, behaviors
• 6-point response format
•
•
•
•
Demographic items
Student activity items
Paper or online administration
Adaptable: “My education at IUPUI” – course, or
major, depends on the research question
Assessment Tools
• Course Evaluation
• SL End –of-Course Survey
• Domains to include for your campus?
• Student Learning Outcomes
• Civic-Minded Graduate Scale
• Civic-Minded Graduate Narrative
• Civic-Minded Graduate Rubric
• AAC&U VALUE Rubric (Civic Engagement)
Implications of CMG--CMP
• Program Design
• Alternative Break; days of service
• Service-scholarship Applications
• Plater Medallion
• Consultations with faculty and
departments
• New work in faculty development and
graduate education
Pair and Share
• Put on your “disciplinary hat”
• Why is it important to you as a…landscape
architect, student affairs professional…to include
civic learning in your course/program?
• What type of civic learning do you hope
occurs for your students?
• Who are your stakeholders, and why would
they care about these outcomes?
Learning Objectives
• List 2 civic learning objectives in your
community engaged program/course.
• (How do these align with broader curriculum?)
• (How do these align with accountability for
external program review, if applicable?)
• (How do these align with the campus mission
for student learning?)
“Deep Learning”
• High Impact Practices (e.g., “HIPs”)
• Distinct value of service learning
• Additive value of multiple experiences
• See AAC&U, Ashley Finley, Director of Research and
Assessment
• NSSE Data (12 items)
• Higher-Order learning
• Integrative learning
• Reflective learning
Participation in Service Learning Courses
The independent variable, participation in serving
learning courses, was derived from NSSE survey
question 1K:
In your experience at your institution during the
current school year, about how often have you done
each of the following?
k. Participated in a community-based project
(e.g., service learning) as part of a regular course
Participation in Service Learning Courses
IUPUI
Urban
13
Public
Research
NSSE
Sample
Freshmen
56%
38%
38%
41%
Seniors
58%
40%
43%
48%
Deep Learning
 The dependent variable deep learning was comprised
of three different scales. Reliability analysis was
conducted for higher-order learning (α=.83),
integrative learning (α=.73), and reflective learning
(α=.83).
 The data file was then split into freshman and senior
students so the analysis could be conducted on these
two populations separately.
Higher Order Learning Questions (α=.83)
During the current school year, how much has your coursework
emphasized the following mental activities?
Applying theories or concepts to practical problems or in new
situations
Analyzing the basic elements of an idea, experience, or theory,
such as examining a particular case or situation in depth and
considering its components
Making judgments about the value of information, arguments,
or methods, such as examining how others gathered and
interpreted data and assessing the soundness of their
conclusions
Synthesizing and organizing ideas, information, or experiences
into new, more
complex interpretations and relationships
Integrative Learning Questions (α=.73)
In your experience at your institution during the current school
year, about how often have you done each of the following?
Worked on a paper or project that required integrating ideas or
information from various sources
Included diverse perspectives (different races, religions,
genders, political beliefs, etc.) in class discussions or writing
assignments
Put together ideas or concepts from different courses when
completing assignments or during class discussions
Discussed ideas from your readings or classes with faculty
members outside of class
Discussed ideas from your readings or classes with others
outside of class (students, faculty members, co-workers, etc.)
Reflective Learning Questions (α=.83)
During the current school year, about how often have you done
each of the following?
Examined the strengths and weaknesses of your own views on a
topic or issue
Tried to better understand someone else’s views by imagining how
an issue looks from his or her perspective
Learned something that changed the way you understand an issue
or concept
The Findings
 An independent-samples t-test evaluated differences
in reported deep learning skills between students
who participated in one or more service learning
courses and those students who did not participate
in service learning courses.
 Deep learning skills of higher-order learning,
integrative learning, and reflective learning were all
higher for both seniors and freshman who
participated in service learning course(s).
IUPUI Freshman
Mean
(Overall)
N=524
Mean
(Service
Learning)
N=305,
58%
Mean
(No Service
Learning)
N=219, 42%
Mean
Difference
(SL and No
SL)
Reliability
Effect
Size
Sig.
3.09
2.99
.10
.83
.08
.085
Construct
# of
Items
Higher Order
Learning
4
Integrative
Learning
5
2.62
2.75
2.43
.32
.73
.27
.000*
Reflective
Learning
3
2.72
2.82
2.58
.24
.82
.16
.000*
*p<.05, 2-tailed significance
3.05
IUPUI Seniors
Mean
(Overall)
N=998
Mean
(Service
Learning)
N=588,
59%
Mean
(No Service
Learning)
N=410, 41%
Mean
Difference
(SL and No
SL)
Reliability
Effect
Size
Sig.
3.36
3.03
.33
.86
.24
.000*
Construct
# of
Items
Higher Order
Learning
4
Integrative
Learning
5
2.81
2.99
2.57
.42
.72
.34
.000*
Reflective
Learning
3
2.86
2.96
2.72
.24
.83
.16
.000*
*p<.05, 2-tailed significance
3.23
Implications
 Results contribute evidence of student learning at
the institution level
 Findings are consistent with prior research on
participation in service learning and improved
student outcome measures (Astin et. al., 2000)
 Provide a rationale for institutions to support faculty
who engage with the community partners to develop
service learning courses
Matrix Framework
Core Concept
AAC&U Learning
Outcome 1
AAC&U Learning
Outcome 2
Key Indicator of
learning
Methods to
gather data
Source of the
data