Assessment
Institute
October 28, 2013
Indianapolis, IN
Dr. Don Levy – Director, Siena
College Research Institute
Agenda
• Fulfilling Higher Education’s Mission for Community Engagement
• NASCE Methodology
• NASCE Findings
• Utility of the NASCE as Institutional Assessment
• Institutional Case Study
NASCE as Institutional Assessment
NASCE is an Institutional Assessment – uses individual-level data to assess
institutional community engagement
Use NASCE Data…
As backbone of Strategic Plan
To assess campus culture of service and facilitate
community engagement discussions with faculty
and administrators
To establish realistic and structured community engagement goals
To implement real change at the systemic level and move all students along the
continuum of service
As bi-annual assessment tool to continuously evaluate institutional progress in community
engagement
NASCE as Assessment Tool
Assess type, frequency, depth, motivations for, and
obstacles to service on the part of students
Assess effects of institutional structures and
programs on student community service
Generates individual level data that allows
institutions to draw conclusions about student
subsets
Comparative analysis with national dataset of more
than 27,000 students and counting
NASCE
• Assessment of
institutional
expression of service
– the Capacity
Contribution
• By measuring the
behavior of students
across 9 service
dimensions
• In order to
– Have a valid and
reliable measure
– Publicize the
findings across
campus
– Plan necessary
implementations
– Re-measure
NASCE Background
Belief that community engagement is
essential to undergraduate education
Lack of substantive data on community
impact in higher ed.
Disconnect between institutional mission
and civic action
Authors saw need for community
engagement measurement
NASCE uses student-reported
service for quantitative analysis
Mission Statements
As a leading metropolitan university, we achieve
our mission by creating a learning environment
where curricula are connected to societal issues
through civic engagement.
We inspire our students to be citizens of
character who demonstrate professional and
civic leadership. We contribute to the vitality
and well-being of the communities we serve.
We offer a unique blend of traditional classroom
knowledge and innovative experiential learning
that provides students with a solid foundation for
a lifetime of learning, service, and leadership
while valuing social responsibility and diversity.
Through academic programs, research, artistic
expression, public service and community-based
learning, the university serves as an educational,
cultural and economic leader for the region.
We view the three traditional pillars of the
public higher education mission—research,
education, and service—as interdependent
endeavors that continually enrich and inform
each other. Groundbreaking research,
transformative educational experiences, and
deeply engaged service to all communities…
The university enriches the world
through scholarship, artistic expression,
community engagement, and entrepreneurship.
Mission and Engagement
George Kuh
Colleges that over-perform on
engagement live their mission
Defining Community Engagement
Community Engagement
The extent to which college and university
students serve the unmet human needs of
people in their communities (i.e. Housing,
Health Issues, Hunger, etc.)
Includes community service and engagement
activities that require the time and effort of
the participant, with the ultimate goal of
bettering the community
Individual Efforts, Work or Internship, Clubs
or Organizations, and Classes
NASCE Methodology
Current Sample
Gender
Male
32%
Total n for study= 27,038
Senior
29%
Class Year
Freshma
n
24%
Average Age: 23 years
Average GPA: 3.26
12% non-traditional students
Female
68%
Junior
24%
I would describe myself as:
Hispanic or Multi-racial
4%
Latino
Other 5%
2%
Asian
7%
Which of the following best describes
your high school experience?
Native
American
<1%
5%
Sophomore
23%
I attended a public school
for my entire time in high
school
12%
I attended a private school
for my entire time in high
school
Black or
AfricanAmerican
8%
81%
Caucasian
76%
I attended both public and
private schools during my
high school
Participating Colleges and Universities
• Through Spring 2013: 46 different schools across 14
states
• Ranging in size from 800-23,000
• 6 re-assessments (52 assessments in total)
• Large and diverse new cohort
• Including:
• 15 New York Campus Compact Schools
• 6 SUNY schools
• 20 Bonner schools
• 15 public schools
• 9 schools with over 10,000 enrolled students
• 13 religiously affiliated schools
NASCE Instrument
1. “DO YOU SERVE?”
Please consider the community service in which you have or are engaged.
Remember, by community service we mean the following: any activity, including
internships, work study, or co-ops, in which you participate with the goal of providing,
generating and/or sustaining help for individuals and groups who have unmet human
needs in areas like shelter, health, nutrition, education, and opportunity.
Do you engage in community service here at XYZ?
NASCE Instrument
Measures student service across 9 areas
Homelessness
Hunger
Civic
Religious
Youth
Environmental
Elder
Economic
Justice
Health
NASCE Instrument
2. “HOW OFTEN DO YOU SERVE?”
For each of the community service activity types that you
checked, how often would you say you did that type of
service. Would you say….
 Once or twice a year
 Several times during the year
 About once a month
 Several times a month
 About once a week
 More than once a week
NASCE Instrument
3. “AT WHAT DEPTH DO YOU SERVE?”
For each of the types of service that you selected, which of the following describes
your level of involvement? [CHECK ALL THAT APPLY]
 I would participate at an event
or short BASIS
term drive.
it was a COMMITTED
one-shot type
REGULAR
FOR Usually DEEPLY
of involvement.
A PERIOD OF TIME
ONE SHOT
 I was involved on a regular basis for a period of time. One example
would
beLiving
a
Volunteer at
a Senior
Mentor
School
regular
commitment
to be there
onceMiddle
a week
for an entire semester,
another
Volunteer
once at
facility twiceor
a week
since
students trip
everyfor
week
for aof each day for a
woulda food
be todrive
participate on a service
most
period
of time.
you
were 16
semester
 I was deeply involved in a project or cause and dedicated to it. Rather than
thinking of my service as a chore or time commitment, I was drawn to serve by the
issue or problem and worked towards its resolution.
Percent of the Possible (POP) Scores
The NASCE computes the POP Score by assigning values
to those students who perform service in each area:
1. Engaged in any of the nine areas.
• Yes (1)
• No (0)
2. Frequency.
• Once or twice a year (1)
• Several times a year or once a month (2)
• Several times a month (3)
• Weekly or more (4)
3. Depth.
• An event or drive; one-shot (1)
• Regular basis for a period of time or a service trip (2)
• Deeply involved and dedicated to a project or cause (3).
Percent of the Possible (POP) Scores
An individual’s responses are multiplied to create area level individual scores
ranging from 0-12. These totals are summed across the institution and divided
by the maximum score.
The raw score for each area is expressed as:
(Service * Frequency * max(Depth))
12
The institutional raw POP Score is computed where n=the number of areas:
 (Service * Frequency * max(Depth))
n*12
The area level scores are averaged to create the institutional percent of the
possible. Both institutional and area scores are then normalized with .33
equaling a POP Score of 100.
POP Scores
• Impressive Service: Students serve frequently and maintain substantive
connections with service sites beyond individual projects. Your institution is
Impressive
making a difference.
60+
40-60
High
• High Level Service: Students are engaged. Small gaps in participation,
frequency, or depth hinder peak service. Potential for greater contribution
exists.
• Moderate Level Service: Students are serving; however, significant gaps in
student participation, frequency, or depth present opportunity for
Moderate
improvement. Data and the POP scores identify areas of weakness.
20-40
0-20
Low
• Low Level Service: Little student involvement. Service takes the form of
"one-shot" activities with little commitment. If the school wishes to
expand its service contribution, data points the way.
Example of POP Score
• Each year, John takes part in one Saturday morning
environmental cleanup and then months later sees a sign for a
food drive and drops by with some cans of soup.
• Areas of Need John Served: Hunger and Environmental
Frequency=Low
Depth=Low
Frequency=Low
Depth=Low
Environmental
Hunger
Total
Participation
1
1
------------
Frequency
1
1
------------
Raw POP
Depth
1
1
------------
Individual POP
6
Overall
Contribution
1
1
2
Overall POP
6
Total
.0208
Example of POP Score
• Mary is also involved in the same environmental cleanup day and
canned food drive as John each year. However, she also has tutored
elementary school students every Tuesday afternoon for an entire
semester.
• Areas of Need Mary Served: Hunger, Environmental, Youth
Frequency=Low
Depth=Low
Frequency=Low
Depth=Low
Frequency=Low
Depth=Low
Environmental
Hunger
Youth
Total
Participation
1
1
1
---------
Frequency
1
1
4
---------
Depth
1
1
2
---------
Overall
Contribution
1
1
8
10
Total
Raw POP
.1042
Individual POP
32
Overall POP
32
Example of POP Score
• Alex has worked diligently to create a program in which students are
linked to elder members of the community. He has also spent his time
weekly at a local city mission and cleaning up at a local park.
• Areas of Need Alex Served: Elder, Homelessness, Environmental
Frequency=Impressive
Depth=Impressive
Elder Homelessness Environmental
Participation
Frequency=Impressive
Depth=High
Frequency=Impressive
Depth=High
Total
1
1
1
---------
Frequency
4
4
4
---------
Depth
3
2
2
---------
Overall
Contribution
12
8
8
28
Total
Raw POP
.2917
Individual POP
88
Overall POP
88
Units of Service
• Each student provides a certain
amount of service measured in units.
 Each student can provide 12 units of
6 units
32 units
88 units
service per service area for a total of 108
units.
 Each surveyed student’s unit of service contributes to the sample
total which is generalized to the total student population.
Example: College XYZ has 1000 students enrolled and every student
participates in a food drive once a year.
Individual Student’s Contribution: 1 unit per person
Institutional Contribution: 1000 units
 The total amount of service this institution could have done
theoretically was 12(9 areas) x 1000 = 108,000 units.
School Comparisons
Variables
School A
School B
School C
Institutional POP
52
28
13
Hunger POP
60
26
13
Homelessness POP
41
25
8
Religious POP
26
7
8
Environmental POP
42
32
11
Civic POP
46
21
15
Health POP
38
35
25
Youth POP
151
66
18
Elder POP
9
9
5
Participated in
Service In College
87%
67%
34%
Drop off From High
School
9%
20%
47%
Student POP Scores Across NASCE Sample
Institutional
17
Civic
17
n = 27,038 students
8
Elder Care
52 total assessments between 2009
and Spring 2013
16
Environmental
22
Health
6 re-assessments
12
Homelessness
15
Hunger
16
Religious
40
Youth
7
Economic
0
20
Low
40
Moderate
60
High
80
Impressive
Institutional POP Scores Across NASCE Sample
More
than 40
P
O
P
S
c
o
r
e
s
2%
POP Score Range: 3 – 52
Institutional Average: 21
10%
31-40
37%
21-30
38%
11-20
13%
0 - 10
0%
5%
10%
15%
20%
25%
30%
35%
40%
Capacity Contribution Examples
Capacity Contribution: Institutional
60
50
30
46% of students contribute to
a POP score of 17.
20
10
54% of students report doing no service.
0%
25%
50%
Percent of Respondents
75%
0
100%
POP Score
40
Capacity Contribution Examples
Capacity Contribution: Youth
60
27% of students
contribute to a
POP score of 44.
50
30
20
10
73% of students do no service addressing youth needs.
0
0%
25%
50%
Percent of Respondents
75%
100%
10% of students account for 63% of the
Youth POP score.
POP Score
40
Low Service
XYZ POP
Score
8
0 Low
20 Moderate 40
High
60 Impressive 80
Do you participate in community
A School with a Low POP Score will:
service here at XYZ?
• Have a low Percentage of Students performing
Service
23% of
No
• Have a small percentage of students contributing
77%
students
to a high percentage of the overall POP Score
report
Yes
23%
Capacity Contribution: Institutional
30
25
20
10% of Students Account
for 84% of the total
service score
0%
23% of students
contribute to a POP
score of 7.
77% of students report doing no
service.
15
10
5
0
25%
50%
Percent of Respondents
75%
POP Score
doing
service
Service
in Need
Areas
Moderate
Service
25
XYZ POP Score
0 Low
Do you participate in community
service here at XYZ?
70% of
Students
Report
Doing
Service
No
30%
Yes
70%
20 Moderate 40
High 60Impressive80
A School with a Moderate POP Score will:
• Have a majority of their students
reporting service.
• Have a small percent of students
contributing to less than half of the
overall POP Score.
Capacity Contribution: Institutional
70
50
10% of students
account for 35%
of the total service
score
30% of students report
doing no service.
70% of students contribute to a
POP score of 25.
40
30
20
10
0
0%
25%
50%
Percent of Respondents
75%
100%
POP Score
60
High Service
XYZ POP
Score
52
0
Low
20
Moderate
40
High
60
Impressive
80
Do you participate in community service A School with a high POP Score will:
• Have the vast majority of their students
here at XYZ?
reporting service.
No
• Have a small percent of students
87% of
13%
contributing to a relatively small percent of
students
the overall POP score.
report
doing
Yes
service.
Capacity Contribution: Institutional
87%
70
10% of students
account for 28% of the
total service score.
87% of students
contribute to a POP
score of 52.
13% of students report
doing no service.
50
40
30
20
10
0%
25%
50%
Percent of Respondents
75%
0
100%
POP Score
60
Ideal Capacity Contribution
Capacity Contribution: Institutional
50
100% of students contribute
to a POP score of 50.
45
0% of students report doing no service.
40
Pop Score
35
30
25
20
15
10
5
0
0%
25%
50%
Percent of Respondents
75%
100%
Drop in Service after High School; “The Melt”
The percentage of students performing service and
the institutional POP scores often decrease
drastically from high school to college
Across the entire sample, 46% of college students
participate in community engagement and service, a
drop of 40 percentage points from high school.
Potential factors:





Less structure and organization
The total Institutional POP drops 16
Decreased promotion of activities
points from high school to college
Fewer requirements
Campus culture
Other activities, including studying and social events, take up more time
High School
College
From High School to College
Four Types of Student Transitions
No: College
Yes: College
Yes: High School
43%
43%
No: High School
3%
11%
46 percent of
Students Report
Doing Service in
College
Yes/No and Yes/Yes: What’s the difference?
Continuing volunteers are significantly
more satisfied with their personal level of
involvement in college than are high
school only volunteers (77% vs. 45%)
Continuing volunteers are more likely to
be involved in clubs in college (90%)
compared to high school only volunteers
(62%)
Service by Need Areas
Total Service by Needs Area
Econ
5%
Environment
10%
Elder Care
5%
Youth
26%
Civic
11%
Health
15%
Religion
10%
Hunger
Homelessness 8%
10%
 Elder Care and Economic
Justice are neglected need
areas.
 Youth Services is the most
impactful area.
 Religion and Youth represent
deepest commitments.
 Environment and Health have
high rates of participation, but
low frequency and depth
 Percentage of students serving
once a week or more:
o
o
o
o
Religion (48%)
Youth (38%)
Economic (21%)
Elder Care (19%)
Institutional Beliefs
Student Opinions (Agree or Strongly Agree):
84%: Overall, I would say that XYZ University promotes community service among the
student body.
72%: I think the college or university does an appropriate job of informing students of all
the ways they can be engaged in the community.
59%: Overall, I am satisfied with my personal level of involvement in community service
here at XYZ University.
Obstacles to Service
84%
I have to study
74%
Too busy with other activities
63%
I have to work
48%
I don't know what is available
44%
Not enough energy
42%
Lack of transportation
32%
Too busy with friends
28%
What I would like to do is…
18%
It makes me uncomfortable
13%
Previous bad experience
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Motivations for Service
96%
I believe I can help people who are in need
95%
It is the right thing to do
94%
It makes me feel good about myself
83%
I want to change the world
83%
I meet people through participating
75%
It is important to have on my record
59%
I have been required to
53%
Because my friends do it
24%
I don't want to feel guilty
0%
20%
40%
60%
80%
100%
Campus Awareness
How have you heard about service opportunities
available here at XYZ?
68%
Campus emails
63%
Word of mouth
62%
Flyers
54%
In a class
49%
Through a club
Organizational Fair
30%
Residence life
29%
24%
My RA or RD
22%
Campus Newspaper
0%
10%
20%
30%
40%
50%
60%
70%
Student Clubs and Organizations
71% involved in at least one type of club
Type of Club
Percentage Reporting Membership
Individual sports
30%
Academic clubs
28%
Cultural groups
22%
Religious groups
21%
Organized sports
18%
Political or environmental advocacy
13%
Student Government
8%
Community Youth Groups
7%
•
57% of students
involved in a club or
activity do service in
college
•
Only 18% of students
not involved in any
clubs or activities do
service in college
Service-Learning – High Impact Practice
“Ideally, institutions would structure the curriculum and other learning opportunities so that
one high-impact activity is available to every student every year…In the short term,
making high-impact activities more widely experienced should have a demonstrable impact in
terms of student persistence and satisfaction as well as desired learning outcomes.”
-- George Kuh
High Impact Educational Practices, AAC&U
 84% of students agree that one of the obstacles to volunteering more is having to
study
 Of the students who serve, 48% say they’ve done so “as part of a course.”
 Students who do service as part of a course:
o
o
88% are also involved in at least one type of club or activity in college
76% are satisfied with their personal level of involvement (sample-wide = 59%).
 Service learning, campus culture of service, and college structures as a solution to:
o
o
o
o
o
Having to study
Too little time
No transportation
Don’t hear about activities
What I’d like to do is not available
High Performers and the Rest
POP Scores:
High Performers (top 5% of Sample)
n = 1,437
The Rest of Students who Serve:
n = 10,953
Institutional: 107
Institutional: 28
Hunger: 109
Hunger: 23
Environmental: 94
Environmental: 28
Youth: 190
Youth: 74
Civic Awareness: 121
Civic Awareness: 27
Religion: 108
Religion: 25
Elder Care: 61
Elder Care: 11
Homelessness: 94
Homelessness: 17
Health: 120
Health: 38
Economic Justice: 65
Economic Justice: 8
High Impact Community Engagement
Leadership Qualities
HP’s
The Rest
Organized or planned a service project
that involved other volunteers:
76%
34%
Helped facilitate a group service project as
a group leader:
75%
31%
Participated in a service-based trip:
58%
25%
NASCE Reliability
NASCE Data
NSSE Data
Q22. “…Do you engage in
community service here at XYZ?”
15e. “About how many hours do you spend in
a typical 7 day week doing community
service or volunteer work?”
Freshmen Seniors
Yes
36%
51%
No
64%
49%
Q28. “…On average during your time at
XYZ, how many hours a MONTH do you
engage in community service?”
Freshmen
Seniors
More than 0 but
less than 10
25%
30%
Between 10 and 20
6%
11%
More than 20
5%
11%
First Year
Students
Seniors
0 hours
56%
49%
1-5 hours
33%
36%
6-10 hours
6%
8%
11 hours or
more
4%
7%
NASCE Validity – An Example
Bonner Data
Bonner POP Scores
76
Institutional
84
Civic
Bonner Sample: n = 198
100% Report doing service in College
28
Elder Care
97% Reporting Service Prior to College
97% Involved in clubs in College
57
Environmental
70% organized/planned a service project that
involved other volunteers
60
Health
Homelessness
Hunger
88
70% helped facilitate a group service project
as a group leader
88
60% Participated in service as part of a course
78% Participated in a service-based trip
34
Religious
200
Youth
76
Economic
0
20
40
60
Low
80
Moderate
100
120
High
140
Impressive
160
180
200
Using NASCE to Enhance Community Engagement
Developmental Pathway
Increase service
involvement and POP
Move students along
“service continuum”
Strategic Plan
NASCE Data
Win-Win-Win
Participation
Improve Students’
Participation,
Frequency and
Depth of Service
NASCE
Target Areas of Need
NASCE
LIVE the Mission
NASCE
Frequency
Depth
Institutional Case Study
2009 Data
2013 Data
HS Participation Rate: 93%
College Participation Rate: 49%
HS Participation Rate: 93%
College Participation Rate: 64%
POP Scores
POP Scores
Institutional: 14
Civic: 14
Elder Care: 3
Environmental: 9
Health: 21
Homelessness: 14
Hunger: 10
Religious: 9
Youth: 32
Economic: --
Institutional: 26
Civic: 28
Elder Care: 7
Environmental: 17
Health: 30
Homelessness: 26
Hunger: 25
Religious: 15
Youth: 73
Economic: 13
Institutional Case Study
The How and Why:
• Established Central Engagement Office
• Developed Large Bonner Program
• Institutionalized Service-Learning
– Community Engaged First-Year Seminar
– Certificate in Community Development
– Interdepartmental Collaborations
• Increased Service Programs/Opportunities
• Increased Service Staff/Administrators
Evaluating NASCE
Is it Perfect Assessment? No.
But…
• NASCE overstates service
• NASCE is largest compiled dataset
on community engagement in
higher education
• POP places relatively large
emphasis on frequent, deeply
committed service compared to
sporadic involvement
• Some institutions administer
NASCE and fail to effectively use
the data
• NASCE goes more in-depth than
any existing assessment
• NASCE is affordable, customizable,
and informs data-driven solutions
Higher Ed’s Decision
• Does Community Engagement matter?
• Are we here to have a meaningful impact on the communities in which we are
nested?
• Are we here to foster responsible, active citizens during and after college?
Data paves the way!
• Assessment and evaluation is first necessary step to making informed decisions
about the effectiveness and importance of community engagement initiatives on
campus.
• Assessment and evaluation provide the necessary structure to foster dialogue
with all relevant parties about community engagement on your campus
• Is it Higher Ed’s duty to play an engaged role in the community?
•
\
• Do we wish to fulfill our Institutional Missions by creating a more sustained
connection between classroom learning and community involvement?
Future Paths and Recommendations
 How can we retain volunteerism at the high school level?
 54% of students reported not doing service
o
How can we get these students involved?
 How can we increase frequency and depth of service?
Pinpoint students with deep commitment to
service in admissions
Facilitate the transition from high school to
college
Create structure and awareness
Enhance academic involvement and service
learning
NASCE Final Analysis
•
•
•
•
•
•
•
•
•
•
Final Report
Comparative analyses to national sample
Additional data breakdowns of subsets
Data Cross tabulations
Data Frequency Tables
Raw dataset
Phone calls/in-person meetings with NASCE Staff
Continued Support
Assessment Tool to incorporate into strategic plan
Follow-up Assessments to measure progress
www.siena.edu/sri