Assessment of Learning in
the Liberal Arts Core
Donna Vinton, Ph.D.
Director of Academic Assessment
January 11, 2010
Topics for Today
• Background on LAC assessment at UNI
• 2008-2009 assessment data for the LAC
• Questions for next steps
LAC Assessment:
Some Background
2000:
NCA re-accreditation team recommended improvement
in student outcomes assessment processes related to
the LAC
2001:
LAC Committee formed a subcommittee to develop a
comprehensive student outcomes assessment program
for the LAC
2002:
Pilot testing of Academic Profile and the Collegiate
Assessment of Academic Proficiency/Critical Thinking.
2004:
• The LAC Assessment Subcommittee decided to use
Academic Profile for LAC assessment. (This
instrument was later renamed Measure of Academic
Proficiency and Progress and, more recently,
Proficiency Profile.)
• Initial results of testing with Academic Profile were
shared with the President and his cabinet, the
Provost and Academic Affairs Council, and the
Faculty Senate
• NCA Progress Report on the General Education
Program was submitted to the North Central
Association and approved.
2005-6:
• First year to administer NSSE
2007-8
• First use of Oral Communication sections to administer
MAPP to first-year students
2008-9
• NSSE & MAPP data placed on password-protected
page on the Academic Assessment web site
• Workshops on LAC data offered on campus
2009-2010
• Presentations on NSSE and MAPP data at department
faculty meetings
3 Key Measures of Learning
• National Survey of Student Engagement (NSSE)
This instrument replaced College Student Experience
Questionnaire, or CSEQ, named in the original plan.
• Graduating Senior Survey from UNI Office of
Institutional Research
• Measure of Academic Proficiency and Progress
(MAPP, now known as Proficiency Profile and formerly
called Academic Profile)
Two Types of Data
• Student perceptions (indirect measures)
– NSSE
– Graduating Senior Survey
• Standardized testing (direct measure)
– Measure of Academic Proficiency and Progress
NSSE Survey Content
Student Behaviors in College
Institutional Actions
And Requirements
Student Reactions to College
Student Background
Information
Student Learning
& Development
Data From NSSE
• Responses for individual survey items by response
stems and item mean
• Calculated mean for 5 “benchmark” areas: level of
academic challenge, active and collaborative learning,
enriching educational experiences, student-faculty
interaction, supportive campus environment
• Comparison of UNI data with data from selected cohort
groups (e.g., UNI peers, Carnegie peers, Foundations of
Excellence institutions)
• Additional data from consortium questions—American
Democracy Project and Consortium for the Study of
College Writing
MAPP
MAPP, or the Proficiency Profile measures:
• proficiency in critical thinking, reading, writing and
mathematics in the context of humanities, social
sciences and natural sciences
• academic skills developed, versus subject knowledge
taught, in general education courses
Source: http://www.ets.org/mapp, highlights added
MAPP: Norm-Referenced Scores
Norm-referenced scores (scaled scores) compare the
scores of one student or group of students to another.
Eight norm-referenced scores:
• Total Score
• Skills subscores (critical thinking, reading, writing
and mathematics)
• Context subscores (humanities, social sciences and
natural sciences)
UNI scores in each of these areas can be compared to scores of
students in comparable (Master’s I and II) institutions.
MAPP: Criterion-Based Scores
Criterion-referenced scores (proficiency
classifications) measure the level of proficiency obtained
on a certain skill set.
Nine criterion-referenced scores:
• Mathematics (Level 1, Level 2, Level 3)
• Writing (Level 1, Level 2, Level 3)
• Reading (Level 1, Level 2, Critical Thinking)
UNI scores in each of these areas can be compared to scores of
students in comparable (Master’s I and II) institutions.
Uses of MAPP/NSSE Data at UNI
• Assessment of learning in the Liberal Arts Core
• Key performance indicators for the Student Affairs
Division
http://www.uni.edu/studentaffairs/documents/Student%20Affairs%20Units
%20KPIs%205-27-09.pdf
• Foundations of Excellence (study of the first-year at UNI)
• Academic Advising assessment plan
• Voluntary System of Accountability College Portrait
http://www.collegeportraits.org/IA/UNI
2008-9 Assessment Data
For the Liberal Arts Core
Tips for Viewing the Data Slides
• Focus on the types of data available for now; examine
specific data later.
• Think about potential goals for student performance and
how we can/should go about setting such goals.
• Identify areas that are most important—to you, your
department, the university.
• Consider what particular data sets might imply for the
LAC and for curricula and performance in academic
majors—in terms of courses, assignments, pedagogy,
grading, etc.
• Ask what additional data might be needed and how/with
whom to share the data to put it to use most effectively.
Creating an Assessment Plan
• Determine learning outcomes
• Describe learning outcomes in terms of specific
behaviors
• Identify assessment strategies/instruments to provide
data related to learning outcomes
• Analyze and use the resulting data
• Evaluate/revise assessment strategies
The LAC Assessment Plan
• Assessment of learning in the LAC Categories
conducted through the Category Review process
• Assessment of student learning related to five overall
areas of skill represented in the Liberal Arts Core:
o
o
o
o
o
Communication
Information
Thinking
Interpersonal
Quantitative
Communication
Students should be able to speak, listen, read,
write, and view effectively, adapting
appropriately to the audience and material at
hand.
Speaking Effectively
Graduating Senior Survey
Rate how well UNI has prepared you for speaking
effectively (Q1)
Excellent
Good
Average
Fair
Poor
Cannot
evaluate
30.3
54.0
12.7
1.9
0.5
0.6
Speaking Effectively, continued
NSSE
11d. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in speaking clearly and effectively?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
28%
47%
22%
3%
Seniors
29%
45%
20%
6%
Listening Effectively
Graduating Senior Survey
Rate how well UNI has prepared you for listening
effectively (Q 4)
Excellent
Good
Average
Fair
Poor
Cannot
evaluate
38.3%
48.7%
11.0%
1.4%
0.4%
0.2%
Writing Effectively
Graduating Senior Survey
Rate how well UNI has prepared you for communicating
through writing (Q2)
Excellent
Good
Average
Fair
Poor
Cannot
evaluate
36.1%
52.3%
9.3%
1.7%
0.2%
0.4%
Writing Effectively, continued
NSSE
11c. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in writing clearly and effectively?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
29%
48%
18%
4%
Seniors
31%
45%
21%
4%
Writing Effectively, continued
MAPP
Norm-referenced subscore for writing
Possible range: 100-130
Year
Mean Score
25th percentile
50th percentile
75th percentile
First-Year
113.72
111
114
117
Seniors
115.55
113
116
117
Writing Effectively, continued
MAPP
Proficiency classifications for writing
2008-9 UNI scores compared to scores from ETS testing in Master’s I and II
institutions, 2003-2007
Skill
Year
Proficient
Marginal
Not Proficient
UNI
ETS
UNI
ETS
UNI
ETS
Writing,
Level 1
First-year
54%
55%
31%
29%
14%
15%
Seniors
67%
71%
27%
21%
6%
8%
Writing,
Level 2
First-year
13%
12%
34%
33%
53%
55%
Seniors
24%
22%
34%
40%
42%
38%
Writing,
Level 3
First-year
4%
5%
19%
21%
77%
75%
Seniors
9%
9%
23%
32%
68%
59%
Information
Students should be able to use both traditional
sources and modern technologies to access,
analyze, and manage information.
Information
Graduating Senior Survey
Rate how well UNI has prepared you for:
Excellent
Good
Average
Fair
Poor
Cannot
evaluate
Understanding
written
communication.
(Q3)
33.0%
52.1 %
11.9 %
1.9 %
0.5 %
0.6 %
Learning new
things (Q11)
44.9%
44.1%
9.7%
0.9%
0.2%
0.2%
Bringing info/
ideas together
from different
areas (Q13)
36.8%
48.3%
12.2%
1.9%
0.4%
0.5%
Using research
skills (Q14)
37.0%
45.6%
13.9%
2.7%
0.5%
0.4%
Information, continued
NSSE
11g. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in using computing and information
technology?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
28%
47%
22%
4%
Seniors
38%
41%
19%
3%
Information, continued
NSSE
11j. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in the area of learning effectively on your
own?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
29%
48%
19%
4%
Seniors
27%
48%
20%
5%
Thinking
Students should be able to address complex
issues and problem situations with sound
reasoning, reflective judgment, creative
imagination, and a critical, analytical bent of
mind.
Thinking
Graduating Senior Survey
Rate how well UNI has prepared you for:
Excellent
Good
Average
Fair
Poor
Cannot
evaluate
Defining
problems (Q9)
30.6%
50.3%
16.7%
1.6%
0.3%
0.5%
Solving
problems (Q10)
33.6%
51.2%
13.3%
1.2%
0.3%
0.4%
Thinking
38.3%
creatively (Q12)
44.7%
13.5%
2.4%
0.8%
0.2%
Making
decisions (Q19)
48.5%
10.5%
0.7%
0.3%
0.2%
39.8%
Thinking, continued
NSSE
11e. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in the area of thinking critically and
analytically?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
37%
44%
17%
2%
Seniors
45%
42%
11%
2%
Thinking, continued
NSSE
11m. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in the area of solving complex real-world
problems?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
20%
38%
35%
7%
Seniors
19%
40%
31%
10%
Thinking, continued
MAPP
Norm-referenced subscore for critical thinking
Possible range: 100-130
Year
Mean Score
25th percentile
50th percentile
75th percentile
First-Year
110.42
106
109
114
Seniors
113.60
109
113
118
Thinking, continued
MAPP
Proficiency classifications for writing
2008-9 UNI scores compared to scores from ETS testing in Master’s I and II
institutions, 2003-2007
Skill
Year
Proficient
Marginal
Not Proficient
UNI
ETS
UNI
ETS
UNI
ETS
Reading,
Level 1
First-year
51%
52%
24%
25%
26%
23%
Seniors
66%
72%
20%
17%
14%
11%
Reading,
Level 2
First-year
25%
22%
17%
19%
59%
59%
Seniors
39%
41%
19%
21%
42%
38%
Reading,
Level 3
(Critical
Thinking)
First-year
3%
2%
11%
8%
86%
90%
Seniors
9%
6%
15%
17%
76%
77%
Interpersonal
Students should understand human emotions,
motivations, and idiosyncrasies, and be able to
participate effectively in relationships, groups,
and citizenship activities.
Interpersonal:
Understand human emotions, motivations, and idiosyncrasies
Graduating Senior Survey
Rate how well UNI has prepared you for:
Excellent
Good
Average
Fair
Poor
Cannot
evaluate
Working with
people of
diverse
backgrounds
(Q21)
40.4%
36.5%
16.7%
4.1%
1.7%
0.6%
Working as a
team (Q22)
44.5%
42.8%
11.2%
0.7%
0.7%
0.2%
Leading
others(Q23)
40.6%
42.1%
14.3%
2.0%
0.5%
0.4%
Interpersonal, continued:
Understand human emotions, motivations, and idiosyncrasies
NSSE
11h. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in working effectively with others?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
33%
42%
23%
2%
Seniors
41%
43%
13%
3%
Interpersonal:
Participate Effectively in Relationships,
Groups, and Citizenship Activities
Graduating Senior Survey
Rate how well UNI has prepared you for:
Excellent
Good
Average
Fair
Poor
Cannot
evaluate
Planning
projects (Q8)
32.9%
46.9%
15.4%
3.4%
0.5%
0.9%
Conducting
yourself in a
professional
manner (Q15)
49.6%
40.2%
8.3%
1.1%
0.3%
0.5%
Upholding
ethical
standards
(Q16)
45.2%
41.8%
10.8%
0.9%
0.6%
0.6%
Interpersonal, continued:
Participate Effectively in Relationships,
Groups, and Citizenship Activities
Graduating Senior Survey
Rate how well UNI has prepared you for:
Excellent
Good
Average
Fair
Poor
Cannot
evaluate
39.4%
45.2%
13.6%
1.1%
0.3%
0.4%
Working under
45.8%
pressure ((Q18)
43.4%
9.3%
1.1%
0.2%
0.2%
Working
independently
(Q20)
42.4%
9.0%
0.6%
0.3%
0.2%
Adapting to
change (Q17)
47.6%
Interpersonal, continued:
Participate Effectively in Relationships,
Groups, and Citizenship Activities
NSSE
11i. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in voting in local, state, or national elections?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
29%
30%
30%
11%
Seniors
16%
25%
32%
28%
Interpersonal, continued:
Participate Effectively in Relationships,
Groups, and Citizenship Activities
NSSE
11l. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in understanding people of other racial and
ethnic backgrounds?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
18%
27%
43%
11%
Seniors
18%
33%
36%
12%
Interpersonal, continued:
Participate Effectively in Relationships,
Groups, and Citizenship Activities
NSSE
11n. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in developing a personal code of values and
ethics?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
24%
32%
35%
9%
Seniors
22%
37%
28%
13%
Interpersonal, continued:
Participate Effectively in Relationships,
Groups, and Citizenship Activities
NSSE
11o. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in contributing to the welfare of your
community?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
14%
27%
38%
20%
Seniors
14%
31%
39%
16%
Interpersonal, continued:
Participate Effectively in Relationships,
Groups, and Citizenship Activities
NSSE/American Democracy Project Consortium
16a. I believe my coursework is preparing me to
participate in politics (e.g., voting, volunteering or
contributing to a campaign, or persuading others about a
candidate or political issue).
Year
Strongly
Agree
Agree
Disagree
Strongly
Disagree
First-Year
5%
44%
43%
8%
Seniors
6%
35%
46%
13%
Interpersonal, continued:
Participate Effectively in Relationships,
Groups, and Citizenship Activities
NSSE/American Democracy Project Consortium
16b. I believe my coursework is preparing me to discuss
politics and government with my friends and family.
Year
Strongly
Agree
Agree
Disagree
Strongly
Disagree
First-Year
14%
53%
26%
8%
Seniors
17%
53%
24%
6%
Interpersonal, continued:
Participate Effectively in Relationships,
Groups, and Citizenship Activities
NSSE
11h. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in working effectively with others?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
33%
42%
23%
2%
Seniors
41%
43%
13%
3%
Quantitative
Students should be able to make effective use of
quantitative data, and to intelligently apply
relevant mathematical and statistical concepts
and methods on appropriate occasions.
Quantitative
Graduating Senior Survey
Rate how well UNI has prepared you for making basic
calculations and computations (Q6)
Excellent
Good
Average
Fair
Poor
Cannot
evaluate
32.4%
41.5%
18.9%
4.7%
0.6%
1.8%
Quantitative, continued
NSSE
11f. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in analyzing quantitative problems?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
28%
44%
25%
3%
Seniors
30%
45%
21%
4%
Quantitative
MAPP
Norm-referenced subscore for mathematics
Possible range: 100-130
Year
Mean Score
25th percentile
50th percentile
75th percentile
First-Year
113.14
109
112
117
Seniors
115.62
112
114
120
Quantitative, continued
MAPP
Proficiency classifications for writing
2008-9 UNI scores compared to scores from ETS testing in Master’s I and II
institutions, 2003-2007
Skill
Year
Proficient
Marginal
Not Proficient
UNI
ETS
UNI
ETS
UNI
ETS
Mathematics,
Level 1
First-year
45%
47%
27%
29%
27%
24%
Seniors
63%
58%
27%
26%
10%
16%
Mathematics,
Level 2
First-year
20%
20%
26%
27%
54%
53%
Seniors
33%
30%
30%
29%
37%
41%
Mathematics,
Level 3
First-year
6%
4%
11%
12%
82%
84%
Seniors
13%
8%
14%
18%
73%
74%
Humanities
MAPP
Norm-referenced subscore for humanities
Possible range: 100-130
Year
Mean Score
25th percentile
50th percentile
75th percentile
First-Year
113.56
109
113
118
Seniors
117.03
112
117
121
Social Sciences
MAPP
Norm-referenced subscore for social sciences
Possible range: 100-130
Year
Mean Score
25th percentile
50th percentile
75th percentile
First-Year
112.78
107
112
117
Seniors
115.38
110
117
121
Natural Sciences
MAPP
Norm-referenced subscore for natural sciences
Possible range: 100-130
Year
Mean Score
25th percentile
50th percentile
75th percentile
First-Year
114.05
110
113
118
Seniors
115.98
112
117
121
Final Thoughts
Evaluating the LAC
Graduating Senior Survey
Indicate the extent to which you agree or disagree with the
following statements.
Strongly
Agree
Agree
Disagree
Strongly
Disagree
Not Sure
Overall quality of most
LAC courses is excellent
14.6%
58.5%
14.5%
3.1%
9.3%
The purposes of most
LAC courses are very
clear
14.9%
52.8%
19.5%
4.2%
8.7%
Most of the LAC courses I
took at UNI were
intellectually demanding.
14.1%
55.6%
19.0%
4.0%
7.3%
I believe the LAC has
been an important part of
my education.
18.2%
50.9%
17.5%
5.4%
8.1%
Evaluating the LAC, continued
NSSE
11a. To what extent has your experience at this institution
contributed to your knowledge, skills, and personal
development in acquiring a broad general education?
Year
Very Much
Quite a Bit
Some
Very Little
First-Year
44%
40%
14%
2%
Seniors
45%
39%
14%
2%
Evaluating the LAC, continued
NSSE
13. How would you evaluate your entire educational
experience at this institution?
Year
Excellent
Good
Fair
Poor
First-Year
41%
50%
9%
1%
Seniors
41%
49%
8%
1%
Next Steps: Some Questions
• How can the data we have be put to use?
• What additional data/evidence of learning would we like to
have, and how can we collect it?
• How will revision of the LAC affect assessment strategies?
• Do we want longitudinal data, and if so, how can we test
enough students to gather the necessary data?
• The sample size for seniors is small (57-123 per year for
2007 through 2009). How can we increase it?
• How do we increase student motivation to complete tests
and surveys and do their best on them?
Thank you!
Questions/thoughts?