7/21/2011
Defining
g and Measuring
g
Student Learning
Outcomes
Dr. Tammy Kahrig
July 14, 2011
Office of Strategic
Information Resources
• Director, Annette Marksberry
• Institutional Research (Susana
Luzuriaga)
• Institutional Assessment (Tammy)
• Projects and Planning (Tim Bucher)
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
1
7/21/2011
Types of Assessment
Measuring
Participation
Needs Assessment
Satisfaction
Assessment
Cost Effectiveness
Benchmarking or
Comparable
Institution
Assessment
National Standards
Assessment
Campus
Environment/Climate
Assessment
Student Learning
Outcomes
Assessment
Types of Assessment
Measuring
Participation
(
(annually)
ll )
Cost Effectiveness
(every 5-7 yrs.)
Satisfaction
Assessment
Needs Assessment
(every 2
2-3
3 yrs.)
Benchmarking or
Comparable
Institution
Assessment
(every 2-3 yrs.)
(
(every
2
2-3
3 yrs.))
National Standards
Assessment
(every 5-7 yrs.)
Campus
Environment/Climate
Assessment
Student Learning
Outcomes
Assessment
(every 5-7 yrs.)
(annually)
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
2
7/21/2011
Conversations about Student Learning
and Development at Xavier University
What are our students’ (as a whole) strengths and weaknesses?
What should we work on as a university?
Curriculum:
Core and Degree Programs
OSIR Data Collection
Co-Curricular Units
Program-Level SLO
Assessment
Student
Surveys
Division-Level
Assessment of Student
Learning
Course-Level SLO
Assessment
Tests
Assessment
(collectively) within Unit
Individual Section-Level
SLO Assessment
Retention
Studies
Individual
Initiative/Event
Assessment
Individual Student
Assessment (Grading and
Feedback)
Alumni Surveys
Individual Student
Assessment and
Feedback
Curriculum:
Core and Degree Programs
Co-Curricular Units
Program-Level
P
L
l SLO
Assessment
Division-Level
Division
Level
Assessment of
Student Learning
Course-Level SLO
Assessment
Assessment
(collectively)
within Unit
Individual SectionLevel SLO
Assessment
Individual
Initiative/Event
Assessment
Individual Student
Assessment (Grading
and Feedback)
Individual Student
Assessment and
Feedback
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
3
7/21/2011
External Expectations
Today’s Goal and Agenda
Learning
Outcomes Part
II: Revising and
Identifying
Measures
More Practice in
Writing Learning
Outcomes; and
Identifying
Measures
Divisional and
Unit-Level
Learning
O t
Outcomes
th
thatt
are Measurable
(and Measures)
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
4
7/21/2011
STEPS of SLO
ASSESSMENT
Identify
goals for
learning.
(Student
Learning
Outcomes)
Gather
information
Discuss, and
Identify
use the
where in the about how
well
information
“curriculum”
students are to improve
the goals
achieving
learning.
are
those
goals
goals.
addressed
addressed.
(Action/
(Curriculum
Mapping)
(Methods:
Direct and
Indirect
Measures)
Continuous
Improvement)
Definition:
Student Learning Outcomes
Learning
g outcomes or learning
g
goals are goals that describe how
students will be different because
of a learning experience. More
specifically, learning outcomes are
the knowledge,
knowledge skills,
skills attitudes
attitudes,
and habits of mind that students
take with them from a learning
experience. (Suskie, p. 117)
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
5
7/21/2011
Characteristics of Program/UnitLevel Student Learning Outcomes
Describes what students
should be able to
d
demonstrate,
t t
represent, or produce
as a result of what and
how they have learned in
a program;
Focuses on what you
expect to see at the end
(outcome) rather than
what you plan to do or
how it will be done (goal
or process);
Focuses on the most
important goals for
students in your unit;
Relies on an action verb
that identify what
students should be able
to demonstrate,
represent, or produce
over time
Characteristics of Program/UnitLevel Student Learning Outcomes
Offers multiple and varied
opportunities for students
to learn through your
efforts/curriculum;
Aligns with collective
program- and institutionl
level
l educational
d
ti
l
intentions for student
learning translated into
the curriculum and cocurriculum;
Incorporates or adapts
professional
organizations’ outcome
statements when they
exist;
Can be quantitatively
or qualitatively
assessed during
students’ undergraduate
or graduate studies.
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
6
7/21/2011
Basic Formula for Program/Unit-Level
Student Learning Outcomes
As a result of
participating
in_____,
students
Will be able
to
________
(action
verb)
b)
Product/
process/
outcome
Broad, Nebulous Terms to
Avoid
•acknowledge
•appreciate
•aware of
•believe
•capable of
•comprehend
comprehend
•conscious of
•enjoy
•experience
•knows/ has
knowledge of
•learn
•memorize
•show interest in
•understand
understand
•familiar with
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
7
7/21/2011
Recognition of
Pitfalls and
Practice
SLL Division Learning
Outcomes
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
8
7/21/2011
Writing and Revising
Your SLOs
Students on Student
g Outcomes
Learning
• http://youtu.be/BWtlhGF3ejY
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
9
7/21/2011
Identifying Measures for
Student Learning
Outcomes
Efficiency in Assessment:
Where to Begin
Current practices
•What are you doing already to assess student
learning? Do you use any existing institutional data?
•Where do conversations about student learning occur?
IInitiative-level
iti ti
l
l tto U
Unit-level
it l
l
Discussions
• What opportunities are in place for taking information about
student learning from the your initiative to the unit level
(everyone teaching that kind of material)?
Unit-Level to Division-Level
Discussions
•What opportunities are in place for taking information
from unit-level to the division-level?
End-of-term Department
Meeting (or Sub-Group Meeting)
Comprehensive
Assignments/Activities
Common Assignments across
Initiatives within Your Unit or the
Division
“Capstone” Experiences
• What regular meetings already exist for
discussing student learning?
•Where (if any) are those student-level assignments
g
in
your unit that get at multiple student learning
outcomes?
•What common assignments exist currently?
•Where are opportunities for a common assignment?
•Are there any initiatives in the division that function as
a “capstone” experience, i.e. programs that address
multiple student learning outcomes?
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
10
7/21/2011
Direct
Measures
IIndirect
di
t
Measures
• Prompt students to represent or
demonstrate their learning or
produce work so that observers
can assess how well students’
texts or responses fit programlevel expectations
• Signs
Si
th
thatt students
t d t are probably
b bl
learning, often based on reports
of perceived student learning
Examples of Direct Measures of
Student Learning
Student Work
• Written work scored using a
rubric
• Performances scored using a
rubric
• Presentations scored using a
rubric
• Projects, case studies, and
exhibitions, scored using a
rubric
• Portfolios
• Tests such as quizzes in key
initiatives
Ratings
R
ti
off
Student Skills
Standardized
Tests
• Supervisors or others
rate students on skills
using a rubric (based
on direct observation)
• Instruments to
Measure Specific
Skills (e.g. California
Critical Thinking Skills
Test)
• Pre-post tests
Direct measures are often limited to students with whom you have
ongoing, sustained, deep interactions –
e.g. student committee members, hall staff, student workers, etc.
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
11
7/21/2011
Examples of Indirect Measures of
Student Learning
Most Efficient:
Student
Response
•Students’ perceptions of
their educational
experiences and the
institution’s impact on
their learning.
•List PSLOs and ask the
students
•a. How well did you
achieve each g
goal?
•b. What aspects of the
education provided
through the
initiative/unit helped you
with your learning?
•c. What might the unit
do differently to help you
learn more effectively?
National Student
Surveys
•ACHA/NCHA
•ACUHO-I/EBI
•Core Survey
•National Survey of
Student Engagement
(NSSE)
•Beginning College Survey
of Student Engagement
(BCSSE)
Student Focus
Groups
•held with representative
students to probe a
specific issue that might
have been identified
Examples of Indirect Measures of
Student Learning
Alumni, Employer
Alumni
Employer,
or Faculty-Staff
Surveys
Admission Rates
into Graduate
Programs
Placement Rates of
Graduates into
Appropriate Career
Positions
• Perceptions of student
learning and the
institutions’ impact on
their learning
• Alumni: ACT Alumni
y
Outcomes Survey
provided by Xavier
University OSIR
• Possible
measure if
established as
relevant to unit
or division’s
division s
student learning
outcomes
• Possible
measure if
established as
relevant to unit
or division’s
division s
student learning
outcomes
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
12
7/21/2011
How does a measure get from being a
student-level or initiative-level assessment
to a unit-level or division-level assessment?
1 yields information about specific
1.
strengths and weaknesses of
students as a whole
2. is shared, discussed, and used to
improve
p
learning
g
Resources to Assist
You
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
13
7/21/2011
Student Learning Outcomes
Essentials
Mission, Goals
Bloom’s
Taxonomy
Cognitive
Development
Learning
Reconsidered 2
CAS 16
FALDOs
AAC & U
Rubrics
Mental
Measurements
Yearbook
Tammy Kahrig, Ph.D.
Xavier University Office of Strategic Information Resources
14
Bloom’s Taxonomy
Cognitive Domain
Learning Outcomes Related To Knowledge
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Student
remembers or
recognizes
information or
specifics as
communicated
with little
personal
assimilation.
Student grasps the
meaning behind
the information and
interprets,
translates,
or comprehends
the
information.
Student uses
information to
relate and
apply it to a
new situation
with minimal
instructor
input.
Student
discriminates,
organizes, and
scrutinizes
assumptions in
an attempt to
identify
evidence for a
conclusion.
Student
creatively
applies
knowledge
and
analysis to
integrate
concepts or
construct an
overall
theory.
Student
judges or
evaluates
information
based upon
standards
and criteria,
values and
opinions.
Cite
Label
List
Enumerate
Identify
Imitate
Match
Name
Quote
Recall
Reproduce
State
Write
Convert
Define
Describe
Discuss
Estimate
Explain
Generalize
Identify
Illustrate
Locate
Paraphrase
Restate
Summarize
Apply
Chart
Compute
Demonstrate
Determine
Dramatize
Establish
Make
Manipulate
Prepare
Project
Solve
Use
Analyze
Compare
Contrast
Correlate
Diagram
Dissect
Differentiate
Distinguish
Infer
Investigate
Limit
Outline
Separate
Assemble
Create
Construct
Design
Develop
Formulate
Generate
Hypothesize
Initiate
Invent
Modify
Reframe
Synthesize
Access
Appraise
Conclude
Critique
Decide
Defend
Diagnose
Evaluate
Judge
Justify
Rank
Recommend
Support
Basic
Knowledge
Level
More Sophisticated
Higher Level Thinking
Critical Thinking
15
Bloom’s Taxonomy
Psychomotor Domain
Learning Outcomes Related To Skills
Observe Model
Recognize Correct
Standards
Apply
Coach
Students
translate
sensory
input into
physical
tasks or
activities.
Students
are able to
replicate a
fundamental
skill or task.
Students
recognize
standards or
criteria
important to
perform a skill
or task
correctly.
Students use
standards to
evaluate their
own
performances
and make
corrections.
Students
apply this
skill to real
life
situations.
Students are
able to
instruct or
train others
to perform
this skill in
other
situations.
Hear
Identify
Observe
See
Smell
Taste
Touch
Watch
Attempt
Copy
Follow
Imitate
Mimic
Model
Reenact
Repeat
Reproduce
Show
Try
Check
Detect
Discriminate
Differentiate
Distinguish
Notice
Perceive
Recognize
Select
Adapt
Adjust
Alter
Change
Correct
Customize
Develop
Improve
Manipulate
Modify
Practice
Revise
Build
Compose
Construct
Create
Design
Originate
Produce
Demonstrate
Exhibit
Illustrate
Instruct
Teach
Train
*Usually no
outcomes
or
objectives
written at
this level.
Basic Knowledge
Basic Skills
Level
More Sophisticated Skills
Higher Level Abilities
Critical Understanding of Performance
16
Bloom’s Taxonomy
Affective Domain
Learning Outcomes Related To Attitudes, Behaviors, and Values
Receiving Responding Valuing
Organizing Characterizing
Students
become
aware of an
attitude,
behavior, or
value.
Students exhibit
a reaction or
change as a
result of
exposure to an
attitude,
behavior, or
value.
Students
recognize
value and
display this
through
involvement
or
commitment.
Students
determine a
new value or
behavior as
important or a
priority.
Students integrate
consistent behavior
as a naturalized
value in spite of
discomfort or cost.
The value is
recognized as a part
of the person’s
character.
Accept
Attend
Describe
Explain
Locate
Observe
Realize
Receive
Recognize
Behave
Comply
Cooperate
Discuss
Examine
Follow
Model
Present
Respond
Show
Studies
Accept
Adapt
Balance
Choose
Differentiate
Defend
Influence
Prefer
Recognize
Seek
Value
Adapt
Adjust
Alter
Change
Customize
Develop
Improve
Manipulate
Modify
Practice
Revise
Authenticate
Characterize
Defend
Display
Embody
Habituate
Internalize
Produce
Represent
Validate
Verify
Elementary Values and Behaviors
More Highly Developed Attitudes
Inherited Value System
Well Thought-out Value System
Egocentric View
Higher Level Abilities to Identify and
Articulate Others’ Values
17
Knowledge
define
identify
describe
label
list
name
state
match
recognize
select
examine
locate
memorize
quote
recall
reproduce
tabulate
tell
copy
discover
duplicate
enumerate
listen
observe
omit
read
recite
record
repeat
retell
visualize
Action Words for Bloom’s Taxonomy
Understand
Apply
Analyze
Evaluate
explain
describe
interpret
paraphrase
summarize
classify
compare
differentiate
discuss
distinguish
extend
predict
associate
contrast
convert
demonstrate
estimate
express
identify
indicate
infer
relate
restate
select
translate
ask
cite
discover
generalize
give examples
group
illustrate
judge
observe
order
report
represent
research
review
rewrite
show
trace
transform
solve
apply
illustrate
modify
use
calculate
change
choose
demonstrate
discover
experiment
relate
show
sketch
complete
construct
dramatize
interpret
manipulate
paint
prepare
produce
report
teach
act
administer
articulate
chart
collect
compute
determine
develop
employ
establish
examine
explain
interview
judge
list
operate
practice
predict
record
schedule
simulate
transfer
write
Center for University Teaching, Learning, and Assessment
analyze
compare
classify
contrast
distinguish
infer
separate
explain
select
categorize
connect
differentiate
discriminate
divide
order
point out
prioritize
subdivide
survey
advertise
appraise
break down
calculate
conclude
correlate
criticize
deduce
devise
diagram
dissect
estimate
evaluate
experiment
focus
illustrate
organize
outline
plan
question
test
reframe
criticize
evaluate
order
appraise
judge
support
compare
decide
discriminate
recommend
summarize
assess
choose
convince
defend
estimate
find errors
grade
measure
predict
rank
score
select
test
argue
conclude
consider
critique
debate
distinguish
editorialize
justify
persuade
rate
weigh
Create
design
compose
create
plan
combine
formulate
invent
hypothesize
substitute
write
compile
construct
develop
generalize
integrate
modify
organize
prepare
produce
rearrange
rewrite
role-play
adapt
anticipate
arrange
assemble
choose
collaborate
collect
devise
express
facilitate
imagine
infer
intervene
justify
make
manage
negotiate
originate
propose
reorganize
report
revise
schematize
simulate
solve
speculate
structure
support
test
validate
http://uwf.edu/cutla/SLO/ActionWords.pdf
References Diamond, R. M. (1998). Designing and assessing courses & curricula: A practical guide. San Francisco, CA: Jossey‐Bass. Fresno City College Curriculum Committee. Student learning outcomes: A guide for faculty. Available at http://online.fresnocitycollege.edu/senate/curriculum/slo.html. Maki, P.L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus. Schuh, J.H., Upcraft, M.L. & associates (2001). Assessment practice in student affairs: An application manual. San Francisco, CA: Jossey‐Bass. Suskie, L.A. (2009). Assessing student learning: A common sense guide. San Francisco, CA: Jossey‐Bass. University of West Florida Center for University Teaching, Learning, & Assessment. Bloom’s action words. Available at http://uwf.edu/cutla. Walvoord, B.E. (2010). Assessment clear and simple: A practical guide for institutions, departments, and general education. San Francisco, CA: Jossey‐Bass.