Assessing and Improving Student
Outcomes
Presentation for TENNAIR Conference
August 7, 2008
Gwen P. Aldridge, Ph.D.
Director of Assessment
J. Nevin Robbins, Ph.D.
Executive Director for Planning and Analysis
Presentation to include:
•
•
•
•
•
•
•
Importance of SLOs
Defining Learning Objectives
Selecting Measures & Setting Benchmarks
Comparing Outcomes with Intended Objectives
Using Results to Improve Learning
Online Documentation – Southwest’s System
SLO Hints
Current Focus on Importance of
Student Learning Outcomes
This focus represents a convergence
of two major trends in higher
education: the assessment
movement and the accountability
movement.
Assessment and Accountability
• Assessment has evolved from
efforts to improve the quality of
teaching and learning by improving
the quality of educational outcomes
• Accountability has evolved from the
efforts of state legislatures to prove
institutional effectiveness in making
higher education more cost effective
Tennessee Academic Audits for Programs
• Criterion 1.1: The faculty completed an honest analysis of
their process for developing learning objectives for the
program, considering measurability, clarity, and what
students need to know.
• Criterion 1.2 : The faculty have documented or proposed a
process for developing learning objectives that are based on
realistic and appropriate evidence.
• Criterion 1.3: The faculty have documented or proposed
specific plans to take best practices and appropriate
benchmarks into account in the analysis of learning
objectives.
Good Assessment Practice
• First - assess what is most important
• Realize that anything that can be taught
or learned can be assessed
• Apply assessment at the course,
program, and institutional levels
• Organize every course and every program
around clearly defined learning goals and
objectives, explicit assessment methods,
and measurable outcomes.
Program Outcomes
• Are needed for internal assessment of
continuing improvement
• Are needed for external assessment of
institutional effectiveness for
accountability purposes
• Are useful in communicating program
results for purposes of marketing
• Insure that the program is more than just
a cluster of courses
The Purpose of Student Outcomes Assessment Effort
Academic Program Improved
Accreditation Requirements Satisfied
Effectiveness Demonstrated
What SLOs can accomplish:
To tell students what is expected of them and thus…
 help students learn more effectively
 make it clear what students can gain
 help instructors design their materials more effectively
 help instructors select the appropriate teaching strategy
 assist in setting examinations based on the material
developed
 ensure that appropriate assessment strategies are employed
SACS Compliance Standard
3.3 Institutional Effectiveness
• 3.3.1 The institution identifies expected
outcomes, assesses the extent to which it
achieves these outcomes, and provides
evidence of improvement based on
analysis of the results in each of the
following areas (Institutional
Effectiveness):
• 3.3.1.1 educational programs, to
include student learning outcomes
• 3.3.1.2 administrative support services
• 3.3.1.3 educational support services
• 3.3.1.4 research within its educational
mission, if appropriate
• 3.3.1.5 community/public service within
its educational mission, if appropriate
Simply… the 4 Column Model:
•
•
•
•
Learning outcome
Method of assessment
Assessment results
Use of results
Assessment
Learning
Cycle
Use
results for
improving
program
Define
learning
objectives
Continuing
Improvement
in Learning
Measure
and state
results
Define
measure
and
intended
result
From SACS-COC Resource Manual
for Review Teams
“The expectation is that the
institution will engage in ongoing
planning and evaluation to ensure
that, for each academic program, the
institution develops and assesses
student learning outcomes.”
Note the “each” in the expectations – it’s not
“some”.
From SACS-COC Resource Manual for
Review Teams - continued
“Program and learning outcomes are
grounded in the faculty’s knowledge
of the content and coherence of the
discipline as well as in the learning
process and reflect expectations for
performance consistent with the
level of the program and the mission
of the institution.”
From SACS-COC Resource Manual for
Review Teams - continued
Sample Documentation for this requirement:
Representative examples of program and learning
outcomes for each educational program
Descriptions of methods for evaluating student
achievement of these outcomes
Reports of the results of the evaluation, examples
of how the results have been used for program
improvement, and examples of how methods of
evaluation have been improved over time
From SACS-COC Resource Manual for
Review Teams - continued
“At appropriate intervals, program and
learning outcomes and evaluation
methods are evaluated and revised
as appropriate.”
From SACS-COC Resource Manual for
Review Teams - continued
“Program and learning outcomes specify the
knowledge, skills, values, and attitudes students
are expected to attain in courses or in a
program.”
“Methods for evaluating the extent to which
students achieve these outcomes are
appropriate to the nature of the discipline and
consistent over time to enable the institution to
evaluate cohorts of students who complete
courses or a program.”
UNIT ANNUAL OUTCOME OBJECTIVES EDUCATIONAL PROGRAMS
Unit/Program
Intended
Outcomes
Intended Method of
Measurement
and Level of
Performance
Assessment/Evaluation
Results
Use of Results
Improvement
What
How
Later...
Either
do you want
students to know
think or do when
they have
completed the
program?
will you measure?
What measurement
activity will
demonstrate that
students
accomplished the
outcome, and what
overall level of
performance do
you wish to set?
What was the
score?
Make a
change and
state that
change
Or state
“No action
necessary.”
Was performance up
to the level you set?
Give Results.
Column 1: Defining
Learning Objectives
The dream begins with a teacher who
believes in you, who tugs and
pushes and leads you to the next
plateau, sometimes poking you with
a sharp stick called "truth."
• ~Dan Rather
Promising practices
in Defining Learning Objectives
Make certain that Learning Objectives
Are specific to the program
Are focused on what is critical to the program
Are descriptive of the what a student should gain as a result of their
completion of the program
Are clear and understandable
Are written to a level of specificity while allowing a certain amount of
interpretation leeway for faculty
Use action verbs
Are realistic given the typical student in the program
Are assessable
Is outcome singular?
Singular – Statement of One Outcome
Don’t bundle multiple outcomes in a single
statement. This leads to problems in formulating
means of assessment.
When writing SLOs, the focus should be on observable
outcomes and an “action verb” can provide that focus.
SLOs usually begin with something like:
By the end of the secondary education program,
students will be able to design curriculum and
appropriate instruction.
Design is the “action verb” in this example.
Level
Action Verbs
Outcome Example
Knowledge
Recite, List
By the end of the chemistry program, students
will be able to list all of the elements on the
Periodic Table.
Comprehension
Translate, interpret,
predict, generalize,
identify examples
By the end of the French program, students will
be able to translate a paragraph of text from
English to French.
Application
Apply, rewrite
By the end of the BIS program, students will be
able to apply basic Web development skills.
Analysis
By the end of the special education program,
Analyze, dissect,
resolve, solve, diagnose, students will be able to diagnose learning
disabilities in K-12 settings.
investigate
Synthesis
Create, synthesize, write By the end of the art program, students will be
able to create at least 12 original works in their
medium.
Evaluation
Evaluate, judge, rate,
appraise
By the end of the music education program,
students will be able to judge student
performances.
Level
Action Verbs
Outcome Example
Perception
Chooses, describes, detects,
differentiates, relates, selects
By the end of the music theatre program, students will
be able to relate types of music to particular dance
steps.
Set
Begins, displays, explains,
moves, proceeds, reacts,
responds, shows
By the end of the physical education program, students
will be able to show the proper stance for batting a ball.
Guided, Complex,
Mechanical Response
Assembles, builds, constructs,
dissects, fastens, fixes, grinds,
heats, measures mends, mixes,
sketches
By the end of the aviation maintenance program,
students will be able to draw sketches of aircraft repairs
and alterations.
Adaptation
Adapts, alters, changes,
rearranges, reorganizes,
revises, varies
By the end of the industrial education program, students
will be able to adapt their lessons on woodworking
skills for disabled students
Origination
Arranges, combines,
composes, constructs, creates,
designs, originates
By the end of the dance program, students will be able
to create a dance step.
Statements of Educational Outcomes
should answer WHAT?
What students should:
Know (cognitive)
Think (affective)
Do (behavioral)
when they have completed a degree program.
How many SLOs are enough?
A program may have only a few SLOs or a long list of
them. A total of five to seven outcomes is typical for a
program.
Most career-technical programs have standardized student competency
profiles and instructors may select the learning outcomes from these. An
instructor should not list each competency, but should include the ones
deemed most important.
Specific (who or what will be changed)
_______
Time-Based
__________ (when will the changes take place)
__________
Measurable (how much change will there be)
Goals are necessary for improvement because
when we shoot at nothing, we usually hit it.
Evaluating SLO Statements
1. Are they reasonable given student ability?
2. Are they clear?
3. Is accomplishment measurable?
4. Is outcome singular (not a bundle of outcomes)?
5. Do they focus on what students can know, think,
or do as a result of completing the program?
A Citation for One Institution Regarding Learning Outcomes:
3.4.1 Non-Compliance: The On-Site Committee should seek
evidence that the college establishes and evaluates student
learning outcomes for each of its educational programs.
•
Comments from the Off-site team: The execution of the student
learning outcome system varied from effective and accurate in
the description of student learning outcomes and their
assessment to ineffectively broad in defining the student learning
outcomes and assessments not keyed to the desired outcomes.
For example, certain programs effectively and descriptively
focused on what students could do, know or believe as a result of
completing their programs and also used assessment tools
directly addressing the expected outcomes. Other programs
presented groups of student learning outcomes from courses
making up the program or from a course sub-section as one
learning outcome.
Identify the problem here:
Unit: Commercial Truck Driving
• The students will demonstrate the
ability to safely operate commercial
trucks.
Outcome:
Funeral Service Technology: The student
will be able to explain the techniques and
theory of all elements of funeral service.
General Agriculture: Students will
demonstrate knowledge of
photosynthesis, plant anatomy, cellular
functions, classifications and genetics.
Commercial Truck Driving: The students
will demonstrate the ability to safely
operate commercial trucks.
Business and Office Technology:
Students will demonstrate the ability to
keyboard alphanumeric material at
specified speed and accuracy levels.
Problem
Outcome:
Problem
Funeral Service Technology: The student
will be able to explain the techniques and
theory of all elements of funeral service.
•Ineffectively
Broad
General Agriculture: Students will
demonstrate knowledge of
photosynthesis, plant anatomy, cellular
functions, classifications and genetics.
•Groups of
Learning
Outcomes
Commercial Truck Driving: The students
will demonstrate the ability to safely
operate commercial trucks.
•Ineffectively
Broad
Business and Office Technology:
Students will demonstrate the ability to
keyboard alphanumeric material at
specified speed and accuracy levels.
•Acceptable
Column 2: Selecting Measures
& Setting Benchmarks
• Are Assessments Keyed to the
Desired Outcome?
• Are Assessments Reasonable Given
Student Ability?
• “Fledgling skiers make the most progress when
they are pushed outside their comfort zone, but
not so far that they are scared off the slopes
altogether.” a ski instructor
Assessment of the Outcomes
Each SLO should have its own assessment.
At the end of the assessment, an instructor should know which
SLOs were achieved, and which were not achieved. If goal related
questions are used on a final exam to assess more than one outcome,
an item analysis should be done.
For example, 80% of students achieved Outcome 1, 60% of students
achieved Outcome 2, 75% of students achieved Outcome #3…
Without such individual assessment of outcomes, the
instructor fails to know in the end what was
accomplished and thus what individual changes in
instruction should be made in improving Student
Learning in the program. The ultimate purpose of
writing and assessing SLOs is Program
Improvement.
Some acceptable measures to evaluate SLOs:
• standardized tests with questions tied to specific SLOs
• locally developed tests with SLO goal-related questions
identified (use item analysis to judge performance on each
individual outcome)
• analysis of theses, portfolios, recitals, speeches
• student performance in lab activity
Assessments…Are they reasonable given
student ability?
Make sure criteria (bench marks) for success and
expected student achievements are not set at levels
far beyond the ability of most students.
We should STRETCH, but not STRAIN.
Individual Student
Grading and Educational Outcomes Assessment
Individual Students
Student 1
Student 2
Student 3
Student 4
Student 5
Criteria/Intended
Educational
Outcomes Average
Cell Biology
4
9
5
7
9
6.8
Metabolism
3
9
7
8
9
7.2
Reproduction
9
9
7
9
9
8.6
Evolution
Taxonomy
2
7
9
9
4
7
4
8
4
9
4.6
8.0
TOTAL
25
45
30
36
40
F
A
D
C
B
SLO
Individual
Student Grade
Class Grades or G.P.A. as a Means of Assessment
Why not just use the course grades as assessment?
 With course grades, we have assessment of individuals,
not programs
 Also, the objectivity of the evaluator is questioned
 Finally, an instructor is not provided with information
that allows that instructor to assess student
accomplishment on the individual SLOs
(Generally grades are not accepted as a mean of
assessment for SLOs).
Biology I Classes
Grading and Educational Outcomes Assessment
Individual Section Averages
SLO
Criteria/Intended
Educational
Outcomes Average
Section 1
Section 2
Section 3
Section 4
Section 5
Cell Biology
3
7
5
6
8
5.8
Metabolism
4
8
8
7
7
6.8
Reproduction
8
9
8
8
9
8.4
Evolution
Taxonomy
3
6
7
9
4
7
3
8
3
7
4.0
7.4
Citation for NWCC from SACS-COC
3.4.1 Non-Compliance Recommendation: The On-Site Committee should seek
evidence that the college establishes and evaluates student learning outcomes for
each of its educational programs.
Comments from the Off-site team: …One program listed all student learning
outcomes from each course, but used class management and course requirement
standards as assessment means. In assessing learning outcomes, most programs
used a retention indicator instead of a learning outcome indicator, that of course
completion, or a measurement tool that did not address the desired outcome. In
those cases a “C” grade in a course or a comprehensive exam grade without
targeting portions of the exam relevant to the learning outcome were presented
as proof of achieving the outcome.
Art: Students will demonstrate mastery of hand
building pottery techniques by creating forms
with pinch pots, coiled symmetrical cylinders,
and crafting coil and slab combinations.
After completing the pottery course, 75% of students will
demonstrate mastery of this technique by making a C or
better in the course.
Students will demonstrate mastery of this techniques by
making a C or better on the final exam.
After competing the pottery course, 75% of students will
demonstrate mastery of this technique by receiving a rating
of “good” or better on pieces produced based on class
critique and the grading of formed pieces. Emphasis will be
on use of knowledge attained, consistency in building, and
creativity.
Art: Students will demonstrate mastery of hand
building pottery techniques by creating forms with
pinch pots, coiled symmetrical cylinders, and crafting
coil and slab combinations.
After completing the pottery course, 75% of students will
demonstrate mastery of this technique by making a C or
better in the course.
No!
Students will demonstrate mastery of this techniques by
making a C or better on the final exam.
No!
After competing the pottery course, 75% of students will
demonstrate mastery of this technique by receiving a
rating of “good” or better on pieces produced based on
class critique and the grading of formed pieces. Emphasis
will be on use of knowledge attained, consistency in
building, and creativity.
Okay
With “Assessments not keyed
to desired outcomes…”
Outcome: Agriculture
Technology
Assessment
1. Student will demonstrate the
ability to disassemble and
reassemble clutch packs from the
16 speed power shift
transmission.
90% of all students completing
the program will demonstrate
skills by completing student
competencies.
2. Student will demonstrate an
ability to repair and replace
component parts of a machine
including eccentrics, shafts,
bearings, fasteners and o-rings.
3. Students will demonstrate the
ability to service the fuel system
in internal combustion engines.
Column 3: Comparing Outcomes
with Intended Objectives
At some point, you just have to
decide that you are not going
to be afraid of the data.
It may not be pretty…
But you can use it.
Column 4: Using Results to Improve
Learning
How might we use the results?
We may decide to…
Change the related SLO
Adjust the Program/Unit Activities
Modify the Means of Assessment
Who Kissed the Mirror?
The End Product of Assessment Activities
Assessment results are used for Improving
Educational Programs…. Thus Column 4,
our “Use of Results”.
Ever tried. Ever failed. Try again. Fail again. Fail better.
Samuel Beckett
Statement of “use of results”
• The Use of Results should be stated in
past tense - an action that has already
taken place to improve the program.
• But, what if the improvement won’t take
place until the following year (semester)?
You may say, “A decision was made to
begin…….. in the upcoming semester.”
Make the decision now.
UNIT ANNUAL OUTCOME OBJECTIVES EDUCATIONAL PROGRAMS
Unit/Program Intended
Outcome
Welding:
Students will be able to
perform welds for all
position fillet and
groove welding within a
limited thickness range
of plain carbon steel
material.
Intended Method of
Measurement
and Level of
Performance
Assessment/Evaluation
Results
75% of students
80% of students in Gas successfully completed
Tungsten Arc Welding this competency profile.
will perform all welds
and welding operations
identified in the Student
Competency Profile.
Use of Results
Improvement
A decision was made to
give students more
hands-on in this skill
and to spend more time
on questions and
answers.
A Yearly Implementation Cycle
Identify Expected Outcomes & Identify Means of
Assessment:
(Columns 1 & 2)
Carry Out Means of Assessment
early Fall
during the school year
Report Assessment and Use of Results (Columns 3 & 4)
Either in May at the end of the school year or the following August
We are continually faced with a series of great opportunities
brilliantly disguised as insoluble problems…John Gardner
SLO Hints:
For SLOs to accomplish anything,
The administration at a college must take steps
to promote inquiry and alleviate fear. Instructors
must be assured that assessment results of the
SLOs will be in no way tied to their own
evaluations.
Lessons Learned…
Some SLO Hints
• There are some unacceptable words that
sometimes jump into the assessment
column that just won’t work (retention,
course grade of C or better,
comprehensive exam grade)
Lessons learned…
• The expectations of Peer Review Teams
are as varied as the individuals who
comprise those teams. You can never
predict in advance what a team will wish
to focus on, or how knowledgeable team
members will be on the topics being
reviewed – but you can be sure of one
thing, and that is that SLOs are currently
a hot topic.
Celebrate Success
Your college is a haven for excellent
student learning.
Embrace assessment, learn to
measure student learning
communicate it, and celebrate it!
Gwen P. Aldridge, Ph.D.
Director of Assessment
J. Nevin Robbins, Ph.D.
Executive Director for Planning and Analysis
Southwest TN Community College
Memphis, TN
Office: 901-333-5257
gpaldridge@southwest.tn.edu
nrobbins@southwest.tn.edu