The Nuts and Bolts of
Outcome Assessment
Terri Manning
Denise Wells
Planning and Research
Central Piedmont Community College
Much confusion!




There are so many words being used in education
right now that involve some sort of measurement
that people are generally confused, throw up their
hands and walk away.
What are the differences if any between testing,
measurement and assessment?
Are objectives and outcomes the same thing?
What is the difference between a learning outcome,
an outcome target, an educational output, an
outcome objective….. and the list goes on and
on……
Our Task…...
1. To explain all these definitions
(with examples) so you can
understand them.
2. To prepare you to guide your
department/unit in establishing and
measuring outcomes.
Stages of Grief for Outcome Measurement
Stage 4
Stage 5
Acceptance & adaptation
Challenge & competition
Catalyst - Proactive
Depression
Compliance - Passive reactive
Stage 3
Bargaining - no time/no money
Seek outside sources
Stage 2
Anger and antagonism
Resistant & Reactive
Stage 1
Disbelief & Denial
Paralysis - Passive resistance
In Education…




We’ve learned that things come and they go
Most of these trends are purely academic
exercises
What we do on a daily basis doesn’t change
regardless of the mission, vision and goals
There will be new “edu-babble” things to do
next year
Changes in Higher Education
We have moved from:



a culture of unexamined
assumptions
a culture of implicit,
individually held hopes,
preferences and beliefs
a model of higher education
as primarily a quantitative,
additive process
To:



a culture of assessment and
evidence
a culture of explicit broadly
shared goals, criteria and
standards
a model that is fundamentally
qualitative and transformative
Changes in Higher Education,
continued
We have moved from:

a culture that tends to ignore
costs

a teaching culture which
ignores what is known about
human learning
a culture that emphasizes and
privileges individual struggle
for private advantage

To:

a culture that attempts to
realistically account for direct,
deferred and opportunity costs

one which applies relevant
knowledge to improve learning

one which encourages
collaboration for the common
good and individual
advancement
Long Story Short….




Accountability in higher education is
here to stay!
Measuring Outcomes is here to stay
and makes good sense!
The national educational climate is
skeptical about accountability!
They want us to prove that students are
learning, that their lives are improving
and that we are good stewards of funds!
The Spellings Commission

Some things the draft report called for:




The creation of an overall measurement of an institution's
"bottom line," including measures of institutional costs and
performance that let parents and policy makers view
institutional results;
A mandate that institutions measure student learning
outcomes, disseminate the results to students, and report
them publicly in the aggregate;
The development of a national student unit-record database to
follow the progress of each student;
The establishment a national accreditation framework that
includes comparable performance measures, and making the
findings of reviews easily accessible to the public;
SACS Requirements and Proposed
Changes


Core Requirement 2.5 (from 2003)
 The institution engages in ongoing, integrated and
institution-wide research-based planning and evaluation
processes that incorporate a systematic review of
programs and services that (1) results in continuing
improvement and (2) demonstrates that the institution is
effectively accomplishing its mission (Institutional
Effectiveness.)
Proposed Change 12/06
 The institution engages in ongoing, integrated, and
institution-wide research-based planning and evaluation
processes that (1) incorporate a systematic review of
institutional mission, goals and outcomes; (2) result in
continuing improvement in institutional quality and (3)
demonstrate that the institution is effectively
accomplishing its mission (Institutional Effectiveness)
SACS, continued


Comprehensive Standard 3.3.1 states “The institution identifies
expected outcomes for its educational programs and its administrative
and educational support services; assesses whether it achieves these
outcomes; and provides evidence of improvement based on analysis
of those results.”
Proposed and Tabled – (12/06) The institution identifies expected
outcomes, assesses whether it achieves these outcomes, and
provides evidence of improvement based on analysis of the results in
each of the following areas:





educational programs, to include student learning outcomes at the
program and individual level
administrative support services
educational support services
research within its educational mission, if appropriate
community/public service within its educational mission, if appropriate
SACS, continued

Comprehensive Standard 3.4.1 The institution
demonstrates that each educational program for which
academic credit is awarded (a) is approved by the faculty and
(b) establishes and evaluates program and learning
outcomes.

Changed (12-06) (Educational programs: all educational

programs (includes all on-campus, off-campus and distance
learning programs and course work)
The institution demonstrates that each educational program
for which academic credit is awarded is approved by the
faculty and the administration.
SACS, continued

Comprehensive Standard 3.5.1 The
institution identifies college-level competencies
within the general education core and provide
evidence that graduates have attained those
competencies.

Changed (12-06) The institution identifies
college-level general education competencies and
uses best practices in assessment to provide
evidence that graduates have attained them.
So an institution needs to have:

Ongoing, integrated, and institution-wide research-based planning and
evaluation processes which includes:




A list of expected outcomes, the assessment of those outcomes and
evidence of improvement based on analysis of those outcome results
in each of the following areas:




a systematic review of institutional mission, goals and outcomes
results in continuing improvement in institutional quality
demonstrates that the institution is effectively accomplishing its mission
educational programs (student learning outcomes at the program and
individual level)
administrative support services
educational support services
Identified college-level general education competencies (based on
best practices in assessment) and provide evidence that graduates
have attained them
But Why?



Are we doing this only because of SACS?
Who is SACS?
Shouldn’t we periodically take a serious look
at our students? Are they learning? Who is
learning best? Are they achieving the
outcomes we expected? Should we make
changes in programs and services? Do we
need more in-depth services? Do we need a
new curriculum or a change in methodology?
The Great Fallacy

Grades




In this day of social promotion, grade inflation and
different teaching/learning philosophies, grades
tell you virtually nothing.
They are not a measure of outcome achievement.
Two teachers will grade a student differently for
the exact same work.
They cannot be used!
How it Works in Most Schools…
Mission
Values
Philosophy
Goals
Unit
Objectives
Goals
Unit
Objectives
Outcome
Objectives
Unit
Objectives
Outcome
Objectives
Goals
Unit
Objectives
Outcome
Objectives
Unit
Objectives
Outcome
Objectives
Unit
Objectives
Outcome
Objectives
Mission Statement

It is the first step in any planning process

It establishes an identity for the
organization that is more expansive than
what one individual or small group can
bring to the organization

It has to be an enduring vision of future
direction and values

It identifies the organizations worthwhile
work
Mission Statement

Helps the organization accomplish goals,
receive recognition, accomplish set tasks,
observe program success, move the
organization forward and earn respect

Is wide in scope, directional in purpose, and
allows for a wide range of goals

It should be broad enough to allow for
creativity, expansion and new possibilities
The Mission Statement
Misunderstanding….





It doesn’t necessarily drive all the daily activities of
a unit.
Counselors are going to advise students into
classes at a community college regardless of the
college’s mission.
However, each unit needs to own the mission and
set some goals to help the college meet its
mission.
Therefore some of what a unit chooses to do is
driven by the mission.
People who say “everything we do is driven by the
mission” are either naïve or don’t really know what
is going on across the college.
What is a Goal



A statement of a quantifiable desired future state or
condition
It is driven by the mission
A goal:

lacks deadlines

is usually long-range

is relatively broad in scope

provides guidance for the establishment of
objectives (the specifics)
What is a Goal

It is reported on anecdotally

It is accomplished by activities

It takes several goals to accomplish a
mission statement

It takes several objectives to accomplish a
goal
Due to the nature of goals,
a unit may never
accomplish a goal…
…but the unit makes progress
toward the fulfillment of a goal.
We are used to setting goals






But goals are not outcomes
Outcomes are program-specific
Outcomes represent a new way of thinking
Outcomes have become widely accepted by
our various publics
They are here to stay
We used to measure ourselves by our
activities
Program Outcome Model
INPUTS
ACTIVITIES
OUTPUTS
Resources
Services
Products or Results of
Activities
Staff
Buildings
Facilities
State funds
FTE
Education (classes)
Services
Counseling
Student activities
Numbers served
FTE (input next year)
# Classes taught
# Students recruited
Constraints
Laws
State regulations
United Way model
Program Outcomes Model
INPUTS
> ACTIVITIES > OUTPUTS
> OUTCOMES
Benefits for People
(Outcomes answer the “so what”
question)
*New knowledge
*Increased skills
*Changes in values
*Modified behavior
*Improved condition
*Altered status
*New opportunities
Which Is It?

An input

An activity

An output

An outcome
1st Activity
GED Preparation







College provides well trained faculty.
200 students complete their GED.
Students move from public housing.
30 courses are offered each semester.
150 FTE are generated.
Students’ reading level improves.
Students are gainfully employed.
Student Services





Four counselors are hired.
Students successfully transfer courses
and enter the university system.
Students receive financial aid at the
university.
Students attend campus activities.
Students are able to pay back their
student loans.
Nursing Program




100% of students pass their state board
exam.
10 faculty members deliver 600 hours of
clinical experience for students.
Two students become actively involved in
the state nursing association.
Agreements are established between the
college and area hospitals for student
training opportunities.
General Education Courses





Students receive creative classroom
experiences.
Faculty members receive a grant to offer
multimedia opportunities to students.
Students become active participants in
County civic activities.
700 FTE are generated in core courses.
Students’ math skills improve.
Outcomes are ……...





Driven by the mission.
Related to overall program goals.
Specific to the teachings/activities of your
program.
Determined by faculty and front-line staff.
Measured carefully and specifically.
Inputs through Outcomes:
The Conceptual Chain
Long-range
Intermediate
Initial
Outputs
Activities
Inputs
OUTCOMES
But are there different
types of outcomes?
Different Types of Outcomes



Learning Outcomes
Program Outcomes
Administrative Outcomes
Definitions and Examples

Learning Outcomes:


What changes in knowledge, skills, attitude, awareness,
condition, position (etc.) occur as a result of the learning
that takes place in the classroom. These are direct benefits
to students.
Examples:
general learning skills (e.g. improved
writing and speaking abilities), ability to apply learning to
the work environment (e.g. demonstrate skills in co-op),
program-specific skills developed or enhanced (e.g. take
blood pressure.)
Definitions and Examples

Program Outcomes:


The benefits that results from the completion of
an entire program or series of courses. Are there
benefits for students who get the AAS in welding
versus those who take a few courses? If so what
are they?
Typical examples are: licensure pass rates,
employment rates, acceptance into 4-year
schools, lifelong learning issues, contributions to
society, the profession, etc.)
Definitions and Examples

Administrative Outcomes


These are outcomes established by programs and services that
have nothing to do with student learning or completion of
programs.
They have to do with benefits to the faculty/staff, the
department and to the college.
 The English Department wants all faculty to attend one
professional meeting annually so they can stay up-to-date in
their field. What are the benefits to the faculty, the college, the
students?
 Counseling may want students to lose few credits when they
transfer or to recruit a new counselor with expertise in working
with first-generation students (huge population for us). What
are the benefits to the counseling department, the college and
students?
 Facilities services may want all college units to feel that they
respond quickly to maintenance needs and security issues. It
they do this, what are the benefits to the college and to
students?
What is an Outcome
Objective?

A short-term, measurable, specific activity
having a time limit or timeline for completion
around a specific outcome

They measure outcomes and are used to show
progress toward goals

They specify who, will do what, under what
condition, by what standard and within what
time period
Go Over Definitions
Example – The College
Foundation

Mission Statement (portion): To enable county residents to
accomplish their educational goals.
Goal: Enhance accessibility to the college through building
private sector support for scholarships.
Objective: The College Foundation will increase private sector
contributions for scholarships in the amount of 5% by June 30,
2006.
Outcome Objective: 100 additional students will receive
need-based scholarships by August 31, 2006.
How to Set Objectives

There’s no magic number





e.g. 80% or 90%
What is reasonable?
What can you afford?
What realistically can your staff accomplish?
What percent shows you’re not committed and
what percent shows you’re naïve?
How to Set Objectives

Examples:

Fifty percent of students will be able to
communicate effectively in writing (complete the
writing exam with a grade of 60 [D] or better)

By the end of the spring term, 95% of faculty and
staff will have completed 20 contact hours of
professional development (workshops, college
courses, conferences, onsite trainings, etc.)
More Realistic

Seventy percent of students will be able to
communicate effectively in writing (complete the
writing exam with a grade of 75 [C+] or better)

By the end of the spring term, the professional
development office will increase their offerings for
faculty and staff by 10% over what was offered
last year (workshops, college courses,
conferences, onsite trainings, etc.)
How to Set Objectives




The first time you set objectives, be
conservative
Allow yourself a pilot semester or year to
determine the appropriate levels of change
that can be expected
Don’t pull a rabbit out of your hat (e.g. let’s
grow enrollment by 10%)
May need to benchmark (what does it mean)
Fall Curriculum Enrollment at ABC
College
17,188
17,500
16,982
17,000
16,660
16,000
15,500
(1.5%)
16,245
16,500
15,208
15,488
15,724
15,997
(1.2%)
(2.6%)
(1.6%)
(1.7%)
14,975
(1.5%)
15,000
(1.8%)
(1.6%)
14,500
Headcount
14,000
13,500
1997
1998
1999
2000
2001
2002
2003
2004
2005
Should they set a goal for a 5% enrollment growth for next fall?
Administrative Objectives




Many units do not directly serve students or
they want results within their units that are
not truly outcomes.
They want to improve services or approach
an old problem in a new way.
They want to become more efficient and
effective.
They will set administrative objectives.
My Administrative Objectives
1. 80% of faculty/staff responding to the faculty/staff
survey will perceive that Planning and Research
responds quickly to their requests for data.
2. 80% of faculty/staff responding to the survey will
perceive that Planning and Research makes a
significant contribution to the College.
3. 80% of faculty/staff responding to the survey will
perceive that Planning and Research contributes
to the effectiveness of CPCC.
4. 80% of faculty/staff responding to the survey will
indicate that Planning and Research produces
enough reports to meet the planning and
information needs of faculty and staff.
Why is This Hard?

Because it is education

Because the best results may not
happen for years

Because we are so busy doing what we
are doing…. we forget why we are doing
it
Let’s Look at Healthcare



When you have strep throat and go to the doctor for
your antibiotics (your intervention)
What are your intended outcomes?
Would the doctor ever tell you:




We believe curing disease is a developmental process
We believe there is value in the activity of taking pills and
receiving shots
We’re not sure if you’ll get better, it is how all doctors have
treated the disease since we learned about it
What would you say to that?
Health Outcomes Are Easy



Your health faculty have been doing this for
many years
Everything in healthcare is about outcomes
People go to doctors, receive treatment,
endure surgery and physical therapy – only
for the outcomes… no other reason
How to Measure
Program/Student
Outcomes
Identifying Outcomes

Faculty/staff in an area are often the worst in defining
outcomes because they are too close to the subject.
New Teachers
1st Year
2nd- 3rd
Middle Years of
Year Teachers Teaching
Older
Teachers
Unconsciously
Unskilled
Consciously
Unskilled
Consciously
Skilled
Unconsciously
Skilled
Don’t realize they
don’t know what
they are doing
Realize they
don’t know
what they are
doing – begin
to ask for help
Know what works Can’t remember
and why
why they do
what they do –
just know it
works
Sources of Ideas for Outcomes
*
*
*
*
*
*
*
*
*
*
program documents
program faculty and staff
national associations/credentialing boards
key volunteers
former students
parents of students
records of complaints
programs/agencies/employers that are the next
step for your students
other colleges with similar programs, services
and students as yours
outside observers of your program in action
How Often

Should we measure objectives or student
learning outcomes every year?

When does measurement become too time
consuming?

Units need time to put into effect the
changes made as a result of outcome
assessment before they are thrown back into
another cycle. They need time to reflect on
changes and results.
Disappointing Outcome Findings:
Why Didn’t We Meet Our Objectives?
Internal Factors:
*
*
*
*
*
Sudden staff turnover
New teaching philosophy/strategy
Curricular change (campus move)
Unrealistic outcome targets
Measurement problems (lack of followthrough, no effective tracking)
Disappointing Outcome
Findings
External Factors:
*
*
*
*
*
Community unemployment increases
State funding changes
Related programs (BS or MS programs) close
Public transportation increases fares or shuts
down some routes serving your campus or
time slot
Employment trends change
Use Your Findings
Internal Uses for Outcome Findings









Provide direction for curricular changes
Improve educational and support programs
Identify training needs for staff and students
Support annual and long-range planning
Guide budgets and justify resource allocations
Suggest outcome targets (expected change)
Focus board members’ attention on
programmatic issues
Help the college expand its most effective
services
Facilitates an atmosphere of change within the
institution
External Uses of Outcome Findings

Recruit talented faculty and staff

Promote college programs to potential students

Identify partners for collaboration (hospitals, businesses,
etc....)

Enhance the college’s public image

Retain and increase funding

Garner support for innovative efforts

Win designation as a model or demonstration site
So will someone help me do
this… help me select and
measure outcomes for my
program?
Ideas for Outcomes

Typical general education goals:









Students will demonstrate the ability to obtain meaning from printed,
electronic, and graphical resources
Students will effectively communicate both orally and in writing.
Students will demonstrate the ability to locate, critically evaluate, and
present information.
Students will apply mathematical concepts and skills to analyze,
manipulate, and interpret quantitative data.
Students will demonstrate the basic computer skills necessary to
function in a technological world.
Students will demonstrate the ability to identify, analyze, question, and
evaluate content as a guide to understanding and action.
Students will demonstrate knowledge of cultural differences.
Students will demonstrate an understanding of the influence of the
individual on group behavior and conversely, the influence of the group
on the individual.
Students will demonstrate comprehension of the major steps of the
scientific method.
Students will demonstrate knowledge of the humanities and critical skills
in assessing cultural/artistic merit and significance.
Let’s work through a few.





How does the welding program improve math skills
in students?
How does the culinary arts program teach students
to write effectively?
How does the Nursing program teach students to
think critically?
How do English faculty help students understand
the impact of the individual on society?
How does the GED program improve student
computer skills?
Top Ten Skills for the Future










Work ethic, including self-motivation and time management.
Physical skills, e.g., maintaining one's health and good
appearance.
Verbal (oral) communication, including one-on-one and in a group
Written communication, including editing and proofing one's work.
Working directly with people, relationship building, and team work.
Influencing people, including effective salesmanship and
leadership.
Gathering information through various media and keeping it
organized.
Using quantitative tools, e.g., statistics, graphs, or spreadsheets.
Asking and answering the right questions, evaluating information,
and applying knowledge.
Solving problems, including identifying problems, developing
possible solutions, and launching solutions.
The Futurist Update (Vol. 5, No. 2), an e-newsletter from the World Future Society, quotes
Bill Coplin on the “ten things employers want [young people] to learn in college”
Let’s work through a few?




How does the early childhood program improve the
work ethic of childcare workers or the children in
childcare centers?
How do IT programs improve teamwork skills?
How does Engineering improve students’ ability to
use spreadsheets and read tables/graphs?
How do you improve a student’s ability to:




Influence people
Have time management skills
Ask the right questions
Solve problems in the work environment
Learning Outcomes for the 21st
Century
Students in the 21st Century will need to be proficient in:
 Reading, writing, speaking and listening
 Applying concepts and reasoning
 Analyzing and using numerical data
 Citizenship, diversity/pluralism
 Local, community, global, environmental awareness
 Analysis, synthesis, evaluation, decision-making, creative thinking
 Collecting, analyzing and organizing information
 Teamwork, relationship management, conflict resolution and workplace
skills
 Learning to learn, understand and manage self, management of
change, personal responsibility, aesthetic responsiveness and wellness
 Computer literacy, internet skills, information retrieval and information
management
(The League for Innovation’s 21st Century Learning Outcomes Project.)
Let’s work through a few.

How does any given program or course improve:









Listening skills
Environmental awareness
Creative thinking
Relationship management
Conflict resolution
Self-management
Wellness
Information management
How can we measure it and use the results.
Outcomes and the Community
College

Outcome evaluation is at the heart of
institutional effectiveness.

Institutional effectiveness is at the heart of
SACS’ philosophy and more so with the
new criteria.

Data from outcome measurement should
be used in as many ways as possible planning, improvement, accreditation.
Challenges




Identifying and defining outcomes is the easy
part.
The devil is in the details.
How do we track it, where does it all go, how
do we score it, compile it, turn it into a
comprehensive report.
How do we “demonstrate improvement in
institutional quality.”
Things to Remember

Outcome measurement must be initiated from the
unit/department level (promotes ownership of
process).

Measure only what you are teaching or facilitating.

Put as much time in to “thinking through” the tracking
process as you do into the definition of outcomes.

Spend the time up front in planning and the process
will flow smoothly.

It will prove to be energy well spent.
Look at Early Childhood Examples
The Insanity Principle

Doing the same thing we have always
done but expecting different results.