Program Outcomes Assessment
Techniques: Overview and Examples
“I have never let my schooling interfere with my education.”
- Mark Twain
Steve Steele, Consultant
Faculty Online Technology Training Consortium (FOTTC)
Applied Sociologist
Anne Arundel Community College
sfsteele@mail.aacc.cc.md.us
410-541-2369
Common Conceptualizations
of Evaluation/assessment
Is it viewed a “a reign of terror?”
Is it “stuck on like a barnacle”; an
afterthought
Is it an ongoing process in itself?
An organized entity in itself
Part of a management information
system
Where are We Going Today?
Evaluation? Assessment? Definitions
Reasons For Doing This
Placement of Evaluation/assessment
The Dynamic Fit Between Planning,
Evaluation/assessment and Doing In the
Improvement Process
Examples
Summary: Strengths and Weaknesses
Evaluation? Assessment?
Definitions
The systematic assessment of the operation and/or
outcomes of a program or policy, compared to explicit
or implicit standards, in order to contribute to the
improvement of the program or policy. (Weiss, 1998)
“The key sense of the term... refers to the process of
determining the merit, worth, or value of something...”
(Scriven, 1991, pp. 139-141)
“evaluation/assessment research... refers to research
purpose rather than research method. Babbie,1989,
326)
Critical Components of
Definitions
A systematic process
A judgment will likely be made
Standards for judgment, for success
Intervention
Orientation toward Improvement?
Reasons For Doing This
Internal
External
Clients/Stakeholders
Clients/Stakeholders
The President
The Dean
Your Chairman
Your students
You!
Societal
Middle States
Professional
Granting agencies
Reasons For Doing This
Have you considered the client’s
“corporate culture” when constructing
this project/design?
Are you sure that you have focused on
the client’s needs for ALL relevant
clients and stakeholders?
Placement of
Evaluation/assessment
Think of evaluation/assessment before
the program, not after.
Integrate evaluation/assessment into
the grant proposal, plan or program.
Use evaluative data “for all it’s worth!”
Consider evaluation/assessment as
constructive criticism, rather than “doing
time!”
The Dynamic Fit Between Planning,
Evaluation/assessment and Doing In
the Improvement Process
Is it part of a larger process?
Act
HERE!!
Check
Plan
Do
Based on E. Deming’s work
“Plan” and Define it.
Act
Check
Plan
Do
What is it? Intervention Definitions
Intervention
Plan
Definition
"...the process of determining what (something) intends to
1
be in the future and how it will get there..."
Strategic Plan
Project
"...the process by which guiding members of an
organization envision its future and develop the necessary
2
procedures and operations to achieve that future..."
"(An) effort that marshals staff and projects toward some...
3
defined... goals"
4
"...time-bounded efforts within a program"
Process
"...what happens between input and output..."
Course, Training or
Workshop
"...an organized effort in which people or persons learn
something..."
Something Other
Phenomenon (SOP)
"I Don't Know"
Whatever it is, it doesn't fit one or more of the above!
Program
This intervention is ill-defined or undefined.
5
Example: A Distance Learning
Business Plan
http://www.aacc.cc.md.us/ola/
http://www.aacc.cc.md.us/ola/Assessment/BusPlanNov98.html
Goals and Objectives...
http://www.aacc.cc.md.us/ola/Assessment/BusPlanNov98.html#To implement
Act
Check
Plan
Do
“Check”
Some Approaches to “Checking”
Keeping track... is it operating (not
necessarily successfully!)
Forming something, improving something
Sum up, impact… did it work?
Based on Michael Scriven’s Formative & Summative Evaluation
Some Tools for Tracking,
Forming and Summing
Frequency of something
Rates or percentages of something
Level of Need
Opinions, behaviors, feelings
Indicators, Estimates and Forecasts
Survey
Use patterns
Focus groups
A Plan Level Example #1
Wells, Kepner, Barnes, Steele
AACC
Toward an Evaluation/assessment Plan
http://www.aacc.cc.md.us/ola/Assessment/Baldrige.html
Scroll
http://www.aacc.cc.md.us/ola/Assessment/assessmentStudentStakeholder.htm
Measures….
http://www.aacc.cc.md.us/ola/Assessment/studentsurveyresults.htm
A Plan Level Example #2
Barnes, Warner, Steele
AACC
Some Tools for Tracking,
Forming and Summing
Frequency of something
Rates or percentages of something
Level of Need
Opinions, behaviors, feelings
Indicators, Estimates and Forecasts
Survey
Use patterns
Focus groups
On Line Enrollment
1100
1000
900
800
700
600
500
Fall 2000
Predicted
Actual (unofficial)
1019
1031
A Course Level Example #3
Steele
AACC
Some Formative Checks on Soc111
Online
Act
Check
Plan
Do
How Many Links on the Module Did You
Visit (all modules, N=188 response)?
"Site_visit"
"All of the sites"
"Very few of the sit
16.5%
5.3%
"Most of the sites"
17.6%
"Some of the sites"
38.3%
"None of the sites"
22.3%
Rate this Module's Learning
Value (all modules, N=188 response)?
"Learning_value"
"3. Excellent"
41.0%
"0 Poor"
2.1%
"1. Fair"
6.4%
"2. Good"
50.5%
Rate this Module's Interest
Value (all modules, N=188 responses)?
"Interest"
"3. Excellent"
48.9%
"0 Poor"
1.6%
"1. Fair"
5.3%
"2. Good"
44.1%
Summary: Strengths and
Weaknesses
Strengths
Opportunity for continuos growth
Possibility for continuous improvement
Makes planning useful
Improves program and course design
Weaknesses
Resource and infrastructure commitment
Requires a variety of skills
Difficult to stay “tangent”
Demands attention