An Integrated Model for Measuring
the Impact of Course, Program, and
Institutional Learning Outcomes
Carli Straight and Giovanni Sosa
Chaffey College
Originally Presented at Strengthening Student Success
Conference October 9th, 2009 (Wurtz, Sosa, & Fillpot, 2009)
Measuring the Impact of Learning Outcomes
How do we collectively examine disparate
learning outcomes assessment efforts?
(representativeness and replicability)
How do we provide evidence about these
disparate efforts? (verifiable)
How can we use evidence to improve upon
learning outcomes practices? Upon district
practices? (actionable)
Measuring the Impact of Learning Outcomes
Assessment often stands alone; relationships
are often not explored between:
 Courses From Different Programs
 Diverse Programs (within Instruction and
Between Instruction and Student Services)
 Course/Program Learning Outcomes and Their
Contribution to Institutional Learning Outcomes
 Types of Learning Outcomes (SLOs & AUOs)
 Differing Methods of Assessment
 Learning Outcomes and College Goals
Measuring the Impact of Learning
Outcomes: Statistical Analysis
How do we statistically assess impact?

Tests of statistical significance are not
necessary




Greatly influenced by sample size (mean =
10.0; 10.1)
Do not speak to the magnitude of the
difference
Not well understood – even by ‘experts’
Effect Size as measure of practical
significance


Unstandardized
Standardized (d, r)
 Cohen’s conventions: d = .20 – small; .50 –
moderate; .80 – large
Meta-Analysis

Combines research results across all
identified studies


Combines results using Effect Size (i.e.,
d) - generates an Average d
Average d may vary across study
characteristics (i.e., Moderators):


Number of Assessment Cycles
Type of Assessment Instrument
Moderators
(Grouping Variables)


Academic Year (AcadYr) – Academic
year (SU, FA, SP) data collection
occurred
Learning Outcome (LearnOutcome)




AUO – Administrative Unit Outcome
SLO – Student Learning Outcome
Program/Department
Course
Effect Size by Academic Year
Number of
Studies
1
9
15
44
48
Effect Size by Learning Outcome
Type (AUO or SLO)
Number of Studies
2
115
Effect Size by Program
Number
of
Studies
3
1
14
1
2
11
10
5
11
59
Effect Size by Course
Number of
Studies
1
21
12
12
14
3
1
4
2
4
2
Moderators (Continued)
(Grouping Variables - Measure)

Criterion – How was the Criteria for Success set by the program?






PrePost – The assessment was a post-assessment or pre-post
Standardized – Measure used to assess outcome was standardized or unstandardized
Instrument








Population – A measure obtained from the population (e.g.: critical thinking, Nursing Licensing
Exam, etc.)
Pre-assessment score – The pre-assessment score in a pre-post assessment
Program – The program set a specific criterion (e.g.: PE, Communication Studies, etc.)
Rubric
True/False
Multiple choice
Matching
Likert Scale
Anchored Scale
Unduplicated – used to identify number of students impacted by learning outcomes
processes. From 2004-005 to 2008-2009 6,714 students have been impacted by
learning outcomes.
Assessor


Self-Assessment (e.g.: student self-assessed)
External Evaluation (e.g.: student evaluated by teacher)
Effect Size by Criterion Type
Number of
Studies
1
31
85
Effect Size by Pre-Post or PostAssessment
Number of
Studies
85
32
Effect Size by whether the Instrument
was Standardized or Unstandardized
Number of
Studies
10
107
Effect Size by Measurement Type
Number of
Studies
7
22
2
8
75
3
Effect Size by whether Student Self-Assessed
Learning or Learning was Assessed by External
Evaluator (e.g.: teacher)
Number of
Studies
82
35
Moderators (Continued)
(Grouping Variables - Planning)

EndsPolicy1


EndsPolicy2




(1) Instructional and Student Services, (2) Comprehensive Education
Program, (3) Collaborative Partnerships, (4) Continuous Improvement,
(5) Learning Outcomes, (6) Core Competencies
(1) Foundation Skills Courses, (2) Occupational Programs, (3) Transfer
Level Programs, (4) Outreach, (5) Student Services, (6) Library and
Learning Support Services
EndsPolicy3

(1) College Facilities, (2) Technology, (3) Human Resources

(1) Balanced Budget, (2) Achievement of Planned Enrollment Growth,
(3) External Funding, (4) Bond Reserve
EndsPolicy4
EndsPolicy5

(1 Qualified Personnel and Professional Development, (2) Commitment
to Diversity, (3) Employment Agreement, (4) Professional Ethics
Effect Size by Ends Policy 1:
Learning Centered College
Number of
Studies
1
7
23
86
Effect Size by Ends Policy 2:
Institutional Effectiveness
Number of
Studies
1
92
22
1
Moderators (Continued)
(Grouping Variables – Learning Outcomes)


KSA – Knowledge, Skill, or Ability (e.g.: critical thinking, speech,
confidence, self-efficacy, etc.)
KSACateg = KSA Categorized by Core Competency and Instruction
1.
2.
3.
4.
5.


Communication – Students will demonstrate effective communication and
comprehension skills.
Critical Thinking – Students will demonstrate critical thinking skills in problem
solving across the disciplines and in daily life
Community/Global Awareness and Responsibility – Students will demonstrate
knowledge of significant social, cultural, environmental and aesthetic
perspectives
Personal, Academic, and Care Development – Students will assess their own
knowledge, skills and abilities; set personal, educational, and career goals; work
independently and in group settings; identify lifestyle choices that promote self
reliance, financial literacy and physical, mental and social health
Instruction – Students will demonstrate KSA specific to course subject (e.g.:
theory identification, research methods, etc.)
Cycles – Number of cycles program has assessed outcome
Level



Course
Program
Institution
Effect Size by Knowledge, Skill, or
Ability
Number of
Studies
65
9
12
20
11
Effect Size by Number of Times
Program has Assessed Outcome
Number of
Studies
74
34
8
1
Effect Size by Course and Program Level Learning
Outcomes
Number of
Studies
77
40
Limitations


Database only includes results from
outcomes assessment for programs
that assessed outcomes through the
Office of Institutional Research
How do we connect information to
student success (e.g.: goals, course
success, transfer, etc.)?
Resources


Microsoft Excel Spreadsheets that
calculate effect size statistics:
http://www.stathelp.com/spreadsheets.html
Effect Size Calculator with
Confidence Intervals:
http://www.cemcentre.org/renderp
age.asp?linkID=30325017