Educational
Effectiveness
Fall Faculty Retreat 2006
Leanne Neilson
Halyna Kornuta
Key Learning Points
Common understandings
 “Educational Effectiveness”
 Evidence
 Use of Evidence
Educational Effectiveness:
What is it?
 What we want our students to know
• Competencies, knowledge, skills, abilities,
applications, and dispositions
 How we know they know it
• Evidence
Why Bother?
Our need for Evidence
 Student Recruitment
 Accreditation
 Fundraising / Grants
 Institutional Improvement, Selfstudy and Reflection
Indirect Evidence
Perceptions and Input
Methods
 Student Satisfaction
Surveys
 Student Exit
Surveys
 Alumni Surveys
 Employer Surveys
 Focus groups
 Exit Interviews with
Graduates
Data Mining
 Faculty/Student ratios
 Percentage of students who
study abroad
 Enrollment trends
 Percentage of students who
graduate within six years
 Diversity of students, faculty,
staff
 Retention and Transfer
Studies
 Percentage of students who
go on to graduate school
 Job Placement Statistics
Direct Evidence
Products of Student Learning
 Student work
samples
 Portfolios
 Capstone projects
 Assessment of
student
performance
 Case Analysis
 Pre-and post-tests
 Blind scored
assignments
 Locally developed tests
 Standardized tests
 National licensure
examinations
What Constitutes Good
Evidence
1. Purposeful
2. Representative
3. Varied
4. Actionable
CLU Evidence
 Indirect
•
•
•
NSSE / BCSS
Noel-Levitz
Library
 Direct
•
•
WIC
SOE
Think About. . .
CLU Indirect Evidence
NSSE
National
Survey of
Student
Engagement
BCSS
Beginning
College
Student
Survey
NSSE 2005 / 2001
Table 6
Diversity: CLU First-years/Seniors compared with Selected Peers and NSSE
Nationwide First-year/Seniors
11: Educational and Personal Growth : To what extent has your experience at this
institution contributed to your knowledge, skills, and personal development in the following
areas? (Scale: 1=Very little 2=Some 3=Quite a bit 4=Very much)
Means Compari son
N= 174 first-year (FY ) students (2005)
N= 159 senior year (S R) students (2005)
CLU
2005
l. Understanding people of other racial
and et hnic backgrounds
NSSE Nationwide
2001
2005
2001
NSSE 2005 / 2001
Table 6
Diversity: CLU First-years/Seniors compared with Selected Peers and NSSE
Nationwide First-year/Seniors
11: Educational and Personal Growth : To what extent has your experience at this
institution contributed to your knowledge, skills, and personal development in the following
areas? (Scale: 1=Very little 2=Some 3=Quite a bit 4=Very much)
Means Compari son
N= 174 first-year (FY ) students (2005)
N= 159 senior year (S R) students (2005)
l. Understanding people of other racial
and et hnic backgrounds
CLU
NSSE Nationwide
2005
2001
2005
2001
FY
2.70
2.45
2.59
2.58
SR
2.66
2.49
2.60
2.64
NSSE 2005 / 2001
Table 6
Diversity: CLU First-years/Seniors compared with Selected Peers and NSSE
Nationwide First-year/Seniors
1: Academic, Intellectual, and Social Experiences: In your experience at your institution
during the current school year, about how often have you done each of the following?
(Scale: 1=Never 2=Sometimes 3=Often 4=Very often)
Means Comparison
CLU
NSSE Nationwide
2005
2001
2005
2001
N= 174 first-year (FY) students (2005)
N= 159 senior year (SR) students (2005)
u. Had serious conversations with
students of a different race or ethnicity
than your own
FY
2.73
2.67
2.60
2.60
SR
2.75
2.58
2.65
2.59
NSSE 2005 / 2001
Table 6
Diversity: CLU First-years/Seniors compared with Selected Peers and NSSE
Nationwide First-year/Seniors
1: Academic, Intellectual, and Social Experiences: In your experience at your institution
during the current school year, about how often have you done each of the following?
(Scale: 1=Never 2=Sometimes 3=Often 4=Very often)
Means Comparison
CLU
NSSE Nationwide
2005
2001
2005
2001
N= 174 first-year (FY) students (2005)
N= 159 senior year (SR) students (2005)
u. Had serious conversations with
students of a different race or ethnicity
than your own
* Note: % for respondents who
selected 1 (never) and 2 (sometimes)
FY
2.73
2.67
2.60
2.60
FY
*
SR
1 – 15%
2 – 27%
2.75
1 – 16%
2 – 29%
2.58
1 – 15%
2 – 34%
2.65
1 – 15%
2 – 34%
2.59
SR
*
1 – 10%
2 – 37%
1 – 13%
2 – 39%
1 – 12%
2 – 35%
1 – 13%
2 – 37%
BCSS
17j. Participate in a
school-sponsored
NSSE
1k. Participated in a
community-based
project (e.g. service
community service
project
learning) as part of a
regular course
Engagement
BCSS / NSSE
High
Low
Low
High
Expectation
Engagement
BCSS / NSSE
High
1%
8%
Low
50%
41%
Low
High
Service to the
Community
Expectation
Noel-Levitz 2006
CLU
Noel Levitz Adult Student Priorities Survey
N=310
National Adult
Students
Item (Scale: 1= not at all satisfied to 7= very satisfied)
Satis / SD
Satis / SD
Mean
Difference
3. Faculty care about me as an individual.
5.96 / 1.09
5.65 / 1.32
0.31
19. My academic advisor is knowledgeable about
requirements in my major.
6.06 / 1.29
5.66 / 1.49
0.40
26. Faculty provide timely feedback about student
progress in a course.
5.77 / 1.21
5.41 / 1.44
0.36
32. My classes provide opportunities to improve my
technology skills.
5.61 / 1.25
5.29 / 1.46
0.32
40. Faculty are usually available for adult students
outside the classroom by phone, by e-mail or inperson.
6.11 / 1.03
5.90 / 1.25
0.21
50. My advisor helps me apply my academic major to
specific career goals.
5.44 / 1.49
4.97 / 1.74
0.47
CLU Indirect Evidence
Library Expenditures and Usage
Total
FTE
Total Library
Expenditures
Per FTE
Circulation
Transaction
(Including
Reserves)
Per FTE
Total
Interlibrary
Loans
Received
2,500
$276
8
1,680
Comparison Group Median
State Median (CA)
National Median
CLU
Whitworth College, WA
Pacific University, OR
Univ of Portland, OR
Pacific Lutheran Univ, WA
FTE: Full Time Equivalent Student 12 month Enrollment
Source: Education Dept. Biannual Survey of Academic Libraries 2003 - 04
CLU Indirect Evidence
Library Expenditures and Usage
Total
FTE
Total Library
Expenditures
Per FTE
Circulation
Transaction
(Including
Reserves)
Per FTE
Total
Interlibrary
Loans
Received
Comparison Group Median
3,362
$446
12
4,223
State Median (CA)
1,464
$179
7
103
National Median
1,431
$277
9
365
CLU
2,500
$276
8
1,680
Whitworth College, WA
2,045
$387
12
2,435
Pacific University, OR
2,222
$660
21
10,054
Univ of Portland, OR
Pacific Lutheran Univ, WA
2,927
3,167
$560
$654
21
15
5,929
6,758
FTE: Full Time Equivalent Student 12 month Enrollment
Source: Education Dept. Biannual Survey of Academic Libraries 2003 - 04
CLU Direct Evidence
Student Learning Outcomes
Written Communication
Information Literacy
Critical Thinking
WIC
WIC 2005
Table 2
Spring 2005 Summary of Criteria and Mean Scores
Criteria
Written Communication
(5 point analytic rubric)
1. Clear thesis/purpose/topic
2. Paper organized/unified
3. Free of errors—grammar, punctuation, spelling
4. Purpose/thesis addressed throughout
5. Supporting argument, evidence, examples
6. Language and Vocabulary
7. Conclusion (synthesis & addresses
thesis/purpose)
8. Maintains level of excellence throughout
Written Communication Mean
Critical Thinking Score and Mean
(4 point holistic rubric)
Course Type Mean Scores
Freshmen
Writing
Capstone
English
Intensive
N =88
N = 152
N = 99
Summary
N = 339
3.0
2.9
3.3
3.0
2.9
3.2
3.3
3.3
3.4
3.4
3.3
3.2
2.9
3.4
3.3
3.2
3.1
3.2
3.3
3.2
3.5
3.5
3.4
3.5
2.7
2.9
3.1
2.9
2.9
3.0
3.2
3.3
3.1
3.3
3.1
3.2
2.5
2.8
2.7
WIC 2006
Table 3
Spring 2006 Summary of Criteria and Mean Scores
Criteria
Course Type Mean Scores
Summary
Freshmen
English
N =56
Writing
Intensive
N = 88
Capstone
N = 37
N = 181
3.1
2.1
2.7
2.7
2.6
3.0
2.6
2.7
2.8
2.9
2.7
2.6
2.5
3.1
2.5
2.7
3.6
3.2
3.3
3.1
3.2
2.9
3.4
3.3
3.0
2.7
2.8
2.7
2.7
3.0
2.7
2.8
Written Communication, Critical Thinking,
and Information Literacy
(4 point analytic rubric)
1. Articulates focus
2. Finds sources
3. Analyzes and evaluates sources
4. Synthesizes information to support arguments
5. Presents arguments
6. Uses conventions of standard U.S. English
7. Creates conclusion
Mean
CLU Direct Evidence
SOE
Course Work
(Portfolio)
Field
Supervisor
Evaluation
Comprehensive
Exam / Thesis /
Webfolio Defense
Entrance / Exit
Interviews
Alumni Survey
Employer /
Administrator
Survey
Additional Evidence
Assignment / Evidence
C

e.g. Complete an analysis of a
journal article requiring students
to identify, interpret, assess,
and evaluate relevant
information. This assignment
occurs again later in the
semester. A rubric is used to
score both assignments
providing evidence of student
achievement.
E
D
SLO
Critical
Thinking
Key Learning Points
Common understandings
 “Educational Effectiveness”
 Evidence
 Use of Evidence
Educational
Effectiveness
Fall Faculty Retreat 2006
Written and Produced by
Leanne Neilson
Halyna Kornuta