How a Culture of Inquiry
Can Help Catalyze
Completion Efforts
Dr. Rob Johnstone
Cabrillo College Visioning Day
Aptos, CA
February 7, 2014
www.inquiry2improvement.com
Acknowledgements
 Much of the content in this presentation in conjunction with
national projects such as Completion by Design, the Aspen Prize for
CC Excellence, and Bridging Research, Information & Cultures (BRIC)
 Content has also been developed by and with a host of national
partners, including:
•
•
•
•
•
•
Community College Research Center (CCRC)
Jobs for the Future
JBL Associates
Public Agenda
The Research & Planning (RP) Group
WestEd
 Infographics were primarily designed by Greg Stoup
An Applied Inquiry Framework for Student Completion – April 2013
National Center for Inquiry & Improvement
www.inquiry2improvement.com
April 2013
The Student Experience
An Applied Inquiry Framework for
Student Completion
www.inquiry2improvement.com
National Center for Inquiry & Improvement
www.inquiry2improvement.com
April 2013
A Culture of Inquiry &
Action
A RESOURCE for INSTITUTIONAL change
www.inquiry2improvement.com
5
What is a Culture of Inquiry?
Institutional capacity for supporting
open, honest and collaborative dialog
focused on strengthening the institution
and the outcomes of its students.
National Center for Inquiry & Improvement
www.inquiry2improvement.com
6
Culture of Inquiry: Features
● Widespread access to user-friendly
information on student outcomes
● Encouraging more people to ask a wider
collection of questions and use their
evidence and conclusions to enhance
decision making
● Reflective and dynamic discussions across
constituency groups
National Center for Inquiry & Improvement
www.inquiry2improvement.com
7
Culture of Inquiry: More Features
● Continuous feedback so adjustments can be
made along the way and processes can be
adapted
● A sense of ownership over improving student
outcomes – not blaming the student
● Using insight generated from inquiry to act at
varying levels of the institution to create the
conditions to improve outcomes
National Center for Inquiry & Improvement
www.inquiry2improvement.com
8
Applied Inquiry Framework for
Student Completion
Developed for Completion by Design
A Resource for Institutional Change
www.inquiry2improvement.com
9
An Applied Inquiry Framework for
Student Completion
Stage 1 – Explore how to improve outcomes
Stage 2 – Gather meaningful evidence
Stage 3 – Discuss evidence broadly
Stage 4 – Use evidence to inform change
Stage 5 – Measure the impact of change
National Center for Inquiry & Improvement
www.inquiry2improvement.com
10
STAGE 1
Explore how to improve student
outcomes
Focus inquiry on designing
approaches that improve
student outcomes
National Center for Inquiry & Improvement
www.inquiry2improvement.com
11
Key Questions
 When was the last time you sat in a
standing committee meeting on your campus
that used evidence to explore a key student
success outcome for more than 20 minutes?
 What types of questions do we spend most
of our organizational resources answering?
National Center for Inquiry & Improvement
www.inquiry2improvement.com
STAGE 2
Gather meaningful evidence
Collect high-quality,
meaningful evidence
at the student support,
classroom, program, and
institutional levels
National Center for Inquiry & Improvement
www.inquiry2improvement.com
13
When gathering
evidence,
make sure you are
focusing on
the right data…
National Center for Inquiry & Improvement
www.inquiry2improvement.com
14
20 year trend for California CC
course success & persistence rates
100%
Persistence Rate
90%
80%
70%
Success Rate
60%
What does that tell us
about the usefulness of
these metrics in setting
institutional strategies?
50%
40%
30%
20%
10%
0%
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
1989
National Center for Inquiry & Improvement
2006
2007
2008
2008
www.inquiry2improvement.com
15
Three Studies to Energize a Campus
Conversation About Student Success
• Grades & Success in Sequenced Courses
• Cohort Tracking in Developmental Education
• Levels of Preparedness in GE Courses –
success of simultaneously enrolled students
in Math / English courses
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Study 1:
Grades in Sequenced Courses
Note: Data tracked from 2006-07 to 2011-12
National Center for Inquiry & Improvement
www.inquiry2improvement.com
17
Grades in Sequenced Courses Backdrop
• Course sequences analyzed in Mathematics,
English, Biology, Chemistry, Physics & Economics
• Relationship between grade in 1st course in
sequence and success in 2nd course in sequence
was examined.
• Enrollment data tracked from 2006-07
to 2011-12
• Cohort Ns ranged from 1,890 to 126,367
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Segmentation of Students Taking
Intermediate Algebra
Taking
Math 308 /
310 in
Same
Semester
8%
A in Math
308
9%
Other
1%
Math 310
Attempt
was First
Math
Course
29%
B in
Math
308
13%
C in Math
308
12%
Math 310
Repeater /
Non
Success
25%
National Center for Inquiry & Improvement
Unclear
Status in
Math 308*
3%
www.inquiry2improvement.com
19
Table for Intermediate Algebra
Count
% of Total N
Success Rate in
Math 310
A in Math 308
5,659
10%
81%
B in Math 308
7,726
13%
60%
C in Math 308
7,234
12%
37%
Unclear Status in Math 308*
1,675
3%
16%
Math 310 Repeater / Non Success
14,568
25%
41%
Math 310 Attempt was First Math Course
16,812
29%
57%
Taking Math 308 / 310 in Same Semester
4,505
8%
68%
Other
487
1%
44%
Total
58,666
100%
53%
Status in Math 310
National Center for Inquiry & Improvement
www.inquiry2improvement.com
20
Developmental Education Math Course Sequence
Pre-Algebra
Introductory
Algebra
Intermediate
Algebra
Transfer
Level
Math 306
Math 308
Math 310
Math 1300s
The grade received by
students in Math 306
Success Rate of those same
students in Math 308
100%
90%
A
B
76%
80%
70%
Those placing
directly into
Math 308
60%
60%
51%
50%
40%
27%
30%
20%
C
10%
0%
Developmental Education Math Course Sequence
Pre-Algebra
Introductory
Algebra
Intermediate
Algebra
TransferLevel
Math 306
Math 308
Math 310
Math 1300s
The grade received by
students in Math 308
Success Rate of those same
students in Math 310
100%
81%
A
B
90%
80%
70%
Those placing
directly into
Math 310
60%
57%
60%
50%
37%
40%
30%
20%
C
10%
0%
Intermediate Algebra – Transfer-Level Mathematics
Intermediate
Algebra
TransferLevel
MATH 310
MATH 1300s
The grade received by
students in Math 310
Success Rate of those same
students in Math 1300s
100%
A
B
81%
90%
80%
Those placing
directly into Math
1300s
63%
66%
70%
60%
44%
50%
40%
30%
20%
C
10%
0%
Calculus Course Sequence
Calculus I
Calculus II
MATH 2413
MATH 2414
The grade received by
students in Math 2413
Success Rate of those same
students in Math 2414
100%
90%
A
B
78%
Those placing
directly into Math
2414
72%
90%
80%
70%
60%
50%
50%
40%
30%
20%
C
10%
0%
Developmental Education Writing Course Sequence
Two Levels
below
Transfer
One Level
below
Transfer
Transfer
Level English
ENGL 306
ENGL 307
ENGL 1301
The grade received by
students in English 306
Success Rate of those same
students in English 307
100%
90%
A
B
77%
Those placing
directly into
English 307
66%
66%
80%
70%
60%
49%
50%
40%
30%
20%
C
10%
0%
Developmental Education Writing Course Sequence
Two Levels
below
Transfer
One Level
below
Transfer
One Level
Transfer
below
Level English
Transfer
ENGL 306
ENGL 307
ENGL 1301
The grade received by
students in English 307
Success Rate of those same
students in English 1301
100%
90%
A
Those placing
directly into
English 1301
74%
71%
61%
80%
70%
60%
B
46%
50%
40%
30%
20%
C
10%
0%
College English Course Sequence
Composition
I
Composition
II
ENGL 1301
ENGL 1302
The grade received by
students in English 1301
Success Rate of those same
students in English 1302
100%
88%
A
B
77%
Those placing
directly into
English 1302
90%
80%
70%
68%
58%
60%
50%
40%
30%
20%
C
10%
0%
College Biology Course Sequence
Biology I
Biology II
BIO 1406
BIO 1407
The grade received by
students in Biology 1406
Success Rate of those same
students in Biology 1407
100%
90%
A
B
Those placing
directly into
Biology 1407
76%
70%
90%
80%
70%
60%
60%
50%
40%
30%
20%
C
10%
0%
College Biology Course Sequence
Anatomy
Physiology
BIO 2401
BIO 2402
The grade received by
students in Biology 2401
Success Rate of those same
students in Biology 2402
100%
88%
A
B
Those placing
directly into
Biology 2402
72%
90%
80%
70%
67%
60%
50%
50%
40%
30%
20%
C
10%
0%
College Chemistry Course Sequence
General
Chemistry I
General
Chemistry II
CHEM 1411
CHEM 1412
The grade received by
students in Chemistry 1411
Success Rate of those same
students in Chemistry 1412
100%
A
B
86%
90%
80%
Those placing
directly into
Chemistry 1412
70%
66%
70%
60%
46%
50%
40%
30%
20%
C
10%
0%
University Physics Course Sequence
General
Physics I
General
Physics II
PHYS 1401
PHYS 1402
The grade received by
students in Physics 1401
Success Rate of those same
students in Physics 1402
94%
87%
A
Those placing
directly into
Physics 1402
78%
100%
90%
80%
69%
70%
60%
B
50%
40%
30%
20%
C
10%
0%
Economics Course Sequence
Macro
Economics
Micro
Economics
ECON 2301
ECON 2302
The grade received by
students in Economics 2301
Success Rate of those same
students in Economics 2302
100%
89%
A
81%
Those placing
directly into
Economics 2302
74%
71%
90%
80%
70%
60%
B
50%
40%
30%
20%
C
10%
0%
Study 2:
Cabrillo College Developmental
Education Cohort Tracking
National Center for Inquiry & Improvement
www.inquiry2improvement.com
33
Developmental Education Cohort
Tracking Backdrop
• Dev ed cohorts tracked in Math and Writing.
• Students taking their first course within each
department in Fall 2010 were tracked forward
through Summer 2013.
• The charts give the % of each cohort passing
each subsequent stage of the sequence.
• Note that in some cases, students may not need
to pass all the courses in the sequence to meet
their goals.
• Cohort Ns ranged from 225 to 635
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Math cohort tracking starting in Pre-Algebra
PRE-ALGEBRA
INTRODUCTORY
ALGEBRA
INTERMEDIATE
ALGEBRA
TRANSFER LEVEL
100%
100%
Attempt
the
Course
72%
Pass
the
Course
90%
80%
70%
60%
50%
Of the
original
cohort
43%
Attempt
the
27%
Pass
Course
the
40%
30%
Of the
original
cohort
Course
National Center for Inquiry & Improvement
20%
18%
11%
Pass
Attempt
Of the
original
cohort
www.inquiry2improvement.com
Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.
10%
5%Pass
Att
4%
0%
Math cohort tracking starting in
Introductory Algebra
PRE-ALGEBRA
INTRODUCTORY
ALGEBRA
INTERMEDIATE
ALGEBRA
TRANSFER LEVEL
100%
100%
Attempt
the
Course
64% Pass
the
Course
90%
80%
70%
60%
50%
50%
Attempt
39% Pass
the
the
Course
Course
Of the
original
cohort
40%
30%
Of the
original
cohort
25%
19%
Pass
Attempt
20%
10%
0%
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.
Math cohort tracking starting in
Intermediate Algebra
PRE-ALGEBRA
INTRODUCTORY
ALGEBRA
INTERMEDIATE
ALGEBRA
TRANSFER LEVEL
100%
100%
Attempt
the
Course
67% Pass
the
Course
90%
80%
70%
60%
50%
Of the
original
cohort
50%
Attempt
37%
Pass
the
the
Course
Course
40%
30%
20%
10%
0%
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.
English Writing cohort tracking starting
at Developmental Writing I
DEV WRITING I
DEV WRITING II
COMPOSITION I
COMPOSITION II
100%
100%
Attempt
the
Course
71% Pass
the
Course
90%
80%
70%
60%
Of the
original
cohort
53%
Attempt
45%
Pass
the
the
Course
Course
National Center for Inquiry & Improvement
50%
40%
Of the
original
cohort
33%
Attempt
26%
Pass
the
Course
www.inquiry2improvement.com
Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.
30%
20%
10%
0%
English Writing cohort tracking starting
at Developmental Writing II
DEV WRITING I
DEV WRITING II
COMPOSITION I
COMPOSITION II
100%
100%
Attempt
the
78%
Pass
Course
the
Course
90%
80%
70%
65%
Attempt
54%
Pass
the
the
Course
Course
Of the
original
cohort
60%
50%
40%
30%
20%
10%
0%
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.
A Direct Look at
Meaningful Analyses
from Four-Year Schools
National Center for Inquiry & Improvement
www.inquiry2improvement.com
40
Simple and Effective Analyses to Inform
Student Progress and Completion Efforts
•Based on analyses from:
 Florida State University
 Georgia State University
 San Diego State University
 University of North Carolina – Greensboro
 University of Texas
 University of Southern California
 University of Wisconsin Eau Claire
 Virginia Commonwealth University
National Center for Inquiry & Improvement
www.inquiry2improvement.com
41
Simple and Effective Analyses to Inform
Student Progress and Completion Efforts
1. Targeting at-risk populations (UNCG)
2. Identifying At-Risk Students Using
Historical Data (GSU, FSU)
3. Identifying At-Risk Students Using
Milestones (FSU, GSU)
4. Tracking beyond Freshmen to
Sophomore Retention Rates (FSU)
5. Tracking Second Year Students with
Sophomore Standing (GSU)
National Center for Inquiry & Improvement
www.inquiry2improvement.com
42
Simple and Effective Analyses to Inform
Student Progress and Completion Efforts
6. Identifying Correlates of Success (VCU,
UWEC, SDSU)
7. Identifying Correlates of Non-Success
(USC)
8. Predictive Model for Retention (VCU,
GSU)
9. Impact of Course Withdrawals (VCU)
10. Additional Analyses (USC, FSU, GSU,
UWEC)
National Center for Inquiry & Improvement
www.inquiry2improvement.com
43
First Year to Second Year Retention,
Georgia State University
100%
90%
80%
80%
82%
81%
Fall '00
Fall '01
Fall '02
83%
80%
81%
82%
83%
83%
Fall '04
Fall '05
Fall '06
Fall '07
Fall '08
70%
60%
50%
40%
30%
20%
10%
0%
Fall '03
Retention
National Center for Inquiry & Improvement
3/30/10
www.inquiry2improvement.com
44
First Year Retention & Progression
Rates - Georgia State University
100%
90%
80%
80%
82%
83%
81%
80%
81%
82%
83%
83%
67%
70%
62%
60%
49%
50%
40%
30%
27%
28%
Fall '01
Fall '02
33%
35%
Fall '03
Fall '04
39%
22%
20%
10%
0%
Fall '00
Retention
National Center for Inquiry & Improvement
3/30/10
Fall '05
Fall '06
Fall '07
Fall '08
Retained & Sophomore
www.inquiry2improvement.com
45
STAGE 3
Discuss evidence
broadly
Engage a variety of campus
stakeholders in evidence-based
discussions about improvements
in practice
National Center for Inquiry & Improvement
www.inquiry2improvement.com
46
Data
do not
speak
for themselves.
National Center for Inquiry & Improvement
www.inquiry2improvement.com
47
The vital role of exploration
●
In order to make data useful, ample time and space
are needed to discuss and analyze the information
and connect it back to the original research question.
●
Answers are not always immediately apparent,
so skilled facilitation may be needed
to dig out the deeper meaning.
●
Multiple perspectives and
types of information are often
needed to make sense of
individual data points.
National Center for Inquiry & Improvement
www.inquiry2improvement.com
!
48
Study 3:
•English & Math Preparedness &
Success in GE Courses –
National Center for Inquiry & Improvement
www.inquiry2improvement.com
49
Success in Psychology 101 for students simultaneously
enrolled in an English course
English 826
(Two Below)
English 836
(One Below)
English 100
(Transfer A)
English 110
(Transfer B)
Success Rate of those same
students in Psychology 101
English course taking profile
for students in Psychology 101
100%
90%
Not in an English
course
68%
64%
Engl 826
Engl 836
Not in an
English
course
67%
Taking an
English
Course
33%
80%
70%
60%
48%
Engl 100
31%
Engl 110
75%
50%
40%
30%
20%
10%
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
0%
Success in Psychology 101 for students simultaneously
enrolled in an Math course
Beginning
Algebra
Fundamentals
Intermediate
Algebra
Math course taking profile for
students in Psychology 101
Transfer
Level Math
Success Rate of those same
students in Psychology 101
100%
90%
Not in an Math
course
64%
Fundamentals
Beginning
Algebra
Not in a Math
course
63%
Taking a Math
Course
37%
Intermediate
Algebra
Transfer Level
76%
80%
70%
63%
60%
51%
43%
50%
40%
30%
20%
10%
0%
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
Success in five highly enrolled GE courses by English
enrollment level
English 826
(Two Below)
Psychology 101
English 836
(One Below)
Speech 101
English 100
(Transfer A)
Economics 101
English 110
(Transfer B)
History 101
HSCI 101
100%
88%
82%
75%
82%
79%
73%
74%
73%
68%
64%
54%
57%
56%
51%
48%
48%
90%
80%
70%
60%
60%
50%
43%
40%
31%
27%
30%
20%
10%
0%
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
Success in five highly enrolled GE courses by English
enrollment level
English 826
(Two Below)
Fitness 334
English 836
(One Below)
Accounting 101
English 100
(Transfer A)
Music 202
English 110
(Transfer B)
Biology 250
Sociology 101
100%
90%
80%
76%
74%
73%
68%
66%
74%
70%
65%
64%
60%
58%
59%
50%
50%
48%
40%
37%
80%
40%
60%
50%
40%
31%
30%
20%
N/A
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
10%
0%
STAGE 4
Use evidence to
inform change
Implement changes in practice
and policy based on analyses
and discussion of college
evidence
National Center for Inquiry & Improvement
www.inquiry2improvement.com
54
Use evidence to
guide innovation
●
In this context, research and applied inquiry
are fundamentally interventionist in nature.
●
We are not seeking absolute truths; rather we
are looking for patterns of evidence that inform
action-oriented decisions.
●
Failure can be seen as an opportunity for
learning, especially when outcomes are shared
and used to inform further improvements in
practice.
National Center for Inquiry & Improvement
www.inquiry2improvement.com
55
And what do we do when the
evidence is ambiguous?
What to do when you reach the limits
of your research and yet still face
multiple
choices?
The
process
of inquiry is
Domain of possible
solutions
not a search for an absolute
truth
Trust your expertise
& choose !
Gregory M Stoup, Cañada College
We answer the questions that
eliminate dead end solutions
STAGE 5
Measure the impact
of change
Evaluate the impact of
practice changes on
student performance
National Center for Inquiry & Improvement
www.inquiry2improvement.com
57
Exploring the Preventing Loss,
Creating Momentum Framework
www.inquiry2improvement.com
Loss & Momentum
Framework
CONNECTION
ENTRY
PROGRESS
Interest to
Application
Enrollment to
Completion of
Gatekeeper
Courses
Entry into Course
of Study to 75%
Requirements
Completed
POLICIES
PRACTICES
PROGRAMS
PROCESSES
National Center for Inquiry & Improvement
www.inquiry2improvement.com
COMPLETION
Complete Course
of Study to
Credential with
Labor Market
Value
Completion by Design Framing Model
Some Known Loss Points
CONNECTION
Students never
apply to college
Students delay
entry into
college
College counseling
patterns that lead
to:
- under enrollment
- little programspecific guidance
- missed financial
aid opportunities
ENTRY
PROGRESS
COMPLETION
Unstructured
programs / too
many choices
Poor workschool balance
Transfer without
credential
Extended onramps
delay entry to
programs of study
Part-time
enrollment
forcing long
completion times
Students
accumulate
credits (& debt)
not aligned with
completion
Students fail to
enroll/pass
Gatekeeper
courses
Poor academic
preparation
National Center for Inquiry & Improvement
Progress not
monitored /
feedback given
Life events /
“Stop out or
drop out”
www.inquiry2improvement.com
Never complete
college level math
Credential
doesn’t support
needed wage &
aren’t stackable
Completion by Design Framing Model
Momentum Strategies
CONNECTION
Firstfoster
Timecollegegoing norms in
Student
High School
expand awareness
of college programs
and requirements
dual enrollment
& AP credit
ENTRY
PROGRESS
mandatory intrusive
advising focused on
programs of study
programs to
incentivize optimal
attendance
accelerate entry
to POS
student progress
to completion
monitored &
feedback provided
effective academic
catch-up programs
take placement test
in high school
aggressive
financial aid
support
educational
planning in high
school
shorter, faster,
cheaper course
design
National Center for Inquiry & Improvement
accelerated
competencybased programs
emergency aid
for students
www.inquiry2improvement.com
COMPLETION
mandatory
Successful
intrusive
advising
Completion
toward certificates
degrees & transfer
incentives to
transfer with
credentials
remove
barriers to
graduation
Learn & Earn and
Career Pathway
programs
Final Thoughts
An Applied Inquiry Framework for Student Completion – September 2013
www.inquiry2improvement.com
Culture of Inquiry: Why All the Fuss?
●
Because this ongoing work is challenging but
necessary!
●
Work needs to marry insight & evidence
●
Problems are large and recurring
●
No silver bullet / evident answers
●
Multiple solutions likely needed
●
Progress is not linear
●
Requires cross constituency interaction
National Center for Inquiry & Improvement
www.inquiry2improvement.com
63
Next Steps
• What are some “quick win” opportunities
for demonstrating the value of a culture of
inquiry to practitioners and stakeholders?
• What are some of the challenges to
developing or further evolving a culture of
inquiry?
Completion by Design? Completion by Accident?
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Find Out More
• The National Center for Inquiry & Improvement
website
www.inquiry2improvement.com
• Dr. Rob Johnstone, Founder & President
rob@inquiry2improvement.com
• CBD Inquiry Guides:
http://www.inquiry2improvement.com/publicationsresources
National Center for Inquiry & Improvement
www.inquiry2improvement.com
65