How a Culture of Inquiry
Can Help Your College
Move the Needle
on Student Success
Dr. Rob Johnstone
Texas Pathways to Progress Institute
Dallas, TX
September 22, 2014
www.inquiry2improvement.com
Acknowledgements
 Much of the content in this presentation in conjunction with
national projects such as Completion by Design, the Aspen
Prize for CC Excellence, and Bridging Research, Information &
Cultures (BRIC)
 Content has also been developed by and with a host of
national partners, including:
•
•
•
•
•
Community College Research Center (CCRC)
Jobs for the Future
JBL Associates
Public Agenda
The Research & Planning (RP) Group
 Infographics were primarily designed by Greg Stoup
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Formalistic Doublespeak…
• Give your 1st reaction to the following list of words:
•
•
•
•
•
•
•
•
•
•
•
•
Accountability
Accreditation
Assessment
Continuous Improvement
Data-Driven Decision Making
Evaluation
Institutional Effectiveness
Key Performance Indicators (KPIs)
Learning Outcomes Assessment
Performance-based Funding
Program Review
Strategic planning
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Recapturing the Movement…
• Vivid imagery from previous slide
• Faculty, Student Services folks & leaders have
been beaten over the head with these phrases
 they are correct to analyze that they too often
haven’t led to authentic improvement
 and the more formal the process, often…
•There is good news for our improvement
efforts – you can do all of the things captured
by those words in more authentic and less
formalistic ways
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Changing the Conversations…
• We have to make the conversation about things
faculty, staff & administrators care about –
students, their learning, and improving their
outcomes and lives
• Not everybody will come along – but we don’t
need everybody
• Org Change Thought: Red light / Yellow light /
Green light
• People need to see their expertise
acknowledged and integrated – and the effect
of their efforts on outcomes
National Center for Inquiry & Improvement
www.inquiry2improvement.com
April 2014
A Culture of Inquiry &
Action
A RESOURCE for INSTITUTIONAL change
www.inquiry2improvement.com
6
An Applied Inquiry Framework for
Student Completion (CBD)
Stage 1 – Explore how to improve outcomes
Stage 2 – Gather meaningful evidence
Stage 3 – Discuss evidence broadly
Stage 4 – Use evidence to inform change
Stage 5 – Measure the impact of change
National Center for Inquiry & Improvement
www.inquiry2improvement.com
7
What is a Culture of Inquiry?
Institutional capacity for supporting
open, honest and collaborative dialog
focused on strengthening the institution
and the outcomes of its students.
National Center for Inquiry & Improvement
www.inquiry2improvement.com
8
Culture of Inquiry: Features
● Widespread access to user-friendly
information on student outcomes
● Encouraging more people to ask a wider
collection of questions and use their
evidence and conclusions to enhance
decision making
● Reflective and dynamic discussions across
constituency groups
National Center for Inquiry & Improvement
www.inquiry2improvement.com
9
Culture of Inquiry: More Features
● Continuous feedback so adjustments can be
made along the way and processes can be
adapted
● A sense of ownership over improving student
outcomes – not blaming the student
● Using insight generated from inquiry to act at
varying levels of the institution to create the
conditions to improve outcomes
National Center for Inquiry & Improvement
www.inquiry2improvement.com
10
S TA G E 1
Explore how to improve
student outcomes
Focus inquiry on designing approaches that
improve student outcomes
www.inquiry2improvement.com
11
How We Spend Our Time Matters…
 When was the last time you sat in a
standing committee meeting on your campus
that used evidence to explore a key student
progress, completion, labor market, learning
or equity outcome for more than 20 minutes?
 What types of questions do we spend most
of our organizational resources answering?
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Malcolm Gladwell talks about
the right question…
• Link to full video:
http://www.ted.com/talks/malcolm_gladwell_o
n_spaghetti_sauce
National Center for Inquiry & Improvement
www.inquiry2improvement.com
13
The Right Question in the CC World…
Placement Tests & Cut Scores
• What was the problem we were trying to solve?
• I’d posit it was something like:
 “Can we find a short instrument that will help us assign
incoming students to various levels of math & English?” Or…
 “How do we ensure higher levels of course success in
transfer-level math & English courses?
• Are these the best questions? Why or why not?
Note: current system of placement tests may not even be
the best solution for this question: LBCC / CCRC
National Center for Inquiry & Improvement
www.inquiry2improvement.com
14
What if we tried to solve…
• What placement process is the most predictive
of transfer-level course success?
Or….
• What is the optimal curricular structure to
ensure that the greatest number of students
pass transfer-level math / English courses with
appropriate rigor?
Or….
National Center for Inquiry & Improvement
www.inquiry2improvement.com
15
More questions…
• What is the optimal math curriculum that
produces computational learning outcomes
that most students will need in the real world?
Or…
• Is writing about literature the optimal way to
teach students the writing skills they need in
their general education courses? What about
in the real world?
National Center for Inquiry & Improvement
www.inquiry2improvement.com
16
Organizing Question of
Improvement Science:
What problem
are you
trying to solve?
National Center for Inquiry & Improvement
www.inquiry2improvement.com
17
An Example from AB Tech in
Asheville, NC…
• We have to make sure our improvement
efforts to trying to solve the right problem
 “Right” is slippery – the problem at hand may be
important, but it might detract us from a larger
problem that is of much higher impact to improving
student outcomes
• AB Tech’s “One Stop” Onboarding process
 Original problem they were trying to solve: “How
do we address the chaotic and disconnected nature
of the onboarding of our new students?
National Center for Inquiry & Improvement
www.inquiry2improvement.com
18
One Stop Address the Chaotic
Onboarding Problem, but…
• As the AB Tech folks reflected on their efforts,
they realized they might unintentionally
communicate to students that it’s possible in
four hours or one day to:
 assess interests
 match interests to careers
 pick a program
 register for classes
 figure out financial stability issues
 be ready for opening day
 be prepared for any bump in the road that might occur
National Center for Inquiry & Improvement
www.inquiry2improvement.com
19
Perhaps…
• A better question / problem to solve might be:
“How do we ensure that students get the
services they need – when they need them – as
they move through their educational trajectory
at our college?”
• Note that this still covers having a more
streamlined onboarding experience – but
recognizes that it sits in a larger context to be
addressed
National Center for Inquiry & Improvement
www.inquiry2improvement.com
20
To sum up the starting line stage…
• More focus on asking the right question, and
ensuring we know what problem we are trying to
solve – actually less focus on the data per se
• Sometimes exploring the data can help you realize
you’ve been asking the wrong question - The Right
Pepsi vs. The Right Pepsi’s
• Traditional questions: enrollment, course success,
material covered
• Emerging questions: improving outcomes completion, progress, learning, labor market, equity
National Center for Inquiry & Improvement
www.inquiry2improvement.com
21
S TA G E 2
Gather meaningful
evidence
Collect high-quality, meaningful evidence at
the student support, classroom, program,
and institutional levels
www.inquiry2improvement.com
22
When gathering
evidence,
make sure you are
focusing on
the right data…
National Center for Inquiry & Improvement
www.inquiry2improvement.com
23
20 year trend for California CC
course success & persistence rates
100%
Persistence Rate
90%
80%
70%
Success Rate
60%
What does that tell us
about the usefulness of
these metrics in setting
institutional strategies?
50%
40%
30%
20%
10%
0%
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
1989
National Center for Inquiry & Improvement
2006
2007
2008
2008
www.inquiry2improvement.com
24
The Aspen Prize’s Take on
Data & Outcomes that Matter
•
•
•
•
Progress / Completion Outcomes
Labor Market Outcomes
Learning Outcomes
Equity in Achieving First Three Outcomes
• Examples of each in Appendix presentation…
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Gather meaningful evidence for the
“Four Big Ideas for Redesign”
1. Structured Pathways / Programs of Study
2. Better On-ramps into POS
 Accelerated / co-curricular Dev Ed, First Year Experience
3. Intentional Student Support Structures
 Intake system, onboarding, advising
4. Monitoring Progress / Customized Feedback
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Progress Example 1:
Grades in Sequenced Courses
National Center for Inquiry & Improvement
www.inquiry2improvement.com
27
Grades in Sequenced Courses Backdrop
• Course sequences analyzed in Mathematics,
English, Biology, Chemistry, Physics & Economics
• Relationship between grade in 1st course in
sequence and success in 2nd course in sequence
was examined.
• Enrollment data tracked over five-year period
• Selected highlight slides follow; full PPT
provided on request
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Data Too Often Presented Like This…
Count
% of Total N
Success Rate in
Math 310
A in Math 308
5,659
10%
81%
B in Math 308
7,726
13%
60%
C in Math 308
7,234
12%
37%
Unclear Status in Math 308*
1,675
3%
16%
Math 310 Repeater / Non Success
14,568
25%
41%
Math 310 Attempt was First Math Course
16,812
29%
57%
Taking Math 308 / 310 in Same Semester
4,505
8%
68%
Other
487
1%
44%
Total
58,666
100%
53%
Status in Math 310
National Center for Inquiry & Improvement
www.inquiry2improvement.com
29
And Not Often Enough Like This…
Pre-Algebra
Introductory
Algebra
Intermediate
Algebra
TransferLevel
Math 306
Math 308
Math 310
Math 1300s
The grade received by
students in Math 308
Success Rate of those same
students in Math 310
100%
81%
A
B
90%
80%
70%
Those placing
directly into
Math 310
60%
57%
60%
50%
37%
40%
30%
20%
C
10%
0%
Calculus Course Sequence
Calculus I
Calculus II
MATH 2413
MATH 2414
The grade received by
students in Math 2413
Success Rate of those same
students in Math 2414
100%
90%
A
B
78%
Those placing
directly into Math
2414
72%
90%
80%
70%
60%
50%
50%
40%
30%
20%
C
10%
0%
Developmental Education Writing Course Sequence
Two Levels
below
Transfer
One Level
below
Transfer
One Level
Transfer
below
Level English
Transfer
ENGL 306
ENGL 307
ENGL 1301
The grade received by
students in English 307
Success Rate of those same
students in English 1301
100%
90%
A
Those placing
directly into
English 1301
74%
71%
61%
80%
70%
60%
B
46%
50%
40%
30%
20%
C
10%
0%
College Biology Course Sequence
Anatomy
Physiology
BIO 2401
BIO 2402
The grade received by
students in Biology 2401
Success Rate of those same
students in Biology 2402
100%
88%
A
B
Those placing
directly into
Biology 2402
72%
90%
80%
70%
67%
60%
50%
50%
40%
30%
20%
C
10%
0%
College Chemistry Course Sequence
General
Chemistry I
General
Chemistry II
CHEM 1411
CHEM 1412
The grade received by
students in Chemistry 1411
Success Rate of those same
students in Chemistry 1412
100%
A
B
86%
90%
80%
Those placing
directly into
Chemistry 1412
70%
66%
70%
60%
46%
50%
40%
30%
20%
C
10%
0%
Economics Course Sequence
Macro
Economics
Micro
Economics
ECON 2301
ECON 2302
The grade received by
students in Economics 2301
Success Rate of those same
students in Economics 2302
100%
89%
A
81%
Those placing
directly into
Economics 2302
74%
71%
90%
80%
70%
60%
B
50%
40%
30%
20%
C
10%
0%
S TA G E 3
Discuss evidence broadly
Engage a variety of campus stakeholders in
evidence-based discussions about
improvements in practice
www.inquiry2improvement.com
36
Key Concept: Data do not
speak for themselves
Time & Space Needed to Explore Data,
Make Meaning & Generate Insight
National Center for Inquiry & Improvement
www.inquiry2improvement.com
37
Exploring Data Example:
English & Math Preparedness
& Success in GE Courses
National Center for Inquiry & Improvement
www.inquiry2improvement.com
38
Success in Psychology 101 for students simultaneously
enrolled in an English course
English 826
(Two Below)
English 836
(One Below)
English 100
(Transfer A)
English 110
(Transfer B)
Success Rate of those same
students in Psychology 101
English course taking profile
for students in Psychology 101
100%
90%
Not in an English
course
68%
64%
Engl 826
Engl 836
Not in an
English
course
67%
Taking an
English
Course
33%
80%
70%
60%
48%
Engl 100
31%
Engl 110
75%
50%
40%
30%
20%
10%
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
0%
Success in Psychology 101 for students simultaneously
enrolled in an Math course
Beginning
Algebra
Fundamentals
Intermediate
Algebra
Math course taking profile for
students in Psychology 101
Transfer
Level Math
Success Rate of those same
students in Psychology 101
100%
90%
Not in an Math
course
64%
Fundamentals
Beginning
Algebra
Not in a Math
course
63%
Taking a Math
Course
37%
Intermediate
Algebra
Transfer Level
76%
80%
70%
63%
60%
51%
43%
50%
40%
30%
20%
10%
0%
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
Success in five highly enrolled GE courses by English
enrollment level
English 826
(Two Below)
Psychology 101
English 836
(One Below)
Speech 101
English 100
(Transfer A)
Economics 101
English 110
(Transfer B)
History 101
HSCI 101
100%
88%
82%
75%
82%
79%
73%
74%
73%
68%
64%
54%
57%
56%
51%
48%
48%
90%
80%
70%
60%
60%
50%
43%
40%
31%
27%
30%
20%
10%
0%
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
Success in five highly enrolled GE courses by English
enrollment level
English 826
(Two Below)
Fitness 334
English 836
(One Below)
Accounting 101
English 100
(Transfer A)
Music 202
English 110
(Transfer B)
Biology 250
Sociology 101
100%
90%
80%
76%
74%
73%
68%
66%
74%
70%
65%
64%
60%
58%
59%
50%
50%
48%
40%
37%
80%
40%
60%
50%
40%
31%
30%
20%
N/A
Note: Enrollments from Summer 2000 to Spring 2009; Success is defined as A/B/C/CR grade
10%
0%
S TA G E 4
Use evidence to inform
change
Implement changes in practice and policy based
on analyses and discussion of college evidence
www.inquiry2improvement.com
43
Use evidence to
guide innovation
●
In this context, research and applied inquiry
are fundamentally interventionist in nature.
●
We are not seeking absolute truths; rather we
are looking for patterns of evidence that inform
action-oriented decisions.
●
Failure can be seen as an opportunity for
learning, especially when outcomes are shared
and used to inform further improvements in
practice.
National Center for Inquiry & Improvement
www.inquiry2improvement.com
44
And what do we do when the
evidence is ambiguous?
What to do when you reach the limits
of your research and yet still face
multiple
choices?
The
process
of inquiry is
Domain of possible
solutions
not a search for an absolute
truth
Trust your expertise
& choose !
Gregory M Stoup, Cañada College
We answer the questions that
eliminate dead end solutions
S TA G E 5
Measure the impact of
change
Evaluate the impact of practice changes on
student outcomes
www.inquiry2improvement.com
46
Final Thoughts
www.inquiry2improvement.com
Final Reflections
• Creating or evolving your culture of inquiry
isn’t magic; there are clear steps and
resulting artefacts of such a culture
• Don’t focus too much on the data – the
questions you ask are exponentially more
important
• Remind yourself and your team to
constantly ask: “What problem are we trying
to solve?”
National Center for Inquiry & Improvement
www.inquiry2improvement.com
48
What we are shooting for…
• Great statement of the desired end state
from a CC President at the Aspen / ATD
Leadership Symposium:
“A wider range of people on a campus ask
a better set of questions about outcomes
& act on their reflections to improve
them.”
National Center for Inquiry & Improvement
www.inquiry2improvement.com
49
Find Out More
• The National Center for Inquiry & Improvement
website
www.inquiry2improvement.com
• Dr. Rob Johnstone, Founder & President
rob@inquiry2improvement.com
• CBD Inquiry Guides on Applied Inquiry & Nuances
of Completion:
http://www.inquiry2improvement.com/publicationsresources
National Center for Inquiry & Improvement
www.inquiry2improvement.com
50