How to Analyze School Data to Drive
Academic Improvement – A Case Study
with Significant Results in One Year
13th National Quality Education Conference
Session 13
November 13, 2005
Arved Harding
Koalaty Kid Coordinator for ASQ Section 1106,
Associate Statistician for Eastman Chemical Company
Chris Hampton
Vice-Principal of Dobyns-Bennett High School
1
Objective
The objective of this presentation is to provide
you with a process that is backed by results that
can be used to gather and analyze relevant
data in a unique manner to help drive schoolwide improvements.
2
Presentation Guide



We will give you an overview of the process for
gathering and studying important data and a preview of
the outcome.
We will walk you through the details of the process for
gathering and studying important data and show you
results as we go.
We will also show you a detailed process for
Implementing Successful Academic Improvement
Projects
3
Basic Philosophy in Traveling on
any Journey
Determine the following:
Where are you?
Where do you want to go?
How are you going to get there?
4
Basic Philosophy in Traveling on
any Journey
Determine the following:
Where are you?
Requires Measures
(Example: Dobyns-Bennett High School-targeted school due to achievement
of sub-groups on English measures.)
Where do you want to go?
Benchmarking with other schools or state and national standards
(Return to the status of “School in Good-Standing”)
How are you going to get there?
Need a strategy or process (See Hank Somer's 11 Step Process)
(DB-School-wide focus on assessment, i.e., this is a DB problem, not an
English problem. Provide data, accept feedback, provide resources, support,
and model the importance to students and stakeholders)
5
Process (Page 1 of 3)



Decide what data you would like to study
(TCAP, Gateway, ACT, Graduation Rate, …)
Decide how many years of data you want to
include (more is usually better)
Decide the schools with which you want to
benchmark (include some close by but also
some of the best in the state and nation if
possible). Collect the data from your state
website or other sources and populate an
Excel spreadsheet. This may take some time.
6
Consider dividing this task up.
Process (Page 2 of 3)



Plot the data (See our recommended
template. Also consider adding any state
mandated goals.)
Interpret the data (think about what each
plot is telling you and if it makes sense) Take
notes.
Note some of the obvious areas of
opportunity, schools to benchmark against,
and those areas your school should feel good
about.
7
Process (Page 3 of 3)




Put a presentation together for your school or
school system.
Present it first to the administrative leadership team
and prioritize the opportunities for your school or
school system.
Filter the presentation on to other teams as
appropriate. Be sure to field questions and help
people draw conclusions. Many people are not used
to looking at this kind of data this way.
Put together teams around each major opportunity
and implement Hank Somers' 11 Step Process for
Successful Academic Improvement Projects. Be
sure to include this as part of your School
Improvement Plan or other appropriate processes.
8
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
Attendance
Attendance (1994-2004)
Dobyns Bennett
Science Hill
Sullivan South Tennessee High Greeneville City
Oak Ridge
Desired Direction
97
96
95
94
93
92
Maryville
State Average
Year within School
• Compared to State Average 2003-2004
• Compared to Best in Class 2003-2004
• Possible Benchmarking Opportunities
1% above State Average
3% below best in class (Greeneville City )
Greeneville City because of their great
improvements and Maryville because of
their consistent high results.
• Comments: It is required by the state to achieve above 93% attendance.
9
95
90
85
80
75
State Average
Maryville
Oak Ridge
Greeneville City
Tennessee High
Sullivan South
Science Hill
Dobyns Bennett
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
Gateway Alg. I % Prof. and Above
Gateway Algebra I (2001-2004)
Desired Direction
%Proficient and Above
Year within School
• Compared to State Average 2003-2004
20% above State Average
• Compared to Best in Class 2003-2004
2% above 2nd best in class (Maryville)
• Possible Benchmarking Opportunities
None
• Comments: Continued improvement for 2 years, now best in class!!!
10
DB Education Excellence Steering
Team







Principal
4 Vice-principals
4 teachers
2 Students
2 Community members (Businesses)
1 Parent
1 Quality in Education Consultant
11
We will walk you through the process
and show you results as we go.

Chris Hampton – Vice Principal of DobynsBennett High School
12
Decide what data you would like to study
(TCAP, Gateway, ACT, Graduation Rate, …)








ACT (1995-2004)
 ACT English
 ACT Reading
 ACT Math
 ACT Science
SAT (1994-2004)
 SAT Verbal
 SAT Math
Gateway Algebra I (2001-2004)
Gateway Biology I (2001-2004)
Gateway English II (2004)
Writing 11th (1997-2004)
Math Foundations (2002-2004)
English 9 (EOC) (2004)
AP
Data (1996-2004)
Attendance (1994-2004)
Cohort Rate % (1995-2004)
# Students *
System $/Student *
Ethnic minority % *
Free/Reduced Lunch % *
Suspensions % *
Expulsions % *
Note that this information is
updated annually and the process
has become part of our school
culture.
* Compiled from Tennessee School
Report Cards 2002&2003
2002 denotes the 2002-2003 school year.
13
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
1994/95
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
2003/04
Attendance
Decide how many years of data you want to
include (more is usually better)
Collect as many as you can and this will vary depending on the measure.
Attendance (1994-2004)
97
96
95
94
93
92
Dobyns Bennett
Science Hill
Sullivan South Tennessee High Greeneville City
Oak Ridge
Maryville
State Average
Year within School
14
Decide which schools you want to benchmark with
(include some close by but also some of the best
in the state and nation if possible)
Dobyns-Bennett chose:

Science Hill – proximity and similar size and demographics/city school

Sullivan South – proximity and reputation

Tennessee High – proximity and city school

Oak Ridge – reputation as a strong academic school

Greeneville City – proximity and city school

Maryville – reputation as a strong academic school

State Average

National Average

We will adjust these for the 2005-06 school year based on info from TVAAS.
On TVAAS website administrators can search for schools most like theirs based on four
demographics: Economically Disadvantaged, Enrollment, Special Education, and
Minority Population. www.tvaas.sas.com
15
16
Collect the data from your state website or other
sources and populate an Excel spreadsheet. This may
take some time. Consider dividing this task up.
School - Year
Dobyns Bennett - 2003/04
State Average - 2003/04
National Average - 2003/04
Science Hill - 2003/04
Sullivan South - 2003/04
Tennessee High - 2003/04
Oak Ridge - 2003/04
Greeneville City - 2003/04
Maryville - 2003/04
Dobyns Bennett - 2002/03
Science Hill - 2002/03
Sullivan South - 2002/03
Tennessee High - 2002/03
Oak Ridge - 2002/03
Greeneville City - 2002/03
Maryville - 2002/03
State Average - 2002/03
National Average - 2002/03
Dobyns Bennett - 2001/02
Science Hill - 2001/02
Sullivan South - 2001/02
Tennessee High - 2001/02
Oak Ridge - 2001/02
Greeneville City - 2001/02
Maryville - 2001/02
State Average - 2001/02
National Average - 2001/02
Attendance
93.1
92.2
93.8
94.9
91.8
93.9
96.4
95.5
94
93.2
94.9
93.2
94.6
95.8
96.4
92.5
93.2
93.6
94.8
93.2
94.7
95.5
95.9
92
Graduation
Rate
78.8
75.7
89.8
79
82.2
82.2
73.3
94.2
ACT
21.6
20.3
20.9
21.6
21.6
21.2
23.9
22.3
23.6
21.2
21.9
22
21.3
23.6
21.6
23.8
20.1
20.8
21
20.9
21.5
20.9
23.6
21.6
23.2
20
20.8
Writing
11th
4.2
3.8
Gateway
Alg_ I %
Prof_ and
Above
97
81
Gateway
Bio_ I % Gateway Eng_ II
Prof_ and
% Prof_ and
Economically
Above
Above
Disadv %
99
97
27.5
96
90
44.6
4
3.9
4.2
4.5
3.9
4.6
4.2
3.8
3.9
4.3
4.4
4
4.6
3.8
90
85
95
94
89
95
94
91
81
93
86
89
95
75
99
95
99
99
98
99
99
100
99
99
99
93
99
95
4.3
3.9
4
4
4.6
3.8
4.7
93
94
80
79
87
94
92
77
98
100
98
99
99
95
97
95
91
94
95
96
94
96
84
92
96
94
97
88
99
87
32.6
24.3
35.3
13.6
25.2
14.6
24.3
30.7
19
30.3
11.6
20
14.5
# of Students
(ADM)
1763
973626
2426
1020
1102
1465
803
1359
1973
2787
1132
1155
1621
866
1432
17
100
95
90
85
2002/03
Dobyns Bennett
2003/04
2002/03
Science Hill
2003/04
2002/03
Sullivan South
2003/04
2002/03
Tennessee High
2003/04
2002/03
Greeneville City
2003/04
2002/03
Oak Ridge
2003/04
2002/03
Maryville
2003/04
2002/03
State Average
2003/04
Gateway Eng. II % Prof. and Above
Plot the data (See our recommended template.
Also consider adding any state mandated goals.)
Year within School
• Compared to State Average 2003-2004
• Compared to Best in Class 2003-2004
• Possible Benchmarking Opportunities
8% above State Average
1% above 2nd best in class
(Maryville & Oak Ridge)
None
Only 2 years of data are available for this measure.
18
Interpret the data (think about what each plot is
telling you and if it makes sense) Take notes.


Due to previous year improvement plan
implementation DB made great
improvement in this measure, going from
worst in class to best in class in one year.
Note improvement from 84 to 97%
proficient and above. This is a major
improvement.
19
Note some of the obvious areas of opportunity, schools
to benchmark against, and those areas your school
should feel good about.
Gaps Compared to Best in Class
School Year 2003-2004
Gap with Best in Class
5
0
-5
DB-Best in Class x 100
Best in Class
-10
-15
Gateway Alg. I % Prof. and Above
Gateway Eng. II % Prof. and Above
Gateway Bio. I % Prof. and Above
Attendance
ACT Math
Writing 11th
ACT Science
ACT
ACT Reading
ACT English
Graduation Rate
-20
Measure
Gateways are now equivalent or better than all of the comparison schools.
Graduation Rate is an obvious opportunity area, especially since our mission is
to graduate each student.
20
Put a presentation together for
your school or school system.

Parts of DB's presentation are in this
document. For more details contact Chris
Hampton at champton@k12k.com
21
DB Mission and Vision Statements


The mission of Dobyns-Bennett High School is to
graduate each student and provide a challenging
and diverse learning environment which fosters
intellectual growth, individual integrity, and
community responsibility.
The vision of Dobyns-Bennett High School is to
be nationally recognized for educational
excellence in a supportive learning community.
22
Present it first to the administrative leadership
team and prioritize the opportunities for your
school or school system.
DB Identified Areas to Proceed to Next Step
2003-2004 School Year


Attendance – This area is suggested because DB averages very close to
the state requirement of 93 and thus underperforms some years. There is
opportunity for improvement and looking at peer schools indicates it can
be done.
Graduation Rate – DB was identified as a Targeted School for 2003/2004
but this was successfully appealed. Nonetheless, at 79% we are well
below the target of 90% and need to make obvious gains to reach this
target by 2009/2010. We are also only better than 1 of the schools we
compared to, Greeneville City. There is opportunity for improvement and
looking at peer schools indicates it can be done.
23
Filter the presentation on to other teams as
appropriate. Be sure to field questions and help
people draw conclusions. Many people are not
used to looking at this kind of data this way.

Groups to present to include
 Department Teams
 School Board
 Chamber of Commerce
 PTSA
 other appropriate groups for your community.
24
Put together teams around each major opportunity
and implement Hank Somers' 11 Step Process for
Successful Academic Improvement Projects. Be
sure to include this as part of your School
Improvement Plan or other appropriate processes.
Due to the the performance of certain sub-groups on the
English Gateway test, DB was listed as a targeted school,
as outlined by the state of Tennessee from the "No Child
Left Behind" legislation. Accordingly, this need was added
to DB's Tennessee School Improvement Plan.
 The students and faculty were made aware of the
importance of this test result and given ownership to help
resolve the issues.
 Data was used to determine which subgroups of students
were negatively impacting the test results.
 A team was formed consisting of diverse stakeholders
including: English teachers, students, administration and
teachers outside of English.
25

Gateway English II (2002)
%Proficient and Above
Desired Direction
95
90
State Average 2002/03
Maryville 2002/03
Oak Ridge 2002/03
Greeneville City 2002/03
Tennessee High 2002/03
Sullivan South 2002/03
Science Hill 2002/03
85
Dobyns Bennett 2002/03
Gateway Eng. II % Prof. and Above
100
Year w ithin School
• Compared to State Average 2002-2003
3.4% below State Average
• Compared to Best in Class 2002-2003
15.2% below best in class (Maryville)
• Possible Benchmarking Opportunities
None
• Comments: D-B is one of three(Science Hill and Tennessee High) schools in this
study that had more than 45 students identified as economically disadvantaged.
26
Put together teams around each major opportunity
and implement Hank Somers' 11 Step Process for
Successful Academic Improvement Projects. Be
sure to include this as part of your School
Improvement Plan or other appropriate processes.
The team
 Restructured the format of our lowest level English course
 Incorporated curriculum mapping into all English courses
 Incorporated technology by using Gateway geared
software
 Used a team teaching approach for that course
 Emphasized school-wide the importance of the core
Gateway English skills in all curriculum areas.
 Incorporated some intrinsic and extrinsic motivators to
promote student success
 The school recognized and celebrated the achievements.
27
100
95
90
State Average
Maryville
Oak Ridge
Greeneville City
Tennessee High
Sullivan South
Science Hill
2002/03
2003/04
2002/03
2003/04
2002/03
2003/04
2002/03
2003/04
2002/03
2003/04
2002/03
2003/04
2002/03
2003/04
2002/03
2003/04
85
Dobyns Bennett
Gateway Eng. II % Prof. and Above
Gateway English II (2002-2004)
Desired Direction
%Proficient and Above
Year within School
• Compared to State Average 2003-2004
• Compared to Best in Class 2003-2004
8% above State Average
1% above 2nd best in class
(Maryville & Oak Ridge)
• Possible Benchmarking Opportunities
None
• Comments: Note improvement from 84 to 97% proficient and above. This is a
major improvement. Last year DB was a targeted school for this measure
this year they are best in class!!!!!!! Congratulations!!!!
28
Onto Next Year
29
New Goals for 2005


To increase the percentage of students that
successfully complete Algebra I by the end of
their 2nd year of high school from 60% to 75%
by the end of the 2006-2007 SY (5% increase
annually), while maintaining Gateway Math
results at the required AYP standards or higher
in all 9 subgroups.
To increase the Graduation Rate from 78.8% to
90% by the 2009-2010 school year.
30
95
90
85
80
75
State Average
Maryville
Oak Ridge
Greeneville City
Tennessee High
Sullivan South
Science Hill
Dobyns Bennett
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
2001/02
2002/03
2003/04
Gateway Alg. I % Prof. and Above
Gateway Algebra I (2001-2004)
Desired Direction
%Proficient and Above
Year within School
• Compared to State Average 2003-2004
20% above State Average
• Compared to Best in Class 2003-2004
2% above 2nd best in class (Maryville)
• Possible Benchmarking Opportunities
None
• Comments: Continued improvement for 2 years, now best in class!!!
31
Discussion of Algebra Gateway


99% of the students who took the Gateway
Algebra test scored proficient or above
What's to improve?


Students who take Algebra I and do not make a C or
above the 1st semester do not take the Algebra
Gateway test.
The objective is to increase the # that make a C or
above and progress to complete the course and take
the Gateway.
For 2004-2005 DB was 95.4%.
32
Graduation Rate (2003-2004)
Desired Direction
Graduation Rate
95
90
85
80
2003-2004 (78.8%), 2004-2005(84.7%).
State Average
Maryville
Oak Ridge
Greeneville City
Tennessee High
Sullivan South
Science Hill
Dobyns Bennett
75
No Child Left Behind mandates
90% by 2009-2010.
School
• Compared to State Average 2003-2004
4% above State Average
• Compared to Best in Class 2003-2004
16% below best in class (Maryville )
• Possible Benchmarking Opportunities
Maryville and Science Hill.
• Comments: DB is better than only Greeneville City in this group.
33
Implementing Successful
Academic Improvement
Projects
“ASQ Quality Tools Training”
H. Somers
June 1–2, 2005
Stolen with permission.
34
Why Process Improvement
Needed?

This Is About Change!





Academic improvement
SIP targets
NCLB improvement
Expectations for Present Actions
Change Is Difficult!
35
Why Use A
Systematic
Improvement
Process?
36
Helps Educators
Achieve Good
Results – Fast!
37
Success Stories

Elementary



Math
Writing
High School


Gateways
English to “Best in Class”
38
11 Step Improvement Process
Five Key Sections
 Start
 Analyze
 Plan
 Implement
 Improve
39
1. Select School Academic Need


Aligned with School Improvement Plan
Normally Area of Greatest Need
40
Barriers to Support the Educational
Excellence Effort





Perceived notion that they did not need to
improve
Working on Educational Excellence was not a
priority, no time
Resistance to change
Using data to drive decisions was not common
practice
Change in administration at school and district
levels
41
2. Appoint Project Team




One member from each grade
Interface for Grade Level Teams
Share data, best practices, and needs
 To project team
 From project team
SIP Team / Project Team – Same?
42
Steering Team
Data &
Improvement
Best
Needs
Practices
Grade Level Teams
43
3. Analyze Data – Develop
Measures



Where Does Our School Stand?
 Detailed Data Analysis
 Compare to Peers
Validate the Need
Develop Measures
 Long term measure (Lagging)
 Short term measure (Leading)
44
4. Chart Historical Performance –
Develop Goal & Gap



Use Line Charts
 Chart School Performance
 Chart Peer Schools
 Chart Superior Performance
Develop a Goal
Establish a Gap
45
C ha rt S t e wa rd:
____________
School
D ate ______
Improvement Process
100
Benchm arked / Best in Class
Goal
Gap
GOOD
Performance
95
School's Perform ance
Process to Close Gap
What Who When
Plan
___ ___ ___
Do
___ ___ ___
Study ___ ___ ___
Act
___ ___ ___
90
85
School
Goal
Benchmark
80
1999
2000
2001
2002
2003
2004
Year
So urce o f Data
____________
46
5. Benchmark Best Practices




Determine Superior Performance
Determine Best Practices
“Honorably Adopt” Best of Best
Saves Huge Time and Effort
47
6. Develop Plan for
Improvement





Develop Strategy for Improvement
Use Best Practices /Quality Tools
Develop Plan
Include Entire Faculty (Build –in
Participation)
Include Reinforcement Plan
48
PDSA Process
Define
Adjust
the Plan
ACT
Assess
PLAN
Specific Steps for
Improvement
STUDY
Evaluate
the
Results
Analyze
Cause
DO
Implement
the Plan
49
PDSA Improvement Plan
WHAT
WHO
WHEN
PLAN
DO
STUDY
ACT
50
Improvement PDSA Model
(Develop Overall Objective: To improve … …… ……. …… …. .. . …. …. .. . .)
Objective
P
L
A
N
D
1.
2.
3.
4.
Responsible
Date
Target Date
Review data (as appropriate)
Benchmark / select best practices
Develop strategy for improvement
Develop specific plan for improvement
a.
b.
c.
d.
e.
f.
g.
h. etc.
1. Implement Plan a – h above
O
S
T
U
1. Check plan progress during regular Project Team
meetings
2. Evaluate performance charts for performance
improvement
3. Evaluate effectiveness of strategy and
improvement plan
D
Y
A
C
T
1. Adjust PDSA plan as required to achieve
objective
2. Adjust strategy for improvement based on
 what works best
 what adds value
 benchmarked best practices
51
Develop Plan


Develop Spring Before Next School Year
Before School Starts
 Train Faculty

Obtain Support of Entire Faculty
52
7. Implement Plan
 Implement
PDSA Plan
 This is the “Hard Part”
 This is the “Hard Work”
 Get Everyone on Board
53
8. Regularly Assess Progress

Project Team Meets Each Month





Review “Leading” Measures – Making
Progress?
Report Best Practices and Improvement
Needs
Identify Needed Training
Make Plan Changes
Take Report to Grade Level Team
54
9. Assess Progress vs. Goal
at School Year-End
Check “Lagging” Measure (Chart and
Gap)
 Did Close Gap?
 Did Meet Goal?
 Why or Why Not?

55
10. Adjust Goal & Plan for Next
School Year
What Went Well?
 What Needs To Be Changed?
 Adjust Plan
 Be Ready for Implementation at Start
of School Year

56
11. Maintain the Gains /
Multi-Year Effort
Continue Practices that Achieved
Gain
 Make Practices Integral Part of
Teaching Process
 Replicate the Process

57
Commit to Multi-Year Plan

Big Mistake – One Year Improvement
and on to Next Objective
58
Keys for Success
Commitment to Change
 Data Analysis
 Use the Process
 Do NOT Shortcut the 11 Steps
 Multi-Year Commitment

59
So?

Looks like a lot of work.

Yes.

Worth it?

You bet!
60
Question Time

Additional information on how to produce
the benchmarking plots in Excel is
available at www.asq1106.org
61