Consider the Evidence
Evidence-driven decision making
for secondary schools
A resource to assist schools
to review their use of data and other evidence
8
Collated examples
Page 1
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven teaching
I had a hunch that Ana wasn’t doing as well as she could in her
research assignments, a major part of the history course. What
made me think this?
Ana’s general work (especially her writing) was fine. She made
perceptive comments in class, contributed well in groups and had
good results overall last year, especially in English.
How did I decide what to do about it?
I looked more closely at her other work. I watched her working in
the library one day to see if it was her reading, her use of resources,
her note taking, her planning, or what. At morning tea I asked one
of Ana’s other teachers about Ana’s approach to similar tasks. I
asked Ana if she knew why her research results weren’t as good as
her other results, and what her plans were for the next assignment.
I thought about all of this and planned a course of action. I gave her
help with using indexes, searching, note taking and planning and
linking the various stages of her research.
Page 2
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
What can we do with evidence?
Shane’s story
A history HOD wants to see whether history students are
performing to their potential.
She prints the latest internally assessed NCEA records for history
students across all of their subjects. As a group, history students
seem to be doing as well in history as they are in other subjects.
Then she notices that Shane is doing very well in English and only
reasonably well in history. She wonders why, especially as both are
language-rich subjects with many similarities.
The HOD speaks with the history teacher, who says Shane is
attentive, catches on quickly and usually does all work required. He
mentions that Shane is regularly late for class, especially on
Monday and Thursday. So he often misses important information or
takes time to settle in. He has heard there are ‘problems at home’
so has overlooked it, especially as the student is doing reasonably
well in history.
Page 3
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Shane’s story (cont...)
The HOD looks at the timetable and discovers that history is period
1 on Monday and Thursday. She speaks to Shane’s form teacher
who says that she suspects Shane is actually late to school virtually
every day. They look at centralised records. Shane has excellent
attendance but frequent lateness to period 1 classes.
The HOD speaks to the dean who explains that Shane has to take
his younger sister to school each morning. He had raised the issue
with Shane but he said this was helping the household get over a
difficult period and claimed he could handle it.
The staff involved agree that Shane’s regular lateness is having a
demonstrable impact on his achievement, probably beyond history
but not so obviously.
The dean undertakes to speak to the student, history teacher, and
possibly the parents to find a remedy for the situation.
Page 4
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Shane’s story - keys to success
The history HOD looked at achievement data in English and
history.
She looked for something significant across the two data sets, not
just low achievement.
Then she asked a simple question: Why is there such a disparity
between in these two subjects for that student?
She sought information and comments (perceptions evidence
and data) from all relevant staff.
The school had centralised attendance and punctuality records
(demographic data) that form teacher could access easily.
The action was based on all available evidence and designed to
achieve a clear aim.
Page 5
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evidence-driven strategic planning
INDICATORS
FROM DATA
.
asTTle scores
show a high
proportion of
Yr 9 achieving
below
curriculum level
NCEA results
show high nonachievement in
transactional
writing
STRATEGIC GOAL
ANNUAL PLAN
YEAR TARGET
To raise the levels of writing
across the school
Develop and implement
a plan to raise levels of
Writing at Year 9
Raise writing asTTle
results Yr 9boys from
3B to 3A
Strategic action
Development plan to be
based on an analysis of
all available data and to
include a range of
shared strategies
etc
Develop a writing
development plan which
addresses writing across
subjects and levels ,
including targets,
professional development
and other resourcing needs
etc
etc
Poor results in
other language
NCEA
standards
EVALUATION
DATA
Appraisal
asTTle writing
results improve by
…
Perception data
from Yr 9 staff
indicates …
Evaluation of
effectiveness of
range of shared
strategies, barriers
and enablers …
etc
PD
Self
review
etc
School
charter
Page 6
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
The evidence-driven decision making cycle
.Reflect
How will we
teach writing in
the future?
Trigger
Significant
numbers not
achieving well
in writing
A teacher has a
hunch - poor
writers might
spend little time
on homework
Explore data
Survey of
students
shows that
this is only
partially true
Evaluate Has
writing improved?
Intervene
Create multiple
opportunities for writing;
include topics that can
use sport as context;
connect speaking and
writing. PD for staff.
Page 7
Interpret
information
Poor writers
likely to play
sport, speak
well, read less,
do little HW
Analyse
NQF/NCEA
results by
standard
Analyse non
NQF/NCEA data
and evidence
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Question
What are the
characteristics
of students who
are poor at
writing?
Assemble
more data &
other
evidence:
asTTle reading,
homework,
extracurric,
Attendance, etc
More purposeful questions
1. How do year 11 attendance rates compare with other year
levels? Do any identifiable groups of year 11 students attend
less regularly than average?
2. Is the new 6-day x 50-min period structure having any positive
effect on student engagement levels? Is it influencing
attendance patterns? What do students say?
3. Should we be concerned about boys’ writing? If so, what action
should we be taking to improve the writing of boys in terms of
the literacy requirements for NCEA Level 1?
4. The new timing of the lunch break was intended to improve
student engagement levels after lunch. Did it achieve this? If
so, did improvements in student engagement improve student
achievement? Do the benefits outweigh any disadvantages?
Page 8
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Assembling the evidence
• We want to know if our senior students are doing better in one
area of NCEA biology than another.
So … we need NCEA results for our cohort.
• It could be that all biology students do better in this area than
others.
So … we also need data about national differences across the two
areas.
Page 9
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Are our data any good?
A school found that a set of asTTle scores indicated that almost all
students were achieving at lower levels than earlier in the year.
Then they discovered that the first test had been conducted in the
morning, but the later test was in the afternoon and soon after the
students had sat a two-hour exam.
Page 10
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Think critically about data
• Was the assessment that created this data assessing exactly
what we are looking for?
• Was the assessment set at an appropriate level for this group of
students?
• Was the assessment properly administered?
• Are we comparing data for matched groups?
Page 11
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Cautionary tale 1
You want to look at changes in a cohort’s asTTle writing levels over
12 months.
Was the assessment conducted at the same time both years?
Was it administered under the same conditions?
Has there been high turnover in the cohort?
If so, will it be valid to compare results?
Page 12
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Cautionary tale 2
You have data that show two classes have comparable
mathematics ability. But end-of-year assessments show one class
achieved far better than the other.
What could have caused this?
Was the original data flawed? How did teaching methods differ?
Was the timetable a factor? Did you survey student views? Are the
classes comparable in terms of attendance, etc?
Page 13
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Hunches from raw data
2.1
Pamela N
1
A
Lee
2
E
Manu
3
Keisha N
4
E
Bron
5
M
Deane
6
N
Slane
7
A
Sam
8
M
Sione
9
A
10 Oran
E
11 Shirin
E
12 Hanna
E
13 Val
N
14 Liam
15 Morgan M
N
16 Hone
A
17 Mahi
Page 14
2.2
A
A
E
A
M
M
A
A
M
A
E
E
E
A
M
A
A
2.3
N
A
E
N
M
E
N
N
N
A
E
M
E
M
M
N
N
2.4*
N
N
E
N
N
M
N
A
N
A
E
M
E
M
M
N
A
2.5*
N
N
N
N
N
N
N
A
N
A
A
A
N
N
N
N
A
2.6* ABS DET
6
20
N
0
12
A
4
18
E
8
7
N
0
3
A
1
2
A
8
22
N
8
12
A
2
2
N
0
7
A
0
6
E
1
0
M
0
0
E
2
10
M
0
15
M
4
17
N
0
10
A
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Hunches from raw data
•
•
•
Page 15
Is the class as a whole doing better in internally assessed
standards than in externally assessed standards? If so, why?
Are the better students (with many Excellence results) not
doing as well in external assessments as in internal? If so, why?
Is there any relationship between absences and achievement
levels? It seems not, but it’s worth analysing the data to be
sure.
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Reading levels – terms 1 and 4
.
Page 16
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Basic analysis
2.1
1
Pamela N
2
Lee
A
3
Manu
E
4
Keisha N
5
Bron
E
6
Deane
M
7
Slane
N
8
Sam
A
9
Sione
M
10 Oran
A
11 Shirin
E
12 Hanna
E
13 Val
E
14 Liam
N
15 Morgan M
16 Hone
N
17 Mahi
A
Page 17
2.2
A
A
E
A
M
M
A
A
M
A
E
E
E
A
M
A
A
2.3
N
A
E
N
M
E
N
N
N
A
E
M
E
M
M
N
N
2.4*
N
N
E
N
N
M
N
A
N
A
E
M
E
M
M
N
A
2.5*
N
N
N
N
N
N
N
A
N
A
A
A
N
N
N
N
A
2.6* ABS DET
N
20
6
A
12
0
E
18
4
N
7
8
A
3
0
A
2
1
N
22
8
A
12
8
N
2
2
A
7
0
E
6
0
M
0
1
E
0
0
M
10
2
M
15
0
N
17
4
A
10
0
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Basic analysis
• Divide the class into three groups on the basis of overall
achievement
• Identify students who are doing so well at level 2 that they
could be working at a higher level
• Find trends for males and females, those who are absent often,
or have many detentions
• Compare this group’s external assessment success rate with the
national cohort.
Page 18
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Making sense of the results
This table shows that
reading levels overall
were higher in term 4
than in term 1.
Scores improved for most students.
20% of students moved into Level 5.
But the median score is still 4A.
Is this information? Can we act on it?
Page 19
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Interrogate the information
What effect is the new 6-day x 50-min period structure having on
student engagement levels?
Do student views align with staff views?
Do positive effects outweigh negative effects?
Is there justification for reviewing the policy?
Does the information imply changes need to be made to teaching
practices or techniques?
Does the information offer any hint about what sort of changes
might work?
Page 20
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Professionals making decisions
You asked what factors are related to poor student
performance in formal writing.
The analysis suggested that poor homework habits have a
significant impact on student writing.
You make some professional judgements and decide
• Students who do little homework don’t write enough
• You could take action to improve homework habits - but you’ve
tried that before and the success rate is low
• You have more control over other factors – like how much time
you give students to write in class
So you conclude – the real need is to get students to write
more often
Page 21
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evaluate the impact of our action
A school created a new year 13 art programme. In the past
students had been offered standard design and painting
programmes, internally and externally assessed against the full
range of achievement standards. Some students had to produce
two folios for assessment and were unsure of where to take their
art after leaving school.
The new programme blended drawing, design and painting
concepts and focused on electronic media. Assessment was against
internally assessed standards only.
Page 22
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.
Evaluate the impact of our action
• Did students complete more assessments?
• Were students gain more national assessment credits?
• How did student perceptions of workload and satisfaction
compare with teacher perceptions from the previous year?
• Did students leave school with clearer intentions about where to
go next with their art than the previous cohort?
• How did teachers and parents feel about the change?
Page 23
www.minedu.govt.nz
© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.