The small schools dilemma –
reliable analysis and target setting
from NAPLAN data
Educational Measurement and School Accountability
Directorate (EMSAD)
Educational Measurement and School Accountability Directorate
Small school data analysis
Medium and large school analysis relies on
comparison of outcomes with those of previous
years and with outcomes of similar schools.
Small schools analysis needs to focus on the
progress of cohorts rather than comparison of
outcomes for different cohorts.
Educational Measurement and School Accountability Directorate
Data instability
• Aggregated data for small schools are
inherently unstable.
• Confidence intervals around means and band
percentages are much greater than for larger
schools.
• Cohort differences are likely to be much
greater.
Educational Measurement and School Accountability Directorate
Percentage in bands – large school
Educational Measurement and School Accountability Directorate
Means trend – large school
Educational Measurement and School Accountability Directorate
Percentage in bands – small school
Educational Measurement and School Accountability Directorate
Means trend – small school
Educational Measurement and School Accountability Directorate
Small school confidence intervals
Confidence interval
of +/- 52
Educational Measurement and School Accountability Directorate
Large school confidence intervals
Confidence interval
of +/- 13
Educational Measurement and School Accountability Directorate
What is a ‘small school’ for analysis and
reporting purposes
• Reporting limits are usually set out of privacy considerations.
ACARA uses a minimum of 5 students. NSW DEC uses a
minimum of 10 students.
• Numbers required for comparison of cohort performance are
greater than for reporting purposes.
• Overall school performance measures used in school
performance graphs require results for a minimum of 15
students to construct a ‘reliable’ estimate. (Performance
measures based on 10 to 14 students are provided but should
be interpreted with caution.)
• The decisions you make regarding analysis and target setting
need to be based on the size of the group and the nature of the
data (e.g. means or band percentages.)
Educational Measurement and School Accountability Directorate
School size data
The average enrolment of NSW DEC primary schools is 260 students
(approximately 37 per year).
Ten per cent of NSW DEC primary schools have less than 23 students
(3 per year).
Twenty per cent of NSW DEC primary schools have less than 50
students (7 per year)
Thirty per cent of NSW DEC primary schools have less than 90
students (13 per year)
About a third of NSW DEC primary schools could described as ‘Small
schools’ for analysis and reporting purposes.
Educational Measurement and School Accountability Directorate
Percentage of government primary
schools with less than 50 students
State
Percentage
ACT
7
NSW
31
NT
Unavailable
QLD
37
SA
27
TAS
21
VIC
28
WA
22
Educational Measurement and School Accountability Directorate
Useful data sources for small schools
• Expected growth data
• Item analyses by syllabus outcomes for tracked
cohorts and individual students
• Relative performance graphs
• Data aggregated over several years
Educational Measurement and School Accountability Directorate
Y3 to Y5 average growth in reading by Y3 reading score
Y3 reading
score
5
362
Y3 reading
Mean
score
293
109
87
310
302
105
416
79
128
252
311
105
426
78
157
231
320
103
437
76
179
192
328
101
448
75
198
180
337
96
461
73
214
160
345
96
474
73
228
147
354
91
489
70
241
141
362
90
506
68
252
131
370
88
527
63
263
125
379
87
553
52
274
114
388
82
591
31
284
111
397
80
669
-25
Mean
Y3 reading
Mean
score
406
81
Educational Measurement and School Accountability Directorate
Expected growth
• Expected growth differs according to the
starting score.
• Approximately 57% of Year 5 students
achieved expected growth in reading
Educational Measurement and School Accountability Directorate
Year 5 student growth chart
Educational Measurement and School Accountability Directorate
Year 7 student growth chart
Educational Measurement and School Accountability Directorate
Cohort item analysis – Year 3 reading 2008
RS 2.5: Makes some inferences about ideas implicit in texts
Educational Measurement and School Accountability Directorate
Cohort item analysis – Year 5 reading 2010
RS 2.5: Makes some inferences about ideas implicit in texts
Educational Measurement and School Accountability Directorate
Individual item analysis – Year 3 reading 2008
RS 2.5: Makes some inferences about ideas implicit in texts
Educational Measurement and School Accountability Directorate
Relative performance – numeracy v reading
Educational Measurement and School Accountability Directorate
Relative performance – writing v reading
Educational Measurement and School Accountability Directorate
Using rolling 3-Year average data
2008
2009
2010
3 year total
3 year %age
Band 1
2
1
2
5
26
Band 2
3
2
1
6
32
Band 3
2
2
1
5
26
Band 4
0
0
1
1
5
Band 5
1
0
1
2
11
Band 6
0
0
0
0
0
Total
8
5
6
19
100
410.8
423.7
422.6
Average
Educational Measurement and School Accountability Directorate
417.9
Target setting for small schools
• Targets should be achievable (realistic) and
(preferably) measurable.
• It is preferable if small school targets don’t rely on
comparison with raw performance of previous
cohorts. Comparison with 3-year averages may be
appropriate.
• Small school targets are best set in relation to the
prior performance of the students to whom the
targets relate.
Educational Measurement and School Accountability Directorate
Target setting options
• Using the school target setting tool (Replace Year 5
data with Year 3 data and express targets in terms of
percentage proficient, for example. )
• Expected growth targets
• Three year rolling average targets
• Improvement targets in terms of syllabus outcomes
(data extracted from individual student learning
plans).
Educational Measurement and School Accountability Directorate
Using the school target setting tool
Target Setting Spreadsheet: Year 5 Reading
Sample Small School
2373
Step 2: Enter the anticipated number of students in Year 5 in each year
Band Percentages 2008 - 2010
0.0
Average*
14.3
6.3
Band 7
0.0
0.0
0.0
0.0
Band 6
0.0
0.0
14.3
6.3
Band 5
0.0
80.0
57.1
50.0
Replace Year 5 data
0
0
6.3
with 2009 Year 13
0
0
band
percentages
0.0
0
6.3
50.0
Band 4
50.0
0.0
0.0
12.5
12.5
Band 3
50.0
20.0
14.3
25.0
25.0
100.0
100.0
100.0
100.0
###
0
0
0
0
0
0
1
8
0
0
0.0
1
2013
2012 Targets
%
0.0
1
6.3
0
0
0
0
0
0
0
6.3
0
8
16
Numb er of students
Unadjusted Adjusted
Proficient
0.0
2010
Proficient
Band 8
2009
2012
2011 Targets
Numb er of students
Unadjusted Adjusted
Minimum
2008
16
50.0
2
2
12.5
4
4
25.0
16
16
100.0
0
0
1
0
1
0
2013 Targets
%
0
0.0
1
1
6.3
8
50.0
2
2
12.5
4
4
25.0
16
16
100.0
0
0
0
0
0
0
6.3
0
8
0
16
Numb er of students
Unadjusted Adjusted
Proficient
2011
0
0
0
0
Minimum
3 Year Average
Minimum
Step 1: Select starting point for analysis 2010 results or 3-year average
0
0
%
1
1
6.3
0
0
0.0
1
1
6.3
8
8
50.0
2
2
12.5
4
4
25.0
16
16
100.0
Step 3: Use the arrows to select the desired band movements. This moves students up from one band to the next.
Note: the band distribution chart reflects the percentages of the adjusted numbers above
Band Distribution
Percentage of students at or below minimum standard
120
60
100
37.5
37.5
20
10
37.5
40
37.5
20
60
14.3
30
80
20.0
40
100.0
Percentage
50
Percentage
Minimum Standard
Step 4: Select Chart - Minimum Standard or Proficient
2008
2009
2010
3 Yr Ave
2011
2012
2013
0
0
Band 3
Band 4
State
Band 5
3 Year Average
Band 6
2011
Band 7
2012
2013
Band 8
School A ctual
Moderate target line
School Target
A mbitious target line
Educational Measurement and School Accountability Directorate
Target
zone
Region
State
Possible types of targets
• Increase percentage of all students with individual learning
plans (K-6) achieving 80% of their learning goals for
reading (from 2011 achievement average of 66%)
• Increase the percentage of all students achieving greater
than or equal to expected growth to 70% in 2011 (from
59% in 2010)
• Increase the percentage of the 2009 Year 3 cohort in the
top 3 NAPLAN bands from X% to Y% in Year 5 2011
• Increase the three year aggregated percentage of Year 5
students achieving in the top three bands to 81% (from
the 2008-2010 average of 87.5%).
Educational Measurement and School Accountability Directorate
Educational Measurement and School Accountability Directorate