Using NAPLAN summative
data to identify instructional
successes and challenges.
Presented by
Philip Holmes-Smith
School Research Evaluation and Measurement Services
Overview of Session
• Using NAPLAN summative data to identify
instructional successes and challenges
• Understanding the National Scale,
• School Summary analysis,
• Five-Year Trend analysis,
• Individual student-level data analysis
• Value added (Growth) analysis
Understanding the
National Scale
Understanding the National Scale
•
The National Scale is an arbitrary scale – at this stage it is not
related to points along a developmental curriculum. But, it is
highly likely that it will be mapped onto the National
Curriculum at some time in the future.
Understanding the National Scale
•
The National Scale is an arbitrary scale – at this stage it is not
related to points along a developmental curriculum. But, it is
highly likely that it will be mapped onto the National
Curriculum at some time in the future.
•
The National Scale was fixed in 2008 as follows:
900
800
700
+1 STD
600
500
2008
National
Mean
400
300
200
100
-1 STD
– Range: 0-1000
– Mean: 500
– Standard Deviation: 100 (i.e. 68% students between 400-600)
National Averages (2008)
My School Website (2010)
School’s Average
National Average
Average for school’s
with similar ICSEA
Only 20.1% of Australian
school communities are
more advantaged
Understanding the National Scale
•
The National Scale is an arbitrary scale – at this stage it is not
related to points along a developmental curriculum. But, it is
highly likely that it will be mapped onto the National
Curriculum at some time in the future.
•
The National Scale was fixed in 2008 as follows:
– Range: 0-1000
– Mean: 500
– Standard Deviation: 100 (i.e. 68% students between 400-600)
•
The Scale has been divided into ten Bands which are used for
reporting to parents.
– Band 1 covers all scores equal to or less than 270.
– Bands 2 – 9 increment by 52 score points each Band
– Band 10 covers all scores above 686.
Understanding the National Scale
•
The National Scale is an arbitrary scale – at this stage it is not
related to points along a developmental curriculum. But, it is
highly likely that it will be mapped onto the National
Curriculum at some time in the future.
•
The National Scale was fixed in 2008 as follows:
– Range: 0-1000
– Mean: 500
– Standard Deviation: 100 (i.e. 68% students between 400-600)
Yr 9
Yr 7
Yr 5 •
The Scale has been divided into ten Bands which are used for
reporting to parents.
– Band 1 covers all scores equal to or less than 270.
– Bands 2 – 9 increment by 52 score points each Band
– Band 10 covers all scores above 686.
Yr 3
•
At this stage Bands have no explicit curriculum meaning but
results show that for Victorian students in 2011:
–
–
–
–
A typical Yr3 level of performance is at the bottom of Band 5
A typical Yr5 level of performance is almost halfway into Band 6
A typical Yr7 level of performance is a third into Band 7
A typical Yr9 level of performance is at the bottom of Band 8
Parent Reports
(NB. Bands do not equate to Victorian Year Level means)
Year 9
Year 7
Band 10
Year 5
Band 9
Band 9
Band 8
Band 8
Band 8
Year 3
Band 7
Band 7
Band 6
Band 6
Band 5
Band 4
Band 3
Band 2
Band 1
Yr 3
Means
Yr 5
Means
Yr 7
Means
Band 7
Band 6
Band 6
Band 5
Band 5
Band 5
Band 4
Band 4
Band 3
Yr 9
Means
Above National
Minimum Standard
At National
Minimum Standard
Below National
Minimum Standard
Victorian State Averages - 2011
(By Year Level and Dimension)
738
Band 10
686
Band 9
634
Band 8
586.6
582
Band 7
530
Band 6
545.9
504.5
478
Band 5
426
434.5
582.0
585.7
534.4
539.8
539.9
510.6
493.5
424.3
592.5
Year 9
552.6
Year 7
500.6
Year 5
414.1
Year 3
580.3
492.1
435.2
417.7
Band 4
374
Band 3
322
Band 2
270
Band 1
218
Reading
Writing
Spelling
Gram. & Punct.
Numeracy
Cautionary Note #1
• Equal scores amongst different dimensions
(on the National Scale) do not equate to
equal levels of performance in terms of
expected VELS levels. e.g.
– A National Yr9 Reading score of 586.6 is equivalent
to a VELS score of 5.25, but
– A higher National Yr9 Numeracy score of 592.5 is
equivalent to a lower VELS score of 4.82.
Comparing National Scale Scores to
Estimated VELS Equivalent scores
• The Victorian means for Year 3 and Year 9 Reading and Numeracy on the
National scale are compared to the estimated VELS equivalent scores
below:
Year 3 Victoria Means
Dimension
Reading
Numeracy
Year 9 Victoria Means
National
Scale
VELS
Equivalent
National
Scale
VELS
Equivalent
434.5
414.1
2.39
1.85
586.6
592.5
5.25
4.82
• Compared to our expected curriculum outcomes for Year 3 students
(2.175), the State Reading mean is about 1½ term ahead of where we
expect a typical Year 3 student to be. However, the State Numeracy mean
is about 2½ terms below where we expect a typical Year 3 student to be.
• Compared to our expected curriculum outcomes for Year 9 students
(5.175), the State Reading mean is just above where we expect a typical
Year 9 student to be. However, the State Numeracy mean is about 2½
terms below where we expect a typical Year 9 student to be.
Comparing AIM scores (VELS May equivalent)
to NAPLAN scores (Est. VELS equivalent)
VELS Level
READING
7.00
6.75
6.50
6.25
6.00
5.75
5.50
Expected
5.25
Yr9 Level
5.00
4.75
4.50
Expected
4.25
Yr7 Level
4.00
3.75
3.50
Expected
3.25
Yr5 Level
3.00
2.75
2.50
Expected
2.25
Yr3 Level
2.00
1.75
1.50
1.25
1.00
0.75
0.50
0.25
0.00
2002
2003
above
Yr 10
Yr 10
Yr 9
Yr 8
Yr 7
Yr 6
Yr3
Yr5
Yr 5
Yr7
Yr 4
Yr9
Yr 3
Yr 2
Yr 1
AIM
NAPLAN
VELS (May equiv.) scores
VELS (Est. equiv.) scores
2004
2005
2006
2007
Year
2008
2009
2010
2011
Prep
2012
Comparing AIM scores (VELS May equivalent)
to NAPLAN scores (Est. VELS equivalent)
VELS Level
WRITING
7.00
6.75
6.50
6.25
6.00
5.75
5.50
5.25
Expected
5.00
Yr9 Level
4.75
4.50
4.25
Expected
Yr7 Level
4.00
3.75
3.50
3.25
Expected
Yr5 Level
3.00
2.75
2.50
2.25
Expected
Yr3 Level
2.00
1.75
1.50
1.25
1.00
0.75
0.50
0.25
0.00
2002
2003
above
Yr 10
Yr 10
Yr 9
Yr 8
Yr 7
Yr 6
Yr3
Yr5
Yr 5
Yr7
Yr 4
Yr9
Yr 3
Yr 2
Yr 1
AIM
NAPLAN
VELS (May equiv.) scores
VELS (Est. equiv.) scores
2004
2005
2006
2007
Year
2008
2009
2010
2011
Prep
2012
Comparing AIM scores (VELS May equivalent)
to NAPLAN scores (Est. VELS equivalent)
VELS Level
SPELLING
7.00
6.75
6.50
6.25
6.00
5.75
5.50
5.25
Expected
Yr9 Level
5.00
4.75
4.50
4.25
Expected
Yr7 Level
4.00
3.75
3.50
3.25
Expected
Yr5 Level
3.00
2.75
2.50
Expected
2.25
Yr3 Level
2.00
1.75
1.50
1.25
1.00
0.75
0.50
0.25
0.00
2002
2003
above
Yr 10
Yr 10
Yr 9
Yr 8
Yr 7
Yr 6
Yr3
Yr5
Yr 5
Yr7
Yr 4
Yr9
Yr 3
Yr 2
Yr 1
AIM
NAPLAN
VELS (May equiv.) scores
VELS (Est. equiv.) scores
2004
2005
2006
2007
Year
2008
2009
2010
2011
Prep
2012
Comparing AIM scores (VELS May equivalent)
to NAPLAN scores (Est. VELS equivalent)
VELS Level
GRAMMAR & PUNCTUATION
7.00
6.75
6.50
6.25
6.00
5.75
5.50
5.25
5.00
4.75
4.50
4.25
4.00
3.75
3.50
3.25
3.00
2.75
2.50
2.25
2.00
1.75
1.50
1.25
1.00
0.75
0.50
0.25
0.00
2002
above
Yr 10
Yr 10
Yr 9
Expected
Yr9 Level
Yr 8
Yr 7
Expected
Yr7 Level
Yr 6
Yr3
Yr5
Yr 5
Expected
Yr5 Level
Yr7
Yr 4
Yr9
Yr 3
Expected
Yr3 Level
Yr 2
Yr 1
2003
AIM
NAPLAN
VELS (May equiv.) scores
VELS (Est. equiv.) scores
2004
2005
2006
2007
Year
2008
2009
2010
2011
Prep
2012
Comparing AIM scores (VELS May equivalent)
to NAPLAN scores (Est. VELS equivalent)
VELS Level
NUMERACY
7.00
6.75
6.50
6.25
6.00
5.75
5.50
5.25
Expected
Yr9 Level
5.00
4.75
4.50
Expected
4.25
Yr7 Level
4.00
3.75
3.50
Expected
3.25
Yr5 Level
3.00
2.75
2.50
Expected
2.25
Yr3 Level
2.00
1.75
1.50
1.25
1.00
0.75
0.50
0.25
0.00
2002
2003
above
Yr 10
Yr 10
Yr 9
Yr 8
Yr 7
Yr 6
Yr3
Yr5
Yr 5
Yr7
Yr 4
Yr9
Yr 3
Yr 2
Yr 1
AIM
NAPLAN
VELS (May equiv.) scores
VELS (Est. equiv.) scores
2004
2005
2006
2007
Year
2008
2009
2010
2011
Prep
2012
Cautionary Note #2
• “Estimated VELS Equivalent Scores” should
be viewed with caution. Specifically:
– It is doubtful that the “Estimated VELS Equivalent
Scores” for Writing, Spelling or Grammar &
Punctuation are truly VELS scores. (The writing
marking rubric does not use VELS criteria and there
are no VELS criteria for Spelling or Grammar &
Punctuation.)
– “Estimated VELS Equivalent Scores” are based on an
equating study that is now three years old.
School Summary Analysis
The School Summary Report
Choose “School
Summary Report”
2010
Choose to look at
All Students or
Girls vs. Boys or
LBOTE Students or
ATSI Students
Choose to look at
National Scale Scores
(Estimated VELS scores
are problematic)
Click on “Preview
Report” to view results
The School Summary Report
There are 30 students in this Year Level.
Therefore:
–
–
50% (or 15 students) are above the median.
50% (or 15 students are below the median.
–
50% (or 15 students) are inside the “box”. Half
of these (7-8 students) are above the median
and half (7-8 students) are below the median.
–
–
–
–
10% (or 3 students) are at or below the 10th
percentile “whisker”.
10% (or 3 students are at or above the 90th
percentile “whisker”.
15% (or 4-5 students) are spread between the
25th down to the 10th percentile.
15% (or 4-5 students) are spread between the
75th up to the 90th percentile.
?
?
3 students on or above the
90th percentile. (Note, all we
know is at least one student
scored on the 90th percentile.)
4-5 students between the
75th up to the 90th
percentile.
7-8 spread above the median.
15 students inside the box.
7-8 bunched below the median.
15 students below median
•
15 students above median
Interpreting “box and whisker” graphs
4-5 students between the 25th
down to the 10th percentile.
?
?
3 students on or below the 10th
percentile. (Note, all we know
is at least one student scored
on the 10th percentile.)
Interpreting the School Summary Report
1. Is the school’s median
above, at or below the
State median?
School’s median is about
half a band below the
State median
Interpreting the School Summary Report
1. Is the school’s median
above, at or below the
State median?
2. For VELS only, is the
Reading median
above, at or below the
Numeracy median?
School’s median is about
half a band below the
State median
Reading is higher than Numeracy
Interpreting the School Summary Report
1. Is the school’s median
above, at or below the
State median?
2. For VELS only, is the
Reading median
above, at or below the
Numeracy median?
3. How does the school’s
spread compare to the
State spread?
School’s median is about
half a band below the
State median
Reading is higher than Numeracy
School has far fewer high
performers than the State
School’s median is at the
State’s 25th percentile
School has far more low
performers than the State
The School Summary Report
Review your own School Summary Report
Go to: https://naplands.vcaa.vic.edu.au/
(Use your school’s NAPLAN Data Service logon and
password to access the report)
Five-Year Trend Analysis
Five-Year Trend Analysis
Difference scores show how much better or
worse you are compared to the State average
This year’s score appears to be part of an
improving trend compared to the State average
Five-Year Trend Analysis
Look at your own SPA data
Go to: http://www.sreams.com.au/
(Use your SPA logon and password to access you SPA account or click on
“Purchase SPA” to purchase and set up a new account for your school)
Individual Student-level
Analysis:
Grouping Students for Teaching
Individual Student-level Analysis
Score that place a student one or more years below
expected are coloured red
Score that place a student at or about expected
level are coloured yellow
Score that place a student one or more years above
expected are coloured green
Individual Student-level Analysis
Click on a dimension
heading to sort the scores
Group requiring more
basic level material
Group requiring grade
level material
Group requiring more
advanced material
Individual Student-level Analysis
Look at your own SPA data
Go to: http://www.sreams.com.au/
(Use your SPA logon and password to access you SPA account or click on
“Purchase SPA” to purchase and set up a new account for your school)
Value-Added (Growth)
Analysis
Growth Analysis (NAPLAN scale)
Letitia appears to have grown 163.6 units from
436.8 in Yr3 to 600.4 in Yr5.
Javier appears to have gone backwards 40.7
units from 506.2 in Yr3 to 465.5 in Yr5.
Growth Analysis (NAPLAN scale)
On Average, the school has grown its students
66.79 units compared to the State’s 83.2 units.
This is 16.41 units less than the state.
Growth Analysis (VELS scale)
Letitia appears to have grown 1.64 VELS units
from 2.52 in Yr3 to 4.16 in Yr5.
Javier appears to have gone backwards 0.43
VELS units from 3.11 in Yr3 to 2.68 in Yr5.
Growth Analysis (VELS scale)
On Average, the school has grown its students 0.68
VELS units compared to the State’s 0.84 VELS units.
This is 0.16 VELS units less than the state.
Growth Analysis (VELS scale)
Difference scores show how much more or less
growth you show compared to the State growth
This year’s growth appears to be part of an
worsening trend compared to the State growth
Value-Added (Growth) Analysis
Look at your own SPA data
Go to: http://www.sreams.com.au/
(Use your SPA logon and password to access you SPA account or click on
“Purchase SPA” to purchase and set up a new account for your school)