Pennsylvania Value-Added
Assessment System (PVAAS):
Communicating PVAAS to Your
School Board
PVAAS Statewide Core Team
Winter, 2011
Session Description



This 60 minute session will provide strategies and
approaches for discussing the concept of PVAAS
with your local school board, as well as sharing
actual PVAAS results. Approaches that have
been used by actual districts in Pennsylvania will
be discussed. Materials and resources to use as
tools for communication about PVAAS to your
local school board will be reviewed.
Opportunities for questions and discussion will be
provided.
There are trainer notes included for this
presentation.
Agenda




Documents & Suggested Handouts
for School Board Members
Considerations for Presenting PVAAS
to School Boards
Examining the School Board
Presentation Template Power Point
Questions
Suggested Approach for Initial Session




Keep it simple…keep it conceptual
Discuss what PVAAS is and is not
Start: Focus on Scatter Plots: Achievement vs.
Growth
Summarize important reports!
• Listed in suggested priority order
• Suggest starting with Reading and Math only
• District Value-Added Report
• Value-Added Summary Report
• District (Single Grade) Projection Summary Report
• Performance Diagnostic Summary Report
Go For The BIG IDEAS!
Supports and
Resources to Prepare &
Present at
PVAAS Sessions with
Your Board Members
Show: DVD for School Boards
PVAAS for
School Board Members
by School Board Members
Tom Gentzel, PSBA
 2 Superintendents
School Board Members
 12 minutes


Suggested Resources &
Documents for Board Members


One-page handout titled “What is PVAAS?
Evaluating Growth, Projecting
Performance”
One-page handout titled “Effective Use of
PVAAS Student Projection Reports”
Suggested Resources &
Documents for Board Members

Copies of Key District Reports
• District Value-Added Summary Reports
• District Value-Added Reports
• District Projection Summary Reports

Copies of District Scatter Plots
• District vs IU region vs State
• See next slide for example template

Copy of Presentation PPT
District Scatter Plot – Template
Statewide
Grade 4 Reading
IU Region
Grade 4 Reading
District
Grade 4 Reading
PDE PVAAS
Webpage
•Help Menus
•PowerPoint
Presentations
with Trainer
Notes
• Resource
Materials
• Video Clips
• District Case
Studies
PDE PVAAS Website, New & Archived Webinars,
Intermediate Unit Supports, Other Resources
Key Concepts for
EVERY
PVAAS Session
Key Concepts:
ANY PVAAS Session
Achievement AND Growth
(PVAAS Scatter Plots)
Key Concept:
ANY PVAAS Session
 PVAAS reporting reflects the
effectiveness of your district’s or school’s
Standards-Aligned System
 PVAAS reporting reflects the district’s or
school’s system regarding curriculum,
assessment, and instruction
Key Concepts:
ANY PVAAS Session
There is NO relationship between demographic variables
such as socioeconomic status and growth.
• There are high achieving schools making high growth;
• There are high achieving schools making low growth;
• There are low achieving schools making high growth;
• There are low achieving schools making low growth.
Growth reporting reflects what WE do with students in terms
of academic growth in schools/districts.
-- VAAS can remove the effects of factors not under the control of the
school. (McCaffrey, Lockwood, Koretz & Hamilton, 2003; Ross, Wang,
Sanders, Wright & Stringfield, 1999a; Wright, Horn & Sanders, 1997).
Key Concepts:
ANY PVAAS Session
One of the inherent benefits of the PVAAS
analyses is that all scores for all students
are included in the analyses. Simplistic
approaches are less precise and
sometimes biased.
While PVAAS does use individual student
data to yield value-added reporting, there
are NO individual student measures of
growth yielded from PVAAS!
Key Concepts:
ANY PVAAS Session
No measures of individual
student growth!
Why not?
Key Concepts:
ANY PVAAS Session

The PSSA observed scores are simply ‘snapshots in time’
making comparisons of the observed scores as a measure of
growth unreliable.

The error in an estimate for a data set with only one record
(one student) is too large to make the estimate
meaningful…and

Error depends on variation in the data AND the sample size
(the number of student records in a data set).
An estimate of progress, or growth, based on only one
student would have a much larger error; therefore, it
would be considerably less precise and useful than when
considering a group of students.
Key Concept:
ANY PVAAS Session
 PVAAS District and School Value-Added Reports do not
provide a measure of teacher effectiveness!
 Often
people ask about the meaning of these reports,
especially in situations where there is only one classroom
teacher in a grade level. This is NOT a report on the
effectiveness of the classroom teacher, even in that
situation!
Teacher effectiveness reporting would be done using a
different statistical methodology than the one used for
reporting on district or school effectiveness. Additional
data would be collected as well.
DECISION POINTS
Who? - What? -When? - How?
Decision Point: Audience
What are the backgrounds of school board
members?
Do the board members have experience
with growth data/PVAAS?
Do the board members have any
background knowledge/foundation in
Statistics?
Decision Point:
Intended Goal for Session (s)

PVAAS Overview?

Review of Actual District/School Data?

Part of Planning Process?
• Strategic Planning
• School Improvement Planning
Decision Point: Focus of Data





District?
School?
Grade?
Subgroups?
Which subjects?
• Math, Reading, Science, Writing

Remember, less is more!
Decision Point: Reports Needed
District-wide Focus
• Scatter Plots

State to IU to District
• District Value-Added

Think ahead if there is only one building per grade
• Value-Added Summary (shows individual
schools)
• District (Single Grade) Projection Summary
Report
• (Option) Performance Diagnostic
Summary
Examples of Reports
Talking About Your Results
Think about your
message before
sharing your results!



What do the results
mean?
Do you have a plan of
action regarding the
areas needing
improvement (Red)?
What is your plan of
action for attending to
the Green areas?
Talking About Your Results
Think about your message before sharing
your results!



What do the results mean?
Do you have a plan of action regarding the
areas needing improvement (Red)?
What is your plan of action for attending to
the Green areas?
Decision Point: Session
Facilitators(s)






Principal
Team of Principals
District Administrator(s)
District Administrator(s) + Principal(s)
Support from Intermediate Unit PVAAS
Contacts
Support from PVAAS Statewide Team for
PDE – call-in for questions
Decision Point: Next Steps

Session Focus
• Awareness- Follow with a second
session to review actual district/school
reports
• Analysis – share plan of action to use
data with faculty, making sure to pair
PVAAS data with other data available on
your students

Result: Plan of ACTION!
OTHER
CONSIDERATIONS
PVAAS Projections are an
Approved Way for Schools
to Make AYP
(Growth Model)
Preparing for Parent Requests




There is NO student level growth data in PVAAS.
This is a common request from parents.
What is available at the student level? Projections!
• % likelihood of a student reaching Basic,
Proficient and/or Advanced levels on future
PSSA
Examples of districts that share PVAAS student
projections with parents: Corry Area SD, Derry
Township SD
Plan in advance for how to respond such requests
Questions?
Comments?
Presenting PVAAS to an
Individual School Board
Power Point Template
with Trainer Notes
Pennsylvania Value-Added
Assessment System (PVAAS):
A Compass for Improvement
Presented at [District] School Board Meeting
[Date]
DVD for School Boards
PVAAS for
School Board Members
by School Board Members
Tom Gentzel, PSBA
 2 Superintendents
School Board Members
 12 minutes


Two Types of PVAAS Information
Looking
Forward/Planning…
PVAAS Projection Reports
For Individual Students
and Cohorts of Students
Looking Back/Evaluation…
Value-added Growth Reports For
Cohorts of Students
Today
NO individual student growth measure!
PVAAS IS:
• Analysis of existing assessment data
– Longitudinal PSSA scores in available
content areas
• Math, Reading, Science, Writing
– Provides reliable estimates
• A measure of a school’s effectiveness
• Critical Relationship:
Growth Data vs. Achievement Data
PVAAS:
•
Is not another test
•
Is not a comparison of single test
scores
•
Critical concept:
Does not provide a measure of
growth for an individual student
Current Data Collection
Approaches
High Achievement
Average Achievement
Low Achievement
Achievement is based
on each student’s ending point.
PVAAS Adds Another Variable
To Achievement…Progress
Below
Growth
Standard
At or Above
Growth
Standard
Progress is based
on each student’s starting point.
Viewing Achievement
and Progress Together
Scatter Plots:
Grades 4-8
Reading and Math
The Scatter Plot
Coordinate System
Vertical Axis
PSSA Percent
Proficient or
Advanced
Achievement and Growth - 4th Math
% Proficient or Advanced
AYP Percent
Proficient Target
Growth Standard
Boundary
One
Year
Growth
56
0
-20
0
PVAAS Growth Measure
20
Horizontal Axis
PVAAS Growth
Measure
Scatter Plot Quadrants
and Their Meanings
Quad 2
Adequate Achievement
Negative Growth
Achievement and Growth - 4th Math
% Proficient or Advanced
Quad 3
Below Adequate
Achievement
Negative Growth
Quad 1
Adequate Achievement
Positive Growth
56
Quad 4
Below Adequate
Achievement
Positive Growth
0
-20
0
PVAAS Growth Measure
20
District Scatter Plots
District vs. IU Region vs. State
Math and Reading
Grades 4 – 8
District Scatter Plot – Grade 4
District Scatter Plot – Grade 5
District Scatter Plot – Grade 6
District Scatter Plot – Grade 7
District Scatter Plot – Grade 8
Viewing Achievement
and Progress Together
Scatter Plots:
Grade 11
Reading and Math
Grade 11 Scatter Plots
The PSSA-PVAAS Scatter Plots visually display the
following data from the statewide distribution of PSSA
results for the most current school year.

Achievement
• The mean predicted performance of the cohort.
• The actual mean performance of all students in the state is
displayed for reference

Growth
• School Effect ≈ Observed Mean – Predicted Mean
• Standardized School Effect = School Effect relative to its
error – displays degree of confidence
The following two slides explain the meaning of each
of the regions of an 11th grade scatter plot.
The Scatter Plot
Coordinate System for Grade 11
Vertical Axis
Mean
Predicted
Scaled Score
2010 Mean
Scaled Score
Growth Descriptor
Boundaries (±2SEs)
Horizontal Axis
Growth Measure in
Standard Errors
Scatter Plot Regions
and Their Meanings for Grade 11
District Scatter Plot – Grade 11
Overview of PVAAS
Reporting
PVAAS Reporting



District Level Reporting
School Level Reporting
Grade Level Reporting
•
•
•
•


Reading (Grades 4-8, 11)
Math (Grades 4-8, 11)
Science (Grades 4, 8, & 11)
Writing (Grades 5, 8, & 11)
Subgroup Reporting (ex. IEP, ED, ELL)
Student Level Reporting
• Projections to Future PSSA Performance
• Reading, Math, Science, Writing
• Grades vary based on subject area
Have all of our groups of
students experienced one
year of academic growth
during the last school year?
Value-Added
Growth Reports
Grades 4-8, Reading and Math
PVAAS Value-Added
Growth Descriptors – Grade 4 – 8
Math and Reading
The PA Growth Standard specifies the minimal acceptable
academic gain from grade to grade for a cohort of students. The
Growth Standard is met when the students in a school or district on
average maintain their relative position when compared to the 2006
statewide distribution of student scores (2006 is the base year).
Estimated mean NCE gain equal or greater than zero. There is
evidence that the school has met or exceeded the Growth
Standard. All schools can achieve this rating.
Estimated mean NCE gain less than zero by one standard error or
less. There is slight evidence that the school has not met the
Growth Standard.
Estimated mean NCE gain less than zero by between one and two
standard errors. There is moderate evidence that the school has
not met the Growth Standard.
Estimated mean NCE gain more than two standard errors less than
zero. There is significant evidence that the school has not met the
Growth Standard.
PVAAS Value-added
Growth Descriptors
Grade 4 – 8, Math and Reading
Favorable Indicator
Caution Indicator
Stronger Caution
Strongest Warning
District/School Value-Added
Report Example
Value-Added Summary
Report Example
Have all of our groups of
students experienced one
year of academic growth
during the last school year?
Value-Added Reports
Science, Writing
Grade 11, Reading and Math
Writing, Science & Grade 11
Reading/Math Growth Descriptors
Progress is defined by the district or school
effect, which is a measure of the difference
between observed performance and predicted
performance.
Progress is significantly above the progress of
the average district in the state.
Progress is not detectably different from the
progress of the average district in the state.
Progress is significantly below the progress of
the average district in the state.
Writing, Science &
Grade 11 Reading/Math
Growth Descriptors
Below
Below Expectations
NDD
Not Detectably Different
from Expectations
Above
Above Expectations
District/School Value-Added
Report Example
How well are we supporting our
highest achieving students, with our
average performing students, and
with our lowest achieving students?
Performance Diagnostic
Summary Report
Performance Diagnostic
Growth Descriptors,
Grade 4-8, Math/Reading
Below Growth Standard
Met Growth Standard
Exceeded Growth
Standard
Performance Diagnostic
Growth Descriptors
Science, Writing &
Grade 11 Math/Reading
Below Expectations
Not Detectably Different
from Expectations
Above Expectations
Performance Diagnostic
Summary Report Example 1
Performance Diagnostic
Summary Report Example 2
What is the likelihood that a
student will be reach proficiency
on a future PSSA? What is their
likelihood of reaching the Basic
or Advanced levels?
PVAAS Student Projections
Student Projection
Methodology
• A separate longitudinal modeling
process focused on the performance
scores of individual students.
• The model uses all of the available
data (both math and reading) to project
future PSSA performances in Math,
Reading, Science and Writing.
• Projections can be calculated for an
individual student or groups of
students.
PVAAS Student Projections
Projections to Basic, Proficient, Advanced
Math and Reading:
 3  4 or 5
 4  5 or 6
 5  6 or 7
 6  7 or 8
 7  8
 8  11
Writing:
 3  5
 4  5
 5  8
 6  8
 7  8
 8  11
Science:
 3  4
 4  NA
 5  8
 6  8
 7  8
 8  11
Grade PSSA
Grade PSSA
Grade PSSA
Last Taken
Last Taken
Last Taken
PVAAS Student Projection Report
PVAAS Projections for
Groups of Students
Student Search
Identified Students
Meeting Search Requirements
District (Single Grade)
Projection Summary Report
PVAAS Help Menus
Questions:
PVAAS Materials or
Statewide Implementation
pdepvaas@iu13.org
717-606-1911
PVAAS Report Web Site
https://pvaas.sas.com
www.pde.state.pa.us
333 Market Street
Harrisburg, PA 17126