Introduction to EVAAS
Robin Loflin Smith, Ed.D.
PD Lead, Southeast Region
NCDPI
Outcomes:
• Understand EVAAS philosophy
• Interpret value-added and diagnostic
reports
• Interpret teacher reports
• Reflect on EVAAS data
Note: This presentation is based on
materials provided by SAS.
Our Agenda
•
•
•
•
•
•
Welcome, Introductions, Agenda Overview
Preassessment
EVAAS Philosophy
Reflective Assessments
Teacher Reports
Questions, Exit Ticket
Ncregion2.ncdpi.wikispaces.net
CCSA
4
Preassessment
www.Socrative.com
Click:
Student Log in
Room:
www.google.com/imgres?num=10&hl=en&safe=active&biw=1366&bih=559&tbm=isch&tbnid=qSVaILVuI8Nkq
M:&imgrefurl=http
Using Data
Data can be defined as
information organized for
analysis or used to make
decisions.
What is Data Literacy?
Understanding needed
to
FIND, EVALUATE, and
UTILIZE DATA
to inform instruction.
Standard 6 for Teachers
Teachers contribute to the academic
success of students.
The work of the teacher results in acceptable,
measurable progress for students based on
established performance expectations using
appropriate data to demonstrate growth.
Benefits and Considerations for
Teachers
• Understand academic
preparedness of students
before they enter the
classroom.
• Monitor student progress,
ensuring growth
opportunities for all students.
• Modify curriculum, student
support, and instructional
strategies to address the
needs of all students.
Professional Development
•
•
•
•
is the Key
Culture of School
Sensitivity of Data
Finger Pointing and Blame
Game
Window vs. Mirror
NC Standards for School Executives
Standard 2: Instructional Leadership
• Focuses his or her own and others’ attention persistently and publicly
on learning and teaching by initiating and guiding conversations
about instruction and student learning that are oriented towards high
expectations and concrete goals;
• Creates processes for collecting and using student test data and
other formative data from other sources for the improvement of
instruction
• Ensures that there is an appropriate and logical alignment between
the curriculum of the school and the state’s accountability program
• Creates processes for collecting and using student test data and
other formative data from other sources for the improvement of
instruction
Standard 8 for School Executives
Academic Achievement Leadership
School executives will contribute to the
academic success of students. The work of
the school executive will result in acceptable,
measurable progress for students based on
established performance expectations using
appropriate data to demonstrate growth.
Benefits for Principals
• Gain a consolidated view of student progress and
teacher effectiveness, as well as the impact of
instruction and performance.
• Bring clarity to strategic planning and function as a
catalyst for conversations that must take place to
ensure that all students reach their potential.
• Understand and leverage the strengths of effective
teachers.
• Use the valuable resource of effective teaching to
benefit as many students as possible.
Achievement vs. GROWTH
Student Achievement
Proficient
End of
School Year
Student Growth
Proficient
Not Proficient
Start of
School Year
End of
School Year
Proficiency vs. Growth
Scenario
5th grader begins the year
reading at a 1st grade level.
Ends the year reading at a 4th
grade level.
5th grader begins the year
reading at a 7th grade level.
Ends the year reading at the
7th grade level.
Proficient
Growth
NO
YES
YES
NO
Achievement vs. Growth
Student Achievement: Where are we?
• Highly correlated with demographic factors
Student Growth: How far have we come?
• Highly dependent on what happens as a result
of schooling rather than on demographic factors
Achievement and Poverty
SAS
Academic Growth and Poverty
SAS
.
Underlying EVAAS Philosophy
• All students deserve to make appropriate academic progress.
• There is no “one size fits all” way of educating students who enter a
class at different levels of academic achievement.
• Adjustments to instruction should be based on the academic
attainment of students, not on socio-economic factors.
• Given reliable information on past effectiveness, educators can make
appropriate adjustments to improve student opportunities.
• "What teachers know and can do is the most important influence on
what students learn." (National Commission on Teaching and America's
Future, 1996)
• One of the most important things educators can know is with whom
they are effective and where they need to develop new skills.
* SAS
Table Share
Pull a slip from the
cup on the table,
and answer/discuss
with your table
mates.
Changes in Reporting for 2012-13
2011-12
2012-13
Above
Exceeds Expected Growth
Not Detectably
Different
Meets Expected Growth
Below
Does Not Meet Expected
Growth
Two models in EVAAS
1. The Univariate Response Model (URM) is used for
EOGs in grades 5 and 8 science, EOCs in high
school, and CTE Post-Assessments; URM will be
used for the Common Exams (predicts)
2. The Multivariate Response Model (MRM) is used for
EOGs in grades 3 – 8 mathematics and English
Language Arts (projects)
3. In both cases, the basic methodology is the same:
using a student’s prior test scores to consider his/her
performance on an assessment he or she will take at
the end of the school year.
Value-Added Reporting
Projections
Predictions
Growth Standard Value Added Report
Interpreting the VA Report
• How does the report differ by grade
level?
• Why is fourth grade marked in blue?
Why is fifth grade marked in red? What
does this imply?
• What can you infer from the NCEs?
Reviewing Terms:
Percentiles show a where a data point falls in
relation to the entire set of data. Percentiles
help us compare data points to each other
Normal Curve Equivalents are used instead of
percentiles when the distribution of a set of data
is not normally shaped.
Normal Curve Equivalent Units - NCEs
SAS
th
5 Grade:
NCE 37
th
6 Grade:
NCE 37
30 40 50 60 70
30 40 50 60 70
NCEs and Growth
6
th
grade NCE 37 ― 5
th
grade NCE 37 = 0 = State Growth Standard
The State Growth Standard (0.0) is achieved when students do not lose ground from year to year,
relative to other students, across the state, who take the same test. It signifies one year’s growth.
SAS
Using the Terms in EVAAS
Standard deviation is not used in EVAAS, but it’s easy to
confuse it with the standard error!
Example of Standard Deviation:
2 s.d.
1 s.d.
1 s.d.
2 s.d.
This is the distribution for the heights of dogs at
the dog show. The distribution is normal.
Most of the dog heights fall within one standard
deviation…so we can identify the dogs that are
truly tall or short
Mean
Height
SAS
Reviewing Terms:
Standard error is calculated for one data point,
like a value-added measure for a teacher.
Standard errors help us know our level of
uncertainty around the data point
Standard deviation is calculated for a set of
data, like 10,000 Algebra I EOC scores.
Standard deviations help describe how the set
of data are distributed
Using the Terms in EVAAS
Example of Standard Error:
A school “meets expected growth” for fifth grade math,
which means the estimated NCE gain is below the
growth standard by at most 2 standard errors but less
than 2 standard errors above it
Interpretation: The “actual” value-added score may be
+1.2 NCE, or -.5 NCE or 1.0 NCE….we actually cannot
be sure, which is why we use the category and not the
score itself for high-stakes purposes
About those Standard Errors…
How much confidence do we have that the estimate is different
80%
80%
from
0.0?
90%
90%
95%+
95%+
99%
99%
-3se
Does Not Meet
Expected Growth
SAS
-2se
-1se
+1se
0.0
+2se
Growth Standard
State Average
Meets Expected Growth
+3se
Exceeds
Expected Growth
Predictive VA Model –
Univariate Response Model
• For reports where testing is not sequential
(gr. 5 and 8 science, EOCs in high school, CTE
post-assessments)
• Common Exams
• Students must have three previous test scores
for a predictive model
• Effect is difference between predicted and
observed scores
SAS
Interpreting the VA Report
• How do the fifth grade and eighth grade
results compare?
• Looking at the fifth grade in 2012,
explain why these results are shown as
“Exceeds Expected Growth.”
• Would you be content with these
results? Why/why not?
Diagnostic Reports
Use to identify patterns or trends
of progress among students
expected to score at different
achievement levels
Diagnostic Report
SAS
Diagnostic Reports – the whiskers
Diagnostic Reports
Looking for Patterns
SAS
Interpreting the Diagnostic Report
• Which quintiles met growth?
• Which quintiles had results very
different from prior cohorts? What does
this imply?
• Based on this one data source, what
kind of professional development would
you recommend in this school?
Shed Pattern
SAS
Tent Pattern
SAS
V Pattern
SAS
Opportunity Gap Pattern
SAS
What would an ideal pattern
on a Diagnostic Report
look like for closing the achievement gap?
Desirable Pattern
Go to the website:
ncdpi.sas.com
Overview of School Effects
activity
Overview of School Effects
Overall Results: if color is red
Blue bars above the green line
Blue bars below the green line
Blue bars at/near the green line or
whiskers cross green line
Making Data Work for Kids:
EVAAS Teacher Reports
Why should you care about your
EVAAS Teacher Value Added Report?
•
Beginning with your 2013 report, it becomes part of your evaluation.
– Standard 6 – Teachers contribute to the academic success of their
students. (Measurable Progress)
– Standard 4 – Teachers facilitate learning for their students
– Standard 5 – Teachers reflect on their practice.
But your report is not just an evaluation component. It is also a powerful
tool for improving your effectiveness as a teacher.
Effectiveness Categories
State Growth Standard/State Average = 0.0
Standard Error = a measure of uncertainty
Usually, the more data you have, the smaller the standard error.
Index = Teacher Estimate divided by its Standard Error
Index is 2 or
higher
Index is equal to or greater than a -2 but less than
+2
0.0
Index is less than -2
1- Intro message
2 – ratings on
Standards 1-5
3 – rating on
Standard 6; if the
rating includes both
an individual and a
school-wide VA
score, both are
displayed, as well as
the weighted
average of the two.
4 – three years of
data for an overall
effectiveness score
5 – overall status; no
educator will have
status until after
2014-15
EVAAS Teacher Value Added Report
Supplemental Information Table
SAS
EVAAS Teacher Diagnostic Report
Making Generalizations
• What generalizations can
we make?
• What do we not know?
• How do we find out?
• What questions would you
have for this teacher?
• As an administrator, what
support would you provide?
SAS EVAAS Questions?
http://ncdpi.sas.com
• Sandy.Horn@sas.com
This presentation is based on training
materials provided by SAS.
Dr. Robin Loflin Smith
NCDPI
Professional Development Lead
Robin.smith@dpi.nc.gov
336-802-6824
ncregion2.ncdpi.wikispaces.net
Intro to EVAAS
CCSA 2013
Robin Loflin Smith
Interpreting the Growth Standard VA Report:
•
How does the report differ by grade level?
•
Why is fourth grade marked in blue? Why is fifth grade marked in red? What does this imply?
•
What can you infer from the NCEs?
Interpreting the Predictive VA Report:
•
How do the fifth grade and eighth grade results compare?
•
Looking at the fifth grade in 2012, explain why these results are shown as “Exceeds Expected
Growth.”
•
Would you be content with these results? Why/why not?
Interpreting the Diagnostic Report:
•
Which quintiles met growth?
•
Which quintiles had results very different from prior cohorts? What does this imply?
•
Based on this one data source, what kind of professional development would you recommend in
this school?
What would an ideal diagnostic pattern look like if we are going to close the achievement gap?
1
2
3
4
5