Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Correlation Data
07/28/14
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Multiple Measures
Principal Effectiveness Ratings as Defined in Act 82
• For the overall principal effectiveness rating, we know that…
– 50% will comprise the observation/practice piece (Framework
for Leadership).
– 15% will be derived from building level data (School
Performance Profile – SPP).
– 15% will be determined by the relative strength of conversation
regarding the connectedness between average teacher
observation/practice ratings and teacher-level measures.
– 20% will incorporate Elective Data / SLOs for principals.
(note: for 2014-15, elective data is optional)
2
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Principal Effectiveness System in Act 82 of 2012
Observation/ Practice
Framework for Leadership Domains
Strategic/Cultural Leadership
Systems Leadership
Leadership for Learning
Professional and Community Leadership
Building Level Data/School Performance Profile
Indicators of Academic Achievement
Indicators of Closing the Achievement Gap, All Students
Indicators of Closing the Achievement Gap, Historically
Underperforming Students
Academic Growth PVAAS
Other Academic Indicators
Extra Credit for Advanced Achievement
Building
Level Data,
15%
Observation
/Practice
50%
Correlation
Data Based on
Teacher-Level
Measures,
15%
Elective
Data
20%
Correlation Data/Relationship
Based on Teacher Level Measures
Elective Data/SLOs
District Designed Measures and Examinations
Nationally Recognized Standardized Tests
Industry Certification Examinations
Student Projects Pursuant to Local Requirements
Student Portfolios Pursuant to Local Requirements
June 26, 2014
3
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Correlation Data
Based Upon Teacher Level Measures
• It was recommended that we adopt a process that focuses on the
conversation between the supervising administrator and the
principal / school leader based on the principal’s knowledge,
understanding and intended use of the evidence / data presented.
• The rating for correlation data is based on a qualitative assessment
by the supervising administrator of the level of understanding (0, 1,
2, or 3) of the quantitative analysis conducted by the principal /
school leader.
4
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Correlation Data
Based Upon Teacher Level Measures
• To aid the supervising administrator in assigning a correlation rating (0, 1,
2, or 3), a Performance Level Descriptor Chart has been developed.
• This chart will provide guidance for rating the conversation between the
supervising administrator and the principal / school leader regarding the
connectedness between Teacher Level Measures and Observation and
Practice (Framework for Teaching) ratings.
• Teacher Level Measures “shall include, but not be limited to any
combination of one or more of the following data”
• Building Level Data / SPP
• Teacher Specific Data (PVAAS, etc.)
• Elective Data / Teacher SLOs
5
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Correlation Data
Performance Level Descriptor Chart
• The conversation will be measured based upon the principal /school
leader’s understanding of these aspects:
– Degree of understanding of evidence presented regarding the
relationship between teacher-level measures and teacher
observation and practice ratings.
– Quality of explanation provided for observed relationships
between teacher-level measures and teacher observation and
practice ratings.
– Plans for how the data will be used to support school and LEA
goals.
6
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Correlation Data
Performance Level Descriptor Chart
7
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Correlation Data
Looking at “Connectedness”
• The conversation will be based upon the connectedness between
teacher level measures and observation/practice ratings.
• The following charts (displayed on subsequent slides) may be used
to determine the relative strength of the connectedness between the
teacher level measures and observation/practice (Framework for
Teaching) ratings.
• Note: The ranges within the charts are based upon the conversion tables
within PA 82-1 (Teacher Rating Tool).
8
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Possible Levels of Connectedness Between Average
Teacher Observation/Practice Ratings and Teacher
Level Measures Data (SPP)
9
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Possible Levels of Connectedness Between Average
Teacher Observation/Practice Ratings and Teacher
Level Measures Data (PVAAS)
10
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Possible Levels of Connectedness Between Average
Teacher Observation/Practice Ratings and Teacher
Level Measures Data (Elective Data / SLO)
11
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
www.education.state.pa.us
Carolyn C. Dumaresq, Acting Secretary of Education
Guidance for Determining Correlation
A Process for Exploring Connectedness
• To assist supervising administrators and principals / school leaders to
engage in meaningful conversations, a guidance document has been
developed that outlines a process for analyzing Teacher Level Measures
Data and Observation and Practice (Framework for Teaching) ratings.
Step
#
Definition of
Performance Level
Indicator
1
Degree of
understanding
2
Degree of
understanding
3
Degree of
understanding
Description
Organize your teacher observation/practice data, demographic data, teacher level measure
data, and any other data deemed relevant (i.e. format an Excel Spreadsheet).
Locate the School Performance Profile (SPP) for your school using the School
Performance Profile website.
Determine the average observation/practice rating for all your teachers within your
building.
12
Measuring Principal Effectiveness
▪
Tom Corbett, Governor
www.education.state.pa.us
Carolyn C. Dumaresq, Acting Secretary of Education
Guidance for Determining Correlation
A Process for Exploring Connectedness
Step
#
Definition of
Performance Level
Indicator
4
Degree of
understanding
5
Degree of
understanding
Description
Use the Connectedness with SPP chart to help determine a possible level of connectedness
between the average observation/practice rating for all teachers in your building and the
teacher-level measure for the School Performance Profile (SPP)**.
Repeat steps #3 and #4 for subsets of teachers in your building.
Compare possible levels of connectedness between teacher observation/practice ratings
and SPP scores for all the teachers (step #4) and the various subsets (step #5).
6
Quality of explanation
provided for observed
relationships
7
Plans for how the data
will be used to support
school and LEA goals
8
Degree, Quality, and
Plan
9
Degree, Quality, and
Plan
Is there evidence indicating differences in possible levels of connectedness when
comparing the average of all teacher observation/practice ratings and SPP versus the
subset(s) of teacher observation/practice ratings and SPP ?
Identify possible opportunities where your analysis may help inform comprehensive
planning, principal SLOs, etc.
Based upon the conversation, the supervising administrator will assign a rating (0, 1, 2 or
3) relative to the principal/school leader’s knowledge, understanding and intended
application of evidence/data presented.
The correlation rating is then entered into the Principal Rating Tool.
13
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #1)
•
Organize your teacher observation/practice data, demographic data, teacher level
measure data, and any other data deemed relevant (i.e. format an Excel
Spreadsheet). Provided below is a screen shot of the Building Summary of
Teacher Ratings spreadsheet that is provided by PDE to assist in collecting data.
14
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #2)
• Locate the School Performance Profile (SPP) for your school using the
School Performance Profile website http://paschoolperformance.org/.
– Find the Academic Performance Tab to display the Academic Performance
Data for your school. Then click on the View Calculation link located at the
bottom of the page for the details.
– For ABC High School, we have developed SPP Simulation #1 for use in this
training. The SPP score we will use is 54.1.
15
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
The SPP
Score is
54.1
16
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #3)
• Determine the average observation/practice rating for all your teachers
within your building.
– Note that if you use the Building Summary of Teacher Ratings spreadsheet,
then the average is calculated for you. For our example, average for ABC High
School is 2.16.
The average
teacher
observation
/practice
rating is 2.16
17
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #4)
• Use the Connectedness with SPP chart to help determine a possible level of
connectedness between the average observation/practice rating for all
teachers in your building and the teacher-level measures data (SPP School Performance Profile).
– On the Connectedness with SPP chart, identify the range where your average
teacher observation/practice rating falls. In our example, the 2.16 average
teacher observation/practice rating falls within the 1.50 – 2.49 range.
– Using the average teacher observation/practice range identified above, identify
the range where your SPP score falls on the Connectedness with SPP chart. In
our example, the 54.1 SPP score falls within the below 60 range.
– Using the SPP range identified above, determine a possible level of
connectedness. In our example, we can explore possible sources of evidence
that may indicate a limited level of connectedness between the average teacher
18
observation/practice rating and SPP.
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
19
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #5)
• Repeat steps #3 and #4 for subsets of teachers in your building.
– Locally, the determination is made as to which subset(s) will be meaningful to
analyze. This decision should made in such a way that you can conduct a
meaningful analysis as to any trends that may be occurring relative to the
ratings of subsets of teachers when compared to all teachers.
• For our ABC High School example, we are going to analyze subsets of
those teachers that have associated PVAAS scores.
– To do this, we need to conduct three independent sorts of our sample Building
Summary of Teacher Ratings spreadsheet (sort by ELA-Literature, MathAlgebra I, and Science-Biology teachers that have associated PVAAS scores).
20
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #5)
• By using Excel and manipulating the data from our sample Building
Summary of Teacher Ratings spreadsheet, we can determine the following:
– The average teacher observation/practice rating is 2.10 with an average
PVAAS score of 62.00 for ELA – Literature teachers.
Average PVAAS reading score
Average teacher rating for ELALiterature teachers with
associated PVAAS reading scores
21
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
www.education.state.pa.us
Carolyn C. Dumaresq, Acting Secretary of Education
Using the Guidance Document
Example: ABC High School (Step #5)
Average PVAAS math score
•
The average teacher
observation/practice rating is 2.53
with an average PVAAS score of
50.00 for Math – Algebra I.
Average teacher rating for Math –
Algebra I teachers with
associated PVAAS math scores
•
The average teacher
observation/practice rating is 1.90
with an average PVAAS score of
62.00 for Science-Biology.
Average PVAAS science score
Average teacher rating for
Science-Biology teachers with
associated PVAAS science scores
22
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #6 - SPP)
• Compare possible levels of connectedness between teacher
observation/practice ratings and SPP scores for all the teachers
(step #4) and the various subsets (step #5).
– Referring back to the Connectedness with SPP chart and using our SPP score
of 54.1, we can determine possible levels of connectedness between teacher
observation/practice ratings and SPP.
23
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
24
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #6 - Comparison)
• Is there evidence indicating differences in possible levels of
connectedness when comparing the average of all teacher
observation/practice ratings and SPP versus the subset(s) of teacher
observation/practice ratings and SPP?
– In our example, it appears that for our PVAAS Math – Algebra I subset we
may have a possible poor level of connectedness with SPP versus a possible
limited level of connectedness for the other subsets.
– Let’s now consider possible levels of connectedness among teacher
observation/practice ratings (all teachers/subsets) and other teacher-level
measures (PVAAS, Elective Data) to see if this helps to explain these
differences.
25
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
26
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #6 – Plausible Explanations)
• Based upon the results, we see varying levels of possible connectedness
within the subsets when considering the possible connectedness with
PVAAS. Now we need to look for plausible explanations. Possible
guiding questions to assist in this analysis include:
• What trends are you discovering?
• Are these the types of relationships you would expect to see? Why or why
not?
• Which trends are the most positive and/or the most problematic? And
why?
• What evidence would define plausible root causes?
• How does data inform your thinking about how to best support school
and/or district goals?
27
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #6 – Plausible Explanations)
• We found a possible poor connectedness level between the Math – Algebra
I subset of average teacher observation/practice ratings and SPP, as well as
with PVAAS.
• The ratings of the teachers could in fact be accurate, but they are teaching a poorly
aligned curriculum to the standards as demonstrated by the lack of static
achievement (Keystone) at 32.1% advanced or proficient and growth (PVAAS) at
50.0.
• The ratings of the teachers may not be accurate due to various factors including
personal relationships, strong personalities, and experience.
28
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #6 – Plausible Explanations)
• We found a possible poor connectedness level between the Math – Algebra
I subset of average teacher observation/practice ratings and SPP, as well as
with PVAAS (continued)…
• When looking at the SPP calculations for Other Academic Indicators, we see an
attendance rate of 90.12. This indicates that students are regularly attending school;
however, academic achievement is lacking, especially in math. Perhaps teachers are
rated highly based on observations of classrooms where students are engaged;
however, the content being delivered is not aligned to the PA Core Standards and/or
inadequate pedagogies are being used?
• Other plausible explanations…
29
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #6 – Plausible Explanations)
• We found a possible limited connectedness level between the ELALiterature subset of average teacher observation/practice ratings and SPP,
but a possible good connectedness level between the ELA-Literature subset
and PVAAS scores.
•
The possible good connectedness level between the ELA – Literature average teacher
observation/practice ratings and PVAAS could be due to the curriculum being aligned to
the standards, resulting in a stronger link between what is tested and what is taught.
•
The ELA department has two teachers with two or fewer years of experience. These
two teachers also have the highest PVAAS scores. This could be due to their more
recent exposure to the standards and instructional practices designed to improve student
achievement (such as student engagement). Also note that both of these teachers have
been rated as quality instructors, which aligns to their student’s growth on PVAAS.
30
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #6 – Plausible Explanations)
• We found a possible limited connectedness level between the ELALiterature subset of average teacher observation/practice ratings and SPP,
but a possible good connectedness level between the ELA-Literature subset
and PVAAS scores (continued)…
• Five of the eight ELA-Literature teachers have PVAAS scores of 61 or lower, which
could support the assumption that the curriculum is not aligned and/or not being
taught. Or research-based effective pedagogies are not being used in the classroom
to improve instruction and thus student achievement.
• Other plausible explanations…
31
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #7)
• Identify possible opportunities where your analysis may help inform
comprehensive planning, principal SLOs, etc.
• The PVAAS Math – Algebra I subgroup has been identified with a possible
poor connectedness level with SPP and PVAAS. Therefore, it might be
important to include the PVAAS Math – Algebra I subgroup in the
development of strategic interventions to increase student achievement. This
could be accomplished through a Principal SLO and/or Teacher SLOs for this
subset.
• The PVAAS ELA-Literature subset was identified with a possible limited
connectedness level with SPP and a possible good connectedness level with
PVAAS. However, the PVAAS score is close to the lower margin in the
“good” range. Therefore, opportunities could exist to work with these teachers
in the analysis of the data and develop strategic interventions to increase
student achievement. Perhaps reviewing the pedagogies of the two newest
teachers would be beneficial since they have the two highest PVAAS scores? 32
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #8)
• Based upon the conversation, the supervising administrator will
assign a rating (0, 1, 2 or 3) relative to the principal/school leader’s
knowledge, understanding and intended application of evidence/data
presented.
• Referring back to the Performance Level Descriptor Chart, the
supervising administrator (in collaboration with the principal), let’s
consider the following for our example:
– Degree  The principal / school leader demonstrates the ability to
disaggregate teacher observation/practice ratings and teacher-level measures, as
well as conducts an analysis to determine plausible connections among the
data.
33
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
Using the Guidance Document
Example: ABC High School (Step #8)
– Quality  The principal / school leader cites plausible causes for the
connections among teacher observation/practice ratings and teacher-level
measures. Also the principal / school leader articulates why the plausible
connections may have occurred among teacher observation/practice ratings and
teacher-level measures.
– Plan  General plans for increasing student performance based upon the
analysis of teacher observation/practice ratings and teacher-level measures
have been provided. Opportunities exist to further define timelines and
implementation strategies for the incorporation into a Principal SLO.
Therefore, the correlation rating for this principal / school leader is a
“2” based upon a general action plan as opposed to a plan that includes
specificity.
34
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
www.education.state.pa.us
Carolyn C. Dumaresq, Acting Secretary of Education
Using the Guidance Document
Example: ABC High School (Step #9)
• The correlation rating is then entered into the Principal Rating Tool.
Enter 0, 1, 2, or 3 here
Select the “Correlation
Data” tab on the Principal
Rating Tool
35
Measuring Principal Effectiveness
Tom Corbett, Governor
▪
Carolyn C. Dumaresq, Acting Secretary of Education
www.education.state.pa.us
The mission of the Pennsylvania Department of Education is to lead
and serve the educational community, to enable each individual to grow
into an inspired, productive, fulfilled lifelong learner.
36