Principal Evaluation
August 2012
Iowa
Joseph Murphy
Vanderbilt University
• IOWA
• August 2012
Standards
evaluation
Conditions
of Work
governance
incentives
professional development
preparation
internship
accreditation/program
approval
l
i
c
e
n
s
u
r
e
Career Line
induction (residency)
professional development
teacher leadership
preparation
relicensure
mentoring
Education
Standards
LEVERAGE POINT MODEL
Leverage Point
National
data
State of Affairs on
Leverage Point
Potential Improvement
Strategies
Iowa
data
Think, Pair, Share
1. Describe the current evaluation system in
your district.
2. List two things that you don’t like about it.
3. On a scale of 1 (not useful) to 10 (very
useful),
how useful is it in:
 showing you who your really strong principals are
 ensuring principal accountability
 providing targeted professional growth for principals
 promoting improvement at each school
 promoting district-level improvement initiatives
Principal Evaluation:
The National Story
PRINCIPAL EVALUATION:
The National Story
PART A:
Problems with Current
Evaluation Designs
PART B:
“Construction Crew” Work in
Other States
PART A:
PROBLEMS WITH
CURRENT EVALUATION
SYSTEMS
Understudied
Leverage
Point
Content
• Little evidence that systems evaluate
what is important, i.e. not valid
• Insufficient attention to leadership for
learning, especially curriculum and
instruction
Process
• Limited architecture – focus on
one approach (e.g. goals)
• Perfunctory – not a deep process
Impact
• Principals not receiving useful
feedback
• Not promoting professional
growth of principals
• Not promoting organizational
improvement
PART B:
CONSTRUCTION
CREW WORK
Principal Evaluation
“Construction Crew” Work
STATES
Components
Goals
Ohio
Delaware
New
York
Illinois
Kentucky
X
X
X
X
X
School Improvement
X
Student Achievement
X
X
X
X
X
360° Assessment
X
X
X
X
X
X
X
ISLLC Domain
Customer Satisfaction
Professional Growth
X
X
X
Learning-Centered
Leadership
Leadership
1. Leadership matters.
2. In difficult times, leadership matters even
more.
3. In periods of significant organizational
transition, leadership is the major
controllable factor in explaining
organizational performance.
Leadership
4. Instructionally-focused and change-
oriented leadership are especially
effective frames for education.
5. Team leadership seems to offer promise
for enhancing organizational performance.
6. Assessment provides a strategic leverage
point for strengthening leadership.
•External
conditions
•Curriculum
•Instruction
•Culture
Personal
Characteristics
Knowledge
Leadership
Behaviors
Value-added
•Standards
Distribution
•Accountability
Level
School
Experience
Student
Success
•Achievement
•Graduation
•College attendance
•Post graduation success
Values &
Beliefs
Classroom
“Context”
School
•Type
•Level
•Nature
•Student composition
•Staff composition
“precursors”
Figure 1. Learning Centered Leadership Framework
“’behaviors”
District & State
“influence pathway”
“’outcomes”
Table 1– Knowledge Base for the
Assessment System
I. Vision for Learning
II. Instructional Program
III. Curricular Program
IV. Assessment Program
V. Communities of Learning
VI. Resource Acquisition and Use
VII. Organizational Culture
VIII. Social Advocacy
I. Vision for Learning
A.
B.
C.
D.
Developing vision
Articulating vision
Implementing vision
Stewarding vision
II. Instructional Program
A.
B.
C.
D.
Knowledge and involvement
Hiring and allocating staff
Supporting staff
Instructional time
III. Curricular Program
A.
B.
C.
D.
Knowledge and involvement
Expectations, standards
Opportunity to learn
Curriculum alignment
IV. Assessment Program
A. Knowledge and involvement
B. Assessment procedures
C. Monitoring instruction and
curriculum
D. Communication and use of data
V. Communities of Learning
A. Professional development
B. Communities of professional
practice
C. Community-anchored schools
VI. Resource Acquisition and Use
A. Acquiring resources
B. Allocating resources
C. Using resources
VII. Organizational Culture
A.
B.
C.
D.
Production emphasis
Learning environment
Personalized environment
Continuous improvement
VIII. Social Advocacy
A.
B.
C.
D.
Stakeholder engagement
Diversity
Environmental context
Ethics
ISLLC
Educational Leadership Policy
Standards: 2008
As Adopted by the National Policy Board for Educational Administration
STANDARD 1: An education leader promotes the success of
every student by facilitating the development, articulation,
implementation and stewardship of a vision of learning that is shared
and supported by all stakeholders
Functions:
• A. Collaboratively develop and implement a shared
vision and mission
• B. Collect and use data to identify goals, assess
organizational effectiveness, and promote
• organizational learning
• C. Create and implement plans to achieve goals
• D. Promote continuous and sustainable improvement
• E. Monitor and evaluate progress and revise plans
STANDARD 2: An education leader promotes the success of
every student by advocating, nurturing, and sustaining a school
culture and instructional program conducive to student learning
and staff professional growth.
• Functions:
• A. Nurture and sustain a culture of collaboration, trust, learning, and high
expectations
• B. Create a comprehensive, rigorous, and coherent curricular program
• C. Create a personalized and motivating learning environment for students
• D. Supervise instruction
• E. Develop assessment and accountability systems to monitor student
progress
• F. Develop the instructional and leadership capacity of staff
• G. Maximize time spent on quality instruction
• H. Promote the use of the most effective and appropriate technologies to
support teaching
• and learning
• I. Monitor and evaluate the impact of the instructional program
STANDARD 3: An education leader promotes the success of
every student by ensuring management of the organization,
operation, and resources for a safe, efficient, and effective
learning environment.
Functions:
• A. Monitor and evaluate the management and
operational systems
• B. Obtain, allocate, align, and efficiently utilize human,
fiscal, and technological resources
• C. Promote and protect the welfare and safety of
students and staff
• D. Develop the capacity for distributed leadership
• E. Ensure teacher and organizational time is focused to
support quality instruction and
• student learning
STANDARD 4: An education leader promotes the success of
every student by collaborating with faculty and community
members, responding to diverse community interests and needs,
and mobilizing community resources.
Functions:
• Collect and analyze data and information pertinent to
the educational environment
• B. Promote understanding, appreciation, and use of the
community’s diverse cultural, social,
• and intellectual resources
• C. Build and sustain positive relationships with families
and caregivers
• D. Build and sustain productive relationships with
community partners
•
STANDARD 5: An education leader promotes the success of
every student by acting with integrity, fairness, and in an ethical
manner.
Functions:
• A. Ensure a system of accountability for every student’s
academic and social success
• B. Model principles of self-awareness, reflective practice,
transparency, and ethical behavior
• C. Safeguard the values of democracy, equity, and
diversity
• D. Consider and evaluate the potential moral and legal
consequences of decision-making
• E. Promote social justice and ensure that individual
student needs inform all aspects of
schooling
STANDARD 6: An education leader promotes the success of
every student by understanding, responding to, and influencing
the political, social, economic, legal, and cultural context.
Functions:
• A. Advocate for children, families, and caregivers
• B. Act to influence local, district, state, and national
decisions affecting student learning
• C. Assess, analyze, and anticipate emerging trends and
initiatives in order to adapt
• leadership strategies
Principal Performance-Based Evaluation
Prototype
2011-2012
Guiding Principles of the Evaluation
System
Foundations
• Highlight teaching and learning
• Include student growth as a significant factor
• Be standards based: ISLLC and Illinois Performance
Standards for School Leaders
• Underscore learning-centered leadership
Guiding Principles of the Evaluation System
Process
● Be evidence based
● Have set benchmarks agreed upon in advance
● Be transparent
● Foster a culture of collaboration between the
principal and the supervisor
● Be valid and reliable
● Be comprehensive but not overly complex
● Be both formative and summative
● Include multiple measures
● Tap into multiple constituents
● Have well-defined timelines
● Provide ongoing feedback to the principal
● Be site specific, connected to the needs of the specific
school
● Be flexible enough to allow for adjustments
Guiding Principles of the Evaluation System
Outcomes
•Motivate principals to improve
•Promote targeted professional growth
opportunities
•Promote school improvement
•Enhance academic and social learning of
students
•Result in positive or negative consequences
Components of the Evaluation System
•VAL-ED – 25%
Assessment on ISLLC Standards/Illinois
Performance Standards for School Leadership
•Organizational and Professional Goals (25%)
Assessment on meeting an organizational goal (520%)
Assessment on meeting a professional
development goal (5-20%)
•Student Growth (50%)
Assessment on growth in student achievement
Components of the Evaluation System
Component 1 (25%):
•360 degree assessment of principal’s
leadership on
instructional leadership
•Provides nationally normed measures of
principal performance
•In-depth discussion of VAL-ED later this
morning
Components of the Evaluation System
Components 2 & 3
Organizational and Growth Goals – One each – 25%
total
Source of goals:
•Student outcomes
•District and school improvement plans
•Board of education and superintendent objectives
•Surveys (e.g., of school climate, parent satisfaction,
and so forth)
•Previous evaluations (i.e., data from assessments of
components in the evaluation system)
•Accreditation reports
Components of the Evaluation System
Components 2 & 3
Organizational and Growth Goals – One each – 25% total
Characteristics of Effective Goals:
•Be linked to the Illinois Performance Standards for Schools
Leaders and ISLLC Standards.
•Be organizationally grounded and emphasize the direct
contributions of the leader.
•Be anchored in an analysis of multiple sources around
relevant data.
•Be specific, measurable, attainable, realistic, timely, and
challenging.
•Have a longitudinal focus.
•Be mutually determined through collaborative dialogue.
•Be collaboratively reviewed with frequent and specific
feedback.
Components of the Evaluation System
Growth In Student Achievement – 50%
Source of Goals:
•State norm-referenced and criterion-referenced
tests
•Promotion and high school completion
•RTI/progress monitoring data
•College entrance examinations
•End-of-course examinations
•Advanced placement examinations
•Other local assessments
The Evaluation Process
Summative
SelfAssessment
February 1
Summative
Assessment
March 1
VAL-ED
Assessment
December
Goal Setting
June 30
Formative
Conference
November 30
The Evaluation Process
Goal Setting (by June 30)
Steps:
1. Review data sources prior to meeting
2. Bring relevant data to the meeting
3. Discuss and agree on:
 Goals
 Sources of evidence
 Performance measures
The Evaluation Process
Formative Conference: (by November 30)
Steps:
1. Examine evidence and discuss the progress
on target
measures.
2. Review any new data available.
3. Make adjustments to the goals as necessary.
4. Add resources and supports to make goals
attainable.
The Evaluation Process
VAL-ED Assessment (In December)
Completion of the VAL-ED assessment by the
supervisor, principal and teachers in December.
The Evaluation Process
Principal’s Pre-Summative Self-Evaluation
(by February 1)
Steps:
1.Gather evidence
2.Reflect on progress
3.Provide written self-assessment
The Evaluation Process
Summative Evaluation (by March 1)
Steps:
1. Meet and discuss self-reflection of principal
2. Evaluator shares her/his perspective
3. Sign off, using scoring system
SCORING SYSTEM
Components
VAL-ED Performance (25%)
Below
Basic
1
Does Not Meet
Expectations
Basic
Proficient
Distinguished
2
Basic
3
Proficient
4
Exceeds
Expectations
SCORING SYSTEM Components
Organizational Performance Goal (5% to 20%)
Misses the
target by a
considerable
margin
Approaches
the target
1
Does Not Meet
Expectations
2
Basic
Meets the
target
3
Proficient
Significantly
exceeds the
target
4
Exceeds
Expectations
SCORING SYSTEM Components
Personal Growth Goal (5% to 20%)
Misses the
target by a
considerable
margin
1
Does Not Meet
Expectations
Approaches
the target
2
Basic
Meets the
target
Significantly
exceeds the
target
3
Proficient
4
Exceeds
Expectations
SCORING SYSTEM Components
Student Achievement Goal (value-added) (50% )
Misses the
target by a
considerable
margin
Approaches
the target
Meets
the
target
1
Does Not Meet
Expectations
2
Basic
3
Proficient
Significantly
exceeds the
target
4
Exceeds
Expectations
Weights (sample scoring)
Score
Weight
Sub-total
VAL-ED
2
25%
50
Organizational
Goal
2
15%
30
Professional
Growth Goal
2
10%
20
Student
Achievement
Goal
3
50%
150
100%
250
Total
Final Evaluation Rating
Unsatisfactory
Needs Improvement
Proficient
Excellent
1.0-1.6
1.7-2.4
2.5-3.2
3.3-4.0
The categories above are required by PERA
The numbers above are an initial cut; they may need
refinement.
I. Vanderbilt Assessment of
Leadership in Education
VAL-ED
The VAL-ED Instrument
• The instrument consists of 72 items defining six core
component subscales and six key process subscales.
• Principal, Teachers, & Supervisor provide a 360degree, evidenced-based assessment of leadership
behaviors.
• Respondents rate effectiveness of 72 behaviors on
scale 1=Ineffective to 5=Outstandingly effective.
• Each respondent rates the principal’s effectiveness
after indicating the sources of evidence on which the
effectiveness is rated.
• Two parallel forms of the assessment facilitate
measuring growth over time.
• The instrument is available in both paper and online
versions.
Purpose & Uses
• The VAL-ED can be used as part of a comprehensive
assessment of the effectiveness of a leader's
behaviors.
• The VAL-ED reports principal performance through
▫ Norm-referenced scores and
▫ Criterion-reference scores.
• VAL-ED can be used annually or more frequently to:
▫ Facilitate a data-based performance evaluation,
▫ Measure performance growth, and
▫ Guide professional development.
Implementation
•
•
•
•
Identify respondents and invite participation.
Discuss use of results & confidentiality.
Decide paper or online version.
Time and Timing
▫ Average respondent requires 20 to 25 minutes.
▫ Schedule completion after respondents have had a
reasonable time to observe/experience the
leader’s work and its effects on the school.
• Designate person(s) to manage collection and
submission of response forms, if paper version used.
• Ensure teacher confidentiality.
Directions for Completing Rating Scale
An Example Set of Responses
Instrument Development and
Early Studies of Psychometric
Properties
Assessing Learning-Centered Leadership:
The VAL-ED vision…
A leadership assessment system that has the following properties:
•
•
•
•
•
•
•
•
Works well in a variety of settings and circumstances,
Is construct valid,
Is reliable,
If feasible for widespread use,
Provides accurate and useful reporting of results.
Is unbiased,
Yields a diagnostic profile for summative and formative purposes.
Can be used to measure progress over time in the development of
leadership, and
• Predicts important outcomes.
Psychometric Evidence
• Item and response scale development
▫ Based on review of learning-centered leadership literature
and alignment to ISLLC standards.
▫ Critiqued by education leaders and leadership researchers.
• Item sorting study
▫ Established content validity by asking education leaders to
sort the items into 36 cells.
• Cognitive interviews of paper/pencil version
▫ Two rounds of cognitive interviews in three districts each.
▫ Three respondents evaluated the format and items.
• Nine-school pilot test (320/440 teachers = 73%)
▫ Estimated reliability of each of 12 scales.
▫ Established construct validity through factor analysis.
▫ Established face validity through questions to respondents.
Psychometric Evidence
• Cognitive interviews of online instrument
including revisions based on 9-school pilot
• Bias review
▫ Submitted to urban districts to evaluate language.
• 11-school pilot test (283 teachers for 57%
response rate)
▫ Confirmed changes made after 9-school pilot test.
National Field Trial
Design of the Sample
• Target of 300 Schools
▫ 100 elementary, 100 middle, 100 high schools
▫ 150 urban schools, 100 suburban schools, 50 rural schools
▫ Of the 150 urban schools, 50 from Wallace grantee districts and
50 from Wallace grantee states
▫ 75 schools from each of four geographic areas
• The Obtained Sample
▫ 60 districts (9 Wallace districts)
▫ 235 schools for principal data, 253 schools for supervisor data,
245 schools for teachers data
▫ 8,863 teachers
▫ 218 schools for which there was data from all three respondent
groups
▫ 50% of the schools had a 68 percent teacher response or better
Setting Performance
Standards
Performance Level Descriptors
• Distinguished
▫ A distinguished leader exhibits leadership behaviors of core components and
key processes at levels of effectiveness that over time are virtually certain to
influence teachers to bring the school to a point that results in strong valueadded to student achievement and social learning for all students.
• Proficient
▫ A proficient leader exhibits leadership behaviors of core components and key
processes at levels of effectiveness that over time are likely to influence
teachers to bring the school to a point that results in acceptable value-added
to student achievement and social learning for all students.
• Basic
▫ A leader at the basic level of proficiency exhibits leadership behaviors of core
components and key processes at levels of effectiveness that over time are
likely to influence teachers to bring the school to a point that results in
acceptable value-added to student achievement and social learning for some
sub-groups of students, but not all.
• Below basic
▫ A leader at the below basic level of proficiency exhibits leadership behaviors
of core components and key processes at levels of effectiveness that over time
are unlikely to influence teachers to bring the school to a point that results in
acceptable value-added to student achievement and social learning for
students.
The Bookmark Procedure
• Twenty-two panelists: Ten principals, four teachers,
four supervisors of principals, two researchers of
school leadership, and two education policymakers
• Item-ordered booklet based on the National Field
Trial (items from Form A)
• Panel met August 12 and 13, 2008
The Results
• The cut between proficient and basic was set at 3.60.
• The cut between distinguished and proficient was set
at 3.77.
• The cut between basic and below basic was set at
3.42.
• The impact data are
▫ 30% of the principals fall below basic
▫ 50% fall below proficient
▫ 70% fall below distinguished
• While the panel was highly positive about the
experience and felt comfortable with where they had
set the proficient cut, some expressed concern about
having set the distinguished cut too low and the
below basic cut too high.
Post-Standard Setting Reconsideration
• In a follow-up email, panelists were given three options to
consider
▫
▫
▫
Option 1 – Leave the cuts where they are.
Option 2 – Move the below basic cut to be consistent with the table that
set the lowest value (3.29) and the cut for distinguished consistent with
the table that set the highest value (3.87).
Option 3 – The same as Option 2 except the distinguished cut is moved
to 4.0, requiring that a principal on average get a score of 4 on the 5point scale.
• Panelists were given impact data for each possibility across
the three options.
• All 22 panelists responded.
▫
▫
21 favored moving below basic to 3.29
18 favored moving distinguished to 4.00
• The decision was to go with Option 3, thus the cut scores are
3.29, resulting in 17% of principals below basic, 3.60,
resulting in 50% of the principals below proficient, and 4.0
for distinguished, resulting in 14.2% of principals designated
as distinguished.
Score Reports
Interpretation of Rating Scale Results
• Descriptive Analysis
▫ Total Score
▫ Core Components Subscale Scores
▫ Key Process Subscale Scores
• Norm-Referenced Profiles
▫ Principal
▫ Teacher
▫ Supervisor
▫ Total respondent composite
• Criterion-Referenced Profiles
▫ Distinguished
▫ Proficient
▫ Basic
▫ Below basic
Aggregated Effectiveness Ratings
Comparisons Across Respondent Groups
Comparisons Across Respondent Groups
VAL-ED and Professional
Development
Cell-by-Cell Feedback
VAL-ED and Professional Growth
• Cell-by-cell feedback highlights up to 6 potential
areas of growth.
• Behaviors from these 6 domains are listed.
• Areas of growth provide principals with
information about key targets for professional
development.
Leadership Behaviors for Possible
Improvement
Example of a potential area of growth: