Measuring Principal Performance:
How Rigorous Are Publicly Available
Assessment Instruments?
Matthew Clifford
Learning Point Associates
matthew.clifford@learningpt.org
Presentation for PEAC
July 2010
Copyright © 2010 Learning Point Associates. All rights reserved.
Learning Point Associates Overview
www.learningpt.org
www.learningpt.org/expertise/educatorquality/METworks.php
www.learningpt.org
What is Your Perspective on
Principal Performance Assessment?
•
What instruments are you or others currently using?
•
How systematically are the instruments being administered?
•
How are results being used?
•
How confident are you in the assessment instruments and process?
www.learningpt.org
What Research Tells Us about
Principal Performance Assessment
• School districts’ principal performance assessments appear to be
inconsistently administered and measured (Goldring et al, 2009;
Thomas et al, 2000; Murphy et al, 2007).
• Principal performance assessments are not always aligned with
existing state or national professional standards for practice, and
lack justification or documentation of psychometric research (Heck
& Marcoulides, 1996).
www.learningpt.org
Learning Point Associates:
Managing Educator Talent
Performance
Management
Coherent
Systematic
www.learningpt.org/expertise/educatorquality/METworks.php
www.learningpt.org
Principal Performance Assessment:
Recommended Attributes
•
Transparent
•
Trusted
•
Consequential
•
Systematic and consistent
•
Involving multiple measures
•
Validate and reliable
(Brown-Sims, 2010)
www.learningpt.org
Our Study: Question & Method
Question:
How valid and reliable are publicly available
principal performance assessment instruments?
www.learningpt.org
Our Study: Review Criteria and Results
Criteria for inclusion:
• Claim of use as a principal performance assessment
• Recent (within past 15 years or so)
• Publicly available technical information
Results of scan:
• 2000 articles identified by keyword search & snowball
sampling
• Approx. 400 examined
• 20 articles reviewed on 8 assessment instruments
www.learningpt.org
Our Study: Instruments Reviewed
Instrument
Change Facilitator Style
Questionnaire
(1988)
Approach
77-item survey focusing
on 6 domains
Validity
Content: Lit review
Reliability
Alpha Range:
.64 to .95
Construct: Factor analysis
360-degree of principal as
change facilitator
Diagnostic Assessment
of School and Principal
Effectiveness
(1992)
213-item survey
Content: Expert review
360-degree
Construct: Factor analysis
and inter-item correlation
Instructional activity
questionnaire
(1987)
34-item assessment
Content: Lit review
Focus on instructional
leadership
Construct: Factor analysis
Alpha Range:
.8 to .97
Alpha Range: .7 to .9
www.learningpt.org
Our Study: Instruments Reviewed
Instrument
Approach
Validity
Instructional activity
questionnaire
(1987)
34-item assessment
Content: Lit review
Focus on instructional
leadership
Construct: Factor analysis
Leadership Practices
Inventory
(2002)
30-items
Content: Interviews and
surveys of leaders
Principal and supervisor
completes
Leadership effectiveness
Performance Review
Analysis and
Improvement System
for Education (1985)
Reliability
Alpha Range: .7 to .9
Test/re-test: .79
Construct: Factor analysis
Concurrent: Other
management measures
81-item with 9 sub-scales Content: Lit review with
Delphi panel
Two-dimensional profile
with +/Construct: Unknown
Alpha Range: .88 to .98
Test/re-test: .59 to .8
www.learningpt.org
Our Study: Instruments Reviewed
Instrument
Approach
Validity
Principal Instructional
Management Rating
Scale
(1985)
71-item questionnaire
with 11 subscales
Principal Profile
(1986, 1987)
Interview-based
protocol characterizes
principal effectiveness
and leadership style
Content: Lit review
72-items
Content: Lit review
360-degree, produces
profile
Construct: Factor analysis
VAL-ED
(2006)
Content: Lit review and
Delphi panel
Reliability
Alpha: .75
Construct: Correlation within
subscale and school document
review
Inter-rater agreement
range from .5 to 1
Construct: Factor analysis
Alpha= .98 for 12 scales
on different forms
www.learningpt.org
Our Study: What did we learn?
•
If there are many principal performance assessment being
used in the field, then evidence of their reliability and
validity is not publicly available.
•
Of those reviewed, 5 of 8 were developed in tested in the
mid-to-late1980s.
•
Survey-based assessment gather self, and others (3),
perceptions of principal performance. One used a trained
rater.
•
No information on consequential validity, and two examples
of concurrent validity.
www.learningpt.org
Matt Clifford
E-Mail: matthew.clifford@learningpt.org
Phone: 630-689-8017
1120 East Diehl Road, Suite 200
Naperville, IL 60563-1486
General Information: 800-356-2735
13
www.learningpt.org