Assessing Learning-Centered Leadership
Andrew C. Porter
University of Pennsylvania
Joseph Murphy, Ellen Goldring, & Stephen N. Elliott
Vanderbilt University
Spring 2008
Acknowledgments


The VAL-ED instrument will be published and distributed
by Discovery Education Assessment starting in July
2008.
We are authors of the VAL-Ed, and while we have made
every effort to be objective and data-based in my
statements about this instrument in this presentation,
readers should judge the facts and related information
materials for themselves and make independent
decisions regarding the use of the instrument.
2008
Outline of Presentation

Background on VAL-ED

The VAL-ED instrument

Psychometric properties

Score reports

VAL-ED and professional development
2008
Background on VAL-ED
The Case for Leadership Assessment



Most school leadership improvement focuses on
professional development, mentoring, licensing policies,
and standards.
Minimal attention has been paid to assessment,
feedback, and subsequent action.
Leadership assessment and feedback is an important
missing link to improving and strengthening school
leadership.
2008
Learning-Centered Leadership



Leaders should be assessed on leadership behaviors
associated with student learning.
Learning-centered leadership is leadership for student
performance.
Learning-centered leadership is the framework for our
leadership assessment system.
2008
Our Conceptual Model
2008
Leadership Behavior Framework
2008
Definitions of Core Components

High Standards for Student Learning —There are individual, team,
and school goals for rigorous student academic and social learning.

Rigorous Curriculum (content) —There is ambitious academic content
provided to all students in core academic subjects.

Quality Instruction (pedagogy) —There are effective instructional practices
that maximize student academic and social learning.

Culture of Learning & Professional Behavior —There are integrated
communities of professional practice in the service of student academic and
social learning. There is a healthy school environment in which student learning
is the central focus.

Connections to External Communities —There are linkages to family and/or
other people and institutions in the community that advance academic and
social learning.

Performance Accountability — Leadership holds itself and others responsible
for realizing high standards of performance for student academic and social
learning. There is individual and collective responsibility among the professional
staff and students.
2008
Definitions of Key Processes

Planning—Articulate shared direction and coherent policies, practices, and
procedures for realizing high standards of student performance.

Implementing—Engage people, ideas, and resources to put into practice
the activities necessary to realize high standards for student performance.

Supporting—Create enabling conditions; secure and use the financial,
political, technological, and human resources necessary to promote
academic and social learning.

Advocating—Promotes the diverse needs of students within and beyond
the school.

Communicating—Develop, utilize, and maintain systems of exchange
among members of the school and with its external communities.

Monitoring—Systematically collect and analyze data to make judgments
that guide decisions and actions for continuous improvement.
2008
The Development of VAL-ED

The development of VAL-ED has been
supported by a 3-year grant from The Wallace
Foundation.

Three phases of our work:



Phase 1 – Leadership conceptualization and assessment system
development
Phase 2 – Field testing the behavior rating scale and exploring its
properties
Phase 3 – Dissemination of results and products
2008
The VAL-ED Instrument
The VAL-ED Instrument






The instrument consists of 72 items defining six core
component subscales and six key process subscales.
Principal, Teachers, & Supervisor provide a 360-degree,
evidenced-based assessment of leadership behaviors.
Respondents rate effectiveness of 72 behaviors on scale
1=Ineffective to 5=Outstandingly effective.
Each respondent rates the principal’s effectiveness after
indicating the sources of evidence on which the effectiveness
is rated.
Two parallel forms of the assessment facilitate measuring
growth over time.
The instrument will be available in both paper and online
versions.
2008
Purpose & Uses

The VAL-ED can be used as part of a comprehensive
assessment of the effectiveness of a leader's behaviors.

The VAL-ED reports principal performance through



Norm-referenced scores and
Criterion-reference scores.
VAL-ED can be used annually or more frequently to:



Facilitate a data-based performance evaluation,
Measure performance growth, and
Guide professional development.
2008
Implementation






Identify respondents and invite participation.
Discuss use of results & confidentiality.
Decide paper or online version.
Time and Timing
 Average respondent requires 20 to 25 minutes.
 Schedule completion after respondents have had a
reasonable time to observe/experience the leader’s
work and its effects on the school.
Designate person(s) to manage collection and
submission of response forms, if paper version used.
Ensure teacher confidentiality.
2008
Directions for Completing Rating Scale
2008
An Example Set of Responses
2008
Psychometric Properties
Assessing Learning-Centered Leadership:
The VAL-ED vision…
A leadership assessment system that has the following properties:









Works well in a variety of settings and circumstances,
Is construct valid,
Is reliable,
If feasible for widespread use,
Provides accurate and useful reporting of results.
Is unbiased,
Yields a diagnostic profile for summative and formative purposes.
Can be used to measure progress over time in the development of
leadership, and
Predicts important outcomes.
2008
Psychometric Evidence




Item and response scale development
 Based on review of learning-centered leadership literature and
alignment to ISLLC standards.
 Critiqued by education leaders and leadership researchers.
Item sorting study
 Established content validity by asking education leaders to sort
the items into 36 cells.
Cognitive interviews of paper/pencil version
 Two rounds of cognitive interviews in three districts each.
 Three respondents evaluated the format and items.
Nine-school pilot test
 Estimated reliability of each of 12 scales.
 Established construct validity through factor analysis.
 Established face validity through questions to respondents.
2008
Psychometric Evidence





Cognitive interviews of online instrument including revisions
based on 9-school pilot
Bias review
 Submitted to urban districts to evaluate language.
11-school pilot test
 Confirmed changes made after 9-school pilot test.
300-school field test
 Conducting differential item functioning to determine biases.
 Establishing norms.
 Setting performance standards.
Proficiency standard setting
 Using bookmark approach with 24 education leaders.
2008
Cognitive Interviews—Validity

First round




Modifications




Instructions bulleted
Stem included in each item
“Not done” added
Conclusions




Sources of evidence cumbersome
Item stem lost
Instructions “wordy”
VAL-ED captured all the relevant leadership behaviors
Response scale was clear
Respondents able to complete task on their own
Cognitive interviews of online prototype
2008
9-School Pilot Study—Validity & Reliability




Methods
Confirmatory factor analysis
 High Parsimonious Goodness of Fit Index (.93-.96)
 First, second, and third-order factor loadings salient
Responses to final survey questions
 Understood items, sources of evidence
Teacher and principal
Scatter of Principal ratings with Teacher Ratings
effectiveness ratings were
correlated r =.47
Cronbach’s Alpha above
.92 for 108-item-form scales
Scores high (generally >4
on the five-point effectiveness
scale)


Principal ratings by school
5
4.5
4
3.5
3
2.5
2
3.5
3.7
3.9
4.1
4.3
4.5
Teacher mean ratings by school
2008
4.7
r=.47
4.9
11-School Pilot Study—Validity & Reliability


Methods
High agreement between schools



r = .79 for teacher-principal,.51 for principalsupervisor, .68 for
supervisor-teacher
Mean effectiveness
ratings lower, more
variable
Alpha above .89 for
all 72-item-form
scales
2008
Respondent Feedback
2008
Score Reports
Interpretation of Rating Scale Results



Descriptive Analysis

Total Score

Core Components Subscale Scores

Key Process Subscale Scores
Norm-Referenced Profiles

Principal

Teacher

Supervisor

Total respondent composite
Criterion-Referenced Profiles

Distinguished

Proficient

Basic

Below basic
2008
VAL-ED Results: Performance Descriptors

Distinguished


Proficient


A proficient leader exhibits leadership behaviors of core components and key
processes at levels of effectiveness that over time are likely to influence teachers to
bring the school to a point that results in acceptable value-added to student
achievement and social learning for all students.
Basic


A distinguished leader exhibits leadership behaviors of core components and key
processes at levels of effectiveness that over time are virtually certain to influence
teachers to bring the school to a point that results in strong value-added to student
achievement and social learning for all students.
A leader at the basic level of proficiency exhibits leadership behaviors of core
components and key processes at levels of effectiveness that over time are likely to
influence teachers to bring the school to a point that results in acceptable value-added
to student achievement and social learning for some sub-groups of students, but not
all.
Below basic

A leader at the below basic level of proficiency exhibits leadership behaviors of core
components and key processes at levels of effectiveness that over time are unlikely to
influence teachers to bring the school to a point that results in acceptable value-added
to student achievement and social learning for students.
2008
Aggregated Effectiveness Ratings
2008
Comparisons Across Respondent Groups
2008
Comparisons Across Respondent Groups
2008
VAL-ED and Professional
Development
Cell-by-Cell Feedback
2008
VAL-ED and Professional Growth



Cell-by-cell feedback highlights up to 6
potential areas of growth.
Behaviors from these 6 domains are listed.
Areas of growth provide principals with
information about key targets for professional
development.
2008
Leadership Behaviors for Possible Improvement
Example of a potential area of growth:
2008
Supporting Research & Publications

Goldring, E., Porter, A.C., Murphy, J., Elliott, S.N., & Cravens, X. (2007, March).
Assessing learning-centered leadership: Connections to research, professional
standards, and current practice. New York, N.Y.: Wallace Foundation.

Murphy, J., Elliott, S.N., Goldring, E., & Porter, A.C. (2007). Leadership for
learning: A research-based model and taxonomy of behaviors. School
Leadership & Management, 27 (2), 179-201.

Murphy, J., Elliott, S.N., Goldring, E., & Porter, A.C. (in press). Leaders for
productive schools. In M. Brundrett & M. Crawford (Eds.), Developing school
leaders: An international perspective, London: Routledge.

Murphy, J., Elliott, S.N., Goldring, E.B., & Porter, A.C. (2006). Learning-centered
leadership: A conceptual foundation. New York, NY: Wallace Foundation.

Porter, A.C., Goldring, E.B., Murphy, J., Elliott, S.N., & Cravens, X. (2006). A
framework for the assessment of learning-centered leadership. New York, NY:
Wallace Foundation.
Murphy, J.F., Goldring, E.B., Cravens, X.C., Elliott, S.N., Porter, A.C. (2007,
August). The Vanderbilt Assessment of Leadership in Education: Measuring
Learning-Centered Leadership. Journal of East China Normal University.
These and other publications are all available for download at
http://www.vanderbilt.edu/lsi/valed/featured.html.

2008
To Learn More …
Visit:
http://www.val-ed.com
Contact :
Discovery Education/ThinkLink Assessment
2416 21st Avenue, South, Suite 300
Nashville, TN 37212
Jackie Shrago
(o) 615-324-6091
(e) Jackie_Shrago@discovery.com
or William Dycus
(o) 615-324-6090
(e) William_Dycus@discovery.com
2008