Douglas C. Orzolek
University of St. Thomas
St. Paul, Minnesota
Developing Evaluation Tools for Music Education
Session Outline

External Factors

Internal Factors

Administrators/Supervisors Literature

On Developing an Evaluation Tool

The Teacher Performance Assessment

An Accountability Report

What Next?
Developing Evaluation Tools for Music Education
External Factors

Attitude (Peck) *

Need for Composite Rating / “% Problem” (Burling) *

Inconsistency Policy from State to State, District to District

VAMs (Darling-Hammond, 2012)

“Transparency or Accuracy?” (Prince, Friedman)

Evaluator training, understanding of music, criteria don’t apply

$ to develop a Proper Assessment

Reliability and Validity

Legal Issues and Policy Validity (Prince)
Developing Evaluation Tools for Music Education
Evaluation Components for
Teachers in Non-tested Subjects or
Grades – D.C. IMPACT Program
(Steele, 2010)

75% Administrator or
Master Educator
Observation

10% Student Growth on a
Teacher-Chosen Measure

10% Commitment to the
School Community

5% School-wide ValueAdded
Developing Evaluation Tools for Music Education
Internal Factors

How Our Work is Percieved (Orzolek) *

Subjective/Affective (Maranzano, Barrett) *

Student Achievement? Connected to Learning? (Bowman)

One Size Does Not Fit All - Multifaceted Jobs – Create, Respond,
Perform; Classroom, Ensemble, Individual (Grant, Taebel)

Criteria for Excellent Music Teaching? (Brophy, Taebel);
“Successful” Teaching versus “Good” Teaching (Barrett)

More Research – Connect to Other Non-tested Areas?

The “My Program” Problem – Evaluate me on Program Success

“If we say that music kids are smart, then why are we afraid of
VAMs? Wouldn’t that prove our point?” (Sean Barker, UST ‘12)
Developing Evaluation Tools for Music Education
MSBA Survey: Accountability in Music (Orzolek, 2004)
No Complaints
“Good” Concerts
Successful Trips
No Letters to the Editor
Trophies
Athletic Bands
Kids Seem Happy
C&I Person is Happy
A Local “Expert”
Mostly at Budget Time
“What else is there?
Developing Evaluation Tools for Music Education
Developing Evaluation Tools for Music Education
Reminders from Music Leaders Literature

Clear Goals and Objectives – Cowden and
Klotman (1991), 97

“Evaluation ought to involve looking critically at
the process of gathering and weighing evidence
that will reveal changes in terms of a desirbable
musical product.” – Weyland (1960), 295

Very Subjective – Snyder (1959), 114

Multifaceted – Dykema and Gehrkens (1941), 368

“Seek competent evaluation of your work.” –
Prescott (1938), 259
Developing Evaluation Tools for Music Education
On Developing Evaluation Tools – External
Burling (2012) (aka Pearson Corporation)

Define a construct – What is an effective educator?

Deploy multiple indicators – What evidence characterizes good teaching
and school leaderhip?

Develop a clear composite rating – What weights should each indicator
have and who should be involved in the decision?

Clarify differentiated performance levels – What distinguishes varying
levels of educator effectiveness?

Build strong data analysis and reporting tools – What does the
information reveal about student, educator, and school performance?

Improve Instructional Leadership and Practice – How can the information
target professional development to boost educator practice, student
learning outcomes, and school efficacy?
Developing Evaluation Tools for Music Education
On Developing Evaluation Tools – External
Others

Promote consistency in any student measures used to assess
teachers (Steele)

Implement tools after experimentation (Steele)

Should supply teachers with feedback – “close the loop”
(Prince)

Teachers should determine the extent to which math/reading
scores are included (Prince)

Portfolio (Prince)

Always use tools that are valid and reliable – Charlotte
Danielson. (Peck)
Developing Evaluation Tools for Music Education
On Developing Evaluation Tools - Internal
Brophy (1993)

Personal characteristics, as evidenced through
student-teacher, teacher-faculty, and teacher
administration interactions;

Music competence and performance skill, as
evidenced when teaching a lesson;

Effective use of nonverbal strategies, such as
modeling/demonstrating;

Effective use of verbal strategies, such as eliciting
performance from students;
Developing Evaluation Tools for Music Education
On Developing Evaluation Tools - Internal
Brophy (1993)

Classroom management, including discipline,
group participation, and the creation of a
positive learning environment;

Effective planning for concept learning and
aesthetic appreciation across a wide age span;

An objective assessment of teaching style based
on empirically supported criteria;

Appropriate professional development activities
should be recommended.
Developing Evaluation Tools for Music Education
Three Categories of Teacher Evaluation
NAfME (2011)

Teacher Evaluation linked exclusively to
student outcomes.

Teacher evaluations linked exclusively to
teacher practice.

Teacher evaluations that combine
measures of student outcomes and
teacher practice.
Developing Evaluation Tools for Music Education
Question
How do we help prepare
teachers for these evaluations?
Developing Evaluation Tools for Music Education
TPA Background



Teacher Performance Assessment – Stanford,
AACTE, CCSSO, Pearson, MACTE. 21 states
actively involved.
Assesses “Effectiveness” of Pre-service Educators
The entire assessment is outlined in a “TPA
Handbook” – “The Performing Arts”.

Completed during Student Teaching

Scored by Trained Scorers via E-folio
Developing Evaluation Tools for Music Education
How does the assessment work?
“An Authentic Assessment of Teaching Competence”

Students complete 4 Tasks:





Describe their Plans for Instruction & Assessment;
Engage Students in Learning;
Assess Student Learning;
Analyze Teaching.
Parameters





“Central Focus of Learning Segment”
3-5 Lessons
Evidence provided on Video
Use of “Academic Language”
Detailed Rubrics Provided
Developing Evaluation Tools for Music Education
Starting Point:
A Voluntary
Accountability
Report
2002 - 2003 Accountability Report
Instrumental Music Education Department - George Washington High School
Contact: Douglas C. Orzolek (dcorzolek@stthomas.edu)
Mission Statement : High Quality, Comprehensive Music Education
Goals/Results for 2003-2004

Increase number of national standards completed / Implemented
composition unit this year

Improve student rankings in state portfolio assessments / See
below

Enhance music theory understanding by adapting new software /
New software purchased and implemented

Provide outstanding performance opportunities for students /
Performances were well received - CD attached!
National Standards
This year we added a composition (Standard 4)
unit to our instruction. Several guests spoke of
their experiences as composers, we took a field
trip to the American Composer’s Forum and
wrote our own pieces that were performed
during our recital at the end of the year.
State/District Standards
11th grade portfolios were sent to our
neighboring schools for evaluation and
assessment by music educators in those
districts. The following are the results based
upon established rubrics:

4 (Exemplary) - 15% (10% in 2002)

3 (Proficient) - 45% (40% in 2002)

2 (Developing) - 35% (45% in 2002)

1 (Poor/Incomplete) - 5% (5% in 2002)
The results suggest that our students are
performing well in these assessments of their
work. This year’s scores show an
improvement from the previous year.
Comments from Professional Reviews
“Strongest performing group I have heard in years.” - Leonard
Bernstein
Comments from Student/Peer Evaluations

“I feel like I have a better understanding of my instrument
after I’ve studied a composer and her intentions.”

“”Billy’s perceptions of the performance were much different
than mine. But, I appreciated his points - they reminded me
that music can be something different for everyone.”

“Opera - I still don’t like it.”
Testing Results for 2003-2004

9th Graders completed the NAEP music skills written portion
and scored in the 72 percentile nationally.

Music reading skills measured by the Watkins-Farnum
performance scale continue to improve. The mean etude
completed by students in grades 9-12 was #6 in 2002 and
rose to #8 in 2003.

Average SAT score of a student in band was 1425 (of 1600.)
Other Highlights

97% of our students participated in the MN solo/ensemble
festival this year. 35% received Superior and 40% received
Excellent ratings.

Our Concert Band earned the top ranking of Superior at the
large group festival.

Our Marching Band was selected to perform in the Rose
Parade.

We commissioned a new work from composer Libby Larsen.

6 band members were selected to perform with the All-State
band.

Our students gave 60 public performances during this past
year.

80% of senior band members are planning to attend a 4 year
college.
Developing Evaluation Tools for Music Education
What Next?

Top of NAfME Agenda – Policy Analysis,
Leadership

Stay Informed – Websites, Readings

Share Stories – Forum, Meetings

Coordinate and Advance Research Efforts - Grants

The Reality - Multifaceted and Messy – Experiment.

This is about People not Composite Ratings.

Until we agree on what constitutes effectiveness, it
will be difficult to measure and reward it.
Developing Evaluation Tools for Music Education
Darling-Hammond & Bransford (2005)
Effective Teachers:

Understand subject matter deeply and flexibly;

Connect what is to be learned to students’ prior knowledge and
experience;

Create effective scaffolds and supports for learning’

Use instructional strategies that help students draw connections, apply
what they’re learning, practice new skills, and monitor their own learning;

Assess student learning continuously and adapt teaching to student
needs;

Provide clear standards, constant feedback, and opportunities for revising;

Develop and effectively manage a collaborative classroom in which all
students have membership.
Developing Evaluation Tools for Music Education
“When a call for accountability
elicits discussions about
education, the results produced
are usually positive.”
R. Colwell, (2003) “The Status of Arts Assessment:
Examples from Music”, Arts Education Policy Review,
105 (2), 16.