Discussion Materials:
Educator Effectiveness and the Measures of
Student Learning (MSL/Common Exams)
Superintendents’ Quarterly Meeting
July 25, 2013
Federal Definition of Student Growth:
Student Growth: “Student growth” is the change in student achievement for an
individual student between two or more points in time. For the purpose of
this definition, student achievement means—
·
For grades and subjects in which assessments are required under
ESEA section 1111(b)(3): (1) a student’s score on such assessments
and may include (2) other measures of student learning, such as those
described in the second bullet, provided they are rigorous and
comparable across schools within an LEA.
·
For grades and subjects in which assessments are not required under
ESEA section 1111(b)(3): alternative measures of student learning and
performance such as student results on pre-tests, end-of-course
tests, and objective performance-based assessments; student learning
objectives; student performance on English language proficiency
assessments; and other measures of student achievement that are
rigorous and comparable across schools within an LEA.
The United States Department of Education has used this definition in Race to
the Top and the waivers from the Elementary and Secondary Education Act.
The Department of Education has also provided the following answer in
response to a question about the original ESEA waiver process.
May an SEA request only a portion of this flexibility [waiver from ESEA]?
No. An SEA may not request a portion of this flexibility or implement only
some of its principles. This flexibility is intended to build on and support the
significant State and local reform efforts already underway in critical areas such
as transitioning to college- and career-ready standards and aligned highquality assessments; developing systems of differentiated recognition,
accountability, and support; and evaluating and supporting teacher and
principal effectiveness. Only through such comprehensive efforts can SEAs and
LEAs truly increase the quality of instruction and improve student academic
achievement.
NCDPI/Superintendents’ Quarterly
Page 2 of 18
Notice of Flexibility on State Waivers from Elementary and Secondary
Education Act (ESEA)
Letter to Chief State School Officers on Graduation Rate Data and Race to the Top -- Printable
7/2/13 3:37 PM
Close
Window
Print
Key Policy Letters from the Education Secretary and Deputy Secretary
June 18, 2013
June 18, 2013
RELATED DOCUMENTS
Dear Chief State School Officers:
Fact sheet
Over the last four years, state and local leaders and educators across America
have embraced an enormous set of urgent and long-overdue challenges: raising
standards and upgrading curricula to better prepare students to compete in the
global economy, developing new assessments, rebuilding accountability systems
to meet the unique needs of each state and better serve at-risk students, and
adopting new systems of support and evaluation for teachers and principals.
Meeting this historic set of challenges all at once asks more of everybody
throughout the education sector, and it is a tribute to the quality of educators,
leaders, and elected officials across this country that so many have stepped up.
Homeroom Blog: New
flexibility for states
implementing fastmoving reforms (June 18,
2013)
State-by-State Timeline
Implementation Chart
Throughout this process, states and districts have established high goals for themselves: college- and career-ready
standards for all; higher graduation rates and college enrollment rates; high expectations for critical thinking,
problem-solving, and other 21st century skills; ambitious and achievable performance targets that really move the
needle for kids at risk; and useful, rigorous systems of evaluation and support for teachers and principals based on
multiple measures, including student growth. The Department has offered flexibility to enable states and districts
to meet these goals. In a country as diverse as ours, where schools and students have different educational
challenges, one-size-fits-all solutions have not worked. We have also aligned our grant programs to support states
willing to lead this important work, and the result is that some states are further along than others, but all states
are engaged in significant improvement efforts and students are better off for it.
In recent months, we have heard from many of you and from thousands of teachers, principals, and education
advocates. While there is a broad sense that these far-reaching changes carry enormous promise for schools,
children, and the future of our country, there is caution that too much change all at once could undermine our
collective progress. I fully appreciate both the courage to tackle so many challenges at once and the burdens this
imposes on front-line educators – teachers, principals, school boards, and administrators – who are committed
to doing this work well.
With that in mind, the Department is open to additional flexibility for states in two critical areas: the first relates to
one particular element of teacher and leader evaluation and support system implementation, and the second
addresses "double-testing" during the transition to new assessments aligned with college- and career-ready
standards.
First, I want to address the implementation of teacher and leader evaluation and support systems. States that
have received a Race to the Top grant or flexibility under the Elementary and Secondary Education Act (ESEA) are
responsible for working with districts to develop systems to evaluate and support principals and teachers based on
multiple measures, including student growth. States have committed to different deadlines to implement these
systems: some are implementing now; others will begin over the coming years. Given the move to college- and
career-ready standards, the dramatic changes in curricula that teachers and principals are now starting to teach,
and the transition to new assessments aligned to those standards, the Department will consider, on a state-bystate basis, allowing states up to one additional year before using their new evaluation systems to inform
personnel determinations. To be specific, states that request and are given this flexibility may delay any personnel
consequences, tied in part to the use of student growth data, until no later than 2016–2017. We recognize that,
http://www2.ed.gov/print/policy/elsec/guid/secletter/130618.html
NCDPI/Superintendents’ Quarterly
Page 1 of 2
Page 3 of 18
Letter to Chief State School Officers on Graduation Rate Data and Race to the Top -- Printable
7/2/13 3:37 PM
for many states, it will not make sense to request this flexibility because they are already well ahead in
successfully implementing these changes or have requirements in state law.
States interested in this extension may request this change, before September 30, 2013, through the current ESEA
flexibility amendment process. Details about the amendment process are available on the ESEA flexibility Web
page (http://www2.ed.gov/esea-flexibility). As each state implements college- and career-ready standards, it must
have a robust plan for supporting teachers and principals as they transition to the new standards and assessments.
States will need to lay out those plans in detail in the ESEA flexibility renewal process, along with indicators of
teacher and principal familiarity and comfort with these new materials.
The second issue I want to address is that of "double-testing" during the transition from the current statewide
assessments to new assessments aligned with college- and career-ready standards. During the 2013–2014
school year, some schools will be involved in the important work of field testing new assessments. We want to
support states that would like to avoid double-testing students, which as you know often happens during the shift
to a new test. Therefore, we would consider requests from states for a one-year waiver, to allow schools
participating in these field tests to administer only one assessment in 2013–2014 to any individual student —
either the current statewide assessment or the field test. We would also consider a request for those schools to
retain their Federal accountability designations for an additional year during which the same targeted interventions
would have to continue, with no relaxation of accountability requirements. Details about the Title I waiver process
are available at http://www.ed.gov/titlei-waiver.
Our country continues to face challenges as we work together toward achieving educational excellence for all
children, and the timing of these actions has real consequences for students in the real world. The point of raising
standards is to prepare students for tomorrow's challenges rather than yesterday's. Their readiness has real
consequences for their lives, and the nation's economic health. Yet this effort will only succeed if all parties have
the time, resources, and support needed to make the journey from the inadequate standards of the past to the
ambitious standards of tomorrow. As the highest-ranking education official in your states, you define the path and
the pace for how states and schools will make that journey. Our job in Washington is to support you. In the
coming days, the Department will provide more information on the flexibility discussed above; my staff will reach
out to you and your teams to provide assistance.
On behalf of the Obama Administration, I deeply appreciate your leadership and courage. I also appreciate your
honest feedback and the feedback of your principals and teachers. Above all, I salute your continuing
determination to advance reforms that will benefit millions of students in states and across America. This is hard
work and the need for change is urgent.
Sincerely,
/s/
Arne Duncan
cc:
Council of Chief State School Officers
Close
Window
Print
Last Modified: 06/20/2013
http://www2.ed.gov/print/policy/elsec/guid/secletter/130618.html
NCDPI/Superintendents’ Quarterly
Page 2 of 2
Page 4 of 18
Options for Moving Forward with Educator Effectiveness Model
North Carolina’s Race to the Top contract and waiver from the Elementary and
Secondary Education Act require the State to measure student growth and
include it explicitly in the evaluation of all teachers.
To meet this requirement, North Carolina has several options:
1.
Continue with each school district using the Measures of Student
Learning (MSL/Common Exams) and collaborate to improve the process.
Possible steps for improvement include:
• State prints the Common Exams and ships to school districts
• State includes greater specificity in Scoring Rubrics (if constructed
response items remain)
• State removes constructed-response items
• State requires that Common Exams replace teacher-made final
exams
• State requires that Common Exams count a defined percentage of
student’s grade statewide
• Continue with research on how to assess Arts Education, Healthful
Living, and World Languages
2.
Allow each school district to elect to develop its own system of measuring
student growth according to the federal definition of student growth.
Each district’s process for determining validity and reliability would
require approval by the State Board of Education, local board of education
chair, local superintendent, and district teacher representative.
3.
Allow each school district to elect to use school-wide growth from EOC
and EOG assessments to populate the sixth standard for all teachers in
non-tested areas. Decision would require approval by local board chair,
local superintendent, and district teacher representative.
4.
Allow each school district to elect to combine Options 2 and 3. Decision
would require approval by local board chair, local superintendent, and
district teacher representative.
5.
Other
NCDPI/Superintendents’ Quarterly
Page 5 of 18
Letter on Measures of Student Learning from North Carolina Teacher
of the Year, Karyn Dickerson(*)
(*) All text emphasis is from Ms. Dickerson’s original letter.
Measures of Student Learning/Common Exams: A Teacher’s Perspective
At the end of the last school year, Common Exams were criticized by several
different parties. I would like to offer some opinions concerning Common
Exams and their implementation that might help put the criticisms into
perspective and offer some suggestions for improvement.
The Format of the Common Exams
First of all, I want to state that I thought the Common Exams were good
measures of student learning. I have been frustrated since I started teaching at
the discrepancies within schools, school districts, and the state in the level of
rigor addressed in teacher-made final exams. Students taking the same class
with different teachers could take very different exams, and the grades
students received on the exams and in the classes varied widely. Furthermore,
some teachers opted out of creating and administering exams that included a
constructed response. Even some of my fellow English teachers did not include
written portions on their exams in the past because they feared it would be too
time consuming to read all of the essays. I often questioned how we were ever
going to implement true writing across the curriculum if even English teachers
were not including written portions on their exams.
The new Common Exams allow for more consistent rigor on final exams
and require that all subject areas include a constructed response portion. I
believe that constructed responses not only help improve the writing skills
of our students, skills that many colleges have stated students are
woefully lacking, but also move our students towards more twenty-first
century ways of thinking. In this day and age, the acquisition of facts is not
as important as the application of those facts. Constructed responses require
students to develop their higher order thinking skills and are essential to help
teachers understand what concepts need to be taught in a clearer, more indepth way. The new Common Exams helped address all of these gaps in
previous teacher-made assessments.
The Common Exams also encouraged important collaboration among
teachers as they had to score students’ responses and discuss ways to
improve the results for their students in the future. I found that during the
school year, teachers worked together much better in preparation for the
exams than they have in previous years. In my experience, it was often only the
teachers of EOCs that collaborated on a regular basis to share best practices,
NCDPI/Superintendents’ Quarterly
Page 6 of 18
but this year with the implementation of the Common Exams, I saw much more
collaboration throughout my school and in different content areas. Now that
teachers know what the exams look like, I foresee even more collaboration next
year.
Concerns about Timing of the Common Exams
In spite of many of the strengths of the format of the exams, the rushed
implementation of the exams created a great deal of stress for many teachers.
I believe that this is the primary reason why teachers complained about the
exams. Teachers have experienced many changes over the past several years.
There have been new systems to learn (NCWise, PowerSchool, McRel
Evaluations, PEP online software in some districts for student intervention,
Home Base) as well as significant changes to content in the Common Core and
Essential Standards. I think most teachers would state that they support the
new standards, but the number of changes in such a short span of time has
exhausted many teachers. They are putting in numerous hours after school
and over the summer learning new standards, creating new lesson plans, and
reworking their assessments. They are also spending a great deal of time
trying to communicate these changes to parents and other educational
stakeholders. For many school districts, last year was the first year that the
Common Core and Essential Standards were fully adopted. To expect teachers
to master the new standards, find the best ways to teach their students to
address the new standards, implement more technology than ever, and
complete all of the above with few corresponding resources led to many
teachers feeling overwhelmed. Numerous hours were spent trying to find
resources to address the standards, and many teachers spent more money out
of their pockets than ever before.
Due to all of the significant changes in education over the past few years, I
think DPI’s suggestion to count the 2013 Common Exams as pilot tests
would help with teacher morale in the state. Teachers want to do well and
want their students to do well. The lack of time to truly master the new content
before administering a new exam caused a great deal of frustration among
teachers. The morale of my fellow coworkers was the lowest I have ever seen
during exams and in the month leading up to them. Adding to the stress for
teachers was the last-minute notification of testing plans and policies within
school districts.
Problems with Flexibility among School Districts
With the flexibility in individual school districts and in individual schools with
how to administer the Common Exams, there was even more undue stress
placed on teachers. As we heard in the presentation at the State Board of
NCDPI/Superintendents’ Quarterly
Page 7 of 18
Education meeting in July, some students had to take two final exams (both the
Common Exam and a teacher-created exam)! That is way too much testing and
grading at the end of the school year. There was also a wide range concerning
when to administer the tests and how they would count for students. Because
of this flexibility, some school districts and individual schools did not have a
clear plan for testing until almost a month prior to the start of testing. The lack
of a set policy until the end of the school year negatively impacted the attitudes
of students, parents, and teachers towards the exams. At my school, we
administered Common Exams the week before End of Course Exams. That
meant that towards the end of the school year, teachers suddenly lost a week of
instruction that they had planned on all year. For teachers like myself who
spend time creating long-term plans and unit plans, this made the end of the
year extremely stressful as I had to make significant changes to my plans at the
last minute.
Additionally, I had some concerns with some of the districts’ implementation of
the constructed response scoring rubrics. Teachers in my school district were
informed about three weeks prior to administering the Common Exams that in
all exams except for English the constructed responses did not have to be
written in paragraphs or even in complete sentences! Some teachers
encouraged students to respond in bullet format on their exams, including
social studies exams. This was surprising to me since history assessments at
the college level require written responses in complete sentences. It seemed
like the discrepancies in how to administer and score the exams defeated the
initial purpose of the exam: to assess twenty-first century skills and the
acquisition and application of important content. We need much more
consistency across school districts in order for the Common Exams to be
as effective as they have the potential to be in assessing the growth of our
students and teacher affect.
Conclusion
In short, the Common Exams have a great deal of potential in assessing both
the gains students have made during the course of a year and the efficacy of
teacher instruction. I believe that these exams with a few modifications and
increased consistency have the potential to serve as valuable forms of
assessment that offer insightful feedback. If used effectively, they will prove to
be much more than just another set of tests.
NCDPI/Superintendents’ Quarterly
Page 8 of 18
Quality Assurance Checks for the Common Exams
During the 2012-13 school year, hundreds of thousands of students took the
Common Exams for the first time. The design of the Common Exams began in
2011 with a gathering of 800 teachers who provided feedback on how to assess
their content standards.
A set of those 800 teachers reviewed and approved the items that were
generated as a result of their feedback. Thirty-nine school districts partnered
with the Department of Public Instruction to administer the Common Exams
during first semester. Their feedback informed decisions to the exams before
second semester administration.
The Department of Public Instruction has committed to rigorous internal and
external quality assurance checks prior to the use of any Common Exam data
for value-added analysis. The results from a Common Exam that does not pass
these checks will not be used to generate teacher value-added scores.
Internal Review by the Department of Public Instruction:
ü Is the Common Exam reliable? Reliability describes the consistency or
stability of student scores if a student were to take the exam multiple
times.
ü Is the Common Exam aligned with the content standards? The
Department of Public Instruction has already completed this step.
Multiple individuals have reviewed each Common Exam to ensure
alignment with the content standards:
§ Department of Public Instruction Test Measurement
Specialists
§ Department of Public Instruction Curriculum Specialists
§ NC State University Content Specialists
§ NC State University Specialists on Assessment for English
Language Learners
§ NC State University Specialists on Assessment for Exceptional
Children
ü Does the Common Exam produce a desirable distribution of scores with a
good spread across most of the scale? If the distribution is extremely
negatively skewed or positively skewed, the test may not be useful for
measuring growth of low ability or high ability students. Score
distribution is a range, not clustered at one point on the scale.
ü Does the Common Exam have any items that need to be removed, for
example, an item that does not meet measurement of content criteria? If
so, the Department removes the item from the test and any data analysis.
NCDPI/Superintendents’ Quarterly
Page 9 of 18
Only Common Exams that pass these quality assurance checks move on to
the next stage of review.
External Review by the EVAAS Team at the SAS Institute:
ü Did the Common Exam “pass” the quality assurance checks completed by
the Department of Public Instruction?
ü Can the Common Exam measure the achievement of both high- and lowachieving students? This idea is referred to as the “stretch” of an
assessment.
ü Did enough students take the Common Exam? Value-added analysis is
not possible when a very small group of students have exam results.
ü Did enough teachers teach the course associated with the Common
Exam? Value-added analysis is not possible when a very small number of
teachers have taught the course.
ü Can students’ prior test scores be used to predict students’ scores on the
Common Exam? For example, when the Department of Public Instruction
and SAS Institute explored value-added modeling for Career and
Technical Education Post-Assessments, the scores for some PostAssessments could not be predicted. For others, a prediction model was
possible.
ü Are there sufficient differences among teacher value-added scores based
on the Common Exam? In other words, do the results of the Common
Exam provide measurable differences among teachers?
When a Common Exam passes both the internal and external review, results will
be used to inform a teacher’s value-added score and sixth standard rating in
the NC Educator Evaluation System.
NCDPI/Superintendents’ Quarterly
Page 10 of 18
Discussion Questions on Improvement of Common Exams
Administration
As part of the teacher evaluation process, the Common Exams were
administered for the first time in 2012–13 to North Carolina elementary,
middle, and high school students. Throughout the school year, the NCDPI
gathered feedback from teachers, students, parents, principals, and
superintendents on the Common Exams and their administration procedures.
Based on this feedback, the NCDPI is determining what changes may need to
occur to the Common Exams for going forward.
Listed below are the key items under discussion for change.
1. Currently the Common Exams are designed to be administered during a
class period or during exam week, and the LEA determines the testing
window. Should the testing window be restricted?
2. Currently the Common Exams are timed tests, and the testing time is 90
minutes for all Common Exams (elementary, middle, and high school).
Schools have the option to administer the test in one school day (i.e., one
90 minute testing session) or in two school days (i.e., one 45 minute
testing session on day one and one 45 minute testing session on day
two). The 90 minutes does not include time for general instructions and a
break. Should we continue with the 90-minute testing time?
3. In 2012–13, LEA test coordinators were responsible for downloading the
appropriate Common Exams and all associated ancillary materials and for
printing and distributing enough copies of these materials for their
schools. LEAs were also responsible for purchasing the answer sheets for
the exams. Should the State print the exams, ancillary materials, and
answer sheets, or should the LEA continue to print the common
exams and purchase the answer sheets?
4. Some LEAs gave a final exam in addition to the Common Exam. Should
the Common Exams, where available, replace local final exams?
5. For 2012–13, LEAs were asked to determine if or how results from the
Common Exams would be used in determining students’ final grades.
Should Common Exams carry the same weight for grading EOCs (i.e.,
25%), and should we change the EOC weight for grading to 15% and
count the Common Exams also as 15%of the final grade?
NCDPI/Superintendents’ Quarterly
Page 11 of 18
6. All Common Exams with the exception of four (i.e., OCS Applied Science,
OCS Financial Management, OCS Introductory Math, and U.S. History)
contained constructed response (CR) items. The number of CR items
varied by test from 2–4 questions. Teachers supported the he inclusion of
constructed response items, but the feedback has been both positive and
negative (see chart below). Should the number of CR items be reduced
or should they be removed from the exams for 2013–14?
Positives/Negatives on the Common Exam Constructed Response Items
Plus +
Delta -
Allows students to demonstrate what
they know and are able to do
Allows higher-level items
Provides more engaging items than
typical multiple-choice items
Provides teachers with professional
development (scoring process)
Makes the tests too long
Delays the turn-around of the scores
at the high school for inclusion in the
student’s final grade
Requires teachers to participate in a
scoring process, which some have
reported is too time consuming
Adds to inconsistency in grading
across schools/LEAs
NCDPI/Superintendents’ Quarterly
Page 12 of 18
NC School District Recommendation for Moving Forward with
NC Educator Effectiveness Model
District Name:
_____________________________________________________________________________
Superintendent Name (please print) & Signature:
_____________________________________________________________________________
Please place a check mark next to your recommendation for moving forward
with the educator effectiveness model.
_____ 1. Continue with each school district using the Measures of Student
Learning (MSL/Common Exams) and collaborate to improve the process.
Possible steps for improvement include:
• State prints the Common Exams and ships to school districts
• State includes greater specificity in Scoring Rubrics (if constructed
response items remain)
• State uses only multiple-choice items
• State requires that Common Exams replace teacher-made final
exams
• State requires that Common Exams count a defined percentage of
student’s grade statewide
• Continue with research on how to assess Arts Education, Healthful
Living, and World Languages
_____ 2. Allow each school district to elect to develop its own system of
measuring student growth according to the federal definition of student
growth. Each district’s process for determining validity and reliability
would require approval by the State Board of Education, local board of
education chair, local superintendent, and district teacher representative.
_____ 3. Allow each school district to elect to use school-wide growth from
EOC and EOG assessments to populate the sixth standard for all teachers
in non-tested areas. Decision would require approval by local board
chair, local superintendent, and district teacher representative.
_____ 4. Allow each school district to elect to combine Options 2 and 3.
Decision would require approval by local board chair, local
superintendent, and district teacher representative.
Comments:
Please submit this form to the DPI staff member in your discussion group.
NCDPI/Superintendents’ Quarterly
Page 13 of 18
North Carolina’s Approval for ESEA Flexibility
NCDPI/Superintendents’ Quarterly
Page 14 of 18
NCDPI/Superintendents’ Quarterly
Page 15 of 18
NCDPI/Superintendents’ Quarterly
Page 16 of 18
NCDPI/Superintendents’ Quarterly
Page 17 of 18
NCDPI/Superintendents’ Quarterly
Page 18 of 18