!
!
!
!
!
!
!
!
!
!
!
!
!
!
Discussion Materials:
Educator Effectiveness and the Measures of
Student Learning (MSL/Common Exams)
Superintendents’ Quarterly Meeting
July 25, 2013
!
1!
!
Federal Definition of Student Growth:
Student Growth: “Student growth” is the change in student achievement for an
individual student between two or more points in time. For the purpose of
this definition, student achievement means—
·
For grades and subjects in which assessments are required under
ESEA section 1111(b)(3): (1) a student’s score on such assessments
and may include (2) other measures of student learning, such as those
described in the second bullet, provided they are rigorous and
comparable across schools within an LEA.
·
For grades and subjects in which assessments are not required under
ESEA section 1111(b)(3): alternative measures of student learning and
performance such as student results on pre-tests, end-of-course
tests, and objective performance-based assessments; student learning
objectives; student performance on English language proficiency
assessments; and other measures of student achievement that are
rigorous and comparable across schools within an LEA.
The United States Department of Education has used this definition in Race to
the Top and the waivers from the Elementary and Secondary Education Act.
The Department of Education has also provided the following answer in
response to a question about the original ESEA waiver process.
May an SEA request only a portion of this flexibility [waiver from ESEA]?
No. An SEA may not request a portion of this flexibility or implement only
some of its principles. This flexibility is intended to build on and support the
significant State and local reform efforts already underway in critical areas such
as transitioning to college- and career-ready standards and aligned highquality assessments; developing systems of differentiated recognition,
accountability, and support; and evaluating and supporting teacher and
principal effectiveness. Only through such comprehensive efforts can SEAs and
LEAs truly increase the quality of instruction and improve student academic
achievement.
NCDPI/Superintendents’ Quarterly
Page 2 of 37
!
Notice of Flexibility on State Waivers from Elementary and Secondary
Education Act (ESEA)
Letter to Chief State School Officers on Graduation Rate Data and Race to the Top -- Printable
7/2/13 3:37 PM
Close
Window
Print
Key Policy Letters from the Education Secretary and Deputy Secretary
June 18, 2013
June 18, 2013
RELATED DOCUMENTS
Dear Chief State School Officers:
Fact sheet
Over the last four years, state and local leaders and educators across America
have embraced an enormous set of urgent and long-overdue challenges: raising
standards and upgrading curricula to better prepare students to compete in the
global economy, developing new assessments, rebuilding accountability systems
to meet the unique needs of each state and better serve at-risk students, and
adopting new systems of support and evaluation for teachers and principals.
Meeting this historic set of challenges all at once asks more of everybody
throughout the education sector, and it is a tribute to the quality of educators,
leaders, and elected officials across this country that so many have stepped up.
Homeroom Blog: New
flexibility for states
implementing fastmoving reforms (June 18,
2013)
State-by-State Timeline
Implementation Chart
Throughout this process, states and districts have established high goals for themselves: college- and career-ready
standards for all; higher graduation rates and college enrollment rates; high expectations for critical thinking,
problem-solving, and other 21st century skills; ambitious and achievable performance targets that really move the
needle for kids at risk; and useful, rigorous systems of evaluation and support for teachers and principals based on
multiple measures, including student growth. The Department has offered flexibility to enable states and districts
to meet these goals. In a country as diverse as ours, where schools and students have different educational
challenges, one-size-fits-all solutions have not worked. We have also aligned our grant programs to support states
willing to lead this important work, and the result is that some states are further along than others, but all states
are engaged in significant improvement efforts and students are better off for it.
In recent months, we have heard from many of you and from thousands of teachers, principals, and education
advocates. While there is a broad sense that these far-reaching changes carry enormous promise for schools,
children, and the future of our country, there is caution that too much change all at once could undermine our
collective progress. I fully appreciate both the courage to tackle so many challenges at once and the burdens this
imposes on front-line educators – teachers, principals, school boards, and administrators – who are committed
to doing this work well.
With that in mind, the Department is open to additional flexibility for states in two critical areas: the first relates to
one particular element of teacher and leader evaluation and support system implementation, and the second
addresses "double-testing" during the transition to new assessments aligned with college- and career-ready
standards.
First, I want to address the implementation of teacher and leader evaluation and support systems. States that
have received a Race to the Top grant or flexibility under the Elementary and Secondary Education Act (ESEA) are
responsible for working with districts to develop systems to evaluate and support principals and teachers based on
multiple measures, including student growth. States have committed to different deadlines to implement these
systems: some are implementing now; others will begin over the coming years. Given the move to college- and
career-ready standards, the dramatic changes in curricula that teachers and principals are now starting to teach,
and the transition to new assessments aligned to those standards, the Department will consider, on a state-bystate basis, allowing states up to one additional year before using their new evaluation systems to inform
personnel determinations. To be specific, states that request and are given this flexibility may delay any personnel
consequences, tied in part to the use of student growth data, until no later than 2016–2017. We recognize that,
http://www2.ed.gov/print/policy/elsec/guid/secletter/130618.html
NCDPI/Superintendents’ Quarterly
Page 1 of 2
Page 3 of 37
!
Letter to Chief State School Officers on Graduation Rate Data and Race to the Top -- Printable
7/2/13 3:37 PM
for many states, it will not make sense to request this flexibility because they are already well ahead in
successfully implementing these changes or have requirements in state law.
States interested in this extension may request this change, before September 30, 2013, through the current ESEA
flexibility amendment process. Details about the amendment process are available on the ESEA flexibility Web
page (http://www2.ed.gov/esea-flexibility). As each state implements college- and career-ready standards, it must
have a robust plan for supporting teachers and principals as they transition to the new standards and assessments.
States will need to lay out those plans in detail in the ESEA flexibility renewal process, along with indicators of
teacher and principal familiarity and comfort with these new materials.
The second issue I want to address is that of "double-testing" during the transition from the current statewide
assessments to new assessments aligned with college- and career-ready standards. During the 2013–2014
school year, some schools will be involved in the important work of field testing new assessments. We want to
support states that would like to avoid double-testing students, which as you know often happens during the shift
to a new test. Therefore, we would consider requests from states for a one-year waiver, to allow schools
participating in these field tests to administer only one assessment in 2013–2014 to any individual student —
either the current statewide assessment or the field test. We would also consider a request for those schools to
retain their Federal accountability designations for an additional year during which the same targeted interventions
would have to continue, with no relaxation of accountability requirements. Details about the Title I waiver process
are available at http://www.ed.gov/titlei-waiver.
Our country continues to face challenges as we work together toward achieving educational excellence for all
children, and the timing of these actions has real consequences for students in the real world. The point of raising
standards is to prepare students for tomorrow's challenges rather than yesterday's. Their readiness has real
consequences for their lives, and the nation's economic health. Yet this effort will only succeed if all parties have
the time, resources, and support needed to make the journey from the inadequate standards of the past to the
ambitious standards of tomorrow. As the highest-ranking education official in your states, you define the path and
the pace for how states and schools will make that journey. Our job in Washington is to support you. In the
coming days, the Department will provide more information on the flexibility discussed above; my staff will reach
out to you and your teams to provide assistance.
On behalf of the Obama Administration, I deeply appreciate your leadership and courage. I also appreciate your
honest feedback and the feedback of your principals and teachers. Above all, I salute your continuing
determination to advance reforms that will benefit millions of students in states and across America. This is hard
work and the need for change is urgent.
Sincerely,
/s/
Arne Duncan
cc:
Council of Chief State School Officers
Close
Window
Print
Last Modified: 06/20/2013
http://www2.ed.gov/print/policy/elsec/guid/secletter/130618.html
NCDPI/Superintendents’ Quarterly
Page 2 of 2
Page 4 of 37
!
Options for Moving Forward with Educator Effectiveness Model
North Carolina’s Race to the Top contract and waiver from the Elementary and
Secondary Education Act require the State to measure student growth and
include it explicitly in the evaluation of all teachers.
To meet this requirement, North Carolina has several options:
1.
Continue with each school district using the Measures of Student
Learning (MSL/Common Exams) and collaborate to improve the process.
Possible steps for improvement include:
• State prints the Common Exams and ships to school districts
• State includes greater specificity in Scoring Rubrics (if constructed
response items remain)
• State removes constructed-response items
• State requires that Common Exams replace teacher-made final
exams
• State requires that Common Exams count a defined percentage of
student’s grade statewide
• Continue with research on how to assess Arts Education, Healthful
Living, and World Languages
2.
Allow each school district to elect to develop its own system of measuring
student growth according to the federal definition of student growth.
Each district’s process for determining validity and reliability would
require approval by the State Board of Education, local board of education
chair, local superintendent, and district teacher representative.
3.
Allow each school district to elect to use school-wide growth from EOC
and EOG assessments to populate the sixth standard for all teachers in
non-tested areas. Decision would require approval by local board chair,
local superintendent, and district teacher representative.
4.
Allow each school district to elect to combine Options 2 and 3. Decision
would require approval by local board chair, local superintendent, and
district teacher representative.
5.
Other
!
!
!
NCDPI/Superintendents’ Quarterly
Page 5 of 37
!
Language from Senate Bill 402
PAY FOR EXCELLENCE
Section 9.5. When a robust evaluation instrument and process that
accurately assesses and evaluates the effectiveness of teachers, especially in
the area of student growth, is wholly implemented in North Carolina, it is the
intent of the General Assembly that the evaluation instrument and process be
utilized in the implementation of a plan of performance pay for teachers in this
State.
NCDPI/Superintendents’ Quarterly
Page 6 of 37
!
Letter on Measures of Student Learning from North Carolina Teacher
of the Year, Karyn Dickerson(*)
(*) All text emphasis is from Ms. Dickerson’s original letter.
Measures of Student Learning/Common Exams: A Teacher’s Perspective
At the end of the last school year, Common Exams were criticized by several
different parties. I would like to offer some opinions concerning Common
Exams and their implementation that might help put the criticisms into
perspective and offer some suggestions for improvement.
The Format of the Common Exams
First of all, I want to state that I thought the Common Exams were good
measures of student learning. I have been frustrated since I started teaching at
the discrepancies within schools, school districts, and the state in the level of
rigor addressed in teacher-made final exams. Students taking the same class
with different teachers could take very different exams, and the grades
students received on the exams and in the classes varied widely. Furthermore,
some teachers opted out of creating and administering exams that included a
constructed response. Even some of my fellow English teachers did not include
written portions on their exams in the past because they feared it would be too
time consuming to read all of the essays. I often questioned how we were ever
going to implement true writing across the curriculum if even English teachers
were not including written portions on their exams.
The new Common Exams allow for more consistent rigor on final exams
and require that all subject areas include a constructed response portion. I
believe that constructed responses not only help improve the writing skills
of our students, skills that many colleges have stated students are
woefully lacking, but also move our students towards more twenty-first
century ways of thinking. In this day and age, the acquisition of facts is not
as important as the application of those facts. Constructed responses require
students to develop their higher order thinking skills and are essential to help
teachers understand what concepts need to be taught in a clearer, more indepth way. The new Common Exams helped address all of these gaps in
previous teacher-made assessments.
The Common Exams also encouraged important collaboration among
teachers as they had to score students’ responses and discuss ways to
improve the results for their students in the future. I found that during the
school year, teachers worked together much better in preparation for the
exams than they have in previous years. In my experience, it was often only the
teachers of EOCs that collaborated on a regular basis to share best practices,
NCDPI/Superintendents’ Quarterly
Page 7 of 37
!
but this year with the implementation of the Common Exams, I saw much more
collaboration throughout my school and in different content areas. Now that
teachers know what the exams look like, I foresee even more collaboration next
year.
Concerns about Timing of the Common Exams
In spite of many of the strengths of the format of the exams, the rushed
implementation of the exams created a great deal of stress for many teachers.
I believe that this is the primary reason why teachers complained about the
exams. Teachers have experienced many changes over the past several years.
There have been new systems to learn (NCWise, PowerSchool, McRel
Evaluations, PEP online software in some districts for student intervention,
Home Base) as well as significant changes to content in the Common Core and
Essential Standards. I think most teachers would state that they support the
new standards, but the number of changes in such a short span of time has
exhausted many teachers. They are putting in numerous hours after school
and over the summer learning new standards, creating new lesson plans, and
reworking their assessments. They are also spending a great deal of time
trying to communicate these changes to parents and other educational
stakeholders. For many school districts, last year was the first year that the
Common Core and Essential Standards were fully adopted. To expect teachers
to master the new standards, find the best ways to teach their students to
address the new standards, implement more technology than ever, and
complete all of the above with few corresponding resources led to many
teachers feeling overwhelmed. Numerous hours were spent trying to find
resources to address the standards, and many teachers spent more money out
of their pockets than ever before.
Due to all of the significant changes in education over the past few years, I
think DPI’s suggestion to count the 2013 Common Exams as pilot tests
would help with teacher morale in the state. Teachers want to do well and
want their students to do well. The lack of time to truly master the new content
before administering a new exam caused a great deal of frustration among
teachers. The morale of my fellow coworkers was the lowest I have ever seen
during exams and in the month leading up to them. Adding to the stress for
teachers was the last-minute notification of testing plans and policies within
school districts.
Problems with Flexibility among School Districts
With the flexibility in individual school districts and in individual schools with
how to administer the Common Exams, there was even more undue stress
placed on teachers. As we heard in the presentation at the State Board of
NCDPI/Superintendents’ Quarterly
Page 8 of 37
!
Education meeting in July, some students had to take two final exams (both the
Common Exam and a teacher-created exam)! That is way too much testing and
grading at the end of the school year. There was also a wide range concerning
when to administer the tests and how they would count for students. Because
of this flexibility, some school districts and individual schools did not have a
clear plan for testing until almost a month prior to the start of testing. The lack
of a set policy until the end of the school year negatively impacted the attitudes
of students, parents, and teachers towards the exams. At my school, we
administered Common Exams the week before End of Course Exams. That
meant that towards the end of the school year, teachers suddenly lost a week of
instruction that they had planned on all year. For teachers like myself who
spend time creating long-term plans and unit plans, this made the end of the
year extremely stressful as I had to make significant changes to my plans at the
last minute.
Additionally, I had some concerns with some of the districts’ implementation of
the constructed response scoring rubrics. Teachers in my school district were
informed about three weeks prior to administering the Common Exams that in
all exams except for English the constructed responses did not have to be
written in paragraphs or even in complete sentences! Some teachers
encouraged students to respond in bullet format on their exams, including
social studies exams. This was surprising to me since history assessments at
the college level require written responses in complete sentences. It seemed
like the discrepancies in how to administer and score the exams defeated the
initial purpose of the exam: to assess twenty-first century skills and the
acquisition and application of important content. We need much more
consistency across school districts in order for the Common Exams to be
as effective as they have the potential to be in assessing the growth of our
students and teacher affect.
Conclusion
In short, the Common Exams have a great deal of potential in assessing both
the gains students have made during the course of a year and the efficacy of
teacher instruction. I believe that these exams with a few modifications and
increased consistency have the potential to serve as valuable forms of
assessment that offer insightful feedback. If used effectively, they will prove to
be much more than just another set of tests.
NCDPI/Superintendents’ Quarterly
Page 9 of 37
!
Letter on Common Exam Implementation from NCAE President, Mr.
Rodney Ellis
NCDPI/Superintendents’ Quarterly
Page 10 of 37
!
NCDPI/Superintendents’ Quarterly
Page 11 of 37
!
Proposal from Group of Superintendents
The following superintendents have submitted a proposal on testing and
educator effectiveness to Dr. Atkinson. The proposal follows this cover page.
Authors:
•
•
•
•
•
•
•
•
•
•
Dr. Eric Beacoats, Durham Public Schools
Dr. Ed Croom, Johnston County Schools
Dr. Mary Ellis, Union County Public Schools
Dr. Beverly Emory (formerly of Pitt County Schools; currently of
Winston-Salem/Forsyth County Schools)
Mr. Maurice “Mo” Green (Guilford County Schools)
Dr. Tim Markley (New Hanover County Schools)
Dr. Donald Martin, Jr. (formerly of Winston-Salem/Forsyth County
Schools)
Mr. Reeves McGlohon (Gaston County Schools)
Dr. Heath Morrison (Charlotte-Mecklenburg Schools)
Dr. Frank Till (Cumberland County Schools)
NCDPI/Superintendents’ Quarterly
Page 12 of 37
!
Proposal for Testing in North Carolina
This past school year (2012-13), Local Education Agencies (LEAs) in North
Carolina administered over 190 state or national exams. The purposes of these
exams ranged from assessing student learning to evaluating educator
effectiveness. We embrace being accountable for student outcomes, but our
state’s introduction of Common Exams as a measure of student learning has
brought us to the point of too much testing. This focus on assessment has
become a distraction from teaching and learning, presenting a hindrance to our
charge to prepare every child, every day to be college and career ready. Overall,
we have two primary concerns with our state’s implementation of Common
Exams:
Inadequate (Validity) Field Testing
The twenty-two (22) Common Exams administered this school year were
informed by teachers throughout the state over several months. This was an
essential element of test production that should be acknowledged. Additional
validity testing or field testing was required before these assessments were
administered for the high stakes purpose of evaluating educator effectiveness.
Concerns about the length of the assessment and the number of constructed
response items surfaced in the fall. In the spring, concerns about the focus and
content of questions were raised. Such issues need to be resolved before the
results of assessments are used to evaluate a teacher’s impact on student
learning.
Drain on Teacher Morale and Time
As our nation moves toward full implementation of the Common Core, our
students are being asked to exhibit what they know and can do in more
complex ways. That in turn requires us to design or adopt assessments that
provide students with more authentic ways to show what they know. The
inclusion of constructed response items provides such an opportunity.
However, this shift in our assessments requires a commensurate shift in how
we score and administer assessments. Our state’s approach to administering
Common Exams has been to put the responsibility for making that latter shift
squarely on LEAs, specifically teachers. Without the additional fiscal or
programmatic support needed to score these new constructed response items,
LEAs had to press teachers into action to score students’ exams. This scoring
process, which LEAs were given the “flexibility” to design, required teachers in
some large schools to dedicate over 20 hours each to appropriately score
constructed response items. This took away from time usually reserved for
lesson planning, work in professional learning communities, and grading other
teacher developed assignments. The fact that this had to be done with no
additional compensation negatively impacted teacher morale as teachers were
asked to score tests they saw as flawed, to determine their own effectiveness.
NCDPI/Superintendents’ Quarterly
Page 13 of 37
!
We propose a three-fold solution to provide increased accountability and
greater insight to teacher effectiveness, while requiring fewer assessments.
They are as follows:
1. Treat the 2012-13 School Year as a Pilot Year
The aforementioned challenges implementing Common Exams this past year,
as well as others, creates a shaky foundation upon which to build our state’s
educator effectiveness work. However, these shortcomings become an occasion
for learning and improvement if leveraged well. Instead of charging forward
despite our missteps, we propose that the 2012-13 school year be treated like
a pilot year. This allows us to learn from last year’s implementation, make
improvements, and launch this initiative with the benefit of those lessons in the
2013-14 school year. This would not compromise the initiative, but improve it.
2. Adopt a New Option for Determining Educator Effectiveness
Based on data from 2010-11, the North Carolina Department of Public
Instruction (DPI) estimates that End-of-Grade (EOG) and End-of-Course (EOC)
assessments cover approximately 40 percent of North Carolina’s teacher
workforce. Common Exams, as a measure of student learning, are intended to
cover the remaining 60 percent of teachers. In our state’s Elementary and
Secondary Education Act (ESEA) flexibility request, North Carolina proposed
three options to determine a teacher’s effectiveness. Each option placed a
weight on individual teacher impact and school-wide impact. We do not believe
that every student needs to be tested in every subject in order to determine
student growth and teacher effectiveness. For teachers in non-tested areas, we
propose that teacher effectiveness be determined by overall school impact, as
long as that impact is positive. If the overall school impact is negative, then
teachers in non-tested areas should be evaluated based on standards 1-5,
without consideration of student growth. Our proposed approach, though
lacking the granularity of our current direction, can incent collective ownership
for student results and school-wide collaboration, tenets seen in effective
schools. We support the inclusion of student growth in teachers’ evaluations,
but we need another option. We’re willing and interested in working to craft
and implement such an alternative.
3. Focus on the Core Areas of English Language Arts, Mathematics, Science,
and Social Studies, through Grade 10, if Common Exams Must Be
Administered
North Carolina is currently on the path of assessing students in every grade in
every subject to determine educator effectiveness. We support the Common
Core and increased rigor, and desire a testing program that aligns with the
Common Core. The expansion of our current testing program lacks that
alignment. Nonetheless, if we must move forward with our expansion, we
NCDPI/Superintendents’ Quarterly
Page 14 of 37
!
propose that the administration of Common Exams target the areas of English
Language Arts (ELA), Mathematics, Science, and Social Studies through the 10th
grade, with a reduced testing program in the middle grades. This scaled back
expansion will increase our ability to determine student growth and teacher
effectiveness while curtailing the number of tests administered. This slow-down
of our expansion allows us to move prudently ahead during this time when our
testing program is evolving. However, if the administration of Common Exams
in subjects and grades currently identified must continue, we propose that
Common Exams be composed of only multiple choice items.
We offer these recommendations as the beginning of a larger dialogue on the
reform of public education, which testing and accountability play prominent
roles. Our aim is to prepare students equipped with 21st century skills to enter
college or the workplace, and lead our communities. We seek modifications to
our testing program to foster this focus on student learning and achievement.
NCDPI/Superintendents’ Quarterly
Page 15 of 37
!
Proposal for Testing In North Carolina
Grade
Level
ELA/Reading
Assessment
Math
Assessment
Science
Assessment
Social
Studies/History
Assessment
3rd
4th
5th
6th
7th
8th
9th
EOG
EOG
EOG
EOG
EOG
EOG
English I
MSL/Common
Exam
EOG
EOG
EOG
EOG
EOG
EOG
Algebra
1/Math 1
EOC
None
Science MSL
EOG
None
Science MSL
EOG
* Physical
Science MSL
None
None
None
None
None
None
Civics
Final Exam or
MSL/Common Exam
10th
English II EOC
Algebra 2/
Math 2 or 3
MSL
Biology EOC
American History I
Final Exam or
MSL/Common Exam
CTE Assessments
Eliminate Local Assessments
Eliminate Local Assessments
Additional
Assessments
Locally Designed
or Administered
Assessments
BOG of 3 ELA
ClassScape/MAP
ClassScape/MAP
ClassScape/MAP
ClassScape/MAP
ClassScape/MAP
ClassScape/MAP
ClassScape
EXPLORE
Agriscience Applications
Biomedical Technology
Career Management
Core & Sustainable Construction *
Fashion Merchandising
Health Team Relations
MS Excel & Access *
MS Word, Power Point & Publisher *
Multimedia & Webpage Design
Personal Finance
Principles of Business & Finance
Public Safety I
Tech Engineering & Design
Accounting I
Animal Science I
AOF Business Economics
AOF Financial Services
Apparel I
Automotive Service
Cabinetmaking I
PLAN
PSAT
ClassScape
4
NCDPI/Superintendents’ Quarterly
Page 16 of 37
!
Carpentry I *
Drafting I
Firefighter Tech I *
Foods I
Health Science I
Horticulture I
Interior Design I
Intro to Culinary Arts & Hospitality
Marketing
Parenting & Child Development
Principles of Tech I
Scientific & Tech Visualization I
Sports & Entertainment Market I
11th
English III
Final Exam
Math 3 or
Math 4
Final Exam
*Chemistry
Final Exam
American History II
Final Exam
Accounting II
Animal Science II - Small Animal
Apparel II – Enterprise
Automotive Brakes *
Automotive Electrical
Business Management
Cabinetmaking II
Carpentry II *
Culinary Arts & Hospitality I
Drafting II – Architectural
Drafting II - Engineering
Early Child Ed I
Entrepreneurship I
Firefighter II *
Firefighter III *
Foods II – Enterprise
Health Science II
Horticulture II
Horticulture II Landscaping
Hospitality & Tourism
Interior Design II
Marketing Management
ACT
ClassScape
5
NCDPI/Superintendents’ Quarterly
Page 17 of 37
!
Principles of Tech II
Scientific & Tech Visualization II
Sports & Entertainment Market II
Strategic Marketing
12th
English IV
Final Exam
Final Exam
Final Exam
Final Exam
AOF Business in a Global Economy
AOF Applied Finance
Auto Computer System Diagnostics
Automotive Electrical Advanced *
Business Law
Carpentry III
Culinary Arts & Hospitality II
Early Child Ed II
Entrepreneurship II
Nursing Fundamentals
*Indicated science course or one most appropriate for the LEA’s science course trajectory.
WorkKeys
We would like to eliminate the following MSLs/Common Exams in elementary and middle school grades: Grade 4 Social Studies,
Grade 5 Social Studies, Grade 6 Science, Grade 6 Social Studies, Grade 7 Social Studies, and Grade 8 Social Studies.
We would like to eliminate the following MSLs/Common Exams in high schools: English III, English IV, Geometry, Advanced
Functions and Modeling, Pre-Calculus, Chemistry, Earth/Environmental Science, Physics, World History, OCS English III, OCS English
IV, OCS Financial Management, OCS Introductory Math, OCS Applied Science, 21st Century Global Geography, Sociology, American
Humanities, 21st Century Civil Liberties and Rights, Discrete Math, Psychology, World Humanities, The Cold War, and Turning Points
in American History.
We ask that schools not be required to administer an operational End-of-Grade or End-of-Course assessment and a field test in the
same subject during any year.
Approved industry recognized credential assessments may be used for technical attainment in lieu of CTE post assessments. CTE
post assessments are required for federal accountability per the Carl D. Perkins CTE Improvement Act of 2006.
6
NCDPI/Superintendents’ Quarterly
Page 18 of 37
!
Quality Assurance Checks for the Common Exams
During the 2012-13 school year, hundreds of thousands of students took the
Common Exams for the first time. The design of the Common Exams began in
2011 with a gathering of 800 teachers who provided feedback on how to assess
their content standards.
A set of those 800 teachers reviewed and approved the items that were
generated as a result of their feedback. Thirty-nine school districts partnered
with the Department of Public Instruction to administer the Common Exams
during first semester. Their feedback informed decisions to the exams before
second semester administration.
The Department of Public Instruction has committed to rigorous internal and
external quality assurance checks prior to the use of any Common Exam data
for value-added analysis. The results from a Common Exam that does not pass
these checks will not be used to generate teacher value-added scores.
Internal Review by the Department of Public Instruction:
! Is the Common Exam reliable? Reliability describes the consistency or
stability of student scores if a student were to take the exam multiple
times.
! Is the Common Exam aligned with the content standards? The
Department of Public Instruction has already completed this step.
Multiple individuals have reviewed each Common Exam to ensure
alignment with the content standards:
" Department of Public Instruction Test Measurement
Specialists
" Department of Public Instruction Curriculum Specialists
" NC State University Content Specialists
" NC State University Specialists on Assessment for English
Language Learners
" NC State University Specialists on Assessment for Exceptional
Children
! Does the Common Exam produce a desirable distribution of scores with a
good spread across most of the scale? If the distribution is extremely
negatively skewed or positively skewed, the test may not be useful for
measuring growth of low ability or high ability students. Score
distribution is a range, not clustered at one point on the scale.
! Does the Common Exam have any items that need to be removed, for
example, an item that does not meet measurement of content criteria? If
so, the Department removes the item from the test and any data analysis.
NCDPI/Superintendents’ Quarterly
Page 19 of 37
!
Only Common Exams that pass these quality assurance checks move on to
the next stage of review.
External Review by the EVAAS Team at the SAS Institute:
! Did the Common Exam “pass” the quality assurance checks completed by
the Department of Public Instruction?
! Can the Common Exam measure the achievement of both high- and lowachieving students? This idea is referred to as the “stretch” of an
assessment.
! Did enough students take the Common Exam? Value-added analysis is
not possible when a very small group of students have exam results.
! Did enough teachers teach the course associated with the Common
Exam? Value-added analysis is not possible when a very small number of
teachers have taught the course.
! Can students’ prior test scores be used to predict students’ scores on the
Common Exam? For example, when the Department of Public Instruction
and SAS Institute explored value-added modeling for Career and
Technical Education Post-Assessments, the scores for some PostAssessments could not be predicted. For others, a prediction model was
possible.
! Are there sufficient differences among teacher value-added scores based
on the Common Exam? In other words, do the results of the Common
Exam provide measurable differences among teachers?
When a Common Exam passes both the internal and external review, results will
be used to inform a teacher’s value-added score and sixth standard rating in
the NC Educator Evaluation System.
NCDPI/Superintendents’ Quarterly
Page 20 of 37
!
Discussion Questions on Improvement of Common Exams
Administration
As part of the teacher evaluation process, the Common Exams were
administered for the first time in 2012–13 to North Carolina elementary,
middle, and high school students. Throughout the school year, the NCDPI
gathered feedback from teachers, students, parents, principals, and
superintendents on the Common Exams and their administration procedures.
Based on this feedback, the NCDPI is determining what changes may need to
occur to the Common Exams for going forward.
Listed below are the key items under discussion for change.
1. Currently the Common Exams are designed to be administered during a
class period or during exam week, and the LEA determines the testing
window. Should the testing window be restricted?
2. Currently the Common Exams are timed tests, and the testing time is 90
minutes for all Common Exams (elementary, middle, and high school).
Schools have the option to administer the test in one school day (i.e., one
90 minute testing session) or in two school days (i.e., one 45 minute
testing session on day one and one 45 minute testing session on day
two). The 90 minutes does not include time for general instructions and a
break. Should we continue with the 90-minute testing time?
3. In 2012–13, LEA test coordinators were responsible for downloading the
appropriate Common Exams and all associated ancillary materials and for
printing and distributing enough copies of these materials for their
schools. LEAs were also responsible for purchasing the answer sheets for
the exams. Should the State print the exams, ancillary materials, and
answer sheets, or should the LEA continue to print the common
exams and purchase the answer sheets?
4. Some LEAs gave a final exam in addition to the Common Exam. Should
the Common Exams, where available, replace local final exams?
5. For 2012–13, LEAs were asked to determine if or how results from the
Common Exams would be used in determining students’ final grades.
Should Common Exams carry the same weight for grading EOCs (i.e.,
25%), and should we change the EOC weight for grading to 15% and
count the Common Exams also as 15%of the final grade?
NCDPI/Superintendents’ Quarterly
Page 21 of 37
!
6. All Common Exams with the exception of four (i.e., OCS Applied Science,
OCS Financial Management, OCS Introductory Math, and U.S. History)
contained constructed response (CR) items. The number of CR items
varied by test from 2–4 questions. Teachers supported the he inclusion of
constructed response items, but the feedback has been both positive and
negative (see chart below). Should the number of CR items be reduced
or should they be removed from the exams for 2013–14?
Positives/Negatives on the Common Exam Constructed Response Items
Plus +
Delta -
Allows students to demonstrate what
they know and are able to do
Allows higher-level items
Provides more engaging items than
typical multiple-choice items
Provides teachers with professional
development (scoring process)
!
Makes the tests too long
Delays the turn-around of the scores
at the high school for inclusion in the
student’s final grade
Requires teachers to participate in a
scoring process, which some have
reported is too time consuming
Adds to inconsistency in grading
across schools/LEAs
!
NCDPI/Superintendents’ Quarterly
Page 22 of 37
!
NC School District Recommendation for Moving Forward with
NC Educator Effectiveness Model
District Name:
_____________________________________________________________________________
Superintendent Name (please print) & Signature:
_____________________________________________________________________________
Please place a check mark next to your recommendation for moving forward
with the educator effectiveness model.
_____ 1. Continue with each school district using the Measures of Student
Learning (MSL/Common Exams) and collaborate to improve the process.
Possible steps for improvement include:
• State prints the Common Exams and ships to school districts
• State includes greater specificity in Scoring Rubrics (if constructed
response items remain)
• State uses only multiple-choice items
• State requires that Common Exams replace teacher-made final
exams
• State requires that Common Exams count a defined percentage of
student’s grade statewide
• Continue with research on how to assess Arts Education, Healthful
Living, and World Languages
_____ 2. Allow each school district to elect to develop its own system of
measuring student growth according to the federal definition of student
growth. Each district’s process for determining validity and reliability
would require approval by the State Board of Education, local board of
education chair, local superintendent, and district teacher representative.
_____ 3. Allow each school district to elect to use school-wide growth from
EOC and EOG assessments to populate the sixth standard for all teachers
in non-tested areas. Decision would require approval by local board
chair, local superintendent, and district teacher representative.
_____ 4. Allow each school district to elect to combine Options 2 and 3.
Decision would require approval by local board chair, local
superintendent, and district teacher representative.
Comments:
Please submit this form to the DPI staff member in your discussion group.
NCDPI/Superintendents’ Quarterly
Page 23 of 37
!
Slides from Today’s Meeting
Educator Effectiveness and the Measures of Student Learning (MSL)
Superintendents’ Quarterly Meeting, July 25, 2013
1"
NCDPI/Superintendents’ Quarterly
Page 24 of 37
!
Overview 1.  Context and progress to date
2.  Measures of Student Learning (MSL) for nontested grades and subjects
•  Common Exams
•  Other MSL
3.  Recommendations for improvements to
Common Exams
4.  Options for moving forward (small groups)
2
NCDPI/Superintendents’ Quarterly
Page 25 of 37
!
Context & Progress •  Federal definition of “student growth” in
Race to the Top (RttT) and ESEA waiver
•  NC’s RttT contract
•  NC’s ESEA waiver
•  USED’s flexibility only for “consequences”
of educator effectiveness determination; full
model must still be implemented
3
NCDPI/Superintendents’ Quarterly
Page 26 of 37
!
Context & Progress
NC’s Research-based Model
Months of Learning Gained or Lost
State Math
Observation Tool
Observation Tool
+ Student Survey Observation Tool
+ Student Survey + Growth (Value-Add)
+1.2
Top 25%
Bottom 25%
-1.4
Top 25%
Bottom 25%
+.2
-.4
+2.8
-2
Top 25%
Bottom 25%
State ELA
+.7
-.9
+4.5
-3.1
+1.2
-1.3
Re-creation of chart from Gathering Feedback For Teaching, http://www.metproject.org/downloads/MET_Gathering_Feedback_Practioner_Brief.pdf
NCDPI/Superintendents’ Quarterly
Page 27 of 37
!
Measures of Student Learning
Type of Assessment Specific to Grade/Subject or Course
6 End of Grade or End of Course
Contribute
to
Academic
Success
6
6 Common Exams
6 CTE Assessment
6 K-3 Assessments
6 Analysis of Student Work
To learn more about the Measures of Student Learning, please visit
http://www.ncpublicschools.org/docs/effectiveness-model/
resources/measures-guide.pdf 5
NCDPI/Superintendents’ Quarterly
Page 28 of 37
!
Improving Common Exams
Using ESEA Flexibility to make 2012-13 a “Pilot” Year
USED is offering the opportunity to request postponing
any consequences of educator effectiveness until 2016-17
2010-11
2011-12
Adding
Standard 6
Standard 6
added, but
(Development) data does not
“count” and is
not published
(Development)
2012-13
Proposal:
MSL given…
but
Standard 6
data doesn’t
“count” (Pilot)
2013-14
2014-15
Year 1
Year 2
2015-16
Year 3
First
Effectiveness
Status
6
NCDPI/Superintendents’ Quarterly
Page 29 of 37
!
Improving Common Exams
by Addressing Administration Challenges
•  Require that the Common Exams replace
teacher-made exams •  Limit testing window
•  Review time for testing
•  Print exams and materials at the State
•  Require that the Common Exams count in final
grades for high school students •  Reduce/eliminate the constructed-response
items NCDPI/Superintendents’ Quarterly
7
Page 30 of 37
!
Options
for Maintaining Educator Effectiveness Model Option 1
Option 2
Continue with each school district using the Measures of
Student Learning (Common Exams) and collaborate to improve
the process.
Allow each school district to elect to develop its own system of
measuring student growth according to the federal definition of
student growth. Each district’s process for determining validity and
reliability would require approval of the State Board, local board chair,
local superintendent, and teacher representative.
!
Option 3
Allow each school district to elect to use school-wide growth
from EOC and EOG assessments to populate the sixth standard
for all teachers in non-tested areas. District plan would require
approval of local board chair, local superintendent, and teacher
representative. !
Option 4
Allow each school district to elect to combine Options 2 and 3.
District plan would require approval of local board chair, local
superintendent, and teacher representative.!
8
NCDPI/Superintendents’ Quarterly
Page 31 of 37
!
Small Group Discussions
1.  Consider options
2.  Record your recommendation
3.  Everyone shares thoughts
9"
NCDPI/Superintendents’ Quarterly
Page 32 of 37
!
North Carolina’s Approval for ESEA Flexibility
NCDPI/Superintendents’ Quarterly
Page 33 of 37
!
NCDPI/Superintendents’ Quarterly
Page 34 of 37
!
NCDPI/Superintendents’ Quarterly
Page 35 of 37
!
NCDPI/Superintendents’ Quarterly
Page 36 of 37
!
NCDPI/Superintendents’ Quarterly
Page 37 of 37