Sean P. Corcoran
New York University
Education Policy Breakfast
April 27, 2012

2
How did we get here?
 Research finds teachers are the most important
school influence on student achievement
 Teachers appear to vary widely in effectiveness, as measured by student gains on standardized tests
 Teachers can have long-run measurable effects on life outcomes (e.g., Chetty et al., 2012)

3
How did we get here?
 By many measures, teacher quality is inequitably
distributed across students and schools
 There is some evidence that teacher quality has
declined over the long-run (Corcoran et al., 2004)

4
How did we get here?
 If teachers are so important, what are we doing to ensure high-quality teachers can be found in every classroom, particularly for those students who need them the most?
 The generally accepted answer among policymakers:
not much, or at least current efforts are not working very well (e.g., see The Widget Effect)

5
Two key issues
 What is teacher quality and how do we measure it?
 What policies are most effective in improving the level and distribution of teacher quality?

6
Potential Teachers
• Self-selection
• Recruitment
Preparation
• Teacher training
• Alternative pathways
Screening
• Certification
• Testing
• Hiring / Placement
Retention
• Tenure decisions
• Involuntary exits
• Turnover/attrition
Professional
Development
Evaluation
The Teacher Quality Pipeline

7
What is teacher quality?
 The easy (non-)answer: skills, practices, personal characteristics that positively impact desired student outcomes
 Not a very helpful definition … but does make clear that it is ultimately outcomes that indicate quality

8
The “old view”
 Research and policy emphasized qualifications and experience as presumed indicators of quality
Certification
Certification test scores
Educational attainment (e.g. MA)
Subject matter preparation
College selectivity
Own academic abilities (e.g. SAT)
In-service professional development

9
The “old view” – why?
 Convenience – these measures are readily available and easily observable; a lack of data on outcomes themselves
 Face validity – on their face, they seem sensible
 Reward structure – traditional salary structure rewards these qualifications (e.g. MA, experience)

10
The “old view”
 NCLB’s Highly Qualified Teacher provision: all teachers of core academic subjects must:
 Have a BA or better in the subject matter taught
 Have full state certification
 Demonstrate subject matter knowledge

11
Policies that set high professional standards and barriers to entry
Potential Teachers
• Self-selection
• Recruitment
Preparation
• Teacher training
• Alternative pathways
Screening
• Certification
• Testing
• Hiring / Placement
Retention
• Tenure decisions
• Involuntary exits
• Turnover/attrition
The Teacher Quality Pipeline
Evaluation

The “old view”
 Surprisingly (or not) research has not found qualifications to be highly predictive of student outcomes (i.e. test scores), although some do better than others
12

13
The “new view”
 “Teaching should be open to anyone with a pulse and a college degree—and teachers should be judged after they have started their jobs, not before”
 Malcolm Gladwell, The New Yorker, 12/15/2008

14
The “new view”
 “Success should be measured by results…That’s why any state that makes it unlawful to link student progress to teacher evaluation will have to change its ways.” President Barack Obama, July 24, 2009

15
The “new view”
 In other words, let outcomes be the arbiter of quality
 Great in theory, but which outcomes do we measure, and how does one measure teachers’ contribution to them?
 How does one incorporate this information into personnel policies in ways that have desired effects?

16
Policies that focus on measurement and incentives
Potential Teachers
• Self-selection
• Recruitment
Preparation
• Teacher training
• Alternative pathways
Screening
• Certification
• Testing
• Hiring / Placement
Retention
• Tenure decisions
• Involuntary exits
• Turnover/attrition
Professional
Development
The Teacher Quality Pipeline

17
Measurement: outcomes
 Outcomes: to date, whatever we have on hand
 Typically, student growth on standardized tests in reading and math, grades 3-8 (though not for long)
 Necessarily a subset of expected skills/outcomes
 Necessarily a short-run outcome
 Is our evaluation measure properly aligned with the goals we have for our educational system?

18
Measurement: value-added
 Value-added:
 The theoretical construct: a teacher’s unique impact on student learning
 In practice, a statistic used to estimate this impact

19
Measurement: value-added
 “Unique impact” implies causality – i.e. ruling other possible explanations for student learning
 Several possible sources of error:
 Systematic error (bias): attributing “value-added” to the teacher when it is really due to some other factor
 Random error (noise): getting a “noisy signal” of the teacher’s contribution to learning

20

21

22
Measurement: value-added
 So how can we attribute
causality to a teacher?
 If teachers were randomly
assigned, this would be easy: systematic differences would almost surely be due to the teacher

23
Measurement: value-added
 In the absence of this, we can instead devise a statistical model to account for other factors that explain differences in achievement

24
Measurement: value-added
 Value-added is then defined as student achievement relative to predicted—in other words, there will always be a distribution of value-added
-
+
0

25
Value-added: bias
 How confident are we that value-added measures isolate the unique contribution of individual teachers?
 Classroom vs. teacher effects (esp. after 1 year)
 Teacher vs. school effects
 Mobile students
 Tracking (e.g. Rothstein falsification test)

26
Value-added: bias
 Does attributing outcomes to individual teachers even make sense?
 Middle and high school settings
 Team teaching
 Evidence that teacher peers matter
 The higher the stakes places on value-added measures, the more these questions matter

27
Value-added: noise
 Even if value-added measures are not biased, they are still noisy—i.e. they are estimates with a high
“margin of error”
 More years of test results helps, although this may be “too late” to provide actionable information

28

29

30

31

32
Implications for policy
 The promise of personnel decisions driven by
outcomes has led to sweeping reforms of
 Performance evaluations
 Tenure and promotion, dismissal
 Compensation
 Principal evaluation
 Evaluation of teacher training programs

33
Implications for policy
 Race to the Top led numerous states to propose
50% or more of performance evaluations to be the “teacher’s impact on student achievement”
 E.g. CO, FL, TN, NJ
 Indiana: “negative” value-added teachers may not receive an effective rating, and tenure requires 3 years of effective ratings in a row
 NY’s APPR: a somewhat more balanced approach

34
Potential Teachers
• Self-selection
• Recruitment
Preparation
• Teacher training
• Alternative pathways
Screening
• Certification
• Testing
• Hiring / Placement
Retention
• Tenure decisions
• Involuntary exits
• Turnover/attrition
Professional
Development
Evaluation
The Teacher Quality Pipeline

35
Implications for policy
 What can we realistically expect from value-added based policies?
 Not as much timely, actionable information as we might like – though perhaps useful as an early warning indicator
 Crude differentiation of teachers at best, but more than current practice

36
Implications for policy
 What are the risks and implications of a system based on high-stakes use of imprecise measures?
 Mechanical applications are dangerous
 Risk of improper attribution and “Type I errors”
 Public reporting has minimal benefits and may do harm
 Unnecessary diversion of resources
 Unclear effects on entry into teaching profession

37
Implications for policy
 Little is know about how value-added measures will be used in practice

38
References
 Excellent and (mostly) non-technical resources:


Corcoran (2010) report for Annenberg
 http://www.annenberginstitute.org/products/Corcoran.php
Harris (2010) Value Added Measures in Education


Koretz (2008) in American Educator
Braun (2005) primer for ETS


“Merit Pay for Florida Teachers: Design and
Implementation Issues” (RAND 2007)
Rivkin (2007) CALDER policy brief
 Harris (2009) and Hill (2009) point/counterpoint in the Journal of Policy Analysis and Management