Student Teacher Linkage Data Quality: Discrepancy Rates and Team Teaching
Jeff Watson
Chris Thorn
UW Madison
Wisconsin Center for Education Research
Value-Added Research Center
Center for Data Quality and Systems Innovation
Paper Presented at the Association for Education Finance Policy 37th Annual Conference
March, 2012
Boston, Massachusetts
Background and Purpose
Urban districts invest significant resources in data systems and information technology infrastructure
with the goal of informing decision making, supporting evaluation, and enabling planning. Increasingly,
districts are trying to use their data systems to inform decisions related to measuring and improving the
rates at which students are learning (e.g., using value-added). One necessary condition of developing
classroom-level estimates of student growth is that a district’s data systems must provide information
about which teachers are teaching which students. While it is unrealistic to expect districts to have
perfect data about who is teaching whom, we should be very concerned about if student – teacher
linkage data are good enough to support this new use. This paper reviews our work with one urban
district that was designed to assess the quality of their ST linkage data as well as the extent to which
team teaching may impact the attribution of student learning to specific teachers.
Linking students to teachers sounds like a function that any district data system should be able to do.
However, many states and districts are challenged when they try to implement systems that require
high quality student–teacher linkage data. The Data Quality Campaign (DQC) has focused on providing
resources state-level policy makers need to guide strategic planning for their states’ information
technology infrastructure. In 2009, the DQC found that 29 states self reported the ability to link
students to teachers (DQC, 2009). However, in a 2010 brief, the DQC reported that most states and
districts lack the capacity to know about team-teaching, schools’ organizational models, or the ability to
verify and improve the quality of their student–teacher linkage data (DQC, 2010). Battelle for Kids
recently published a white paper that identified several sources of errors in district data systems that
can impact the quality of student– teacher linkages. These deficiencies included not knowing about
course schedules, school- and grade-level regrouping students during the year, student and teacher
mobility, team teaching methods, course content and course-assessment alignment, as well as charter
school teaching and learning (Battelle for Kids, 2010).
As the stakes associated with these new policies increase, the need for accurate and valid student –
teacher linkage data also increases. As a result, the process of validating student – teacher linkage data
is becoming more common place. While organizations like the DQC and Battelle for Kids are helping
raise awareness of the factors that can impact the data quality of student teacher linkages, little
attention has been given to documenting the extent to which student – teacher linkage data accurately
portrays who is teaching whom. Watson et. al (2010) reported results from a pilot study where 10% of
records were validated as incorrect by teachers and another 15% of records were flagged as having
received team teaching. In other words, only 3 out of 4 records were verified as accurate and valid.
We designed the current study to expand the 2010 pilot study to a much larger sample of schools and
teachers. We assessed the accuracy of student-teacher linkage data by asking teachers to verify the
math and reading courses they were teaching as well as the student rosters for those courses. In
addition to assessing the accuracy of the district’s data, we also asked teachers to report the extent to
which they used team teaching.
Methods
Sample
Working with district leaders, we identified 49 schools to invite to participate in this study. Teachers
were invited to participate via email as well as through printed letters of support and periodic requests
for their participation from their principals. Over a three week period, 348 teachers from 47 schools
reviewed their student-teacher linkage data for 8824 students. As a result, 16,318 student records were
verified by teachers.
Procedure
We developed a web-based application called the Student Teacher Verification System (STVS) for the
purpose of efficiently verifying teacher – course assignments and student rosters. Initial student-teacher
linkage data was extracted from the district’s student information system and loaded into the STVS. We
also worked with the district prior to verification to recruit schools, train principals on the system, and
describe the purpose of the study. STVS was field tested in 4 schools 2 weeks prior to the
implementation of the study to ensure that they system was working properly on the computers that
were typical for the district.
Teachers completed the following steps:
1)
2)
3)
4)
5)
6)
7)
Log In
Confirm courses
Indicate content area
Indicate course-level team teaching
Confirm students for Math and Reading courses
Indicate student-level team teaching
Log out
The entire process required about 15 minutes to complete. Teachers who did not wish to participate
simply did not log in to the system. After 3 weeks, the teacher participation window was closed and
Principals logged into the system to resolve any discrepancies that may have resulted from teachers'
participation. Specifically, STVS asks principals to review and assign any unclaimed students or classes.
Findings and Conclusions
Student-teacher linkage discrepancies
In general, our findings suggest that teacher-course assignments were pretty accurate and student
rosters were also in pretty good shape for this district. Out of 16318 records verified, only 428 (2.6%)
were flagged by teachers as being discrepant. Incorrect teacher – course assignments accounted for .7%
of these while incorrect student-roster data accounted for 1.9% (see Figure 1).
17443
13296 (81.5%)
Records
reviewed
No Team Instruction
100%
Self-Attribution
15890 (97.4%)
16318 (100%)
Records Included
in Analysis
Correct at both
course- and studentlevel
2594 (15.9%)
Some Team
Instruction
775 (4.7%)
Data Quality
111 (.7%)
428 (2.6%)
Incorrect at either
course- or studentlevel
Course-level
Course-level
discrepancy
1422 (8.7%)
317 (1.9%)
Student-level
58.4%
Average SelfAttribution
63.5%
Course Level
Self-Attribution
49.7%
Student Level
Self-Attribution
Student-level
discrepancy
397 (2.4%)
1125 incomplete records excluded
BOTH courseand studentlevels
38.4%
Both Levels
Self-Attribution
Team Teaching
Figure 1- Data Quality Logic Model. Discrepancy frequencies for disputed records, Team teaching
frequencies, and the average self-attribution of instruction.
However findings showed that discrepancy rates varied by school and teacher. Using a weighted
average, we observed that 1.8 records per teacher were disputed (SDweighted = 7.3). Furthermore, 80%
of the discrepancies originated from just 6% of participating teachers (N=22) (Figure 2). The highest
discrepancy rate observed for a teacher was 98% (83 out of 84 records) and another 4 teachers had
discrepancy rates at 50% or higher. Ten teachers reported discrepancy rates at 10% or higher and less
than 50%. 19 teachers reported rates between 5% and 9.9%. Thirty-seven teachers had rates higher
than 0% but less than 5% and 278 teachers reported no discrepancies.
Cumulative Percent of Discrepancies by
Teacher
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
13
25
37
49
61
73
85
97
109
121
133
145
157
169
181
193
205
217
229
241
253
265
277
289
301
313
325
337
0
Figure 2- Cumulative percent of discrepancies by teacher - Twenty-two teachers accounted for 80% of
reported discrepancies.
We observed similar patterns when we aggregate discrepancies at the school-level. Using a weighted
average, we observed only 9.1 discrepancies per school (SDweighted = 13.7). However, eleven schools
accounted for 80% of the discrepancies reported and seventeen schools reported no discrepancies
(Figure 3). The highest observed school-wide discrepancy rate was 49% (based on 169 records). Two
other schools had rates above 10% and three more had rates between 5 and 10%. Twenty-five schools
had rates between 0% and 5%, whereas sixteen schools had no discrepancies.
One of the challenges that remain, despite the generally high quality data in most schools, is how to
successfully identify the schools and classrooms in which error rates are high. Our finding of 80% of the
errors in 11 of the schools 49 schools suggests that the deployment of a modern student information
system is not enough to address data quality challenges by itself. The task for policy makers is to explore
the sources of error in this group of schools. Anecdotal evidence from other urban districts suggests that
issues of staff training, leadership turnover, and novel organizational models (such as looping or multigrade classrooms that don’t fit the standard SIS use case) have all been associated with higher error
rates. Without a systemic understanding of the source of this error to more explicitly address its causes,
system-wide verification may be the only way to ensure fairness for high stakes use of student-teacher
linkage data.
Cumulative Percentage of Discrepancies
by School
1
0.8
0.6
0.4
0.2
0
V
I AF C AP D AJ H AO S AC AA AQ AL AT AB E G
L
P
Z AH AK AS
Figure 3- Cumulative Percentage of Discrepancies by School. About 80% of discrepancies were observed
in about 20% of the schools included in this study.
Team teaching discrepancies
Our data suggests that team teaching practices are commonly used, and on average teachers only
attributed about 60% of a student’s instruction to themselves when team teaching methods were used.
Of the 15890 records that were verified as accurate, teachers flagged 2594 as receiving instruction
through team teaching practices such as shared classrooms, pull-out instruction for English language
learners, or push-in services for special education instruction (Figure 1). When aggregated by school,
team teaching was reported unevenly across schools in that about half of the schools accounted for
about 80% of the records flagged as receiving team teaching (Figure 4). When teachers reported team
teaching at the course-level (e.g., shared classrooms), the percent of instruction they attributed to
themselves was bimodally distributed at 50% and 75% levels. Self-attribution of instruction for studentlevel team teaching also appeared bimodal, but a higher percentage of teachers estimated selfattribution below <40% (Figure 5).
Cumulative Percentage of Team Teaching
by School
100.0%
80.0%
60.0%
40.0%
20.0%
0.0%
M AI AF Y
F N T AN J AR B R P A G AU AT U E
L AM AA AJ AS
Figure 4- Cumulative Percentage of Team Teaching by School. Team teaching practices were less
clustered than data quality discrepancies, but were nonetheless distributed unevenly across schools.
0.4
0.35
0.3
0.25
0.2
0.15
Course
0.1
Student
0.05
91-100
81-90
71-80
61-70
51-60
41-50
31-40
21-30
11-20
1-10
0
0
% of Instruction Attributed to Self
Figure 5- Distribution of Self-Attribution of Instruction in Team Teaching Situations.
Team teaching presents a similar, but slightly different challenge than the “distribution of error”
problem presented above. While team teaching at the course and individual student level reflects of the
reality of a substantial number of students in the sample, it is also analytically much more complicated
and is a policy challenge for school leaders who are now required in many states and districts to provide
individual effectiveness ratings to individual educators . The fact that approximately 16% of the sample
included some level of team teaching indicates that for those teachers, the system will only be able
produce a single measure of the teachers’ joint effort – not a measure of each teacher’s contribution.
While not a problem for program evaluation of school-level effectiveness studies, it does provide a
substantial challenge for those charged with producing individual measures of educator productivity.
References
Battelle for Kids, 2009 – The Importance of Accurately Linking Instruction to Students to Determine
Teacher Effectiveness (http://static.battelleforkids.org/images/BFK/Link_whitepagesApril2010web.pdf)
Data Quality Campaign, 2010 - Strengthening the Teacher-Student Link to Inform Teacher Quality Efforts
(http://www.dataqualitycampaign.org/files/TSDL_abstract.pdf)
Data Quality Campaign, 2010 - Effectively Linking Teachers and Students
(http://www.dataqualitycampaign.org/files/DQC7_14.pdf)
Watson, J., Thorn, C., Ponisciak, S. & Boehm, F. (2011, March). Measuring the Impact of Team Teaching
on Student-Teacher Linkage Data. Paper presented at the 36th annual meeting of the Association for
Education Finance and Policy, Seattle, Washington.