The Effectiveness of California State University’s Teacher Preparation Programs on K-12
Student’s Achievement
Kilchan Choi
National Center for Research on Evaluation, Standards, and Student Testing
(CRESST)
UCLA
This Precis presents a proposal to evaluate teacher preparation programs implemented in
California State University Northridge (CSUN) in terms of impact and effectiveness in K-12
students achievement, particularly in reading and math. This proposal is mainly focused on
presenting statistical models and analysis plan for analyzing the existing LAUSD and CSUN
data base.
Overarching Research Questions
The following are the specific research questions addressed in this study.
1.
2.
In what ways do CSU Northridge Teacher Preparation programs impact K-12 pupil
achievement, particularly in reading and math?
a.
Do pupils of teachers who graduated from different CSUN teacher preparation
programs produce significantly different achievement patterns as measured by
standardized NRT’s and district- developed tests?
b.
Is there a relationship between teacher preparation factors (such as course
grades & test scores) and pupil performance on these tests?
c.
Which set of variables have a greater impact on pupil test scores, those relating
to teacher preparation programs, those relating to teacher inservice experience
(number of years teaching, waivers for special education or emergency
credentialing), or those relating to school characteristics (school characteristics
index, proportion of pupils in school lunch program)
Are there significant differences in K-12 pupil achievement patterns that are related to
whether the teachers completed their credential program at CSUN, other CSU’s and
non-CSU institutions?
a.
If there are differences, are these related primarily to school characteristics,
pupil characteristics, or teacher inservice characteristics?
We need to look at patterns within and across grade levels, comparing schools with similar
school characteristics, and, to the extent possible, teachers with the same number of years
teaching.
Statistical Models
The comprehensive data base from LAUSD and CSUN includes the following three
categories of information: 1) student information including test scores, characteristics (e.g.,
free/reduced price lunch status, LEP, ethnicity, gender, etc); 2) teacher information including
teacher preparation program, course grade, years of service, gender, age, etc; 3) school
information including demographic information, school characteristics index, % of students in
free/reduced price lunch program, etc). More importantly, student information is link to teacher
information and in turn also link to school information. The available data traces back to year
1999, and the most recent year is 2003.
As such, the data have a hierarchical structure and a cross-classified structure as well.
First, each student has a time-series of test scores. And students and teachers are nested within
schools. However, students are very likely taught by a different teacher at each grade. For
example, a student was taught by teacher A at grade 3, teacher B at grade 4, and teacher C at
grade 5. In this situation, grade and teacher are crossed not nested. Looking at the data year by
year, the data has three-level hierarchical structure: students are nested within teachers, who in
turn are nested within schools. However, in the array of year spectrum, the data has a nested
structure and a cross-classified structure: time-series of test scores are nested within students,
student’s grades are crossed with teacher, and students and teachers are nested within schools.
Compliant to data structure and research questions addressed above, I will employ two
different statistical models for this study. The first one is a 3-level hierarchical model (HM3:
Raudenbush & Bryk, 2002; Choi& Seltzer, 2006) and the second one is a value-added model
(VAM: Raudenbush, & Bryk, 2002; Sanders, Saxton, & Horn, 1997; McCaffrey, Koretz, Louis,
& Hamilton, 2004;.Choi, Goldschmidts, & Yamashiro, 2005).
One way of examining research questions is to apply the HM3 to the cross-sectional data,
repeatedly. For example, using students’ math scores in the 3rd grade at year 2003 as outcomes,
CSUN’s teacher preparation program effect can be estimated within a framework of HM3 as
presented in detail in the following section. This approach is a relatively less statistically
challenging, well-established model compared to the value-added modeling approach. In
addition, within the framework of HM3 it is readily possible to answer the research questions in
a cross-sectional viewpoint. One cumbersome of this approach is that it requires many runs of
the HM3, because the HM3 is applied to the each grade-level data in each year in order to
examine the patterns of the teacher preparation program effect on student achievement across
grades and years as well.
The other methodology is employing a value-added model using the entire data base at a
single time. VAM is an appropriate statistically modeling technique applied to this kind of
longitudinal cross-classified data in order to estimate teacher effect by examining student gain
(Sanders’ Tennessee Value-Added Assessment System: TVASS model; McCaffrey’s RAND
VAM) and/or growth (Raudenbush’s Chicago Public School model: CPS model). In TVASS
model and RAND model, the teacher effect is defined as what remains of the year-to-year gain
after removing the district mean gain and the contribution of factors idiosyncratic to student.
The TVASS model is a special case for RAND model in a sense that RAND model does not
assume a persistence of teacher effects over time and that it allows us to adjust for differences in
student characteristics in the model. Compared to TVASS and RAND model, CPS model can be
considered as a special case of TVASS, because it explicitly models student growth over time
either in a linear or a quadratic form to which district teacher effect is added and accumulated
over time. It also considered as a special case of RAND model in a sense that it includes the
covariate adjustment feature in the model, but it dose assume a persistence of teacher effects over
time. However, the communality across the VAMs is to focus on estimating the effect of each
individual teacher as a random effect, that is, an expected deflection to the students’ gain or
growth associated with encountering a particular individual teacher.
Despite of appropriateness of the VAMs to apply it to this kind of data and research
questions, VAM approaches have not been widely used in formal district accountability system
and program evaluation area in part because it requires very complex modeling extensive
computing resources as well. Furthermore, the standard routine available in statistical package
software, SAS, HLM, S-Plus, and R does not support all the modeling features of VAMs. For
example, estimation process in TVASS is proprietary and the three VAMs aforementioned can
be implemented only with the special customized programs developed by the each of authors.
In connection with the research questions in this study, it is necessary to expand current
VAMs or develop a new model which combines some of features of the CPS model with the
RAND model. As addressed in research questions, it demands not only estimation of individual
teacher effect in a cross-classified setting, which is the main goal of currently implemented
VAMs, but also comparing teacher effect between CSUN’s teacher program graduates and the
others controlling for teacher characteristics. Furthermore, variability of this comparison in
terms of teacher effect need to be examined across schools and is modeled as a function of
school characteristics. In other words, it needs to examine the extent to which differences in
teacher effect between the different groups of teachers relate to differences in school
characteristics such as % of FRL students, % minority, and so on.
I will propose a new value-added model to examine the research questions in this study.
I will present a fully Bayesian approach to a new VAM based on the use of Markov Chain Monte
Carlo (MCMC) algorithm, termed Gibbs sampler. The fully Bayesian approach entails
calculating the marginal posterior distributions of parameters of interest (e.g., a parameter
capturing different teacher effect between CSUN graduates and the others) from the joint
posterior distribution of all unknowns in the model. The MCMC technique is a very useful
estimation method for fitting such a very complex model. I will use WinBUGS program
(Spiegelhalter, Thomas, and Best, 1999) for fitting a new VAM using the Gibbs sampler, and use
HLM 6 software program (Raudenbush, Bryk, Condon, & Cheong, 2005) to fit 3-level
hierarchical models depicted in the following section.
3-level hierarchical model:
For a heuristic purpose, I pose a simple 3-level HM as follows. In a level-1 (between
students; within teacher) model, math achievement score, Yijk, for student i in teacher j, in school
k, is modeled as a function of his or her free/reduced lunch (FRLijk) status.
Yijk = 0jk + 1jk (FRLijk - FRL.. ) + ijk
tij ~ N (0, 2),
(1)
where 0jk is adjusted mean score for teacher j in school k, and 1jk represents the expected
difference for teacher j in school k in terms of math achievement score between FRL students
and non-FRL students. In a fully model, we can include more student characteristics variables as
covariates so that 0jk represents the expected mean score for teacher j in school k after
controlling for the differences in student characteristics in a model.
The following Equations 2a and 2b present a level-2 (between teachers; within school)
model. I include a key indicator variable which takes on a value 1 for teachers who graduated
from CSUN, 0 for otherwise.
0jk = 00k + 01k CSUNjk + r0jk
1jk = 10k + 11k CSUNjk + r1jk
r0jk ~ N (0, 0)
r1jk ~ N (0, 1) .
(2a)
(2b)
00k represents the expected mean score for non-CSUN teachers in school k, and 01k captures the
expected difference in mean score between CSUN teachers and non-CSUN teachers in school k.
Thus, CSUN teachers’ expected mean score in school k is (00k + 01k). If CSUN teachers
perform better than non-CSUN teachers, then 01k takes on a positive value. As in the level-1
model, it is possible to include more teacher characteristics variables in Equations 2a and 2b. As
such in level-2 model, we can examine CSUN teacher effect controlling for various teacher
characteristics and interaction effect of CSUN teacher by including a product term of CSUN
teacher indicator variable (CSUNjk) and a teacher characteristic variable (e.g., gender).
In level-3 (between schools) model, I include percentage of FRL student as a school-level
variable. The key equation of interest is Equation 3b where the expected difference between
CSUN teachers and non-CSUN teachers in school k is modeled as a function of % of FRL
student.
00k = 000 + 001(%FRLk – FRL. ) + u00k
u00k ~ N(0,00)
(3a)
01k = 010 + 011(%FRLk – FRL. ) + u01k
u01k~ N(0,01)
(3b)
10k = 100 + 101(%FRLk – FRL. ) + u10k
u10k~ N(0,10)
(3c)
11k = 110 + 111(%FRLk – FRL. ) + u11k
u11k~ N(0,11) .
(3d)
In other words, Equation 3b model the extent to which CSUN teacher effect relates to difference
in % of FRL student in a school. Thus, 011 represents the expected increase or decrease in
CSUN teacher effect when one % increases in % of FRL student. Using this level-3 model, we
can examine relationship between CSUN teacher effect with school characteristics variables.
A new Value-Added Model:
When examining contextual effects on individual growth using a 3-level growth
hierarchical model, the example involves repeated measures on students who were nested within
teachers or schools. The teacher or school differences in student’s growth rate are of central
interest. However, this example using a 3-level growth hierarchical model limits situations
where students remain in a single context during the period of investigation. In other words,
students should be taught by the same teacher each year. If students were taught by a different
teacher each year, the data no longer have a nested structure and it is not possible to apply the 3level growth model to it. Rather, it is called as a cross-classified data structure and often requires
using so called value-added model.
For simplicity, I present a simple value-added model where it is assumed that student’s
linear growth trajectory in math during grades 3 to 5. Note, however, that as in TVASS model
and the RAND model, we do not make explicit assumptions about growth. Instead, we can
allow the means to vary across grades, which implicitly accommodates certain growth models.
Posing either a linear growth, quadratic growth, or no explicit specification of growth is
dependent upon researcher’s interest and the degree of data fitness.
The following equation presents a level-1 model:
Ytijk = 0ik + 1ik (Gradeik - 3) + ik
ik ~ N (0, 2),
(4)
where Ytikj is math achievement score at time t (= 0, 1, 2) for student i in teacher j in school k.
0ik is expected status at grade 3, initial status, for student i in school k, and 1ik is expected
growth rate.
At level 2, we have
0ik = 00k + 01k FRLik+ b00ik +
K
t
 D
c
hijk 00 jk
,
(5a)
k 1 h  0
1ik = 10k + 11k FRLik+ b10ik
(5b)
  0    b 00  b 00, 10  
 b00ik 

 ~ N  ,

 b10ik 
  0   b10, 00  b11  
C00jk ~ N (0, c00).
Where Dhijk = 1 if student i encounters teacher j at time h, 0 otherwise. The double summation in
Equation 5b cumulates the teacher effects in school k C00jk over time. The design matrix Dhijk
defines cross-classified structure of student’s grade and teacher j in school k. In other words,
Dhijk matrix contains information which teacher in school k taught which students in each grade.
The key parameter of interest, C00jk , is a vector containing each individual teacher effect. That is
the extent to which an expected deflection to the growth trajectory is associated with
encountering teacher j in school k. The teacher effects assume mean 0 and variance c00. Note
that student FRL variable is included so that teacher effect is estimated controlling for student
characteristics.
More importantly, I now expand Equation 5a by modeling random teacher effect as a
function of CSUNjk variable and teacher’s gender variable. Thus Equation 5b can be re-written:
0ik = 00k + 01k FRLik+ b00ik +
K
t
 D
hijk
[  01k (CSUNjk )   02 k ( AGEjk )  c00 jk ] ,
(5c)
k 1 h  0
where 01k is the adjusted mean difference in teacher effects in school k between those who
graduated from CSUN teacher preparation program and those who did not, holding constant
teacher’s age.
In level-3 model, different teacher effect of CSUN variable in school k is modeled as a
function of school-level variable.
00k = 000 + 001(%FRLk – FRL. ) + u00k
u00k ~ N(0, 00)
(6a)
01k = 010 + 011(%FRLk – FRL. ) + u01k
u01k ~ N(0, 01)
(6b)
10k = 100 + 101(%FRLk – FRL. ) + u10k
u10k ~ N(0, 10)
(6c)
11k = 110 + 111(%FRLk – FRL. ) + u11k
u11k ~ N(0, 11)
(6d)
01k = _011 + _012(%FRLk – FRL. ) + u01k
u01k ~ N(0, 01)
(6e)
02k = _021 + _022(%FRLk – FRL. ) + u02k
u02k ~ N(0, 02)
(6e)
By doing this specification in Equation 6e, the relationship between school-level variable and
different teacher effect across schools can be examined.