Modeling student achievement in K-12 math and science classrooms: An experimental approach
M.J. Bryant, R.A. Cardullo, K. Bocian, K.A. Hammond. University of California, Riverside, CA 92521
The initial study group includes teachers and students from 16 elementary and 3 middle
schools. In dealing with a small group a number of questions arise concerning the research
design, the quality of assessments used for evaluating both content and pedagogy, and the
types of analysis used to maximize statistical power in quantifying the effectiveness of
treatments. Since scientifically designed and controlled experiments are rare in education,
Mathematical ACTS provides an opportunity to quantify the effectiveness of various teacher
interventions.
Question: How can student achievement in mathematics be improved in public schools so
that a larger fraction of students is eligible for Algebra I by eighth grade?
Hypothesis: Since students receive mathematics training from teachers, improving student
achievement in elementary and middle school math classes can be attained by improving the
math proficiency and pedagogy of teachers.
Prediction: As a result of ACTS professional development, student achievement, as
measured by standardized tests, will significantly increase resulting in a measurable increase
in Algebra I enrollment.
A core team consisting of
mathematicians, scientists, and
educators develops a treatment
Treatment
The treatment is
administered to teachers in
the treatment group
Teachers teach with
learned techniques
Efficacy of the treatment
is measured through
student test scores
4th Grade
Teachers
4th Grade
Students
5th Grade
Teachers
5th Grade
Students
6th Grade
Teachers
6th Grade
Students
7th Grade
Teachers
7th Grade
Students
Time (2002-2006)
Figure 1.
From
to student
This
severalprofessional
year long studydevelopment
measures the growth
in both achievement
teachers and students over time
We predict that student achievement will improve by providing teachers with improved mathematics
content and pedagogy. These treatments are applied to teachers in grades 4-8.
Score on Mathematics Pedagogy Test
Relationship Between Mathematics Proficiency and Pedagogy
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
Year 2
Year 3
Year 4
Year 1
Year 2
Year 3
Year 4
Figure 2. Comparison of possible experimental designs
The treatment () can be applied to either entire schools or to individual classrooms within a school. Schools or
classrooms that do not receive the treatment serve as controls (). Ignoring issues of human subjects, the
treatment would be applied to half of the schools or classrooms. The bottom two panels illustrate and example
of increasing the proportion of treatment to control groups which is desirable in an educational experiment.
The experimental design with the highest statistical power is
dependent on where the variation is greatest.
0.20
Variation greatest among schools
Classroom design has the
most statistical power
0.05
School design has the
most statistical power
0.6
0.8
1.0 0.0
0.2
0.4
0.6
0.8
1.0
Probability of P < Alpha
Figure 3. Monte Carlo based power analyses inform which design has the highest statistical power.
The choice between schools or classrooms as the experimental unit is subject to many logistical considerations.
The design impacts statistical power (the probability that a significant difference is found when a difference
exists). Computer simulations reveal that statistical power is optimized as a function of where random variation
lies. For example if all classrooms in a school are equal but variation exists among schools, randomly assigning
treatment to classrooms has greater statistical power.
Schools matched by demographic variables
for stratified random assignment
3
2
1
2.
0
3.
-1
4.
-2
-3
-4
-3
-2
-1
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
0
1
2
3
4
100
90
Sample Questions
Question 39
Question 1
1) 3.01 + 0.301+ 30 = ?
(A) 0.631
(B) 3.611
(C) 33.311
(D) 33.32
80
39) In the right triangle ABC
shown to the right, what is
the length of AC ?
(A) 10
(B) 14
(C) 24
(D) 48
70
60
A
50
80
90
100
70
50
60
Percentage of Elementary School Teachers
Correctly Answering Question
6
B
8
C
Figure 6. Baseline data can serve to inform professional development content.
Primary and secondary school teachers do not understand all mathematics subject areas equally. The data
points above the 1 to 1 line suggest relatively higher proficiency on a given question by secondary school
teachers. By examining the relative performance on each question by the different groups of teachers we can
identify concepts where a need for professional development is not indicated (i.e. Question 1). Questions
with a great disparity between the two groups can be used to fine tune professional development in areas of
content knowledge where it is most needed (Question 39 builds off of State Standards from the grade 4
through grade 7).
1.
-5
0.6
Variation greatest among classrooms
0.10
0.4
0.8
Figure 5. Mathematics content knowledge and pedagogy are related but distinct.
Primary and secondary school teachers tend to have higher pedagogical skills (as measured by a
University of Michigan designed pedagogy assessment) when their mathematics content proficiency is
higher (CSU/UC Algebra Readiness Test). Secondary school teachers have significantly higher
mathematics pedagogical skills, perhaps due to the fact they are single subject teachers. Despite the
correlation, there remains considerable variation in pedagogical skill for all levels of content knowledge.
The secondary school teachers’ baseline data serves as benchmark goal for the primary schools teachers.
0.15
0.2
Elementary School Teachers
Secondary School Teachers
Effect of Teacher’s Grade level: P = 0.0002
Score on Mathematics Proficiency Test
Treatment Applied to:
Schools
Classrooms
0.00
0.0
1.0
Percentage of Secondary School Teachers
Correctly Answering Question
The effect of professional development on teaching and the subsequent achievement of
students are studied in a research design that includes:
•Comparison of teacher baseline and post ACTS intervention knowledge and use of
mathematics and pedagogy.
•Assessment of the implementation of ACTS teaching strategies in the classroom.
•Assessment of the value added by the treatment in the classroom by tracking the rate of
student achievement growth in targeted and control (wait-listed) classrooms.
•Case studies of teacher and school implementation within one school site.
•Assessment of benchmarks of systemic change for both UCR and JUSD.
Treatment Applied to Classrooms
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
Year 1
Alpha (< 0.05 considered significant)
Mathematical Achievement through Collaboration with Teachers and Students (Mathematical
ACTS) is a multi-faceted NSF-funded research project designed around specifically tailored
initiatives for teachers and students in the 4th – 8th grades. ACTS seeks to raise achievement
and close gaps in mathematics for all students in the Jurupa Unified School District through a
series of programs (150 hours annually) for teachers and students. For the purpose of this
project, student achievement in mathematics is defined as preparation for and success in
Algebra I, the gatekeeper class for higher mathematics courses and entrance colleges and
universities.
We targeted students in the Jurupa Unified School District 4th through 9th grades, beginning in
September 2003 through June 2007. Low mathematics performance in K-12 impedes
progress in college mathematics and science and its toll is being felt at many institutions of
higher education. At the University of California, Riverside the percentage of students who
take remedial mathematics (defined as pre-calculus or below) has risen from 65% in Fall,
2000 to 73% in Fall, 2003 (UCR Learning Center Fall, 2003 report). In addition, the majority
of under-represented high school students in the 66th State Assembly District are not prepared
for college. Less than 15% of students from this region attend any university (California
Postsecondary Education Commission Report, 2003) and the percentage of students that take
any mathematics beyond Algebra I is exceedingly low. Left unchecked, any cohort of
students that is not well prepared in both mathematics and science will not be prepared to
function in a technologically advanced society.
Index Based on Characteristics of the School
(i.e. Class Size)
INTRODUCTION AND STATEMENT OF PROBLEM
Treatment:Control Increases Treatment:Control Constant
(School)
(School)
Treatment Applied to Schools
CONCLUSIONS
Any experimental design in education must take into account both traditional scientific design
motifs as well as ethical considerations (i.e., ultimately providing treatments for all subjects
during the study).
To optimize statistical power the choice of schools or classrooms as experimental units depends
on the amount of variation within that unit.
Pairs of experimental and control groups must be established by taking into account multiple
variables (i.e., socio-economic status, school test scores, etc.).
In our study, there are significant differences in both math content and pedagogy between
elementary (Grades 4-6) and Middle School (Grades 7 & 8) teachers. Further, there are defined
areas within mathematics where there are significant differences between these two groups.
5
Index Based on Aggregated Student Level
Characteristics (i.e. % reduced or free lunch)
Figure 4. Assignment of treatment and control groups as a stratified random design.
In educational settings, demographic factors (i.e. socio-economic status or SES) are known to have a great
impact on student performance, but random assignment of treatment is essential. Random treatment assignment
may lead to unintended pooling on either end of an SES continuum. By selecting schools at random from
matched pairs, we can control for differences in SES. Here schools were matched by principal components
analysis. Blue arrows connect matched pairs ( = Treatment,  = Control). Perfect pairing may not be possible
for all schools (▲ = schools with SES characteristics outside the range of the other schools).
ACKNOWLEDGEMENTS
1. This work is funded by a Math Science Partnership grant from the National Science Foundation.
2. We would like to acknowledge the support of the ALPHA Center at the UCR for assisting in the preparation of
assessment materials and data collection.
3. We thank the Jurupa Unified School District for their cooperation and assistance with this project.