Does Diagnostic
Math Testing Improve
Student Learning?
Julian Betts, Youjin Hahn and
Andrew Zau
This project was supported with funding from the Donald Bren Foundation.
The California Academic Partnership Program also
provided support to the authors for related research.
Published Report and Technical
Appendix are available at:

http://www.ppic.org/main/publication.asp?i=953
2
Outline




Math Diagnostic Testing Project
Examining San Diego Unified
Results from San Diego Unified
Summary and Conclusions
3
Math Diagnostic Testing Project



Tests, diagnoses student math skills, problems
– Provides fast, detailed feedback to teachers
A joint CSU, UC project (1977)
– Available to all California schools
– Ten regional centers process tests, provide
resources
Differs from California Standards Test
– Student/class intervention tool vs. school
accountability measure
– CST test results available only months later
4
MDTP Testing in State is Growing
800
2,000
700
600
1,500
500
(1000s)
400
1,000
300
500
200
100
-
-
Participating Schools
Tests Administered
5
Students Receive Detailed Reports
of Discrete Math Tasks
Here are the diagnostic results from the "M athematics Course Readiness Test", TR45K08, that
you have recentlytaken. Your Score, the Critical Level, and the Total Possible score for each topic are
reported below. The Critical Level for each topic is what MDTP considers to be the minimum number of
correct responses for you to show adequate preparation in that topic.
Your Critical Total
Score Level Possible
Congratulations! Your results indicate that you have done well in
each of the following topics:
Literals & Equations
Data Analysis, Probability, & Statistics
3
5
3
4
4
6
However, your results indicate you need review in the following
topics:
Integers
Fractions and their Applications
Decimals, their O perations & Applications; Percent
Exponents and Square Roots; Scientific N otation
5
4
5
2
6
5
6
3
8
7
9
4
Your results indicate you need substantial review in the following
topics:
Geometric Measurement and Coordinate Geometry
2
5
7
Your total score is 26 out of 45, which is 58%. We hope you find this information helpful. Please
contact your teacher for specific activities and assignments that will aid in any necessary review.
6
Outline




Math Diagnostic Testing Project
Examining San Diego Unified
Results from San Diego Unified
Summary and Conclusions
7
Dual Use of MDTP in San Diego



SDUSD testing mandated for middle, high
school
– Began 2000-2001, ended after 2007-2008
Goals
– Ensure students placed in appropriate class
– Identify students needing summer school
Individual teachers testing independently,
voluntarily
– Before, during, and after mandate period
– Most common use throughout state
8
Different Research Approaches
Required for Two Testing Methods


We compare growth of student math CST score
– In years and grades with a mandated MDTP
test, to years without testing
– A natural experiment
No similar natural experiment available for
voluntary testing
– Observed impact may stem from unusual
factors of teachers and/or of classes tested
9
Data Sources



Results from 2001-02 to 2006-07 school years,
165,863 tests
Mandated tests
– Annual student-level records
– Other relevant information: previous scores,
classroom setting, et al.
Voluntary tests
– San Diego MDTP office provided students,
schools data
– Matched to a database of complete student
records
– Identified about 80 percent of tested students
10
Participation Rates in Mandated
MDTP Testing in San Diego
(percent)
Grade 6
Grade 7
Grade 8
Grade 9
Grade 10
2001-02
0
.3
81
51
.4
2002-03
0
.3
60
43
.3
2003-04
0
88
.2
0
0
2004-05
23*
91
0
0
0
2005-06
95
88
0
0
0
2006-07
63
63
.1
0
0
*Selected schools only
Bold indicates mandated testing
11
CST and MDTP on Separate
Schedules
Voluntary MDTP
Sept.
Dec.
CST
March
MDTP,
Voluntary MDTP
May
June
Sept.
12
Outline




Math Diagnostic Testing Project
Examining San Diego Unified
Results from San Diego Unified
Summary and Conclusions
13
Mandated MDTP Clearly Boosted
Math Skills for All Participants
60
One MDTP
Test
Achievement percentile
58
56
Two MDTP
Tests
54
52
50
48
0
1
Year
2
14
Smallest Positive Effects for Those
With Lowest Initial Achievement
0.25
Based on most
recent CST
math score
0.2
Math
achievement 0.15
gain
on CST
0.1
Based on first
available CST
math score
0.05
0
Low
Medium
High
Past mathematics achievement level
15
Voluntary Testing Had Little Effect


Voluntary testing from September to March
associated with lower March CST score
– Correlatonal, not causal
– No relationship when class average of prior
math achievement scores factored in
– Teachers appear to be selectively testing
classes with weaker achievement
Voluntary testing late in prior year not linked to
current year CST score
16
How Did Mandated MDTP Have
Such a Large Effect?



Students more accurately tracked into
appropriate classes
– Explains 5–11 percent of positive effect
Better identification of students needing
summer school
– Explains 1 percent of effect
Could be that mandated testing worked better
than voluntary testing because it engaged
entire math departments in working together to
spot and improve areas of student weakness
17
Outline




Math Diagnostic Testing Project
Examining San Diego Unified
Results from San Diego Unified
Summary and Conclusions
18
Summary: Mandated Testing



District-mandated MDTP testing associated
with positive gains in subsequent year
Gains increase if students tested two years in a
row
Just under a third of effect dissipates within a
year if students tested only once
19
Summary: Voluntary Testing



Voluntary MDTP testing fails to deliver similar
benefits
In some cases, negative effects seen due to
testing on already low-achieving classes
– Reflects only decision by some teachers to
test a struggling class
Unclear whether these effects apply more
generally in other school districts
20
Lessons for California’s
Accountability System




CST not designed to fill diagnostic test role,
but…
Could be made more useful if results, feedback
made more timely, detailed
Conversion to computerized state test could
help
Room for both diagnostic and accountability
testing exists in current education landscape
21
The Future of State Testing


California one of 18 governing states in the
SMARTER Balanced Assessment Consortium
– Developing tests for the national Common
Core standards in math and English
SMARTER planning computer adaptive testing
– Could improve ability of teachers to more
timely use test results
22
Does Diagnostic
Math Testing Improve
Student Learning?
Report available at:
http://www.ppic.org/main/publication.asp?i=953
This project was supported with funding from the Donald Bren Foundation.
The California Academic Partnership Program also
provided support to the authors for related research.
Note on the use of these slides
These slides were created to accompany a
presentation. They do not include full
documentation of sources, data samples,
methods, and interpretations. To avoid
misinterpretations, please contact:
Julian Betts: 858-534-3369; jbetts @ucsd.edu
Thank you for your interest in this work.
October, 2011
Report available at:
http://www.ppic.org/main/publication.asp?i=953
24
Does Diagnostic
Math Testing Improve
Student Learning?
Supplemental Slides
Julian Betts
25
Estimated Effects from an Alternative
Statistical Model
Achievement Percentile
60
58
56
One MDTP
Test
54
Two
Consecutive
MDTP Tests
Baseline
52
50
48
0
1 Year
2
26
Summer School



Students in general were 2% more likely to
attend summer school in summers after they
had been given a mandated MDTP test
But real question is whether MDTP increases
probability that struggling students attend
A one-standard-deviation drop in math CST
scores is associated with 1.1% increase in
summer school attendance in years without
MDTP. This doubles to 2.1% in years with
mandated MDTP
– Large effect relative to average rate of
summer school attendance of 8.1%
27
Empirical Strategies
Y
(1) icgst
 Yicgs ,t 1     s  MDTPi ,t 1  Familyit  
Classicst 1  Schoolist  2  Teachericst  3 
Gradei  4  Yeart  5  Testit  6   it


Y: the standardized math CST score
Characteristics
– Family: Parental education
– Class: Annual average math class size
– School: % of students on free lunch; Asian,
white,
and other race; English learners
– Teacher: Credentials; Authorization; Master
& Bachelor's degree in math;
Race
28
Constructing the Outcome Variable



We calculate changes in Z-score.
Up until grade 7: one type of CST is offered in
each grade. We standardize the CST score by
district-wide mean and std dev, for a given
grade
Starting in grade 8: students take different
version of the math CST depending on the
subject matter they study.
29
Dep. Var: Gains in CST Score, Standardized by Grade and Test
Dep var: Gains in Std. CST score
MDTP t-1
0.219***
0.196***
0.184***
(0.019)
(0.017)
(0.017)
-0.059**
-0.076***
(0.025)
(0.027)
MDTP t-2
MDTP t-1 * MDTP t-2
0.097**
(0.042)
Constant
Observations
0.220
0.505***
0.517**
(0.199)
(0.114)
(0.114)
342571
311415
311415
Standard errors in parentheses. Standard errors are clustered by school.
* p<0.10 ** p<0.05 *** p<0.01
30
30
Results Robust to…




Exclusion of potentially endogenous teacher
characteristics and math class size
Controls for type of CST taken not just this year
but last year
Interactions between type of CST test and
grade dummies
Alternative specification:
Yicgst   i   s  Yicgs,t 1  MDTPi ,t 1  Family it  0  Class ist 1  Schoolist  2  Teacherist  3
 Yeart   Test it    it
– Estimate by Anderson and Hsiao method.
MDTP remains significant, but slightly
smaller effect
31
Instrumental Variable and Intention to
Treat Effect
MDTP t-1
(1)
(2)
(3)
Baseline
IV
ITE
0.219***
0.317***
(0.019)
(0.009)
Mandate t-1
0.218***
(0.020)
N
342571
306894
342567
32
The Effect of VMDTP
(1)
(2)
(3)
(4)
(5)
(6)
Dep var: Gains in CST std by Gr-Test
MDTP t-1
0.232*** 0.218*** 0.232***
(0.019)
VMDTP Sep-Mar this year or AprAug last year
(0.019)
-0.032**
0.055***
(0.014)
(0.015)
VMDTP Sep-Mar this year
-0.034**
-0.039**
(0.016)
(0.017)
VMDTP Apr-Aug last year
Constant
Observations
(0.021)
-0.027
0.134***
(0.021)
(0.020)
0.226
0.223
0.228
0.226
0.223
0.216
(0.200)
(0.203)
(0.200)
(0.174)
(0.166)
(0.186)
342571 342571 342571 342571 342571 342571
33
Why Less than 100% Compliance?

Linear Probability model
Pr( MDTPt  1| Mandateit  1)   i   s  Yit  Gradei ,t   Yeart   Testit    it



Schools differ significantly in compliance rates
but CST scores do not predict whether student
tested
Removes some but not all concerns about
endogeneity of MDTP testing
Instrument MDTPt-1 with dummy for whether
district mandated testing in given grade
34
Next Steps

Field survey of math teachers
– How do they use MDTP?
– Does math department hold math faculty
meetings to go over MDTP results, or does it
have other ways to let this year’s teacher
know of results on MDTP at end of prior year
35
Proportion of students who took MDTP and VMDTP
02-03
03-04
04-05
05-06
06-07
MDTP
VMDTP
02-03
(1) (2)
Grades
(3)
(4)
3
3
4
3
4
5
3
4
5
6
(1)
(2)
(3)
(4)
0
0
0
0
0
0
03-04
04-05
05-06
06-07
(5)
3
4
5
6
7
(5)
0
0
0
0
0
0
0
0
0
0
0 0.04
0
0 0.95
0 0.02 0.85
0 0.63 0.63
0.02 0.54 0.78
(6)
4
5
6*
7
8
(7) (8)
5
6
6
7
7
8
8
9
9
10
COHORT
(6) (7) (8)
0
0
0
0
0
0
0
0 0.88
0 0.07 0.81
0.23* 0.91 0
0.36 0.9 0.13
0.88 0
0
0.79 0.19 0.3
0.01 0
0
0.28 0.32 0.29
(9)
7
8
9
10
11
(9)
0.03
0.06
0.02
0.07
0
0.17
0
0.24
0
0.23
(10) (11) (12) (13)
8
9
10 11
9
10 11
10 11
11
(10)
0.6
0.04
0
0.23
0
0.22
0
0.27
(11) (12) (13)
0.43 0.03 0.01
0
0
0
0
0
0.2 0.26
0
0.23
Note: Numbers in red indicate the mandated grade/year. *
mandated at several schools.
36
Appendix:
Summary
Statistics
of selected
variables
Variable
female (1=female, 0=male)
% on free lunch
avg years at sducsd
avg years teaching
avg % bachelors in math
Mean
Std. Dev.
0.49
0.50
54.67
27.92
10.60
6.45
12.28
7.12
21.55
40.63
schl pct asian
schl pct white
schl pct hisp
schl pct black
schl pct english learners
16.80
26.11
41.36
13.21
17.36
14.45
20.36
22.55
9.64
18.96
math class size
avg yrs svc in math classes
avg sducsd yrs in math classes
avg full cred among math tchrs
avg intern in math classes
16.86
8.57
7.37
62.79
0.70
13.78
10.16
9.08
48.22
8.03
avg auth among math tchrs
avg supp auth among math tchrs
avg female tchr in math class
avg clad in math classes
avg any ma/ms in math
0.33
0.12
34.08
29.81
32.84
0.47
0.33
46.74
45.21
46.47
avg of white tchrs in math cls
avg of blk tchrs in math cls
avg of asian tchrs in math cls
avg of hisp tchrs in math cls
48.14
3.68
7.88
3.23
49.53
18.44
26.49
17.47
37
Appendix
Table 2: 7 popular course-sequences
Cohort: 8th grade in 2001-2002
8
9
10
11
freq
pct
cum
AL1
Gmtry
AL2
HsMath
1269
30.71
30.71
AL1
AL1
AL1
AL1
Gmtry
AL1
AL1
AL1
AL1
Gmtry
AL2
AL1
Gmtry
Gmtry
Gmtry
AL2
HsMath
AL1
AL2
I. Math
Gmtry
AL2
HsMath
Gmtry
956
397
250
177
153
110
23.14
9.61
6.05
4.28
3.70
2.66
53.85
63.46
69.51
73.79
77.49
80.15
Note: We followed the cohort who was 8th grade in 2001-2002 academic year up to when they became 11th grade 2004-2005.
Subject names are: AL1: Algebra 1; AL2: Algebra 2; Gmtry: Geometry; HsMath: High school math; I. Math: Integrated Math
38
Appendix: Sample reports to teachers (source: http://mdtp.ucsd.edu)
39
Appendix: Sample reports to teachers
40
41
Appendix: Sample reports to students
42