Standing on the Shoulders
of Chess Masters:
Using RCTs to Produce Evidence about
Teaching Chess in Italian Elementary Schools
Gianluca Argentin
Alberto Martini
Barbara Romano
romanob@gse.upenn.edu
Why should chess be
taught in school?
According to the
European Parliament for
several reasons
The EU Parliament claims that
“Chess can improve children’s
concentration, patience and persistence
and can develop sense of creativity,
intuition, memory, analytic and decisionmaking skills. Chess also teaches
determination, motivation and
sportsmanship”
The evidence on the effects of learning chess on
cognitive outcomes is literally scattered all over
the world: from chess association websites to
academic journals in a variety of disciplines.
However
Rarely this evidence is
based on rigorous evaluation design
such as randomized controlled trials
We will present:
1. the results of a simple RCT to measure the effects
of learning chess on math achievement (SAM)
2. the design of a more complex RCT to answer
several questions that arise when incorporating
chess in the regular curriculum (SC3)
Our true long run objective is promoting the use
of random assignment in evaluation and
education research in Italy.
PROGETTO SAM
Scacchi e
Apprendimento
della
Matematica
We recruited 33 schools that had
been involved in chess teaching before
(a non probabilistic sample,
yet nicely spread all over Italy)
Each school agreed to exclude at least
one 3° grade class, chosen
randomly
All the other 3° grade classes
were taught 30 hours of chess by
professional instructors
during 2009-10 school year
A total of 113 classes - about 2000
pupils - participated either as
treatment or control
All pupils were pre-tested and posttested using modified versions of
INVALSI tests
High level of compliance to the
design
Impact
Pre-test Score distribution
0
20
40
60
Gruppo di controllo
Gruppo sperimentale
80
Post-test Score distribution
100
0
20
40
60
80
100
Gruppo di controllo
Gruppo sperimentale
Clearly something is happening to the experimentals
Estimated effect on math scores
(in standard deviations)
ITT
Effect0:size
Mod.
nessun
0.34
0.28
controllo, tutti i casi
P-value
p value
.01***
.01***
A third of a standard deviation
is an exceptionally good result
This makes people (including us) doubt it
Is it because you included questions too
favorable for chess players?
a. Draw the snail path
b. Which shape corresponds to the snail path?
We found an effect on almost
each item on the test
Differential probability of correct response for each item
IT MUST BE THE EFFECT
OF CHEATING!
-1
0
Media
1
2
CHEATING SHOULD SHOW UP HERE, BUT IT’S EMPTY
.4
.6
.8
1
Dev. St.
Mandati
Controlli
1.2
1.4
Chess does work!
We need to prove it with an RCT
based on a probabilistic sample
of schools
Sc3
CONTROLLATA
SPERIMENTAZIONE
SCUOLA
DI SCACCHI A
(Springer = Knight in German)
Does learning chess at school (at some point
between 3rd and 4th grade) affects achievement in
mathematics and reading measured in 5th grade?
Does chess taught by professional instructors obtain
better or similar results than when based on web
teaching tools with the support of regular teachers?
Which solution is more cost-effective and
sustainable at scale?
How persistent are the effects of learning chess?
Does any beneficial effect remain one or two years
after training? Which fraction of pupils continue to
play chess and possibly benefit from it, after it is no
longer taught in class?
First question: does learning chess improve achievement in math and reading,
as measured by INVALSI standardized tests
Second
grade
INVALSI
National
test
240 schools selected with
probability prop. to size
out of 18,000 primary
schools in Italy
160 treated schools
RA
80 control schools
Fifth
grade
INVALSI
National
test
The second question:
External instructor or regular teacher+web?
Second
grade
INVALSI
National
test
80 schools
“teaching by external instructor”
RA
80 schools
“web-based with support by regular
teachers”
Fifth grade
INVALSI
National
test
The third question:
Do school-based activities increase persistency of the
effect of learning chess?
Test of
knowledge
of chess
and
frequency
of use
(early in
third
grade)
80 schools
promote chess in 5° grade
RA
80 schools
do not promote chess in 5° grade
Test of
knowledge
of chess
and
frequency
of use
(late In
fifth
grade)
The two dimensions crossed represent a factorial design
Comparisons
treatment A with treatment B
treatment A or B or C or D with controls
A+B (or C+D) treatment with controls
A+B+C+D treatments with controls
Treated
Controls
Proportion
treated
40
40
80
160
40
80
80
80
0,50
0,33
0,50
0,67
MDES
ICC=0.15
ICC=0.25
0,22
0,19
0,16
0,13
0,28
0,24
0,19
0,17
We are looking for sponsor/supporter
And for other countries willing to
“play (it again) SAM”
Anyone interested?