Large scale quantitative studies in
educational research
Nic Spaull
SAERA conference
Presentation available online: nicspaull.com/presentations
|
Durban
| 12 August 2014
Objectives of the workshop
• For participants to leave with…
1. A good idea of what large-scale data exist in SA
and which assessments SA participates in.
2. To appreciate why we need them
3. Which areas of research are most amenable to
analysis using quantitative data?
(The focus here is on non-technical, usually descriptive, analyses of large-scale
education data. There is obviously an enormous field of complex multivariate
research using quantitative data. See Hanushek and Woessman, 2013)
1. What do we mean by “large-scale
quantitative research”?
1. What the heck do we mean by
“large-scale quantitative research” ?
Firstly, what do we mean when we say “large-scale
quantitative studies”
– Large-scale: usually implies some sort of representivity of an
underlying population (if sample-based) or sometimes the
whole population.
– There are two “main” sources of large-scale data in
education
1. Assessment data and concomitant background info
(PIRLS/TIMSS/SACMEQ/ANA/Matric/NSES)
2. Administrative data like EMIS, HEMIS, PERSAL etc..
– Quantitative: The focus is more on breadth than depth.
• As an aside in the economics of education, qualitative research that
uses numerical indicators for the 15 (?) schools it is looking at would
not really be considered quantitative research. The focus is still
qualitative.
Personal reflections – please challenge me on these…
Qualitative
Number of schools
Quantitative
Usually a small number of schools (1- Usually a large number of schools
50?) selected without intending to be (250+) that may be representative of
representative (statistically speaking)
an underlying population or not
Over-arching interest
Depth over breadth
Breadth over depth
Can make populationwide claims?
No. This is one of the major
limitations.
Yes. This is one of the major
advantages
Scope of research
Numerical summaries of
data
Often quite broad but shallow (one
Usually very specific getting detailed
dataset might be analysed from a SLM
information pertinent to the specific
perspective, a content perspective, a
research topic.
resourcing perspective etc.)
Less important
More important
1. What are we talking about?
A.
Types of research questions that are amenable to quantitative research:
–
–
–
–
–
B.
How many students in South Africa are literate by the end of Grade 4?
What proportion of students have their own textbook?
What do grade 6 mathematics teachers know relative to the curriculum?
Which areas of the grade 9 curriculum do students battle with the most?
How large are learning deficits in Gr3? Gr6? Gr9?
Types of research questions that are LESS amenable to quantitative research:
– Which teaching practices and styles promote/hinder learning?
– Questions relating to personal motivation, school culture, leadership style etc. (all of
which require in-depth observation and analysis)
– All the ‘philosophical’ areas of research: what is education for? What is knowledge? Says
who? Who should decide what goes into the curriculum? How should they decide?
Should education be free?
That being said, researchers do focus on some of “type-B” questions (nonphilosophical ones) using quantitative data – (and have often made important
contributions) but the scope of questions is usually quite limited, but the
breadth/coverage and ability to control for other variables often makes the
analysis insightful
1. What are we talking about?
• To provide one example. If we look at
something like school leadership and
management (SLM), there are various
approaches to researching this including:
– In-depth study of a small number (15) of schools
(something like the SPADE analysis of Galant &
Hoadley)
– Using existing large-scale data sets to try and
understand how proxies of SLM are related to
performance. To provide some examples…
The above analysis is taken from Gabi Wills (2013)
The above analysis is taken from Gabi Wills (2013)
Sample-based
Censusbased
Number
of
schools?
Number of
students?
Comparable
over time?
TIMSS 1995, 1999,
2003, 2011
-
285
11969
Yes
-
392
9071
Yes
-
92
3515
Sort of
341
15744
NA
-
2340
54
Sort-of
ANA
2011/12/13/1
4
24
7mil
Definitely not
Cross-national studies SACMEQ 2000, 2007,
2013
of educational
achievement
PIRLS 2006, 2011
(Eng/Afr only)
prePIRLS 2011
Systemic Evaluations
2004 (Gr6), 2007
(Gr3)
National assessments
(diagnostic)
National assessments
(certification)
Verification-ANA
2011, 2013 (Gr 3 & 6)
2164 (125/
prov)
NSES* Gr3 (2007) Gr4
(2008) Gr5 (2009)
266
24000
(8383 panel)
6591
about 550,000
-
Matric
No
*Number of schools and students is for the most recent round of assessments
Yes
(+ longitudinal)
Differences between national assessment
and public exams
Like
TIMSS/PIRLS/S
ACMEQ
Like matric
Source: Greaney & Kellaghan (2008)
There are also other assessments
which SA doesn’t take part in…
School-based
• PISA: Program for International Student Assessment [OECD]
• ICCS: International Civic and Citizenship Education Study
[IEA]
Home-based
• IALS: International Adult Literacy Survey [OECD]
• ALLS: Adult Literacy and Life Skills Survey [OECD]
• PIAAC: Programme for the International Assessment of
Adult Competencies [OECD]
For more information see: http://www.ierinstitute.org/
Source: IERI Spring Academy 2013
Source: IERI Spring Academy 2013
Source: IERI Spring Academy 2013
An aside on matrix sampling…
Because one
1.
2.
can only test students for a limited amount of time (due to practical reasons and cognitive fatigue),
and because one cannot cover the full curriculum in a 2 hour test (at least not in sufficient detail for
diagnostic purposes)
It becomes necessary to employ what is called matrix sampling.
•
•
•
•
•
•
If you have 200 questions that cover the full range of the maths curriculum you could split
this into 20 modules of 10 questions.
If a student can cover 40 questions in 2 hours then they can write 4 modules.
Different students within the same class will therefore write different tests with overlapping
modules.
Matrix sampling allows authorities to cover the full curriculum and thus get more insight into
specific problem-areas, something that isn’t possible with a (much) shorter test.
TIMSS/PIRLS/PISA all employ matrix sampling. SACMEQ 2000 and 2007 did not employ
matrix sampling (all children wrote the same test) but from 2013 I think they are doing
matrix sampling as well.
This highlights one of the important features of sample-based assessments: the aim is NOT
to get an accurate indication of any specific child or specific school but rather some
aggregated population (girls/boys/provinces/etc.)
TIMSS 2007 Test Design
Booklet
1
2
3
4
5
6
7
8
9
10
11
12
13
14
TIMSS 2007 Booklets
Pos 1
Pos 2
Pos 3
Pos 4
M01
M02
S01
S02
S02
S03
M02
M03
M03
M04
S03
S04
S04
S05
M04
M05
M05
M06
S05
S06
S06
S07
M06
M07
M07
M08
S07
S08
S08
S09
M08
M09
M09
M10
S09
S10
S10
S11
M10
M11
M11
M12
S11
S12
S12
S13
M12
M13
M13
M14
S13
S14
S14
S01
M14
M01
The IEA/ETS Research Institute (www.IERInstitute.org)
11
PIRLS 2006 Test Design
•
•
•
•
10 passages; 5 literary & 5 informational
126 items; 167 score points
Multiple-choice and constructed-response
questions
PIRLS Reader
PIRLS 2006 Booklets
1
2
3
4
5
6
7
8
Part 1 L1 L2 L3 L4
I1
I2
I3
Part 2 L2 L3 L4
I2
I3
I1
The IEA/ETS Research Institute (www.IERInstitute.org)
9
10
11
12
R
I4 L1
I2
L3
I4
L5
I4 L1 I1
L2
I3
L4
I5
13
Sample-based assessments (cont.)
• The aim of sample-based assessments is to be able to gain insight
(and make statements) that pertain to an underlying population
AND NOT the sampled schools.
• For example in SACMEQ the sample was drawn such that the
sampling accuracy was at least equivalent to a Simple Random
Sample of 400 students which guarantees a 95% confidence
interval for sample means that is plus or minus 1/10th of a student
standard deviation (see Ross et al. 2005).
– This is largely based on the intra-class correlation coefficient (ICC)
which is a measure of the relationship between the variance between
schools and within schools.
– In South Africa this meant we needed to sample 392 schools in
SACMEQ 2007
• Important to understand that there are numerous sources of error
and uncertainty, especially sampling error and measurement error.
Consequently one should ALWAYS report confidence intervals or
standard errors.
Sample-based assessments (cont.)
• Once you know the ICC and therefore the
number of schools you need to sample, you
need a sampling frame (i.e. the total number
of schools).
• One can also use stratification to ensure
representivity at lower levels than the whole
country (i.e. province or language group)
• Randomly select schools from sampling frame.
• For example, for the NSES 2007/8/9….
Brown dots = former black schools
Blue dots = former white schools
Purple dots = school included in NSES
(courtesy of Marisa Coetzee)
What kinds of administrative
data exist?
• Education Management Information Systems (EMIS)
– Annual Survey of Schools
– SNAP
– LURITZ. System aimed at being able to identify and follow
individual learners using unique IDs
– SA-SAMS
•
•
•
•
•
HEMIS – EMIS but for higher education
PERSAL – payroll database
School Monitoring Survey
Infrastructure survey
ECD Audit 2013
Overview
• Main educational datasets in South Africa:
•
•
•
•
•
•
•
•
•
PIRLS
TIMSS
1995
1999
SACMEQ
2000
V-ANA
ANA
NSES
EMIS (various)
Matric (annual)
Household surveys (various
2006
2002
2011
2011
2007
2013
2011
2011 2012
2007 2008 2009
•
When and Who:
•
•
•
PIRLS 2006 (grade 4 and 5)
PIRLS* 2011 (grade 5 Eng/Afr only)
prePIRLS (grade 4)
Examples of how we can use it?
•
•
•
Issues related to LOLT
Track reading performance over time
International comparisons
.004
.003
.001
0
•
Progress in International Reading and Literacy
Study
Tests the reading literacy of grade four children
from 49 countries
Run by CEA at UP on behalf of IEA
(http://timss.bc.edu/)
0
200
400
reading test score
600
800
English/Afrikaans schools
African language schools
600
prePIRLS reading score 2011
•
.002
What:
kdensity reading test score
PIRLS
.005
PIRLS 2006 – see Shepherd (2011)
560
576
531 525
520
480
440
452 443
436 429 428 425
461 463
407
400
395 388
360
320
280
240
Test language
prePIRLS 2011 – see Howie et al (2012)
PIRLS South Africa
2006
Grade 4
11
Languages
Grade 5
11
Languages
PIRLS South Africa
2011
prePIRLS
PIRLS
Grade 4
Grade 5
11
Languages
Afrikaans
English
Afrikaans
English
isiNdebele
isiXhosa
isiZulu
prePIRLS 2011
Grade 4
Sepedi
Sesotho
Setswana
siSwati
Tshivenda
Xitsonga
PIRLS 2011
Grade 5
Afrikaans
English
prePIRLS 2011 Benchmark Performance by Test Language
47
Xitsonga
53
53
Tshivenda
47
24
siSwati
0
0
76
0.25
Setswana
34
66
0.1
Sesotho
36
64
0.1
57
Sepedi
43
29
isiZulu
71
38
isiXhosa
0.8
0.4
62
31
isiNdebele
0
69
0.2
English
10
90
19
Afrikaans
12
88
15
South Africa
29
Did not reach
High International Benchmark
71
6
Low International benchmark
Advanced International benchmark
Intemediate International Benchmark
.008
TIMSS 2003 Maths – see Taylor (2011)
•
When and Who:
•
•
•
TIMSS 1995, 1999 (grade 8 only)
TIMSS 2002 (grade 8 and 9)
TIMSS 2011 (grade 9 only)
Examples of how we can we use it?
•
•
•
Interaction between maths and science
Comparative performance of maths and
science achievement
Changes over time
.004
0
0
200
400
Grade 8 mathematics score
South Africa Quintile 5
Chile Quintile 5
Singapore Quintile 5
600
800
Chile
Singapore
600
560
520
480
440
400
360
320
280
240
200
Middle-income countries
TIMSS 2011 Science – see Spaull (2013)
Quintile 1
Quintile 2
Quintile 3
Quintile 4
Quintile 5
Independent
•
Trends in International Mathematics and
Science Study
Tests mathematics and science achievement of
grade 4 and grade 8 pupils
Run by HSRC in SA on behalf of IEA
(http://timss.bc.edu/)
Russian Federation
Lithuania
Ukraine
Kazakhstan
Turkey
Iran, Islamic Rep. of
Romania
Chile
Thailand
Jordan
Tunisia
Armenia
Malaysia
Syrian Arab Republic
Georgia
Palestinian Nat'l Auth.
Macedonia, Rep. of
Indonesia
Lebanon
Botswana (Gr 9)
Morocco
Honduras (gr 9)
South Africa (Gr 9)
Ghana
•
.002
What:
TIMSS 2011 Science score
Density
.006
TIMSS
South Africa (Gr9)
TIMSS 2011
South African mathematics and science performance in the Trends in International Mathematics and
Science Study (TIMSS 1995-2011) with 95% confidence intervals around the mean (Spaull, 2013)
480
440
400
360
TIMSS score
320
280
240
352
160
120
443
433
200
276
275
264
285
1995
1999
2002
2002
332
260
243
244
268
1995
1999
2002
2002
80
40
0
Grade 8
2011
Grade 9
TIMSS Mathematics
2011
TIMSS
middleincome
country
Gr8
mean
Grade 8
2011
Grade 9
TIMSS Science
2011
TIMSS
middleincome
country
Gr8
mean
.0 08
SACMEQ III – see Spaull (2013)
•
•
Southern and East African Consortium for
Monitoring Educational Quality
Tests the reading and maths performance of
grade six children from 15 African countries
Run by DBE – Q.Moloi
(http://www.sacmeq.org/)
0
•
.0 02
What:
.0 04
D en sity
.0 06
SACMEQ
0
Mean
950
•
•
•
900
SACMEQ II – 2000 (grade 6)
SACMEQ III – 2007 (grade 6)
SACMEQ IV – 2013 (grade 6)
Examples of how can we use it?
•
•
•
Regional performance over time
Teacher content knowledge
Understanding the determinants of
numeracy and literacy
Maths-teacher mathematics score
When and Who:
200
800
Poorest 25%
Second poorest 25%
Second wealthiest 25%
Wealthiest 25%
Lower bound confidence interval (95%)
1000
Upper bound confidence interval (95%)
KEN
850
ZIM
UGA
TAN
SEY
SWA
BOT
NAM
MALSOU
LESZAMMOZ
800
750
700
650
400
600
Learner Reading Score
Q5-SOU
Q4-SOU
Q3-SOU
Q2-SOU
Q1-SOU
ZAN
600
SACMEQ III – see McKay & Spaull (2013)
SACMEQ III (Spaull & Taylor, 2014)
ANA – see Spaull (2012)
School Categorisation by District (KZN)
ANA
60
40
0
20
P ercent
80
100
Universal ANA 2011
What:
•
•
•
OB
Annual National Assessments
Administrative data on enrolments, staff,
schools etc.
Collected by DBE
ON
JE
NI
VR
YH
E ID
S IS
ON
I
N
KE
LA
EN
OW
KE
NG
ET
HU
PA
P IN
OT
EM
IL E
MB
E
AM
PO
AJ
RT
UB
A
E
SH
PS
TO
I
U
TH
OV
YA
DL
Z IN
UN
G
UM
UN
G
UM
NE
Dysfunctional schools: <30%
Underperforming schools: 30-40%
Poor schools: 40-50%
Good schools: 50-60%
Great schools: 60-70%
Excellent schools: 70%+
Correlation Between Avg. School Gr3 and Gr6 Numeracy Score (KZN)
interview)
1 00
80
60
40
20
0
20
40
60
80
Average school grade 6 numeracy score
Correlation Between Avg. School Gr3 and Gr6 Numeracy Score (WC)
1 00
U-ANA 2011
0
•
•
•
0
80
•
•
Analyse performance at primary grades,
potentially at the micro-level (district/circuit)
Create indicators for dashboards
Report cards (once ANA is externally evaluated
at one grade)
Early indicators of problems/deficits
Planning at primary school level
Serious comparability problems between ANA
2011 and ANA 2012 (see SVDB and Spaull
S cho ol a ve ra ge g ra de 3 nu m e racy score
•
60
Examples of how can we use it?
40
Grades 1-6 and 9 (maths and language - FAL
and HL)
20
A verag e scho ol g rad e 3 n um e ra cy sco re
•
LA
School categorization (Average school numeracy and literacy score)
U-ANA 2011
When and Who:
UM
0
20
40
School average grade 6 numeracy score
60
80
ZI
ANA
Language by
grade/quintile (KZN)
Race Distribution by Quintile (KZN)
U-ANA 2011
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
1
3
3
1
14
11
8
3
100
100
98
Other
Asian
Indian
91
White
65
Coloured
Black
Q1
Q2
Q3
Q4
Q5
Correlation 0.82
Correlation 0.51
EMIS – see Taylor (2012)
1200000
EMIS
1000000
800000
600000
What:
•
•
•
400000
Education Management Information System
Administrative data on enrolments, staff,
schools etc.
Collected by DBE
(http://www.education.gov.za/EMIS/tabid/57/
Default.aspx)
200000
0
grade 10
The ratio of grade 2 enrolments ten years prior to matric to matric passes by province
When and Who:
•
Various
Examples of how can we use it?
•
•
Analyse flow-through
Create indicators for dashboards
–
•
•
Grade 12
PTR, school size, LOLT etc
Provide an up-to-date and accurate
picture of elements of the education
system
Planning
EMIS – see Taylor (2012)
“In 1999 and 2000 the numbers enrolling in grade
1 dropped substantially, by about half a million.
Crucially, it is these cohorts who make up the bulk
of the matric class of 2011. This was due to a
change in the policy stipulating age of entry
into grade 1. According to Notice 2433 of 1998,
it was stipulated that children should only be
allowed to enrol in grade 1 if they turned seven in
that calendar year. Therefore children who
previously might have entered in the year in which
they turned six were now not allowed to.
The
policy change was announced in October 1998
and schools were expected to comply by
January 2000. This would explain why grade 1
enrolments declined somewhat in 1999 and
then again even more so in 2000. The reason
why numbers declined as the policy was phased
in is that some children who turned 7 in the
2000 calendar year had already entered in the
previous year under the previous policy. “
- Taylor 2012
EMIS – see Taylor (2012)
Matric
Grade 10 (2 years earlier)
Grade 12
Those who pass matric
Pass matric with maths
1200000
60%
1000000
50%
800000
40%
600000
30%
400000
20%
200000
10%
•
•
•
Grade 12 examinations results
Performance data
Collected by DBE
When and Who:
•
Various
Examples of how can we use it?
•
•
Analyse subject choices/combinations
Create indicators for dashboards
–
–
•
•
Number of students
What:
0
0%
Matric 2008 Matric 2009 Matric 2010 Matric 2011
(Gr 10 2006) (Gr 10 2007) (Gr 10 2008) (Gr 10 2009)
% taking maths/science
Proportion of Gr 8’s passing matric
Relatively trustworthy and regular
indication of student outcomes in SA.
Planning
EMIS – see Taylor (2012)
Proportion of matrics (%)
Proportion of matrics taking mathematics
Household
Surveys
What:
•
•
•
Grade 12 examinations results
Performance data
Collected by DBE
When and Who:
•
Various
Examples of how can we use it?
•
•
Research
Link education to other social outcomes
like employment and health
HH-Surveys – see Taylor (2012)
Household Surveys
Employment/LFA Rate for
18 - 24 -year -olds
Percentage of youth in employment by highest educational attainment (Van Broekhuizen, 2013)
80%
70%
60%
50%
40%
30%
20%
10%
All Youth
Youth with Less than Matric
With Matric
Youth With Diploma
Youth With Degree
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
1999
1998
1997
1996
Working-Age Population
100%
80%
60%
40%
20%
With Less Than Matric
With Matric
With Diploma
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
1999
1998
1997
1996
0%
1995
Proportion of youth with
Qualification
1995
0%
With Degree
Composition of 18 - 24-year-olds by highest level of education completed (Van Broekhuizen, 2013)
Some other research…
(Discuss if time permits)
.005
Kernel Density of Literacy Score by Race (KZN)
.0 06
.0 04
D en sity
.003
.0 02
.002
.0 1
0
0
0
.0 05
.001
.0 15
kdensity reading test score
.004
.0 2
U-ANA 2011
D en sity
.0 08
Context: low and unequal learner
performance
0
20
40
60
Literacy score (%)
80
0
100
0
Black
White
Indian
Asian
200
400
reading test score
600
200
English/Afrikaans schools
African language schools
400
600
Learner Reading Score
800
800
Poorest 25%
Second poorest 25%
Second wealthiest 25%
Wealthiest 25%
1000
.025
PIRLS / TIMSS / SACMEQ / NSES / ANA / Matric… by Wealth / Language / Location / Dept…
Kernel Density of School Literacy by Quintile
.0 1
.0 2
D e n s ity
.015
.01
0
0
.005
Density
.0 3
.02
.0 4
U-ANA 2011
0
0
20
40
60
Numeracy score 2008
Ex-DET/Homelands schools
80
Historically white schools
100
20
40
60
Average school literacy score
80
Quintile 1
Quintile 2
Quintile 3
Quintile 4
100
Quintile 5
47
Comparing WCED Systemic
Evaluation and DBE ANA WC 2011
Quantifying learning deficits in
Gr3
Figure 1: Kernel density of mean Grade 3 performance on Grade 3 level
items by quintiles of student socioeconomic status (Systemic Evaluation
2007)
.01
.01 5
.02
16%
51%
.00 5
K e rn el d e nsity o f G ra d e 3- le vel s core s
.02 5
(Grade-3-appropriate level)
11%
0
Only the top 16% of grade 3 students are
performing at a Grade 3 level
0
10
20
30
40
50
60
70
80
90
Systemic 2007 Grade 3 mean score (%) on Grade 3 level items
Quintile 5
•
(Spaull & Viljoen, 2014)
Quintile 1-4
Following Muralidharan & Zieleniak (2013) we
classify students as performing at the gradeappropriate level if they obtain a mean score of
50% or higher on the full set of Grade 3 level
questions.
49
NSES question 42
NSES followed about 15000 students (266 schools) and tested them in Grade 3 (2007),
Grade 4 (2008) and Grade 5 (2009).
Grade 3 maths curriculum:
“Can perform calculations
using appropriate symbols to
solve problems involving:
division of at least 2-digit by
1-digit numbers”
100%
Even at the end of Grade 5
most (55%+) quintile 1-4
students cannot answer
this simple Grade-3-level
problem.
90%
35%
80%
70%
59%
57%
57%
55%
60%
50%
40%
13%
14%
14%
15%
20%
13%
10%
12%
12%
10%
16%
19%
17%
17%
Q1
Q2
Q3
Q4
30%
13%
Still wrong in Gr5
14%
Correct in Gr5
Correct in Gr4
Correct in Gr3
39%
0%
“The powerful notions of ratio, rate
and proportion are built upon the
simpler concepts of whole number,
multiplication and division, fraction
and rational number, and are
themselves the precursors to the
development of yet more complex
concepts such as triangle similarity,
trigonometry, gradient and calculus”
(Taylor & Reddi, 2013: 194)
Q5
Question 42
(Spaull & Viljoen, 2014)
50
Insurmountable learning deficits: 0.3 SD
South African Learning Trajectories by National Socioeconomic Quintiles
Based on NSES (2007/8/9) for grades 3, 4 and 5, SACMEQ (2007) for grade 6 and TIMSS (2011) for grade 9)
13
12
11
10
Effective grade
9
8
Quintile 1
7
Quintile 2
6
Quintile 3
5
Quintile 4
4
Quintile 5
Q1-4 Trajectory
3
Q5 Trajectory
2
1
0
Gr3
Gr4
Gr5
(NSES 2007/8/9)
Gr6
(SACMEQ
2007)
Gr7
Gr8
Projections
Gr9
(TIMSS 2011)
Gr10
Gr11
Gr12
Projections
Actual grade (and data source)
(Spaull & Viljoen, 2014)
51
Data and analysis
• In order to answer research questions and engage with the data requires
some level of analytic proficiency with a statistical software package like
STATA or SPSS (or R if you are hardcore)
• Education faculties in South Africa really need to up their game as far as
quantitative analysis is concerned. For whatever reason there seems to be
an anti-empirical, anti-quantitative bias across the board. This filters
through into course-load priorities and expectations (or lack of
expectations) on graduate students.
• Without an ability to interact with a large data set and do BASIC data
analysis any graduate student’s research opportunities are severely (and
unnecessarily) limited (the same applies to faculty members)
• SALDRU (UCT) runs a free online STATA course to teach the basics of data
analysis
–
–
http://www.saldru.uct.ac.za/training/online-stata-course
There is also a two-week ”UCT Summer training Programme in Social Science research Using Survey Data”
run in January every year and well worth going to if you already have a basic background in statistics
Conclusion
• Data is essential for making informed decisions
• To be able to use these data sets requires some level of
analytic proficiency. Basic proficiency can take as little as 4
months but is infinitely valuable.
• Nationally representative datasets allow us to draw
conclusions for each province and the whole country –
something that is not possible from small local studies.
• DBE has access to a wealth of useful but under-utilized data
– ANA, EMIS, MATRIC, HH-SURVEYS (also PERSAL & SYSTEMIC)
• Many datasets are publicly available on request
– SACMEQ, TIMSS, PIRLS (SACMEQ 2013 soon to be available)
• “Without data you are just another person with an opinion”
– Andreas Schleicher
References and useful websites
•
•
•
•
•
•
•
•
Fleisch, B. (2008). Primary Education in Crisis: Why South African Schoolchildren underachieve
in reading and mathematics (pp. 1–162). Cape Town: Juta & Co.
Greaney, V., & Kellaghan, T. (2008). Assessing national achievement levels in education (Vol.
1). World Bank Publications.
Reddy, V., Prinsloo, C., Visser, M., Arends, F., Winnaar, L., & Rogers, S. (2012). Highlights from
TIMSS 2011: The South African perspective. Pretoria.
Ross, K. N., Dolata, S., Ikeda, M., Zuze, L., & Murimba, S. (2005). The Conduct of the SACMEQ
II Project in Kenya. Harare.
Taylor, N., Van der berg, S., & Mabogoane, T. (2013). What makes schools effective? Report of
the National School Effectiveness Study. Cape Town: Pearson.
Taylor, S., & Yu, D. (2009). The importance of socioeconomic status in determining educational
achievement in South Africa (No. 1). Stellenbosch.
Van der berg, S., Burger, C., Burger, R., De Vos, M., Du Rand, G., Gustafsson, M., … Von Fintel,
D. (2011). Low quality education as a poverty trap. Stellenbosch.
http://www.sacmeq.org/
www.oecd.org/pisa
http://timssandpirls.bc.edu
Group exercise
• Last 45 minutes
– Split into groups of 5 (8 groups)
– Using questionnaires provided, come up with at least 5
research questions that could (potentially) be answered
using that data
1.
2.
3.
Explain which variables you would use and how (what would the
graph/table look like or be populated with? Sketch the axes)
Why did you choose those research questions?
Which other large-scale data do you think you could look at to
further investigate the issue?
Thank you
www.nicspaull.com/research
nicholasspaull@gmail.com
@NicSpaull
Difference between TIMSS & PISA
Difference between TIMSS & PISA