EVALUATING MATHEMATICS
KNOWLEDGE FOR TEACHING AS AN
OUTCOME OF PRE-SERVICE TEACHER
EDUCATION: LESSONS FROM THE
HIGH PERFORMING COUNTRIES IN
THE IEA-TEDS-M STUDY
Maria Teresa Tatto
Michigan State University
Michael Rodriguez
University of Minnesota
Improving Education through Accountability and Evaluation:
Lessons from Around the World
October 3-5, 2012
Center for American Studies
Via Michelangelo Caetani 32
00186 Rome, Italy
http://www.invalsi.it/invalsi/ri/improving_education/
IEA Teacher Education Study in Mathematics
Teacher Education and Development
Study in Mathematics
Funding provided by:
– A grant from the National Science Foundation to
Michigan State University REC-0514431;
– The IEA
– The participating countries and their institutions
– Support on item development and expert
judgment came form the international
mathematics education community.
IEA Teacher Education Study in Mathematics
2
The Teacher Education and
Development Study in Mathematics
or TEDS-M
• First cross-national data-based study
about initial mathematics teacher
education (e.g. Tatto, Senk, Rowley, &
Peck, 2011)
• 23.000 future teachers from 17 countries:
nationally representative samples
• Launched by IEA, directed by MSU, in
collaboration with ACER and National RCs
Session Outline
• Introduction to TEDS-M
• Methods: Challenges & successes
• Results
IEA Teacher Education Study in Mathematics
5
Introduction
IEA Teacher Education Study in Mathematics
6
TEDS-M Main Research Questions
What are the policies that support prospective primary and secondary
teachers’ achieved level of mathematics and related teaching
knowledge?
What learning opportunities are available to them via their
teacher education programs and requirements?
level and depth of the mathematics and
related teaching knowledge attained by
What is the
prospective primary and secondary teachers at the end of their preservice teacher education?
IEA Teacher Education Study in Mathematics
Representative Samples in 17
Participating Countries
Botswana
Canada
Chile
Georgia
Germany
Malaysia
Norway
Oman
Philippines
Poland
Russia
Singapore
Spain
Switzerland
Chinese
Taipei
Thailand
USA
TEDS-M Collected Data 2008-09:
• 15,163 Primary Future Teachers
• 9, 389 Secondary Future Teachers
• In 500 institutions which included
– 451 units preparing future primary teachers, and
– 339 units preparing future secondary teachers
• 4837 Teacher Educators
IEA Teacher Education Study in Mathematics
Four Surveys
•
•
•
•
Teacher Education Programs
Teacher Educators
Primary Future Teachers
Secondary Future Teachers
–
–
–
–
Background
Opportunities to learn
Beliefs
Assessments of knowledge
• Mathematics Content
– University level
– School level
• Mathematics Pedagogical Content
IEA Teacher Education Study in Mathematics
Challenges and Successes of
Building Measures Relevant to
Research on Teacher Education
in International Contexts
IEA Teacher Education Study in Mathematics
Validity
Validity refers to the degree to which
evidence and theory support the
interpretations of test scores entailed by
proposed uses of tests.
Standards for Educational & Psychological Testing
(AERA, APA, NCME, 1999, p. 9).
Intended Inferences
• Includes core content of OTL for teacher
preparation in Mathematics
• Captures core beliefs about teaching and
learning mathematics
• Measures are appropriate, meaningful, and
useful across international contexts
TEDS-M Conceptual Framework
Sources of Validity Evidence for
Opportunities to Learn in TE
• Each of the OTL indices must be
analyzed for psychometric
quality
–Content related evidence
–Internal structure of measures
•Score consistency (reliability)
•Measurement invariance
–Relations to other variables
Using of Rasch Scaling for
Program Measures
• Rasch scaling provides an efficient way to
estimate trait values for individuals who
have not responded to every item.
• IRT places items and persons on the
same scale – providing a tool for score
interpretation.
• Concurrent calibration across Primary
and Secondary Levels; then applied to
Educators.
Reporting & Interpretation
• OTL Topics Studied
– Original Metric: Number of Topics Studied
• OTL & Beliefs Scales
– Scales are centered at 10.0
– Indicating the “midpoint” of the rating scale
• Perceptions: Agree to Disagree
• Frequencies: Never to Often
Trends Across Measures
• All OTL and Beliefs measures with PFT vary
more at the FT level, much less so at program
level
• For most OTL measures, less than 10% of
variance occurs between programs.
• Mathematics content and mathematics
education pedagogy tend to be more
consistent within program
• Proportion of variance due to programs for
beliefs measures was very small (all less than
5%)
OTL Interpretation
• Consistency within programs and
significant variation across
programs & countries
–Evidence of coherence in
enacted curriculum
• Significant variation within
programs
–Evidence of variation in a
enacted curriculum
Design of the Assessment
A critical part of an international study is deciding what to
include in the assessment component.
 The assessment needs to be appropriate to the
target population.
 For this study, the content was selected based on
research on the mathematics knowledge needs of
teachers.
 Mathematics content knowledge – MCK
 Mathematics pedagogical content knowledge –
MPCK
 The mathematics content knowledge was based on
the content assessed by TIMSS
When trying to get comprehensive information about MCK
and MPCK, the content is greater than the time
available.
Design of the Assessment
Content Coverage




Number and Operations
Algebra and Functions
Geometry and Measurement
Data and Chance
Cognitive Levels
 Knowing
 Applying
 Reasoning
Mathematics Pedagogical Content Knowledge
 Mathematics Curricular Knowledge
 Knowledge of Planning for Mathematics Teaching
and Learning
 Enacting Mathematics for Teaching and Learning
Design
of the Assessment
Three Item Types
 Multiple-choice
 Complex Multiple-choice
 Constructed Response
Sixty minutes were allocated for the
mathematics knowledge survey.
Two-thirds of items on MCK and one-third
on MPCK
The Primary Assessment
There was not enough time for all prospective
teachers to take all items.
Items were distributed over five blocks that
would take 30 minutes
Booklets were made up of two blocks. There
were five booklets total with a spiraled design
 Blocks 1 and 2, 2 and 3, 3 and 4, 4 and 5,
5 and 1.
 Each booklet overlapped with two other
booklets.
 This allowed all of the results from the
booklets to be linked together to get results
on the full set of items.
The Secondary Assessment
The number of prospective teachers in the
sample was much less than the number of
prospective primary teachers.
 Because of the sample size needed to do
the data analysis, only three blocks of
items could be used.
The knowledge survey had three booklets with
two blocks each.
 The booklets were composed of blocks 1
and 2, 2and 3, and 3 and 1.
 The overlap between booklets allows the
results from the booklets to be combined.
Computing the Reported Scores
The item responses from the booklets were
analyzed using item response theory (IRT).
This allowed the results of all the booklets for
each of primary and secondary to be reported
on the same scale. There were four scales.




Primary MCK
Primary MPCK
Secondary MCK
Secondary MPCK
The results are reported on a scale that has an
international mean of 500 and standard
deviation of 100.
Interpretation of MCK and
MPCK
 The relative performance of groups can be
determined by comparing the results to the
international mean, but this does not tell much
about the meaning of score points.
 To help give meaning to the score scale, anchor
points were selected and performance at the
anchor points was described.
 Content experts were given items that persons at
those points could answer correctly with confidence.
 They were also given items that persons at those
points would find challenging.
 They created descriptions from those items.
 The descriptions were then cross-validated on
second sets of items.
Anchor Point Description -Primary
Persons at Point 1 on the Primary MCK scale were
likely to correctly answer items involving
 basic computations with whole numbers
 identification of properties of operations with whole
numbers
 reasoning about odd or even numbers.
They were generally able to
 solve straightforward problems using simple fractions
 to achieve success at visualizing and interpreting
standard two-dimensional and three-dimensional
geometric figures, and solving routine problems about
perimeter.
 understand straightforward uses of variables and
equivalence of expressions
 solve problems involving simple equations.
Anchor Point Description Primary
Persons at Point 1 tended to
 over-generalize
 have difficulty solving abstract problems and problems requiring
multiple steps.
They had limited knowledge of
 proportionality
 multiplicative reasoning
 least common multiples
They had difficulty
 solving problems that involved coordinates
 problems about relations between geometric figures
 reasoning about multiple statements
 relationships among several mathematical concepts (such as
understanding that there is an infinite number of rational numbers
between two given numbers)
 Finding the area of a triangle drawn on a grid
 identifying an algebraic representation of three consecutive even
numbers.
Example Item
for Primary Anchor Point 1
Options A, B, and C answered with confidence at
Anchor Point 1
Option D was challenging at Anchor Point 1
Final Report on MCK and MPCK
 Both Primary and Secondary had anchor point
descriptions for MCK (2) and MPCK (1).
 Samples of items that could be done with
confidence and those that were challenging
are available for each anchor point.
 The scales based on the international average,
the anchor points, the anchor point
descriptions, and the sample items give
meaning to the results reported on the scale.
Results
IEA Teacher Education Study in Mathematics
43
Teacher Education and
Mathematics Knowledge for
Teaching
The results of our study are presented in detail
in the TEDS-M international report (IEA – see
www.iea.nl or TEDS-M website):
Tatto, M. T., Schwille, J., Senk, S. L., Ingvarson,
L., Rowley, G., Peck, R., Bankov, K., Rodriguez,
M. & Reckase, M. (2012). Policy, Practice, and
Readiness to Teach Primary and Secondary
Mathematics in 17 Countries. Findings from the
IEA Teacher Education and Development Study
in Mathematics (TEDS-M). Amsterdam, The
Netherlands.
IEA Teacher Education Study in Mathematics
Future Primary Teachers MCK
IEA Teacher Education Study in Mathematics
Future Secondary Teachers MCK
IEA Teacher Education Study in Mathematics
What May Help Explain
These Results?
IEA Teacher Education Study in Mathematics
What may explain these findings?
Quality Assurance Mechanisms in Teacher Education in the
TEDS-M Countries
Entry into Teacher Education
Control over
Promotion
Selection
supply of
of teaching
standards
teacher
as attractive for entry to
education
career
teacher
students
education
Accreditation
of teacher
education
programs
Botswana
Canada
Chile
Chinese Taipei
Georgia
Germany
Malaysia
Norway
Oman (Sec)
Philippines
Poland
Russian Fed.
Singapore
Spain (Prim)
Switzerland
Thailand
USA
Key
Strong QA procedures
Entry to the
teaching
profession
Relative
strength of
QA system
Moderate
Moderate/High
Low
High
Low
Moderate/High
Moderate
Moderate/Low
Low
Low
Moderate
Moderate
High
Moderate/Low
Moderate
Low
Moderate/Low
Moderately strong QA
procedures
Limited QA procedures
A closer look at the countries
where future teachers had the
highest scores in our
assessments
IEA Teacher Education Study in Mathematics
IEA Teacher Education Study in Mathematics
TIMSS, PISA & TEDS-M
IEA Teacher Education Study in Mathematics
Regressions Primary Programs
IEA Teacher Education Study in Mathematics
Regressions Secondary Programs
IEA Teacher Education Study in Mathematics
HLM Model of Future Teacher MCK and PCK, Given Future
Primary Teacher Level and Program Level Characteristics
IEA Teacher Education Study in Mathematics
What may explain these findings?
Individual differences among future
teachers within programs
The following characteristics of Future teachers are
positively correlated with higher levels of
knowledge:
• socioeconomic status, age, gender and previous
school performance:
that is, wealthier, younger future teachers who
reported that they did well in high school tended
to perform better in the TEDS-M knowledge tests,
and males do better than females (except for
Russia).
What may explain these findings?
Contextual and programmatic
influences
The following characteristics of education programs
are positively correlated with higher levels of
knowledge.
• Opportunity to learn
Primary level - Advanced level school
mathematics, specifically function, probability
and calculus
Secondary level - University level mathematics,
specifically geometry, and the opportunity to
read research in teaching and learning
….. continued
What may explain these findings?
Contextual and Programmatic
Influences
• A conceptual, problem solving and active learning
orientation view of mathematics
• Coherent curricular content and orientation
• Rigorous selection standards
• Demanding and sequential (versus repetitive)
university and school mathematics curriculum
• Formative evaluations at critical points in the
program (written and oral)
• Stringent graduation requirements
In sum…
• To date research on the effects of teacher education has
been underfunded – NSF and IEA support changed this,
more high quality research by teacher educators is
needed.
• TEDS-M is a first step in the direction to better
understand what makes a difference in preparing
effective and knowledgeable teachers.
• Some critics of teacher education believe it is possible
to bypass colleges of education and prepare teachers in
an easier, faster way but TEDS-M does not support
that: the countries that prepare the most
knowledgeable teachers rely on university-based
teacher education programs.
IEA Teacher Education Study in Mathematics
Future Research
FIRSTMATH, a study of novice teachers’ development of
mathematical knowledge for teaching
FIRSTMATH will explore:
• the connections between what teachers bring with them
when they begin to teach, and what is learned on the
job as it concerns knowledge, skills and curricular
content; and
• the degree to which standards, accountability and other
similar mechanisms operate to regulate the support that
beginning teachers of mathematics receive during their
first years of teaching.
For more information on TEDS-M and FIRSTMATH consult
the websites http://teds.educ.msu.edu/ and
http://firstmath.educ.msu.edu/
• FIRSTMATH next meeting Cork, Ireland November 2530, 2012
IEA Teacher Education Study in Mathematics