The Implementation of National
Educational Standards in
Austria
Michael Bruneforth
BIFIE - Austrian Federal Institute for Education Research, Innovation
and Development of the Austrian School System
EURASIAN EDUCATIONAL DIALOGUE
17.-19. April, Yaroslavl
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
BIFIE - Austrian Federal Institute for Education Research,
Innovation and Development of the Austrian School
System
Areas of work
 International Studies (PISA, TIMSS, PIRLS, TALIS)
 Implementation of national education standards
 Formative and summative Evaluation of standards
 Central examination for upper secondary education preparing for
tertiary education
 National Education Reports
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Spotlight on Austrian education
Very rich country (4th highest GDP per Capita in EU)
High expenditure per student (140% of EU Average) in
primary and secondary education
Low student teacher ratio and class size
But PISA shock in 2003 and 2009:
 Very low PISA results in reading (470, similar to Russia 459), no
EU country scores less
 Strong relation between education outcomes and family
background
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Changing in paradigm in educational
governance
Since more than a decade there is a change in
governance of schools and education in German speaking
countries from input oriented control to evidence based
output oriented governance and extension of school
autonomy
In Austria at the centre is shift towards competence
oriented teaching based on the definition of nationwide
compulsory education standards and a mechanism of
formative and summative evaluation
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Legal Basis and Conditions
From 2004: pilot phase with selected pilot schools
2008: alteration of the law of teaching in schools to allow
for introduction of educational standards
2009: educational standards become compulsory for
German, Math and English
2009 (grade 8) and 2010 (grade 4): baseline-tests in
samples of about 10.000 students each
From 2012 onwards yearly assessment of national
standards (one subject per year)
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Annex of the legal regulation
Competency Models and descriptors are formulated as
Can-Do-Statements
Competency Models and descriptors are legally
established
Educational Standards can be interpreted as mandatory
and agreed upon social objectives of instruction
Educational Standards are derived from national
curricula and clearly define learning outcome.
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Areas covered
German (language of
instruction) – grades 4 & 8
- Listening & Speaking
- Reading
- Writing texts
- Spelling/Grammar
Math – grades 4 & 8
4 content dimensions
4 process dimensions
English Assessment:
Common European
Framework of Reference
for Languages (CEF)
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
English, grade 8
- Listening
- Reading
- Speaking
- Writing
Competencies are …
„fashionable term with a vague meaning“ (Weinert 2001)
„cognitive prerequisites to coping with a specific range of
situations“ (Klieme, Hartig & Rauch, 2008)
„cognitive abilities and skills available to/acquired by
individuals that enable them to solve specific problems, as
well as the related motivational, volitional and social
readiness and skills to utilize the solutions successfully
and responsibly in variable situations“ (Weinert, 2002)
 BIFIE use: subject (domain) specific competency
models, focusing on cognitive skills and proficiencies
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Competency Models
Models of competence structures
 Dimensions of competence: E.g. TIMSS, grade 8: 4 content
domains (number, algebra, geometry, data and chance); 3
cognitive domains (knowing, applying, reasoning)
Models of competence levels
 Describe specific skills and proficiencies usually shown by
individuals located at specific points of the ability continuum
 Used to provide a criterion-referenced interpretation of
measurement results
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Competency Models
Models of competence structures
Models of competence levels
Development models
Models describing the acquisition of skills and
proficiencies
May be linked to neurological/psychological
theories of development
Only a few exist
Would be great to have a theoretical
foundation for tests in different grades! (IKM –
BIST)
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
EXAMPLE
MATHEMATICS, GRADE 8
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
M8 – Competency Model
Process domains
(Handlungsbereiche)
Content domains
(Inhaltsbereiche)
Level of complexity
(Komplexitätsbereiche)
demonstrating, modelling
numbers and units
use of basic skills and
knowledge
calculating, operating
variable, functional
dependency
build connections/connect
interpreting
geometric figures and
shapes
reflect, use knowledge of
reflection
explaining, reasoning
statistical representations,
measures of central
tendency and variance
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
M8 – Competency Model
Complexity
Competency (H3, I2, K2)
Content
Process
The 4x4 grid leads to 16 nodes, using complexity as a 3rd
dimension = 48 nodes
Not very handy and the theoretical concept of complexity does
not lend itself to assessment
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Example: Competency Model M8
P: „demonstrating, modelling “ C: „numbers and measures“
Formulated as can-do-statements: The students are able to
 transform given
arithmetical data into (another) mathematical
P1/C1/K1
representation. In doing so, direct use of basic skills is
necessary
P1/C1/K2
transform given arithmetical data into (another) mathematical
representation. In doing so, connections to other mathematical
contents (terms, definitions, representations) or activities have
to be established
P1/C1/K3
make and evaluate statements about the appropriateness as
well as weaknesses and strengths of different mathematical
representations (models) of arithmetical data
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Educational Standards –
The Austrian Version
Three areas of activities
 Implementation of standards at school and capacity building for
teacher: Support of teachers and school heads in implementation
 Formative evaluation to support teachers diagnostic: On-line
evaluation system to be used by teachers
 Census to monitor and summative evaluation at system, subsystem, school and classroom level
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Educational Standards –
The Austrian Version
School-specific development of instruction
 Mandatory educational standards
 Census-survey of all students and schools
 Feedback to all students, teachers, schools
 Feedback moderation („Rückmeldemoderation“)
 Stimulation of quality development and support by MoE
 Strong linkage of monitoring and practical school and instructional
development
 Strengthening of schools, school management and regional quality
development
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Main functions of standards
according to the legal regulation
Output orientation: Sustainable output orientation in
planning and implementation of instruction
Promotion („Förderfunktion“): Diagnosis and specific
individual remediation and challenge
Evaluation: reliable information about the output of
instruction, schools and education system
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Implementation of Standards
Supporting materials for instruction
 Praxishandbücher
 Themenhefte
 Aufgabenbeispiele, Interaktive Beispiele
 Best-Practice-Beispiele
Integration with teacher training institutions
 In-service teacher training (partially mandatory)
 Pre-Service teacher trainin
 Research and development work by members of teacher training
institutions
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Diagnosis tool for Informal Monitoring
of Competencies
Teacher administered on-line system
Grade 3 and 7, one year before Assessment
All Areas of standards
Only teachers themselves have access to results
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Assessment of Educational Standards
Legal assessment cycle: three (school) years
In each cycle, the whole target population has to be
assessed in all prescribed competency subdomains
 Grade 4: German, Mathematics
 Grade 8: German, Mathematics, English
Students, teachers, schools and regional education
authorities must get feedback about the results (to be used
for quality improvement)
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Domain-orientated Test Design
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Pilot testing: about 10% of population, up to 9000
Students
Main test: Census up to 83000 students
 Speaking domain only with sample for system monitoring
10% of census with external administration and items for
next census to ensure linking over time
Example: Test session Mathematics Grade 4
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
This picture
shows about
1/10 of
the test
materials
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Item Development
Items are developed to measure sub-competencies or
domains, eg. Content = numbers and units and process =
demonstrating
Dimension „complexity“ is not used as such, since it can‘t
be measured empirically
We have 16 nodes, upon which an item should fall. We
strive for unidimensional items, i.e. items that can be
classified as belonging to one specific node only
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Item Development
Items are developed to measure sub-competencies or
domains, eg. Content = numbers and units and process =
demonstrating
Dimension „complexity“ is not used as such, since it can‘t
be measured empirically
We have 16 nodes, upon which an item should fall. We
strive for unidimensional items, i.e. items that can be
classified as belonging to one specific node only
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Item Development
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Item Development
Test blueprint specified the items to be developed:
 Mathematical node
 Item format: Multiple Choice, Constructed response, …
 Estimated difficulty of item
Further requirements:
 adequat for age group,
 no biased towards gender, race, religion, also taking into account
rural/urban differences.
 does not require other knowledge than maths contents.
 Items should be uni-dimensional
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Validation
In order to secure validity of the test items, they were …
 written by school and university teachers
 reviewed by teachers (peer review and outsider review)
 trialled on a sample of grade 8 students – each item was answered
by at least 200 students
 item properties were reviewed by a team of psychometricians and
school teachers
 items with less than desirable or surprising properties were retired
and not used in the main test
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Test Design
Multiple matrix sampling
Two-step approach for the test design
 Allocation of „empty“ blocks to forms that satify several global
prerequisites for the design:
• positional and contextual balance, item security (S90 vs. S10),
longitudinal link (BL and BIST-UE 2015)
 Allocation of items to blocks that satisfy several local prerequisites:
• Balanced difficulty, format, representative for the competence
construct, dependency (enemy items), test motivation
• Test information, especially along cutoff-scores
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Scaling
Using Rasch model
Allows to express difficulties of items and students ability
on one common scale
Allows to match test cycles over time
Allows for complex test design
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Standard Setting
Standard setting goal: meaningfully described levels
criterion-referenced
Phases:
Determination of number of levels
Labelling of levels (PLLs)
Description of levels (PLDs)
Establishing the cut scores between levels
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
4 levels in Austria in most domains
0: standards not reached
1: partially reached standards
2: reached standards
3: exceeded standards
Easy to understand for „laymen“
Proficiency Level Descriptions M8
3 – standards exceeded
Students possess fundamental knowledge and skills
in all parts of the mathematics curriculum and
advanced knowledge structures, which exceed the
requirements of level 2, specifically more pronounced
abilities of abstraction and higher proficiency in
combining parts of knowledge, methods or rules.
They are able to apply these independently in novel
situations in a flexible way.
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
2 – standards reached
Students posess fundamental knowledge and skills in
all parts of the mathematics curriculum and can use
these in a flexible way. They are able to find and
apply problem solving strategies, to describe and
1 – standards partially reached
reason an approach. They are able to handle verbal,
Students possess fundamental knowledge and skills
graphical and formal representations of mathematical
in all parts of the mathematics curriculum and can
facts in a flexible way and can apply these
master reproductive tasks and carry out routine
appropriately. Theay are able to extract relevant
procedures.
information from a differently represented facts (e.g.
texts, data material, graphics) and can interpret them
in the respective context. They are able to relate their
mathematical knowledge and can check, evaluate
and/or reason mathematical statements.
Standard Setting
„ A standard setting method is a mechanism for helping
panelists translate their intentions to the score scale.“
(Reckase, 2006)
Goal: Match predetermined descriptors of competency
levels to an empiric scale defined by test items
Determine cut scores between levels of competence
Bifie invited a group of stakeholders to set the cut scores:
teachers (lower and upper secondary), teachers of didactics of
math; representatives of parents associations; representatives of
the MoE; representatives of chamber of commerce; bifie staff
Method used: Item descriptor matching method
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
(Modified)
Item descriptor matching method
Item-descriptor matches
Difficult Item
Level 3
Level 3
Level 2
Level 1
Level 2
Level 1
Difficulty-ordered
items
Easy Item
Ordered Item
Booklet
Cut
Score
1
Cut
Score
2
Freunberger & Yanagida, in press.
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Standard Setting
Standard setting is more about finding a societal
consensus than finding a constant
The goal should be clear before test development etc
starts
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Feedback and Reports
Information about results of Educational Standards
SCHOOL LEVEL
School report (for school
4600 schools
about 3200 primary and
1400 secondary I
management) per school,
online
School and instruction development
Feedback to teachers
10.000 classes
about 6000 primary and
4000 secondary I
er teacher, per class/group;
nline
Feedback to students
per participant, online
170.000 students
about 85.000 in grade 4 and 8 each
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Feedback and Reports
Information about results of Educational Standards
SYSTEM LEVEL
National report (Minister for Education)
aggregated
MoE
System development
Province reports (pres & vice-) for each procince
and grade
Supervisory reports (federal superv.
authorities for PS & LS/AS) aggregated on
fed. state and supervised-school type
School development
Supervisory reports (district superv.
authorities for PS and LS) per district
and grade
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
9 provinces
9 provinces
Primary/4
Lower sec/8, academic sec/8
99 school districts
15 Cities – 84 districts
School report: overview of content
Information about school results:
compared to AUT
compared to the defined goals (in the regulation)
compared to other schools with similar basic
conditions („fair comparison“)
Areas of conflict:
 Educational Standards apply to all students – and
therefore to all schools
 Conditions of schools vary widely
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
FEEDBACK
The school takes centre stage
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Feedback – example (school report)
„Competency levels“
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Feedback – example (teacher
feedback)
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Feedback – example (school report)
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Support system for schools
School
feedback
IKM (informal
self
evaluation)
Teacher
feedback
A SCHOOL
Quality
developmentprocess
supervision
School
authorities
Feedback
Moderation
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Outlook
Bringing IKM (grade(s) before BIST) and BIST assessments closer
together. Using items in both assessments could lead towards a
better understanding of learning
Backwash from doing the assessments towards research in the areas
of item development, standard setting, feedback … through
evaluation of results
Outcomes of the assessments should not only allow us to give
feedback to schools, teachers and students but also to improve the
competence models we use for the assessment
Prepare for computer based testing
Open databases for external researchers and improve system level
monitoring
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg
Thank you
for your attention!
MichaeI Bruneforth
m.bruneforth@bifie.at
Bundesinstitut BIFIE
Salzburg | Zentrum für Bildungsmonitoring &
Bildungsstandards
www.bifie.at
The Implementation of National Educational Standards
in Austria, 18/05/2013 © BIFIE Salzburg