Quality Management
in FL Education
Large Scale Assessment, Procedural
Controls and Self Evaluation
A critical analysis, with some references to
the post-PISA and post-DESI German scene
Konrad Schroeder, University of Augsburg
Main Points to be addressed
• The Traditional System and the Impact of PISA and the CEF
• Standards in Large Scale Language Assessment
• New Standards in Classroom Assessment: Procedural Controls
• The Role of Self-Assessment: Portfolio-Work etc.
• Establishing a „New Evaluative Culture“ for Our Schools: Bridging
Gaps and Healing Traumas
• A Changing Role for National Testing Institutions
• Some Golden Rules for Teachers
Vilnius Convention May 2011
2
Traditional Modes of Assessment in the FL
Classroom
• Modes of assessment based on school traditions rather
than didactic insight
• No proper criteria for the construction of tasks. No
construction standards
• Subjective tasks, often reproducing the structure of
exercises in the course books. Very little task orientation.
Very little communicative reliability
• Pseudo norm-orientation (→normal probability curve)
instead of transparent criteria
• Traditional modes of marking and a negative approach
based on the counting of errors
Vilnius Convention May 2011
3
The Advent of a New Era in FLT (1)
PISA and DESI
•
PISA was targeted at reading comprehension as a basic tool for cultural
participation and insight: it was mother-tongue based, but had a decidedly
communicative orientation, and its results had a definite bearing on FLT
performance.
Finland won, but Germany as another wealthy EU country with a wellestablished educational system, failed: a fact that generated the famous „PISASchock“ and a call for a complete change in the system: from “inputorientation“ to “output-orientation“, and consequently for a “new culture of
evaluation.”
•
•
Internationally speaking, PISA produced two positive results:
▪
Firstly: The call for National Educational Standards, with teaching
syllabi and student books based on them.
▪
Secondly: A move towards a monitoring of educational systems based
on truly empirical external evaluation (comparative proficiency tests,
centralised exams), also with the idea of making it easier to compare
national profiles.
DESI later confirmed the PISA results for Germany with regard to English as
a FL: Students had substantial deficits in listening and reading
comprehension.
Vilnius Convention May 2011
4
The Advent of a New Era in FLT (2)
The Common European Framework (CEF)
The CEF: A tool in many ways revolutionary. Its assets:
▪
It offers a positive approach to language learning by using
can do-statements. No counting of errors.
▪
It honours the lower levels of proficiency, giving each of them
their own gestalt and thus: right to exist, value.
▪
It emphasises the various facets of oral proficiency, and
describes them at the six levels.
▪
It includes scales for a fair number of sub-skills.
▪
The underlying competence model is consensus-based and
empirically scaled.
▪
The criteria for the descriptors are taken from the outside
world. They have their equivalents in real life.
Vilnius Convention May 2011
5
Some Flaws of the Common European Framework
• The CEF is unable to scale intercultural competence,
which nowadays is a key issue in and an integral part of
FLT.
• The CEF does not scale language awareness, the
development of which nowadays is, again, a key issue in
FLT.
• The CEF does not scale language learning awareness,
the development of which is absolutely necessary in
view of European demands and the results of
globalisation.
• The CEF does not deal with literary understanding (text
and media competence).
Vilnius Convention May 2011
6
Large Scale Language Assessment as a Cure (1)
The Aims
Two related objectives:
• Comparing the levels of proficiency of groups of learners
nationally or internationally, in the context of quality
development
• Evaluating the level of proficiency of individual learners
by way of centralised tests, in order to produce
comparable individual results (e.g. meeting an
educational or a career requirement, → certificate)
Vilnius Convention May 2011
7
Large Scale Language Assessment as a Cure (2)
Standards
• In the context of quality development: a probabilistic approach. The
individual testee is of no interest.
• In the context of centralised tests: focus on individual proficiency
• Standardised, properly validated tasks (a clear view of the test
construct, a clear-cut competence-model, a transparent criterion
orientation, an empirical operationalisation through phases of prepiloting, piloting)
• A standardised test administration (sampling of items and students)
• An objective scoring and grading (rating scales, benchmarks, rater
training)
• Empirical development of proficiency scales (Rasch-modeling,
standard setting – a process both empirical and based on expertise)
• Transparent reporting (generalised results, e.g. in terms of
proficiency levels: norm orientation, → normal probability curve)
Vilnius Convention May 2011
8
How to establish a „New Culture of Evaluation“ in the FL
Classroom (1)
Traditional
evaluation
Needed in the
Classroom:
External
evaluation:
Subjective
tasks
Competence
models
Pseudo
normorientation
Explicit criterionorientation
Error
counting
?
Objective rating
Standardisation
Levels of
proficiency
Grades
Vilnius Convention May 2011
9
Aims and Standards of Assessment
in the Innovative FL Classroom:
Aims:
• Measuring the achievement of individual learners and, implicitely, the quality
of their learning processes over a defined period
• Finding out degrees of achievement, and spotting deficits
Standards:
• An individual approach: the individual testee and their learning processes are
in the centre of interest.
• Tasks based on the didactic expertise of the teacher (as an expert in the fields
of tuition and [process-oriented] testing): criterion orientation
• A test administration standardised within the quality level which the individual
school can offer
• Scoring and grading done within the quality level which the individual school
can offer: ad hoc rating scales, ad hoc benchmarks, ad hoc rater training.
• Using the CEF as benchmark (as far as language skills are concerned)
• Transfering degrees of achievement into marks, following a positive, processoriented approach
Vilnius Convention May 2011
10
How to establish a „New Culture of Evaluation“ in the FL
Classroom (2)
Traditional
evaluation
Subjective
tasks
Pseudo
normorientation
Error
counting
Grades
Needed in the
Classroom:
Innovative
tools
to diagnose
and
to measure
individual
achievement
and progress
Grades
Vilnius Convention May 2011
External
evaluation:
Competence
models
Explicit criterionorientation
Objective rating
Standardisation
Levels of
Proficiency
11
The Role of Self-Assessment
• Self-assessment in general: a key competence. In the EU the selfassessment of FL competences is an important asset.
• The Portfolio format of the Council of Europe: a training-tool for the
development of language- and culture-oriented self assessmentskills
• Self-assessment in FL course books: Test-Yourself-pages etc.
• The psychological truth: If properly trained, learners become well
aware of their communicative assets and deficits. Further empirical
research is needed in this field.
• Schools experimenting in this field (e.g. by including selfassessment into the achievement statements of the school report)
report on a dramatic rise in student motivation.
Vilnius Convention May 2011
12
How to establish a „New Culture of Evaluation“ in the FL
Classroom (3)
Traditional
evaluation
Subjective
tasks
Pseudo
normorientation
Error
counting
Grades
Needed in the
Classroom:
Innovative
tools
to diagnose
and
to measure
individual
achievement
and progress
Grades
External
evaluation:
Competence
models
Explicit criterionorientation
Objective rating
Standardisation
Levels of
Proficiency
Self-assessment
Vilnius Convention May 2011
13
How to establish a „New Culture of Evaluation“ in the FL
Classroom (4)
Traditional
evaluation
Subjective
tasks
Pseudo
normorientation
Error
counting
Needed in the
Classroom:
Innovative
tools
to diagnose
and
to measure
individual
achievement
and progress
Grades
Preparation
for, and
“making use“
of central
exams and
different
kinds of
external
evaluation
Levels of
Proficiency
Grades
Self-assessment
Vilnius Convention May 2011
External
evaluation:
Competence
models
Explicit
criterionorientation
Objective rating
Standardisation
Levels of
Proficiency
14
A Modified Role for National Testing Institutions (1)
• The traditional role of national testing institutions has been to
develop and administer instruments of external evaluation, basically
in the context of quality management and educational
standardisation, but also as (semi-) standardised exams for schools
(cf. e.g. the British test boards).
• These duties will not change, but new responsibilities have recently
been added (cf. e.g. the IQB in Berlin):
▪
Writing educational standards (or bringing them up to date)
▪
Developing “learning tasks“ („Lernaufgaben“) as a preparation
for central exams and different kinds of external evaluation,
thus helping to avoid a narrow-minded “teaching to the test“approach.
Vilnius Convention May 2011
15
A Modified Role for National Testing Institutions (2)
▪
▪
Developing achievement tests based on these tasks, as
examples of „good practice“
Preparing material in the context of advising teachers how to
best make use of centralised exams and different kinds of
external evaluation.
Vilnius Convention May 2011
16
A „New Evaluative Culture“ on the Move
Areas of Action
Initiatives at level of
individual school:
“Assessment and
Evaluation”
Teacher training
at university and
in-service
…
…
Teaching materials
reflecting assessment
needs
Proper educational policies
Implementing evaluation
systems. Supplying training and
materials.
Vilnius Convention May 2011
17
And finally: Some Golden Rules (1)
As an FL teacher
• Be not dismayed: In no other domain of FL teaching has there been such a
dramatic change as in student evaluation. Things are changing for the
better. Witchcraft is not involved. You can understand the new approach and
follow it.
• External testing, using validated and standardised tasks and methods, is not
negative or educationally dangerous. It is imperative if we take quality
management (at group and/or individual level) seriously.
• Testing too often (“testeritis“) is demotivating and therefore counterproductive. Traditional testing with its lack of transparence, validity, and
reliability is extremely counterproductive.
• Do not confound large scale assessment (proficiency testing) with the
homemade, process-oriented achievement tests of the FL classroom. The
standards of large scale assessment cannot be reproduced on a smaller
scale. There is no „large scale assessment light“.
Vilnius Convention May 2011
18
And finally: Some Golden Rules (2)
• When you write a process-oriented achievement test measuring
learning success
▪
have recourse to your didactic and testing expertise. Do not
merely follow traditions.
▪
make your criteria transparent.
▪
do not fall victim to a norm-orientation which does not exist:
There is no „normal“ distribution of achievement in an FL
classroom.
▪
In marking the documents try to follow a positive approach (no
error-counting) based on your own, task-specific rating scales.
Use the CEF as a reference work only, not as a set of preestablished descriptions.
▪
Develop a clear view of the many domains of FLT that cannot
be objectively tested.
Vilnius Convention May 2011
19
THANK YOU FOR YOUR
ATTENTION
My thanks go to my former student Dr. Claudia Harsch,
Assistant Professor for Language Assessment at the University of Warwick,
with whom I collaborated in writing this presentation.
Konrad.schroeder@phil.uni-augsburg.de
Prof. em. Dr. Konrad Schröder
Department of Didactics of English
University of Augsburg
Germany