Testing Writing
Prepared by :
Lecturer : Hanan Sanad
The best way to test people’s writing is to
get them to write.
How?
1. We have to set writing tasks which are
representative of the population of writing
tasks we expect students to be able to perform.
2. Tasks should represent valid samples of writing
that represent students’ ability.
3. Samples of writing should be scored validly and
reliably.
1. How to choose
representative tasks?
*
*
This refers to the content of the test. Elements specified are:
1.
Operations: the tasks that candidates have to be able to perform such as;
describing, explaining , comparing, and contrasting, … They are based on
the stated objectives of the course
2.
Types of a text: this may include letters, forms , academic essays, fax,
message, report, set of instructions,…
3.
Addressees: this refers to kinds of people the candidate is expected to be
able to write to ( native speakers or nonnative speakers )
4.
Topics: topics may be selected according to the suitability of the academic
level and type of test. Topics must be neutral: l not specialized.
5.
Dialect and style : the dialect and accent that test takers are meant to
understand or those in which they are expected to write . ( Standard
American or British English Dialect )
6.
Length of text: the length of the pieces to be written.
*
•
The ideal test would be one which requires candidates to
perform all the relevant writing tasks . The total score of
the test would be the best estimate of ss’ ability.
•
The more tasks that we set ,the more representative of
candidates’ ability and the more valid will be the totality
of samples of the candidate’s ability.
2 How to elicit a valid
sample of writing ability:
1.
.
2.
3.
•
This requirement is closely related to the need
to include representative sample of the content
specified . Students’ performance are not
perfectly consistent , therefore, we have to
offer as many fresh starts as possible .
•
By doing this, we achieve greater reliability and
greater validity.
*
For the sake of validity, we shouldn’t set
tasks which measure abilities such as
creativity or intelligence or wide
knowledge. Just writing should be tested.
An effective way to test writing only is to
give ss pictures to describe or compare
and contrast or by giving them a series of
pictures to form a story…
•
Candidates should know just what’s required
of them and they should not be allowed to go
far astray (missing).
•
A useful device is to provide information in the
form of notes or pictures
•
Care has to be taken when notes are given not
to provide ss with too much information of
what they need. Full sentences are avoided.
3. How to Ensure valid
and reliable scoring
*
*
:
* 1. set tasks which can be reliably scored
A number of suggestions made to obtain representative
performance will facilitate reliable scoring.
2. Set as many tasks as possible
The more score for each candidate the more reliable should be the
total score.
3. Restrict candidates:
The greater the restrictions imposed on candidates the more
directly comparable will be the performances of different
candidates.
4. Give no choice of tasks:
Making candidates perform all tasks makes comparison easier.
5. Ensure long enough samples:
Samples of writing have to be long enough for judgment to be
made reliably. Or example, in order to receive reliable
information about organizational ability , the writing should be
long enough in for organization to reveal itself
6. Create appropriate scales for scoring:
There are two basic approaches to scoring:
Holistic
Analytic
Holistic scoring (rubric):
• It’s a rating scale often referred to as "impressionistic" scoring
• It Involves the assignment of a single score to a piece of writing on the basis
of an overall impression of it.
• Individual features of a text, such as grammar, spelling, and organization,
should not be considered as separate entities.
• It Has the advantage of being very rapid
• Since, in holistic scoring, the entire written text is evaluated as a whole, it
is important to establish the specific criteria upon which the evaluation is to
be based prior to undertaking the evaluation. This does not mean
establishing a catalogue of precise individual errors that might appear, but
rather deciding what impact the errors that are present have on the overall
tone, structure, and comprehensibility of the writing sample
Analytic Scoring (rubric):
•A rating scale that requires a separate score for each of a number of
aspects of a task, such as grammatical accuracy, vocabulary, idiomatic
expression, organization, relevance, coherence.
•Scorers are compelled to consider aspects of performance which they
might otherwise ignore.
•The very fact that the scorer has to give a number of scores will tend to
make the scoring more reliable.
•In some schemes, each of the components is given equal weight. In other
schemes, the relative importance of the different aspects, as perceived
by the tester (with or without statistical support), is reflected in
weightings attached to the various components.
•Advantages:
1. it disposes of the problem of uneven development of sub-skills.
2. Scorers are compelled to consider aspects of performance which they
might otherwise ignore .
3. the scorer has to give a number of scores will tend to make the scoring
more reliable
•Disadvantages:
• It takes more time than holistic scoring.
• Concentration on the different aspects
may divert attention from the overall
effect of the piece of writing. Inasmuch
as the whole is often greater than the
sum of its parts, a composite score may
be very reliable but not valid (Hughes
1989: 93-94)
What are the steps of constructing a rating scale?
7. Calibrate the scale to be used:
• Collecting samples of performance . A team of experts
looks at these samples and assigns each of them to a
point on the relevant scale . The assigned samples
provide reference points for all future uses of the
scale as well as being necessary training material.
8. Select and train scorers:
• Not everyone is equally good at rating writing . So all
stuff who will be involved in the test process should
be trained. This is specially important where scoring is
more subjective like the scoring of writing tests.
9. Follow acceptable scoring procedures:
Identifying benchmark scripts
Presenting copies of these to the scorers for initial scoring
Scoring begins
Each task is scored independently by two or three scorers
A senior member of the team collates scores and identify
discrepancies in scores
*