Writing tests at Universities
ESF Testing project, Vilnius, January 29-30, 2007
Vita Kalnberzina, Dr.Phil., Assist.Prof., Latvia University
Contents
Writing tests at Universities ......................................................................................1
Contents ......................................................................................................................................................... 1
List of Tables .................................................................................................................................................. 1
Course Programme ........................................................................................................................................ 2
'Writing' as a concept ...................................................................................................................................... 3
What do you understand by the term 'writing'? ........................................................................................... 3
What does the term 'writing' mean to the student/test-taker? .................................................................... 6
How is written language different from spoken? ......................................................................................... 6
Is writing informational, interactional or both? ............................................................................................ 7
Teaching Writing at University Level............................................................................................................... 8
Product approach ....................................................................................................................................... 8
Conventions For Term Paper Writing ..................................................................................................... 8
Process approach ...................................................................................................................................... 9
Help for students at Dartmouth University ............................................................................................. 9
Instructions to Writing assistants:......................................................................................................... 10
Moving from Writer-Based to Reader-Based Prose ............................................................................. 10
Writing test development .............................................................................................................................. 10
Indirect approach...................................................................................................................................... 11
Direct approach ........................................................................................................................................ 11
What do you need to know to develop a task? ......................................................................................... 12
Writing Tests at University ........................................................................................................................ 14
Qualspell .............................................................................................................................................. 14
What task types can be used to test writing proficiency at the University? .......................................... 14
Which text/task-types are appropriate for your students? .................................................................... 14
How do you know that you have produced a good task? ..................................................................... 15
What does a test-taker need to know to do a task: develop the test specification ............................... 16
Marking writing tests ..................................................................................................................................... 16
Formative Assessment ............................................................................................................................. 16
Summative Assessment ........................................................................................................................... 17
Holistic marking scales ........................................................................................................................ 17
Analytic marking scales ....................................................................................................................... 19
Assurance of test quality............................................................................................................................... 21
Validity...................................................................................................................................................... 21
Reliability of the examination .................................................................................................................... 23
Inter and Intra-marker reliability ........................................................................................................... 23
Bibliography .................................................................................................................................................. 26
Online sources .............................................................................................................................................. 26
List of Tables
Table 1 You as a writer ..................................................................................................... 3
Table 2 What is writing? .................................................................................................. 4
Table 3 Writing from the point of view of the test-taker ................................................. 6
Table 4 Comparison between written and spoken language ........................................... 6
Table 5 Interactional versus informational writing ........................................................ 7
Table 6 Comparison between direct and indirect writing tests ..................................... 12
Table 7 Topics for writing tasks, Compare with Language Domains in CEF ............... 12
Table 8 Writing as a process (Callaghan, Knapp, and Noble 1993, p.193) ................... 15
Table 9 Impressionistic marking scale ........................................................................... 17
Table 10 CEF levels of writing production ..................................................................... 17
1
Table 11 Holistic scale (British Council IELTS written production evaluation) .......... 18
Table 12 BC analytical writing evaluation scale ........................................................... 19
Table 13 Validity of the writing tasks ............................................................................ 23
Table 14 Reliability of the Year 12 examination in 1999 .............................................. 24
Table 15 Intercorrelations between the different tests in 2001 .................................... 24
Course Programme
January 29
9:00 – 10:30
session
Introduction, Writing as a concept
10:30
coffee break
11:00 – 12:30
session
13:30
Lunch
15:00 – 16:30
session
16:30
tea break
17:00 – 18:30
session
Marking scale types and marking
reliability
9:00 – 10:30
session
Development of new writing tasks
10:30
coffee break
11:00 – 12:30
session
13:30
Lunch
15:00 – 16:30
session
16:30
tea break
17:00 – 18:30
session
Approaches to testing writing
Writing tests and tasks
30 January
Development of marking scales
Pretesting of the new writing tasks
and marking scales
Feedback on the new writing tasks
and marking scales and evaluation
of the course
2
'Writing' as a concept
Objective: Discuss the concept of writing and hence validity of writing tests
What do you understand by the term 'writing'?
Take a couple of minutes to answer the following questions:
Table 1 You as a writer
Question
What do you write?
Answer
How do you write?
What are three main things you want
your students to learn about writing?
How would you define 'writing' as a skill?
Read the 5 texts and fill in the table 3 below:
1. Writing is both tool and art, but it is not a body of knowledge. Those who have
taught writing and observed writers at work are convinced that writing is a
complex, multifaceted, dynamic skill, affected by cognitive demands, affect
(whether the writer feels good on that day, whether he or she likes the topic, or
the reader of the writing, or the reasons of writing) and variations in test
conditions. Above all writing is a cognitive activity that calls on many
components of personal experience, attitude, knowledge of ideas, issues and
factual information, but that requires a person to take and shape some parts of
this complex web of material into a rhetorical, syntactic and mechanical whole
(Hamp-Lyons 1993, p.11).
2. The writing test tests the test-takers’ ability to write correct and appropriate
sentences, use conventions peculiar to written language, think creatively
and develop thought excluding irrelevant information, manipulate
sentences and paragraphs to use language effectively, write in an
3
appropriate manner with a particular audience in mind, organise and
order the written material (Year 12 test specification).
3. Textual competence includes the knowledge of the conventions for
joining utterances together to form a text consisting of two or more
sentences which are structured according to the rules of cohesion
(marked relationships among sentences) and rhetorical organisation
(development of the text) (Bachman 1990)
4. A successful writer needs to have content knowledge (knowledge of the
concepts involved in the subject area), context knowledge (social context
in which the text will be read, the reader's expectations, knowledge of
the cotexts alongside which this text will be read), language system
knowledge (lexis, syntax) and writing process knowledge (the most
appropriate way of preparing for a writing task) (Tribble 1995).
Table 2 What is writing?
Year 12 test
specification
(1999)
Hamp-Lyons (1993)
Tribble/Bachman
(1990)
What is
writing?
What subskills
does writing
consist of?
What is it
affected by?
4
5
What does the term 'writing' mean to the student/test-taker?
Table 3 Writing from the point of view of the test-taker
Question
Answer
What do your students write?
In what settings do they like/do not like to
write?
What factors inhibit/encourage them to
write?
What contexts stimulate them to write
well?
How is written language different from spoken?
Read the extract below and say what language aspects are lost when we
transform spoken language to written language? How can we compensate for
the loss? Is this enough?
Contemporary views of the differences between written and spoken language
support the idea that
1. written and spoken language do possess distinctive features and
2. that texts can be distributed along the continuum from the most
typically spoken to the most typically written.
Apart from the loss of the contextual information that is available to the people
who participate in a conversation (for example, knowledge of the background
and the histories of the individuals concerned) two of the most important
aspects of speaking that are normally lost in any transcription are prosodic
(rhythm, phrasing, pauses) and paralinguistic features (the way someone is
speaking, facial expressions and gestures). To compensate for this loss we use
punctuation and other features (Tribble 1995, p.17, 18).
Fill in the table choosing from the box below:
Table 4 Comparison between written and spoken language
spoken
written
vocabulary
6
Grammar
Text
Incomplete utterances, full word forms, rare words, 'incorrect' grammar, vocabulary of
Latin origin, complete sentences, hesitations, reduced word forms, informal
vocabulary, many phrasal verbs,
common words, conventional abbreviations, polysyllabic words, high density of content
words, sophisticated and intricate grammatical resources, heavily nominalized style,
clauses instead of sentences as a minimum unit, carefully planned paragraphs
Is writing informational, interactional or both?
Weir (1995) considers that there are two types of writing operations:
interactional and informational.
Fill in the table choosing the operations from the box below
Table 5 Interactional versus informational writing
Type of
Operations/action
operation
Interaction
al in social
and service
texts:
Product
Informatio
nal in
academic
contexts
Summary checklist of writing operations (Weir 1995)

Describing process which might involve: purpose, describing means,
results, process, change of state, sequential description, instructions,
summary
7

Describing
phenomena
and
ideas:
definition,
classification,
identification, comparison and contrast, exemplification,

Arguing:
stating
deduction,
a
proposition,
substantiation,
stating
concession,
assumptions,
summary,
induction,
generalisation,
speculation, comment, evaluation

Expressing thanks, requirements, opinions, attitude, confirmation,
apology, wants, needs, lacks, ideas, information, complaints, reasons,
justifications

Eliciting information, directions, service, clarification, help, permission

Directing: ordering, instructing, persuading, advising, warning
Teaching Writing at University Level
Product approach
Conventions For Term Paper Writing
http://www.lu.lv/mvf/Studij_bak.htm
Students having registered for the academic BA programme (English Philology) will be
eligible to write 3 term papers before writing their Bachelor Paper (BA paper).
Students can choose themes for term papers in the following areas:
1. Aspects of the English Language (Phonetics, Grammar, Lexis);
2. British (American, Canadian) studies;
3. English Language Teaching (ELT) Methodologies;
4. Linguistics;
5. Translation and Interpreting;
6. Literature.
Length Of The Term Paper
The term paper will be between 5000 words and 7500 words.
You are not required to produce a term paper embodying totally original work and ideas
that will be a contribution to knowledge, although if you do, you will receive appropriate
credit for it.
Organisation And Content Of The Term Paper
Conventionally, there will be:
Title page
Declaration of academic integrity
Abstract in English and Abstract in Latvian
Table of contents
8
Introduction;
Chapters;
Conclusions;
Theses;
Bibliography;
Appendix (or appendices).
Plagiarism
You plagiarize if you take someone else’s work, ideas, words and use them as if they were
your own. You can avoid plagiarism by using correct methods for quoting, paraphrasing
and referencing. Quoting means using the exact words of the writer/speaker, whereas
paraphrasing means restating the words and ideas from a book, an article, or a lecture in
your own words.
Failure to produce references amounts to plagiarism, and your work will be deemed invalid.
Layout And Presentation
Use A4 size white paper on one side.
Word process the text using Times New Roman.
Use twelve point for the main text; fourteen point for headings.
Use margins: top/bottom/right -2.5 cm; left-3.5 cm (for binding);
Use 1.5 line spacing;
Indent quotes longer than three lines by five characters; use single space between the lines;
Process approach
http://www.dartmouth.edu/~writing/services/support_student.shtml
Principles of process pedagogy at Dartmouth University:
1. writing can be understood as the culmination of several steps in a complicated
process, which include prewriting, writing, rewriting, and all their attendant
strategies
2. use of dialogue as a teaching technique.
3. challenge the traditional, authoritative models of teaching, in which professors
(who know everything) talk "at" students (who know very little).
4. empower students by getting them to talk about their writing at every step of
the writing process.
Help for students at Dartmouth University

If you are a first-year student, we will help you understand the expectations of
college writing.

We will help you generate, research, and organize ideas.

We will talk to you about your structure, grammar, and style.

If you are a student for whom English is a second language, we will help you
work on English grammar and style. We will also help you understand the
conventions for writing an essay in English.
9

If you are a senior, we will provide strategies for writing a thesis or
culminating experience paper.

We will even help you with multi-media compositions, such as Web sites,
videos, or PowerPoint presentations.
Instructions to Writing assistants:

Re-read the assignment, if the professor has written one. Make careful
notes regarding what, precisely, the professor is asking the students to do.

Read through all (or several) of the papers once, without marking. This
strategy helps you to get a sense of the range of essays you'll be reading.

Read through each paper twice: once to get the gist, the second time to
respond. This strategy helps you to see where the writer is going before you
jump in with your advice.

Respond the first time through, but in pencil. That way, you can erase
your remarks if something the writer says on page four changes the way you
feel about page two.

Take notes, on the side. Sometimes writing assistants want to keep a
running commentary of their reading responses, but don't wish to share these
responses with the writer. Keep a pad and a pen nearby.
Moving from Writer-Based to Reader-Based Prose
http://www.dartmouth.edu/~writing/materials/tutor/pedagogy/process.shtml
At some point in the writing process, writers must turn their attention from the
writing process to the written product. They must transform writer-based prose into
reader-based prose. To do this, they must be ready to revise, paying close attention to
their prose and to the effect it may be having on their audience.
Often it's precisely at this point in the process that a writer will come to you for help.
He has a draft completed, his argument seems logical and even persuasive, but he
wants a second opinion. You read the essay and have some trouble following his line of
reasoning. You point to a paragraph that is particularly confusing, and you ask what
the writer is trying to say. He responds, "But it's all right there!" and goes on to
summarize a point that he clearly hasn't made. What's happened? The writer's point is
so firmly entrenched in his mind that he really believes that it's "in" the essay. You
show him that it's not. You've just given him a lesson in the difference between writerbased and reader-based prose.
Writing test development
How do you develop writing tests? How do you test writing?
How often do you test? How do you mark writing?
What problems do you have?
10
Indirect approach
Hamp-Lyons (1993, p.5):
Indirect tests (recognize correct word order or structure, choose the correct phrase for
the gap, underline the phrase that contains error etc.) do not require the test-taker to
write continuous prose, there is little room for personal interpretation, a key is
provided for the marker, the role of human judgement is less obvious, as scores are not
tied to essay samples or rating scales, but to the judgement of the test developers who
have developed questions and decided on the correct answers. In this case training of
markers is not necessary and scoring can be done clerically.
In indirect discrete element writing tests (TOEFF, MELAB) knowledge of grammar
rules and vocabulary items tend to dominate. While these skills may be related to
proficient writing, most of us do not accept that they can represent what proficient
writers do. Therefore indirect writing assessment is invalid. It has extremely negative
educational consequences on the learning/teaching process as it creates the impression
that writing is the same as recognizing errors in sentences, knowing grammar rules,
and of not writing about ideas and emotions.
Direct approach
A direct test of writing has the following characteristics:
1. each individual taking the assessment must actually physically write at least
one piece of continuous text of 100 words and may write several considerably
longer pieces
2. the writer is provided with a set of instructions and a text, picture or other
prompt material, he or she is given considerable room within which to create a
response to the prompt
3. every text written by a candidate is read by at least one , usually two or more,
human reader-judges who have been through some form of preparation process
4. the judgements of the readers are tied in some way to some common yardstick,
such as a set of sample essays, description of the expected performance or one
or several marking scales
5. scores on the text are recorded and can be retrieved for review
The ability of a writing test to give reliable results is affected by the amount of
variation in things like:
1. topic he or she is asked to write about
2. the type of written task (narrative, argumentative, etc.)
11
3. the amount of time available
4. whether it is done on word processors
5. how many samples we get from each writer
Fill in the table:
Table 6 Comparison between direct and indirect writing tests
Sample task
Positive
Negative
Direct approach to
testing writing
Indirect approach to
testing writing
What do you need to know to develop a task?
Questions:
1. What is a test task? How is it different from a learning task?
2. What does a test task consist of?
3. What are the criteria in a test task evaluation?
Hughes (1989) offers a following writing task framework:
1.
2.
3.
4.
Operations (describe, explain, compare, contrast, argue)
Text type: letter, form, essay, postcard
Addressee: native speaker, non-native speaker or as specified under topics
Topic (see Table 7)
Table 7 Topics for writing tasks, Compare with Language Domains in CEF
Topic:
Rank order of
appropriacy for
your student
social interaction with native and non-native speakers of
English
shopping and using services
travelling and arranging for travel
dealing with official and semi-official bodies
visiting places of interest and entertainment
using media for information and entertainment
12
studying for academic, occupational and social purposes.
medical attention and health
Use Hugh's (1989) framework to analyse Year 12 examination tasks in 2001
What operations, addressee, text type and what topic?
Task 1
Task 2
Task 3
Operations
Text type:
Addressee:
Topic
Year 12 Writing test
Task 1
Your friend wants to rent his seaside house to Mr and Mrs Crawford for the summer
holidays. They asked him to describe it in detail, but his English is not very good. Help
him to write the letter (100 words). Here are his notes:
A wooden cottage near the Lielupe
A kitchen and living room downstairs
Two bedrooms and a bathroom upstairs
Garden, a garage
Task 2
Describe the diagram in 100 words and compare the popularity of the different study
programmes (Diena June 2000).
Task 3
The role of sports in the modern world
Justice Earl Warren says that the sports page records people's accomplishments, but
the front page nothing but their failures.
Write an essay on the role of sports in the modern world (200-250 words) and discuss:
What sports means to you and your friends
Whether you follow the results of Latvian athletes in international competitions
Whether it is important for Latvian athletes to win international competitions or not (why)
Whether the government of Latvia should support sports clubs, teams (if yes then how
and why) or spend money for more urgent needs of the Latvian people
13
Writing Tests at University
Qualspell
ESP:C, an English Course for Chemists With the support of
http://moodle.esp-c.org/mod/resource/view.php?id=727
Task 1
Give written instructions to a new laboratory assistant on how to prepare a solution of salt in water.
Say where the necessary chemicals can be found, in which vessel(s) they must put in order to
dissolve the salt, how the solution is treated after use and how it is stored.
(It is important that your instructions are logically structured, easy to understand and very clearly
expressed. Give your instructions in 8 – 10 short sentences.)
Task 2
Genetics is most important in modern science.
Do you agree with this statement or would you rather disagree with it?
Present your arguments in 120 - 150 words. Structure your ideas logically. Do not forget to make
clearly visible paragraphs.
Task 3
Write an e-mail to an important customer in which you inform the company about a serious delay
and in which you apologize for this delay.
The reasons for the delay: You bought a new analytical instrument but you have problems with this
machine as the machine cannot be operated. You hope the maintenance works will not take too
long.
You have had very good relations with this customer for many years. So you hope that despite this
delay you will be able to continue this good the co-operation.
Your reply should be very polite and explain the situation as given.
What task types can be used to test writing proficiency at the University?
Year 12 test specification says: The test-takers demonstrate their ability to do
three of the following: to write a letter, a postcard, a set of instructions, a report
or a guided essay or fill in forms. In all cases test-takers are asked to respond to
leaflets, notices, announcements, personal notes and messages, directions, tables
and graphs.
Which text/task-types are appropriate for your students?
Look through the Table 11and tick the text types that would be appropriate for your
students, choose one type and develop the task as you would present to your students
Include the text type, length, addressee and time limit, criteria, optional.
14
Table 8 Writing as a process (Callaghan, Knapp, and Noble 1993, p.193)
Action
Process
Describe Through the process of
sequencing phenomena in
temporal and causal
relationships
Product
Yes/No
Personal descriptions
Technical descriptions
Information reports
Scientific reports
Definitions
Explain
Through the process of
sequencing phenomena in
temporal or causal
relationships
Explanations of how
Explanations of why
Elaborations
Illustrations
Accounts
Explanation essay
Instruct
Through the process of
Procedures
logically ordering a sequence
Instructions
of actions or behaviours
Manuals
Science experiments
Recipes
Directions
Narrate
Through the process of
sequencing people and
events in time and space
Personal recounts
Stories
Fairytales
Myths and fables
Narratives
Argue
Through the process of
Essays
persuading readers to accept
Expositions
a logical ordering of
propositions
Discussions
Debates
Reviews
Evaluations
How do you know that you have produced a good task?
How difficult the task will be?
15
Personal
Public
Concrete
1
3
Abstract
2
4
Weir 1995 p.22-25: Specific guidelines for moderation of a task: format
1. Is the level of difficulty appropriate?
2. Will the task discriminate?
3. Is it appropriate?
4. Is it clear?
5. Is the timing appropriate?
6. Is the layout appropriate?
7. Is there a bias against some group of students?
What does a test-taker need to know to do a task: develop the test
specification
What are the student's rights? What do they know before they start writing?
Hamp-Lyons (1993, p.61) considers that a student has the right to full knowledge
about the test she or he is taking and should be informed about:
test purpose
format
test-length
number of questions and score weighting if any
kinds of writing to be evaluated
criteria
scoring method
score reliability
qualification of judges
Tick the information in the table above that was available for the year 12
exam test takers in year 2001 (see below)
Marking writing tests
http://www.provost.cmich.edu/assessment/toolkit/formativesummative.htm
Formative Assessment
Formative assessment is often done at the beginning or during a program, thus providing
the opportunity for immediate evidence for student learning in a particular course or at a
16
particular point in a program. Classroom assessment is one of the most common formative
assessment techniques. The purpose of this technique is to improve quality of student
learning and should not be evaluative or involve grading students.
Summative Assessment
Summative assessment is comprehensive in nature, provides accountability and is used to
check the level of learning at the end of the program. For example, if upon completion of a
program students will have the knowledge to pass an accreditation test, taking the test
would be summative in nature since it is based on the cumulative learning experience.
Holistic marking scales
Holistic scoring involves the assignment of a single score to a piece of writing
on the basis of an overall impression of it (hence the name 'impressionistic
scoring'). It is rapid and therefore each paper can be scored several times.
Sample scale used at an English medium university
Table 9 Impressionistic marking scale
NS
NSMA
MAA
D
NA
FBA
Native speaker standard
Close to a native speaker standard
Clearly more that adequate
Possibly more than adequate
Adequate for study at this university
Doubtful
Clearly not adequate
Far below adequacy
Holistic scales are based on overall level descriptions, which can also be expressed in
'can statements'.
CEF describes its levels of overall written production in a following scale
starting with A1 (the lowest level) and finishing with C2 (the top level). Fill in
the names of the levels in Table 13
Table 10 CEF levels of writing production
Description
Level
Can write straightforward connected texts on a range of familiar
subjects within his field of interest by linking a series of shorter
discrete elements into linear sequence
Can write simple isolated phrases and sentences
Can write clear, well structured texts of complex subjects, underlining
the relevant salient issues, expanding and supporting points of view at
some length with subsidiary points, reasons and relevant examples, and
rounding off with an appropriate conclusion
17
Can write clear, detailed texts on a variety of subjects related to his/her
field of interest, synthesising and evaluating information and
arguments from a number of sources
Can write a series of simple phrases and sentences linked with simple
connectors like 'and', 'but', 'because'
Can write clear, smoothly flowing, complex texts in an appropriate and
effective style and a logical structure which helps the reader to find
significant points
What are the criteria used to differentiate between the levels in CEF writing
production level description?
Table 11 Holistic scale (British Council IELTS written production evaluation)
9
The writing displays an ability to communicate in a way which gives the reader full
satisfaction. It displays a complete logical organisational structure which enables the
message to be followed effortlessly. Relevant arguments are presented in an
interesting way, with the main ideas prominently and clearly stated, with completely
effective supporting material, arguments are effectively related to the writer's
experience or views. There are no errors of vocabulary, spelling punctuation or
grammar and the writing shows an ability to manipulate the linguistic systems with
complete appropriacy
8
The writing displays an ability to communicate without causing the reader any
difficulties. It displays a logical organisational structure, which enables the message
to be followed easily. Relevant arguments are presented in an interesting way, with
main ideas highlighted, effective supporting material and they are well related to the
authors own experience or views. There are no significant errors of vocabulary,
spelling, punctuation or grammar and the writing reveals an ability to manipulate
the linguistic systems appropriately.
7
Writing displays an ability to communicate with few difficulties for the reader. It
displays good organisational structure, which enables the message to follow without
much effort. Arguments are well presented with relevant supporting material and an
attempt to relate them to writer's experience or views. The reader is aware of but not
troubled by occasional minor errors of vocabulary, spelling, punctuation and
grammar or some limitations to the writer's ability to manipulate the linguistic
systems appropriately.
6
The writing displays an ability to communicate although there is occasional strain
for the reader. It is organized well enough for the message to be followed throughout.
Arguments are presented but it may be difficult for the reader to distinguish main
ideas from supporting material; main ideas may not be supported; their relevance
may be dubious; arguments may not be related to the writer's experience or views.
The reader is aware of errors of vocabulary, spelling, punctuation or grammar, and/or
limited ability to manipulate the linguistic systems appropriately, but these intrude
only occasionally.
5
The writing displays an ability to communicate although there is often strain for the
reader. It is organized well enough for the message to be followed most of the time.
Arguments are presented but may back relevance, clarity, consistency or support;
they may not be related to the writer's experience or views. The reader is aware of
errors of vocabulary, spelling, punctuation or grammar, which intrude frequently,
18
and of limited ability to manipulate the linguistic systems appropriately.
4
The writing displays a limited ability to communicate, which puts strain on the
reader throughout. It lacks a clear organizational structure and the message is
difficult to follow. Arguments are inadequately presented and supported; they may be
irrelevant; if the writers experience or views are presented, their relevance may be
difficult to see. The control of vocabulary, spelling, punctuation and grammar is
inadequate, and the writer displays inability to manipulate the linguistic systems
appropriately.
3
The writing does not display an ability to communicate although meaning comes
through spasmodically. The reader cannot find any organizational structure and
cannot follow a message. Some with an argument is mainly irrelevant. The reader is
primarily aware of gross inadequacies of vocabulary, spelling, linguistic appropriacy,
although there is no evidence of sentence structure.
2
The writing displays no ability to communicate. No organisational structure or
message is recognizable. A meaning comes through occasionally but it is not relevant.
There is no evidence of control of vocabulary, spelling, punctuation or grammar and
no sense of linguistic appropriacy
1
A true non-writer who has not produced any assessable strings of English writing.
An answer which is wholly or almost wholly copied from the input text or task is in
this category
0
Should only be used where a candidate did not attend or attempt this part of the test
in any way
Analytic marking scales
Methods of scoring that require a separate score for each of a number of aspects of
tasks are said to be analytic.
Transform the holistic marking scale in Table 11 into an analytical marking
scale in Table 12
Table 12 BC analytical writing evaluation scale
9
8
7
19
6
5
4
3
2
1
QUALSPELL Marking scale
Grammar
Mechanical
Accuracy
Vocabulary
Structure
Content and
Style
5
Wide range and
correct use of
complex
structures, very
few minor errors
Almost no
errors of
spelling,
punctuation,
capitalization
Wide range of
vocabulary,
appropriate
register, consistent
style
Completely
logical, varied
use of cohesive
structures,
complex
sentence
structures
Arguments effectively
related to specific area
within topic,
interesting
presentation of topic,
appropriate tone and
register
4
Good general
command of
grammar with no
systematic errors
Occasional
errors that do
not obscure the
meaning
Good range of
vocabulary (not
necessarily
outstanding, some
circumlocutions,
some errors of
word forms,
choice, usage, but
meaning not
confused
Generally logical,
sufficient use of
various cohesive
devices, complex
sentence
structure with
occasional errors
Arguments mostly
related to the specific
area within the topic,
mostly appropriate
tone and register
3
Accuracy high,
some gross or
systematic errors
and more minor
errors which do
not distort
communication
Frequent or
systematic
errors, but
meaning
mostly not
confused or
obscured
Limited
vocabulary, but
largely correct,
sometimes
“adventurous”
choice of words,
meaning not
confused
Inconsistent
structure, but
easily to follow
most of the time,
occasionally
repetitive, some
variety of
sentence
structure, some
use of cohesive
Arguments related to
the general topic,
occasional
inappropriate tone and
register
20
structure
2
Has a basic
grasp of the
grammar system
which is not
applied
consistently,
frequent
occurrence of
errors
Frequent or
systematic
errors,
meaning
confused or
obscured, poor
handwriting
Range inadequate
to express ideas
effectively,
frequent errors of
words, idioms,
form, usage,
meaning
sometimes
confused
Loosely
organised, but
main ideas
expressed, can
be followed, but
requires some rereading,
sentences short
and stereotyped,
attempts at using
cohesion
Written mostly to the
point with occasional
lapses, inconsistent in
tone and register
1
Very frequent
occurrence of
gross errors,
grammatical
patterns based
on direct
translation
Dominated by
errors,
meaning
confused or
obscured,
serious
misspellings of
elementary
words
Limited range,
Lack of basic
organisation,
only simple
sentences or
serious, frequent
errors in
sentence
structure, not
cohesive
Limited treatment of
topic, major points left
out, insufficient length,
mostly inappropriate
style
Completely
ungrammatical
Misspellings of
basic words,
illegible
handwriting
0
Misuse of words
makes
communication
difficult, essentially
translation from L1
Misuse of words
breaks down
communication
Vague, confused
and
disconnected
ideas
Assurance of test quality
Validity
http://exchanges.state.gov/forum/vols/vol40/no2/p14.htm
Validity is the degree to which a test or assignment actually measures what it is
intended to measure. There are five important aspects of validity (Hamp-Lyons 1991;
Jacobs et al. 1981):
1. Face validity Does the test appear to measure what it purports to measure?
2. Content validity Does the test require writers to perform tasks similar to what
they are normally required to do in the classroom? Does it sample these tasks
representatively?
3. Concurrent validity Does the test require the same skill or sub-skills that other
similar tests require?
4. Construct validity Do the test results provide significant information about a
learner’s ability to communicate effectively in English?
5. Predictive validity Does the test predict learners’ performance at some future
time?
Hughes (1989): Specific guidelines for moderation of a task: contents
1. Is the task properly representative of tasks that we should expect the testtakers to be able to perform
21
2. Will the tasks elicit samples of writing which truly represent the students'
ability?
3. Will the tasks help markers score reliably?
4. Does the task test writing ability and nothing else? In language testing we are
not normally interested in knowing whether students are creative, imaginative,
or even intelligent, have wide general knowledge or have good reasons for the
opinions they hold. For that reason we should not set tasks which measure
these abilities.
22
What do you think the following tasks test?
Table 13 Validity of the writing tasks
Task
Write the conversation you
have with a friend about
the holiday you plan to
have together
What does the task test?
You spend a year abroad.
While you are there, you
are asked to talk to a group
of young people about life in
your country. Write down
what you would say.
The advantages and
disadvantages of being born
into a wealthy family.
'Envy is the sin which most
harms the sinner' Discuss
Reliability of the examination
http://exchanges.state.gov/forum/vols/vol40/no2/p14.htm
Reliability is the degree to which the scores assigned to students’ work accurately and
consistently indicate their levels of performance or proficiency.
Inter and Intra-marker reliability
Correlation coefficients of .80 and above between readers’ scores (inter-rater
reliability) as well as between the scores assigned by the same reader (intra-rater
reliability) to the same task are considered acceptable for decision making (Bachman
1990).
There is research that indicates that the gender, background, and training of the
reader can affect the reliability of scores (Brown 1991; Cushing-Weigle 1994).
Thus, to maintain reliability many programs put heavy emphasis on the training of
raters and as a result have obtained high positive correlations (Jacobs et al. 1981;
Hamp-Lyons 1991).
Hughes (1989) suggests the following steps to make the scoring of writing tasks
more reliable:
23
1. set as many tasks as possible : the more scores that scorers provide for
each candidate, the more reliable the total score
2. restrict candidates: the greater the restrictions imposed on the
candidates (writing tasks should be well defined and candidates should
know just what is required of them) the more reliable should be the total
score
3. give no choice of tasks: making the candidates perform all tasks also
makes comparisons between candidates easier
4. ensure long enough samples: the samples should be long enough for
judgements to be made reliably, especially if you are evaluating
organisation
5. Once we have written a task and pretested it, we can start developing a
marking scale (holistic or analytical)
http://www.rtlib.com/vita/home.nsf/content/y12
Table 14
Reliability of the Year 12 examination in 1999
Task
Reading
Matching
Gap-filling
Multiple-choice
Listening
Gap-filling
Multiple-choice
true/false
Language Use
Multiple choice
Editing
Gap-filling
Writing
Postcard
Letter
Report
Speaking
Total
points
SEM
Alpha
37
11
20
6
40
17
12
11
51
19
16
16
60
15
20
25
25
100%
2.4
1.3
1.7
1.0
2.3
1.2
1.4
1.3
2.9
1.7
1.7
1.5
1.3
0.2
0.5
0.7
0.6
0.2
0.895
0.711
0.866
0.622
0.776
0.708
0.555
0.421
0.932
0.815
0.799
0.875
No data*
No data
No data
No data
.572**
Not appropriate
Correlation
with the
total***
0.885
0.640
0.858
0.643
0.812
0.683
0.683
0.558
0.700
0.642
0.608
0.671
0.761
0.542
0.670
0.725
0.823
Not appropriate
* In 1999 CEC data package did not contain the results of the first and second marking
**Correlation between the first and the second marking
***See Intercorrelations between the different tests in Appendix 12
Table 15 Intercorrelations between the different tests in 2001
Total
Listening
Total
1.000
0.915
Listening Reading
0.915
0.884
1.000
0.814
Lang.use
0.872
0.766
Speaking
0.849
0.688
Writing
0.869
0.709
24
Reading
Language use
Speaking
Writing
0.884
0.872
0.849
0.869
0.814
0.766
0.688
0.709
1.000
0.742
0.651
0.682
0.742
1.000
0.638
0.685
0.651
0.638
1.000
0.763
0.682
0.685
0.763
1.000
25
Bibliography
Bachman, L.F. 1990: Fundamental considerations in language testing. Oxford:
Oxford University Press.
Common European Framework of language learning, teaching and assessment,
Strasbourg 1996
Cushing-Weigle S., Assessing Writing Cambridge University Press, 2002
Hamp-Lyons, L. & W. Condon (1993) ‘Questioning the assumptions about portfolio-based
assessment’ College Composition and Communication, 44/2: 176-190.
Hughes, A. Testing for Language Teachers. Cambridge: Cambridge University Press,
1989.
Tribble C. Writing. Oxford University Press,1995
Weir, C. Communicative Language Testing. Prentice Hall, 1995
Online sources
http://www.lu.lv/mvf/Studij_bak.htm
http://exchanges.state.gov/forum/vols/vol40/no2/p14.htm
http://www.dartmouth.edu/~writing/services/support_student.shtml
http://moodle.esp-c.org/mod/resource/view.php?id=727
http://www.provost.cmich.edu/assessment/toolkit/formativesummative.htm
www.britishcouncil.org/lithuania-education-exams-ielts.htm
http://www.rtlib.com/vita/home.nsf/content/y12
26
Testing Writing Seminar Evaluation Sheet
Please evaluate the usefulness of the seminar from 1 (not useful)- 5 (very useful),
please give your comment and suggestions for the further seminars
The topic of the seminar 1…2…3…4…5……………………………………………………
The format of the seminar 1…2…3…4…5……………………………………………………
Session 1: Writing as a concept
1…2…3…4…5…………………………………………………
Session 2: Approaches to testing writing
…2…3…4…5…………………………………………
Session 3: Writing tests and tasks
1…2…3…4…5…………………………………………………
Session 4: Marking scale types and marking reliability
1…2…3…4…5…………………………
Session 5: New task development
1…2…3…4…5…………………………………………………
Session 6: Marking scale development
1…2…3…4…5……………………………………………
Session 7: Pretesting test tasks
1…2…3…4…5……………………………………………………
Session 8: Feedback on the tasks developed
1…2…3…4…5………………………………………
27
Suggestions for the further
seminars…………………………………………………………………
Thank you for your help!
28