Evaluating the effectiveness of a
task-based syllabus for teaching
speaking to advanced learners of
English
Melanie Ellis
Foreign Language Teacher Training
College, Zabrze, Poland
melanie@ellis.pol.pl
Defining terms
Evaluation: the process of determining how a
program worked in practice, by collecting
varied informative data from a variety of
sources
Why? When? Whom? What? How? For whom?
(Alderson 1986)
Task-based syllabus
• Process merging ‘what’ is to be learnt and
‘how’ this is to take place (Nunan 1989)
Specification of tasks: type, topic
Specification of performance conditions:
support, planning time, rehearsal, interaction
Sequencing criteria: progession from easy to
more difficult
(after Ellis, R. 2003:206)
Plus syllabus design principles based on Stenhouse (1975)
Stenhouse (1975:4)
An attempt to communicate the essential
principles and features of an educational
proposal in such a form that it is open to
scutiny and capable of effective translation
into practice
*Seen as key, as little research available on TB
syllabuses in practice
Definition of task
• Activity where communication of meaning
is of first importance
• Successful completion of the task is one of the
main aims
• Usually involves some communicative
challenge to solve
• Is open in terms of language (‘unfocused’)
(based on Skehan 1998:95)
Why evaluate?
• Accountability: Is the program effective? Does
the syllabus meet its objectives? Is it valid?
• Curriculum development and betterment:
Is a task-based syllabus feasible? Does it give
learners a sense of progress?
• For purposes of teacher self-development
Doctoral research conducted by the teacher as
researcher
(Rea-Dickens & Germaine 1992:26)
When? Whom? What?
• 3 year foreign language teacher training
college in Zabrze, Poland
• First year undergraduates, school leavers and
mature students. 3 class groups, n=39
• Academic year (2002-3) one 2 hour class a
week, over 30 weeks
• Learners at B2-C1 (CEF) Target on graduation
from college is C2
• ‘Conversation’ course
Syllabus design principles:sequencing
1. Task category
Description>description/instruction>narration>
opinion-giving/analysis/synthesis (Brown & Yule 1983, Bloom
1956)
2. Performance conditions-organization & support, feedback
Closed pair………………………………………open forum
Prompts + planning + rehearsal…………………………..spontaneous
speech
(Bygate 2001, Nunan 1989)
3. Task difficulty code & cognitive complexity
Familiarity. No. of elements, Structure etc.
(Skehan 1998, Robinson 2001)
Syllabus objectives
The course aims to help learners:
become more confident in speaking English
develop fluency
increase vocabulary
increase awareness of their strengths and
weaknesses in speaking and how to improve
these
develop compensation strategies
How was the evaluation done ?
• Questionnaires after first and second semesters,
closed and open questions
• 9 Case studies, 8 guided interviews conducted
• Teacher notes
• Quantitative evaluation of speech samples for
evidence of development of fluency (MLU, words
/min., % pause ) and increase in lexical
complexity (type-token ratio, no. of clauses)
• Intact class group results compared with sub grps
divided according to ability
Expected outcomes
Learners
• become more confident in speaking
• develop greater fluency
• produce more complex speech, particularly
richer in lexis
• feel that there is a progression from easier to
more difficult in terms of task sequencing
More confident?
Questionnaire 1
In what ways has your speaking improved in
this class in this semester?
40% ‘more confident’, ‘more self-assured’ ‘more
brave’ in speaking
35% more fluent
55% vocabulary has increased
20% better pronunciation
More confident?
Questionnaire 2
26% more confident, particularly at speaking in
front of the group
Interview data: In what way has your speaking
improved?
6/8 mention confidence
I gained lots of confidence
Now I’m more confident in my use of the language.
I’m less stressed
I’m not afraid now to talk in English
I break the barriers. The barriers were my shyness
Was there a gain in fluency?
Caveats
Learners are in an English-medium program 24
hours a week
Comparability of first and final tasks- problematic
Picture description is not representatative, ‘task’ ?
Sample picture from final assessment
First assessment: October
• Picture description: repeated task
• Describe picture for partner to identify
• Describe from a prompt: new task, no
preparation eg. Describe sb you know well;
Describe a member of your family; Describe
one of your relatives
October-June: Fluency
Group
MLU
MLU
% pauses
% pauses
WPM
WPM
A
11.9
20.96
19.08
11.83
70.07
91.6
B
8.01
20.34
20.69
12.87
63.27
86.92
C
10.64 15.48
18.17
14.2
65.12
66.36
October to June
/Well the lady is is about I think she’s in her
fifties/she’s a bit plump/she’s wearing a black
pleated skirt/under a um striped coat/um:um she
has:blond hair tied er in a plait/
/OK/: these are two mothers with their
children/one is with a boy with a boy and the
other one is with a little girl/: um the woman the
mother on the left is wearing a white
sweater/she’s also a necklace/: and her hair is
brown/
Sub-groups: October to June
• Significant increases in fluency in all measures
in Strong group and Below Average group
• Average group: significant gains in MLU,
significant reduction in amount of pause, no
gain observed in words per minute (SD =
17.17 October; 21.97 June)
• WPM as useful measure of fluency across a
group? Very high SDs in all cases
Is there a sense of progression?
Sample task sequence
• Known tasks (matura); rehearsed: describe
picture of house
• Parallel: you visited this place last weekend,
tell a friend at work about your impressions
• Personalise: place you often visited as a child
Take your partner on a guided tour. They will
later have to explain to someone else where
they went and what they experienced.
Is there a sense of progression?
Data from 2 questionnaires
Learners rated tasks 1 (very difficult for you) – 5
(very easy for you)
Open questions:
Explain which tasks were most difficult for you
and why
Explain which tasks were easiest for you and
why
Most difficult
Picture story (new task, no preparation) : 52%
People and places tasks with no support: 10%
People task no support: 10%
All unfamiliar tasks: 10%
Why?
Lack of vocabulary 40%
Not enough preparation time 22%
Didn’t know what to say 17%
Easiest
All pictures seen before 35%
Re-telling a story 35%
Describing people 25%
Places: pair task (describe and identify) 25%
Why?
I had practised and learnt vocabulary 37%
Task was familiar 12%
Task was interesting 10%
Sequencing criteria confirmed
•
•
•
•
•
•
•
Familiarity 
Repeated, rehearsed tasks 
Describing people 
Minus support 
Minus familiarity 
Minus preparation 
Minus structure 
Increase in vocabulary?
Contradictory evidence
Questionnaire 1: In what ways has your speaking
improved in this class in this semester?
55% report increase in vocabulary
Questionnaire 2: In what ways has your speaking
improved and how much (1-5 where 5 = a lot)
Vocabulary : mean value 4.67
But: No gains in Type Token Ratio although increase
in number of clauses
> > > More research needed
Complexity: October- June
Group
Clauses
Clauses
TTR
TTR
A
1.05
2.09
0.31
0.24
B
0.77
1.89
0.28
0.22
C
0.81
1.63
0.33
0.28
Conclusions
• Syllabus achieves objectives in the eyes of the
learners
• Fluency increases
• Sense of progression perceived
• Design principles confirmed
• Vocabulary development? Other measures ?
Other tests? More research needed
Thank you for your attention
melanie@ellis.pol.pl
References
•
•
•
•
•
•
•
•
•
•
•
Alderson, J.C. 1986. The nature of the beast. Trends in Language Programme evaluation. Bangkok:
Chulalongkorn University Language Institute
Bloom, B.S. (ed.) 1956 Taxonomy of Educational Objectives. New York: David McKay Company Inc.
Brown, G & Yule, G. 1984. Teaching the Spoken Language. Cambridge: Cambridge University Press
Ellis, M. 2004. Developing speaking skills in the teaching of English at the advanced level. Design,
implementation and evaluation of a task-based syllabus for trainees in pre-service education.
Unpublished PhD thesis . University of Warsaw, Poland
Ellis, M. 2008. Design & evaluation of a task-based syllabus for developing speaking skills. In Pawlak,
M. (ed.) Investigating English Language Learning & Teaching. Poznan-Kalisz: Faculty of Pedagogy &
Fine Arts in Kalisz, Adam Mickiewicz University , Poznan
Ellis, R. 2003. Task-based Language Learning and Teaching. Oxford: Oxford University Press
Nunan, D. 1989. Designing Tasks for the Communicative Classroom. Cambridge: Cambridge
University Press
Rea-Dickens, P & Germaine, K. 1992. Evaluation. Oxford: Oxford University Press
Robinson, P. 2001. Task complexity, task difficulty and task production: exploring interactions in a
componential framework. Applied Linguistics 22/1: 27-57
Skehan, P. 1998. A Cognitive Approach to Language Learning. Oxford: Oxford University Press
Stenhouse, L. 1975. An Introduction to Curriculum Research and Development. London: Heinemann