v>
EVALUATION REPORT
EVALUA
TION OF THE WRITING
TEACHING
PROGRAM
COMPONENT
AT A PUBLIC
OF AN ENGLISH
UNIVERSITY:
A CASE
LANGUAGE
STUDY
The lasl step o f the context-adaptive evaluation model (Lynch, 1990) is formulating
the evaluation report. The findings o f this evaluation study are discussed with special focus on
the participants' views about different dimensions o f the program called by Brown (1995) as
the elements o f a language curriculum (i.e. objectives, materials and content, teaching and
testing). The results obtained from different data collection tools arc presented focusing on
each o f the above-mentioned elements o f the writing program and the findings are discussed
in the light o f the perspectives o f students and instructors at the D B E as well as the content
course instructors. Also, some implications are provided for the improvement o f the writing
program at the D B F .
The following tables are the summary o f the D B E participants.
Tabic 1 Demographic data about DBE students
Gender
Age
N
%
Female
198
47.8
Male
216
52.2
17-19
322
77.8
92
22.2
278
67.1
20-22
Anatolian High S c h o o l
T y p e o f High S c h o o l
S u p e r High S c h o o l
3
0.7
S c i c n c c High S c h o o l
44
10.6
V o c a t i o n a l High S c h o o l
15
3.6
Private High S c h o o l
11
2.7
M a i n s t r e a m High school
34
8.2
Anatolian T e a c h c r High
29
7.0
Pre-intcrmediale
136
32.9
Intermediate
114
27.5
Upper-intermediate
98
23.7
Advanced
66
15.9
20-25
15
3.6
26-31
108
26.1
32-37
236
57XT]
38-43
55
13.3
414
100
School
P r o f i c i e n c y L e v e l in the
Second T e r m
First T e r m G r a d e
out o f 45
TOTAL
Tabic 2 Demographic data about DBE instructors
Male
Gender
Female
Years of Teaching 1-10
N
%
9
18.0
41
82.0
32
64.0
30.0
E x p e r i e n c e a t the
11-20
15
DBE
21-30
3
6.0
I'rc-intcrmediate
14
28.0
13
26.0
15
30.0
8
16.0
50
100
L e v e l t e a c h i n g in
Intermediate
the second
Upper-
semester
intermediate
Advanced
TOTAL
Students' and Instructors' Perspectives
The analysis o f the syllabus o f the writing program at the D B E revealed the objectives
o f the program and these objectives were listed as skills in the instructor and the student
questionnaires to obtain data about how competent students became in these objectives. Both
students and instructors at the D B E held the idea that these objectives were generally
achieved by the students. Still, there was a common objective (using appropriate register in
writing) responded more negatively than others by both students and instructors. The
importance o f this skill was also emphasized in the literature. For instance, as one o f the
characteristics lor an effective writing course, Holbrook ( 1 9 8 4 ) emphasized the skill o f
writing appropriately for many audiences.
Also, it was found that not only instructors but also students argued that the writing
program at the D B E was very prescriptive because o f the discourse patterns students had to
memorize. Tarnopolsky and Kozhushko ( 2 0 0 7 ) describes such an approach to writing
instructions as textual approach requiring students to read sample texts and then write by
using the linguistic and the stylistic patterns they learnt through the sample texts. They also
underline that in this approach, there are strict regulations as for issues like thesis statement
and topic sentences. From their perspective, such an approach hinders the creativity o f the
academic writing and turns it into a process o f imitating the sample texts.
With regards to the prescriptive nature o f discourse-level writing at the D B E , students
and instructors recommended that a free writing approach should be adopted to avoid
focusing too much on the discourse patterns. In other words, they expressed that free-writing
should be included as an objective in the writing program o f the D B E . Similarly, some
instructors suggested that the number o f discourse types taught at the D B E should be reduced
and more focus should be placed on the content o f the written text rather than the fixed
discourse patterns. As Mohamed ( 2 0 0 4 ) points out, students should investigate the language
and experiment freely with it to write more creatively. The need for less controlled and guided
tasks was supported by some Turkish researchers as well. For instance, Atay and Kurt ( 2 0 0 6 )
suggested that writing in English should not be restricted to controlled writing exercises. The
reason why Turkish students are not very good at expressing their original thoughts in writing
in L2 (Alagozlu, 2 0 0 7 ) might be related to the controlled and guided manner Turkish E F L
instructors follow in teaching writing.
Another point students and instructors agreed upon as for program objectives was the
need to include essay writing as an objective o f the writing program. The same issue was also
suggested in the sharing and feedback sessions held at the D B E last year (Retrieved on
September
1, 2 0 1 0
from http://dbe.metu.edu.tr/port/documents).
In that meeting,
some
instructors argued that essays are more realistic than condensed paragraph writing. As for
preparing students better for their departments, essays can be favored more than paragraphs as
some o f the content course instructors who took part in this study also stated that they
required their students to write essay type written works, rather than paragraphs. In terms o f
whether the program objectives were fully achieved by the students o f not, some instructors
arguing that the objectives were not fully achieved claimcd that the reason for this failure
might have to do with the lack o f consensus among instructors on what to emphasize in the
program, students' poor English background, the large size o f the class and the lack o f time
allocated for each objective.
The second element o f the program, materials and content, were generally evaluated
positively by both students and instructors. The results with regards to the evaluation o f the
materials and content used to teach writing at the D B E demonstrate that the majority o f the
students and the instructors held the idea that the current materials and content was rather
effective despite some points to consider for the improvement o f the materials and the
relevant contents. Among these points, the visual unattractiveness o f the materials (i.e.
handouts) was the common concern for both students and instructors. As for ideal writing
materials to be used in E F L classrooms, Qian ( 2 0 1 0 ) asserts that writing materials should
provide stimulating activities by means o f audiovisual aids, such as tapes, pictures, and
diagrams.
With respect to the materials, favoring the idea o f a writing book, some students
suggested that the handouts should be compiled in the form o f a book or a writing coursc
book available should be used to teach writing at the D B E . Some students also voiccd their
concerns about the number o f sample texts in the handouts and suggested that there was a
need to include more sample paragraphs to analyze. On the other hand, some instructors
believed that although some o f the writing sections o f the Language
Leader
were good to
practice students' fee-writing skills, they could not focus on these tasks due to time
constraints and the inconsistency between the writing parts o f the book and the writing exam
questions. When it comes to the content o f these materials, both students and instructors
reached the consensus as to the need to give students the flexibility to choose the topics to
write about. Analyzing the writing needs o f E L T students at METU, Ozbek (1995) also found
that students would like to given a variety o f topics in order that they can choosc the topic that
interests them most. Similarly, as stated by Tribblc (1996), students should be assigned
writing tasks that are related to their real problems to increase students' motivation to start
writing.
Also, parallel to some D B E students' opinions, some instructors argued that the content
o f the materials to teach writing should have some rclcvancc to the students' own
departments. Content course instructors in various departments and the students as well as
instructors at the DML made similar suggestions. Although Gcrcdc (2005) stated that it is not
very likely for preparatory programs to focus on students' language needs relevant to each
subjcct area because o f the variety o f students at different departments, it could be argued that
students at the D B E could be grouped according to their departments and students can at least
be familiarized the terminology relevant to their fields. As emphasized by Mirici and Demirel
(1999), students at the preparatory classes should be provided with general
English
knowledge as well as some terminology related to their departments. As also stated by Mirici
and Saka (2004), the English preparatory programs should includc Technical English. They
point out that especially students o f engineering faculties should learn English both for
general and for specific purposes so that they can keep up with the developments in their field
using the Technical English background.
As suggested by Lcki (2003), the common characteristic in most L2 academic writing
classcs is that they are "freestanding, self-contained, and dctachcd from the rest o f students'
academic lives". Therefore, it can be argued that students should be taught writing skills
through topics about their departments or the topics they are mostly interested in. A needs
analysis study aiming to reveal the topics students would like to have in writing materials
could provide some valuable data about the issue.
The third element o f the L2 writing program at the DBE was the tcaching-lcarning
process embodying the activities and the methods used to teach writing. Like students,
although generally positive about most o f the relevant issues like the teacher feedback,
instructors expressed uncertainty and disagreement with the idea that there was enough
variety o f activities and the conduct o f the writing lessons was interesting. Besides, both
students and instructors ranked the following activities as not frequently used: student
presentations, role-plays, projects, games, journal keeping, brainstorming and dictation. Also,
some instructors proposed the following activities and methods for the improvement o f the
teaching-learning process in writing-focused lessons: integration o f the writing skill to the
other basic skills, use o f process-based writing activities like drafting and outlining, activities
integrating technology, more self-correction, the analysis o f poor, adequate and good written
samples, projects and student presentations; more use o f pair and group work activities, more
student-centered tasks and more pcer-to-peer feedback applied with the use o f relevant
checklists. As one can see from the list o f activities and methods proposed by the instructors,
the proposed activities and methods were student-centered activities that were not adequately
done in the lessons. Looking into these activities, one can draw the conclusion that there is a
need for more studcnt-ccntcrcd teaching at the S F L (the D B E and the DML), METU. The
reason why these activities cannot be done sufficiently in class might be the large class size
which was also evaluated negatively by some D B E students. Similarly, in his evaluation o f
the theme-based curriculum at the D B E , Top^u ( 2 0 0 5 ) found that pair/group work was
considered as ineffective due to time constraints in the program. Some other research studies
conducted in different preparatory schools in Turkey also revealed the need to incorporate
more student-centered activities. T o illustrate, Gokdcmir ( 2 0 1 0 ) carried out an evaluation
study in 5 different universities and found that the lessons at preparatory classes are generally
teachcr-ccntcrcd. Dcmirta§ and Sert ( 2 0 1 0 ) also drew attention to the teachcr-centered
traditional approaches as the cause o f the low proficiency level o f students in Turkish
universities. As for peer feedback using a checklist, Fearn and Farnan ( 2 0 0 1 ) warned that the
teacher should train their students to use the checklists effectively, which will enable teachers
and students to work collaboratively.
The fourth element o f the writing program evaluated by the students and instructors
was the assessment. Despite high level o f agreement with the effectiveness o f the assessment
procedure to test writing at the D B E , both students and instructors agreed on the idea that the
mid-term exam results do not demonstrate students' actual ability in writing very much.
Parallel to the suggestions made by some students for the improvement o f the assessment
dimension o f the writing program, some instructors rccommcndcd that there was a need for
the standardization procedure applied to assess writing. For standardization purposes, both
students and instructors argued for checking inter-rater reliability while some instructors
underlined the need for a more detailed holistic marking scale and the training o f the
instructors by means o f workshops related to the standardization o f the assessment. With
regards to inter-rater reliability, Coombe ( 2 0 1 0 ) underlined the importance o f double marking
as a means to maintain writing test reliability. On the other hand, for a more dependable
assessment procedure for writing, Enginarlar ( 1 9 9 1 ) suggests the following three factors to
take into account: established scoring systems, tested writing prompts and carefully trained
raters. At the end o f the same study, he presented some invaluable recommendations for
scoring students' written works with a holistic scoring key. He suggested that instructors
"should first read the papers quickly and placc them in batches, ranging from 'very good' to
'nilP according to first impressions" (p. 43). He recommended that that instructors should
then read and reassess the written works again. Also, it was also suggested in his study that
"instructors should not grade papers for longer than one hour in one session (i.e., give a break
when you arc tired" (p. 44).
These guidelines pave the way for a more reliable testing
procedure for both departments.
Another suggestion made not only by students but also by instructors at the D B E was
that writing tasks as tested in the E P E should be included in the program because E P E
includes some parts (e.g., responding to a reading text in a written form) that arc hardly ever
covcred in the lessons. It was also recommended by a few instructors that there was no need
to tcach discourse-level writing to D B E students as the writing section o f the E P E does not
necessitate any use o f discourse patterns. On the other hand, complaining about the lack o f
writing pop-quizzes and the lack o f time allocated for the writing parts o f the midterms, some
students underlined the need for more pop-quizzes and more for writing tasks in the exams. A
similar suggestion was also made by Coombe and Evans ( 2 0 0 0 ) who argued that instead o f
testing writing only in mid-term or final exams, students' writing ability should be assessed
more frequently for a more fair assessment. When it comes to the frequent assessment o f the
writing skills, the issue o f portfolio assessment comes into play and this assessment tool was
favored by a few D B E instructors who held the idea that portfolio assessment should be a part
o f their assessment procedure applied to test writing. In informal interviews with an
instructor, I was informed that portfolios were used in the past to test students' writing ability
as the instructors were not trained enough, this alternative assessment tool was abandoned.
Still, it can be argued that portfolio assessment could be a good solution to deal with validity
and reliability conccrns expressed by some participants o f the current research (Enginarlar,
1994b) i f the necessary training could be provided to writing instructors at the D B E .
Finally, whether the D B E prepares students for the English writing requirements o f the
D M L courses and the departmental courses was not responded by the majority o f the
instructors as they stated that they were not fully aware o f the writing program at the D M L or
the writing requirements at different departments. For this reason, it would be fair to suggest
that cooperation between the instructors at the D B E and the D M L is necessary to raise
instructors' awareness o f the common goal o f both departments to prepare students for their
departments.
Content Course Instructors' Opinions
Afler summarizing findings as to how D B E and D M L students and instructors
evaluated their program, results o f the content course instructors who have
firsthand
experience with the written works o f students who completed the D B E and the D M L writing
programs are discussed in this part o f the study. The questions intended to reveal the context
o f the departments with regards to L2 writing, demonstrated that while some courses require a
lot o f L2 writing, some others do not. It was also realized that English writing was commonly
required to give written answers to exam questions, to write term papers, research proposals,
essays, reflection papers, reports (e.g., lab. reports) and book reviews.
In order to evaluate the objectives from the perspective o f the content course
instructors, they were first asked to indicate how competent their students were in the given
skills which were actually the objectives o f the D B E and the D M L writing program. Most o f
the content course instructors were found to hold positive attitudes towards their students'
competencies in English writing. Still, some objectives as can be listed as follows received
comparatively more negative responses by the content course instructors: 'using appropriate
discourse patterns', 'using correct, appropriate language structures, vocabulary and discourse
markers',
'writing academic paragraphs with appropriate topic, supporting and concluding
sentences', 'identifying rcfcrcncc information' and 'achieving unity in writing through the use
o f appropriate signal words'. Some instructors made additional claims about students'
speaking performance as they thought that speaking skills should also be emphasized more at
the S F L . A similar suggestion was also made by a couple o f D M L students.
When asked how well the S F L prepared students for the writing requirements o f the
courses they offered, instructors held different views. Even though three instructors responded
positively to this question, three other instructors shared their bad memories about their
students' written works and the written errors students made in their courses. As to the
preparation o f S F L students for their departments, three other instructors stated that it was not
the S F L to put the blame on i f students could not write very well in their departments. Instead,
these instructors rccommcnded that more written work should be done in the departments.
The same suggestion was also made by a few o f the D M L instructors.
The second dimension o f the program evaluated by the content coursc instructors was
the materials and contents at the D B E and the DML. Considering that the participants were
unlikely to know enough about materials and contents to make an evaluation o f this
dimension, a short explanation about D B E and the D M L materials and contents used to teach
writing at the D B E . With respcct to the materials and content o f the writing program at the
SFL, more than half o f the instructors argued for the need to integrate writing contents and
materials that have some relevance to students' departments. The issue o f topics relevant to
students' own departments was also brought up by a great majority o f the D B E and the D M L
students and instructors.
In order to improve the writing program at the S F L , some instructors highlighted the
importance o f cooperation between S F L and contcnt coursc instructors in order that the S F L
instructors could be more aware o f the students' future needs as for the writing skill. As most
o f the D B E and the D M L instructors expressed that they were not fully aware o f the writing
needs o f their students, there is a need for more cooperation between the instructors at the
S F L and the departments at M E T U . Finally, it was recommended by one instructor that
students hold negative altitudes towards writing in their departments; as a result, the S F L
ought to encourage students to start writing by writing about themselves before leading them
to formal writing.
Implications of Ihc Study
The current study has some implications for the improvement o f the writing program at
the D B E . As identified in the first step o f the context-adaptive evaluation model, the main
goal o f this evaluation was to make recommendations to improve it so that they can utilize the
findings o f this study and make necessary changes. Considering the findings o f the current
evaluation study, the following
recommendations
can be made for the improvement o f the
writing program at the D B E at Middle East Technical University. These recommendations
could be taken into account by the Administration and the instructors at the S F L who will see
the evaluation report.
1. The writing program at the D B E was generally considered to be prescriptive in nature.
Therefore, it is suggested that writing should be less focused on different types o f
discourse and the discourse patterns that students generally memorize and be more
focused on the content o f the written texts. As a reaction to controlled writing, Baskan
( 2 0 0 2 ) proposed a free writing approach reflecting the basic characteristics o f the
process approach for the D B E . A process-based approach including stages like
drafting and outlining was also suggested by the content course instructors.
2.
As it was suggested by both students and instructors, essay writing should be included
as an objective o f the writing program at the D B E .
3.
For the achievement o f the D B E writing objectives, it is recommended to have fewer
students ( 1 5 - 2 0 ) and more time should be allocated for each objective.
4.
In terms o f the materials o f the writing program at the D B E , the common conccrn for
students and instructors was that there should be more visually attractive materials (i.e.
handouts). As for the topic to write about there is a need to give students the flexibility
to choose the topics to write about. Also, it is recommended that the content o f the
materials to teach writing should have some relevance to the students'
own
departments. As tried at the D B E in the past, students can be grouped according to
their future departments and after a careful need analysis involving both the students
and their department
instructors,
the topics
found to be
related
to
students'
departments could be incorporated into the writing materials.
5.
As far as the teaching-learning process applied in writing-focused lessons at the D B E ,
it can be recommended that activities like journal keeping, projects and student
presentations should be more frequently done in the lessons. More student-centered
tasks and self-correction and peer-to-peer feedback applied with the use o f relevant
chccklists are necessary to apply in writing lessons.
6.
For the standardization o f the assessment procedure for the writing ability, there is a
need for double rating to ensure a satisfactory level o f inter-rater reliability. It is also
suggested that D B E instructors should be trained on the standardization o f the
assessment o f the writing skills.
7.
Because E P E includes some parts (e.g., responding to a reading text in a written form)
that are hardly ever covered in the lessons, it can also be recommended that the
writing tasks as tested in the E P E should be included in the program. Also, as the
discourse-level writing is not tested in the paragraphs writing section o f the EPE, it
can be recommended that there is no need for too much focus on ccrtain discourse
patterns. 'Hie feedback sessions about the program o f the D B E last year (Retrieved
from http://dbc.mctu.edu.tr/port/documents) also implied that students unfamiliar with
"discourse" patterns were more successful in the writing section o f the Proficiency
Exam while our students get confused trying to fit their writing to a "discourse" rather
than answering the question.
As suggested by D B E students, there is a need for more writing pop-quizzes and more
time should be given for the writing parts o f the midterm exam. Moreover, portfolio
assessment should be implemented as an alternative way to test writing at the D B E
and the necessary training on how to effectively exploit portfolios should be provided
to instructors at the D B E .
1. The writing program at the D B E was generally considered to be prescriptive in nature.
Therefore, it is suggested that writing should be less focused on different types o f
discourse and the discourse patterns that students generally memorize and be more
focused on the content o f the written texts. As a reaction to controlled writing, Baskan
( 2 0 0 2 ) proposed a free writing approach reflecting the basic characteristics o f the
process approach for the D B E . A process-based approach including stages like
drafting and outlining was also suggested by the content course instructors.
2.
As it was suggested by both students and instructors, essay writing should be included
as an objective o f the writing program at the D B E .
3.
For the achievement o f the D B E writing objectives, it is recommended to have fewer
students (15-20) and more time should be allocated for each objective.
4.
In terms o f the materials o f the writing program at the D B E , the common concern for
students and instructors was that there should be more visually attractive materials (i.e.
handouts). As for the topic to write about there is a need to give students the flexibility
to choose the topics to write about. Also, it is recommended that the content o f the
materials to teach writing should have some relevance to the students'
own
departments. As tried at the D B E in the past, students can be grouped according to
their future departments and after a careful need analysis involving both the students
and their department
instructors,
the topics
found to
be
related
to
students'
departments could be incorporated into the writing materials.
5.
As far as the teaching-learning process applied in writing-focused lessons at the D B E ,
it can be recommended that activities like journal keeping, projects and student
presentations should be more frequently done in the lessons. More student-centered
tasks and self-correction and peer-to-peer feedback applied with the use o f relevant
chccklists are necessary to apply in writing lessons.
6.
For the standardization o f the assessment procedure for the writing ability, there is a
need for double rating to ensure a satisfactory level o f inter-rater reliability. It is also
suggested that D B E instructors should be trained on the standardization o f the
assessment o f the writing skills.
7.
Because E P E includes some parts (e.g., responding to a reading text in a written form)
that are hardly ever covered in the lessons, it can also be recommended that the
writing tasks as tested in the E P E should be included in the program. Also, as the
discoursc-Icvel writing is not tested in the paragraphs writing section o f the EPE, it
can be recommended that there is no need for too much focus on ccrtain discourse
patterns. The feedback sessions about the program o f the D B E last year (Retrieved
from http://dbc.mctu.edu.tr/port/documents) also implied that students unfamiliar with
"discourse" patterns were more successful in the writing section o f the Proficiency
Exam while our students get confused trying to fit their writing to a "discourse" rather
than answering the question.
As suggested by D B E students, there is a need for more writing pop-quizzes and more
time should be given for the writing parts o f the midterm exam. Moreover, portfolio
assessment should be implemented as an alternative way to test writing at the D B E
and the necessary training on how to effectively exploit portfolios should be provided
to instructors at the D B E .