Department of Academic Literacy
Spring 2015
BE 205 Course Assessment Report
BE205 Course Assessment Overview
In the Department of Academic Literacy, BE205, Advanced Composition for ESL Students, is
designed for students who speak English as a Second Language and have had some experience in
English composition, but who still require remedial work before taking content area courses. It is
also the final course of the sequence for ESL students with serious writing deficiencies. The
objective is to provide students with reading and writing strategies that prepare them to
understand and analyze texts and to write well-developed, organized, and coherent analytic
essays. At the end of the course, students who have successfully completed this course will take
the CUNY Assessment Test in Writing known as the CATW.
ESL students who are enrolled in Advanced Composition for ESL Students may come from four
sources. First, some exit from BE203 Intermediate Composition for ESL Students after passing
the departmental examination at the end of the previous semester. Second, some are placed
directly in this advanced composition course based on the performance on the College
Assessment Tests. Third, a few could come from the College’s Language Immersion Program
when their proficiency reaches the entry level of advanced composition. Finally, some students
who fail the course once will retake this advanced composition course.
In Fall 2013, Dr. Jennifer Maloy in the Academic Literacy Department conducted an assessment
of the BE205 Course. She was instrumental in devising the original assessment plan. The present
assessment project will first replicate Jennifer’s study from Fall 2013, assessing BE205 in the
Fall of 2014 by comparing the final exam scores in Fall 2013 with those in Fall 2014. In doing
the replication study, this project will assess the areas of strength and weakness in BE 205
students’ writing. We will use the same methods but with different subjects and a different
researcher. If different results are obtained, we will speculate why these differences occur and
what type of actions that could be taken to help our students advance. In addition to conducting
the replication study, we will also



Compare scores on the Fall 2014 CATW with those on the Fall 2014 final,
Compare scores on the Fall 2013 CATW with those on the Fall 2013 final, and
Compare scores on the CATW in Fall 2013 with those in Fall 2014
This assessment project will use the CATW Analytic Scoring Rubric to identify the average
score of 205 students in each of the five individual domains that cover critical response,
development of ideas, organization, sentence structure/vocabulary, and grammar/mechanics.
Each of the five scoring domains corresponds to one or more of the student learning outcomes
for this course, as demonstrated in Table B.
1
Student Learning Outcomes
Table A includes a list of the learning outcomes for BE 205 students. Each learning outcome
corresponds to one or more of the following General Education Objectives.
General Education Objectives:
1. Communicate effectively through reading, writing, listening and speaking
2. Use analytical reasoning to identify issues or problems and evaluate evidence in order to
make informed decisions
3. Reason quantitatively and mathematically as required in their fields of interest and in
everyday life
4. Use information management and technology skills effectively for academic research and
lifelong learning
5. Integrate knowledge and skills in their program of study
6. Differentiate and make informed decisions about issues based on multiple value systems
7. Work collaboratively in diverse groups directed at accomplishing learning objectives
8. Use historical or social sciences perspectives to examine formation of ideas, human
behavior, social institutions, or social processes
9. Employ concepts and methods of the natural and physical sciences to make informed
judgments
10. Apply aesthetic and intellectual criteria in the evaluation or creation of works in the
humanities or the arts
Table A
General
Education
Objective
BE 205 Student Learning Outcome
1,2
1. Students will write analytic essays of multiple paragraphs (500 words) that
introduce, develop, and conclude the discussion of an essay's topic with a
coherent focus.
1, 2
2. Students will write body paragraphs that develop one idea and support the
central focus of the essay.
1, 2
3. Students will write essays that demonstrate the logical development of an
essay by using transitional words between and within paragraphs.
1, 2, 6
4. Students will summarize and analyze a variety of texts, identifying and
engaging in important ideas from the text and relating these ideas to other
readings or personal experiences.
1
5. Students will follow conventions of Standard Written English (SWE),
specifically using coordination and subordination to achieve sentence variety
as well as an appropriate and consistent level of diction in their essays.
2
1
6. Students will write essays, in and out of class, with minimal global errors,
showing a command of sentence boundaries and will be able to write an essay
that contains very few local errors related to fragments and/or run-ons,
subject-verb agreement, verb tense, pronoun agreement and reference, and
basic punctuation and capitalization.
1
7. Students will be able to proofread effectively for surface errors such
misspellings, as well as missing or misused apostrophes, articles, possessive
nouns, prepositions, and content words.
Student Assignment for Assessment
The assignment chosen for assessment was a timed in-class essay that served as BE 205
students’ final exam. During the last week of November 2014, all students enrolled in BE 205
took a standardized final exam that resembles the CATW in instructions, requirements, and
scoring (see Appendix II).
The final exam is completed approximately two weeks before classes end so that students have a
clear assessment of their performance, but still have time to rectify their writing issues by
seeking tutoring or working with their instructors. Generally, a student’s score on the final exam
serves as an indicator of the score that the student will receive on the CATW exam, which all BE
205 students in good standing take at the end of the semester. The writing directions,
accompanying reading passage, scoring rubric and process, and the physical conditions of the
exam closely resemble the CATW.
In the final exam, students have 90 minutes to write an essay that responds to a reading passage
they are given. Students may use only a dictionary or thesaurus and must write in pen in a blue
book. The writing instructions for the exam are as follows:
Read the article provided and compose an essay that summarizes the short text, identifies
a significant idea in the text, and relates it to your own reading, observations, or
personal experience. Your essay should consist of an introduction, a body, and a
conclusion.
Students are then presented with a description of what to include in their introduction, body, and
conclusion, and the CATW Analytical Scoring Rubric and a scoring sheet is provided to students
along with the reading they are asked to use.
The article to which the students were asked to respond was titled, “Is There Really Such a Thing
As a ‘Morning Person’? adapted from a World Science Festival on-line newsletter by Clare
Smith Marash (see Appendix II).
Grading Policy for 2013
Upon collecting exams from students, all instructors teaching BE 205 scored their students’
3
writing using the CATW Analytic Scoring Rubric and then arranged for another BE 205
instructor to score the exams. The scoring of the exams by two readers resembles the scoring
protocol used for the CATW. All BE 205 instructors participated in norming sessions prior to
scoring the exams.
Grading Policy for 2014
The AL Department instituted a new grading policy and provided additional norming sessions so
that the AL instructors’ grading was more accurately aligned with the official CATW readers’
grading. In Fall 2014, the midterm and final exams were exchanged among the writing
instructors who blindly read and graded the exam from two other classes. If an instructor
disagreed with the final score, he/she could provide a departmental committee with a student
portfolio and/or offer another writing exam to guarantee that each student was assessed
accurately. The purpose of implementing the new grading policy was to reduce the subjective
effect of an instructor’s grading his/her own students’ tests.
Evidence for Assessment
When scoring the final exam, instructors use the CATW Analytic Scoring Rubric, which
assesses student essays in the following five areas:
1. Critical Response to the Writing Task and the Test: This category focuses on
whether students understand the main ideas in the text and understand the nature of the
writing task, which is to discuss these ideas and to critically analyze and integrate them
with their own ideas and experiences.
2. Development of the Writer’s Ideas: In this category students are assessed on whether
they are able to develop their ideas through summary, narrative and/or problem/solution.
Students should support statements with details and examples from what students have
experienced, read, or learned about. Students also must refer to specific ideas from the
reading to support their ideas.
3. Structure of the Response: This category focuses on students’ ability to express ideas
that connect to a central focus or thesis and to use an organizational structure and
transitions that help to support the thesis.
4. Language Use: Sentences and Word Choice: This category focuses on clarity and
sentence control.
5. Language Use: Grammar, Usage, and Mechanics: This category focuses on
students’ ability to follow conventions of Standard American English.
The domains of the scoring rubric correspond to the Student Learning Outcomes as indicated in
Table B.
4
Table B
CATW Analytic
Scoring Rubric
Domain
Critical Response to the
Writing Task and the
Text
205 Student Learning Outcome
Development of the
Writer’s Ideas
1. Students will write analytic essays of multiple paragraphs (500 words) that
introduce, develop, and conclude the discussion of an essay's topic with a
unified, logical, and coherent focus.
Structure of the
Response
2. Students will write body paragraphs that develop one idea and support the
central focus of the essay.
4. Students will summarize and analyze a variety of texts, identifying and
engaging in important ideas from the text and relating these ideas to other
readings or personal experiences.
3. Students will write essays that demonstrate the logical development of an
essay by using transitional words between and within paragraphs.
Language Use:
Sentences and Word
Choice
5. Students will follow conventions of Standard Written English (SWE),
specifically using coordination and subordination to achieve sentence variety
as well as an appropriate and consistent level of diction in their essays.
Language Use:
Grammar, Usage, and
Mechanics
6. Students will write essays, in and out of class, with minimal global errors,
showing a command of sentence boundaries and will be able to write an
essay that contains very few local errors related to fragments and/or run-ons,
subject-verb agreement, verb tense, pronoun agreement and reference, and
basic punctuation and capitalization.
7. Students will be able to proofread effectively for surface errors such
misspellings, as well as missing or misused apostrophes, articles, possessive
nouns, prepositions, and content words.
In this assessment, each of the five individual domains listed above has been scored on a scale of
1-6 by two 205 instructors. The scores from each instructor are calculated according to CATW
guidelines. Domains 1, 2, 3 (Content) are double-weighted. Domains 4, 5 (Language Use) are
added to Content domains, and these are single-weighted. And then the scores from both
individual instructors are combined for a totaled score.
The researcher also would average the two individual scores the student essay receives in each
domain. The average of the two scores will be used to calculate the mean scores of all essays
within the samples across the five domains.
Please refer to Appendix for a copy of the CATW Analytic Scoring Rubric.
5
Example of a Final Exam Scoring Sheet
The CATW Analytic Scoring Rubric (Appendix I) includes detailed descriptions of each score in
each domain; however, Table C presents a general description of each score, ranging from 1 to 6.
Table C: Description of Scores within the CATW Analytic Scoring Rubric
Score
1
2
3
4
5
6
Description
The student demonstrates a minimal ability to accomplish the task within the essay.
The student demonstrates a weak ability to accomplish the task within the essay.
The student demonstrates a general or uneven ability to accomplish the task within the
essay.
The student demonstrates a competent ability to accomplish the task within the essay.
The student demonstrates an effective or skillful ability to accomplish the task within
the essay.
The student demonstrates a thoughtful or insightful approach to accomplishing the
task within the essay.
6
The total score may range from 16 (in which the student writer received a score of 1 from each
reader in each of the five domains) to 96 (in which the student writer received a score of 6 from
each reader in each of the five domains). In terms of the CATW, students must receive a score of
56 in order to pass the exam and exit writing remediation. The borderline score of 56 indicates
that a student has received a majority of individual scores of 4, described as “competent” in the
rubric; however, the student also scored a 3 in more than one area of the exam, meaning some
aspects of the essay were deemed “uneven” or “general.”
Assignment to Assess
The student assignment that was assessed across the course was the final exam in BE 205, which
was a timed in-class essay given department-wide in the last week of November in 2014. Below
is a description of the writing instructions that students were given, along with a short passage
(approximately 300 words), which was selected by the department’s Director of Writing in midNovember.
Writing Directions: Read the passage above and write an essay responding to the ideas
it presents. In your essay, be sure to summarize the passage in your own words, stating
the author’s most important ideas. Develop your essay by identifying one idea in the
passage that you feel is especially significant, and explain its significance. Support your
claims with evidence or examples drawn from your own experiences or what you have
read or learned about outside of class.
Remember to review your essay and make any changes or corrections that are needed to
help your reader follow your thinking. You will have 90 minutes to complete your essay.
Analysis of Assessment Results
The researcher recorded the scores received by each student, examining the scores from
individual readers as well as the average scores in each domain to determine which areas BE205
students scored the highest, indicating achievement of particular learning outcomes by the end of
the semester, and areas in which the students scored the lowest, indicating a need for
improvement in particular learning outcomes.
The current project will take the following steps to assess the BE205 course:
1. Section A: Replicating Jennifer’s study from Fall 2013, assessing BE205 in the Fall of
2014,
2. Section B: A comparison of scores on Fall 2014 CATW with those on Fall 2014 in-class
final,
3. Section C: A comparison of scores between Fall 2013 CATW and Fall 2013 in-class
final, and
4. Section D: A comparison of the average Scores on the CATW between Fall 2013 and
Fall 2014.
5. Section E: Resulting Action Plan
7
Section A
A Comparison of Final Exam Scores between Fall 2013 and Fall 2014
Table 1: Comparing Final Exam Scores in Fall 2013 with Those in Fall 2014
Fall 2013 vs. Fall 2014
Score
2013
In-Class
Final
2014
In-Class
Final
Average
Total n
55.8
123
76
52.26
100
30
Passing
percentage
Failure
percentage
47
62%
38%
70
30%
70%
Passed n Failed n
Figure 1: Comparing Final Exam Scores in Fall 2013 with Those in Fall 2014
Table 1 and Figure 1 represented the comparison between the 2013 and 2014 in-class final
scores. The data were as follows:

123 students took the in-class final in Fall 2013, whereas 100 students (2 students missed
the final) took the final in Fall 2014.
8


The average total final score in 2013 is 55.8, which was close to a 56, a passing score on
the CATW, while the average total in 2014 was 52.26, 3.74 below a CATW passing
score.
76 students, or 62% of the students passed the 2013 final, whereas 30, or 30% of the
students passed the 2014 final.
The above data demonstrated that 76 students, or 62% of the students in Fall 2013 received a
passing score on the final in agreement with the CATW rubric. This 62 % pass rate gave an
indication of a similar pass rate on the CATW exam. However, Department data reveals that
only 26.63% of the BE205 students passed the CATW. The results illustrate that the AL readers’
scoring of the finals was not accurately aligned with the official CATW readers’ grading. Seen in
this light, there might be some misrepresentation of the grading accuracy. As already noted on
Grading Policy in 2013 on page 4, this final exam was graded by the BE205 students’ teacher
and another writing instructor.
In contrast, under a new grading policy (see p.4 Grading Policy in 2014), the pass rate of the
Fall 2014 final was 30%, 32% lower than 62% on the Fall 2013 final. Only 30 students out of
100 passed the in-class final. In other words, a majority of the BE205 students, or 70 % of the
students received a score lower than 56. The outcomes indicate that the passing percentage on
the Fall 2014 final considerably declined, possibly as a result of the implementation of the new
grading policy, even though other possible variables could also have an impact on student
performance, such as different groups of students and different writing topics.
However, according to the current study, 41% of the BE205 students passed the actual Fall 2014
CATW exam, which was 11% higher than the in-class final pass rate of 30%.
The instructors’ grading on the final indicates that they graded the final a bit too harshly this
time, but to a certain extent, they increased the representation of the grading accuracy. In contrast
to a pass rate of 26.63% in Fall 2013, a pass rate of 41% of the BE205 students on the CATW in
Fall 2014 demonstrated their ability to write multiple-paragraph essays that introduce, develop,
and conclude the discussion of an assay’s topic with a unified, logical, and coherent focus. This
increased pass rate could be largely explained by the Department’s multifaceted, dynamic
support, including implementing a new grading policy, running a series of norming sessions
arranged by the Best Practice Committee to help writing instructors improve grading accuracy,
offering Friday CATW writing workshops to multiple BE205 course repeaters, and requiring
writing instructors to include High Impact Practices in BE205 courses that allowed ESL students
to develop summarizing, paraphrasing, developing ideas, etc.
9
Table 2: A Comparison of Domain Scores
A Comparison of Each Domain Score: Fall 2013 vs. Fall 2014
2013 In-Class
Average
Score
2014 In-Class
Average
Score
Differences
CR: Critical Response to Writing Task and
Text
3.65
3.25
-0.40
DI: Development of Writer’s Ideas
3.50
3.36
-0.14
SR: Structure of the Response
3.71
3.34
-0.37
3.20
3.04
-0.16
3.05
3.19
0.14
In-Class
LUSW: Language Use: Sentences and
Word Choice
LUGM: Language Use: Grammar, Usage,
Mechanics
Figure 2: A Comparison of Final Exam Scores in Each Domain between 2013 and 2014
4
3.5
3.65
3.71
3.50
3.25
3.36
3.34
3.20
3.04
3.05
3.19
3
2.5
2
2013 In-Class Final
1.5
2014 In-Class Final
1
0.5
0
CR Average DI Average SR Average
Score
Score
Score
LUWS
Average
Score
LUGM
Average
Score
Table 2 and Figure 2 represent two different readers’ average score of each of the five individual
domains for both 2013 and 2014 in-class finals. The average score comparison was as follows:


The average score in CR for 2013 in-class final was 3.65, while the 2014 average CR
domain score was 3.25, which was 0.40 lower.
The average score in DI was 3.50 for Fall 2013, and 3.36 for Fall 2014, which was 0.14
lower.
10



The average score in SR was 3.71 for Fall 2013, and 3.34 for Fall 2014, which was 0.37
lower.
The average score in LUSW was 3.20 for Fall 2013, and 3.04 for Fall 2014, which was
0.16 lower.
The average score in LUGM domain was 3.05 for Fall 2013, and 3.19 for Fall2014,
which was 0.14 higher.
Table 2 and Figure 2 represent the domain in Fall 2013 in which students scored highest in
Structure of the Response and lowest in Language Use: Grammar, Usage, and Mechanics,
whereas in Fall 2014 students scored highest in Development of Ideas and lowest in Language
Use: Sentences and Word Choice. Overall, students in both 2013 and 2014 performed better in
the first 3 double-weighted Content domains than in the last 2 single-weighted Language Use
domains.
Within the 3 Content domains, in Fall 2013 students received the highest score in the Structure
of the Response domain (3.71), which may demonstrate that the students could organize their
ideas around a central focus, and the lowest in the Development of Writer’s Ideas domain (3.50).
In contrast, the students in Fall 2014 received the highest score in the DI domain (3.36), and the
lowest in the CR domain (3.25).
These findings reveal that in 2013, among the Content categories the DI domain was identified
as the weakest area for the BE205 students. In 2014, on the contrary, the DI domain received the
highest average scores. At least, the students’ performance in the DI domain did not seem to
become the most pressing issue, even though 3.36 was not really an accurate score to
demonstrate the writers’ ability to competently develop ideas. Data in Table 2 and Figure 2 show
that the weakest area identified in 2014 was the CR domain (3.25 in contrast to 3.65 in 2013).
As shown above, to help students improve college writing and achieve learning outcomes,
it may be necessary to examine why the students’ average scores in this CR domain on the Fall
2014 final were the lowest of the 3 Content domains.
First, the BE205 students faced many obstacles while trying to express an opinion about text and
demonstrating their understanding of the key ideas in the reading. Many ESL students may not
have sufficient opportunity to practice expressing their opinions about a particular issue. Also,
most CATW reading texts are culture bound, so English language learners may not be aware of
the information that the author left unsaid. As indicated in the CUNY CATW Analytic Scoring
Rubric, to receive a passing score in the CR domain, students have to critically discuss ideas in
the text and consistently demonstrate an understanding of the main ideas and some of the
complexity in the text. Hence, it would be a more helpful effort for the instructors in the
Department of Academic Literacy to spend an appropriate amount of time teaching ESL students
how to express their opinions by activating ESL learners’ prior knowledge or building and
enhancing their background information.
11
In conclusion, the present project first assessed the BE205 course in the Fall of 2014 by
replicating the previous study from Fall 2013. In replicating the previous study, the investigator
obtained different results among the three Content domains. These results indicate that




In 2013, the weakest area identified for the BE205 students was the Development of
Ideas domain.
In 2013, the CR domain received the highest score.
In contrast, in 2014 the DI domain received the highest scores, and
In 2014, the weakest area identified was the Critical Response domain.
There are a variety of factors that may have contributed to these differences, such as groups of
students, the topic of the reading, students’ test anxiety, students’ attitudes toward learning, and
instructors’ grading accuracy. Therefore, in terms of the three Content categories for this
replication study, it is difficult to say which specific Content domain is the strongest, and which
is the weakest and needs more time for improvement. In fact, each Content domain continues to
deserve much attention in order to help ESL students to write a strong reading-response essay.
Section B
A Comparison of Scores on Fall 2014 CATW with Those on Fall 2014 In-Class
Final
Table 4: Comparing Scores on the Fall 2014 CATW with Those on the Fall 2014 Final,
Fall 2014 Actual CATW Scores versus Fall 2014
In-Class Final Scores
Score
Average
Total n
Actual
CATW
52.26
102
42
In-Class
Final
52.26
100
30
Passing
percentage
Failure
percentage
60
41%
59%
70
30%
70%
Passed n Failed n
12
Figure 3: A Comparison of Scores between 2014 CATW and 2014 In-Class Final
Table 3 and Figure 3 represent an average total score and passing rate for the Fall 2014 CATW
and in-class final. In Table 3 and Figure 3, we see



102 students took the actual CATW exam, whereas 100 students (2 students missed the
final) took the final,
The average total final score for 2014 CATW was 52.26, while the average total for the
in-class final was 52.26, and
42 students, or 41% of the students passed the 2014 CATW, while 30, or 30% of the
students passed the in-class final.
Overall, the average total scores for the CATW and in-class final in Fall 2014 were coincidently
identical. The true reason behind this identicalness remains unknown. Future research needs to
explore how to decrease misrepresentation of grading accuracy. One way to improve consistency
is to encourage all instructors to become certified CATW readers. As shown in Table 3 and
Figure 3, the pass rate of the CATW in Fall 2014 increased by 11% compared to the pass rate of
30% in the in-class final.
Table 4: A Comparison of Average Scores in Each Domain between Fall 2014 CATW and Fall
2014 In-Class Final
Fall 2014 Actual CATW Average Scores versus
In-Class Average Scores in Each Domain
Actual
In-Class Final
CATW
Average
CATW Domain
Differences
Average
Score
13
Score
CR: Critical Response to writing task and
text
3.39
3.25
-0.14
DI: Development of writer’s Ideas
3.34
3.36
0.02
SR: Structure of the Response
3.29
3.34
0.05
LUSW: Language Use: Sentences and
Word Choice
3.10
3.04
-0.06
LUGM: Language Use: Grammar, Usage,
Mechanics
3.00
3.19
0.19
Figure 4: A Comparison of Average Scores in Each Domain between Fall 2014 CATW and Fall
2014 In-Class Final
Table 4 and Figure 4 represent the average score of each domain for both the actual CATW test
and in-class final in Fall 2014.



The average score of the CR domain for the actual CATW test was 3.39, while the inclass final average score was 3.25, which was 0.14 lower.
The average score of the DI domain for the actual CATW test was 3.34, while the inclass final was 3.36,
The average score of the SR domain for the actual CATW test was 3.29, while the inclass final was 3.34,
14


The average score of the LUSW domain for the actual CATW test was 3.10, while the
in-class final was 3.04, and
The average score of the LUGM domain for the actual CATW test was 3.00, while the inclass final was 3.19, which was 0.19 higher.
Basically, the average score in each individual domain for the in-class final was adjacent to that
of the CATW in 2014. The outcomes exemplified that since the AL instructors’ scoring of the inclass final was better aligned with the official CATW readers’ grading, the scores on the in-class
final more adequately reflected what the BE205 students had actually learned, even though the
grading process was still not completely consistent, particularly in the domains of Critical
Response and Language Use.
Section C
A Comparison of Scores between Fall 2013 CATW and Fall 2013 In-Class Final
Table 5: Comparing Scores on the Fall 2013 CATW with Those on the Fall 2013 Final
Fall 2013 Actual CATW Scores versus
Fall 2013 In-Class Final Scores
Score
Average
Total n
Actual
CATW
51.57
199
53
In-Class
Final
55.8
123
76
Passing
percentage
Failure
percentage
146
26.63%
73.37%
47
62%
38%
Passed n Failed n
Figure 5: Comparing Scores between Fall 2013 CATW and Fall 2013 In-Class Final
15
Table 5 and Figure 5 represent an average total score and passing rate for the Fall 2013 CATW
and Fall 2013 in-class final. The data in Table 5 and Figure 5 were presented as follows:



199 students took the actual CATW exam in Fall 2013, whereas among the students who
took the in-class final exam, a sample size of 123 students was selected for the purpose of
the course assessment in Fall 2013,
The average total final score for 2013 CATW was 51.57, while the average total for the
in-class final was 55.8, which was 4.23 points higher than the actual CATW passing
score of 56, and
53 students, or 26.63% of the students passed the 2013 CATW, while 76, or 62% of the
students passed the in-class final.
In general, in the in-class final, the pass rate was 62% while in the CATW, 26.63% of the
students passed. These results illustrated that the AL readers’ grading was not rigorous enough.
Table 6: Comparing the Fall 2013 CATW Scores with Those on the Fall 2013 Final in Each
Domain
Fall 2013 Actual CATW Average Scores versus
Fall 2013 In-Class Average Scores in Each Domain
CATW Domain
Actual
CATW
Average
Score
In-Class Final
Average
Differences
Score
CR: Critical Response to Writing Task and
Text
3.36
3.65
0.29
DI: Development of Writer’s Ideas
3.28
3.50
0.22
SR: Structure of the Response
3.25
3.71
0.36
3.05
3.20
0.15
2.96
3.05
0.09
LUSW: Language Use: Sentences and
Word Choice
LUGM: Language Use: Grammar, Usage,
Mechanics
16
Figure 6: A Comparison of the Average Scores in Each Domain between Fall 2013 CATW and
Fall 2013 In-Class Final
Table 6 and Figure 6 represent the average score of each domain for both actual CATW test and
in-class final in Fall 2013. The data in Table 6 and Figure 6 were presented as follows:





The average score of the CR domain for the actual CATW test in Fall 2013 was 3.36,
while the in-class final average score was 3.65, which was 0.29 higher.
The average score of the DI domain for the actual CATW test was 3.28, while the inclass final was 3.50, which was 0.22 higher,
The average score of the SR domain for the actual CATW test was 3.25, while the inclass final was 3.71, which was 0.36 higher,
The average score of the LUSW domain for the actual CATW test was 3.05, while the
in-class final was 3.20, which was 0.15 higher, and
The average score of the LUGM domain for the actual CATW test was 2.96, while the inclass final was 3.05, which was 0.09 higher.
As demonstrated in Table 6 and Figure 6, the scores on the final in Fall 2013 were not aligned
with the actual CATW scores because the grading of the in-class final was not a reliable measure
of the student learning outcomes.
17
Section D
A Comparison of the Average Scores on the CATW between Fall 2013 and Fall
2014
Table 7: Comparing Scores on the CATW in Fall 2013 with Those in Fall 2014
Scores on the CATW: Fall 2013 vs. Fall 2014
Score
Average
Total n
Fall 2013
CATW
51.57
199
53
Fall 2014
CATW
52.26
102
42
Passing
percentage
Failure
percentage
146
26.63%
73.37%
60
41%
59%
Passed n Failed n
Figure 7: Comparing Scores on the CATW in Fall 2013 with Those in Fall 2014
Table 7 and Figure 7 represented a comparison of scores on the CATW between Fall 2013 and
Fall 2014. The data were as follows:


199 students took the CATW in Fall 2013, whereas 102 students took the CATW in Fall
2014.
The average total CATW score in Fall 2013 was 51.57, while the average total in Fall
2014 was 52.26.
18

53 students, or 26.63% of the students passed the 2013 CATW, while 42, or 41% of the
students passed the 2014 CATW.
These results demonstrated that in Fall 2014, 42 students out of 102, or 41% of the BE205
students passed the CATW, which was 11% higher than the pass rate of 30% in the Fall 2013
CATW, indicating that 41% of the BE205 students successfully achieved the learning objectives
of the course.
Table 8: A Comparison of Scores on the CATW in Each Domain between Fall 2013 and Fall
2014
Comparing the Average Scores Given by Two Official CATW Readers in Each
CATW Domain: Fall 2013 vs. Fall 2014
CATW Domain
2013 CATW 2014 CATW
Average
Average
Differences
Domain Score Domain Score
CR: Critical Response to writing task and
text
3.36
3.39
0.03
DI: Development of writer’s Ideas
3.28
3.34
0.04
SR: Structure of the Response
3.25
3.29
0.04
LUSW: Language Use: Sentences and
Word Choice
3.05
3.10
0.05
LUGM: Language Use: Grammar, Usage,
Mechanics
2.96
3.00
0.04
19
Figure 8: A Comparison of the Scores on the CATW in Each Domain between Fall 2013 and
Fall 2014
Table 8 and Figure 8 represent two official CATW readers’ average scores of each of the five
individual CATW domains for both Fall 2013 and Fall 2014. Basically, the average score of each
domain for both 2013 and 2014 was not high enough compared to a score of 3.5 [2(3+4) + 2(3+4)
+ 2(3+4) + (3+4) + (3+4) = 56], a minimum average passing score for each domain. This clearly
indicates the big challenges that ESL students face in each domain when they do college-level
writing in English. The data in Table 8 and Figure 8 were presented as follows:





The average score in CR for 2013 CATW was 3.36, while the 2014 average CR domain
score was 3.39, which was 0.03 higher.
The average score in DI was 3.28 for Fall 2013, and 3.34 for Fall 2014, which was 0.06
higher,
The average score in SR was 3.25 for Fall 2013, and 3.29 for Fall 2014, which was 0.04
higher.
The average score in LUSW was 3.05 for Fall 2013, and 3.10 for Fall 2014, which was
0.05 higher.
The average score in LUGM domain was 2.96 for Fall 2013, and 3.00 for Fall2014,
which was 0.04 higher.
The data shown above reveal that the Structure of the Response domain among the Content
categories consecutively received the lowest scores, 3.25 in Fall 2013 and 3.29 in Fall 2014,
suggesting that the SR domain was identified as the weakest area for the BE205 students.
To have a better understanding of the BE205 students’ academic performance, a closer look at a
score distribution in the SR domain will be particularly helpful to the AL Department.
Figure 9: Fall 2013 CATW: A Score Distribution in the Structure of the Response Domain
20
Fall 2013 CATW Score Distribution in SR
Domain
300
255/398
250
200
119/398
150
100
2013 CATW SW
Score distribution
23/398
5.78%
1/398
0.25%
50
0
score of 2
score of 3
score of 4
score of 5
398 means a total
number of scores out
of 199 individual
essays
In Fall 2013, 199 students took the actual CATW exam. According to Figure 9, 199 students
were graded by two certified CATW readers. Each of the 199 students’ essays was graded by
two different readers, so a total of 398 scores were identified. The spread of scores was as
follows:
 23 were a score of 2, or 5.78% of students’ structures were
 255 were a score of 3, or 64.07%
 119 were a score of 4, or 29.90%, and
 1 was a score of 5, or 0.25%
The Structure of the Response domain measures students’ ability to organize ideas into a
coherent essay that backs up a central focus, or thesis. These figures reveal that only 29.90% of
students’ essay organization competently supported the central focus and showed evidence of
logical progression of ideas in this domain. On the other hand, 255 scores out of 398 were 3.
This means 64.07% of students’ structures were uneven, and ideas were sometimes disconnected
because they could be characterized by digression, thus failing to answer the question or
substantiate an argument or position in writing through the use of “evidence.” These results
emphasize a serious issue among ESL students when they compose off-topic statements that are
not closely related to the significant idea they tried to develop. The ESL students may have made
some good points, but they have not substantiated them. As a result, their response may be too
wordy, confusing, or complicated.
Figure 10: Fall 2014 CATW: A Score Distribution in the Structure of the Response Domain
21
Fall 2014 CATW Score Distribution
in SR
113/204
55.39%
120
100
76/204
37.25%
80
60
40
20
2/204
0.98%
13/204
6.37%
0
Score of 1 Socre of 2 Score of 3 Score of 4
Fall 2014 CATW Score
Distribution in SR
204 means a total
number of scores out
of 102 individual
essays
Figure 10 indicates that102 students were graded by two readers. In addition, because each of the
102 students’ essays was graded by two different readers, a total of 204 scores were identified.
The spread of scores was as follows:




2 were a score of 1,
13 were a score of 2,
113 were a score of 3, and
76 were a score of 4.
As discussed earlier, on the Fall 2013 CATW, only 29.90% of the organizational structure of the
response supported progression of the writer’ ideas. Likewise, the above figures again
demonstrate that in Fall 2014, no more than 37% of students’ structure was evident and had a
logical progression of ideas that competently supported the writers’ central focus. Clearly, more
than half of the organizational structures, or 55.39 % of students’ structures were uneven, and
relationships among ideas may not be consistently linked, thereby failing to achieve the purpose
of the writing.
Research shows that ESL students, unlike their American counterparts, are less direct in text and
paragraph organization. From the perspective of contrastive rhetoric, this digression issue could
be culture specific. It may be very challenging for ESL students because many of them would
wander off the subject without being aware that their writing is straying off the topic because
indirectness is common or acceptable in their native languages. A simple trick is to just train
students to have an outline of what it is they intend to write about to make sure their response
answers the question.
Finally, the students scored lowest in the two Language Use categories. These findings in 2014
are congruent with the results of 2013 in terms of language use. Given the nature of second
22
language acquisition, these college-level ESL learners’ writing presents unique features of syntax
and choice of words when compared to English. These results may be attributed to BE205
students’ linguistic backgrounds. Some of the specific challenges they faced may include







Difficulty expressing concepts and ideas in English: Because of different linguistic
backgrounds, BE205 students feel that their words often fail to convey the meaning,
An abundance of idioms and figurative language in a reading text,
Density of unfamiliar vocabulary,
Word order and sentence structure: English writers sometimes use unusual word order or
invent rules that could be extremely difficult to BE205 students. For example, they will
will write The book that you gave me I’d read already. The significant differences
between English and some other languages, such as Korean and Chinese, particularly in
sentence structures, make it extremely hard for most Korean and Chinese ESL students to
acquire English at the same rate as, for example, their Spanish-speaking peers whose
sentence structure is similar to English. Therefore, it is difficult for many BE205 students
to build correctly ordered sentences in English,
English strongly stresses cohesion of form, while in some other languages, surface links
are optional because meaning is understood from context. Thus, when BE205 students
are instructed to use transitions to create the logical relationship among ideas, they would
often use them inappropriately.
Difficult text structure. The reason is that CATW practice readings are OFTEN lack topic
sentences,
Unfamiliarity with the connotative and denotative meanings of words
It can be very frustrating for students to readily and clearly articulate complex thoughts in an
English-dominant context. Therefore, BE205 students need explicit instruction in how to use the
language to convey the meaning expected by an English reader.
Conclusion
The outcomes of this BE205 course assessment have pointed out the areas of strength and
weakness in BE205 students’ academic writing. Overall, the outcomes indicate that BE205
students’ ability to think and write in English has been reinforced through the collaborative joint
efforts of the whole Department, which was reflected in the increased pass rate of 41% of the
BE205 students on the Fall 2014 CATW in contrast to that of 26.63% on the Fall 2013 CATW.
However, the urgent need for improvement may lie in BE 205 students’ ability to develop their
related ideas to support their essay’s central focus through careful analysis of texts as well as
their own experience, even though continuing efforts should be made to increase competence in
the three Content domains.
Another area for improvement for BE 205 students is language use. BE205 students have
different needs than native speakers of English and need frequent, guided practice in using
language to develop as academic writers. Finally, to decrease the misrepresentation of grading
accuracy, the Department needs to continue to encourage more BE 205 instructors to train to
become certified CATW readers.
23
Section E: Resulting Action Plan
In this section, action plan includes –but are not limited to – the following:
Here are some suggestions for addressing the areas of improvement. The goal is to improve
student learning outcomes in that specific area in which BE205 students have demonstrated
weaknesses.
1. Avoiding Digressions in the Critical Response Domain:
This domain is double weighted and measures students’ ability to organize related ideas into a
coherent essay that clearly and consistently supports the writer’s central focus. When writers
focus on irrelevant details, the score on this domain decreases. Findings show that instead of
maintaining focus by directly responding to the question throughout the response, the BE205
students may sometimes focus on irrelevant details or loosely related details (64.06% of students
on the Fall 2013 CATW and 55.39% of students on the Fall 2014 CATW had this issue).
“Digression” prevents students from answering the question, so teachers should dedicate a little
more time training students to organize the relationships among ideas, so that students learn how
to group relevant ideas together to show how each idea is interconnected.
2. Designing a Series of Best Practice Lessons on the “Digression” Issue
Since the Structure of the Response domain presents an issue for BE205 students, it is
recommended that a series of Faculty Development lessons on the “digression” issue be designed
and presented within the Department to improve students’ performance as well as instructors’
awareness of the issue.
3. Integrating High Impact Practices into the BE205 Course
To highly motivate students and significantly improve student learning outcomes and retention of
critical content knowledge, it is suggested that AL instructors incorporate High Impact Practices
into the BE205 curriculum. This should be directly related to the CATW writing in the areas of
paraphrasing, summarizing, and development of ideas, sentence structure, sentence variety,
grammar, and logical connections between sentences within and beyond paragraphs.
4. Discussing Writing Student Samples in Class
Sentence structure and grammar are important, but showing student writing samples in class—
not just isolated grammar mistakes, is even more important. As demonstrated in this assessment,
many of the BE205 students’ writing problems are weak vocabulary, a lack of sentence variety,
cohesion, and logic. These are not writing problems that most ESL students will be able to
identify easily. For this reason, as suggested above, the method of discussing writing samples
may assist in the process.
5. Continuing Cross-Grading of Upper Level Writing Classes
24
Based on the findings in Fall 2014, Departmental cross-grading of upper-level writing classes
worked well. To raise grading accuracy, this grading policy should continue, and the Department
should continue to offer norming sessions to maintain consistence in grading.
6. Raising the BE205 Placement Score to 52
Findings indicate that 42 students out of 102, or 41% of students passed the Fall 2014 CATW
because their skills were too weak when they entered BE205. Although the pass rate in Fall 2014
has increased considerably compared with the pass rate in the Fall 2013, this result suggests that
the CATW pass rate is still not satisfactory among BE205 learners. For this reason, the cutoff for
a BE205 placement score should be raised to a score of 52 rather than the current 50. This would
ensure that students who enter BE205 have higher level skills so that they will be able to
improve enough to pass the CATW at the end of the term. Thus, it is recommended that we raise
the BE205 placement score from 50 to 52.
7. Reducing Class Size for BE205
Smaller classes for BE205 are particularly effective at raising student learning outcomes as well
as the CATW pass rate. With a smaller BE205 class size of 15-20 students, teachers can provide
individualized attention and hone in on students’ writing issues.
Copy of Assessment Materials
Appendix I: CATW Analytic Scoring Rubric
Appendix II: Standardized BE 205 Final Exam
25
26
27
28
29