1
Weier Ye
Assessment Institute
Spring 2015
BE 205 Course Assessment Report
BE205 Course Assessment Overview
In the Department of Academic Literacy, BE205, Advanced Composition for ESL Students, is
designed for students who speak English as a Second Language and have had some experience in
English composition, but who still require remedial work before taking content area courses. It is
also the final course of the sequence for ESL students with serious writing deficiencies. The
objective is to provide students with reading and writing strategies that prepare them to
understand and analyze texts and to write well-developed, organized, and coherent analytic
essays. At the end of the course, students who have successfully completed this course will be
scheduled to take the CUNY Assessment Test in Writing known as the CATW.
ESL students who are enrolled in Advanced Composition for ESL Students may come from four
sources. First, some exit from BE203 Intermediate Composition for ESL Students after passing
the departmental examination at the end of the previous semester. Second, some are placed
directly in this advanced composition course based on the performance on the College
Assessment Tests. Third, a few could come from the College’s Language Immersion Program
when their proficiency reaches the entry level of advanced composition. Finally, some students
who fail the course once will retake this advanced composition course.
In Fall 2013, Dr. Jennifer Maloy conducted an assessment of the BE205 Course. The present
assessment project replicated Jennifer’s study from Fall 2013, assessing BE205 in the Fall of
2014 by using the same methods but with different subjects and a different researcher. By doing
the replication study, this project assessed the areas of strength and weakness in BE 205
students’ writing. This project focused on the final essay that BE205 students wrote in class
before they took the CATW exam at the end of the Fall 2014 semester. For the purpose of the
assessment, a total of 103 students’ scores on the final were collected with the help of a secretary
of the Department, representing all the six sections of BE205 offered in Fall 2014. It used the
CATW Analytic Scoring Rubric (see Appendix I) to identify the average score of 205 students in
each of the five individual domains that cover critical response, development of ideas,
organization, sentence structure/vocabulary, and grammar/mechanics. Each of the five scoring
domains corresponds to one or more of the student learning outcomes for this course, as
demonstrated in Table B.
Student Learning Outcomes
Table A includes a list of the learning outcomes for BE 205 students. Each learning outcome
corresponds to one or more of the following General Education Objectives.
General Education Objectives:
1. Communicate effectively through reading, writing, listening and speaking
2. Use analytical reasoning to identify issues or problems and evaluate evidence in order to
make informed decisions
2
3. Reason quantitatively and mathematically as required in their fields of interest and in
everyday life
4. Use information management and technology skills effectively for academic research and
lifelong learning
5. Integrate knowledge and skills in their program of study
6. Differentiate and make informed decisions about issues based on multiple value systems
7. Work collaboratively in diverse groups directed at accomplishing learning objectives
8. Use historical or social sciences perspectives to examine formation of ideas, human
behavior, social institutions, or social processes
9. Employ concepts and methods of the natural and physical sciences to make informed
judgments
10. Apply aesthetic and intellectual criteria in the evaluation or creation of works in the
humanities or the arts
Table A
General
Education
Objective
BE 205 Student Learning Outcome
1,2
1. Students will write analytic essays of multiple paragraphs (500 words) that
introduce, develop, and conclude the discussion of an essay's topic with a
coherent focus.
1, 2
2. Students will write body paragraphs that develop one idea and support the
central focus of the essay.
1, 2
3. Students will write essays that demonstrate the logical development of an
essay by using transitional words between and within paragraphs.
1, 2, 6
4. Students will summarize and analyze a variety of texts, identifying and
engaging in important ideas from the text and relating these ideas to other
readings or personal experiences.
1
5. Students will follow conventions of Standard Written English (SWE),
specifically using coordination and subordination to achieve sentence variety
as well as an appropriate and consistent level of diction in their essays.
1
6. Students will write essays, in and out of class, with minimal global errors,
showing a command of sentence boundaries and will be able to write an essay
that contains very few local errors related to fragments and/or run-ons,
subject-verb agreement, verb tense, pronoun agreement and reference, and
basic punctuation and capitalization.
1
7. Students will be able to proofread effectively for surface errors such
misspellings, as well as missing or misused apostrophes, articles, possessive
nouns, prepositions, and content words.
3
Student Assignment for Assessment
The assignment chosen for assessment was a timed in-class essay that served as BE 205
students’ final exam. During the last week of November 2014, all students enrolled in BE 205
took a standardized final exam that resembles the CATW in instructions, requirements, and
scoring (see Appendix II).
The final exam is distributed by individual instructors to their students approximately two weeks
before classes end. Generally, a student’s score on the final exam serves as an indicator of the
score that the student will receive on the CATW exam, which all BE 205 students in good
standing take at the end of the semester. The writing directions, accompanying reading passage,
scoring rubric and process, and the physical conditions of the exam closely resemble the CATW.
In the final exam, students have 90 minutes to write an essay that responds to a reading passage
they are given. Students may use only a dictionary or thesaurus and must write in pen in a blue
book. The writing instructions for the exam are as follows:
Read the article provided and compose an essay that summarizes the short text, identifies
a significant idea in the text, and relates it to your own reading, observations, or
personal experience. Your essay should consist of an introduction, a body, and a
conclusion.
Students are then presented with a description of what to include in their introduction, body, and
conclusion, and the CATW Analytical Scoring Rubric and a scoring sheet is provided to students
along with the reading they are asked to use.
The article to which the students were asked to respond was titled, “Is There Really Such a Thing
As a ‘Morning Person’? adapted from a World Science Festival on-line newsletter by Clare
Smith Marash (see Appendix II).
Grading Policy for 2013
Upon collecting exams from students, all instructors teaching BE 205 scored their own students’
writing using the CATW Analytic Scoring Rubric and then arranged for another BE 205
instructor to score the exams. The scoring of the exams by two readers resembled the scoring
protocol used for the CATW. All BE 205 instructors participated in norming sessions prior to
scoring the exams.
Grading Policy for 2014
The AL Department instituted a new grading policy and provided additional norming sessions so
that the AL instructors’ grading was more accurately aligned with the official CATW readers’
grades. In Fall 2014, the midterm and final exams were exchanged among the writing instructors
who blindly read and graded the exam from two other classes. If an instructor disagreed with the
final score, he/she could provide a departmental committee with a student portfolio and/or offer
another writing exam to guarantee that each student was assessed accurately. The purpose of
4
implementing the new grading policy was to reduce the subjective effect of an instructor’s grading
his/her own students’ tests.
Evidence for Assessment
When scoring the final exam, instructors use the CATW Analytic Scoring Rubric, which
assesses student essays in the following five areas:
1. Critical Response to the Writing Task and the Test: This category focuses on
whether students understand the main ideas in the text and understand the nature of the
writing task, which is to discuss these ideas and to critically analyze and integrate them
with their own ideas and experiences.
2. Development of the Writer’s Ideas: In this category students are assessed on whether
they are able to develop their ideas through summary, narrative and/or problem/solution.
Students should support statements with details and examples from what students have
experienced, read, or learned about. Students also must refer to specific ideas from the
reading to support their ideas.
3. Structure of the Response: This category focuses on students’ ability to express ideas
that connect to a central focus or thesis and to use an organizational structure and
transitions that help to support the thesis.
4. Language Use: Sentences and Word Choice: This category focuses on clarity and
sentence control.
5. Language Use: Grammar, Usage, and Mechanics: This category focuses on
students’ ability to follow conventions of Standard American English.
5
The domains of the scoring rubric correspond to the Student Learning Outcomes as indicated in
Table B.
Table B
CATW Analytic
Scoring Rubric
Domain
Critical Response to the
Writing Task and the
Text
205 Student Learning Outcome
4. Students will summarize and analyze a variety of texts, identifying and
engaging in important ideas from the text and relating these ideas to other
readings or personal experiences.
Development of the
Writer’s Ideas
1. Students will write analytic essays of multiple paragraphs (500 words) that
introduce, develop, and conclude the discussion of an essay's topic with a
unified, logical, and coherent focus.
Structure of the
Response
2. Students will write body paragraphs that develop one idea and support the
central focus of the essay.
3. Students will write essays that demonstrate the logical development of an
essay by using transitional words between and within paragraphs.
Language Use:
Sentences and Word
Choice
5. Students will follow conventions of Standard Written English (SWE),
specifically using coordination and subordination to achieve sentence variety
as well as an appropriate and consistent level of diction in their essays.
Language Use:
Grammar, Usage, and
Mechanics
6. Students will write essays, in and out of class, with minimal global errors,
showing a command of sentence boundaries and will be able to write an
essay that contains very few local errors related to fragments and/or run-ons,
subject-verb agreement, verb tense, pronoun agreement and reference, and
basic punctuation and capitalization.
7. Students will be able to proofread effectively for surface errors such
misspellings, as well as missing or misused apostrophes, articles, possessive
nouns, prepositions, and content words.
In this assessment, each of the five individual domains listed above has been scored on a scale of
1-6 by two 205 instructors. The scores from each instructor were calculated according to CATW
guidelines. Domains 1, 2, 3 (Content) were double-weighted. Domains 4, 5 (Language Use) were
added to Content domains, and these were single-weighted. And then the scores from both
individual instructors were combined for a totaled score.
Please refer to Appendix for a copy of the CATW Analytic Scoring Rubric.
Example of a Final Exam Scoring Sheet
6
The CATW Analytic Scoring Rubric (Appendix I) includes detailed descriptions of each score in
each domain; however, Table C presents a general description of each score, ranging from 1 to 6.
Table C: Description of Scores within the CATW Analytic Scoring Rubric
Score
1
2
3
4
5
6
Description
The student demonstrates a minimal ability to accomplish the task within the essay.
The student demonstrates a weak ability to accomplish the task within the essay.
The student demonstrates a general or uneven ability to accomplish the task within the
essay.
The student demonstrates a competent ability to accomplish the task within the essay.
The student demonstrates an effective or skillful ability to accomplish the task within
the essay.
The student demonstrates a thoughtful or insightful approach to accomplishing the
task within the essay.
The total score may range from 16 (in which the student writer received a score of 1 from each
reader in each of the five domains) to 96 (in which the student writer received a score of 6 from
7
each reader in each of the five domains). In terms of the CATW, students must receive a score of
56 in order to pass the exam and exit writing remediation. The borderline score of 56 indicates
that a student has received a majority of individual scores of 4, described as “competent” in the
rubric; however, the student also scored a 3 in more than one area of the exam, meaning some
aspects of the essay were deemed “uneven” or “general.”
Assignment to Assess
The student assignment that was assessed across the course was the final exam in BE 205, which
was a timed in-class essay given department-wide in the last week of November in 2014. Below
is a description of the writing instructions that students were given, along with a short passage
(approximately 300 words), which was selected by the department’s Director of Writing in midNovember.
Writing Directions: Read the passage above and write an essay responding to the ideas it
presents. In your essay, be sure to summarize the passage in your own words, stating the
author’s most important ideas. Develop your essay by identifying one idea in the passage that
you feel is especially significant, and explain its significance. Support your claims with evidence
or examples drawn from your own experiences or what you have read or learned about outside
of class.
Remember to review your essay and make any changes or corrections that are needed to help
your reader follow your thinking. You will have 90 minutes to complete your essay.
Analysis of Assessment Results
The researcher recorded the scores received by each student, examining the scores from
individual readers as well as the average scores in each domain to determine which areas BE205
students scored the highest, indicating achievement of particular learning outcomes by the end of
the semester, and areas in which the students scored the lowest, indicating a need for
improvement in particular learning outcomes.
8
A Comparison of Final Exam Scores between Fall 2013 and Fall 2014
Table 1: Comparing Final Exam Scores in Fall 2013 with those in Fall 2014
Fall 2013 vs. Fall 2014
Score
2013
In-Class
Final
2014
In-Class
Final
Average
Total n
55.8
123
76
52.26
100
30
Passing
percentage
Failure
percentage
47
62%
38%
71
29.70%
70.30%
Passed n Failed n
Figure 1: Comparing Final Exam Scores in Fall 2013 with Those in Fall 2014
70
62%
60
55.8
52.26
50
40
29.7%
30
2013 In-Class Final
20
2014 In-Class Final
10
0
Average Score
Comparison
Passing Percentage
Table 1 and Figure 1 represented the comparison of scores between Fall 2013 in-class final and
Fall 2014 in-class final. The data were as follows:


123 students took the in-class final in Fall 2013, whereas 100 students (2 students missed
the final) took the final in Fall 2014.
The average total final score in 2013 was 55.8, which was close to a 56, a passing score
on the CATW, while the average total in 2014 was 52.26, 3.74 below a CATW passing
score.
9

76 students, or 62% of the students passed the 2013 final, whereas 30, or 29.70% of the
students passed the 2014 final.
The above data demonstrated that 76 students, or 62% of the students in Fall 2013 received a
passing score on the final in agreement with the CATW rubric. This 62 % pass rate gave an
indication of a similar pass rate on the CATW exam. However, Department data reveals that
only 26.63% of the BE205 students passed the CATW. The results illustrate that the AL readers’
scoring of the finals was not accurately aligned with the official CATW readers’ grading. Seen in
this light, there might be some misrepresentation of the grading accuracy. As already noted on
Grading Policy in 2013 on page 3, this final exam was graded by the BE205 students’ teacher
and another writing instructor.
In contrast, under a new grading policy (see Grading Policy in 2014), the pass rate in Fall 2014
was 29.70%, 32.3% lower than 62% in 2013. Only 30 students out of 100 passed the in-class
final. In other words, a majority of the BE205 students, or 70.30% of the students received a
score lower than 56. The outcomes indicate that the passing percentage on the Fall 2014 final
considerably declined, possibly as a result of the implementation of the new grading policy, even
though other possible variables could also have an impact on student performance, such as
different groups of students and different writing topics.
However, according to the Department data, 41.75% of the BE205 students passed the actual
Fall 2014 CATW exam, which was 12.05% higher than the in-class final pass rate of 29.70%.
These findings demonstrate that the instructors may have graded the final a bit too harshly this
time, but to a certain extent, they increased the representation of the grading accuracy.
Apparently, with a pass rate of 41.75% on the CATW in Fall 2014, the BE205 students gave a
better performance, in contrast to a pass rate of 26.63% in Fall 2013. This increased pass rate
could be largely explained by the Department’s multifaceted, dynamic support, including
implementing a new grading policy, running a series of norming sessions arranged by the Best
Practice Committee to help writing instructors improve grading accuracy, offering Friday CATW
writing workshops to multiple BE205 course repeaters, and requiring writing instructors to
include High Impact Practices in BE205 courses that allowed ESL students to develop
summarizing, paraphrasing, developing ideas, etc.
Table 2: A Comparison of Domain Scores
Comparison of Each Domain Score: Fall 2013 vs. Fall 2014
2013 In-Class
Average Score
2014 In-Class
Average Score
CR: Critical Response to Writing Task and
Text
3.65
3.25
-0.40
DI: Development of Writer’s Ideas
3.50
3.36
-0.14
SR: Structure of the Response
3.71
3.34
-0.37
3.20
3.04
-0.16
3.05
3.19
0.14
In-Class
LUSW: Language Use: Sentences and Word
Choice
LUGM: Language Use: Grammar, Usage,
Mechanics
Differences
10
Figure 2: A Comparison of Final Exam Scores in Each Domain between Fall 2013 and Fall 2014
4
3.5
3.65
3.71
3.50
3.25
3.36
3.34
3.20
3.04
3.05 3.19
3
2.5
2
2013 In-
1.5
2014 In-
1
0.5
Table02 and Figure 2 represent two different readers’ average score of each of the five individual
domains forCR
both
2013 and 2014
in-class finals.SR
The
average score comparison
follows:
Average
DI Average
Average
LUWS was asLUGM





Score
Score
Score
Average
Average
The average score in CR for 2013 in-class final was 3.65, while the 2014 average CR
Score
Score
domain score was 3.25, which is 0.40 lower.
The average score in DI was 3.50 for Fall 2013, and 3.36 for Fall 2014, which was 0.14
lower.
The average score in SR was 3.71 for Fall 2013, and 3.34 for Fall 2014, which was 0.37
lower.
The average score in LUSW was 3.20 for Fall 2013, and 3.04 for Fall 2014, which was
0.16 lower.
The average score in LUGM domain was 3.05 for Fall 2013, and 3.19 for Fall2014,
which was 0.14 higher.
Table 2 and Figure 2 represent the domain in Fall 2013 in which students scored highest in
Structure of the Response and lowest in Language Use: Grammar, Usage, and Mechanics,
whereas in Fall 2014 students scored highest in Development of Ideas and lowest in Language
Use: Sentences and Word Choice.
Overall, students in both 2013 and 2014 performed better in the first 3 double-weighted Content
domains than in the last 2 single-weighted Language Use domains.
Within the 3 Content domains, in Fall 2013 students received the highest score in the Structure
of the Response domain (3.71), which may demonstrate that the students could organize their
ideas around a central focus, and the lowest in the Development of Writer’s Ideas domain (3.50),
11
which could define the students’ inability to competently develop ideas. In stark contrast, the
students in Fall 2014 received the highest score in the DI domain (3.36), and the lowest in the
CR domain (3.25).
These findings reveal that in 2013, among the Content categories the DI domain was identified
as the weakest area for the BE205 students. In 2014, on the contrary, the DI domain received the
highest average scores. At least, the students’ performance in the DI domain did not seem to
become the most pressing issue, even though 3.36 was not really a good score to demonstrate the
writers’ ability to competently develop ideas. Data in Table 2 and Figure 2 show that the weakest
area identified in 2014 was the CR domain (3.25 in contrast to 3.65 in 2013).
As shown above, to help students improve college writing and achieve learning outcomes,
it is necessary to examine why the students’ average scores in this CR domain on the Fall 2014
final were the lowest of the 3 Content domains.
First, the BE205 students faced many obstacles while trying to express an opinion about text and
demonstrating their understanding of the key ideas in the reading. Many ESL students may not
have sufficient opportunity to practice in expressing their opinions about a particular issue. Also,
most CATW reading texts are culture bound, so English language learners may not be aware of
the information that the author left unsaid. As indicated in the CUNY CATW Analytic Scoring
Rubric, to receive a passing score in the CR domain, students have to critically discuss ideas in
the text and consistently demonstrate an understanding of the main ideas and of some of the
complexity in the text. Hence, it would be a more helpful effort for the instructors in the
Department of Academic Literacy to spend more time teaching ESL students how to express
their opinions and emphasize activating ESL learners’ prior knowledge or build their background
information.
Moreover, among the five domains, the lowest scores were identified in the last two Language
Use domains in both 2013 and 2014. These low average scores may be attributed to BE205
students’ linguistic backgrounds. Some of the specific challenges they faced may include





Difficulty expressing concepts and ideas in English: Because of different linguistic
backgrounds, BE205 students feel that their words often fail to convey the meaning,
An abundance of idioms and figurative language in a reading text,
Density of unfamiliar vocabulary,
Word order and sentence structure: English writers sometimes use unusual word order or
invent rules that could be extremely difficult to BE205 students. For example, The book
that you gave me I’d read already. The significant differences between English and some
other languages, such as Korean and Chinese, particularly in sentence structures, make it
extremely hard for most Korean and Chinese ESL students to acquire English at the same
rate as, for example, their Spanish-speaking peers. Therefore, it is difficult for many
BE205 students to build correctly ordered sentences in English,
English strongly stresses cohesion of form, while in some other languages, surface links
are optional because meaning is understood context. When BE205 students are instructed
to use transitions to create the logical relationship among ideas, they would often use
them inappropriately.
12


Difficult text structure. CATW practice readings are OFTEN presented to BE205
students without topic sentences,
Unfamiliarity with the connotative and denotative meanings of words
Clearly, BE205 students have different needs than native speakers of English and need frequent,
guided practice in using language to develop as academic writers.
Conclusion and Implication
The present project has assessed the BE205 course in the Fall of 2014 by replicating the previous
study from Fall 2013. As pointed out earlier, the final exam the BE205 students took at the end
of the semester resembles the CATW in instructions, requirements, and scoring. Findings
demonstrate that 62% of the students in Fall 2013 received a passing score on the final, but
Department data indicates that only 26.63% of the students passed the CATW. The difference
suggests that the passing percentage of the Fall 2013 final exam was not consistent with the
official CATW readers’ grading. In contrast, under a new grading policy, the pass rate on the
2014 final exam was 29.70%. However, according to Department data, 41.75% of the BE205
students in Fall 2014 passed the CATW. These results show that the instructors’ grading may be
a little too hard. Therefore, to increase grading accuracy, more BE205 instructors are encouraged
to become certified CATW readers.
Since the learning and writing skills taught in the BE205 course are reflected in the CATW, this
increased pass rate on the CATW was a reflection of the student learning outcomes that have
been achieved by the BE205 students. Specific outcome is focused on a specific domain of
student learning (see Table B). 41.75% of the BE205 students demonstrated on the CATW their
ability to write multiple-paragraph essays that introduce, develop, and conclude the discussion of
an assay’s topic with a unified, logical, and coherent focus.
The investigator finds that in both 2013 and 2014 final exams, among the five domains, the
students scored lowest in the two Language Use categories. Put differently, the findings in 2014
are congruent with the results of 2013 in terms of language use. Given the nature of second
language acquisition, these college-level ESL learners’ writing presents unique features of syntax
and choice of words when compared to English. It can be very frustrating for them to readily and
clearly articulate complex thoughts in an English-dominant context. Therefore, BE205 students
need explicit instruction in how to use the language to convey the meaning expected by an
English reader.
In replicating the previous study, the investigator obtained different results among the three
Content domains. These results indicate that




In 2013, the weakest area identified for the BE205 students was the Development of
Ideas domain.
In 2013, the CR domain received the highest score.
In contrast, in 2014 the DI domain received the highest scores, and
In 2014, the weakest area identified was the Critical Response domain.
13
There are a variety of factors that may have contributed to these differences, such as groups of
students, the topic of the reading, students’ test anxiety, students’ attitudes toward learning, and
instructors’ grading accuracy. Therefore, in terms of the three Content categories, it is difficult to
say which specific Content domain is the strongest, and which is the weakest and needs more
time for improvement. In fact, each Content domain continues to deserve attention in order to
help ESL students to write a strong reading-response essay.
Copy of Assessment Materials
Appendix I: CATW Analytic Scoring Rubric
Appendix II: Standardized BE 205 Final Exam
14
15
16
17
18