50% Sabbatical
Fall 2009
Lisa Marchand
Report Submitted April 16, 2010
Sabbatical Report
Fall 2009 Type B 50%
Lisa Marchand
Creating Student Learning Outcomes Assessment Tools
Page 1 of 8

A B S T R A C T
This sabbatical project sought to determine a methodology for measuring how well students in English as a Second Language writing courses can demonstrate the skills named in student learning outcomes (SLO’s) for the prerequisite course. To achieve this end, the release time provided opportunity to

review assessment literature to refresh knowledge of sources of bias

collect existing rubrics and terminology in use by ESL faculty
 analyze course SLO’s to isolate specific measurable skills

discuss the above with colleagues to agree upon valid measures

create holistic scoring rub rics aligned to the subjective prerequisite SLO’s
 create subjective writing prompts appropriate to the program’s five levels
 construct objective assessment materials to measure metalinguistic SLO’s

revise drafts of assessment materials subsequent to input from department members

train all ESL writing faculty in the use of the assessments

prepare and distribute assessment administration packets to all ESL writing faculty

submit conference presentation proposal to California Teachers of English to
Speakers of Other Languages (CATESOL)
The SLO assessments were administered in the second week of the spring 2010 semester. The data collected provides the department with a comprehensive profile of the degree to which all writing students can demonstrate the skill(s) described by each
SLO. This data can now be analyzed to determine correlations among many variables and be repeated every semester to then identify patterns in student performance ability.
This longitudinal data will be used in program review and play an important role in making future improvements in ESL writing instruction.
Sabbatical Report
Fall 2009 Type B 50%
Lisa Marchand
Creating Student Learning Outcomes Assessment Tools
Page 1 of 8

S A B B A T I C A L R E P O R T
ESL Writing Strand – Creating Student Learning Outcomes Assessment Tools
Activities
The primary purpose of this project was to discover the extent to which students are able to demonstrate that they have achieved the student learning outcomes described for the course prerequisite to that in which they are currently enrolled. SLO’s to be measured included those for ESLW 20; ESLW 30; ESLW40; ESLW 50; and ESLW 310.
At the time of this project, the college had not yet added ESLW 340, which has ESLW
320 as its prerequisite. For this reason, this newly prerequisite level was not included when this project was conceived.
Before beginning materials development, it was necessary to review test development principles. The book, Test Better, Teach Better: The Instructional Role of Assessment , by W. James Popham was especially helpful. Specific bias review protocols created by the Sacramento City College ESL department for use in vetting their placement assessment writing sample prompts also guided the final selection of writing prompts for subjective SLO assessment.
Another preliminary step was to collect existing scoring rubrics from ESL faculty. None of these was specifically aligned to the Student Learning Outcomes, which confirmed that the project was both useful and necessary. Also an important preparatory step was to ask faculty members to provide me with lists of specific terms they and their texts use to discuss the principles of writing and grammar.
The first objective for the project was to create scoring guides aligned to the expressive writing SLO’s for each of five levels of the program. This required analysis of the SLO’s to identify metrics that would provide evidence whether or not a student had achieved them. Examples of identified measures include: syntactic structures and variety, frequency and accuracy of word forms, verb tense selection, range of vocabulary, rhetorical pattern, logical development, use of cohesive devices, and other technical aspects of writing.
Sabbatical Report
Fall 2009 Type B 50%
Lisa Marchand
Creating Student Learning Outcomes Assessment Tools
Page 2 of 8

The second objective of the project was to pilot test the scoring rubrics. After drafting the guides, they were distributed to the writing faculty for review and comment. This resulted in numerous suggestions for improvement and further discussion of metalinguistic terminology. Also critical in the development of the scoring guides was taking time to align the language to more accurately reflect the teachers’ perceptions of stude nts’ writing proficiency.
To accomplish this, the teachers collected writing samples from all of their students and marked them according to how well, in their opinion, the individual students had been prepared for the challenges of their current course. I then read and considered these scored papers against the descriptions in the draft scoring guides and made additional adjustments to the descriptors. Now complete, each scoring guide consists of a fivepoint scale for evaluating students’ skill upon entering a writing course having a prerequisite:
Demonstrates insufficient preparation for the level score of 1
Demonstrates under-preparation for the level
Demonstrates fair preparation for the level score of 2 score of 3
Demonstrates good preparation for the level
Demonstrates skills above the level score of 4 score of 5
In addition to the holistic score described above, teachers give a score of 0 – 3 for evidence of how well each student has demonstrated control of expressive SLO component skills. The holistic score becomes one variable for analysis, but it is accompanied, for example, by individual scores for abilities to plan and organize ideas, construct topic sentences, develop cohesive ideas, employ syntactic variety, and appropriately select and form verb tense and aspect structures.
After the instructor has marked the ratings on a scoring sheet, these, along with the results of the objective segment of the assessment, were transcribed into Excel spreadsheets then analyzed statistically. Because the process looks at component skills as variables rather than composite SLO’s, analysis can indicate what abilities are most
Sabbatical Report
Fall 2009 Type B 50%
Lisa Marchand
Creating Student Learning Outcomes Assessment Tools
Page 3 of 8

well controlled by groups of students, and possibly predict were facility with a particular skill will lead to more successful writing overall.
The next objective of this sabbatical project was to create objective assessments to measure students’ metalinguistic skills. Student Learning Objectives in writing courses include claims that, upon completion of the course, students will be able to identify, recognize, or analyze structures and features of writing. For this reason, holistically evaluating writing samples does not provide a complete set of metrics to measure all
SLO’s. An objective instrument had to be developed to measure SLO’s completely.
Just as midterm and final exams are constructed following field-specific knowledge and are based upon text and lecture content, the objective instruments were constructed as though the SLO’s were the actual course outline. It may be useful to mention here that most ESL course content includes instruction leading to SLO attainment and more . This is why existing tests would not have accomplished the same goal as an instrument specifically aligned to the SLO’s themselves. To enhance the accessibility of test items, text material was reviewed to look for item types and range of vocabulary typical at each level. Some items, however, were designed to assess the students’ abilities to deal with decontextualized linguistic structures. These items utilized nonsense words in place of sentence content while grammatical words were left intact.
For each SLO (or, in some cases, sub-SLO), several test items were constructed. This allows an individual student to apply a learned skill more than once, and in most instances, in more than one context. Each item on the objective assessment has a multiple-choice solution so the responses can be marked on a SCANTRON score sheet. After being electronically scored, the number of correct items under each SLO is recorded individually for each student. This data is then transferred into an Excel spreadsheet so that it can be turned over to the college research department for statistical analysis, including correlating subjective and objective variables. SLO assessment data will also be correlated with additional information about the students.
The college research department can provide each student’s gender, language
Sabbatical Report
Fall 2009 Type B 50%
Lisa Marchand
Creating Student Learning Outcomes Assessment Tools
Page 4 of 8

background, previous educational experience, parents’ educational background, and grade in the prerequisite writing course.
It is important to note that the focus is on student performance of learning. Data from these assessment instruments is not to be used to infer teaching success or ineffectiveness. Individual instructors will be provided with data reports from which they can make their own observations and contribute suggestions for instructional improvement through the self-study and program review processes.
The final objective for this project was to prepare the instructors to administer the SLO assessments during the second week of instruction in Spring 2010. This involved writing detailed instructions and scoring grids for each level then assembling packets of testing instructions, scoring guides, an appropriate number of objective assessments and writing prompts, SCANTRON forms, and score tally sheets for the instructor of each section.
Critique
The objectives of this sabbatical project have been met. Materials have been created for assessing Writing SLO’s for all of the original five levels of prerequisite courses. These instruments have been administered, scored, tallied, transcribed to Excel spreadsheets and submitted to the college research department for cross-tabulation and identification of correlated variables. Results will also be correlated with demographics and student grades.
The process of creating the assessment materials has been interesting on several levels. For example, in the first phase of preparation, the department engaged in a lively
Email exchange about terminology. We discovered differences and similarities in the way our books and training have led us to approach teaching grammar, syntax, and the writing process. A beginning has been made toward increasing parallel terminology, without jeopardizing the academic individuality of the faculty members. This is a primary benefit to the students. Where instructional consistency occurs, persistent students will
Sabbatical Report
Fall 2009 Type B 50%
Lisa Marchand
Creating Student Learning Outcomes Assessment Tools
Page 5 of 8

have an opportunity to continue to build upon a single foundation, adding new information to their body of knowledge without having to first reconsider old information in new terms.
Students will also benefit from what the ESL department learns about correlations between overall writing success and the writer’s control of component skills. Anything we observe about the demographics of our students and their writing issues will also improve the teachers’ ability to predict where emphasis will be most effective.
Another benefit of the project is that the materials created will continue to provide the program with comparable data through the foreseeable future. As the ESL program builds a longitudinal collection of information, patterns will become evident, and effects of changes in instruction, text materials, student demographics, and curriculum can be surmised.
The ESL department may choose to also develop SLO assessments for the other program strands. The writing assessment may serve as a model for the development of listening, speaking, and reading SLO measures.
Sharing the Outcomes of the Project
…with students: Once SLO assessments have been analyzed and reported back to the faculty, what we learn can become part of our instructional strategy. If we learn, for example, that ability to employ logical cohesive devices is highly correlated with a student’s holistic writing score, the skill can be emphasized and the students can be made tacitly aware of its importance overall.
…within the ESL department: At our next department meeting, on April 30, each fulltime faculty member will receive a copy of The Writing Strand SLO Assessment
Handbook . The handbook contains copies of all instructions, assessment instruments,
SLO’s, scoring guides, objective SCANTRON scoring key, and data recording grids.
They will also be given access to electronic copies of the assessment instruments and an Excel spreadsheet workbook set up to receive the data they collect. With the
Sabbatical Report
Fall 2009 Type B 50%
Lisa Marchand
Creating Student Learning Outcomes Assessment Tools
Page 6 of 8

exception of answer keys and electronic media, a copy of the handbook is attached to this report.
…with other departments:
The first run of writing SLO assessment was only partially conducted by individual faculty members. Once the process has become regularized, ESL writing teachers can explain the ease and utility of this kind of retrospective, SLO measurement. Another way to share the project outcomes with other college departments will be to report our findings in the CASSL newsletter, in the
Learning Outcomes Dialogue sub-committee, and during FLEX.
…with the community:
As part of the sabbatical project, I submitted a proposal to
California Teachers of Speakers of Other Languages. I will present a detailed methodology for creating writing SLO assessment instruments at the state conference on April 23, 2010.
Attachments
1. Fall 2009 Sabbatical Application
2. CRC Department of English as a Second language Writing Strand SLO
Assessment Handbook (product of sabbatical)
3. Data Collected January 2010 (in Excel)
4. Analysis of Data (Provided by Jeanne Edman--in PASW)
5.
Sabbatical Report
Fall 2009 Type B 50%
Lisa Marchand
Creating Student Learning Outcomes Assessment Tools
Page 7 of 8