Assessing
Student Learning Outcomes
in
Writing Courses
Lisa Marchand
Cosumnes River College
Sacramento, California
ACCJC Accreditation Standards
California Community Colleges are
required to establish “well-documented,
ongoing processes for evaluating”
effectiveness at all levels of the
institution.
The foundation of this evidence must be
data collected and analyzed in a
consistent way, over time.
By the Year 2012
Engage in an ongoing dialog about student
learning
 Have a well documented process for
evaluating Student Learning Outcomes
 Show tangible evidence that your
program is continuously moving toward
improved student learning.

Think about your own classes or
program…
How do you assess students writing skills?
 Are Student Learning Outcomes in place?
 Are they measurable?
 How are they measured?

Take five minutes to talk it over!
Cosumnes River College
Questions:
 How well can students demonstrate that
they have met the Student Learning
Outcomes for writing courses that they
have passed?
 How well can students who have placed
into a level through the assessment
process demonstrate the skills specified in
the prerequisite course’s SLO’s?
Existing Diagnostics

Not specifically aligned to course SLO’s
(Either too general or too specific)

Mining data from these diagnostics would
be too difficult
Concerns
We wanted to assess SLO skill sets but we
didn’t want to have these assessments affect
the students’ grades.
 We didn’t want to have the SLO
assessments be used to evaluate the
teachers in any way.
 Assessing SLO’s at the end of a course
wouldn’t provide the best information on
students who entered through the
placement process.

Our Goal:
Sustainable Continuous Quality Improvement
Measure
Student
Learning
Outcomes
Ramping Up
Define
metrics to
measure
SLO’s
Develop
student
learning
outcomes
Experiment
with various
measures
and discuss
observations
Creating Departmental Diagnostic
Writing Assessments

Measure expressive writing skills
A holistically scored writing sample

Measure meta-linguistic abilities
An objective multiple choice assessment
Analyze SLO’s to Identify Metrics
Measurable elements of writing include:
◦
◦
◦
◦
◦
◦
◦
◦
Syntactic structures and variety
Accuracy of verb tense form and use
Appropriate word forms
Range of vocabulary
Rhetorical patterns
Development
Use of cohesive devices
Other
…include measures of meta skills
Meta-linguistic skills are included in SLO’s
 Examples:

◦
◦
◦
◦
◦

Identify X
Label X
Analyze X
Recognize X
Differentiate between X and Y
An objective instrument can provide
discrete measures of each of these
Build a Scoring Guide

CRC chose an ETS style holistic scoring
guide model with 5 levels:
◦
◦
◦
◦
◦
Insufficiently prepared = 1
Under-prepared = 2
Fairly prepared = 3
Well prepared for the level = 4
Demonstrates skill above level = 5
…write descriptors for each level
Start with the SLO’s – write a descriptive
narrative to match expressive skills they
describe
 This will be the score 4 – “well prepared
for the course”
 To write descriptors for scores 1 – 3 and
for score 5, examine the sample papers
already marked by faculty

Select A Writing Prompt

For the lowest levels, we chose a picture
and guiding questions:
“Look at the picture. Who is the man? Where is he?
What happened? How does he feel? Why does he
feel that way? Where is he going? What will happen
next? Write the story!”

For intermediate SLO’s and beyond, we
selected this prompt:
“Some people like to live in big cities, others do not.
Tell where you prefer to live and explain why.”
Create an Objective Instrument
Align assessment items directly to SLO
 Make more than one item to measure
each skill…vary the contexts
 Use multiple choice format & SCANTRON©
 Check to see that the terms used are
consistent with what students have
encountered in classes and texts
 Collaborate to check for validity, bias, and
clarity

Scrutinize the Process
Validity Check
According to Test Better,Teach Better by W.
James Popham, content-related evidence
of validity can be determined by relying
on “human judgement”
 A team of expert colleagues agrees that
each test item measures the skill it was
designed to reflect.
 The team must also agree that the test
items are accessible and free of bias.

Check for Clarity
Conduct item-by-item study of the test
 Be sure that the language level matches
the texts
 Be sure that vocabulary is appropriate
and standard
 Be sure the content doesn’t unfairly affect
a student’s ability to respond—it is not
biased.

Check for Bias

Content Bias
◦ Gender
◦ Culture
◦ Experiential

Linguistic Ambiguity
◦ Vague wording
◦ Confusing grammar
◦ Low-frequency vocabulary

Difficulty
Create a Score Tally in Excel
student
id
Verb
Identifyin Identifyin Verb Tenses in
g
g Parts of Tense Question
SLO #2
Sentence Speech Forms
s SLO
SLO #1
Logic
Parts
SLO #6 SLO #7
#2
SLO's 1 - Evidence
and
SLO #3 SLO #5 Items 8- Items 15- Items 214 Holistic
of
Develop SLO #4 Mechanic Items 1-7
14
20
24
Score Planning ment
Syntax
s
(7Q's)
(7 Q's) (6 Q's) (4 Q's)
475549
4
2
3
2
2
5
4
1
2
485646
2
2
2
1
2
4
4
4
3
922766
3
2
2
2
2
4
4
6
4
987991
3
2
2
2
1
5
6
5
3
1020104
3
2
3
3
2
2
3
3
2
1036656
2
2
1
1
1
4
3
4
2
…include

Columns for
◦
◦
◦
◦
Student ID #’s
Holistic Writing Score
Scores for component skills evaluation
Scores from each section of the objective
assessment
Analyze Data
Collect descriptive statistics
 Make correlations
 Focus on evidence of student learning
 Provide data sets to all teachers for use in
self-studies and program review
 Use the assessments over several
semesters to enable longitudinal study of
a significant sample of student work

Continue the Dialog

Use collective observations to suggest
and implement changes to
◦ Curriculum
◦ Methods
◦ Materials
Select appropriate data to support
resource requests like facilities and FTE
 Have faculty experienced in the process
mentor and norm their colleagues

Other than ACCJC Standards,
Why Assess SLO’s?
Find out what you’re “teaching” that the
students aren’t learning.
 Move faculty dialogue away from
anecdotal discussion
 Confirm intuition (or not)
 Gather substantive support for resource
requests

Example SLO’s
SLO 1:
Identify and apply basic essay components
SLO 2:
Employ the writing process
SLO 3:
Interpret and respond to writing prompts
SLO 4:
Develop paragraphs appropriate to rhetorical mode
SLO 5:
Identify and correct grammatical errors through
editing
SLO 6:
Write simple, compound, and complex sentences with
correct punctuation
SLO 7:
Identify and correctly use grammar structures and
word forms
Example of an SLO with Metrics
SLO 7: IDENTIFY AND USE GRAMMAR STRUCTURES
AND WORD FORMS

identify and use all basic verb tenses with very rare errors in form (simple
and progressive aspect of present, past and future)

identify and use some perfect verb tenses with occasional error in form and
usage (present perfect, present perfect progressive, past perfect)

identify and use singular, plural and non-count forms and appropriate
quantifiers and articles without errors in form

identify parts of speech and use correct word forms in writing
W 50 SLO Attainment Fall ‘09
average percent success
80
70
60
50
40
30
20
10
0
SLO 1
SLO 2
SLO 3
SLO 4
SLO 5
SLO 6
SLO 7
W 50 SLO#7 IDENTIFY AND USE GRAMMAR
STRUCTURES AND WORD FORMS
10
9
8
7
6
Total Possible
5
Average
4
Best Score
3
Lowest Score
2
1
0
Holistic Score
Grammar
Correcting Errors
Convert to a Comparable Form

Do some simple arithmetic:
◦ Divide the total possible into 100
 100÷5 = 20
◦ Multiply the answer by the average score
 20 x 2.18 = 43.6%
W 50 SLO#7 IDENTIFY AND USE GRAMMAR
STRUCTURES AND WORD FORMS
Student Success
80
70
60
50
40
Student Success
30
20
10
0
Holistic Score
Grammar in
Writing
Correcting Errors
on objective test
Observations from the Data
Neither the students’ ability to correct
grammar errors on the objective
assessment nor the teachers’ evaluation
of students’ grammar in the writing
sample were significantly correlated with
their success on the holistically graded
writing sample.
 There was no significant correlation
between students writing sample and the
objective assessment.

Significant Correlations
The students holistic writing scores were
directly correlated with the score the
rater gave them on their essay
organization.
 The students holistic writing scores were
directly correlated with the score the
rater gave them on their essay
development.

Thank you for your participation!