Tanya Zhelezcheva
Assessment Institute Fall 2013
Assessment Report
English Department
EN-102 English Composition II: Introduction to Literature
General Education Objective
Use analytical reasoning to identify issues
or problems and evaluate evidence in order
to make informed decisions.
Course Learning Outcome
Summarize, analyze, and synthesize
diverse readings including
multidisciplinary academic articles, essays,
literary works, or other relevant genres.
Differentiate and make informed decisions
about issues based on multiple value
systems
Assessment Activity
Tools
Kim Banks and Tanya Zhelezcheva used the attached rubric to evaluate whether and how well
students have achieved the EN-102 course learning outcome # 3.
Assignments
Six faculty members contributed several randomly selected artifacts of student writing in
response to a graded, substantive assignment that asks students to select and integrate relevant
evidence into their writing.
Overview of Assignments
1. Students will write a 3-4 page paper arguing for how a work of their choice should be
incorporated into the course based on the theme “Death, Disease, and Disorder.” They
will incorporate evidence from the new text and at least two texts discussed in class.
2. Students will write a 3-4 page paper synthesizing two short stories. They will also tie the
paper’s theme to their own experience in a SWIG project.
3. Students write a 3-4 page paper by identifying a difficult aspect of a short story they've
selected, assessing at least two hypotheses and incorporating at least one classmate's
opinion either in support or as a way to challenge any one of their hypotheses.
4. Students will write a 1600-2000 word essay comparing two plays and incorporating
evidence from at least three library research sources.
5. Students will write a 3-4 page essay interpreting one text (story or poem which we read in
class) with reference to two secondary sources written about the chosen text.
1
Data Collection
A four-point scale criterion was used to assess each writing sample. The rubric is designed to
measure student achievement in terms of manipulating evidence. This exercise was designed to
assess the reliability of the rubric for measuring what it purports to measure.
Analysis and Summary of the Assessment Results
We modified the rubric as follows:
• We redefined three of the five criteria.
• We removed the term “primary source” to open up possibilities for the multimodal
projects many students in the department complete.
• We realized that the rubric does not provide sufficient attention to the importance of
students providing a clear and relevant context for their analysis. So we adjusted the first
row to reflect this.
• We replaced “quotations” with “examples” to account for the many ways evidence can
be presented.
• Many bullet points were moved from one criterion to another or deleted to limit
duplication.
Initially, the first criterion was focused on students’ ability to summarize primary sources.
However, we realized that this criterion was more relevant to the fifth criterion. We also realized
that the way we want student to analyze primary and secondary sources is essentially the same.
This revised category, “Appropriately contextualized the project,” emerged when we divided the
bullet points listed in the original third criterion.
In the original rubric, we assumed students would be working with literary sources. However, as
we were scoring student papers, we realized that in some classes students discuss different
genres, sometimes movies or visual works. In these contexts, assessing use of quotations is
inadequate. We changed the second criterion to reflect the range of primary materials with
which faculty require students to work. With the understanding that providing quotations is still
appropriate in some contexts, we revised the emphasis on quotations to examples.
The third criterion underwent the least amount of transformation; we felt that some of the
explanation in the graded boxes was more connected to the first criteria, and we moved these
features into the relevant graded boxes.
As we scored student samples, we realized that there was an important difference between the
ability to select appropriate quotations/examples and the ability to analyze them. So we re-named
the category to more accurately reflect our expectations. We felt that the fourth criterion should
not emphasize the ability to summarize secondary sources; nor should it presume an emphasis on
working first hand with secondary sources. Instead, we thought that emphasizing the information
that students learn orally, during class discussions within their discourse communities should be
valued and that it should be reflected in the assessed criterion. Therefore, our understanding of
secondary sources is very broad. In addition, we realized that what we value in student writing is
students’ courage to tackle quotations that both support and challenge as well as provide a
different perspective on their thinking. We revised the A-level box to reflect this value.
2
Finally, with the fifth criterion, we realized that it was too broad. So half of the bullet points
were moved to the fourth criterion or eliminated. This allowed us to focus on the quality of
students’ discussions of secondary sources as well as the extent to which they were able to
formulate an opinion in a clear dialogue with those sources.
Assessment Results and Effects on Student Learning Outcomes
Since our task was to determine the applicability of the rubric and to revise it, we did so as we
were scoring all artifacts. Typically, we scored 5 to 6 artifacts at a time; this was followed by a
discussion of our results and inevitably by a revision of the outcomes rubric. Therefore, the total
results for all scored artifacts will not be meaningful. Instead, the data below shows the
breakdown of the scores for the first and last set of artifacts which were scored with the first and
last version of the rubric.
Focused solely on the final set of papers, the differences between the average scores of the two
scorers are significant: 3.04 compared to 2.28. This could indicate a problem with the rubric
overall or could indicate that this rubric will be instrumental in helping the department
establishing more consistent criteria.
Given that the 48 essays scored did not represent a statistically significant sample of EN-102
students, the results were used to revise the assessment instrument as described above and as
attached to prepare for a more comprehensive assessment using the revised rubric to be
conducted in the fall of 2014. Overall, we changed some of the key language associated for each
grade. For instance, initially, the rubric emphasized students’ ability to provide “insightful”
discussion for A-level papers; we revised this language to “in-depth” discussion to reflect
students attempt to think deeply about the project. In the original rubric, the emphasis for a Blevel work was on “unevenness” of the final work; in the revised rubric, it is on “the lack of
polish” in the presentation of the ideas. We retained the emphasis on extremes (either too many
or too few quotations) for C-level work and the emphasis on the absence of certain features for
D- or F-level student work. To this end, both the learning outcome and rubric have been revised
to clarify the expected outcome and its criteria.
We tentatively propose that at the end of EN 101 English Composition I, students should be
Satisfactory in all five categories. Consequently, we tentatively propose that, to demonstrate
progress, at the end of EN 102 English Composition II, all students who have taken EN 101 and
EN 102 in a two-semester sequence should be At Standard in all categories of the rubric.
3
Table 1. Scorer 1. First Set. (5 essays)
Average Score
0
0
0
0
0
2
1
3
4
3
3
1
1
0
1
1
0
6
12
7
0.00
%
7.89
%
15.7
9%
9.21
%
16
13
15
13
76
3.20
2.60
3.00
2.60
3.04
1. Appropriately
contextualizes the
project
2. Carefully
analyzes examples
(e.g. quotations or
descriptions)
appropriate to the
genre
3. Adequately
synthesizes
primary and
secondary sources
4. Appropriately
selects examples
from secondary
sources
5. Appropriately
analyzes
secondary sources
Totals
Percentages
Average Score
TOTAL
3.80
TOTAL
A
19
Criteria
B
4
A
Percentages
1
B
Totals
0
C
5. Appropriately
analyzes
secondary sources
0
F/D
3. Adequately
synthesizes
primary and
secondary sources
4. Appropriately
summarizes
secondary sources
C
1. Appropriately
summarizes
primary sources
2. Appropriately
analyzes primary
sources
F/D
Criteria
Table 2. Scorer 1. Final Set. (5 essays)
0
0
4
1
16
3.20
0
0
5
0
15
3.00
2
2
1
0
9
1.80
2
2
1
0
9
1.80
2
3
0
0
8
1.60
6
7
11
1
57
2.28
10.5
3%
12.2
8%
19.3
0%
1.75
%
4
0
0
2
3
18
3.60
0
1
4
0
14
2.80
0
2
1
2
15
3.00
0
3
2
0
12
2.40
0
6
10
9
78
3.12
0.00
%
7.69
%
12.8
2%
11.5
4%
1. Appropriately
contextualizes the
project
2. Carefully
analyzes examples
(e.g. quotations or
descriptions)
appropriate to the
genre
3. Adequately
synthesizes
primary and
secondary sources
4. Appropriately
selects examples
from secondary
sources
5. Appropriately
analyzes
secondary sources
Total
Percentages
Average Scores
3.80
Totals
19
Outstanding
4
At Standard
1
Satisfactory
0
Poor
0
Criteria
Average Scores
Percentages
Total Points
Total
Outstanding
5. Appropriately
analyzes
secondary sources
At Standard
3. Adequately
synthesizes
primary and
secondary sources
4. Appropriately
summarizes
secondary sources
Satisfactory
1. Appropriately
summarizes
primary sources
2. Appropriately
analyzes primary
sources
Table 4. Scorer 2. Last Set. (5 essays)
Poor
Criteria
Table 3. Scorer 2. First Set. (5 essays)
0
0
4
1
16
3.20
0
0
4
1
16
3.20
0
4
0
1
12
2.40
2
2
1
0
9
1.80
2
2
1
0
9
1.80
4
8
10
3
62
2.48
6.45
%
12.9
0%
16.1
3%
4.84
%
In the fall of 2014, ten faculty members will use the revised rubric to evaluate a statistically significant and
representative sample size of roughly 600 student essays in EN 101 collected from the 2013–2014 academic
year.
Appendices
Appendix A: Original Scoring Rubric
Appendix B: Revised Scoring Rubric
Appendix C: Assignment # 1 with Two Sample Papers
Appendix D: Assignment # 2 with Two Sample Papers
Appendix E: Assignment # 3 with Two Sample Papers
Appendix D: Assignment # 2 with Two Sample Papers
Appendix E: Assignment # 3 with Two Sample Papers
Appendix F: Assignment # 1 with Two Sample Papers
Appendix G: Assignment # 2 with Two Sample Papers
5
Appendix A. Original Rubric
Rubric for Learning Outcome: “Summarize, analyze, and synthesize diverse readings including
multidisciplinary academic articles, essays, literary works, or other relevant genres.”
1. Appropriately
summarizes primary
sources
•
•
2. Appropriately
analyzes primary
sources
•
•
3. Adequately
synthesizes primary
and secondary
sources
•
4. Appropriately
summarizes
secondary sources
•
5. Appropriately
analyzes secondary
sources
•
•
F/D
Short summary
Poor grammar
•
No quotations or •
mostly
inadequate ones
No or extremely
•
choppy
discussion of the
chosen quotations
No or minimal
•
connections
among sources
No or minimal
and inadequately
described context
•
Mostly confusing •
or no distinction
among
perspectives
(student’s own
•
and that of the
secondary
source)
The discussion of •
secondary
sources is mostly
confusing and
•
choppy or absent
•
•
C
Summary takes
over the essay
•
Some quotations •
are appropriate
but not most
•
Uneven
discussion of the
chosen quotations
Connections
among sources
sound unsure;
they are typically
expressed in
choppy prose
Minimal context
Minimal clarity
in the distinction
among
perspectives
Minimal clarity
among secondary
authors’ sources
•
Either too many
or too few
quotations
There is a
tendency to
summarize rather
than to discuss
the quotations
The quality of the
quotations is
uneven
The distinction
between the
student’s
perspective and
the sources’ is
not
systematically
clear
•
•
•
•
•
•
B
Summary is
appropriate but
not with a
purpose
•
Most quotations
are appropriate
Most quotations
are discussed but
occasionally the
connection to the
thesis is not clear
Connections are
appropriate but
not insightful
Appropriate
context but it’s
not insightfully
used
Uneven
distinction among
perspectives
Uneven
distinction among
secondary
authors’ sources
•
A good balance
between
quotations and
good but not
insightful
discussion of
quotations
Most quotations
are appropriately
chosen
Exhibits reliance
on secondary
author’s
perspective rather
than on one’s
own
•
•
•
•
•
•
•
•
A
Purposeful
summary
All or many of
the quotations are
well chosen
All or most
quotations are
discussed
insightfully
Insightful
connections
Appropriate use
of context
Mostly clear
distinction among
perspectives
Mostly clear
distinction of
secondary
authors’ sources
A good balance
between
quotations and
insightful
discussion of
quotations
Quotations are
consistently
appropriate
Clearly
distinguishes
between the
student’s own
point of view and
the secondary
sources’ point of
view
6
Appendix B. Revised Rubric
Rubric for Learning Outcome: “Summarize, analyze, and synthesize diverse readings including
multidisciplinary academic articles, essays, literary works, or other relevant genres.”
Criteria
1. Appropriately
contextualizes the
project
Poor
• Contextualization is
absent
Satisfactory
• Contextualization is
brief and not coherent
At Standard
• Contextualization is
appropriate and
polished but is either
incomplete or overly
lengthy
• Most examples are
appropriate
• Most examples are
discussed
• Occasionally the
connection to the
thesis is not clear
• Connections are
appropriate but not
in-depth
Outstanding
• Contextualization is
purposeful and
functional
2. Carefully
analyzes examples
(e.g. quotations or
descriptions)
appropriate to the
genre
• No examples or
mostly inadequate
ones
• No or extremely
choppy discussion of
the chosen examples
• Some examples are
appropriate but not
most
• Uneven discussion
of the chosen
examples
3. Adequately
synthesizes primary
and secondary
sources
4. Appropriately
selects examples
from secondary
sources
• No or minimal
connections among
sources
• Connections among
sources are tentative
and brief
• No quotations
• Drops random
quotations
• Either too many or
too few quotations
• The quality of the
quotations is uneven
• Drops appropriate
quotations
• Shows
understanding of the
secondary source
• Most quotations are
appropriately chosen
• In-depth work with
secondary sources
• Quotations are
consistently
appropriate
• A good balance
between quotations
expressing a range of
points of view
5. Appropriately
analyzes secondary
sources
•-No distinction and
no discussion among
perspectives
• There is a tendency
to summarize rather
than to discuss the
quotations
• The distinction
between the student’s
perspective and the
sources’ is not
systematically clear
• Minimal clarity
among secondary
authors’ sources
(with or without
quotations)
• Discussion of
quotations is in depth
but not insightful
• Exhibits reliance on
secondary authors
who hold a similar
perspective rather
than including a
mixture of supporting
and opposing points
of view
• The distinction
between secondary
sources' perspectives
is not very clear
• Includes a mixture
of supporting and
opposing points of
view
• Clearly
distinguishes between
secondary sources'
perspectives
• All or many of the
examples are well
chosen
• All or most
examples are
discussed insightfully
•In-depth connections
7
Appendix C/page 1
8
Appendix C/page 2
9
Appendix C/page 3
10
Appendix C/page 4
11
Appendix C/page 5
12
Appendix C/page 6
13
Appendix C/page 7
14
Appendix C/page 8
15
Appendix C/page 9
16
Appendix C/page 10
17
Appendix C/page 11
18
Appendix C/page 12
19
Appendix C/page 13
20
Appendix C/page 14
21
Appendix D/page 1
22
Appendix D/page 2
23
Appendix D/page 3
24
Appendix D/page 4
25
Appendix D/page 5
26
Appendix D/page 6
27
Appendix D/page 7
28
Appendix D/page 8
29
Appendix D/page 9
30
Appendix D/page 10
31
Appendix D/page 11
32
Appendix D/page 12
33
Appendix D/page 13
34
Appendix D/page 14
35
Appendix D/page 15
36
Appendix E/page 1
37
Appendix E/page 2
38
Appendix E/page 3
39
Appendix E/page 4
40
Appendix E/page 5
41
Appendix E/page 6
42
Appendix E/page 7
43
Appendix E/page 8
44
Appendix E/page 9
45
Appendix E/page 10
46
Appendix F/page 1
47
Appendix F/page 2
48
Appendix F/page 3
49
Appendix F/page 4
50
Appendix F/page 5
51
Appendix F/page 6
52
Appendix F/page 7
53
Appendix F/page 8
54
Appendix F/page 9
55
Appendix F/page 10
56
Appendix G/page 1
57
Appendix G/page 2
58
Appendix G/page 3
59
Appendix G/page 4
60
Appendix G/page 5
61
Appendix G/page 6
62
Appendix G/page 7
63
Appendix G/page 8
64
Appendix G/page 9
65
Appendix G/page 10
66
Appendix G/page 11
67