English 105 Assessment Results
Presented in the following order:
Spring 2007
Fall 2006
Comparative Results Fall 2006/Spring 2007
Comp 1 Assessment Analysis
Spring 2007
Purpose
The Spring 2007 Comp 1 writing assessment was undertaken as part of DMACC’s institutional
accreditation process (AQIP). The writing assessment itself has been evolving over several
semesters.
Methodology
The Spring 2007 semester Comp 1 (ENG 105) assessment represents an on-going district-wide,
holistically scored, portfolio assessment program. All face-to-face Comp 1 classes on all six
campuses were part of the project, representing a total of 950 students. (There were 1,728
students who took Comp 1 in the fall.).
Each student portfolio submitted for the assessment included a reflective letter, generated by the
student, to be assessed by trained faculty readers.
 A scoring rubric was used to evaluate the letters
 Each letter was read and evaluated by two different readers
Prior to the assessment, the following steps were taken:
1. In January, just prior to the Spring 2007 semester, a memo was sent to all Comp 1
instructors which included:
 A reminder to include the five Comp 1 goal statements on the course syllabus
 A reminder to collect portfolios from Comp 1 students
 FAQs regarding the assessment
 Business letter format requirements for the reflective letters
2. One reader-training session was held in January.
3. Joe DeHart, Executive Director of Planning and Research, determined the sample size of
282 students. After the March drop date, a random list of 500 names was generated.
4. Susan Pagnac, Assessment Coordinator, contacted each Comp 1 instructor requesting
them to submit portfolios from those students selected for the assessment. A total of 345
student portfolios (out of 500) were actually collected. Of those portfolios collected, 336
were useful for the assessment (67.2%).
DMACC Comp I Writing Assessment
Page 1 of 22
Reasons for not submitting or using requested portfolios included:
 Students were registered, but were no longer attending class
 Portfolios submitted did not include the reflective letter
On Saturday, June 16, 2007, twenty-four highly trained readers scored the reflective letters in
two hours. Two different readers scored each letter. The scores were entered on an Excel
spreadsheet and the scoring graphs (shown below) were created from that data. The readers also
wrote a reflective evaluation of the writing assessment. Their comments will help us refine the
next assessment (January 2008).
DMACC Comp I Writing Assessment
Page 2 of 22
Findings
Summary Data Graph 1
COMP 1 Assessment - Fall 06 & Spring 07
Mean Score by Goal
Source: DMACC English Department
5.00
Fall 06 Mean
4.50
Spring 07 Mean
4.00
3.35
3.50
3.19
3.10
Mean Score
3.00
2.83
2.77
2.76
2.53
2.50
2.58
2.41
2.55
2.69
2.39
2.00
1.50
1.00
0.50
0.00
Goal 1- Recursive
Writing
Goal 2- Critical Reading
Goal 3- Standard
Written English
Goal 4- Research
Techniques
Goal 5- MLA Format
Overall Score
This bar graph breaks down the mean scores by each one of the five goal statements (see below)
and also shows the overall score. The graph also compares the data to the Fall semester mean
scores. Both the individual mean scores and the Fall/Spring mean score comparison will provide
useful benchmarks for future assessments.
The five Comp 1 course goals are:
1.
2.
3.
4.
5.
Demonstrate effective writing as a recursive process.
Demonstrate critical reading strategies.
Demonstrate the application of the conventions of standard written English.
Demonstrate the effective use of research techniques.
Demonstrate the proper use of Modern Language Association (MLA) format, including
page layout, in-text citations, and a Works Cited page.
While a bar graph provides one kind of aggregate information, a line graph is a better way to
express the distribution of the scores.
DMACC Comp I Writing Assessment
Page 3 of 22
COMP 1 Assessment- Fall 2006
Number of Responses per Score by Goal
Number of Responses
Summary Data Graph 2A
Source: DMACC, English Department
160
140
120
100
80
60
40
20
0
Score
1 - 1.9
2 - 2.9
3 - 3.9
4 - 4.9
5
Recursive Writing
23
43
65
91
41
Critical Reading
42
68
79
57
17
Standard Written English
0
27
112
107
17
Research Techniques
39
59
95
57
13
MLA Format
29
53
91
76
14
Overall
10
81
107
59
6
COMP 1 Assessment- Spring 2007
Number of Responses per Score by Goal
Number of Responses
Summary Data Graph 2B
Source: DMACC English Department
160
140
120
100
Score
80
60
40
20
0
1 - 1.9
2 - 2.9
3 - 3.9
4 - 4.9
5
Recursive Writing
56
64
79
88
38
Critical Reading
81
83
65
75
21
Standard Written English
2
37
98
154
34
Research Techniques
93
65
76
78
13
MLA Format
77
59
90
71
28
Overall
31
114
100
70
10
DMACC Comp I Writing Assessment
Page 4 of 22
These composite line graphs (2A & 2B) make useful comparisons of the scoring distribution of
each goal statement and also show each line in reference to the overall score. Each of the five
goals measured are interpreted individually. While both sets of overall scores were distributed in
fairly typical bell curves, the respective Fall/Spring line graphs reveal some interesting patterns.
Interpretation
As is evident by the bar graphs (Summary Data Graph 1), Goals 1 & 3 for both fall and spring
were the highest scoring goals. There was also significant movement to the right of the Spring
2007 Goal #3--Standard Written English. While this does not mean that the other measures were
not successful, it does point out some places where the department can focus its attention for
improvement. Summary Data Graphs 2A and 2B (rather than the bar graph) better illustrate
areas that can be improved.
Composite Score Graph
COMP 1 Assessment
Percentage of Responses per Overall Score
Source: DMACC English Department
Fall 2006
40.68%
Spring 2007
35.08%
30.77%
30.80%
22.43%
21.54%
9.54%
3.80%
3.08%
2.28%
1 - 1.9
2 - 2.9
3 - 3.9
4 - 4.9
5
Score
The Fall 2006 composite score graph line represents a fairly typical bell curve, though it is
weighted slightly to the left. The Spring 2007 graph line shows a distinct shift to the left. There
was roughly a 10% drop in the number of letters that scored a three and there were
corresponding increases in the left portion of the graph line. Future departmental assessment
goals might include class strategies for pushing this aggregate score to the right and increasing
the number of portfolios rating higher than a three on the five-point scale. A faculty discussion
of best practices would aid in this endeavor. This will be discussed at the English Faculty
Retreat on September 28.
DMACC Comp I Writing Assessment
Page 5 of 22
DMACC Comp I Writing Assessment
Page 6 of 22
Goal 1: Demonstrate effective writing as a recursive process.
COMP 1 Assessment
Percentage of Responses for Goal 1 - Recursive Writing Score
Source: DMACC English Department
Fall 2006
Spring 2007
0.35%
0.25%
0.27%
0.20%
0.24%
0.17%
0.16%
0.16%
0.12%
0.09%
1 - 1.9
2 - 2.9
3 - 3.9
4 - 4.9
5
Score
The shape of the data dispersion indicates that the majority of Comp 1 students understand the
recursive nature of writing. It also indicates that those Comp 1 students who understand the
process know it well, while those students who don’t understand the process REALLY don’t
understand it.
DMACC Comp I Writing Assessment
Page 7 of 22
Goal 2: Demonstrate critical reading strategies.
COMP 1 Assessment
Percentage of Responses for Goal 2 - Critical Reading Score
Source: DMACC English Department
Fall 2006
Spring 2007
30.04%
24.92%
25.86%
23.08%
25.54%
21.67%
20.00%
15.97%
6.46%
1 - 1.9
2 - 2.9
3 - 3.9
4 - 4.9
6.46%
5
Score
The mean score of Goal 2 in the fall semester was 2.53 (the lowest average score), and the shape
of the curve drops off considerably on the right. The comparable mean score in the spring
semester was 2.41 (the second lowest score). There were a significant number of spring
semester reflective letters that did not mention Goal 2. The reflective letter scores suggest that
not all of the Comp 1 faculty is addressing Goal 2 in their respective classrooms. It also appears
that while students in Comp 1 possess some sense of basic critical reading strategies, more could
be done to prepare them for the kinds of reading necessary for the research-writing component of
Comp 2. This is something to discuss at the September 28 English Faculty Retreat.
DMACC Comp I Writing Assessment
Page 8 of 22
Goal 3: Demonstrate the application of the conventions of standard written English.
COMP 1 Assessment
Percentage of Responses for Goal 3 - Standard Written English Score
Source: DMACC English Department
47.38%
Fall 2006
Spring 2007
42.59%
40.68%
30.15%
11.38%
10.46%
10.27%
6.46%
0.62%
1 - 1.9
0.00%
2 - 2.9
3 - 3.9
4 - 4.9
5
Score
The shapes of these graphs clearly demonstrate that most Comp 1 students have a basic
competence in standard written English. This is not to say, however, that the goal has been met.
The department needs to meet and set a standard for future assessments.
DMACC Comp I Writing Assessment
Page 9 of 22
Goal 4: Demonstrate the effective use of research techniques.
COMP 1 Assessment
Percentage of Responses for Goal 4 - Research Techniques Score
Source: DMACC English Department
Fall 2006
Spring 2007
36.12%
28.62%
24.00%
22.43%
23.38%
21.67%
20.00%
14.83%
4.94%
4.00%
1 - 1.9
2 - 2.9
3 - 3.9
4 - 4.9
5
Score
The Fall 2006 midpoint score of 2.58 and the overall shape of the line graph suggest that while
students may have a basic understanding of research techniques, more could be done to
prepare students for Comp 2. The Spring 2007 line graph shape, especially in relation to the fall
graph raises some questions, especially in terms of the rise of scores in the one range and the
drop in the three ranges. Like the interpretation of Goal 2 (demonstrate Critical reading
strategies), there are concerns that not all faculty are teaching to this competency. This graph
and the Goal 2 critical reading graph track very closely and suggest that the teachers who focus
on research techniques in Comp 1 may also do a better job of focusing on critical reading skills.
As noted before, this is something that will be discussed at the English Faculty Retreat.
DMACC Comp I Writing Assessment
Page 10 of 22
Goal 5: Demonstrate the proper use of Modern Language Association (MLA)
format, including page layout, in-text citations, and a Works Cited page.
COMP 1 Assessment
Percentage of Responses for Goal 5 - MLA Format Score
Source: DMACC English Department
Fall 2006
Spring 2007
34.60%
28.90%
23.69%
27.69%
20.15%
21.85%
18.15%
11.03%
8.62%
5.32%
1 - 1.9
2 - 2.9
3 - 3.9
4 - 4.9
5
Score
Again, the shape of the graph has a strong correlation to the critical reading and the research
techniques goals (Goals 2 & 4). The reasons for the spring semester 12.66% rise in scores in the
1 to 1.9 range are related to the parallel increases in the Goal 2 and Goal 4 scores. Focusing
more on these three goals (Numbers 2, 4 & 5) is also something that the department needs to
discuss.
Conclusion
While it is premature to make any specific claims about Comp 1, the information collected by
the analysis provides a statistically valid set of benchmarks for future assessments. After at least
three semesters of collecting and interpreting district-wide data, we will be able to see trends and
determine the success of any practices we implement.
Where do we go from here?







Determine when we have been successful with our measures
Share best practices/ strategies for teaching lower scoring goals
Suggest changes in course competencies
Discuss Comp 2 pilot assessment process
Discuss developmental assessment pilot process (ENG 060 College Prep Writing 1 &
ENG 061College Prep Writing 2)
Discuss on-line course assessments
Report on the inclusion of dual credit classes in future assessments
DMACC Comp I Writing Assessment
Page 11 of 22
An English faculty retreat was held on Sept. 28 with all district comp teachers (full and
part time) and the above items were discussed. A retreat report will follow.
+++++++++
Comp 1 Assessment Analysis
Fall 2006
Purpose
The Fall 2006 Comp 1 writing assessment was undertaken as part of DMACC’s institutional
accreditation process (AQIP). The writing assessment itself has been evolving over several
semesters.
Methodology
The Fall 2006 semester Comp 1 (ENG 105) assessment represents the first district-wide,
holistically scored, portfolio assessment program. All face-to-face Comp 1 classes on all six
campuses were part of the project, representing a total of 1,728 students.
Each student portfolio submitted for the assessment included a reflective letter, generated by the
student, to be assessed by trained faculty readers.
 A scoring rubric was used to evaluate the letters
 Each letter was read and evaluated by two different readers
Prior to the assessment, the following steps were taken:
5. In August, just prior to the fall 2006 semester, a memo was sent to all Comp 1 instructors
which included:
 A reminder to include the five Comp 1 goal statements on the course syllabus
 A reminder to collect portfolios from Comp 1 students
 FAQs regarding the assessment
 Business letter format requirements for the reflective letters
6. Two reader-training sessions - November 2006 and January 2007.
7. Joe DeHart, Executive Director of Planning and Research, determined the sample size of
325 students. After the November drop date, a random list of 400 names was generated.
8. Susan Pagnac, Assessment Coordinator, contacted each Comp 1 instructor requesting
them to submit portfolios from those students selected for the assessment. A total of 262
student portfolios (out of 400) were actually collected.
Reasons for not submitting or using requested portfolios included:
 Students were registered, but were no longer attending class
 One adjunct instructor moved at semester’s end without submitting the portfolios
 Portfolios submitted did not include the reflective letter
DMACC Comp I Writing Assessment
Page 12 of 22
On Saturday, January 20th, 2007, twenty-five trained readers scored the reflective letters in just
under two hours. Each letter was scored by two independent readers. The scores were entered
on an Excel spreadsheet and the scoring graphs (shown below) were created from that data.
DMACC Comp I Writing Assessment
Page 13 of 22
Findings
Summary Data Graph 1
COMP 1 Assessment- Fall 2006
Average Score by Goal
Source: DMACC, English Department
5.00
Mean Score
Standard Deviation
4.50
4.00
3.50
3.19
3.10
Score
Score
3.00
2.77
2.83
2.58
2.53
2.50
0.84
1.00
0.99
0.71
1.02
1.08
1.50
1.14
2.00
0.50
0.00
Goal 1- Recursive
Writing
Goal 2- Critical
Reading
Goal 3- Standard
Written English
Goal 4- Research
Techniques
Goal 5- MLA
Format
Overall Score
Goal
Goal
This bar graph breaks down the mean scores by each one of the five goal statements (see below)
and also shows the overall score. Of the five goals measured, Goal 3 (demonstrate the
application of the conventions of standard written English) scored the highest at 3.19 (on a five
point scale) and Goal 2 (demonstrate critical reading strategies) scored the lowest at 2.53. The
overall score of 2.83 reflects the bell shaped dispersion of the data. Both the individual mean
scores and the respective standard deviations will provide useful benchmarks for future
assessments.
The five Comp 1 course goals are:
6. Demonstrate effective writing as a recursive process.
7. Demonstrate critical reading strategies.
8. Demonstrate the application of the conventions of standard written English.
9. Demonstrate the effective use of research techniques.
10. Demonstrate the proper use of Modern Language Association (MLA) format, including
page layout, in-text citations, and a Works Cited page.
While a bar graph provides one kind of aggregate information, a line graph is a better way to
express the distribution of the scores.
DMACC Comp I Writing Assessment
Page 14 of 22
Summary Data Graph 2
COMP 1 Assessment- Fall 2006
Number of Responses per Score by Goal
Source: DMACC, English Department
Number of Responses
Number of Responses
120
100
Recursive Writing
Critical Reading
Standard Written English
Research Techniques
MLA Format
Overall
80
60
40
20
0
1
2
3
4
5
Recursive Writing
23
43
65
91
41
Critical Reading
42
68
79
57
17
Standard Written English
0
27
112
107
17
Research Techniques
39
59
95
57
13
MLA Format
29
53
91
76
14
Overall
10
81
107
59
6
Score
Score
This composite line graph makes a useful comparison of the scoring distribution of each goal
statement and also shows each line in reference to the overall score. Each of the five goals
measured are interpreted individually. While the overall scores were distributed in a fairly
typical bell curve, the respective line graphs reveal some interesting patterns.
Interpretation
As is evident by the bar graph (Summary Data Graph 1), Goals 1 & 3 were the highest scoring
goals. While this does not mean that the other measures were not successful, it does point out
some places where the department can focus its attention for improvement.
DMACC Comp I Writing Assessment
Page 15 of 22
Composite Score Graph
COMP 1 Assessment- Fall 2006
Number of Responses per Overall Score
Source: DMACC, English Department
120
Number of Responses
Number of Responses
100
80
60
40
20
0
Overall
1
2
3
4
5
10
81
107
59
6
Score
Score
The composite score graph represents a fairly typical bell curve and is weighted slightly to the
left. Future assessment goals might include pushing this aggregate score to the right, by
increasing the number of portfolios rating higher than a three on the five-point scale. A faculty
discussion of best practices would aid in this endeavor.
DMACC Comp I Writing Assessment
Page 16 of 22
Goal 1: Demonstrate effective writing as a recursive process.
COMP 1 Assessment- Fall 2006
Number of Responses per Recursive Writing Score
Source: DMACC, English Department
120
Number of Responses
Number of Responses
100
80
60
40
20
0
Goal 1- Recursive Writing
1
2
3
4
5
23
43
65
91
41
Score
Score
The aggregate score of 3.1 and the shape of the data dispersion indicate that the majority of
Comp 1 students understand the recursive nature of writing. It also indicates that those Comp 1
students who understand the process know it well, while those students who don’t understand the
process REALLY don’t understand it.
DMACC Comp I Writing Assessment
Page 17 of 22
Goal 2: Demonstrate critical reading strategies.
COMP 1 Assessment- Fall 2006
Number of Responses per Critical Reading Score
Source: DMACC, English Department
120
Number of Responses
Number of Responses
100
80
60
40
20
0
Goal 2- Critical Reading
1
2
3
4
5
42
68
79
57
17
Score
Score
The average score of Goal 2 was 2.53 (the lowest average score), and the shape of the curve
drops off considerably on the right. It appears that while students in Comp 1 possess some sense
of basic critical reading strategies, more could be done to prepare them for the kinds of reading
necessary for the research-writing component of Comp 2.
DMACC Comp I Writing Assessment
Page 18 of 22
Goal 3: Demonstrate the application of the conventions of standard written English.
COMP 1 Assessment- Fall 2006
Number of Responses per Standard Written English Score
Source: DMACC, English Department
120
Number of Responses
Number of Responses
100
80
60
40
20
0
Goal 3- Standard Written English
1
2
3
4
5
0
27
112
107
17
Score
Score
The shape of this graph clearly demonstrates that most Comp 1 students have a basic competence
in standard written English. This is not to say, however, that the goal has been met. The
department needs to meet and set a standard for future assessments.
DMACC Comp I Writing Assessment
Page 19 of 22
Goal 4: Demonstrate the effective use of research techniques.
COMP 1 Assessment- Fall 2006
Number of Responses per Research Techniques Score
Source: DMACC, English Department
120
Number of Responses
Number of Responses
100
80
60
40
20
0
Goal 4- Research Techniques
1
2
3
4
5
39
59
95
57
13
Score
Score
The midpoint score of 2.58 and the overall shape of the graph suggest that while students may
have a basic understanding of research techniques, more could be done to prepare students for
Comp 2. It is also interesting to note that this graph and the critical reading graph track very
closely. This suggests that the teachers who focus on research techniques in Comp 1 may also
focus on critical reading skills.
DMACC Comp I Writing Assessment
Page 20 of 22
Goal 5: Demonstrate the proper use of Modern Language Association
(MLA) format, including page layout, in-text citations, and a Works
Cited page.
COMP 1 Assessment- Fall 2006
Number of Responses per MLA Format Score
Source: DMACC, English Department
120
Number of Responses
Number of Responses
100
80
60
40
20
0
Goal 5- MLA Format
1
2
3
4
5
29
53
91
76
14
Score
Score
Again, the shape of the graph has a strong correlation to the critical reading and the
research techniques goals (Goals 2 & 4). Focusing more on these three goals (Numbers
2, 4 & 5) is something that the department needs to discuss.
Conclusion
While it is premature to make any specific claims about Comp 1, the information
collected by the analysis provides a statistically valid set of benchmarks for future
assessments. After at least three semesters of collecting and interpreting district-wide
data, we will be able to see trends and determine the success of any practices we
implement.
Where do we go from here?
English faculty retreat Sept. 28 (tentative) all comp teachers—full and part time
 Determine when we have been successful with our measures
DMACC Comp I Writing Assessment
Page 21 of 2




Share best practices/ strategies for teaching lower scoring goals
Suggest changes in course competencies
Discuss Comp 2 pilot data
Discuss developmental assessment pilot data (ENG 060 College Prep Writing 1 &
ENG 061College Prep Writing 2)
+++++++++
Comparative Results Fall 2006 with Spring 2007
4.00
2.74
2.83
2.69
2.74
2.77
2.43
2.58
2.39
2.55
Fall 2007 Mean
3.05
3.19
Spring 07 Mean
2.50
2.41
2.50
2.53
2.97
2.76
3.10
3.00
3.35
Fall 06 Mean
3.50
2.00
1.50
1.00
0.50
0.00
Goal 1- Recursive
Writing
Goal 2- Critical
Reading
DMACC Comp I Writing Assessment
Goal 3- Standard
Written English
Goal 4- Research
Techniques
Goal 5- MLA Format
Overall Score
Page 22 of 2