Program Assessment of English Early Start
August 30, 2012
Asao B. Inoue
Director of English Early Start
Special Assistant to the Provost for Writing Across the Curriculum
This report briefly assesses the summer 2012 English Early Start (ES) program at Fresno State. The
English ES program has four outcomes that were measured in final portfolios a few days after the end of
the second session (Aug 6). The outcomes listed in the course syllabus for ESE 1 and used in the reading
sessions with teachers/readers to determine the portfolio ratings are:




Reading Practices: Demonstrate a successful and purposeful use of at least two distinct reading
practices or strategies on a single text;
Academic Conversation: Demonstrate an understanding (e.g., explanation or discussion) of
what an “academic conversation” is;
Conversation Analysis: Demonstrate one attempted explication of an academic conversation for
a purpose, identifying significant trends, questions, aspects, or themes in the conversation;
Reflection on Practices: Reflect meaningfully upon at least two distinct reading practices or
strategies, discussing perhaps their development since the beginning of the course, questions
they still have about their own reading practices, new insights discovered, or the how they read
the texts included and discussed in their portfolios.
The average ratings on final portfolios show that most students achieved all four learning outcomes. The
portfolios were read by teachers from other ESE 1 courses (but not the teacher of record). Teachers
rated portfolios along the four outcomes, using a scale of 1-6, where scores in the 3-4 range means
adequate or acceptable quality demonstrated in the portfolio. The full scale of ratings used in training
readers is as follows:
1
consistently inadequate, of poor quality, and/or significantly lacking
2
consistently inadequate, of poor quality, but occasionally showing signs of
demonstrating competence
3
adequate or of acceptable quality but inconsistent, showing signs of competence
mingled with some problems
4
consistently adequate and of acceptable quality, showing competence with perhaps
some minor problems
5
consistently good quality, showing clear competence with few problems, and some
flashes of excellent or superior work
6
mostly or consistently excellent/superior quality, shows very few problems and several
or many signs of superior work
1
2
Generally
poor quality
throughout
3
4
Generally
acceptable
or adequate
quality
throughout
5
6
Generally
better to
superior
quality
throughout
Inoue 2
Portfolio Ratings
As illustrated in the graph below, the outcomes that were rated highest in portfolios were
demonstrations of reflection on reading practices (average rating of 3.39) and of the use of reading
practices on academic texts (average rating of 3.23). Demonstrations of an understanding of an
academic conversation (2.70) and of an analysis of an academic conversation (2.93) were lower. During
the reading session, without any knowledge of these findings, teachers felt that these two outcomes
(academic conversation and conversation analysis) where more difficult to accomplish under the tight
constraints of the 1 unit course. Mostly, there wasn’t enough time to allow for students to read an
actual academic conversation of more than one or two voices (sources or texts) and analyze them as a
conversation, or write about them.
Early Start Average Portfolio Ratings
3.39
3.23
2.70
Reading Practices
Academic
Conversation
2.93
Conversation
Analysis
Reflection
Overall, the portfolios demonstrated most students doing the work needed to pass the course and
presumably preparing themselves for their college writing courses.
Student Surveys
The end of the course, anonymous surveys taken by students online suggest also that students
accomplished the course outcomes. There were a total of 563 invitations sent, and 193 of those
students responded. Most of the respondents were female (by a ratio of about 2:1) and either Latino/a
or Asian/Asian Pacific Islander:


Male: 32.99% (64)
Female: 67.01% (130)





White: 3.09% (6)
Latino/a: 47.94% (93)
Asian/Asian Pacific Islander: 44.85% (87)
African American: 2.58% (5)
Native American: 1.55% (3)
Inoue 3
According to the responses, initially students taking ESE 1 had ambivalent feelings about taking the
course, with most saying they “liked the idea of getting prepared for college” (39.69%, 77), almost as
many saying they “did not like the idea” (37.11%, 72), or that they “did not have any feelings either way”
(23.20%, 45). However, when asked if they found the ESE 1 course helpful in preparing them for college
reading and writing, the majority of them said it did prepare them (87.11%, 169), while only 12.37% (24)
saying they weren’t sure, and 0.52% (1) saying it did not prepare that student. Additionally, of the 98
open responses given to the question, “What would you like to tell us that might help make future
versions of ES English better or more helpful for students?” the vast majority said something like the
following:






overall ilked how the ESE helped me personally to prepare me for the college classes
I wouldn't change anything had such a great experience :)
I think there is no need to make it better. It was perfect.
I just want to said that I really enjoy this class, It was really helpful for me. This class no only help
me to improve my reading skills it also helps me to become a better writer.
Had a good experience would not change a thing.
I really like how it was.
From the survey, it appears that the vast majority of ESE 1 students found the course beneficial and
liked it.
Of the reading practices and strategies used in the curriculum, most students found the unit on “reading
for deeper meaning / close reading” (43.81%, 85) and “TRIAC” (31.96.%, 62) to be the most helpful (see
graph below that shows all five units in the course). It should be noted that during the portfolio reading
session teachers mentioned wanting better or different textual samples for each of the reading
strategies of the course. Additionally, in the 98 open responses by students on the survey, no one
mentioned any of the reading strategies listed below in a negative way. Even GASCAP was mentioned as
helpful in those open responses.
There were also three academic readings used in the curriculum. Students read one each week of the
three week course. The majority found the Jones article the most interesting (50.26%, 96) and the one
they learned the most from (65.46%, 127). Students found the Nash essay least interesting (9.95%, 19),
the hardest to read (54.64%, 106), and learned the least from it (7.22%, 14). The three graphs below
show the results of the three questions concerning the three readings. It should also be noted that
several open responses in the surveys mentioned changing the readings. Students either noted that the
readings were not interesting enough or were too hard.
Inoue 4
Students were also asked about the learning outcomes of the course. When asked which outcome was
learned or practiced the best by the student, most said reflection on practices (31.77%, 61), while the
fewest said analyzing academic conversations (15.63%, 30). The graph below shows all the responses for
this question. These findings match closely the average ratings on portfolios of these same outcomes,
discussed above. It appears students know what they perform the strongest in and what they are
weakest in, at least in this course context. Perhaps the course also is helping students become more selfaware of their learning?
When asked which outcome students felt they did not learn well, again, they were consistent with their
portfolio ratings given by teachers. Their responses showed that most found analyzing academic
conversations (38.92%, 72) to be the outcome they learned the least. The graph below shows their full
responses to this question.
Inoue 5
What is most striking about the survey responses is how consistent they are with the portfolio ratings
along outcomes and with teacher’s perceptions of their students and courses during the final portfolio
reading sessions. All results point to positive learning experience for students, and most teachers
expressed similar positive experiences teaching the courses.
Conclusion and Closing the Loop
By all accounts, the English ES program during the summer of 2012 was a success if the measures of
success are the outcomes of the program, students’ abilities to recognized their own strengths and
weaknesses in reading academic work, and students’ and teachers’ positive feelings toward the course
and its outcomes.
There are a few revisions to the curriculum that I will make for next year’s implementation:




Replace both the Nash and the Postman readings for ones more like the Jones’ text, since most
students found them too difficult and uninteresting, while teachers also wanted readings that
matched Jones better.
Remove the “Conversation Analysis” outcome, since both students and teachers found it too
complex and difficult to accomplish in the short, 1-unit, 3 week course. In reflection, this
outcome seems too aggressive and too much to expect from students in the ESE 1 course.
Keep and perhaps find other short, teachable examples for the reading strategies offered in the
curriculum.
Continue the same teacher orientation before the courses begin and portfolio reading session
after the courses end.