Final Written Report Form for Learning Outcomes Assessment Grants 2010
(Submit by Monday, May 30, 2011 to the Director of Assessment)
Instruction: Please type your answers in the respective text areas.
Part A: General Information
Date:
Assessment
May 27, 2011
Primary investigator:
Project Title:
John Sundquist
Program and college or unit:
Beginning and Intermediate German Vocabulary
Email:
jsundqui@purdue.edu
Dept. of Foreign Languages and Literatures, CLA
Other directly involved collaborators and their programs and colleges/units:
None directly involved
Part B: Description of the Assessment Project
1) Which learning outcome(s) of your college or unit did this assessment project focus on?
Students will have an active vocabulary of 1000 German words in speech and in writing by the end of the
second semester of German language instruction (GER 102) and 2000 words by the end of the fourth
semester of language instruction (GER 202).
These benchmarks are based on previous research done by scholars in the area of applied linguistics on the
relationship between the number of contact hours and the acquisition of foreign language vocabulary.
2) What was the purpose of your assessment project? That is, what did you wish to find out about student
learning? As you began, what did you actually expected to find?
The purpose of this project was to analyze assessment data from students in the first four semesters of the
German language sequence at Purdue (GER 101-202). We wanted to assess the extent to which students are
actively using German vocabulary learned from course materials in their writing and in speech.
When we began, we expected to find that students were close to the assessment goals of 1000 words at the
end of two semesters and 2000 words at the end of four semesters.
In addition, we expected to find that students use a similar inventory of words in their speech as in their writing.
3) What activities comprised this assessment project? That is, what did you do in order to investigate your
hypothesis? If you could do it over, what would you change or not do?
The project included the collection of written and spoken language samples from students at the beginning and
end of each course in the four-semester sequence. In the first round of data collection, we collected electronic
copies of written essays from students. Students in all sections of GER 101, 102, 201, and 202 met in computer
labs on the day of the assessment projects in early September. A total of 425 students took part. They were
given a prompt and asked to write in German for 45 minutes without the aid of a dictionary. Students were
given the same prompt for their compositions in all four courses so as to allow comparison across levels while
keeping the task the same. In December, students in all sections of each course met again in computer labs.
The tasks varied across levels for the second essay. Again, students were provided with a prompt and not
allowed to use a dictionary. The essays were saved to Blackboard and then downloaded by the researcher.
In the second round of assessment assignments, students completed a speaking assessment assignment. Using
Professor Atsushi Fukada’s “Speak Anywhere” technology provided by the TELL Center at Purdue, in September
2010, students were asked to record their response to a video prompt. “Speak Anywhere” is a program that
runs through any internet browser and can be embedded in Blackboard. Students clicked on a video prompt
that asked that they record their response in German. They could complete this task anywhere where they had
access to a microphone and the internet (in their dormitory room, at home, in an ITaP lab). First, they were
provided with a practice round which allowed them to get used to the procedure and the recording program.
Next, they were asked to go online and record their voices to a prompt. As with the writing assessment, all four
levels/courses had the same task (describe a family member in as much detail as possible). Students did a
second speaking assessment task in December 2010, but the task was varied across all four levels. The
recordings were stored on the TELL server and then burned to a CD for transcription.
The procedures for Round 1 were repeated in January with new students enrolled in GER 101-202 in the Spring
2011 Semester. In total, there were three rounds of assessment:
Round 1: Writing Assessment #1 and Speaking Assessment #1 (conducted in September 2010)
Round 2: Writing Assessment #2 and Speaking Assessment #2 (conducted in December 2010)
Round 3: Writing Assessment #1 and Speaking Assessment #1 (repeated with new students in Spring
Semester courses)
The design was set up this way in order to evaluate student progress over one semester (with two rounds per
semester), comparison across levels (with the same task for each course for Writing/Speaking Assessment #1)
and to control for any effects caused by collecting data from only one semester (Round 3).
In the second phase of the project, a graduate Research Assistant (RA) transcribed the recorded speech into a
written format. In order to do this, the RA listened to each student’s recording in German and wrote the
transcribed speech in MS Word documents. She followed standard conventions for linguistic transcription to
indicate pauses, unintelligible words and false starts.
Due to time constraints, a comprehensive statistical analysis of the vocabulary in the writing and speech has not
yet been completed.
If I had to do this project again, I would find a different way to collect the data from the various sections of the
German courses. The teaching assistants who teach these courses vary in their level of familiarity with
technology, and some of them had difficulties in understanding how to collect the speech and writing samples
and relaying them to the researcher. Many of the problems arose from technical problems with Speak Anywhere
interacting with each student’s browser—sometimes in very different ways. Some students are more capable
than others at trouble-shooting in these situations. As a result, I would avoid the Speak Anywhere program and
go with a program that would allow students to record their voices in a computer lab with the researcher and an
assistant present (rather than allowing students to record themselves at home). This would avoid the individual
problems that had to be handled on a case-by-case basis.
Another change that I would make if I were doing this project again would be that I would request additional
funds for transcription. This process took much longer than expected, so I would hire two research assistants
so that it could be completed more quickly. Although the RA was well qualified, the process of transcribing each
individual recording is very tedious and proved to take almost twice as long as I expected.
4) What did you actually find out about student learning? What data support your conclusion?
The statistical analysis of the data collected is still underway. The results at this point are tentative and are
based on close reading and general analysis of both the transcripts and written compositions. Preliminary
results suggest, however, that students’ vocabulary development does not meet the goals of 1000 words at
the end of the second semester of instruction or 2000 words at the end of the fourth semester.
The general descriptive analysis of the data is that students in GER 101 performed better than expected and
used a wide range of vocabulary that they have learned from the course material in GER 101. However,
results indicate that the students in GER 102 have only a slightly wider range of vocabulary than those in GER
101. Students in GER 102 seem to be able to express themselves more freely in their writing, showing
general increases in vocabulary during Rounds 1 and 2. However, in speech, students in GER 102 don’t have
that much of a wider range in vocabulary than the students in GER 101. The general assumption at this point
is that the 100-level students do not meet the goal of 1000 words by the end of the second semester.
The gradual increase in vocabulary expressed in student writing seems to continue into the 200-level courses
where students show more frequent use of less commonly used words. However, In speech, the trend from
the 100-level continues: in general, students show greater strides in their expressiveness and range of
vocabulary in writing than in speech. In addition, the plateau effect in the 100-level is evident at the 200level. Students generally showed the most increases in GER 201 (third semester) and only a slight increase in
the final semester (GER 202).
5) What next? Based on what you found, what next steps or interventions to improve student learning do you
plan to take? When and where do you plan to do this?
We are also looking at the differences in the vocabulary that students are able to put into use in speech
versus writing. The results suggest that the differences here are larger than what we initially expected. That
is, students are able to use a significantly larger vocabulary in writing than they are In speech.
In order to improve vocabulary learning in the four-semester language sequence, the next step would be to
change the curriculum to increase the focus on vocabulary learning. This would include in-class activities, work
outside the classroom, and an emphasis on vocabulary on the quizzes and tests. After further analysis of the
data, I might also consider supplementing the course materials in order to encourage vocabulary acquisition, for
example through more extensive reading.
In order to address the discrepancy between speech and writing that the preliminary analysis has revealed, we
might consider focusing on more regular practice of in-class speaking activities. The “Speak Anywhere”
program could, for instance, allow students to respond to narrowly focused exercises that target vocabulary
from that day’s lessons. Students could record themselves, responding to instructor’s video/audio prompts
online. Instead of only assessing vocabulary in speech once or twice a semester, it would be beneficial to
students to practice using the vocabulary after each class meeting.
Furthermore, I intend to implement more dialogic speaking tasks and more interpersonal writing tasks in
classroom instruction. The dialogic tasks would require interacting with a partner (rather than speaking into a
microphone/computer outside of class). Conversations with a partner are more likely to be beneficial to the
learner in acquiring vocabulary because both listening and speaking skills are being exercised simultaneously.
Secondly, interpersonal writing tasks, such as an interactive journal or letter writing, will help students improve
their vocabulary in ways in which traditional essay writing may not. Letter writing, in which students ask
questions or write in a more informal register, help students solidify vocabulary words that are more common in
spoken language and in authentic situations.
6) Appendix: Please attach publications and presentation abstracts resulting from this work.
I am the sole (or primary) collaborator on this project. My collaborators (if appropriate) and I are available at a
mutually acceptable time to present an oral report, with our findings and data, to the University community.
Signature of Investigator