REPORT
Assessment of LI 112
Methodology
At the end of the fall 2013 semester, eighty-nine students attending Beginning II Italian
courses were assessed. They were tested on whether they were able to communicate effectively
through writing, and speaking at a mid-novice level as indicated in the American Council on
Teaching Foreign Languages (General Education Objectives # 1) . They were also evaluated on
whether they could apply their analytical reasoning to identify and correctly use, from a lexical
and morpho-syntactic and semantic perspective, what they have learned during the course
(General Education Objectives # 13).
In order to determine whether students had successfully reached these objectives, they
were asked to provide an oral and written description of a past event avoiding unnecessary repetition of words or phrases (Course Objective). With that in mind, they had to undertake a conversation with an imaginary friend (played by the instructor) where they would describe an unforgettable vacation they took (Learning Outcome). They had to provide information regarding its
destination and duration. They also had to include the means of transportation used, and a brief
description of the travelling companions. In addition, they had to provide a description of the activities done and the places visited. The conversation concluded with some comments on their
vacation and some brief explanation of their positive or negative remarks.
Students were evaluated on whether they were able to understand the questions asked by
the instruction with no great difficulty, and answer them with a relative degree of grammatical
accuracy. Particular attention was also paid on the presence of pauses or interruptions in their
replies as well as their level of native-like pronunciation. The use of appropriate of the vocabulary was also taken into account (see Appendix 1 for a detailed explanation of these parameters).
1
Their writing task consisted of providing a description of what their life was like during
their High School years. They were suggested to include some information regarding the name
and location of their school, their favorite subject and/or preferred teacher and explain the
reasons of their choice. They were also asked to indicate what they liked or did not like about the
school they attended. They also had to briefly describe what a typical day was like, and whether
they were pursuing any hobbies or sports.
Their 80-word paragraph was evaluated on whether it provided a detailed description of
the student’s time spent in HS. Furthermore, particular attention was paid on the level of discourse and appropriateness of the vocabulary used in their writing, as well as their degree grammatical accuracy (see Appendix 2).
Data analysis
As indicated in Rubric 1, students’ oral performance was rated according to five
parameters: listening comprehension, fluidity, pronunciation, vocabulary and grammar. Each
parameter was divided in four different level based on whether student met or exceeded the
expected proficiency level, or completely or barely failed to do so. Each level was assigned one
point. The total points a student could achieve (adding up all five parameters, each worth a
maximum of four points) was 20. The scale defined the following ranges: 0-5 points= student
performance does not meet expectations; 6 – 10 points= student performance almost meets
expectations; 11-15 points= student performance meets expectations; 16-20= student
performance exceeds expectations.
The average of total points achieved by the eighty-nine students tested was 16.2.
According to the scale, this percentile slightly exceeds the 11-15 range, indicating that, in
general, students’ met the expected oral proficiency level. Actually, 97% of them achieved or
2
exceed their performance level, whereas only 3% of them were unable to reach the minimum
standards (see Table 1 below)
Table 1
Number and percentage of students in each performance level for the speaking task
Students (n = 89)
Percentage
Does not meet
expectations
1
1.5
Almost meet
expectations
1
1.5
Meet expectations
29
32.6
Exceed
expectations
58
64.4
n = number of students
Students’ written performance was also rated according to different parameters, i.e. Task
completion, Level of discourse, Vocabulary (richness) and Grammar (accuracy). Similar to the
speaking task, each parameter was divided in four different proficiency levels, namely
Level 1 = Student does not meet expectations
Level 2 = Almost meet expectations
Level 3 = Meet expectations
Level 4 = Exceed expectations
For each level, student will be assigned one point for a total of 4 points. The maximum
of points a student could receive was 16. The scale defined the following ranges: 0-4 points=
student performance does not meet expectations; 5 – 8 points= student performance almost meets
expectations; 9-12 points= student performance meets expectations; 13-16= student performance
exceeds expectations.
Students’ average score on the written task was 12.9, suggesting that, in general, they
met, actually slightly exceeded the anticipated proficiency level. More specifically, 89% were
writing at or higher level than expected, whereas only 8% of them were approaching their
required level, and any of them did not meet the minimal standards. This is illustrated in Table 2
below.
3
Table 2
Number and percentage of students in each performance level for the written task
Students (n = 89)
Percentage
n = number of students
Do not meet
expectations
0
0
Almost meet
expectations
8
9
Meet expectations
30
33.7
Exceed
expectations
37
57.3
In sum, data have shown that the majority of students have reached the objective of the course
tested here, which was that of describing a past event, orally or in writing.
A closer look at the data indicates that, in the speaking task, the average score in each parameter was as follows:
a.
b.
c.
d.
e.
listening comprehension 3.5
fluidity
3.3
pronunciation
3.5
vocabulary
3.1
grammar
2.9
As we can see, for all parameters, except for grammar, students met or slightly exceeded the expectations. In any case, the highest average score was reached in the Listening comprehension
and Pronunciation, followed by Fluidity, Pronunciation and Grammar. Furthermore, the majority
of students exceeded the expectations for the Listening comprehension and the Pronunciation,
whereas a slightly less number of students reached that level in the Fluidity, and definitely less of
them for the Grammar, as shown in Table 3 below.
Table 3
Number of students in each competency type and level for the speaking task
Levels
Listening compr.
4
46
3
41
2
2
1
0
Total
89
Fluidity
28
48
13
0
89
Pronunciation
42
39
8
0
89
Vocabulary
27
39
23
0
89
Grammar
15
50
21
3
89
4
As for the writing task, the average score for each parameter was as follows:
a.
b.
c.
d.
Task completion
Level of discourse
Vocabulary
Grammar
3.5
3.5
3.1
2.6
Similar to the speaking activity, students reached or slightly exceeded the expected proficiency
level. Unfortunately, the grammatical accuracy of their writing was not as good the other aspects
tested. In any case, students were quite accurate in completing the task and use the appropriate
language and vocabulary. As we can see from Table 4 below, the majority of them exceeded the
expectations for Task completion and Level of discourse. A slightly less number of them reached
that level for their use of the vocabulary, and visibly less students scored as high for the correct
application of grammar rules of the Italian language.
Table 4
Number of students in each competency type and level for writing task
Levels
Task Completion
4
59
3
22
2
6
1
2
Total
89
Comprehensibility
41
11
13
1
89
Level of Discourse
46
29
13
1
89
Vocabulary
32
36
19
2
89
Grammar
14
25
40
10
89
Conclusion and future action plan
Data have displayed an interesting scenario regarding students’ abilities to narrate past
events either through writing or oral description. Their narrations are quite accurate and detailed
showing that students have mastered a great variety of vocabulary words. Furthermore, even
though a slightly foreign accent may be detected in their oral descriptions, it does not interfere
5
with their communication. Unfortunately, the same cannot be said with regard to their use of
syntactic structures. Students do not appear to be as grammatically accurate as they are in other
oral and writing competencies. Their speech patterns and writing products still show some
morpho-syntactic inaccuracies; they, nevertheless, do not seem to completely hinder the
understanding of their message.
In sum, despite the encouraging results obtained, students need to further improve their
morpho-syntactic knowledge of their target language. This is quite surprising since they
frequently exposed and formally instructed regarding the grammar rules and the syntactic
structures of Italian. Their less-developed grammar competency with respect to the other areas
suggests that, for some reasons, they do not fully retain what has been presented to them.
In light of these results, I would suggest that more attention be paid to further develop
that particular language ability. With that in mind, the syllabus of this particular course should
include more activities whose objective is to provide students with further practice and drilling of
particularly difficult structures such as the formation of the Italian past tense. Additional
verification practices in form of weekly quizzes and daily homework assignments should also be
part of daily lesson plans to make sure that a particular structure has been fully learned. This
procedure would certainly help the mastery of the necessary vocabulary, which does not seem to
develop as fast as the other abilities.
6
Appendices
Appendix 1
Speaking Task - Holistic Rubric
Performance
exceeds
expectations
Listening
comprehension
Student understands
the examiner’s
questions and
responds easily and
without probing
Fluidity
Speech
continuous with
few pauses or
stumbling
4 points
Performance
meets
expectations
4 points
Student understands
the examiner’s
questions and knows
how to respond but
needs occasional
probing
Some hesitation
but manages to
continue and to
complete
her/his thoughts
Pronunciation
Vocabulary
Grammar
Enhances
communication
Rich use of
vocabulary
Correct use
of basic
language
structures
(1-5 errors)
4 points
Adequate and
accurate use of
vocabulary for
this level
4 points
Adequate use
of basic
language
structures
4 points
Does not
interfere with
communication
(6-10 errors)
3 points
Performance
almost meets
expectations
3 points
Student only
understands the
examiner’s questions
after probing
Performance
does not meet
expectations
2 points
Student fails to
understand most
questions even after
probing
0-1 points
Speech choppy
and/or slow
with frequent
pauses. Few or
incomplete
thoughts
2 points
Speech halting
and uneven
with long
pauses or
incomplete
thoughts
0-1 points
3 points
Occasionally
interferes with
communication
3 points
Somewhat
inadequate
and/or
inaccurate use
of vocabulary.
3 points
Emerging use
of basic
language
structures
(11-15 errors)
2 points
Frequently
interferes with
communication
2 points
Inadequate
and/or
inaccurate use
of vocabulary
2 points
Inadequate
and/or
inaccurate use
of basic
language
structures
(more than 16
errors)
0-1 points
0-1 points
0-1 points
7
Appendix 2
Writing Task - Holistic Rubric.
Exceeds
expectations
Task
Completion
Level of
Discourse
Vocab.
Grammar
Superior completion
of the task. Students
fully address the
information
requested, and
provide additional
details
Sentences are fully
developed and
interconnected with
conjunctions (e.g.
AND, BUT, or
BECAUSE
Rich use of
vocabulary
Perfect control of the
syntactic structures
required (Accuracy
level 90% - 100%).
4 POINTS
Completion of task.
Students fully
address the
information
provided, but do not
provide additional
details
4 POINTS
Sentences are fully
developed. Cohesive
devices are
sporadically used
4 POINTS
Adequate and
accurate use of
vocabulary
4 POINTS
Meets
expectations
Adequate control of
the syntactic
structures. Some
grammatical
imperfections
(Accuracy level 79%
- 89%).
3 POINTS
3 POINTS
Almost meets
expectations
Does not meet
expectations
3 POINTS
3 POINTS
Emerging control of
the syntactic
structures Several
grammatical
imperfections
(Accuracy level 61%
- 78%).
2 POINTS
Partial completion of
task. Students
complete no more
than 60% of the
information
requested
Sentences are
somewhat complete.
Rare use of cohesive
devices
Somewhat
inadequate and/or
inaccurate use of
vocabulary
2 POINTS
2 POINTS
2 POINTS
Minimal completion
of task. Students
complete less than
40% of the
information
requested.
Sentences are mostly
incomplete. No use
of cohesive devices
Inadequate and/or
inaccurate use of
vocabulary
Minimal control of
the syntactic
structures.
Numerous
grammatical errors
(Accuracy level 0%60%)
1 POINT
1 POINT
1 POINT
1 POINT
8
Appendix 3
Writing samples
9
10
11
12
13