Dr. Franca Ferrari, Professor Barbara Lynch and Professor Rosanne Vogel _ Speech and Theatre
Data Analysis and Report by Dr. Franca Ferrari
Middle States Listening Skills Report for SP 211-Fall 2013
List your student learning outcomes as described in your syllabus. Please list ALL of the Student Learning
Outcomes that are listed in your syllabus
Gen Ed. Obj.
1.
Communicate
effectively through
reading, writing,
listening and
speaking
2. use analytical
reasoning to identify
issues or problems
and evaluate
evidence in order to
make informed
decisions
Outcome desired
To develop students
as more effective
listeners and
evaluators of
communication, in
order to make them, in
turn, more capable
learners and
intelligent decisionmakers.
Outcome desired
To develop the public
speaking skills
necessary to
effectively present
informative and
persuasive speeches
Outcome desired
To learn the major
communication
theories in public
speaking,
interpersonal
communication, selfcommunication,
intercultural
communication, and
group communication.
Students will be able
to incorporate these
theories into their own
speaking styles
Outcome desired
To work towards
understanding and
overcoming
communication
apprehension
To develop critical
thinking and problemsolving skills that
enable students to
understand the
intricate link between
audience, speaker,
and occasion
3. reason
quantitatively and
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 1
mathematically as
required in their fields
of interest and in
everyday life
4. use information
management and
technology skills
effectively for
academic research
and lifelong learning
To develop skills in
diverse
communication
contexts including
small groups,
computer-mediated
communities and
professional
communities
5. integrate
knowledge and skills
in their program of
study
To understand the
overwhelming
importance of
effective
communication in all
aspects of academic,
professional, and
everyday life
6. differentiate and
make informed
decisions about
issues based on
multiple value
systems
7. work
collaboratively in
diverse groups
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 2
directed at
accomplishing
learning objectives
8. use historical or
social sciences
perspectives to
examine formation of
ideas, human
behavior, social
institutions, or social
processes
9. employ concepts
and methods of the
natural and physical
sciences to make
informed judgments
10. apply aesthetic
and intellectual
criteria in the
evaluation or creation
of works in the
humanities or the
arts
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 3
Describe the assessment activity and the (student learning outcome(s) it addresses ) that occurred in your
course.
Assignment: Students will watch and listen to two video recorded speeches from Lucas’s DVD included in “The Art of
Public Speaking”. The first speech is “The Hidden World of Chili Peppers” (Needs Improvement) and the second
speech is “Medical Robots: From Science Fiction to Science Fantasy” (Needs Improvement). The two speeches are
substandard, i.e., they do not have all the required speech components. Students will use the speech evaluation form
included here to evaluate the two speeches indicating which speech components are present and which are missing,
indicating how many topics and transitions the speaker introduce and how many sources are cited. Moreover the
students will answer three listening and comprehension questions about the content of the each speech. The form
used is intended to facilitate student critical listening by identifying missing parts and by processing and understanding
what heard.
This assignment is designed primarily with the intention to test student listening skills as far as their ability to identify
relevant speech outline components (QUESTIONS 1-12) and show understanding of the content of the speeches
(QUESTIONS 13-15). Because the goal is to track student’s listening, each student should repeat this exercise twice
during the semester, once while listening to the first Lucas speech and the second time while listening to the second
Lucas speech.
Instructions: Instructors will administer the listening test twice during the semester. The first time the test should be
administered after the instructor has taught the class on how to create a speech outline for their first informative (or
persuasive) speech. The second time the test should be administered after students have been instructed on how to
create the speech outline for their second persuasive (or informative) speech or preferably after they have delivered
their second speech.
All instructors need to follow the speech outline format indicated in your speech text book or conform to it while
instructing students on how to create an informative or persuasive speech outline. Students will first complete the
attached scoring SPEECH EVALUATION form and then they will transfer their results on the scantron sheet.
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 4
When to administer the assessments:
The first assessment needs to be administered before the 6th week of the semester after instructors have taught
students about the Informative (or persuasive Speech) outline. Instructors will show the following speech “The Hidden
World of Chili Peppers (Needs Improvement)” to two sections of SP211. By the 12th week of the semester instructors
have to show the following speech: “Medical Robots: From Science Fiction to Science Fact (needs improvement)” to
the same two sections of SP211.
List the data collection instrument (s) used for assessment.
Speech evaluation forms used by students to critically listen on the two speeches are given below. Each student’s will
complete two forms one in connection with the first speech and the second one with the second speech. Students will
transfer their results on a scantron sheet. All the scantron sheets will be graded at the end of the semester in two
batches. The first batch contains the scantron sheets for the first speech and the second batch contains the scantron
sheets for the second speech.
For Fall 2013 we made few changes to the speech evaluation forms by adding few footnotes to help the students with
difficult terminology and with the definition of what a proper citation is.
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 5
ASSESSMENT#1
SPEECH EVALUATION FORM
First Name________________________ Surname___________________________________
Speech Title: _____________________________________________________________________________
Listen carefully indicating which speech components are present in this speech and rating the speaker on each point. Remember that your goal is to
identify which speech components are missing from the speaker’s speech.
INTRODUCTION
NO
YES
1. Attention getter


2. Reason to listen


3. Reveal Topic


4. Established credibility


5. Preview body of speech


BODY
0
1
2
>2
6. How many main ideas




Dr. Franca Ferrari 02/04/2014
Speech 211
Page 6
7. How many sources are properly cited1



8. How many transitions




CONCLUSION
NO
YES
9. Cues the audience


10. Review of main points


11. Restate thesis


12. Vivid ending/Closure


QUESTIONS
13. This speech is about
A. Chili peppers
B. The shortage of good peppers
C. A brief history of chili peppers
D. a and c
1
Please remember that for a source to be cited properly, you need to hear one of the following combinations of information: (1) author and year of publication; (2) author and
title of article/ book; or (3) author, title of article/book and year of publication. Mentioning just the author or a title does not count as a proper citation.
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 7
14. If you eat a pepper that is too hot you should use:
A. Water and salt
B. Milk or ice cream
C. Ice water
D. Milk or yogurt
15. Chili peppers can be used for:
A. Protection from muggers
B. Food storage
C. Fatigue
D. Fertility
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 8
ASSESSMENT # 2
SPEECH EVALUATION FORM
First Name________________________ Surname___________________________________
Speech Title: _____________________________________________________________________________
Listen carefully indicating which speech components are present in this speech and rating the speaker on each point. Remember that your goal is to
identify which speech components are missing from the speaker’s speech.
INTRODUCTION
NO
YES
1. Attention getter


2. Reason to listen


3. Reveal Topic


4. Established credibility


5. Preview body of speech


BODY
0
1
2
>2
6. How many main ideas




7. How many sources are properly cited 2







8. How many transitions
2
Please remember that for a source to be cited properly, you need to hear one of the following combinations of information: (1) author and year of publication; (2) author and title of article/ book; or
(3) author, title of article/book and year of publication. Mentioning just the author or a title does not count as a proper citation.
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 9
CONCLUSION
NO
YES
9. Cues the audience


10. Review of main points


11. Restate thesis


12. Vivid ending/Closure


COMPREHENSION
13. Orderly3 robots can do the following:
A. let doctors see patients from a remote location
B. communicate with each other
C. allow a doctor to see a patient’s vital signs
D. none of the above
14. Robots called ‘Da Vinci’4 are used for:
A. delivering items in a hospital
B. allowing patients to create art
3
4
Definition of ‘orderly’: an attendant in a hospital responsible for the nonmedical care of patients and the maintenance of order and cleanliness.
Leonardo Da Vinci was an Italian Renaissance polymath: painter, sculptor, architect, musician, mathematician, engineer, inventor, anatomist, geologist, cartographer, botanist, and writer.
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 10
C. helping hospital staff accomplishing routine tasks
D. surgery
15. The speaker told a story about a woman in New Jersey who:
A. had heart surgery
B. was operated on by a robot
C. had an orderly robot
D. invented a medical robot
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 11
Provide an analysis (and summary) of the assessment results that were obtained.
197 students took part to both listening exercises. First we will report the results of Questions 1-12 testing ‘critical
listening’ and then Question 13-15 testing comprehension.
QUESTIONS 1-12: Critical Listening
The results for number of correct answers for each questions, means, SD, and percentages are summarized in the
following table.
LISTENING 1
LISTENING 2
%CHANGE
Question
# correct
Mean - SD
%
# correct
Mean -SD
%
% Change
1
102
0.52 – 0.50
52
105
0.53 – 0.50
53
1%
2
104
0.53 – 0.50
53
121
0.61 – 0.49
61
8%
3
159
0.81 – 0.39
81
139
0.71 – 0.46
71
-10%
4
139
0.71 – 0.46
71
140
0.71 – 0.45
71
0%
5
124
0.63 – 0.48
63
149
0.75 – 0.43
75
8%
6
40
0.20 – 0.40
20
51
0.26 – 0.43
26
6%
7
116
0.59 – 0.49
59
109
0.55 – 0.50
55
-4%
8
79
0.40 – 0.49
40
89
0.45 – 0.50
45
5%
9
63
0.32 – 0.45
32
89
0.45 – 0.50
45
13%
10
144
0.73 – 0.44
73
152
0.77 – 0.42
77
4%
11
119
0.60 – 0.49
60
125
0.63 – 0.48
63
3%
12
79
0.40 – 0.49
40
147
0.75 – 0.44
75
35%
1268
0.54 – 0.50
50
1416
0.60 – 0.49
60
10%
Total
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 12
The following two charts visually shows the students’ number and percentage of correct answers for both listening
assessments.
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 13
percentage of correct answers
PERCENTAGES OF CORRECT ANSWERS
90
80
70
60
50
40
30
20
10
0
Series1
Q1
52
Q2
53
Q3
81
Q4
71
Q5
63
Q6
20
Q7
59
Q8
40
Q9
32
Series2
53
61
70
71
75
25
55
45
45
Q10 Q11 Q12
73
60
40
77
68
74
We also performed a repeated measure ANOVA to see whether there was a significant difference between the overall
results for the assessments taken at Time 1 and at Time 2; whether there was a significant difference between the results
for each individual question at Time 1 and Time 2 and; finally, if there was significant difference between the interaction of
questions and Time. The results are presented in the following table.
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 14
ANOVA
Source of
Variation
Sample
Columns
Interaction
Within
Total
SS
df
MS
F
P-value
F crit
4.632826
108.6007
13.11844
1033.99
1
11
11
4704
4.632826
9.872789
1.192586
0.219811
21.07643
44.91495
5.425511
4.53E-06
7.03E-94
1.22E-08
3.843436
1.790679
1.790679
1160.342
4727
0.245523
105.4856
0.043919
As the F values in the table indicate there was a significant difference for Time, Questions and Interaction, which together
with the positive direction of change indicated by the mean values, suggest that students improved their critical listening
skills between the first and second administering of the assessment.
QUESTIONS 13-15: COMPREHENSION
The results for number of correct answers, means and percentages are summarized in the following table and in the
chart.
13
14
15
# correct
108
182
167
LISTENING 1
mean
0.55
0.92
0.85
Dr. Franca Ferrari 02/04/2014
%
55
92
85
LISTENING 2
# correct mean
87
0.44
153
0.77
170
0.86
Speech 211
%CHANGE
%
44
78
86
-11
-14
1
Page 15
The following two charts visually show the students’ number and percentage of correct answers for both listening
assessments.
Number of Correct Answers
200
150
100
50
0
Q13
Q14
TIME1
Dr. Franca Ferrari 02/04/2014
Speech 211
Q15
TIME2
Page 16
Percentage of Correct Answers
percentages
100
80
60
40
20
0
Q13
Q14
Q15
TIME1
Q13
55
Q14
92
Q15
85
TIME2
44
78
86
As will be discussed in the next section, though students listening comprehension skills did not improve from one
assessment to the next, the Fall 2013 overall students’ performance on comprehension has notably improved if compared
with the Spring 2013 students’ cohort.
Describe how the assessment results that were obtained affected (or did not affect) the student learning
outcomes you identified. As part of your discussion, describe any plans you have to address the areas where
students need to improve.
Students were tested on the individuation of main speech components here listed as ‘intro’, ‘body’, ‘conclusion’ and
‘comprehension’ of the speech twice.
Question 1-12 measured students' ability to recognize parts of speech, a concept reinforced by the syllabus of SP 211. On
the ability to recognize parts of a speech students showed improvement on 9 areas, i.e., (#1) attention getter (+1%),
(#2) reason to listen (+8%),(#4), (#5) preview body of speech (+8%), (#6) main ideas (+6%), (#8) number of
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 17
transitions (+5%), (#9) cues the audience (+13%), (#10) review main points (+4%), (#11) restate thesis (+3%) and
(#12) vivid ending (+35%). There was no change for (#4) credibility statement between Time 1 and Time2.
There was instead deterioration for 2 areas: (#3) reveal topic (-10%) and (#7) number of cited sources (-4%).
The deterioration on identify the speech topic may have been confounded due to the fact that the two speeches presented
to the students may not have been equally strong in these areas. The slight deterioration attested for (#7) number of cited
sources (-4%) is possibly due to students’ inability to follow instructions rather than on students’ lack of knowledge of what
a proper citation is. Not only did the students receive instructions on how to cite properly from the instructors and from the
librarians, and not only they cited themselves while delivering their speeches, but also before the administering of the
assessments instructors were supposed to read aloud the footnotes on the student’s evaluation forms where clearly it was
specified how to properly cite a source in an oral presentation.
Overall the Fall 2013 students’ cohort improved their abilities to identify speech components showing a 10% gain score
from the first to the second administering of the assessment.
With respect to the Spring2013 assessment cohort students improved in the recognition of content components and
similarly less students showed to have difficulties in identifying a properly cited source in comparison with the results of
the Spring2013 cohort as the following table indicates.
Preview (#5)
Main Topic (#6)
Review (#10)
Restate Thesis (#11)
Citing sources (#7)
Spring13
2%
2%
2%
2%
-14%
Fall13
12%
5%
4%
8%
-4%
We believe students’ improvement in these areas is the result of the recommendations we made to all the faculty in the
action plan of our Fall 2013 report.
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 18
Question 13-15 measured the student's ability to extract content from a spoken message. Students’ comprehension
deteriorated in two questions (#13, -11%) (#14, -14%) but improved in question (#15, +1%).
However, if we compare Fall13 students’ results with Spring13 students’ results, we can notice that the Fall students
improved their performance possibly because of the changes we made to the student evaluation form with the addition of
cultural background explanation for the second speech and to our recommendations to the faculty as stated in our Spring
2013 Actions Plan. As the data in the table indicate, in Fall 2013 students improved dramatically at least for Q14 and
Q15.
Q13
Q14
Q15
Spring13
-10
-23
-17
Fall13
-11
-14
1
We believe that the different presentation styles of the two speakers and the different degree of structural complexity of
the two speeches in terms of content and organization might have contributed to the deterioration in comprehension of the
second assessment. Moreover, as the literature on listening research points out improvement in students’ listening and
comprehension skills take longer than a 6-8 week period and is influenced by many individual and environmental factors.
In our case one the fact that students were assessed by different instructors under different circumstances might have
well played a role in determining the results.
Action plan:
On ability to recognize parts of the speech, we believe faculty should give more emphasis to the concept of topic
identification, main ideas identification and citing sources. With regards to citing sources faculty should organize for their
classes more than one trip to the library and create target exercises to identify citation in speeches and to incorporate
citation while delivering speeches following the MLA citation style for oral citations.
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 19
With regards to topic and main ideas identification together with the ability to extract content from spoken language
(listening and comprehension) we believe that faculty should assign more in class activities asking students to summarize,
analyze and report on content from spoken language.
However, as speeches were both substandard, the skill of recognizing parts of the speech at or above 80 % level from the
first to the second assessment might be too advanced for students and it might require longer than a semester to see a
substantial improvement, especially with regards to comprehension. To test if students’ deterioration in comprehension is
due to the different content of the two speeches, for Spring 2014 we will modify how students are assessed without
altering the assessment tool for the moment. Keeping the same testing conditions used in the Fall, this semester we will
divide instructors in two groups and each group will test students’ listening skills using one speech only. Group 1 will test
their students twice using the speech “The Hidden World of Chili Peppers” (Needs Improvement); while Group 2 will use
“Medical Robots: From Science Fiction to Science Fantasy” (Needs Improvement). We believe that this change will
enable us to assess whether the content differences between the two speeches have an effect on students’ results. At the
end of this round of assessments we will be able to better determine how to intervene to help our students to improve their
comprehensive listening skills. However, the results of this assessment will be discussed at the February Faculty Meeting
and future plan and possible recommendations will be distributed to all faculty once approved at the meeting.
Dr. Franca Ferrari 02/04/2014
Speech 211
Page 20