Dr. Franca Ferrari-Bridgers, Professor Barbara Lynch and Professor Rosanne Vogel _ Speech and Theatre
Data Analysis and Report by Dr. Franca Ferrari-Bridgers
Middle States Listening Skills Report for SP 211-Fall 2015
List your student learning outcomes as described in your syllabus. Please list ALL of the Student Learning Outcomes that are listed in your
syllabus
Gen Ed. Obj.
1. Communicate effectively
through reading, writing,
listening and speaking
Outcome desired
To develop students as
more effective listeners
and evaluators of
communication, in order
to make them, in turn,
more capable learners and
intelligent decisionmakers.
2. use analytical reasoning to
identify issues or problems
and evaluate evidence in
order to make informed
decisions
To develop critical
thinking and problemsolving skills that enable
students to understand
the intricate link between
audience, speaker, and
occasion
3. reason quantitatively and
mathematically as required
in their fields of interest and
in everyday life
4. use information
management and technology
Outcome desired
To develop the public
speaking skills necessary
to effectively present
informative and
persuasive speeches
Outcome desired
To learn the major
communication theories in
public speaking, interpersonal
communication, selfcommunication, intercultural
communication, and group
communication. Students will
be able to incorporate these
theories into their own
speaking styles
Outcome desired
To work towards
understanding and
overcoming
communication
apprehension
To develop skills in diverse
communication contexts
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 1
skills effectively for academic
research and lifelong learning
including small groups,
computer-mediated
communities and
professional communities
5. integrate knowledge and
skills in their program of
study
To understand the
overwhelming importance
of effective
communication in all
aspects of academic,
professional, and
everyday life
6. differentiate and make
informed decisions about
issues based on multiple
value systems
7. work collaboratively in
diverse groups directed at
accomplishing learning
objectives
8. use historical or social
sciences perspectives to
examine formation of ideas,
human behavior, social
institutions, or social
processes
9. employ concepts and
methods of the natural and
physical sciences to make
informed judgments
10. apply aesthetic and
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 2
intellectual criteria in the
evaluation or creation of
works in the humanities or
the arts
Describe the assessment activity and the (student learning outcome(s) it addresses ) that occurred in your course.
Assignment: In order to measure the impact that in class listening activities may have on the overall students’ listening skills and whether
these activities might results in an overall improvements in the students’ listening skills we divided students in two groups. Students’ in the
experimental group completed four in class listening activities that were very similar in format and content to the listening assessment
described here below. The students in the control group did not performed any in class listening activity.
For the assessment all students watched twice Lucas’s video “Medical Robots: From Science Fiction to Science Fantasy” (Needs
Improvement)
This speech is substandard, i.e., it does not have all the required speech components. Students used the speech evaluation form included
here to evaluate this speech indicating which speech components were present and which are missing, indicating how many topics and
transitions the speaker introduces and how many sources are cited. Moreover, the students answered three listening and comprehension
questions about the content of the each speech. The form used is intended to facilitate student critical listening by identifying missing parts
and by processing and understanding what was heard.
This listening assignment is designed primarily with the intention to test student listening skills as far as their ability to identify relevant
speech outline components (QUESTIONS 1-12) and show understanding of the content of the speeches (QUESTIONS 13-15). Because the
goal is to track student’s listening, each student should repeat this exercise twice during the semester.
Instructions: In order to standardize the assessment procedures three designated test administrators will administer the listening test for all
the faculty twice during the semester. The first time the test should be administered after the instructor has taught the class how to create a
speech outline for their first informative (or persuasive) speech approximately between the 5th and the 6th week of class. The second time
the test should be administered after students have been instructed on how to create the speech outline for their second persuasive (or
informative) speech or preferably, after they have delivered their second speech between the 12th and the 14th week of class.
All instructors need to follow the speech outline format indicated in your speech textbook or conform to it while instructing students on how
to create an informative or persuasive speech outline. Students will first complete the attached scoring SPEECH EVALUATION form .
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 3
List the data collection instrument (s) used for assessment.
Speech evaluation forms used by students for the assessment are given below. Each student will complete two forms one in connection with
the first speech and the second one with the second speech.
SPEECH EVALUATION FORM
Speech Title: _____________________________________________________________________________
Listen carefully indicating which speech components are present in this speech and rating the speaker on each point. Remember that your goal
is to identify which speech components are missing from the speaker’s speech.
INTRODUCTION
NO
YES
1. Attention getter


2. Reason to listen


3. Reveal Topic


4. Established credibility


5. Preview body of speech


BODY
0
1
2
>2
6. How many main ideas




7. How many sources are properly cited 1



8. How many transitions




1
Please remember that for a source to be cited properly, you need to hear one of the following combinations of information: (1) author and year of publication; (2) author and title of article/ book; or
(3) author, title of article/book and year of publication. Mentioning just the author or a title does not count as a proper citation.
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 4
CONCLUSION
NO
YES
9. Cues the audience


10. Review of main points


11. Restate thesis


12. Vivid ending/Closure


COMPREHENSION
13. Orderly 2 robots can do the following:
A. let doctors see patients from a remote location
B. communicate with each other
C. allow a doctor to see a patient’s vital signs
D. none of the above
14. Robots called ‘Da Vinci’ 3 are used for:
A. delivering items in a hospital
B. allowing patients to create art
C. helping hospital staff accomplishing routine tasks
D. surgery
2
3
Definition of ‘orderly’: an attendant in a hospital responsible for the nonmedical care of patients and the maintenance of order and cleanliness.
Leonardo Da Vinci was an Italian Renaissance polymath: painter, sculptor, architect, musician, mathematician, engineer, inventor, anatomist, geologist, cartographer, botanist, and writer.
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 5
15. The speaker told a story about a woman in New Jersey who:
A. had heart surgery
B. was operated on by a robot
C. had an orderly robot
D. invented a medical robot
Provide an analysis (and summary) of the assessment results that were obtained ( in addition, see also statistical addendum)
Overall 157 students were assessed in Fall 2015. I report in Table 1 “Overall students’ performance questions 1-12” and Table 2 “Overall
students’ performance questions 13-15” the overall and group specific raw and percentage scores of correct answers for both the Pre-and the
Post-assessment. The control group had 110 subjects, while the experimental group had 47 subjects. Students in the experiment group
completed four listening activities between the pre and the post assessment; whereas students in the control group did not complete any
activity. We tested the hypothesis of whether increasing the number of listening activities in a semester had a positive effect on students’
listening skills. I will present two analyses, one for each of the two groups and then I will summary the two analyses generalizing the results to
all the students assessed this semester.
DATA:
TABLE 1 Overall students’ performance questions 1-12
LISTENING_1_PRE
% CORRECT_PRE_1
LISTENING _2_POST
% CORRECT_POST_2
% CHANGE
Attention Getter
92
59%
100
64%
9%
Reason to Listen
83
53%
89
57%
7%
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 6
Reveal the topic
141
90%
133
85%
-6%
57
36%
51
32%
-11%
103
66%
115
73%
12%
How many main points
41
26%
57
36%
39%
How many properly cited sources
84
54%
70
45%
-17%
How many transitions
52
33%
65
41%
25%
Cues to the audience
86
55%
90
57%
5%
108
69%
122
78%
13%
54
34%
66
42%
22%
Closure
130
83%
139
89%
7%
TOTAL
1031
86%
1097
91%
6.4%
Credibility Statement
Preview
Review
Restate the topic
We performed a one way ANOVA to see whether there was a significant difference between the overall results for the assessments taken at
Time 1 and at Time 2, The ANOVA resulted as no significant, indicating than overall there was no significant change on the number of correct
answers between the pre and the post.
TABLE 2: Overall students’ performance questions 13-15
LIST_PRE Assessment
% correct answers
88
116
56%
74%
COMPREHENSION 1
COMPREHENSION 2
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
LIST_POST Assessment % correct answers
91
132
58%
84%
% Change
3.4%
13.8%
Page 7
COMPREHENSION 3
120
324
TOTAL
76%
108
121
334
77%
114
0.83%
6%
We performed a one way ANOVA to see whether there was a significant difference between the overall results for the assessments taken at
Time 1 and at Time 2, The ANOVA resulted as no significant, indicating than overall there was no significant change on the number of correct
answers between the pre and the post.
QUESTIONS 1-12: CRITICAL LISTENING_CONTROL GROUP
The results for number of correct answers for each questions (raw scores), percentages and percentage change are summarized in the following
table. The control group is made of 110 students.
TABLE 3: Control Group Questions 1-12
LISTENING_1_PRE
LISTENING _2_POST
% CORRECT_PRE_1
% CORRECT_POST_2
% CHANGE
Attention Getter
62
66
56%
60%
6%
Reason to Listen
60
56
55%
51%
-7%
Reveal the topic
101
95
92%
86%
-6%
Credibility Statement
44
47
40%
43%
7%
Preview
72
73
65%
66%
1%
How many main points
29
51
26%
46%
76%
How many properly cited sources
57
46
52%
42%
-19%
How many transitions
35
35
32%
32%
0%
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 8
Cues to the audience
59
66
54%
60%
12%
Review
75
83
68%
75%
11%
Restate the topic
36
42
33%
38%
17%
Closure
88
94
80%
85%
7%
718
54%
754
57%
5%
TOTAL
We performed a one way ANOVA to see whether there was a significant difference between the overall results for the assessments taken at
Time 1 and at Time 2, the ANOVA resulted as no significant, indicating than overall there was no significant change on the number of correct
answers between the pre and the post. We also performed a repeated ANOVA we found that though there was no overall change in the way
students answered the 12 questions, there was a significant difference for certain questions, (F=35.5, p.05). Looking at the positive direction of
change indicated by the mean values, students improved their critical listening skills only for few of the 12 constructs.
QUESTIONS 13-15: COMPREHENSION_CONTROL GROUP
Questions 13 through 15 regards students’ listening comprehension abilities. The results for number of correct answers, means and percentages
are summarized in table.
TABLE 4: Control Group Questions 13-15
LIST_PRE
% correct answers
LIST_POST
% correct answers
% Change
62
74
85
56%
67%
77%
63
92
82
57%
85%
2%
24%
-4%
COMPREHENSION 1
COMPREHENSION 2
COMPREHENSION 3
75%
TOTAL
221
67%
237
72%
7%
We also performed a one way ANOVA to see whether there was a significant difference between the overall results for the assessments taken at
Time 1 and at Time 2, The ANOVA resulted as no significant, indicating that overall there was no significant change on the number of correct
answers between the pre and the post.
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 9
QUESTIONS 1-12: CRITICAL LISTENING_EXPERIMENTAL GROUP
The results for number of correct answers for each questions (raw scores), percentages and percentage change are summarized in the following
table. The experimental group is made of 47 students.
TABLE 5: Experimental Group Questions 1-12
PRE_LISTENING_1
% CORRECT ANSWERS
POST_LISTENING_2
% CORRECT_ANSWERS
%
CHANGE
Attention Getter
30
64%
34
72%
13%
Reason to Listen
23
49%
33
70%
43%
Reveal the topic
40
85%
38
81%
-5%
Credibility Statement
13
28%
4
9%
-69%
Preview
31
66%
42
89%
35%
How many main points
12
26%
6
13%
-50%
How many properly cited sources
27
57%
24
51%
-11%
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 10
How many transitions
17
36%
30
64%
76%
Cues to the audience
27
57%
24
51%
-11%
Review
33
70%
39
83%
18%
Restate the topic
18
38%
24
51%
33%
Closure
42
89%
45
96%
7%
313
55%
343
61%
11%
TOTAL
We performed a one way ANOVA to see whether there was a significant difference on how students answered the 12 questions between the pre
and the post. The ANOVA resulted significant, (F= .057, p.05). Looking at the positive direction of change indicated by the mean values, students
improved their critical listening skills only for a few of the 12 constructs.
QUESTIONS 13-15: COMPREHENSION_EXPERIMENTAL GROUP
Questions 13 through 15 regards students’ listening comprehension abilities. The results for number of correct answers, means and percentages
are summarized in table.
TABLE 6: Experimental Group Questions 13-15
LIST_PRE
% correct answers
LIST_POST
% correct answers
% Change
26
42
55%
89%
28
40
60%
85%
8%
-5%
35
103
74%
34%
39
107
83%
36%
11%
4%
COMPREHENSION 1
COMPREHENSION 2
COMPREHENSION 3
TOTAL
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 11
We performed a one way ANOVA to see whether there was a significant difference between the overall results for the assessments taken at
Time 1 and at Time 2, The ANOVA resulted as no significant, indicating than overall there was no significant change on the number of correct
answers between the pre and the post.
Describe how the assessment results that were obtained affected (or did not affect) the student learning outcomes you identified. As part of
your discussion, describe any plans you have to address the areas where students need to improve.
In this section we first look at the at the overall results of Group 1 and Group 2, then we will compare the results of between the control and the
experimental group.
Overall Students were tested on the individuation of main speech components here listed as ‘intro’, ‘body’, ‘conclusion’ and ‘comprehension’ of
the speech twice.
Question 1-12 measured students' ability to recognize parts of speech, a concept reinforced by the syllabus of SP 211. On the ability to recognize
parts of a speech the Anova results suggest that students overall did not improve between the pre and the post assessment. Looking at the
percentage change for each of the 12 components students students showed improvement on 9 areas, i.e., (#1) attention getter (+9%); (#2)
reason to listen (+7), (#5) preview (+12%), (#6) main ideas (39%),(#8) transitions (+25%); (#9) cue to the audience (5%); (#10) Review (13%);
(#11) restate topic (22%) and (#12) closure (7%). There was instead deterioration in (#3) reveal the topic (-6%); (#4) credibility (-11%), (#7)
number of cited sources (-17%)
We believe that students’ moderate improvement in these areas is due to the students’ inability to listen critically to the information and on the
way data were collected. As far as the data collection, we had to eliminate many students’ assessment sheets because they were incomplete.
Though we asked students to check that they had answered all the questions, they still returned incomplete assessment sheets.
We believe that the missing answers were probably due to the fact that the students did not know all the answers. Next year we will add a “I do
not know” column next to the yes/no ones to see if students will check this column rather than simply leaving the answers blank.
Overall Fall 2015 students’ cohort minimally improved their abilities to identify speech components showing a 6.4% gain score from the first to
the second assessment. Fall 2015 students improved moderately with respect to Fall 2014 students: from 5.2% (Fall 14) to a 6.4% gain score.
Students also improved in 9 out of 12 possible areas versus the 7 areas improved in Fall 2014.
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 12
However, as the following graph indicates, trends of improvement and deterioration vary from one cohort to the next. This suggests that our
results might be determined by the type of students attending our courses. Next year we will plan to teach two honors sections of SP211 and
we will compare our listening results between the honors and non honors sections to see if whether students’ GPA has an effect on the results.
We will also try to collect information about how many of our students are taking remedial courses. We will correlate students’ academic status
(honors vs. non-honors, remedial vs non remedial) to the listening assessment results to see whether the students’ academic status has an effect
on the students’ critical listening skills.
Percentage
Comparison between Fall 2014 and Fall 2015
60
50
40
30
20
10
0
-10
-20
-30
AG
FALL 2014 -15
FALL 2015 9
RL
18
7
RT
8
-6
Cr
39
-11
PR
-18
12
MI
6
39
CS
0
-17
Tr
-4
25
Cue
0
5
Re
-3
13
RT
13
22
Cl
53
7
AG=Attention Getter; RL=Reason to Listen; RT=Reveal Topic; Cr= Credibility; PR=Preview; MI= number of Main Ideas/Points, CS= cited sources;
Tr=number of transitions; Cue=Cues to the audience; Re=Review; RT= Restate Topic; Cl=Closure
Question 13-15 measured the student's ability to extract content from a spoken message. Overall Students comprehension did not improve
significantly, (6%). We registered a modest improvement for questions #13(3.4%) and #14 (13.8%) only.
If we compare Fall 15 students’ results with Spring 14, Fall 13 and Spring13 students’ results, students of Fall 15 improved with respect to
students of Spring 13, Fall 13 and Spring 14, but they did not do well as students from Fall 14.
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 13
Table 7 : Summary of results for questions 13-15 across different semesters
Q13
Q14
Q15
Spring13
-10%
-23%
-17%
Fall13
-11%
-14%
1%
Spring14
-3.8%
5.1%
16.4%
Fall 14
59%
81%
71%
Fall 15
3.4%
13.8%
.08%
Control and experimental data
As far as the comparison between the control and experimental data, the fact that many records were not used because they were incomplete
reduced the number of students in the experimental group to 47 from the original 99. However, the data showed that students improved only
in few areas, though their improvement was greater than the improvement obtained by the students in the control group. Therefore, we
conclude that the addition of four listening activities had a positive effect on students’ overall listening skills.
Action plan:
On ability to recognize parts of the speech, we believe faculty should give more emphasis to the concept of topic identification, main ideas
identification, credibility statement and citing sources. But most importantly faculty should include several listening activities in their curriculum.
Asking, for instance, students to summarize, analyze and report on content from spoken language.
Next semester we will continue to standardize the procedure of the data collection by asking faculty to complete their listening assessments
around the same time: Week 5 for the Pre-Assessment and week 13 for the Post- Assessment. Moreover, we will suggest faculty members to
add the number of in class listening exercises in order to help students to increase their listening skills.
Finally, given the inconsistency of errors across different cohorts of students, we plan to teach two honors sections of SP211. We will compare
honors’ and non honors students’ listening results. We will also try to collect information about how many of our students are taking remedial
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 14
courses. We will correlate students’ academic status (honors vs. non-honors, remedial vs non remedial) to the listening assessment results to
see whether the students’ academic status might have an effect on the students’ critical listening skills.
Dr. Franca Ferrari-Bridgers - 03/15/2016
Speech 211
Page 15