Is Automatic Speech Recognition Ready
for Direct Use by Classroom Teachers?
PEPNet 2004 - Presentation
Pittsburgh, PA, Sheraton Station Square, April 24, 2004, 10:15 –
11:30 AM
Presenter/Author: Kathleen Eilers Crandall, Ph.D.
Contributors: Donna E. Gustina, and Stephen S. Campbell
National Technical Institute for the Deaf
Rochester Institute of Technology
The Glossograph
• Fay wrote about an experimental
mechanical device used to transcribe
human speech, and said,
• “… it is not unreasonable to hope that
some instrument will yet be contrived …“
Fay, E.A. (1883). The glossograph. American Annals of the Deaf, 28,
67-69.
Sci-Fi or Reality?
"The pen was an archaic instrument,
seldom used even for signatures...Apart
from very short notes, it was usual to
dictate everything into the speak-write…”
(Nineteen eighty-four. Orwell, 1949)
Project
• Direct teacher use of Continuous
Automatic Speech Recognition:
– English Classroom/Lab
Funded by a grant from the Parsons Foundation of California
English Classroom/Lab
Project
English Classroom/Lab Project
Purpose
Investigate direct use of ASR by classroom
teacher to learn:
• Is acceptable recognition level attained?
• Under what conditions?
– Style of speaking
– Communication mode
– Language complexity
Related Work
Use of ASR by an intermediary
• Intermediary, a ‘captionist,’ re-speaks
professor’s words into a computer
• Intermediary summarizes professor’s
words into a computer (‘interpreted
speech’)
• Intermediary may use C-print (a shorthand
typing system) in combination with ASR
http://cprint.rit.edu/
Related Work
Use of ASR by the primary speaker
• iCommunicator™
http://www.myicommunicator.com/product
_info.html
• Liberated Learning Environment
http://www.liberatedlearning.com (St.
Mary’s University, Halifax, Nova Scotia)
English Classroom/Lab
Project
English Classroom/Lab Project
Teacher -- Students
• Teacher -- Speaker
– Native speaker of American English
– User of ASL as a second language
– Trained the ASR equipment
• Students -- Readers
– Young adult college students who are deaf or hard-ofhearing
– Reading and writing skills at the lowest quartile of
entering students
– Enrolled in basic level English language reading and
writing courses
English Classroom/Lab Project
Evaluation Procedures
• ASR Software:
– Dragon Naturally Speaking
– IBM ViaVoice
– Microsoft Office
• Speaking styles:
– Spontaneous conversation
– Dictation-like speech
• Communication modes:
– Speaking
– Simultaneously speaking and signing
English Classroom/Lab
Teacher station
Control system
Smart Board & LCD Projector
Student Stations
English Classroom/Lab Project
Accuracy Needs
• Vary by population and message predictability
– New vs. Known information
– Fluent readers vs.
Language learners
– Reading for pleasure vs. Reading to master new
information
• CLOZE research and prediction of missing
information
English Classroom/Lab Project
Results: ASR Software
100%
95%
90%
Conversation
Dictation
85%
80%
75%
Drag on
V iaV oice
XP
English Classroom/Lab Project
Results: Communication Mode
98%
96%
94%
92%
90%
Conversation
Dictation
88%
86%
84%
82%
80%
S im u ltan eou s Com m m u n ication
S p eech O n ly
English Classroom/Lab Project
Results: Language Complexity
98%
96%
94%
92%
Conversation
90%
Dictation
88%
86%
84%
82%
< 7th G rad e
> 7th G rad e
English Classroom/Lab Project
Correcting Text
• Error correction
– What to correct
– When to correct
– How to correct
Multitasking Demands
• Normal tasks for speaker/teacher
– Formulating ideas relevant to topic
– Attending to learning needs of students
– Meeting lipreading and sign language needs
• Added tasks for speaker/teacher
– Speaking to produce readable ASR text
– Monitoring text
– Making corrections
Recommendations
Discussion
Questions
Grammatical Correctness
• Is ASR accuracy affected by the
grammatical correctness of the user’s
speech?
• Student written responses spoken as
written: Accuracy – 93.8%
• Student written responses spoken after
corrected: Accuracy - 94.3%
Style of Speaking
1. Style of speaking that more closely
resembles dictation approaches a usable
accuracy rate.
2. Lowering the complexity does not
improve accuracy.
Conditions of Use
Direct use of ASR by a language teacher -Useful only under very controlled conditions.
• Illustrating the generation of written
language
• Demonstrating the use of notes and
outlines to produce written text
• Translating selected sign language
utterances into English text during
discussions
ASR: Classroom Use
Prepared Outline
Teacher’s Screen
Student’s Screen
Considerations
• Training
– Critical to reach over 90% accuracy
– Training with conversation
• Corrections
– Familiarity with strategies
– Dictate, Spell, Right click
• Equipment
– Microphone headsets - design, comfort, and size
– Demand on computer processor
– Effect of optional settings
Tips for Better Accuracy
•
•
•
•
•
•
•
Powerful computer
No other programs running
Consistent microphone placement
Environment
Training
Profile
Join user groups, such as msspeech@yahoogroups.com
Language Processing
Teaching/Learning Issues:
• Does ASR promote the learning of reading
and writing for Deaf and Hard-of-Hearing
students?
• How do students process this information?
• Do students attend to multiple inputs?
• Can teachers attend to this many tasks
effectively?
Presenter – Contact Information
Kathleen Eilers-crandall, Ph.D.
Department of English
National Technical Institute for the Deaf
Rochester Institute of Technology
Lyndon Baines Johnson Building - 2264
Phone: (585) 475-5111
Fax: (585) 475-6500
Email: kecncp@rit.edu
Web: http://www.rit.edu/~kecncp