Mark Simson, Chris Weeden, AJ Stohr, Bhargav Shivkumar, Balu Swarna
Making a Kinection
CSE 442 Phase 1
Introduction
Languages have helped mankind to communicate effectively over centuries. Though oral
communication constitutes a major portion of communication, body language helps to emphasize
and enhance the expression and thus enriches the experience. But what if body language is your
only means of communication?
In order for the deaf, hard of hearing, or speech impaired people to communicate via
telephone with hearing people, they must use a Video Relay Service (VRS.) A VRS is a service
which involves relaying through a sign language interpreter, all communications to and from a
deaf person. These services are useful but are also expensive to taxpayers (in the United States)
because a certified sign language interpreter must be employed for each simultaneous call.
An alternative form of communication between deaf and hearing people is a TTY or
Teletypewriter. This is a tool that a deaf person can use to type out messages to a hearing
person, and an operator can read their messages. The operator then transcribes into text the
hearing person’s response. This form of communication is not ideal either as it requires the deaf
person to communicate in English, which is not the primary language of most deaf individuals.
Proposal/Problem Statement
Imagine a Kinect-based sign language recognition system, which could interpret standard
sign language into English text, and perhaps text then to speech. This would be a multi-use
application for the deaf, to include many forms of sign language and gesturing.
Targeted Demographic
There are many types of users for this system. First and foremost would be individuals
who are deaf, hard of hearing, or in some other way speech impaired. Second, the system also
helps anyone who wishes to communicate with someone when they don’t understand sign
language themselves. For instance, this could be a teacher trying to teach hearing impaired
students, or a doctor needing to treat a patient.
Usefulness
The system would improve the users’ quality of life by allowing them to communicate
with people who can hear, using sign language. It also allows other parties to communicate back
with the hearing impaired even when they may not know sign language. Interactions that would
be cumbersome and would require a human translator would be much easier. The next time a
hearing impaired person walks into a doctor’s office, the doctor will turn on their own system,
and the patient will be able to communicate freely and smoothly with the doctor.
Requirements
The main requirement is to build a Kinect based system that can convert the signs of ASL
to English text or speech and vice versa.


The system focuses the signs, speech, and facial expressions the user makes
A database of all the signs of American Sign Language




A user interface that shows the resultant text after processing the signs and expressions
In order to use the system, a laptop, and kinect system are needed
The hearing/speech impaired person will need to understand ASL, which is an
assumption the system makes
An interface that recognizes speech and converts it to sign language, or text