AN INTELLIGENT ASSISTANT FOR NAVIGATION
OF VISUALLY IMPAIRED PEOPLE
N.G. Bourbakis*# and D. Kavraki #
#AIIS Inc., Vestal, NY, email: aiis@stny.rr.com
*WSU, ITRI, Image-Video-Machine Vision Research Lab, Dayton OH.
Outline
•
•
•
•
•
•
•
•
Introduction
Tyflos System
Human Tyflos Interaction
Visual to Audio Communication
Detection of Other Moving Objects
Navigation with L-G Landmarks
IMPLEMENTATION OF THE TYFLOS PROTOTYPE
CONCLUSIONS
Introduction
• This paper presents the navigation methodology employed by an
intelligent assistant (agent) for people with disabilities partial
independence.
• Tyflos
– Work in a 3-D dynamic environment
– System carries two vision cameras and captures images from
the surrounding 3-D environment.
– These images into verbal descriptions for each image into a
verbal communication.
Introduction
• In USA
– there are more than 3 millions with low vision and millions more
completely blind.
• A great challenge
– robotic-like vision
• Be able to obtain 3-D representations of the surrounding
world.
• provide verbal descriptions of it
Introduction
• Of particular importance
–
–
–
–
Unknown environment
High resolution
Analyze
Events based on simple queries
– Vision system convert visual descriptions into verbal communication
Introduction
• Applied artificial intelligence and software technologies
– QBIC
• Query By Image Content
– EUROTRA
• Machine languages translation
– STARS
• Newsletter
– MEL
• An autonomous intelligent vehicle
– Intelligent walking robots, 3-D space maps generation, autonomous vision
systems, neuromorphic vision chips, the robot with feelings
Introduction
• verbally see
– an intelligent system to make visually impaired people via a
machine vision system equipped with speech and natural
language understanding technologies.
• challenges and technological
– the navigation methodology of an intelligent assistant for visually
impaired people's partial independence.
TYFLOS SYSTEM
• System's Main Components
–
–
–
–
–
–
–
The
The
The
The
The
The
The
Pair of Glasses
Vision Cameras
Laser Range Scanner
Ear Speaker
Microphone
Portable Computer
Communication System
TYFLOS SYSTEM
• The Pair of Glasses
– The pair of glasses is a important part of the Tyflos system, and
several system’s parts are mounted on it.
TYFLOS SYSTEM
• The Vision Cameras
– The vision camera is mounted on the top of the glasses in order
to provide a better view of the targeted scene.
• Camera
– High resolution
– Color
– Small in size and weight.
TYFLOS SYSTEM
• The Laser Range Scanner
– "scan“
• the same view with the camera.
– For determination of the distances of the existing objects
– For 3-D view of a captured image
• The size of the laser scanner
– resolution at least 1cm radius in a distance of 15 meters.
TYFLOS SYSTEM
• The Ear Speaker
– high fidelity component
• has only one ear speaker through it the user receives the
detailed description of the surrounding environment.
TYFLOS SYSTEM
• The Microphone
– User communicates with computer and the
appropriate Databases.
– Filter
• To eliminate the noise from the user's environment.
TYFLOS SYSTEM
• The Portable Computer
– It hosts all the important software tools that make the Tyflos
system functional.
– DBs
• Image
• Speech
• Natural Language
– Software Interfaces
– Knowledge Conversion Tools
– Communication Tools
TYFLOS SYSTEM
• The Communication System
– the user needs to establish communication with other
resource
• portable or mobile information DBs
TYFLOS SYSTEM
Source:http://tweakers.net/nieuws/31815/computer-met-ogen-begeleidt-slechtzienden.html
HUMAN TYFLOS INTERACTION
• Tyflos is able to assist the user are：
–
–
–
–
where am I ?
describe the surrounding scene.
guide me to a particular point
where is a particular object?
HUMAN TYFLOS INTERACTION
• Independent
– Human assistant by describing to the user the 3-D visual
environment making him/her more independent.
• find an “object” → microphone → scanning the surrounding
space → vision camera → analyzed and recognized → ear
speaker
• What Object
– describes to the user via the ear speaker what object made a
certain sound.
Visual to Audio Communication
• Where am I ?
– GPS (Global Positioning System)
• Tyflos
– uses a Speech synthesizer
– provides to the user the names of the streets and / or the names
of certain buildings.
– provides to the human user a verbal description of the
surrounding 3-D scene.
Visual to Audio Communication
• Describe the Surrounding Scene
– SPN graph
• Generation and Description of a 3-D Surrounding Unknown
Environment
• Fusion of Images & range Data for 3-D Modeling of the Free
Space
Visual to Audio Communication
• Generation and Description of a 3-D Surrounding
Unknown Environment
– stereo speaker
– vision camera
– laser range data
• Dr.Albus and Dr. Bourbakis
– develop 3-D images via audio signals generated by two
speakers during the motion of a sound generation equipment.
Visual to Audio Communication
• Fusion of Images & range Data for 3-D Modeling of
the Free Space
– Cameras Stabilization Issues
• synthesis of images
– Generation & Representation of the Navigation Space
• Free Navigation Space (FNS)
–
–
–
–
orientation-angle
magnetic north
current position
inclination angle
• shape SH
– ” open-navigation corridors ”
Visual to Audio Communication
• Where is a Particular object?
–
–
–
–
particular object is converted into a hierarchical SPN graph
the SPN graph is compared to the system's KB
recognized or appropriately classified
speech DB and ear speakers
Visual to Audio Communication
• Guide me in a Particular Place
– Collision Free Path Planning
• the system attempts to select the First Best Choice (FBC) open
navigation corridors.
– Detection of Other Moving Objects
•
•
•
•
How Tyflos detects other moving objects in the same navigation space
What is the perceived size of the moving object
What is the direction and the velocity of the moving object
How many different moving objects are in the same free space
Detection of Other Moving Objects
• How Tyflos detects other moving objects in the same
navigation space
– the space shape changing methodology
• shape SH & Dt
• do(Dt) = VRa*Dt → P'
• P' → SH'
– IF SH" = SH' THEN there is no other moving object in the same navigation space
– ELSE there is at least one moving object in the same free navigation space.
– the differences detected from the comparison of two consecutive
2-D images.
• ”motion”
Detection of Other Moving Objects
• What is the perceived size of the moving object
– When Tyflos, detects another moving object, Rb, in the same
navigation area, then it attempts to define the size (dimensions)
"perceived" from its current position P.
• Rb = |dk'(t(i+1)) - dk(t(i+1))|
– Corresponding window
• 2-D image
• Knowledge Base
Navigation with L-G Landmarks
• L-G Landmarks
– local-global graphs
• Knowledge Base (KB)
– successful and flexible generation of Space Maps is the development of
a knowledge base able to store, process and access the Space Maps'
forms.
• Communication and Mobile DBs Issues
– to carry CDs with different DBs
– its computer to be connected with a multimedia network in order to
download the appropriate DB
IMPLEMENTATION OF THE TYFLOS
PROTOTYPE
• Integration of AI Methodologies
– The actual implementation of the Tyflos system combines the
state-of-the-art in AI methodologies.
• The Prototype
– AIIS (American Institute of Indian Studies)
– Tyflos system consists of the pair of glasses, a Hitachi high
resolution, color vision camera, a portable computer Mc booknote, an ear speaker, a voice synthesizer and a microphone.
CONCLUSIONS
• It will provide the ability to the user to understand better
the surrounding environment, and give more
independence in his/her motion.
– Vision
– Speech
• However, a significant progress has been done, and new
methods in both vision and speech developed by other
researchers will be appropriately used.
Thank you for your attention