AEI Conference 2012
ACCESSIBLE CHESSBOARD FOR BLIND THAT IS MOUSE AND KEYBOARD FREE
MSc. Pavel Jetensky
Faculty of Electrical Engineering and Informatics,
University of Pardubice Studentská 95, 532 10 Pardubice, Czech Republic, tel. +420/777 045 366, e-mail: st33370@student.upce.cz
ABSTRACT
This paper describes a design of an interactive chessboard that enables blind players to play comfortably against the computer.
It consists of a physical wooden chess game set, depth camera, visible light video camera and a computer with processing software.
Each player’s chess move is detected and transferred to the computer. An opponent’s move is spoken aloud using a speech engine so
that the player can adjust his chessboard.
Keywords: chess; computer vision; pattern matching; accessibility; depth camera; computer human interface; chessboard; visual
impairment; blindness; blind; user interface; low vision
1. INTRODUCTION
Statistics from 2011 show that in developed countries
there are 3.9 million blind people and 24.6 million people
who have low vision [1]. A lot of them are either casual or
professional chess players. When playing against real
opponents with a real chessboard, they can touch the
chessboard and get the necessary tactile feedback from the
chess pieces. There is no comfortable way for them to
play against a computer or distant human opponent over
the internet.
Existing special chess software already contains an
audio-move announcement feature. It is an audible
feedback of an opponent’s move [2]. The comfort of play
with just this feature is not sufficient. No software yet has
a natural user interface allowing players to easily input
their own moves.
The most intuitive way of playing with a computer
would be to allow a VI (visually impaired) player to move
physical chess pieces on real a chessboard. The necessary
process would involve detection of the move using some
of the techniques of computer vision. The task at hand is
to recognize what chess pieces are on the chessboard and
how their position evolves over time.
A simple approach of utilizing a video camera placed
above the table may not be sufficient. Results from
analyzing a camera picture can be poor, with quality
highly dependent on lighting conditions. Such a detection
algorithm may fail to properly recognize chess figures.
From the top, figures look very similar – f.e. it is difficult
to distinguish a black king from a black queen if they are
on a black square.
This paper introduces a design of an interactive
accessible chessboard that allows tracking of player’s
moves. Working prototype of chess software for the VI
has been created. The design of its accessible user
interface is described in section II. The basic hardware
consisting of video and depth camera, computer and chess
set is described in section III, together with used software
components. Computer vision algorithms that translate
raw input data to player’s moves are explained in section
IV.
2. SUBJECT
2.1. DESIGN OF ACCESSIBLE USER INTERFACE
OF CHESSBOARD SOFTWARE
The primary purpose of this aid is to serve visually
impaired people. The standard tools for user interaction
normally used by people without VI could not be used.
Specific approaches have been developed both for chess
game player’s input and output.
2.1.1. Player’s input - replacing computer mouse
As a user cannot drag and drop a chess piece on the
screen, a special hardware and software solution has been
created to detect movement of the real chess piece on the
real chessboard. This involves analyzing data from depth
and RGB cameras and is described in section IV.
2.1.2. Chess game output – text to speech engine
A visually impaired player cannot see an opponent’s
move, so a text to speech engine has been used to
pronounce text representation of the move. During
application testing it turned out that longer sentences with
some information redundancy are easier to understand
than shorter versions. “Tab. 1” shows shorter and longer
versions of sample output sentences:
TABLE I.
SAMPLE AUDIBLE FEEDBACKS
Short Version
(Chess Notation)
Longer Version
(Human Friendly)
Probability that user will
understand the meaning is
low
Probability that user will
understand the meaning is
high
e2-e4
white pawn from square e2
goes to e4
e4xf5
white pawn from square e4
captures black pawn at
square f5
Be4xf5
white bishop from square e4
captures black pawn at
square f5
Ng1-f3
white knight from square g1
goes to f3
2.1.3. User interface for promoting a pawn
When a player advances with a pawn to the eighth rank
and promotes it, the user interface must enable her to
choose a piece for the pawn’s transformation. As VI
ISBN XXXX-XXXX © 2012
Accessible chessboard for blind that is mouse and keyboard free
players cannot use a mouse, an audible hint is pronounced,
guiding the player to press key 1-4 on the numeric
keyboard which chooses the desired piece.
3. METHODS
3.1. HARDWARE AND SOFTWARE SETUP
This section describes architecture of whole system,
consisting of both hardware and software components.
3.1.1. Hardware setup
A Microsoft Kinect depth camera is placed on a stand 73
cm above the physical wooden chessboard. It is connected
through a USB cable to a processing laptop. ThinkPad
R61 with 3GB of RAM memory and dual 2 GHz CPU
core was used for the working prototype.
3.1.2. Software setup
The whole solution consists of several software layers,
starting from layers working with physical hardware
devices up to the software part written in high
programming language. “Fig. 1” shows all the layers:
1) Low
HW
abstraction
layer
Purpose of this layer is to retrieve raw data from
depth and RGB cameras and pass them to
components in higher layers for processing. There is a
dedicated thread that is providing scene frames at a
rate
of
30
Hz.
The following components are used to deliver this
functionality:
a) PrimeSense Kinect drivers – communicate with
physical Kinect device through USB port.
b) OpenNI and NITE libraries – compose depth
bitmap representing every scanned frame.
c) Avin Kinect Mod – adapter specific to Microsoft
Kinect depth sensor that is plugged into generic
OpenNI processing framework.
2) Computer vision utility library – serves as main
component for analyzing 2D images created from 3D
information about the scene. It is used to calculate
blob contours that represent the position of chess
pieces on the chessboard.
a) OpenCV C++ library – open source computer
vision library with C++ interface.
b) EmguCV C# Wrapper – library wrapping
functionality of OpenCV C++ library, so that it is
accessible from managed C# code.
3) TouchTable library – this component is responsible
for extracting data coming from the HW abstraction
layer into depth image and extracting information
about blobs using the OpenCV library. Detected blobs
are passed to the KinectChess application.
4) KinectChess application analyzes flow of the detected
blobs in the flow of the frames coming from the
TouchTable library. Based on this data it detects
moves of the chess pieces performed by the chess
player.
Fig. 1 Architecture of KinectChess software
components
3.2. ALGORITHM FOR DETECTING PLAYER’S
MOVE
The algorithm is based on the analysis of the depth frame.
Depth frame is a map of pixels with resolution 640x480,
where each pixel carries information about the distance of
the pixel from a depth camera.
3.2.1. Main algorithm description
Move detection proceeds in following steps:
1) 3D blobs are detected in the frame, representing chess
pieces on the chessboard.
2) Detected blobs are associated with the chess squares.
3) Retrieved information is compared to the game
engine to verify consistency of the chessboard setup
with the chess game model.
4) Chess piece move is detected by comparing position
of the chess pieces between successive frames from
depth camera.
3.2.2. Algorithm for detection of blobs representing
chess pieces
Originally I created a simple algorithm for the blobs
detection, using a flood fill linear algorithm. The problem
with this approach was its performance – it took more
than five seconds to analyze one frame. This was
insufficient for the purpose. I have improved it by
replacing this algorithm by utilizing 2D image processing
algorithms from OpenCV library. The The improved
blobs detection now works in following way:
1)
Based on the data from the depth sensor, a grey
OpenCV image is created [3] with grey pixels
representing areas that are standing out above the
tabletop.
2)
Dilate filter is applied to the image to decrease
the amount of noise [4]
3)
Find contours function is used to extract blob
contours from the blobs of neighboring pixels
ISBN XXXX-XXXX © 2012
Acta Electrotechnica et Informatica Vol. Y, No. X, 2008
4)
Detected contours’ centers are matched against
the calibrated position of the chessboard to
associate each blob of the chess piece with a
particular chess square.
3.2.3. Algorithm for associating detected blobs to
chess squares
3.2.3.1. Simple and naïve algorithm based on region of
interest with piece overlapping problem
Firstly, a simple algorithm was used to associate detected
blobs to chess squares:
1) All chess squares were iterated through files and
ranks.
2) Blobs consisting of pixels standing above the table
were detected in the depth picture, limiting the search
only to the current processed square. To limit the
search was used ROI (region of interest) - a feature of
image processing library.
3) If detected blob’s pixel count was higher than a
minimal chess piece pixels threshold, square was
considered to be occupied.
However in some situations it happened that one physical
chess piece was counted twice as if occupying two
neighboring chess squares. This happened usually if a
piece was larger (f.e. queen or king) and on the depth view
was overlapping more than one square, as indicated on
“Fig. 2”. The red square indicates an incorrectly detected
chess square as occupied while the piece overlapping the
two squares was counted twice.
Fig. 2
Overlapping piece problem
3.2.3.2. Improved algorithm solving piece overlapping
problem
This algorithm first detects all blobs in the region of the
whole chessboard and then associates them with
appropriate chess squares. This is done by finding the
closest chess square to the blob centers. This approach
does not allow the blob of the chess piece to be counted to
two different chess squares, thus solving the piece
overlapping problem.
3.2.4. Opponent’s piece taking move detection
problem
In some situations analyzing the depth frame does not
provide enough information to uniquely determine a
3
player’s move. “Fig. 3” shows an example of such a
situation – it is the white player’s move. He can either
take left black pawn or right black pawn:
Fig. 3 Chess game situation with potential detection
problem
By analyzing only changes in the depth frames we cannot
determine, what piece was taken by the player. That is
because in both possible situations we would detect the
same set of occupied chess squares.
To determine which opponent’s piece was taken by the
player we need to analyze not only depth frames changes,
but also RGB frames changes:
1) RGB image of chessboard is stored before and after a
player’s move.
2) Possible legal target opponent’s pieces are retrieved
from the game engine. In our example legal targets
are two black pawns, left and right from a white
pawn.
3) For all possible targets, the difference between
appropriate regions of an RGB chessboard image is
calculated. From all candidates, the square with the
biggest difference is chosen. This is most likely the
piece that was taken by the player, because the
content of the image at its square will change most.
“Fig. 4” shows RGB image before and after such
move.
Fig. 4
RGB square before and after taking move
3.2.5. Algorithm to determine chess move based on
comparing successive frames
When the position of the chess pieces is collected for
frame before and after the move, the application needs to
calculate the proper chess move that was made by the
player. Following situations can happen:
a) a)
Piece disappears on square S2 and a new
piece is detected on square S1. This is interpreted
as a move from S1 to S2.
b) Piece disappears on square S2 and no new piece
is detected. This is interpreted as a move such as
a player is taking opponent’s piece.
Accessible chessboard for blind that is mouse and keyboard free
4. RESULTS
ACKNOWLEDGMENTS
By using approaches and algorithms described in this
paper, a working chess game application has been created.
It allows a VI chess player to start and finish a chess game
without any need of a computer screen or a mouse. The
whole interface is solely based on moving real chess
pieces which is much more comfortable and enjoyable for
the players comparing to typing his move in chess
notation to existing chess software.
Of course there are some aspects of the solution that needs
improving. Here are two of these limitations:
1)
When a depth sensor is placed for the first time
above the table, both depth and RGB cameras require
calibration that must be performed by somebody without
visual impairment. Calibration of the depth camera is
performed by placing two rooks at two opposite corners of
the chessboard and pressing the calibrating key.
Calibration of the RGB camera is performed with a mouse
by clicking on the opposite corners of a chessboard on a
camera preview, which obviously cannot be done by a
blind person. Future research should be done to eliminate
or automate calibration so that a VI player can manage the
process.
2)
The current prototype enables a game against a
PC chess engine but there is no support for an online game
against remote human opponent, which would help VI
players to better socialize with their human chess
opponents.
This work was supported in part by Specific research
grant provided by the University of Pardubice and in part
by grant for the selected innovative projects provided by
Sense Technologies. Author would also like to thank
Professor Simeon Karamazov for providing a researchfriendly environment at the University of Pardubice.
REFERENCES
[1] „WHO | Visual impairment and blindness“. [Online].
Available:
http://www.who.int/mediacentre/factsheets/fs282/en/
. [Accessed: 2012/02/12].
[2] „B.C.A. Website“. [Online]. Available:
http://www.braillechess.org.uk/. [Accessed:
2012/02/12].
[3] „Jetyho blog » How to get OpenCV (EmguCV)
Image from Primesense Kinect depth generator“.
[Online]. Available:
http://jetensky.net/blog/2012/02/24/how-to-getopencv-emgucv-image-from-primesense-kinectdepth-generator [Accessed: 2012/03/24].
[4] „Image Filtering — opencv v2.2 documentation“.
[Online]. Available: http://opencv.jp/opencv2.2_org/cpp/imgproc_image_filtering.html#cvdilate. [Accessed: 2012/03/24].
CONCLUSIONS
As is my humble wish, this paper can suggest to other
researchers what steps and algorithms that may use for
real applications of depth and RGB camera sensors.
In the next phase of this research I would like to offer
this prototype to real visually impaired chess players for
testing. By doing this I could identify potential flaws in
the design and bring the idea to the next levels of
usability.
BIOGRAPHIES
Pavel Jetensky has 11 years of SW development
experience as SW Java Engineer and architect, including 3
years in Ireland. For last 2 years he runs a PhD research
“3D user interfaces”. He focuses on analyzing gestures
and physical objects tracked by depth cameras like
Microsoft Kinect, to be used in Computer Human
interaction.
ISBN XXXX-XXXX © 2012