Sixth Sense Teaching Aid
Shilpa Chaman
A.P EXTC dept. SFIT
(Mumbai University)
Mumbai,India
shilpachaman@sfitengg.org
Aniket Ghadi
Student EXTC dept. SFIT
(Mumbai University)
Mumbai,India
aniketghadi50@gmail.com
Abstract—This paper proposes a novel Sixth Sense Teaching
Aid (SSTA) which incorporates sixth sense technology in
projectors for educational purpose. The Projector-PC system was
earlier used only fordisplaying the presentations but with the aid
of the proposed system, users can touch on any projected
surfaces for interaction purpose. In the SSTA system, the
graphical user interface (GUI) buttonsare projected on any flat
surface like wall andit deals with touch detection of the projected
screen using red color parameter both for still image and real
time images. The algorithm to perform touch detection is
executed in two stages: 1) Feature extraction and button’s touch
detection using red color thresholding algorithm which reduces
the computational complexity of the processing module; and 2)
Performance of assigned operation according to touch action
judgment. New born technology named Sixth Sense technology is
also implemented in SSTAfor getting relevant information from
the internet, whenever we touch any projected figure or
headline.The proposed SSTA system is able to do real time touch
detection with 97 percent accuracy which is demonstrated
through projected GUI and using a data set collected under
different settings of illumination variation, hand orientation and
occlusion.
Keywords—Hand gesture recognition; projector camera system;
sixth sense; SSTA;GUI.
I.
INTRODUCTION
Sixth sense can also be called as Extra Sensory Perception
(ESP) since the reception of information is done apart from
the five senseswhich are hear, touch, smell, sight and
taste.Thus, sixth sense technology has integrated the real world
objects with the digital world with the assistance of PC, web,
camera, cell phone and projector.
Steve Mann was the father of Sixth Sense Technology. He
implemented the concept as the neck worn projector with the
camera system [1], [2].The concept was further developed by
Pranav Mistry who was a PhD student and Research Assistant
in the Fluid Interfaces Group at MIT Media Lab [3].He
developed aprototype device which is calledWear Ur World
(WUW), whose working mechanism was very simple. It uses
wired or wireless connection to connect projector and camera
to a computing device which utilizes web via 3G, 4G or
wireless Wi-Fi. In the computing device, computer vision
algorithm and gesture recognition technology are used to
identify the objects or understand the gestures fed by the user
through camera. These streaming gestures or images of objects
are taken as instructions by the computing device. The device
then works on the instruction and the accordingly performs
query to the internet if required and passes the information
978-1-5386-5367-8/18/$31.00 ©2018 IEEE
Ninad Ketkar
Student EXTC dept. SFIT
(Mumbai University)
Mumbai, India
ninadsketkar@yahoo.com
back. That information is projected out through the projector
and the mirror reflects it to the desired surface or object. The
user can then interact with the projected image to give another
instruction.
During recent years, touchscreen interactive projectorsfor
both computers andmobile equipment have become
animportant and dominant area of research.Now-a day’s
mobile users wants to display their digital contents on any
projected surfaces so that they can enjoy large screens and also
it helps in protection of their eyes from the digital eye strain
caused by small screens of mobile phones. Portability and low
cost of projectorshave made them replaced the traditional
display monitor screen. With the help of computer vision and
projection technology interactive projectors are made which
can transform any projected surface into a touch screen and the
user can havea fantastic experience of touching the virtual
screen, opening or closing files or applications and dragging
items.
“VirtualKeyboard” is the key concept of such
interactive projector system. Using finger recognition
algorithm we can type the content without any physical
clicking using such a keyboard.
The biggestchallenge to achieve the touch interaction lies in
determining the fact whether the fingers actually touchedthe
projected surface or not. Multi-cameras or a depth camera were
used by most of the researchers [4]–[6] to obtain the
relativeposition between the fingertip and the projected
surface.Utilizing complex triangulation and adjustment to
compute the separation amongst fingertip and anticipated
surface, Wang et al. [7] proposed touch recognition by
anticipated image and a 3D sensor display. By evaluating the
separation from fingertip to the projected surface with the help
ofgeometric relationship, Cai et al. [8] identified fingertip
location.
This paper presents a novel Sixth Sense Teaching Aid
(SSTA) system which has a touchscreen interactive projector
integrated with sixth sense technology and also proposes an
approach which uses only one standard camera to detect the
touch operation with high accuracy.Integration and
collaboration of sixth sense technology in the classrooms will
be a new concept over the traditional classrooms. SSTA
system includes applications such as highlighting a text,
inserting new text, zooming of an image, creating multimedia
reading experience and projection of an analog clock and
calculator on the projected area. The system basically relies on
touch detection of the projected screen using red color
parameter both for still image and real time images and on
object recognition to call up virtual gadgets, database
information or information from the internet that we want to
know while viewing that particular object/image. It can project
information on a wall or any other surface. Theproposed SSTA
system makes use of a regular projector and camera instead of
depth camerawith which millimeter accuracyis achieved. It
also makes the whole system cost and power efficient. Future
generations will likely marvel at the fact that how classrooms
can be made so interactive by a teacher or a trainer.
The paper is divided into following sections. Section II
presents the methodology of the system with the detailed
description of each processing modules used in SSTA system.
It basically provides the real time interactive projector system.
The results are discussed in Section III and Section IV presents
the conclusion and the future work.
II.
METHODOLOGY
The proposed SSTA systemis shown in Fig.1.
SSTAsystem has 5 main phases of processing.The complete
gesture controlled SSTA system has following4 main phases
of processing.
A) Projection and Virtual Touch Screen.
camera output is fed to PC for further processing to detect
touch detection.
2) Projected Virtual Touchscreen :Touch Screen has
revolutionized
the
mobile
&
tablet
market.
With touch-screen user can simply touch the screen and
buttons are pressed. Virtual Touchscreen utilizes camera for
implementing buttons. Hence buttons can be anywhere
within range of camera view. Button can be on table, on
blackboard, on walls, on roof, on hand, on refrigerator
everywhere. Here an interactive GUI is displayed which
behaves as a virtual touchscreen. Image acquired from
webcam is divided into required number of parts. Each part
represents a virtual button.
B) Feature Extraction and Motion Detection:
The GUI buttons or digital content are projected on the
planar surface. Whenever the red colored object is put on the
specific button it gets distorted and it is intercepted as click of
the key on the screen. This model of distortion caused by the
red object is very important.
1)
Feature extractionsteps are depicted in the Fig.2.
B)Feature extraction and motion detection.
C) Mapping to assigned applications.
Fig. 2 .Block diagram of Feature Extraction
Fig. 1.Block diagram ofSSTA System
A) Projection andVirtual Touch Screen.
The first hardware key component is a Sony VPL DX-100,
XGA Desktop projector with 1024 x 768 dot resolution
[9].The projector projects the interactive GUI on the wall from
2.5-3 m.A Logitech C170 web camera [10] is used that
provides a 960×720 image at 30FPS. In this section projector–
camera system and projected virtual touchscreen is discussed.
1) Projectorcamera System : The image is captured by the
camera when it is projected by the projector on the desired
planar surface. A fixed distance is maintained between the
projector–camera system and the surface. It uses camera for
implementing Virtual touch screen.In order to cover the
entire region of the projected surface, the viewing angle of
the camera is taken larger than that of the projector.The
978-1-5386-5367-8/18/$31.00 ©2018 IEEE
a) Image Acquisition : An image is a matrix of pixel
values. MATLAB accepts each input as a matrix that is
why, it is most commonly used platform for image
processing. An image can be represented using many
color models like gray-scale, RGB, HSV etc. Read the
input image in RGB format which is the most
commonly used format to represent colored images, if
the resolution of the image is MxN, then the RGB
format of the image will be a three-dimensional matrix
of size MxNx3 where each dimension of the matrix
represents the red, green and blue color components of
the image.
b) Convert RGB to GrayscaleImage : The strategies
required for the identification of color in pictures are
change of three dimensional RGB picture into dark
scale picture.The grayscale format of the image will be
a two-dimensional image containing the intensity value
of each of the pixel of the image. Usually a grayscale
c)
d)
e)
f)
g)
image increases the speed of processing, ease of
visualization, and reduces complexity of code by
converting a three-dimensional image into twodimensional image resulting in reduction in number of
bits used to represent each pixel of an image.
Subtracting Gray scale Image from Red band Image :
This grayscale image is then subtracted from red band
image for the detection of red color in the image.
Subtraction of grayscale image from red band of the
input image is done to extract red color components of
the image.
Removal of Noise using Median filter: Median
filtering is used to remove unwanted noise from the
image while maintaining the originality of image
[11][12]
Conversion to Binary Image:This filtered image is
then converted into binary image to work only on the
area of interest (1 for yes and 0 for no). This binary
image is generated from the filtered image using
thresholding process[13]. In thresholding process each
pixel of the image is given 1 or 0 on the basis that – if
the pixel value is greater than the set threshold value
then it is assigned ‘1’(white) else ‘0’{black). The
threshold value of the image is calculated using the
intensity histogram of the image [14].
Labelling of the connected components :Connected
components labeling is usedto recognize the connected
regions in binary digital images where all the objects of
detected color having less than 300 pixels are removed
from the image.
Recognizing the boundaries of an object:Here the
bounding box and its properties are utilizedfor
calculating
the
metrics
of
each
labeled
region.Boundaries of an object are searched
diagonally(in 8 connected pixels) [15].The detected
object is shown as abounded white object.
2) Motion Detection: The algorithm for motion detection
is discussed below:
a) We consider the standard background image or the
first frame of the video as the reference image for
comparing the other frames of the video in order to
detect any kind of motion.
b) Background (Reference image) to foreground
subtraction is done to identify any sufficient change in
the pixel values. If any red object is detected and the
changes are observed in the pixel value according to
the above feature extraction algorithmand if the
changes are sufficient enough thenit can be marked as
an indicator of motion else the change is too small to
be considered as a motion of insignificant amount.
c) If some motion is detected, then the next few
consecutive frames will be analysed. If the difference
comes out to be zero or equal for the next frames, it
will indicate that the object has stopped. Get X,Y
coordinates of touched button in the reference image.
Thus button is considered to be pressed, if it is pointed
by the red object.This is considered as ‘touch’ of a
particular button and that respective application is
performed.
978-1-5386-5367-8/18/$31.00 ©2018 IEEE
d) Hence it is this frame which will be considered as the
new reference frame for recognizing any sort of
movement after that.
These above steps are coded in MATLAB and executed to
detect any kind of motion with respect to the input video. The
motion tracking continues throughout the video with respect to
the standard background which changes as per the motion of
the moving red color object in the captured video.
C) Mapping to assigned applications : Wheneverthebutton is
considered as pressed, assigned application like displaying
clock, calculator, controlling the PPT, information from web,
etc. is performed using the calculated coordinates.
III.
RESULTS AND DISCUSSION
A set of 100 images of different resolution and clarity were
used for testing of this algorithm and the results were mostly
accurate. Here a discussion of processing the camera input
image by using GUI in MATLAB is done.Fig.3 shows the
GUI for SSTA implementation.
Fig.3 GUI in MATALB for SSTA
The following features are implemented using GUI in
MATLAB
A. Presentation
Classrooms can be made more interesting by a teacher or a
trainer in our system as it allows the presentation pages (PPTs)
to be projected on any surfaces for interaction purpose using
touch detection. In the traditional method, the trainer needs to
make use of a keyboard or a mouse to turn to the next or
previous slide. However, this may lead to distraction in the
audience’s attention. Instead, the quality of the
teaching/trainingimproves if the trainer can directly operate
the PPT with touchinteractive buttons like next slide, previous
slide, etc.Fig.4 demonstrates thenext slide button
implementation.
5) Sixth SenseApplication
In a presentation there will be images in some of the slides.
When a presenter want to get any information about the
presented image/object, then he/she can simply tap on that
image and with the help of internet we can display all the
relevant information available on the projected screen. This
will eventually avoid the efforts of using any search engine
manually and gets the result quite easily.Fig.7 and Fig.8,
demonstrates the Sixth Sense implementation using web.
When the red object is kept on any image, web-enabled PC
processesthe image data, get all the relevant web information.
Fig.4Next Slide button implementationfor SSTA
2) Calculator
The Calculator is used to compute the values for different
operations. If the user wants to make any calculations while
presenting the topic then he/she can simply access the
calculator. The calculator can perform basic math operations
such as addition, subtraction, multiplication, division etc. It
can be further improvised to a scientific calculator which may
perform many complex operations.All the keys required for
calculator are implemented using virtual keyboard on the
projected screen. Fig.5. demonstrates the calculator
implementation using virtual keyboard.
Fig.6Keeping the colored object on the image
Fig.5Implementation of Calculator using virtual keyboard for SSTA
4)Clock :If the user wants to check the time at any instant
while presenting the topic he/she can easily access the
clock.Fig.6 demonstrates the Scotts clock implementation.
Fig.6Implementation of Scotts clock for SSTA
978-1-5386-5367-8/18/$31.00 ©2018 IEEE
Fig.7 Relevant information displayed on projected screen using web
IV.
CONCLUSION
Integration and collaboration of sixth sense technology in
the classrooms will be a new concept over the traditional
classrooms. SSTA system includes applications such as
opening new PPT, controlling the PPT and projection of an
analog clock and calculator on the projected area. The
proposed approach utilizes the feature extraction and red
object motion detection to sense the touch action. Sixth sense
technology is also implemented to call up virtual gadgets,
database information or information from the internet that we
want to know while viewing that particular image on the slide.
It can project information on a wall or any other surface.
Theproposed SSTA system makes use of a regular projector
and camera instead of depth camerawith which millimeter
accuracyis achieved. It also makes the whole system cost and
power efficient. SSTA system will definitely revolutionize the
perspective of the student towards attending the lectures as the
classrooms will be made more interactive by a teacher or a
trainer.
REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
Steve Mann, "Telepointer: Hands-free completely self-contained
wearable visual augmented reality without headwear and without any
infrastructural reliance,”2000, pp.177-178.
Steve Mann, Hal Niedzviecki, "Cyborg: Digital Destiny and Human
Possibility in the Age of the Wearable Computer,”ISBN 0-385-65825-7
(Hardcover), Random House Inc,2001,pp.304
Mistry, P. 2013. Available: http://www.pranavmistry.com/. Accessed 17
February 2014
Z. Mo, J. P. Lewis, and U. Neumann, “SmartCanvas: A gesturedriven
intelligent drawing desk system,” in Proc. ACM IUI, 2005,pp. 239–243.
H. Benko and A. Wilson, “DepthTouch: Using depth-sensing camera to
enable freehand interactions on and above the interactive surface,” in
Proc. IEEE Workshop ITS, vol. 8, 2009.
C. Harrison, H. Benko, and A. D. Wilson, “OmniTouch:
Wearablemultitouch interaction everywhere,” in Proc. ACM UIST,
2011,pp. 441–450.
W. Qun, C. Jun, P. Jianxin, and S. Sanming, "Fingertip-based interactive
projector-camera system," in Information and Automation (ICIA),IEEE
International Conference on, 2013, pp. 140-144.
C. Xilei, X. Xiang, L. Guolin, S. Wei, Z. Yi, and W. Zhihua, "A new
method of detecting fingertip touch for the projector-camera HCI
system," in Circuits and Systems (ISCAS),IEEE International
Symposium on, 2014, pp. 526-529.
Sony Ltd., [Online]. Available:
http://pro.sony.co.in/pro/product/projectors-desktopportable/vpldx100/specifications/#specifications.
Logitech Ltd., [Online]. Available: https://www.logitech.com/enin/product/webcam-c170
Wang, Zhou, and David Zhang. "Progressive switchingmedian filter for
the removal of impulse noise fromhighly corrupted images." IEEE
Transactions onCircuits and Systems II: Analog and Digital
SignalProcessing 46.1 (1999): 78-80.
Brownrigg, D. R. K. "The weighted medianfilter." Communications of
the ACM 27.8 (1984): 807-818.
The Multi-stage Approach to Grey-Scale Image Thresholdingfor
Specific Applications, Van Solihin and C. G. Leedham
N. Otsu, “Threshold Selection Method from Gray-level Histograms”,
IEEE Transactions on Systems, Man, Cybernetics, vol. SMC-9, no. 1,
pp. 62–66, Jan. 1979.
F. Meyer, “Color image segmentation”, Proceedings of4th International
Conference on Image Processing, pp.523–548, 1992
978-1-5386-5367-8/18/$31.00 ©2018 IEEE