Internati onal Journal of Electronics, Electrical and Computati onal System
IJ EECS
ISSN 2348-117X
Volume 4, Issue 3
March 2015
Implementation of Multi-Touch Surface using Combined
Diffused Illumination and FTIR techniques
Prannoy Pokharna
Sakshi Taneja
Dept. of Electronics and Co mmunicat ion
Dept. of Electronics and Co mmunicat ion
Abstract—The paper talks about the prevalent technologies
in the M ulti-touch surfaces which operate exclusive of each
other and our research on combining FTIR and Diffused
Illumination techniques and the counter party benefits this
combination provides.
Keywords—multi-touch, CCV, Touchlib.
I.
INT RODUCTION
Multi-touch denotes a set of interaction techniques which
allo ws the computer users to control applications with
several fingers. Multi-touch consists of a touch screen or
touchpad, as well as software that recognize mult iple
simu ltaneous touch points. There are several ways to
make a mult i-touch surface.
Our whole setup of the multi-touch surface is based on the
FTIR (Frustrated Total Internal Reflection) technique. It
utilizes the IR spectrum of electromagnetic spectrum of
light. Basic idea of this technique is to scatter and reflect
the waves through the touch surface which is gathered by
the camera [1].
Total Internal Reflection describes a condition present in
certain materials when light enters one material fro m
another material with a higher refractive index, at an angle
of incidence greater than a specific angle. The specific
angle at which this occurs depends on the refractive
indexes of both materials, and is known as the crit ical
angle, which can be calculated mathemat ically using
Snell’s law [2]. When this happens, no refraction
occurs in the material, and the light beam is totally
reflected.
An FTIR (short for Frustrated Total Internal Reflection)
setup involves three vital components: a sheet of
transparent acrylic, a chain of infrared LEDs, and a
camera with an IR filter. The LEDs are arranged around
the outside of the sheet of acrylic so that they shine
directly into the thin side surfaces.
Fig 1: Scattered frustrated light waves through the acrylic sheet
(source: cs.nyu.edu)
11
Prannoy Pokharna, Sakshi Taneja
Our aim was to flood the inside of a piece of acrylic with
infrared light by trapping the light rays within the acrylic
using the principle of Total Internal Reflection [3]. Once
the IR light is inside the acrylic, it strikes the top and
bottom surfaces of the acrylic at a near-parallel angle.
This causes it to be wholly maintained in the acrylic.
When the user comes into contact with the surface, the
light rays are said to be frustrated, since they can now
pass through into the contact material (usually skin),
and the reflection is no longer total at that point within
the sheet it, the light rays are reflected down. This
frustrated light is scattered downwards towards an
infrared webcam, capable of picking these ‘blobs’ up, and
relaying them to tracking Soft ware.
Fig
2: Block Diagram of the Touch Surface Implementation
The camera then sends the input to the computer which
encompasses the whole touch surface by calibration.
These received inputs are processed as specific
coordinates relating them to the calibration. The computer
comprehends the coordinates with respect to the screen
and sends .xml output to the running flash application.
The application then produces the desired output.
II.
ORIGIN OF MULTIT OUCH
The origin of mult i-touch technology takes us back to
1982 when Nimish Mehta from the University of Toronto
was the first one developing the finger pressure display
using this technology. The year 1982 was an important
year in mu lti-touch history because after the discovery of
Nimish Mehta, other engineering companies like Bell
Labs got involved in the development of this technology
[5].
Internati onal Journal of Electronics, Electrical and Computati onal System
IJ EECS
ISSN 2348-117X
Volume 4, Issue 3
March 2015
In this way, in 1983 there were many discussions related
to mu lti-touch screens and this new revolutionary
technology, which led to the development of a touch
screen that would change images by using more than just
one hand. Over time, Bell Labs decided to focus more on
software development rather than hardware and had
registered important success in the field o f mu lti-touch
technology [6].
Until the 20th century, the mu lti-touch technology was not
so popular due to the lack of b reakthroughs in this
domain. Ho wever, the success, popularity and use of this
technology started to change once with the breakthrough
of Pierre Wellner in 1991. In his paper "Digital Desk",
Pierre Wellner presented the advantages and mechanism
of mult i-touch technology, supporting the idea of mult ifinger use [6].
Furthermore, starting with 2001, all these papers and
inventions were further analyzed, imp roved, expanded
and developed in today's multi-touch devices, software
programs and hardware. The main player in the mult itouch technology market was Apple who in 2007
launched the iPhone. This product led to an increase in
popularity and use of this technology as more customized,
robust and gesture-based devices were developed since
then on.
In other words, even if the roots of mu lti-touch
technology are related to the year 1982 the real
development of mu lti-touch solutions as we see them
today is related to the discoveries reached since 2007 and
on.
Finally, the future of mult i-touch technology seems bright
as more and more solutions are developed on a daily basis
and imp lemented in all types of businesses and fields of
activity like medicine, banking and most importantly - the
engineering domain.
During the last two years there have been many
developments of solutions, either as devices, software
programs or hardware. Whether we are thinking of mult itouch tables, multi-touch phones or multi-touch displays,
this technology definitely represents an area of expertise
which has yet a lot to reveal.
III.
SYST EM SPECIFICATIONS
Multi-touch denotes a set of interaction techniques that
allo w co mputer users to control graphical applicat ions
with several fingers. Multi-touch devices consist of a
touch screen (e.g., computer display, table, wall) or
touchpad, as well as software that recognizes mult iple
12
Prannoy Pokharna, Sakshi Taneja
simu ltaneous touch points, as opposed to the standard
touch-screen (e.g. computer touchpad, ATM), which
recognizes only one touch point.
Our system consists of the following:
 Wooden box
 IR Camera
 IR Transmitters
 Acrylic sheet
 Projection surface
 Co mputer
Each of the requirements was very specific in their
configuration to an extent that would affect the
preciseness of the touch surface. Therefore, each
component was chosen after research and also tried and
tested prior to their use with other components, so as to
know that they are compatib le with them or not.
The above requirements had certain specificat ions that
needed to be fulfilled in order that our system worked
properly.
A. WOODEN BOX
The wooden box holds our setup. On the top the acrylic
sheet is mounted. Through which the IR transmitters pass
the infrared light. When the surface is touched it reflects
the lights toward the camera wh ich is positioned at the
base near the projector to receive the IR light. The focal
length of the camera virtually determines the height of the
box we are using. But there is another important factor,
which is the “size of the touch surface” which we can
calibrate using the software.
B. IR CAMERA
It is required to catch the reflected beam fro m our touch
surface to know the exact location of the touch. The IR
camera filters the visible radiations and allows only the
infra-red spectrum to pass through the lens. The
wavelength transmitted by our LED’s are in range fro m
850-950 n m so the camera should be able to detect
infrared in that range of spectrum. To achieve this a filter
(i.e. basically a tape inside floppy) is put instead of
original filter of webcam.
Its specifications included:






Resolving power: 640*480
Video format:24bit RGB
Frame rate: 320* 240 up to 30 frames per second
S/N Ratio :48 d B
Focus range:3 cm – infin ity
Manual focus
Internati onal Journal of Electronics, Electrical and Computati onal System
IJ EECS
ISSN 2348-117X
Volume 4, Issue 3
March 2015
C. INFRARED TRANSMITTERS
These are the infra-red led’s which transmit infra-red
spectrum of light which cannot be seen by naked eyes.
This light travels through our touch surface. When
reflected from the surface is collected by the IR camera.
Infrared Emitt ing Diode (IR333/H0/ L10) is a High
intensity diode, molded in a blue transparent plastic
package. These led’s have a peak wavelength of 940 n m
and lead spacing of 2.54 mm.
It is used due to its optical properties. Acrylic is
lightweight, strong and very clear. Acrylic is very clear so
FTIR can’t occur which means that if we tried to project
onto the acrylic as the light would pass right through. In
order for FTIR to take place a projection surface was
made.
D. PROJECTION SCREEN
Projection Screen included the acrylic sheet and the
surface. The foundation of the screen is the sheet of
acrylic wh ich serves as the mediu m for the infrared light.
It is used due to its optical properties. Acrylic is
lightweight, strong and very clear. Acrylic is very clear so
FTIR can’t occur which means that if we tried to project
onto the acrylic the light would pass right through. In
order for FTIR to take place a projection surface was
made. The surface was made comp laint to the FTIR
properties by using a layer of higher RI than the acrylic
sheet itself. In our case, we used a layer made of Silicon
Sealant. This was cost effective, easy and the layer can be
modified as per our requirements.
E. SOFTWARE
The software takes the input fro m the camera and
calibrates it. The Software used here is CCV [4] , an open
source software used for comprehension of multi touch
surfaces. It converts the colour space or compresses it.
The function for the compression of image in RGB colour
space to luminance is:
Y = (0.299 * R) + (0.587 * G) + (.114 * B)
Here, R, G and B are variables with values corresponding
to their wavelengths in nanometers.
Then, compares the source image fro m the camera and the
tracked image post calibration. It performs three
functions:
1. Adjusting
the
background:
Setting
itself
automatically to a new background with the changes
in the external environ ment.
2. Stabilizing the in line image: Using the image
threshold and the movement filters.
3. Filtering the Image: The filtering is done via the high
pass and the amplification filters.
13
Prannoy Pokharna, Sakshi Taneja
IV.
IMPLEMENTATION
CCV receives, burst of images fro m the camera it takes
input from. Then, the background it adjusted in
accordance with the intensity of light entering the surface
fro m outside. Thus, each time there is a significant change
in the intensity of light CCV automatically adjusts to it.
Once the intensity of inco ming light is considered, then
the image is stabilized using the threshold filters and the
movement filters. These filters restrict the false blobs
which might occur due to change in the intensity. A
minimu m threshold is the level of intensity of light which
would not be considered as a point. Thus, when the
intensity of light entering is high, then the thresholds are
kept high and vice versa. The Smooth filter taken into care
the hard or the soft transitions in the light intensity. The
noise is eliminated by the High pass filters and the filtered
image, then can be amplified post the noise reduction.
Post this, the actual touch point recognition happens. The
whole process of referencing the coordinates can be done
as:
Considering 2 pixels α at (x, y) and β at (x, y - θ) in the
source image. Equation given below is applied to each
pixel under consideration: Ynew(x, y) = Yα – Yβ
This Ynew gives the difference in the intensity at two different
points in the screen space. If,
•
Ynew >= Threshold(δ) then y=0 (Part of the fingertip)
•
Ynew <
Threshold(δ)
then
y=255 (Part of the
background)
Fig 3: CCV Startup Screen
For each frame, when the precise touch points are figured
out by CCV in the way mentioned above. It can lend the
coordinates of these touch points to the external
applications for integrated communications. I can
communicate with various applications running Tangible
user interface programs or other flash applications in the
.xml format.
Internati onal Journal of Electronics, Electrical and Computati onal System
IJ EECS
ISSN 2348-117X
Volume 4, Issue 3
March 2015
There were other problems with this surface as well. First,
it could not work in the places with excessive light. This
made it impossible to work in the outdoor environmen ts
where it could prove really useful. Secondly, while the
infra-red LED’s were working perfectly fine until a
change in the external environ ment occurred. As soon as
the light entering above the surface changed, the exposure
settings from CCV has to be readjusted to current
scenario.
VI.
IMPROVEMENT S MADE
A. Working solely with Diffused Illumination
A comprehensive working of the Tangible User Interface
Applications has been shown here:
We researched a bit about the diffused illu mination and
tried to use the touch surface without FTIR in the acrylic
sheet. This techniques also known as diffused illu mination
had two advantages. First, now the surface could be
operated with high levels of exposure. Secondly, the light
was not reflected now, it came out as a shadow. So we
had to invert the image which came through the camera.
Fig 4: Working of the Tangible User Interface Applications
source: tuio.org
V. PRIMARY RESULTS
During the in itial adjus tments done in the CCV software;
we observed the there was a lot of noise due to the
interference of the light present above the surface. This,
can be attributed to the fact that the difference in the
wavelengths of the visible light and the infra-red light.
This is shown below:
Fig 5:Configured touch screen
As you can see above, the images are too much affected
by the noise because of the interference by the light
entering from outside. This created a soft edged
comprehension of the touch points.
14
Prannoy Pokharna, Sakshi Taneja
Fig 6 (a) Image of finger on screen
b)Resultant digital image
It also had a major issue: In this case, the camera was not
receiving light where the fingers were present and
receiving elsewhere. So we had to invert the image and
apply the high pass filters. As the image was being
inverted, it made a lot of noise in the deduction of the
coordinates. This was due to the fact that there was a
difference in the amount of light entering fro m the other
areas than entering where the palm was.
B. Combining FTIR with Diffused Illumination
Thus, to improve the overall efficiency, we decided to
combine both the techniques in order to receive
complimentary benefits fro m both of them. We gave an
array of the infra-red LEDs all the edge of the acrylic
sheet and let the acrylic sheet operate in an environment
with excess light. Now the camera was receiving image as
a blob only where the fingers blocked the outside light
and received light elsewhere. This created a sharp edge
which was easily separated by the noise filters of CCV.
This setup solved the earlier p roblem of distortion of an
inverted image by the CCV due to inadequate noise
filtering.
Internati onal Journal of Electronics, Electrical and Computati onal System
IJ EECS
ISSN 2348-117X
Volume 4, Issue 3
March 2015
work. Th is work would not have been possible without his
valuable guidance.
We would also like to thank Mr. Puneet Sharma and Mr.
J. B. Sharma for their kind and valuable guidance in the
project.
REFERENCES
Fig 7: Calibration of the touch surface
VII.
FINAL RESULT S & CONCLUSIONS
As shown in the image above, when we use the mixture of
both the techniques, it made the image detection much
more accurate by the CCV. The coordinates were picked
up with a much lesser lag time and were mo re precise than
before. Thus, none of the techniques are solely exhaustive
enough. Thus we used both the techniques i.e. Diffused
Illu mination and the FTIR (Frustrated Total Internal
Reflection) to compensate for each other’s shortcomings.
While, Diffused Illu mination helped by converting our
shortcoming of excess interference of light to our benefit,
at the same time, FTIR increased the accuracy of the
intercepted coordinates by reducing the noise.
A CKNOWLEDGMENT
We owe our profound gratitude to our project guide and
instructor, Prof. Dr. S. L. M isra for being a constant
source of inspiration throughout the course of this project
15
Prannoy Pokharna, Sakshi Taneja
[1]
[2]
[3]
[4]
[5]
[6]
Liu, Hongyi, and Xinfu Liu. "The design and implementation of
multi-touch screen system based on FTIR theory." e-Education,
Entertainment and e-Management (ICEEE), 2011 International
Conference on. IEEE, 2011.
Bergman, David R. "Symmetry and Snell’s law." The Journal of
the Acoustical Society of America 118.3 (2005): 1278-1282.
Rong Chang, Feng Wang, Pengfei You, "A Survey on the
Development of Multi-touch Technology", APWCS, 2010,
Wearable Computing Systems, Asia-Pacific Conference on,
Wearable Computing Systems, Asia-Pacific Conference on 2010,
pp. 363-366, doi:10.1109/APWCS.2010.99
Schöning, Johannes, et al. "Building interactive multi-touch
surfaces."Tabletops-Horizontal Interactive Displays. Springer
London, 2010. 27-49.
Chang, Rong, Feng Wang, and Pengfei You. "A Survey on the
Development of Multi-touch Technology." Wearable Computing
Systems (APWCS), 2010 Asia-Pacific Conference on. IEEE, 2010.
White, Jeremy. Multi-touch interfaces and map navigation. Diss.
UNIVERSIT Y OF WISCONSIN-MADISON, 2009.