Bare finger 3D air-touch system with embedded multiwavelength optical sensor arrays
for mobile 3D displays
Guo-Zhen Wang
Yi-Pai Huang (SID Senior Member)
Tian-Sheuan Chang
Abstract — A camera-free 3D air-touch system was proposed. Hovering, air swiping, and 3D gestures for
further interaction with the floated 3D images on the mobile display were demonstrated. By embedding
multiwavelength optical sensors into the display pixels and adding angular-scanning illuminators with
multiwavelength on the edge of the display, the flat panel can sense images reflected by a bare finger from
different heights. In addition, three axis (x, y, z) information of the reflected image of the fingertip can be
calculated. Finally, the proposed 3D air-touch system was successfully demonstrated on a 4-inch mobile
3D display.
Keywords — 3D interactive, air touch, embedded optical sensors, mobile 3D display.
DOI # 10.1002/jsid.190
1
Introduction
The widespread adoption of smartphones and tablets has
accelerated the transformation of user interfaces and has
paved the way to multi-touch technologies in the beginning
of 21st century.1 Meanwhile, 3D technologies2–5 have
dramatically changed the user experience. Today’s 3D display
systems can provide new advantages to end-users and are able
to support an autostereoscopic, no-glasses, 3D experience
with significantly enhanced image quality over earlier technology. Furthermore, numerous 3D interactive systems for
providing friendlier and more intuitive user interfaces have
been proposed for 3D display applications.
Currently, there are two categories of 3D interactive
systems6,7: machine-based and camera-based. Machine-based
systems8 involve devices worn by the user for motion
detection and supports feedback vibration. For instance, the
Haptic WorkstationTM can detect six axes of hand movement
through data gloves9 and can render force-feedback on the
wrists. Although machine-based systems have force-feedback
functions, they are often thought to be inconvenient because
of the bulk and weight of the devices.
For camera-based systems,10 3D information (x, y, z) can
be calculated by one of a variety of camera technologies.
For instance, the popular Wii and Kinect game consoles11,12
are able to detect relative 3D position by using infrared (IR)
cameras with corresponding IR light sources.13 However,
these camera-based systems are limited by their fields of view
and by factors that prevent them from detecting an object’s
proximity to the display. These factors prevent them from
being integrated into mobile devices such as smart phones,
tablets, or laptops. Additionally, high-resolution data are
necessary for the camera-based system to calculate 3D
position, as the resolution is proportional to the size of
charge-coupled device of the camera; this factor also impedes
the system from being integrated into portable devices.
For the integration of these technologies into mobile
devices, embedded optical sensors that can be integrated
within the display pixels were proposed to keep the system
light and thin. Embedding the optical sensors onto a thin film
transistor array substrate was first proposed for 2D touch
applications by W. D. Boer et al. in 2003.14 As shown in Fig. 1,
in which a black matrix does not block a sensor, the photocurrent will be generated when the sensor receives external light.
Thus, the embedded optical sensor-based system becomes a
kind of 2D touch technology.15
To extend the embedded optical sensor-based system to a
3D interface, light-pen touch was proposed.16–18 However, an
additional light pen was necessary for light-pen touch method.
To achieve a more intuitive and friendlier user interface, a 3D
air-touch system for users to ‘touch and interact’ with the
virtual stereoimages on mobile 3D display was proposed, as
shown in Fig. 2. This scheme is illustrated in this paper.
2
Structure
To achieve the air-touch function, we proposed an embedded
multiwavelength optical sensor-based system with added
multiwavelength angular-scanning illuminators on the display
sides, as shown in Fig. 3. The proposed air-touch system is
composed of a traditional display, multiwavelength embedded
optical sensors, an IR backlight, and multiwavelength angularscanning illuminators. For calculating the 2D (x and y) posi-
Received 05/05/13; accepted 10/09/13.
G-Z. Wang, and T-S. Chang are with the Department of Electronics Engineering and Institute of Electronics National Chiao Tung University Taiwan, China.
Y-P. Huang is with the Department of Photonics and Display Institute National Chiao Tung University Taiwan, China; e-mail: boundshuang@mail.nctu.edu.tw.
© Copyright 2014 Society for Information Display 1071-0922/14/2109-0190$1.00.
Journal of the SID 21/9, 2014
381
FIGURE 1 — Schematic structure of an embedded optical sensor onto a thin film transistor substrate, and a depiction
of the sensed image.
FIGURE 2 — Schematic of the embedded optical sensor panel for different 3D interactive applications.
FIGURE 3 — Cross-section of the bare-finger 3D interactive system in (a) Infrared (IR) backlight illuminating mode
(for x, y determination) and (b) IR angular-scanning mode (for z determination).
tion of fingertip, a planar IR backlight was designed to pass
through the whole panel so that it can be reflected by the fingertip for detection, as shown in Fig. 3(a). For calculating the
depth (z) information, a multiwavelength angular-scanning
382
Wang et al. / Bare finger 3D air-touch system
light bar, which can be placed at the edge of the panel, was
proposed, as illustrated in Fig. 3(b).
However, to temporarily scan the depth (z) information with
single wavelength, the operation of the system will suffer from
the sensing rate. Thus, to increase the sensing rate for real-time
operation, we propose a multiwavelength sensing method to
reduce the capturing time by using parallel-processing
concepts. Before we describing this ‘multi-wavelength’ sensing,
the timing diagram (Fig. 4) of the bare finger 3D interactive
system with ‘single-wavelength’ sensing must be established.
At first, the IR backlight and IR angular-scanning illuminators are synchronized with embedded optical sensors.
During the first sensing frame (Frame 0), the IR backlight
passes through the panel twice, and the optical sensors capture
the reflected light for detecting the 2D position (x and y) of
the fingertip. The working time (tx,y) for this detection is t. Then,
during the sensing frames (Frame 1 to Frame n), the IR angularscanning illuminators on two sides of the panel will emit a singlewavelength light at different tilt angles sequentially. These
sensing frames are correlated with tilt angle θ to n × θ. The
working time (tz) for z determination is n × t. Finally, by analyzing the accumulated intensity of the each frame, the scanning
angle to yield the maximum reflectance can be found for the
position of fingertip. With the two axes (x and y) and scanning
angle (θ), the depth (z) information of fingertip can be
calculated. Finally, the 3-axis (x, y, z) information of fingertip
can be obtained over a total working time of (1 + n) × t.
By utilizing the parallel-processing concept, the sensed
images can be captured simultaneously rather than sequentially. The timing diagram of the bare finger 3D interactive
system with multiwavelength sensing is shown in Fig. 5. The
2D axis working time (tx,y) for the multiwavelength sensing
for 2D detection is the same as that for single-wavelength
sensing. However, the depth sensing working time (tz) of the
multiwavelength method of z determination is n/2 × t due to
parallel processing. The concept of parallel processing is that
the depth sensing step can be operated by the red and blue
subpixels at the same time. Therefore, the total depth
sensing working time will be reduced in half. In the following experiment, the proposed multiwavelength sensing is
demonstrated with IR and deep-blue wavelengths for less
interruption of human vision. For this purpose, the optical
sensors were embedded in the red and blue subpixels. The
reflected images of the IR and deep-blue light are filtered
by R and B color filters, respectively.
3
Algorithm
Based on the bare finger 3D air-touch system with
multiwavelength embedded optical sensors and multiwavelength
sequential illuminators, the proposed algorithm can be used to
calculate the 3D (x, y, z) positions of fingertip. The flow chart
of proposed algorithm is shown in Fig. 6. At first, the raw data
are retrieved from embedded optical sensors. To decrease noise
from the environment and the system, a noise suppression
system, including denoising and debackgrounding, is adopted.
FIGURE 4 — Timing diagram of the bare finger 3D interactive system with ‘single-wavelength’ embedded optical
sensors and single-wavelength sequential illuminators.
Journal of the SID 21/9, 2014
383
FIGURE 5 — Timing diagram of the bare finger 3D interactive system with ‘multi-wavelength’ embedded optical
sensors and multiwavelength sequential illuminators.
Next, the intensity of captured image from IR backlight is
accumulated. When the accumulated value is larger than a touch
threshold value, which was determined by experiment calibration, the object (fingertip) will be sensed as touching on the
panel. Then, the full search method can be used to detect
the 2-axis (x and y) position of a touch point. If the accumulated value is smaller than the touch threshold value, the object (fingertip) is treated as a hover of the panel, thus the
proposed bare finger 3D (x, y, and z) touch algorithm can
be used to calculate the 3-axis (x, y, and z) information. The
details of proposed bare finger 3D (x, y and z) touch
algorithm will be described in the following.
The ‘full search’ method (Fig. 7) is used to calculate the
2-axis(x, y) positions of fingertip from the captured image,
384
Wang et al. / Bare finger 3D air-touch system
which is reflect by IR backlight. The full search method
uses a filter to cover the image and sum up the intensity
in order to find the position of maximum accumulated
intensity. Then, the group of images from the IR angularscanning illuminators is processed using a ‘region-based’ algorithm to obtain depth (z) value of hover fingertip, as
shown in Fig. 8.
The ‘region-based’ algorithm classifies the hover region
broadly into three regions: region-1 (overlapping-central),
region-2 (overlapping-wings), and region-3 (non-overlapping).
In region-1 (overlapping-central), the fingertip was at the
center position of display. In other words, the fingertip is
reflected by two side illuminators with the same scanning
angle (θ). Therefore, the intensity versus scanning angle curve
4
FIGURE 6 — Flow chart of the bare finger 3D (x, y, and z) touch
algorithm.
has only one peak, corresponding to the angle (θ) that
yields the maximum reflectance of the fingertip. Finally,
scanning angle (θ) and 2D (x, y) position of fingertip are used
to calculate the depth (z) information of fingertip by a simple
trigonometric relationship (z = x × tanθ). In region-2
(overlapping-wings), the fingertip is reflected by two side
illuminators with different scanning angles (θ1 and θ2).
Therefore, the intensity versus scanning angle curve will has
two peaks. To obtain higher accuracy, the maximum peak
(θ1) that is closest to the fingertip is used to calculate the
depth (z) information. In region-3 (non-overlapping), there
is only one peak. This condition implies that the fingertip
cannot be reflected by the one side illuminator on the nearer
side at the maximum tilt angle (θ1). In the other words, only
the one illuminator on the far side with the smaller scanning
angle (θ2) was used to calculate depth (z) information.
Experiment results
To verify the proposed concept, our multiwavelength sensing
system was implemented on a 4-inch mobile display (Fig. 9) in
which the optical sensors were embedded under red and blue
color filters. Then, we set up the different conditions for proving the accuracy of the 3D (x, y, z) positions of fingertip in this
4-inch mobile display.
In the following steps, the experiment results of (x, y, z) accuracy, the limitation of detecting height of current system,
and the resolution of depth will be discussed. First, the fingertip was placed at the different (x, y) coordinates to analyze the
(x, y) accuracy. The (x, y) detection has an error of less than
2 mm when the fingertip is at center and at the four corners
of the 4-inch mobile display. (Fig. 10)
To analyze the depth (z) accuracy, the fingertip was raised
from 1 to 3 cm over different working regions, as shown in
Fig. 11. The error of depth (z) value was smaller than
3 mm. Therefore, in our experiments, the maximum error
in the (x, y)-plane and the z-plane were 2 mm and 3 mm,
respectively. However, the detected height was limited to
3 cm because of the sensitivity of the embedded
photosensors. As shown in Fig. 12, the reflected IR intensity
decreased to almost zero when approaching 4 cm. The low
sensitivity occurred because of the embedded photo sensor
was made by Si-based material. A Ge-based photo sensor,
which is more sensitive to the IR wavelength, is under
development. Although the sensitivity can be further
improved, the detectable height also can be increased.
Finally, the increment of scanning angle is strongly related
to the resolution of depth value. As shown in Fig. 13, the
resolution of z increases when the increment of tilt angle
decreases. However, a smaller increment of scanning
angle will be required with additional sensing frames to
calculate one fingertip position. The sensing rate also
depends on the signal-to-noise ratio. With a Ge-based
photosensor in an optimized circuit design, the signal-to-noise
ratio can be improved to have higher sensing rate, enhancing
resolution of depth value.
To further verify the 3D gesture application, we tested the
original system with a single wavelength and the proposed system with multiple wavelengths. In an ideal system, the gesture could be fully reconstructed at an infinite sampling
FIGURE 7 — Full search method for 2D (x, y) position.
Journal of the SID 21/9, 2014
385
FIGURE 8 — Flow chart of the region-based algorithm.
FIGURE 9 — Experimental platform with red and blue optical sensors.
rate. However, in the original single-wavelength system, the
sampling rate was only 3 points per second; thus, the gesture recognized by the original system was far from the real
case. In contrast, in the proposed system with IR and deepblue sensors, the capturing rate was as a factor of two of the
386
Wang et al. / Bare finger 3D air-touch system
single wavelength one; thus, the recognized gesture was
much similar to the real S-curve, as depicted in Fig. 14.
Accordingly, a real-time 3D air-touch system could
be achieved through the proposed multiwavelength
sensing system.
FIGURE 10 — Accuracy of (x, y) coordinates in different x/y coordinates.
FIGURE 11 — Accuracy of the depth (z) value of the object at different
working ranges.
FIGURE 12 — Measurement results for the reflected intensity of a fingertip at different depths.
FIGURE 14 — 3D tracking comparison between the single-wavelength
and the multiwavelength results.
5
FIGURE 13 — Relationship between different increments of tilt angle and
accuracy.
Conclusion
In conclusion, we had presented a camera-free 3D air-touch
system with bare finger interaction. It detects hovering, swiping, and virtually ‘touching’ of the 3D images. By embedding
the multiwavelength optical sensors in the display pixels and
adding multiwavelength angular-scanning illuminators on the
edge of the display, the flat panel can sense images reflected
by the fingertip. From the sensed images, the 3D (x, y, z)
position of fingertip can be calculated without complex image
processing. Finally, the experimental results exhibit precise
Journal of the SID 21/9, 2014
387
2D position (x, y) detection at the pixel level and a linear
response to depth (z) from 0 to 3 cm. The detected depth
range can be further increased by improving the sensitivity
of optical sensors. The 3D gesture sensing was also verified.
For current mobile 3D displays, which can only provide few
centimeter depth images, the proposed system, which can
yield 0–3 cm working range, is applicable for the near-field
air-touch of mobile 3D displays. To summarize, the proposed
approach is workable under the single bare finger. However,
the multiple bare-fingers operation will fail because of the
occlusion effect, which means the blocked fingertip cannot
reflect IR light. For overcoming the occlusion issue and achieving 3D multiple bare finger’s functionality, the interpolation and
motion vector methods may be adopted in the future.
17 G. Z. Wang et al., “A Virtual Touched 3D Interactive Display with Embedded Optical Sensor Array for 5-axis (x, y, z, θ, ϕ) Detection,” SID Symp.
Digest 42, 737–740 (2011).
18 Y. P. Huang et al., “Three dimensional virtual touch display system for
multi-user applications,” Early Access by IEEE/OSA Jol. Of Display Technology, (2013).
Guo-Zhen Wang received an M.S. degree from the
Display Institute at the National Chiao Tung University (NCTU), Hsinchu, Taiwan, R.O.C., in 2008 and
is currently working toward a Ph.D. at the Department of Electronics Engineering, National Chiao
Tung University (NCTU). His current research
involves developing 3D interaction systems and
focuses on image processing and computer architecture technologies.
Acknowledgments
The authors would like to acknowledge financial support
from the National Science Council (Grant No. NSC1012221-E-009-120-MY3) of the Republic of China and
hardware support from AU-Optronics.
References
1 K. Bredies et al., “AVI ’08: Proceedings of the working conference on
Advanced visual interfaces 466–469,” (2008).
2 M. Tsuboi et al., “Design conditions for attractive reality in mobile-type 3-D
display,” J. Soc. Inf. Disp. 18, 698–703 (2010). DOI: 10.1889/JSID18.9.698.
3 Y. P. Huang and C. W. Chen, “Superzone Fresnel Liquid Crystal Lens for
Temporal Scanning Auto-stereoscopic Display,” J. Disp. Technol. 8, 650–655
(2012) DOI: 10.1109/JDT.2012.2212695.
4 Y. P. Huang et al., “2-D/3-D switchable autostereoscopic display with
multi-electrically driven liquid-crystal (MeD-LC) lenses,” J. Soc. Inf. Disp.
18, 642–646 (2010). DOI: 10.1889/JSID18.9.642.
5 Y. P. Huang et al., “Autostereoscopic 3D Display with Scanning Multielectrode Driven Liquid Crystal(MeD-LC) Lens,” J. 3D Res. 1, 39–42
(2010). DOI: 10.1007/3DRes.01(2010)5.
6 C. Hand, “A Survey of 3D Interaction Techniques,” Comput. Graph.
Forum 16, 269–281 (1997).
7 C. Brown et al., “A system LCD with integrated 3-dimensional input
device,” SID Symp. Digest 41, 453–456 (2010).
8 R. Ott et al., “Advanced virtual reality technologies for surveillance and
security applications,” Proceedings of the 2006 ACM international conference on Virtual reality continuum and its applications, 163–170 (2006).
9 K. N. Tarchanidis and J. N. Lygouras, “Data glove with a force sensor,”
IEEE Trans. Instrum. Meas. 52, 984–989 (2003).
10 A. Olwal et al, “An immaterial, dual-sided display system with 3D interaction,” Virtual Reality Conference, 279–280 (2006).
11 T. Schlömer et al., “Gesture recognition with a Wii controller,” Proceedings of the 2nd international conference on Tangible and embedded
interaction, 11–14 (2008).
12 B. Lange et al., “Initial usability assessment of off-theshelf video game
consoles for clinical game-based motor rehabilitation,” Phys. Ther. Rev.
14, 355–363 (2009).
13 T. Shiratori and J. K. Hodgins, “Accelerometer-based User Interfaces for
the Control of a Physically Simulated Character,” ACM SIGGRAPH Asia,
123 (2008).
14 W. d. Boer et al., “Active matrix LCD with integrated optical touch
screen,” SID Symp. Digest 34, 1494–1497 (2003).
15 A. Abileah et al., “Integrated Optical Touch Panel in a 14.1 AMLCD,”
SID Symp. Digest 35, 1544–1547 (2004).
16 H. Y. Tung et al., “Multi-user and Multi-touch System for 3D-interactive
Display,” SID Symp. Digest 42, 1834–1837 (2011).
388
Wang et al. / Bare finger 3D air-touch system
Yi-Pai Huang received his BS degree from
National Cheng Kung University in 1999
and earned a PhD in Electro-Optical
Engineering from the NCTU in 2004. He is
currently a full-time Associate Professor in
the Department of Photonics and Display
Institute, NCTU, Taiwan. He was also a visiting
Associate Professor at Cornell University from
2011 to 2012. Additionally, he is the chairman
of the SID Taipei Chapter, the Chair of the SID
Applied Vision subcommittee, and a senior
member of SID. His expertise includes 3D
displays and interactive technologies, display optics and color science, and micro-optics. In these fields, he has published
over 150 international journal and conference papers (including 60 SID
conference papers and 17 invited talks) and has been granted 40 patents,
with another 51 currently publicly available. In addition, he has received
the SID’s distinguished paper award three times (2001, 2004, and 2009).
Other important awards include the 2012 National Youth Innovator Award
of the Ministry of Economic Affairs, the 2011 Taiwan National Award of Academia Inventors, the 2010 Advantech Young Professor Award, the 2009
Journal-SID Best Paper of the Year award, and the 2005 Golden
Dissertation Award from the Acer Foundation.
Tian-Sheuanm Chang (S’93–M’06–SM’07) received BS, MS, and PhD degrees in electronic
engineering from NCTU, Hsinchu, Taiwan, in
1993, 1995, and 1999, respectively. From
2000 to 2004, he was a Deputy Manager with
the Global Unichip Corporation, Hsinchu,
Taiwan. In 2004, he joined the Department of
Electronics Engineering, NCTU, where he is
currently a Professor. In 2009, he was a visiting
scholar at Interuniversity Microelectronics
Centre, Belgium. His current research interests
include system-on-a-chip design, very-largescale integration signal processing, and
computer architecture. Dr. Chang received the
Excellent Young Electrical Engineer award from
the Chinese Institute of Electrical Engineering in 2007, and the Outstanding
Young Scholar award from the Taiwan IC Design Society in 2010. He has
been actively involved in many international conferences, either as a part
of an organizing committee or as a technical program committee member.
He is current an Editorial Board Member of the Institute of Electrical and Electronics Engineers Transactions of Circuits and Systems for Video Technology.