Inkjet-Printed Flexible Sensate Surfaces and the Design of Expressive Multimodal
Interactive Space
PhD Thesis Proposal in Media Arts & Sciences
Nan-Wei Gong
Responsive Environments Group, MIT Media Lab
nanwei@media.mit.edu
Oct 2012
Executive Summery
We live in a world where everyday artifacts are designed and augmented as media
interfaces. Technologies are created based on this mission, which enable everyday
objects to sense, interact, and communicate with us. Since the design and
deployment of any interactive sensing system requires pre-defined content and
sensor mapping between the hardware and software systems, having a platform
that is low-cost but highly customizable, flexible, and capable of multimodal
sensing and ad hoc sensing alteration will present great opportunities for the
development of novel interactive applications.
I envision a platform that provides designers and engineers to construct
interactive space with a graphic design approach. A library of sensing element,
which allows users to define the resolution of this element in any computer-aided
design (CAD) software and implement the area and shape of the sensing element
matrix while integrating the artistic aspect of design into our everyday objects. A
sensing surface can be quickly printed with a conductive inkjet printer and the
shape and modalities can be modified as the product development and interaction
design progress.
In this thesis, I will explore the creation of low-cost flexible electronic skin
specifically designed for customized human-computer interaction (HCI) platforms.
The case studies include my previous work in flexible electronics, which
demonstrate the idea in different scales and scenarios. My proposed thesis work
aims at constructing a toolkit for customizable sensate surface that combines
additive and subtractive fabrication methods with sensing capabilities especially
designed for HCI gestural detection such as pressure, touch, folding, proximity
sensing and galvanic skin response. The work will be evaluated with a series of
user studies and workshops that could demonstrate the usability and advantage of
transforming planar electronics into systems that can be wrap, bend, cut and paste
for low-cost customized HCI designs.
Inkjet-Printed Flexible Sensate Surfaces and the Design of Expressive Multimodal
Interactive Space
PhD Thesis Proposal in Media Arts & Sciences
Nan-Wei Gong
Responsive Environments Group, MIT Media Lab
nanwei@media.mit.edu
Oct 2012
_______________________________________________________________________________
Joseph A. Paradiso
Associate Professor of Media Arts and Sciences, MIT Media Lab
_______________________________________________________________________________
Pattie Maes
Professor of Media Arts and Sciences, MIT Media Lab
_______________________________________________________________________________
Jürgen Steimle
Head of the Embodied Interaction Group, Max Planck Institute for Informatics, Germany
1. Introduction
As computer systems become more intelligent and complex, efforts to create
simple and intuitive user interfaces become progressively important. While many
systems were built for users to easily embed sensing functionalities and establish
connectivity between smart objects, the rigid, planar form factor of traditional
electronics remains a major challenge for the integration between hardware
systems and the desired form, size and shape. This constrains the associated
circuitry not only in terms of physical flexibility, but also in terms of surface area
– because they are rigid, PCBs that are larger than around 1 m2 are typically
impractical because of the high cost and they are hard to transport and deploy.
The latest development of low-cost conformal, stretchable and flexible electronics
manufacturing processes open up new possibilities to the exploration of creating
highly customizable, affordable and reusable platforms for pervasive sensing.
The objective of my research is to develop a framework for the general circuit
design and manufacturing principals of inexpensive, high resolution, flexible and
multimodal sensing surface, which depends only on conductive inkjet printing.
The motivation of such framework is to provide a highly accessible sensing
interface that can be designed and manufactured in every household with any
graphic software combined with a conductive inkjet printer. With this system, we
are able to rethink the design of interactive space with a graphic approach –
sensing elements are presented in graphic libraries that can be manipulated in
color, size, and function. And the output printouts are handled like a paper-like
medium, which can be cut, paste, fold and bend around objects.
My previous projects have demonstrated the ability of using a universal sensing
element design for detecting multiple inputs for near-surface interaction such as
touch, hovering, pressure, and galvanic skin response; and for environmental
monitoring such as humidity and light. My proposed research goals are as follows:
1. Developing a library of sensing elements for different sensing targets, which
allow flexibility in designing customized user interfaces. Designers and end
users can change the element size and type in a graphic design environment.
2. Building toolkits for customizable user interface (UI) creation based on
additive and subtractive methods, similar to techniques for digital fabrication
or cutting and pasting in traditional craft.
3. Implementing fundamental wiring schemes of printed sensor grids that
support a wide variety of cutting and pasting applications - linear sensor tape
for additive sensing construction and sensor sheet/film for subtractive
construction.
4. Exploring novel user interaction design for flexible UI and displays associated
with the new possibilities combining the flexibility of this surface and the
sensing capabilities such as pressure, touch, proximity, folding and affective
computing such as emotion during the interaction.
2. Background
It has long been a desire in Human-Computer Interaction to give computing
devices and interactive displays a skin-like surface that is covered with sensors, in
order to make them react to users’ input and environmental conditions [1]. My
proposed work leverages the capabilities of flexible electronics, depending only
on conductive inkjet printing. This presents an exciting opportunity for
customizable low-cost flexible “sensor sheet”. In contrast to previous work on
multimodal sensors that add different sensor components for each modality [2-3],
one single array of printed electrodes is capable of sensing multiple modalities.
The roll-to-roll printing process allows inexpensive mass production of sensors
that are basically unrestricted in length. It can also be printed on inkjet printers,
enabling small quantity rapid “sensing” prototyping. The related background
includes the development of sensate surfaces, especially in robotics, the recent
advancement in flexible electronics and lastly digital fabrication and user
customization.
2.1 Flexible Electronics and Sensor Skin Surfaces
The concept of sensate media [1], multimodal electronics skins as dense sensor
networks, was introduced in recent years as the future of sensing and networking
of smart objects [2, 3]. Unlike computer vision based tracking and interaction, a
sensate surface is low-cost and requires less computing power [4]. It also provides
various resolutions depending on the sensory density for application specific
requirements. In order to integrate the sensor design into an everyday object,
researchers started working on the implementation of sensor arrays on flexible
electronics. Research in materials and mechanics for flexible and stretchable
electronics [5-6] promise an exciting future in wearable computing and smart
object manufacturing, but are still far away from user customized design. The
most widely accessible flexible electronic manufacturing process (copper on
kapton substrate [7], normally used as flexible connectors, keyboard switch
matrixes) is expensive and has size constraints. However, recent advancements in
manufacturing based on flexible films (conductive inkjet printing) [8] coupled
with conventional rigid components are opening up new possibilities in the design
of a large, flexible and cheap substrates for circuitry. Resent work on flexible
sensor surfaces include optical detection such as FlexAura [9], a flexible range
sensor based on IR LEDs and phototransistors; Sugiura et. al [10] designed a
stretchable tangential force measurement based on measuring IR passing through
an elastic membrane. The unMousePad [11] is an interpolating force-sensitive
resistor sheet for multi-touch pressure input. Our approach aims at providing a
one-layer thin sensor printout without extra hardware components on the surface.
The “sensors” can be easily accessible and produced by end users from either
ordering conductive ink printing services online or printing at home.
2.2 Augmenting Smart Objects with Near Surface Sensing
The advantage of having a thin layer of printed sensor is the ability of quickly
design, modify, and reiterate the shape and size while embedding sensing
capabilities into smart objects. Augmenting smart objects with sensing abilities,
especially touch input has been demonstrated on many input devices for
extending interaction capabilities [12]. Mouse 2.0 [13] demonstrated a series of
new designs to support traditional mouse with additional touch sensing on the unutilized surface. Wigdor et al. presented Under the Table Interaction [14], which
explores the design space of a two-sided interactive touch table. Other researchers
attempted to capture users’ intention by detecting the different ways of implicit
grasp inputs users hold their devices. Examples include Graspables [15], the bar
of soap [16], and multi-touch pen for context sensing [17-18]. My proposed
research can serve as a great platform for prototyping smart objects for gesture
recognition and interaction and enhancing the input capabilities of existing
devices. Also, it allows designers and end users to print their desired sensing
shape cheaply, which can be folded, cut and integrated with rapid prototyping
technologies.
2.3 Toolkits for physical computing and prototyping
Many toolkits have been developed to support physical computing during
prototyping process. Traditionally, designers use sketches to demonstrate user
interface ideas and then prototype, test, analyze and then redesign. Toolkits like
the Phidgets [19], d.tools [20], Calder [21], iStuff [22], LEGO Mindstorms
[23], .NET Gadgeteer [24] and Maaestro [25] provide physical widgets for
designer to interface and iterate during the design process. Platforms like Midas
[26] enable end users to fabricate custom capacitive touch sensors to prototype
interactive objects. The problem with most toolkits is that they are disconnected
from the design of physical objects and the fabrication process. My proposed
research not only provides a fast prototyping sensing surface, also emphasis on
the integration of functional element with industrial design practice, as well as
different techniques for physical constructions.
3. Proposed Research
My thesis will present several projects in details to enable the HCI community to
use this architecture and for developing novel sensors that are based on its generic
implementation. To show the applicability, I will demonstrate a number of
example applications in desktop, mobile, tangible and wearable computing that
demonstrate the utility of this technology. Applications include more expressive
controllers for computing devices, enhanced touch screens, sensate paper, smart
handles/objects, and smart wearable devices. The technical and theoretical
foundations of my thesis proposal are demonstrated in four projects, which cover
different use cases to support the theory and application domains of my thesis.
Below are projects that are already implemented as demonstrations of this
infrastructure in different scale and application areas. These examples include
large area electromagnetic field sensing floor for locating people and devices [27],
customizable sensate surface for music control [28], linear environmental sensing
strip [29] and using printed conductive traces as a template for multimodal
sensing applications. I have demonstrated the fundamental framework of the
design process, technical infrastructures in those projects as the building blocks
for my proposed research.
3.1 Technical Framework
I. Large scale coverage - flexible floor sensor system
In this project I experimented
in
the
design
and
implementation of a new
versatile, scalable and costeffective sensate surface for
large-scale coverage. The
system is based on conductive
inkjet
technology,
which
allows
capacitive
sensor
electrodes and different types
of RF antennas to be cheaply
printed onto a roll of flexible
substrate that may be many meters long. By deploying this surface on (or under) a
floor it is possible to detect the presence and whereabouts of users through both
passive and active capacitive coupling schemes. We have also incorporated GSM
and NFC electromagnetic radiation sensing and piezoelectric pressure and
vibration detection. A number of experiments were performed for evaluating this
sensing performance on a 2.5m x 0.3m hardware test-bed. Our design
incorporates many different sensing capabilities based on the ability to create a
large-scale non-rigid substrate with conductors printed onto it at a relatively low
cost. This was not previously practical - it now opens the possibility of easily
deploying a large-area surface sensing system. We described the design and
implementation of passive and active capacitive sensing, coupled with GSM and
NFC RF signal pickup – all based on copper electrodes and antennas printed on
the substrate. The experiments show the possibilities of using this platform for
gait analysis through footstep signatures localization from the personal mobile
device signals.
II. Customizable Sensate Surface for Music Control
To demostrate the importance of
combining graphic design and
sensing components for creating
interactive
interfaces
as
“functional decoration” while
preserving the asthetics of a music
instrument, we experimented with
printint music controllers that can
embed capacitive touch sensing
and pressure sening on the surface
of a ukulele. This novel music control sensate surface enables integration between
any musical instruments with a versatile, customizable, and cost-effective user
interface. The sensate surface is based on conductive inkjet printing technology
which allows musicians to design a functional decoration, which can be mapped
to control inputs in a software synthesizer or an effect paddel. The high dynamic
range capacitive sensing electrodes can not only infer touch, but near-range, noncontact gestural nuance in a music performance. With this sensate surface, users
can “cut” out their desired shapes, “paste” the number of inputs, and customize
their controller interface, which can then send signals wirelessly to effects or
software synthesizers. I seek to find a solution for integrating the form factor of
traditional music controllers seamlessly on top of one’s music instrument and
meanwhile adding expressiveness to the music performance by sensing and
incorporating movements and gestures to manipulate the musical output. The
example implementation was performed on an electric ukulele and provide
several design examples to demonstrate the versatile capabilities of this system.
My main interest lies in developing a system for producing new forms of
performances and instruments through empowering musicians with the ability to
improve, modify, and extend the capabilities of traditional instruments.
III. Environmental Sensing with Sensor strip
Fine grid environmental sensing has always
been a major challenge for the design and
integration of sensor systems. Deploying a
dense sensor network nowadays involves
implementing individual sensors and
complex wiring between the processor and
sensors. The cost of individual sensor
manufacturing and system integration is
high; hence, in most cases, engineers use
interpolation to compensate the missing
gaps. In this project, I demonstrated a lowcost, large coverage sensing tape based on
roll-to-roll manufacturing where the
substrate, electronic circuitry, electrode, and
sensing elements are implemented in the
same process. Different sensing elements,
either electromagnetic
field
sensing
components that are embedded in the copper
trace design or chemical sensing that is
based on coating resistive polymers on top of the copper trace “template” for
multiple sensing targets. This one-step fabrication process opens up the possibility
for inexpensive dense sensor system deployment that is unlimited in length and
also different applications for large area environmental sensing. In order to have
the flexibility of grid compactness for both narrow and wide area coverage, we
decided to experiment with a “tape” structured linear geometry with dense
sensing capabilities. This shape enables the user to physically manipulate and
configure the sensing density for varies measurement. The sensory units can be
more precisely placed in a targeted location, like how the density and receptors’
functionalities on our skin vary based on the location of the skin. We present the
possibility of creating sensing elements by adding a layer of conductive polymer,
which can potentially be integrated into the same roll-to-roll manufacturing
process as the circuitry printing process in the future. Similar approach can be
applied to other resistive sensors and we envision a sensor strip with multiple
functionalities and can be used in not only in the electromagnetic field sensing but
also scenarios such as smart material for building-scaled water leakage detection.
IV. Printed Sensors for Expressive Multimodal Interactions with
Smart Objects and Surfaces
In this project, I demonstrated the
possibility of printing multimodal
sensor sheets for near-surface
interactions as an enabling
technology for more expressive
interactions on smart objects.
Our prototype leverages the
powerful capabilities of flexible
electronics, depending only on
conductive inkjet printing. This
presents an exciting opportunity for customizable low-cost flexible “sensor sheet”.
In contrast to previous work on multimodal sensors that add different sensor
components for each modality, one single array of printed electrodes is capable of
sensing multiple modalities. The roll-to-roll printing process allows inexpensive
mass production of sensors that are basically unrestricted in length. It can also be
printed on inkjet printers, enabling small quantity rapid “sensing” prototyping.
For use on displays or on transparent surfaces, the sensor can be made transparent
by using a transparent substrate and transparent conductive materials such as
Indium Tin Oxide (ITO). The implementation of this system is presented in detail,
to enable the HCI community to use this architecture and for developing novel
sensors that are based on its generic implementation. Applications include more
expressive controllers for computing devices, enhanced touch screens, sensate
paper, smart handles/objects, and smart wearable devices.
3.2 Research Method
Based on the fundamental technical work as described from the precious section, I
would like to continue the research for my thesis work as follows. First, I will
combine the technical aspects of each project and construct a robust and modular
hardware toolkit for easier implementation of processing the sensor inputs from
the printouts. This toolkit will be evaluated by a side-by-side comparison with
existing platforms such as multi-touch input panel, pressure sensors, remote
sensing (hovering), galvanic skin response, large-area interactive sensing
platforms. Second, I will compile a list of sensing patterns and methods to apply
chemicals for more diverse sensing possibilities. Once both goals are achieved, I
will perform a user study among UI designers and non-technical users in a series
of workshops. In the workshops, each participant will be given a printed circuit
board that is per-programed with the ability of sending input information to a PC.
Participants will be able to use the sensor graphic library to construct an
interactive surface, which is designed for near-surface interaction such as touch,
pressure, hovering… etc. A conductive inkjet printer will be provided at the
workshop for supporting several iterations of designs.
The evaluation of my work will be both quantitative and qualitative. The
quantitative evaluation will be focused on the resolution, accuracy, and response
time of the system. Several interactive applications integrating the system with
consumer electronics devices, for example, flexible displays and mobile phones
will be used as a measure of demonstrating the new possibilities flexible sensate
surfaces presents. The qualitative aspect will be conducted during the workshops.
I will interview the participants about the usefulness of this system in rapid
prototyping and the value of designing interactive space with a graphic design
approach.
4. Contributions
The major contribution of my thesis is to provide a technical platform for the
design of expressive, multimodal interactive space that is highly customizable,
flexible and inexpensive for everyday design. The technical contributions are as
follows: 1) Create multimodal sensing element patterns and the associate circuitry
for fast sampling and multiplexing of each modality. 2) Combine emerging
technologies, i.e. flexible electronics, conductive inkjet printing, as a new way of
HCI interface design. 3) Rethink the way of sensing through a graphic design
approach and create sensate surfaces that can be manipulated like a sheet of paper
that can be cut, paste, fold and bend around smart objects. 4) Develop a low-cost
sensate surface that is highly customizable and scalable for large area interaction.
5. Timeline
Dec 2012
thesis proposal critique
Jan 2013
finish first iteration of hardware toolkit
Feb 2013
host workshop for design evaluation
Mar 2013
second iteration of hardware and software design
Apr 2013
submit results to UIST
June 2013 finish thesis first draft for committee members
July 2013
dissertation defense
Aug 2013
submit final thesis
6. References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
Rekimoto, J., Smartskin: An infrastructure for free hand manipulation on
interactive surfaces. Proceedings of CHI 2002, pp. 113-120.
Paradiso, J.A., Lifton. J., and Broxton, M., Sensate Media - Multimodal
Electronic Skins as Dense Sensor Networks, BT Technology Journal, Vol.
22, No. 4, October 2004, pp. 32-44.
Stiehl, W.D., Breaeal, C., A Sensitive Skin for Robotic Companions
Featuring Temperature, Force, and Electric Field Sensors, 2006, IEEE/RSJ
International Conference on Intelligent Robots and Systems, pp.1952-1959.
Mistree, B.F.T., and Paradiso, J.A., ChainMail – A Configurable Multimodal
Lining to Enable Sensate Surfaces and Interactive Objects, in Proc. of the
TEI 2010, pp. 65-72.
J. Rogers, T. Someya, and Y. Huang, Materials and Mechanics for
Stretchable Electronics Science 327 (5973), 26 Mar 2010
T. Sekitani et al., A Rubberlike Stretchable Active Matrix Using Elastic
Conductors. Science 321(5895), 12 Sep 2008
Kreuz, J. A. et al., Advanced Flexible Dielectric Substrates for FPC/TAB
Applications, application note from dupont.com.
http://conductiveinkjet.com
Liu, S. and Guimbretière, F., FlexAura: a Flexible Near-Surface Range
Sensor, Proceedings of UIST 2012.
Sugiura, Y., Inami, M., Igarashi, T., A Thin Stretchable Interface for
Tangential Force Measurement, Proceedings of UIST 2012.
Rosenberg, I. and Perlin, K. The UnMousePad: an interpolating multi-touch
force-sensing input pad. In ACM SIGGRAPH 2009, Article 65.
Gong, N.-W., Hodges, S. and Paradiso, J. A.” Leveraging Conductive Inkjet
Technology to Build a Scalable and Versatile Surface for Ubiquitous
Sensing” InProc. of UbiComp '11, pp 45-54.
Villar, N et al., Mouse 2.0: multi-touch meets the mouse. In Proceedings of
UIST’09, pp. 33-42.
Wigdor, N., et al., Under the table interaction. In Proceedings of UIST '06,
pp. 259-268.
Taylor, B. T. and Bove, M. Jr., Graspables: grasp-recognition as a user
interface. In Proceedings of CHI '09, pp. 917-926.
B. T. Taylor and V M. Bove. 2008. The bar of soap: a grasp recognition
system implemented in a multi-functional handheld device. In CHI EA '08.
Pp 3459-3464.
Sun, M. et al., Enhancing naturalness of pen-and-tablet drawing through
context sensing. Proceedings of ITS 2011, pp. 83-86.
18. Song, H. et al., Grips and gestures on a multi-touch pen. Proceedings of CHI
2011, ACM Conference on Human Factors in Computing Systems. pp. 13231332.
19. http://www.phidgets.com/
20. http://research.microsoft.com/en-us/projects/gadgeteer/
21. Hartmann, B. et al., 2006. Reflective physical prototyping through integrated
design, test, and analysis. In Proceedings of the ACM symposium on user
interface software and technology (UIST). pp. 299–308.
22. D. Avrahami , S. E. Hudson, Forming interactivity: a tool for rapid
prototyping of physical interactive products, Proceedings of the conference
on Designing interactive systems: processes, practices, methods, and
techniques, June 25-28, 2002, London, England
23. R. Ballagas, M. Ringel , M. Stone , J. Borchers, iStuff: a physical user
interface toolkit for ubiquitous computing environments, Proceedings of the
SIGCHI conference on Human factors in computing systems, April 05-10,
2003, Ft. Lauderdale, Florida, USA
24. LEGO
Mindstorms
Robotic
Invention
System.
http://www.mindstorms.lego.com/
25. Maestro. Cybelius. http://www.cybelius.com/products
26. V. Savage, X. Zhang, and B. Hartmann. 2012. Midas: fabricating custom
capacitive touch sensors to prototype interactive objects. In Proceedings of
the 25th annual ACM symposium on User interface software and technology
(UIST '12).
27. N.-W. Gong, S. Hodges and J. A. Paradiso,” Leveraging Conductive Inkjet
Technology to Build a Scalable and Versatile Surface for Ubiquitous
Sensing” InProc. of the 13th international conference on Ubiquitous
computing (UbiComp '11), 45-54.
28. N.-W. Gong, N. Zhao and J. A. Paradiso,” A Customizable Sensate Surface
for Music Control,” In Proc. of the 12th international conference on New
Interfaces for Musical Expression (NIME 12), pp. 417-420.
29. N.-W. Gong, C. Y Wang and J. A. Paradiso,” Low-cost Sensor Tape for
Environmental Sensing Based on Roll-to-roll Manufacturing Process” In
Proc. of the IEEE SENSORS conference, 2012