rishika alva
G E S T U R E B A S E D T E X T F O R M AT T I N G F O R
M U LT I T O U C H TA B L E
G E S T U R E B A S E D T E X T F O R M AT T I N G F O R M U LT I T O U C H
TA B L E
rishika alva
Department of Human-Computer Interaction
University of Paderborn
Rishika Alva: Gesture Based Text Formatting for Multitouch Table ©
01.09.2013
supervisors:
Prof. Dr. Gerd Szwillus
Prof. Dr. Gitta Domik
location:
Paderborn, Germany
ABSTRACT
Multitouch computers have rapidly grown in popularity since the
beginning of the 21st century [15]. Advancements in the area of multitouch technology have made computers more interactive than ever
before. Gesture based interaction is one of the ways to communicate with multitouch computers, which enables a user to interact
with multitouch devices, with her hands. Being able to communicate
with computers with bare hands has brought the physical and digital
worlds very close, giving end-users an enriched experience.
Text formatting is an important aspect of communication using text
messages. A well formatted text makes the underlying message better
looking, better legible and more meaningful.
The crux of this thesis is to define the concept of using hand gestures to perform text formatting on multitouch tabletop computers.
The goal is to illustrate, how the usage of gestures can alleviate the
problems existing in traditional menu/button based text formatting.
In order to make this illustration clear, the goal is to implement a few
commonly used text formatting options as gestures for multitouch
tabletop computers and comparing their usage with the usage of
traditional menu/button based text formatting.
v
ACKNOWLEDGMENTS
vii
CONTENTS
1
Introduction
1.1
1.2
1.3
2
3
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
5.2
.
.
.
.
.
11
11
13
15
15
19
Example Word Processing Applications . . . . . . . . .
Microsoft Word . . . . . . . . . . . . . . . . . . . . . . .
4.2.1 Text Formatting Options . . . . . . . . . . . . . .
25
25
25
27
Text Formatting . . . . . . . . . . . . . . . . . . .
3.1.1 Text Formatting in Electronic Documents
Multitouch Computing . . . . . . . . . . . . . . .
3.2.1 Multitouch Tabletop Computers . . . . .
3.2.2 Gesture Based Interaction . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
31
31
31
32
33
39
45
.
.
.
.
.
.
49
49
51
51
51
53
56
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . .
Future Work . . . . . . . . . . . . . . . . . . . . . . . . .
59
59
60
The Concept . . . . . . . . . . . . . . . . . . . .
5.1.1 Text Selection . . . . . . . . . . . . . . .
5.1.2 Text Formatting using GBTF . . . . . .
5.1.3 Gestures Requiring Text Selection . . .
5.1.4 Gestures Not Requiring Text Selection .
GBTF vs Menu Based Formatting . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Illustration Of The Usage of GBTF
6.1
6.2
6.3
System Design . . . . . . . . .
Implementation . . . . . . . .
6.2.1 Tools Used . . . . . . .
6.2.2 Behaviour . . . . . . .
6.2.3 User Interface . . . . .
User Study And The Results .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Conclusion
7.1
7.2
1
1
2
3
4
6
7
Gesture Based Text Formatting (GBTF)
5.1
7
.
.
.
.
.
Traditional Menu/Button Based Text Formatting
4.1
4.2
6
.
.
.
.
.
Basis
3.2
5
.
.
.
.
.
Related Work
3.1
4
Typography . . . . . . . .
Motivation . . . . . . . . .
1.2.1 Problem Statement
1.2.2 Gist Of Solution . .
Organization . . . . . . . .
Bibliography
63
ix
LIST OF FIGURES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
x
Mister T. MTT computer at the University of Paderborn.
Pinch-To-Zoom, Pan and Rotate Gestures. . . . . . . . .
An example of a word-processing application: Microsoft
Word. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Text editors with some gesture support: Tyype text editor.
Text editors with some gesture support: Rich Notes text
editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Anatomy of Typography [8]. . . . . . . . . . . . . . . .
Examples of font families [10]. . . . . . . . . . . . . . .
Anatomy of a Miltitouch Computer [3]. . . . . . . . . .
Frustrated Total Internal Reflection Technique [7]. . . .
Laser Light Plane Technique [7]. . . . . . . . . . . . . .
Diffused Illumination Technique [7]. . . . . . . . . . . .
Diffused Surface Illumination Technique [7]. . . . . . .
Light Emitting Diode - Light Plane Technique [7]. . . .
Tap Gestures [2]. . . . . . . . . . . . . . . . . . . . . . .
Rotate Gestures [2]. . . . . . . . . . . . . . . . . . . . . .
Scale Gestures [2]. . . . . . . . . . . . . . . . . . . . . . .
Scroll Gestures [2]. . . . . . . . . . . . . . . . . . . . . .
Drag Gestures [2]. . . . . . . . . . . . . . . . . . . . . . .
Split Gestures [2]. . . . . . . . . . . . . . . . . . . . . . .
Anchor Gestures [2]. . . . . . . . . . . . . . . . . . . . .
Flick Gestures [2]. . . . . . . . . . . . . . . . . . . . . . .
3D Gestures [2]. . . . . . . . . . . . . . . . . . . . . . . .
An example of a word-processing application: Microsoft
Office. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Formatting options on Microsoft Word. . . . . . . . . .
More formatting options on Microsoft Word. . . . . . .
Gesture to achieve superscripts. . . . . . . . . . . . . . .
Gesture to achieve subscripts. . . . . . . . . . . . . . . .
Gesture to achieve bold text. . . . . . . . . . . . . . . . .
Gesture to achieve italicised text. . . . . . . . . . . . . .
Gesture to decrease text size. . . . . . . . . . . . . . . .
Gesture to increase text size. . . . . . . . . . . . . . . . .
Gesture to achieve left aligned text. . . . . . . . . . . . .
Gesture to achieve right aligned text. . . . . . . . . . . .
Gesture to achieve center aligned text. . . . . . . . . . .
Gesture to achieve justified text. . . . . . . . . . . . . .
Gesture to achieve right indented text. . . . . . . . . . .
Gesture to achieve left indented text. . . . . . . . . . . .
Flowcharting depicting the design of the system. . . . .
2
3
4
8
9
12
13
16
17
17
18
18
19
20
21
21
21
22
22
22
23
23
26
27
27
34
35
36
37
38
39
40
41
42
43
44
45
50
The Gesture Based Text Editor.
The Circular Menu. . . . . . . .
The Colour Selection Menu. . .
The Hyper Link Menu. . . . . .
39
40
41
42
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
54
55
55
56
ACRONYMS
GBTF
Gesture Based Text Formatting
MTT
Multi-Touch Tabletop
FTIR
Frustrated Total Internal Reflection
LLP
Laser Light Plane
DI
Diffused Illumination
DSI
Diffused Surface Illumination
LED
Light Emitting Diode
LED-LP
Light Emitting Diode - Light Plane
WPF 4.0
Windows Presentation Foundation 4.0
RTF
Rich Text Format
xi
1
INTRODUCTION
With the advent of the 21st century, multitouch computing has seen
great advancements [15]. Due to extensive research being carried out
and due to the great inventions in the area of multitouch technology,
computers are inching closer to our physical world at a rapid pace.
Being empowered with the ability to respond to touch, which is one
of the senses on which we humans rely, multitouch computers have
become very resourceful, at the same time making our lives more
enjoyable. Multitouch devices come in many shapes, sizes and forms.
Examples of multitouch devices are multitouch tabletop (MTT) computers, mobile phones, tablet computers and music players. Computers
have not just evolved to respond to human touch but today’s computers are also able to respond to commands provided by humans
using their other senses. With such large scale advancements, it is
hard to imagine how an electro-mechanical machine called computer,
that occupied spaces equivalent to multiple room lengths only a few
decades ago, has come this far.
Computers are used for a large array of purposes today. They are
used in gadgets, used to keep in touch with loved ones, used to keep
oneself healthy, used for entertainment and so much more. Due to
such distinct omnipresence, they play an integral role in our society.
Although decades have passed, in this great evolution of computers, one aspect of using a computer has remained nearly unchanged,
namely: the ability to express one’s thoughts with electronic text documents. To achieve this, a traditional keyboard (or software emulated
keyboard on multitouch devices) is widely used. Despite various advancements in computer hardware and software, there has not been a
large breakthrough in this front, with a potential to completely replace
the standard way of inputting text into a computer. Using a keyboard
still remains the quickest way and the easiest way to input text. Due
to this reason, it is likely to remain as the standard way of inputting
text for many years to come, until a very powerful new invention
completely revolutionizes this trend, ending the reign of keyboards
(for instance: a technology which can accurately create electronic
documents just by analysing the thoughts of a human brain).
1.1
typography
Typography is an art and the technique used to depict text on a document in such a way that the text becomes clearer, better visible and
conveys the underlying message quicker. This technique has been
1
2
introduction
Figure 1: Mister T. MTT computer at the University of Paderborn.
in practice for many centuries now and has played a vital role in
handwritten paper-based text documents [14] [22] [16]. It is so powerful that representing a same sentence using different typographic
styles can completely transform its meaning. The role of typography
has remained as significant in an electronic text document as with
its paper-based counterpart. This master thesis work mainly focuses
on these areas: MTT computers, gesture interaction technique for MTT
computers and typography in electronic text documents.
1.2
motivation
An MTT computer (seen in Figure 1) can be communicated with using different interaction techniques like gesture interaction, tangibleobjects interaction, stylus interaction and so on. Gesture interaction is
one of the most flexible and also popular techniques to interact with a
multitouch device. Using this technique, a user is enabled to perform
tasks on a multitouch device using only her bare hands. There is
absolutely no requirement for traditional input devices like a physical
keyboard or a physical mouse.
Many MTT computers are programmed to identify and respond to
gesture interactions. This makes them truly life-like and enjoyable,
when compared to traditional computers. Gesture based interaction
enables a user to relate closely to a computer and makes her feel as if
the computer is very similar to a physical being. Also, since a user is
able to use certain gestures to perform certain tasks on the multitouch
computer which can be programmed to be similar to those gestures
which she would use to perform an equivalent actions in the real
1.2 motivation
Figure 2: Pinch-To-Zoom, Pan and Rotate Gestures.
world, the entire idea of using gestures as a form of input is attractive
and also convenient to end-users. Several gestures can be defined,
implemented and used on multitouch computers. Figure 2 illustrates
three common gestures, namely: pan, zoom and rotate. These gestures
are used on multitouch devices to manipulate virtual objects (like
photos), in a way very similar to how the real counterparts of these
virtual objects would be manipulated, in the real world.
As gesture based interaction technique is so powerful and flexible,
it is used in many applications which are created for MTT computers.
However the specific idea of using gestures to perform text formatting
(or typography), has not been investigated, to a great extent yet. This
investigation is the primary motivation behind this thesis work.
1.2.1
Problem Statement
Many word processing and text formatting applications supported by
Windows 7, enable users to create text documents on MTT computers
too, due to the multitouch capabilities of Windows 7. Figure 3 shows
an example of a word processing application called Microsoft Word
which is developed and maintained by Microsoft Corporation. There
exist many such word processing applications, which are created and
maintained by other organizations. Namely Libre, MacWrite, Abiword
and so on. Each of these applications is similar in appearance to the
others and has very similar functionality and features too.
When creating a document using a word processor, a user has the
opportunity to format text to enhance the appearance and readability
of the document as a whole. To do so, the user often needs to suspend inputting of text, perform formatting by finding the appropriate
button(s)/menu item(s) and then resume inputting of text after the
formatting is complete. Performing this action on a standard desktop
computer is simple because of the availability of a physical hardware
like a keyboard and a mouse. Often, experienced users make use of
keyboard shortcuts while performing most text formatting, saving a
lot of time which is otherwise required to navigate using the mouse
to perform the same tasks. Although there exist virtual (emulated)
3
4
introduction
Figure 3: An example of a word-processing application: Microsoft Word.
alternatives to the mouse and the keyboard for MTT computers, due
to the lack of hepatic feedback and the feeling of using the physical
device, these alternatives are comparatively cumbersome to use.
Due to these reasons, the process of text formatting often distracts a
user from her main task of putting her ideas onto the document and
forces her to find the appropriate formatting option in between. As a
result, the typing speed and fluency of the user is hindered. If users
were provided with an alternative way to perform text formatting
in a more fluent, easy and better suited way on MTT computers, it
would help the users maintain their typing speed and at the same
time concentrate on the task of expressing ideas through electronic
text documents.
1.2.2
Gist Of Solution
Performing text formatting on MTT applications using the standard
button/menu based approach can be time consuming as well as distracting at the same time. As described in Section 1.2.1, the virtual (or
emulated) alternatives to mice and keyboards are not as easy to use
as their physical counterparts. Mice are often substituted with the tap
gesture, which has disadvantages over using a mouse. Mice are also
substituted with emulated mice (jut like emulated keyboards) which
also have disadvantages. All these disadvantages have been discussed
in Section 1.2.1 and in Section 5.2.
Despite this fact, most multitouch table applications still rely on
them. As a text input technique to most word processing applications
or other such applications on MTT computers, an emulated keyboard
is used as the standard choice. There are some alternatives to emulated keyboards (like the transparent keyboard made out of silicon as
described in [23]), but these are not widely used yet. Therefore, while
1.2 motivation
creating documents on an MTT device, a user needs to spend more
time in just typing the document as compared to what she would
spend on a standard personal computer, simply because of the usage
of an emulated version of input devices which are not as convenient
to use as the real hardware. Therefore if a user would have to perform
text formatting by tapping, she would first have to locate the required
buttons/menu items. This would add complexity in using the system,
resulting in the user feeling inconvenient.
As MTT computers are capable of being programmed to respond
to gestures, the idea is to use gestures as the alternative technique
to perform text formatting. Of course, while using the gestures for
the first few times, the user does need to learn the gestures and
perform them accurately. But once a user is accustomed to all the
gestures, the process of performing text formatting completely blendsin with the process of document creation, allowing the user to type
fluently, quickly and also focus more of her attention on the topic of
the document.
To perform text formatting, all that a user would have to do is
(optionally) select a block of text in the text editing area, then perform
a short gesture which would result in a desired formatting style being
applied. This way, gestures can serve as an ideal way of achieving text
formatting, alleviating the discomfort a user would need to face while
using the standard button/menu based approach.
1.2.2.1 Goals
The goals of this thesis work are as follows:
• To define the concept of using gesture interaction technique to
perform text formatting on an application executing on an MTT
computer. This is the primary goal and also the crux of the thesis.
• To highlight the advantages of this approach, first the standard
button/menu based interaction is investigated thoroughly. This
would give a detailed picture of the disadvantages of using the
standard approach on MTT computers. To illustrate these aspects,
one word processor application, namely: Microsoft Word, will
be looked into in detail.
• To further emphasise the idea of using gestures the goal is to design and implement a system with a few of the commonly used
formatting options with gesture support. This implementation is
aimed at making the concept of GBTF for MTT computers clearer.
• To perform user evaluation of this application and interpreting
the results of this evaluation .
5
6
introduction
1.3
organization
The rest of this document is structured as follows. Chapter 2 briefly
discusses some prior work done by other researchers in the area of
using gestures to perform various tasks on multitouch table as well
as in the area of text processing. This is followed by Chapter 3, which
discusses about various technologies involved in this master thesis
work. After this, Chapter 4 discusses traditional (button/menu based)
text formatting approach. This discussion is carried out using an
example word processing application, which uses traditional button/menu based text formatting technique. Then follows Chapter 5, which
discusses in detail, the actual concept of using gesture based text formatting and its benefits over using the traditional approach. Chapter
6 then proceeds by making this concept clear by discussing the design
and implementation of a system which performs gesture based text
formatting. This system is designed and implemented as a part of
this master thesis. Finally the discussion is concluded by Chapter 7,
which summarises the entire discussion and also points out about the
possible future work which can be performed in this area, to reap the
benefits of using gesture based text formatting on MTT computers.
2
R E L AT E D W O R K
There has been a lot of related work in the areas of text formatting,
word processing and also using gestures as replacement to certain actions on multitouch devices. The idea of designing information using
human factors approach by maximizing the typographic potential is
discussed in [18]. The author discusses that this can be achieved by
preserving traditional typographic practice and at the same enrich
documents further with the recent findings from the perpetual, cognitive and ergonomic sciences. And achieving all this by taking human
factors into account, where human factors refer to that aspect of documents which transcends changes in technology, namely: legibility,
aestheticism and the ergonomic effect. The primary argument of this
work is that designers can always rely on these human factors while
designing documents, irrespective of the changes or advancements
in technology, in the area of document creation. The work is aimed
at designing documents in such a way that are more understandable,
they attract readers and motivate them to act on the information they
read. It was also aimed at making the process of locating information
efficient.
Likewise, there has been more work in the area of usage of typography (or text formatting) within documents. The effect of the usage
of typography on the tone of a document [17] is one such research
work. Here the document under consideration is not an electronic one
but a standard paper based one. The discussion presented suggests
that stylistic variables such as passive and active voice, formal and
non formal punctuation, personal pronouns and verb contractions
influence a user’s perception of the formality of a document. It also
suggests that letter spacing does not effect the formality of a document.
This research work is aimed at helping technical writers feel confident
about how their stylistic decisions while creating the document are
perceived by readers.
There has also been research work in the area of usability of word
processing applications. One such evaluation of the Microsoft Word
application is presented in [21]. This work was carried out to study
the strengths and the weaknesses of Microsoft Word, from an userfriendliness point of view. The authors describe that the process of
evaluation was carried out with the help of surveys and that the
participants of the surveys were asked to answer a questionnaire,
which was designed to gather every user’s experience. This work
suggests that users who participated in this usability evaluation were
satisfied with the product (Microsoft Word). However, it also indicates
7
8
related work
Figure 4: Text editors with some gesture support: Tyype text editor.
that the users were overwhelmed with too many options and features,
which are far more than an average user needed. Due to this aspect,
they found the application complex. In the case multitouch computers,
it is this complexity, which can further add up to the difficulty in
performing text formatting on such word processors. The motivation
behind GBTF is alleviating the users of such burdens.
Using gestures for different other purposes on multitouch computers has also been an active research area. One example of work in this
area is the idea of combining user interface elements and gestures on
a multitouch surface [19]. Here the authors discuss how user interface
elements can be combined with a set of relevant gestures to enable the
usage of multitouch systems in multi-user collaborative innovation
scenarios. For this purpose some custom user interface elements are
created and some gestures which can be used with those elements
are implemented. The authors claim that in the usability tests, their
subjects provided them positive feedbacks about their system and that
multiple users could work smoothly on their system. They say that
the usage of gestures can indeed be useful on multitouch innovation
scenarios. The GBTF system being discussed in this document, takes
the usage of gestures one notch further, in the area of text formatting.
The idea is to minimize (or eliminate) the usage of menu/button based
text formatting from word processing applications and replace them
with gestures, on MTT computers.
While the usage of gesture to perform text formatting on MTT computers (i.e. GBTF) is a novel approach, the idea of using gestures to
perform some other basic tasks on text editors has been around. There
are many such applications created for iOS, Android and other multitouch platforms. Some examples of such applications are Tyype [11]
seen in Figure 4 and Rich Notes [5] seen in 5. Each of these applications use gestures to perform actions during the process of creating a
text document, like inserting images, moving the cursor, moving text
content and some limited text formatting (usually in combination with
related work
Figure 5: Text editors with some gesture support: Rich Notes text editor.
traditional buttons). However, no application or a solution was known
during this thesis work, which used solely gestures to perform entire
text formatting tasks for MTT computers. In this sense, the concept of
GBTF is unique.
9
3
BASIS
The idea of using gestures to perform text formatting on MTT computers, involves three different domains, namely: multitouch technology,
gesture interaction technique and text formatting. Each of these domains, has had its share of great advancements over the past years.
This section primarily introduces these areas. It also briefly discusses
the various know-hows, concepts and terminology of these domains,
which have been used to elaborate the idea of Gesture Based Text
Formatting (GBTF), in Chapter 5 and also used to implement the illustration of GBTF, discussed in Chapter 6.
3.1
text formatting
Text formatting (also known as Typography), is a technique of representing text in such a way that the underlying message which the text
is intended to convey, is very legible and clear. The idea of altering
the shape, size and color of text is not new to this era of electronic
media. It has been around since many millenniums now. It was first
used to make punches and dies, which were used to print currencies
and seals. The idea of using typography to represent complete text
blocks, was first realized in the Phaistos Disc, an enigmatic Minoan
print item from Crete, Greece, which dates between 1850 and 1600 BC
[14] [22] [16].
Typography comprises of different elements, which are used, at
times individually and at times in combination with each other, to
make text better in appearance and appropriate to a particular context
[9]. These elements are also used to format text on electronic documents. These elements, as illustrated by Figure 6, can be broadly
classified as follows:
• The size of the characters.
• Spacing between individual lines of text.
• Spacing between individual characters.
• Width of an entire block/paragraph of text.
• Hierarchy of text, which is used to highlight the importance and
purpose of certain words or phrases or blocks.
Characters, which include alphabets, numbers and special characters, are represented in typography with different dimensions, as
11
12
basis
Figure 6: Anatomy of Typography [8].
appropriate to a certain contexts. Their size determines the amount
of space they occupy on a page and also how prominent they appear
on a page. The dimensions of a character are defined using its height
(called as x-height) and its width (called as set width), which together
are termed as the size of that character. The spacing between the lines,
termed as leading, describes how far apart two lines are from each
other. This dimension can vary between individual pairs of lines.
The spacing between individual characters in a text document is
defined using concepts termed as tracking and kerning. Tracking defines
the general spacing between individual characters, which applies to
an entire line of text. Kerning, on the other hand, also defines spacing
between individual characters. However, it applies to individual pairs
of characters. For example, in certain instances, some characters (like
A and V) look better when they are placed farther or closer to each
other, compared to their distances to the other surrounding characters.
Under such a scenario, kerning is used to achieve the desired spacing.
Width of entire blocks / paragraphs of text is another important aspect
to be taken care of when creating text documents. This element of
typography is termed as the measure. By adjusting the measure of
blocks of text to appropriate values, the reading experience of a reader
can be enhanced.
Just like the size of blocks of data, their position, thickness, color,
style etc., too play an important role in the creation of a document.
These aspects help in determining the hierarchy and purposes of
certain blocks of text in the document. Examples creating such a hierarchy are: According to the mood and purpose of a document, it is
appropriate to align blocks of text to the center, left, right or scattered
(or justified) on a document; Certain blocks of text like, headings and
subheadings look more appropriate when highlighted. Their hierar-
3.1 text formatting
Figure 7: Examples of font families [10].
chies can be established, by aligning, sizing, colouring and styling
them differently; On many instances, like in online advertisements,
making text colourful makes it attractive and more meaningful to the
prospective readers; It is appropriate in many cases to style text using
a certain font family. Font families (also called typeface) are definitions
of appearance of text. By applying them to the text in the document,
the appearance of the text can be enhanced. Examples of a few such
font families, as seen in Figure 7, are Arial, Verdana and Georgia.
3.1.1
Text Formatting in Electronic Documents
Text formatting plays an important role in the creation of electronic text
document, just like in the case of traditional paper based documents.
Despite the tremendous advancement in communication technologies
in the areas of telephony, internet etc., communication using text
documents is still performed with the help of the same typographic
techniques, which have been used since millenniums. This is likely to
remain so for many more years to come, until there is a breakthrough
in the area of text based communication, which would revolutionize
this entire concept. Therefore, many computer applications of today,
which facilitate the creation of electronic text documents also enable
users to perform text formatting.
While creating electronic text documents using a computer application (for example Microsoft Word), a user is provided with several
ways to perform text formatting. These options include:
• Changing the size of the characters (also called as the font size).
The font size can be set as positive number decimal number
ranging from 1 to 1638, where 1 represents the smallest possible
size of text and 1638 the largest.
• Changing the thickness, inclination and style of text. The thickness and inclination can be set as bold and italic respectively. Bold
increases the thickness of text and italic makes text italicised
(or slant) in appearance. Additionally, blocks of text can also be
13
14
basis
underlined. Further styling on text can be performed by assigning
certain family of fonts to it (as discussed in Section 3.1. All these
aspects which describe the appearance of text are connectively
termed as the typeface of that block of text.
• In many mathematical formulations and scientific calculations,
it is often required to represent certain characters or words or
phrases of text smaller in size and slightly above or below the
line they are present in. This is called as the subscripting or the
superscripting of text.
• The alignment of certain words or phrases or paragraphs can be
altered by making them aligned to the center of the page, to the
left of the page, to the right of the page and distributed from the
right corner to the right of the page (also called as justified text).
• The spacing of certain blocks of text to the left margin can
be adjusted to be more than its surrounding blocks of text by
increasing its indentation. By indenting a block of text, it is moved
away from the left margin by a predefined character distance.
• Changing the foreground color or background color of blocks of text.
To accomplish this, a user is provided with the choice of a range
of colors which she can select from.
• Adjusting the spacing between pairs of lines of text. On most
applications like Microsoft Word, this is performed by assigning
a positive integer, ranging between 1 and 3 to represent the
spacing between a pair of lines, where 1 is the least distance
possible and the distance can be increased by entering values
higher than 1.
• While creating documents, users often require enlisting or categorize various ideas related to a concept, in the form of points.
This is enabled by providing the user to add certain shapes to
the beginning of every point, signifying the beginning of each
idea or category. These shapes are termed as bullets and they
exist in many shapes, sizes and color. The most common and
basic bullet is a dark coloured circle.
• The appearance of pages in a document can be adjusted by
adding borders to each of the 4 sides of a page, as desired. These
borders too exist in various forms, thicknesses and colors.
These are the ways in which a user is enabled to perform text formatting while creating electronic text documents. Of course, the method to
apply these options differs from one computer application to another.
A more detailed discussion on the application of these formatting
techniques, with the help of an example application (Microsoft Word),
will be performed later in this document, in Chapter 4.
3.2 multitouch computing
3.2
multitouch computing
Multitouch computers have grown in popularity during the past
decade. Multitouch interaction is a popular way to interact with a
computer because it alleviates the need to use traditional input devices,
namely a keyboard and a mouse. With multitouch interaction, a user
can perform all tasks on a computer using her bare hands. In the
beginning of the 21st century, although multitouch technology existed,
it was not available to the masses. They were mostly expensive and
bulky devices. With the introduction of smartphones like the iPhone
in 2007 followed by other Android operating system powered devices,
even a layman of today is able to reap the benefits of multitouch
interaction.
Research work in the area of touch screen dates back to late 1960s,
when IBM built the first touch screens [4]. The idea of multitouch
systems began in 1982 at the University of Toronto, when Nimish
Mehta built the first multitouch system [20] [15]. Later in 1983, using rich gestures on multitouch systems were envisioned and also
developed by Myron Krueger. The first commercial multitouch tables
were developed only in the year 2001 by companies line Microsoft
(PixelSense) and Mitsubishi Research Labs (DiamondTouch) [15].
3.2.1
Multitouch Tabletop Computers
Multitouch tabletop (MTT) computers have been around for a little
over a decade now. As discussed in 3.2, the first multitouch table was
created in the year 2001. These devices are very useful in various
areas. For example: In schools they can be resourceful by providing
teachers with an interactive way to impart education on pupils; In
official scenarios, MTT devices can be used while conducting meetings,
to interactively discuss various ideas or strategies; In museums and
other tourist attractions, these devices can serve as an interactive
resource of information, where multiple individuals can fetch desired
information simultaneously; They can be used in the field of medicine,
by providing surgeons, an interactive means of practising surgical
procedures on a computer based virtual subjects. This way, there exist
endless possibilities for the usage of MTT devices.
A multitouch table (seen in Figure 8) consists of a touch screen,
which forms the interactive top surface of the table. The purpose of
this screen is to display the results of various tasks to a user and also
to collect the user’s touch / contact information. The screen itself can
be made out of various materials like glass and fibre. Information is
displayed on this screen by projecting it from either the rear side or
the front side of the screen, using a video projector. This projector
is connected to a computer, which provides the projector, with the
information to be projected onto the screen. Further, the touch screen
15
16
basis
Figure 8: Anatomy of a Miltitouch Computer [3].
is enabled to recognize contact information with the help of various
techniques, which will be discussed next. The contact information
gathered by this screen is passed on to a software, running on the
same computer which is connected to the projector. This computer
analyses the contact information, interprets them and performs actions,
the results of which are again displayed on the multitouch screen.
Capturing the contact information generated, when a user makes
contact with an MTT (like in Figure 8), can be performed in many ways.
To accomplish this, the surface itself should be constructed in such a
way that it generates touch information, which is later captured using
sensors. This can be performed by illuminating the surface using light
or other forms of energy and later capturing changes to this energy
caused by the user’s touch. Some of the existing techniques [6] [7] to
achieve such a mechanism are the following:
• Frustrated Total Internal Reflection (FTIR): This technique, seen in
Figure 9, of illuminating the surface involves the usage of a
physical property of light called Total Internal Reflection. Total
internal reflection occurs when a light beam strikes the boundary of the medium at an angle greater than a particular critical
angle with respect to the normal of the boundary surface. When
this occurs, light is completely reflected by the boundary of
that medium, preventing any refraction. In FTIR technique, this
property is used to completely flood an acrylic sheet with infrared light. This sheet forms the multitouch screen. When a user
touches the surface of this screen, infrared light at that point no
3.2 multitouch computing
Figure 9: Frustrated Total Internal Reflection Technique [7].
Figure 10: Laser Light Plane Technique [7].
longer gets totally internally reflected, but starts flowing partially
into the skin of the user. This change on the surface is treated as
the user’s contact information.
• Laser Light Plane (LLP): In this technique (seen in Figure 10),
infrared laser light is scattered, forming a thin layer, on the
MTT’s surface with the help of multiple lasers. The surface here
is generally made up of acrylic or other such materials. When a
user touches the surface, she breaks the surface of the laser light
layer and thus generates contact information.
• Diffused Illumination (DI): In this technique, infrared light is
shined at the MTT screen, which is again made up of materials like acrylic, as seen in Figure 11. A diffuser is placed on
top or on bottom of the touch surface. When a user touches
the screen, it reflects more light than the diffuser or objects in
the background; this extra light is treated as the user’s contact
information.
17
18
basis
Figure 11: Diffused Illumination Technique [7].
Figure 12: Diffused Surface Illumination Technique [7].
• Diffused Surface Illumination (DSI): This technique (seen in Figure
12) is similar to DI and FTIR. The setup is exactly same as in the
case of FTIR, the only difference being the material used as the
screen. In DSI, a special acrylic material is used. This material
contains small mirror like particles within, which redirect the
shined infrared light and spread it evenly across the surface
of the acrylic screen. The effect produced when a user makes
contact with this surface is same as in the case of DI.
• Light Emitting Diode - Light Plane (LED-LP): This technique too is
very similar to FTIR. As illustrated by Figure 13 however, light
is not scattered within the screen but above it. Infrared Light
Emitting Diodes (LEDs) are placed around the screen of the MTT
computer. These LEDs illuminate the entire area above the screen
and the user’s finger is sensed even before it touches the screen.
3.2 multitouch computing
Figure 13: Light Emitting Diode - Light Plane Technique [7].
In each of these techniques, the contact information is picked up
using infrared cameras, positioned either above or below the MTT’s
screen’s surface (depending on the position of the illuminating source).
As already discussed in this section, the collected information is passed
on to a software for processing. On receiving the captured contact
information, the application running on the computer, interprets and
displays the calculated results on the MTT’s screen, through the projector.
3.2.2
Gesture Based Interaction
A gesture is a form of communication in which messages are communicated non verbally, either in conjunction with speech or without it
[1]. These actions are performed by the movement of fingers, hands,
face, legs or other parts of the body. Gestures are used by humans
to express various feelings, wishes and thoughts. They are used by
many animals too. There are hundreds of gestures used by us in daily
life. Many of them are culture specific and could convey different
meanings in different cultural settings. Some of them are:
• Salute: Often associated with armed forces and other organizations, it is used as an action symbolizing respect.
• Crossed fingers: To wish for good luck.
• Applause: Clapping of hands together as an expression of approval and appreciation.
• Thumb up or down: An expression of approval or disapproval.
• Bowing: Performed by lowering the torso or the head, as a show
of respect. And so on.
19
20
basis
Figure 14: Tap Gestures [2].
In the field of computer science, gestures are often limited to those
gestures which are performed with a user’s bare hands and are used
as a means of communicating with multitouch computers (for example
an MTT computer). Gestures are usable as a means of input which
can be interpreted via mathematical algorithms. This interpretation
makes a gesture a meaningful command to the computer. This process
of interpretation is performed by collecting data from a multitouch
device’s screen which can sense such gestures. The usage of gestures
as a means of input brings humans and computers closer to each other
because the computer begins to understand the body language and
actions of a human. Gestures are appreciated because it enables users
to communicate by merely making use of her hands, without having
to use any other device.
3.2.2.1
Hand Gestures
Hand gesturing is the most common form of interaction with an MTT.
Many devices like mobile phones, tablet computers have already integrated this way of interaction to some extent. But MTTs have a greater
potential for this form of interaction since they support human-human
collaborative activity. Due to the increased workspace and the possibility for multiple users to work on the device simultaneously, gestures
can be used to perform a large array of tasks on MTT computers.
Hand gestures range from very basic single finger gestures to complex multi-finger gestures. In most cases, the complex gestures are
formed by the combination of more than one basic gestures. Some of
the basic gestures used are:
• Tap gestures: These gestures, seen in Figure 14 are used to select or
point at an object by tapping on the surface of an MTT computer.
• Rotate gestures: These gestures are used to move an object around
an axis, on an MTT computer, by holding it with fingers, as seen
in Figure 15.
• Scale gestures: These gestures, seen in Figure 16 are used to
increase or decrease the size of objects on an MTT computer by
holding them with fingers. Most commonly used to perform
tasks like zooming into or panning out of a picture or a map.
3.2 multitouch computing
Figure 15: Rotate Gestures [2].
Figure 16: Scale Gestures [2].
• Scroll gestures: These gestures are used to scroll through documents or files, just like using a mouse wheel. They can be
performed with one or more than one finger as illustrated in
Figure 17.
• Drag gestures: As seen in Figure 18, these gestures are used to
drag objects on an MTT computer. They are performed by holding
an object with one or more fingers and moving it around.
• Split gestures: These gestures are performed when the object on
the MTT computer has to be split or opened. It needs atleast one
finger of each hand to make contact with the surface. More than
one finger from each hand can be used too. Figure 19 accurately
shows how this is performed.
• Anchor Gestures: These gestures are performed when an object
is first anchored or pinned down by fingers of one hand and
another finger from the other hand is used to manipulate it. It is
performed with both hands and is well illustrated by Figure 20.
Many basic actions like tap, flick, scale and rotate actions can be
performed on the anchored object.
Figure 17: Scroll Gestures [2].
21
22
basis
Figure 18: Drag Gestures [2].
Figure 19: Split Gestures [2].
Figure 20: Anchor Gestures [2].
3.2 multitouch computing
Figure 21: Flick Gestures [2].
Figure 22: 3D Gestures [2].
• Flick Gestures: These gestures are performed with one or multiple fingers of the same hand. Figure 21 shows the use of this
gesture. It is performed by placing the fingertip(s) on the surface and making a short line(s) in any direction. It is usually
used to convey to the application to perform some action as a
consequence.
• 3D Gestures: These gestures are generally used to rotate and
manipulate objects in 3 dimensions on an MTT computer. Just
like the anchor gesture here fingers of one hand are used to pin
down the object and also in a way define the axis for the rotation.
After this, using finger(s) of the other hand the rotation can be
performed. There are variations of these gestures using 3 fingers
which are used to pan or tilt an image. Figure 22 shows this
gesture being used.
23
4
TRADITIONAL MENU/BUTTON BASED TEXT
F O R M AT T I N G
4.1
example word processing applications
Majority of the official documents and many of the non-official documents are created today using computer applications. These applications enable users to create electronic versions of documents, which
can then be used for communication, as in (in the electronic form)
or as a paper based documents (by printing them on paper). Word
processing applications are computer applications, which are created
to do exactly this.
Additional to their basic ability to create electronic documents, these
word processing applications provide users with many features, which
make the process of document creation quick, easy and also enjoyable,
compared to creating a traditional paper based document. Most of
these applications provide some basic features while creating electronic documents, like: Correction of blocks of text without having to
recreate the document from scratch; Formatting the text and pages
to achieve the desired appearance of the document; Using documenttemplates to create future documents in a certain format by simply
modifying or/and adding contents to the template; A graphical user
interface which makes the process of document creation interactive
and easy; and so on. Additionally, these applications provide many
other features which vary slightly from application to application,
since each of them is created and maintained by a different organization or individual. Many such word processing applications exist, for
example: Microsoft’s Word, Apple’s MacWrite, AbiSource’s Abiword,
Mark de Does’ Ted and The Document Foundation’s Writer. Many of
these applications are open source and many others are commercial.
4.2
microsoft word
Microsoft Word, which is created, sold and maintained by Microsoft
Corporation, is the most widely used word processing application
today. It is a commercial application and was first introduced by
Microsoft in the year 1983 under the name Multi-Tool Word. With
various improvements in terms of its features over the past years,
it is today the most used word processing application with about
500,000,000 users worldwide, according to Microsoft’s estimates in
the year of 2009 [13]. The latest version of this application (Microsoft
25
26
traditional menu/button based text formatting
Figure 23: An example of a word-processing application: Microsoft Office.
Word 2013) was released in January 2013. Figure 23 shows the 2010
version of Microsoft Word, which was released in June 2010.
Microsoft Word is used in various fields for an array of purposes.
In the business world, it is used to prepare memos, official letters,
reports and legal copies. In the field of education these applications
are taught to school children to prepare them for the competitive
business world. Additionally they are used in schools by students to
write essays and other such documents. In universities and colleges, it
is used to write complex scientific research documents like project and
thesis reports, which decide the careers of students. Even for home
users, Microsoft Word has become very useful and handy. Tasks like
homework assignments, preparation of curriculum vitae and writing
letters can be performed in a much faster and easier manner compared
to the traditional paper based documents. Users are provided with
a great feature by this application, by enabling them to easily file
their documents internally, on their computers and distribute them
electronically or by printing, when required. This saves a lot of time
and resources spent on maintaining large files of physical documents.
With massive advancements in the area of cloud based computing,
users are enabled to go one step further and save their documents
on cloud storage systems, thus being able to access their files from
any corner of the world over the internet. With such large array
of usages, word processing applications like Microsoft Word have
become an integral part of our daily lives, both professionally and
personally. Since this thesis work deals with text formatting and a way
of performing text formatting on MTT computers using hand gestures,
further discussion on Microsoft Word’s features will be limited to
those features pertaining to text formatting.
4.2 microsoft word
Figure 24: Formatting options on Microsoft Word.
Figure 25: More formatting options on Microsoft Word.
4.2.1
Text Formatting Options
Text formatting has been used to create documents for many millenniums now, as described in Section 3.1. The importance of text
formatting has remained great ever since the beginning of its usage.
Word processing applications of today provide user with a wide variety of text formatting options to choose from. Using these options,
users are enabled to make their documents better in legibility and
better in appearance. Microsoft Word too provides an array of text
formatting options to its users. These options can be characterized on
the basis of the least quantum of text which they effect, when used in
a document. There are some formatting options which effect single
characters (at least); There are other options which effect blocks of
text at a time, with each block comprising of one or more paragraphs
of text; Finally there are those options which effect entire pages. The
formatting options which fall under these categories can vary from
one word processing application to another and also from one version
of a certain application to another. Next, the most commonly available
text formatting options under each of these categories are discussed.
• Formatting options which effect at least one character of text
– Font size: The size of text font can be changed to smaller or
larger in appearance. This can be set by selecting a value
in the drop-down menu (seen in Figure 24) on Microsoft
Word. Font size of text can range from 1 to 1638, where
1 represents the smallest possible font size and 1638, the
largest. Additionally, the user also has an option of entering
a size manually in the form of any decimal number between
1 and 1638, directly into this drop-down menu.
– Subscript/Superscripts: Text can be converted into subscript
or superscript text by clicking on the corresponding buttons
from the menu, seen in Figure 24. By doing so, text is moved
slight above or below the margin of the line it is present
27
28
traditional menu/button based text formatting
in. It is also resized into a proportionally smaller size and
made bolder in appearance.
– Bold font: Users can use this option (seen in Figure 24), to
make text bolder or darker in appearance. This helps when
certain portions of text are required to be emphasised.
– Italic font: This option, seen in Figure 24, is an alternative
method of laying emphasis on certain portions of text. It
transforms text, making it italicised or slant in appearance.
– Underlined font: This formatting option, again seen in Figure
24 is yet another method of laying emphasis on text and is
performed by underlining the required portions of text.
– Foreground/Background font color: By clicking on the appropriate menu option (seen in Figure 24) makes text colourful
and therefore more appealing in many scenarios. It also
helps in highlighting portions of text.
– Strike out text: Sometimes users require striking out text,
while performing corrections on a document. The strike out
option helps users in performing this task. Figure 24 shows
the button which is used to perform this action.
– Remove format option: Many text formatting options can be
applied to text at the same time. Users may want to remove
already applied formatting options at once, without having
to remove each option one after the other through the menu.
To perform this quick action, the appropriate option (seen
in Figure 24) can be clicked from the menu, transforming
(selected) formatted text into unformatted text.
• Formatting options which effect at least one block of text
– Text alignment: Paragraph(s) of text can be aligned in four
fixed ways, to the center of a page, to the left margin of a
page, to the right margin of a page and spread from the left
margin to the right margin. By clicking the corresponding
buttons in the menu, seen in Figure 25, this action can be
performed on either the selected paragraphs of text or the
current paragraph (if there is no text selected).
– Text indentation: Required paragraph(s) of text can be indented to the left or the right by one tab length (which is
usually 8 characters wide). The selected paragraph(s) or the
current paragraph (when there is no text selected) can be
indented by clicking on the corresponding buttons in the
menu, seen in Figure 25.
– Text bullets and numbering: In order to enlist paragraphs of
text in a point wise manner, a user can use the bullet option.
By clicking on the bullet button (seen in Figure 25) from
4.2 microsoft word
the menu, a bullet is placed at the beginning of the current
paragraph. The user is also provided the choice of different
bullet styles, which she can pick from an additional bullet
format dialog.
– Text sorting: Many times, users require sorting paragraphs of
text, according to the character they start with. To perform
this task, a user can simply select the required paragraphs
of text and use the sort option (seen in Figure 25) from the
menu, to sort the selection alphabetically or/and numerically.
– Line spacing: Line spacing between individual lines within
a paragraph/paragraphs can be increased or decreased.
This can be performed by clicking the corresponding buttons from the menu, seen in Figure 25. The spacing can be
adjusted to values like 1, 1.5 and 2, where 1 is the least possible spacing and the spacing can be increased by selecting
a higher value.
• Formatting options which effect at least one entire page
– Page border: It is possible to use a border along any of
the four margins of a page, namely top margin, bottom
margin, left margin or right margin. These borders can be
customized to different color, line style and line thickness. A
user can perform this action by simply selecting the option
from the menu, seen in Figure 25 and then selecting a border
for the required page(s) from the dialog that appears.
– Header/Footer: A standard header and footer can be set
across many pages (also across an entire document, if required). This enables users to enter text like page numbers,
organization name, date etc,. which need to be displayed
on multiple pages of a document. To perform this task,
the user just needs to select the header/footer option from
the menu, then select the header and its contents from the
dialog which appears.
This was a brief description of most of the text formatting options
available on Microsoft Word. As observed, the formatting options
discussed in this section can all be performed using standard menu
based approach. This thesis work aims to take forward the same concepts of text formatting to MTT computers, but apply them using hand
gestures. The primary reason for the need to use gestures to perform
text formatting on MTT computers is to alleviate the inconveniences
which users would otherwise need to face, if they are required to
perform all the text formatting using menu based approach on MTT
computers. Since bare hand gesturing is the easiest way to interact
with multitouch computers, it can serve as an excellent alternative.
29
5
G E S T U R E B A S E D T E X T F O R M AT T I N G ( G B T F )
Gesture Based Text Formatting (GBTF) has many advantages over the
traditional menu/sub-menu based text formatting approach, on an
MTT computer. Finding the right text formatting option by navigating through the menu items and buttons is simple to perform on
traditional desktop computers and laptops, due to the availability
of physical devices like mice and touchpads. On an MTT computer
however, this task can get tedious and repetitive. As discussed in
Chapter 1, on an MTT computer, formatting text using traditional
menus and buttons is not only time consuming but also distracting
to a user creating an electronic text document. The user is frequently
required to concentrate on finding the right formatting option, rather
than performing the primary task of expressing her thoughts in the
document. This task can be easily managed on traditional computers
due to the availability of the hardware which make this task easier and
faster. Additionally, these hardware devices provide users with hepatic
feedback and provide other features like the ability to hover over areas
to retrieve information, right clicking, keyboard shortcuts and so on.
On MTT computers however, since all this has to be performed solely
by touching the screen, the process of text formatting becomes more
time consuming and complicated.
5.1
the concept
If the process of text formatting on an MTT device can be performed
using gestures, it reduces the repetitive cognitive burden on users
greatly, which they would be otherwise subjected to, during the process of finding the appropriate formatting options on the graphical
user interface of an application. As users require to use their bare
hands to interact with MTT devices on most occasions, using gesture
based interaction is the most natural and convenient way to perform
tasks on these devices. GBTF intends to perform exactly this, by providing a user with an alternative method of performing text formatting,
with the help of short and convenient hand gestures.
5.1.1
Text Selection
Selecting text is a very important aspect of all those computer applications, which enable the creation of electronic documents. In order
to format already entered portions of text from a text document, the
required amount of text needs to be first selected. On traditional com-
31
32
gesture based text formatting (gbtf)
puters, this action can be performed easily using a physical mouse
or a keyboard. On an MTT devices, this task can be performed using
hand gestures.
Many multitouch devices of today, like smart-phones, tablet computers use different gestures to perform the task of text selection.
Applications for MTT computers, built using Microsoft technologies
(like Windows Presentation Foundation) and running on Microsoft
Operating Systems (like Windows 7), can make use of the readily
available, native Windows mechanisms to recognise and perform text
selection. Using Microsoft technologies, text can be selected using tap
and drag gestures.
By tapping on the text on a multitouch enabled document editor
application results in placing the cursor at the tapped location. By
tapping twice at a location at short intervals, results in an entire
word at that location getting selected. By tapping thrice at short
intervals, results in the entire paragraph getting selected. In order
to select individual characters or words or multiple paragraphs, the
drag gesture can be performed, after placing the cursor at a desired
starting location. In the illustration of GBTF, discussed in Chapter 6,
Windows Presentation Foundation has been used for implementation
and Microsoft’s native text selection mechanisms have been used to
select text.
Further, on a document editing application running on an MTT
computer, it is convenient to select the entire document quickly, with
very few taps, rather than performing the drag gesture through the
entire document. This feature is included in the illustration described
in Chapter 6. In the illustration, this is achieved using a long press
gesture, with all five fingers of one hand.
5.1.2
Text Formatting using GBTF
One of the most convenient ways to interact with an MTT computer is
using hand gestures. GBTF aims to provide the possibility to perform
text formatting on electronic documents, on MTT computers, using
hand gestures.
Text formatting using GBTF has been broadly classified into two
categories: First of them is those formatting options which require
text to be pre-selected before they can be formatted. Second is those
formatting options, which do not require text selection. The first
category directly change the appearance of the text, by changing
aspects like text font size, text emphasis (bold, italic, underlined)
and text font color. The second category only change the position of
text blocks or change text organization or change the appearance of
an entire page or the change the appearance of an entire document.
Examples of these actions are text alignment, text indentation, page
5.1 the concept
borders and text bullets. GBTF aims replace menu based text formatting
using simple hand gestures.
5.1.3
Gestures Requiring Text Selection
Using GBTF, users are enabled to perform text formatting using simple hand gestures. There are some formatting options, for which the
required portions of text need to be selected, before being formatted. These options are as follows: Converting text to subscript or
superscript, changing the emphasis on certain portions of text (using
bold/italic/underlined font), changing the font size of text, removing
format from text, changing the foreground and background color of
text and hyper-linking text.
5.1.3.1 Subscript or Superscript
A Subscript or a superscript is a number, a symbol, a character, a
word or a phrase which appears below or above other normal words/phrases. It is slightly smaller in size and are often used for various
purposes, for example: in mathematical formulae and in the description of chemical compounds. This formatting option is often used in
scientific documents, it is very convenient and fast to perform them
using gestures on MTT computers.
33
34
gesture based text formatting (gbtf)
Figure 26: Gesture to achieve superscripts.
Figure 27 and Figure 26 depict the gestures, which can be used
to convert text to either a subscript or a superscript. Once the text
which requires to be subscripted or superscripted is selected, the users
performs the vertical swipe gesture. If the gesture is performed in
the upward direction, the text is converted into a superscript. If it is
performed in the downward direction, the text is converted into a
subscript.
5.1 the concept
Figure 27: Gesture to achieve subscripts.
As the location of a superscript is slightly above normal text and
the position of subscript is slightly below it, the upward and the
downward gestures to convert text into subscript and superscript,
respectively, is an absolutely natural and obvious gesture for a user to
think and perform. This is because, even in a real world scenario, in
order to push an object above or below a certain point of reference, one
usually pushes the object upward or downward. After the gesture is
performed by a user and as soon as it is interpreted by the underlying
GBTF software, the selected text is formatted and the result of the
formatting is displayed to the user on the MTT computer screen (within
the application).
5.1.3.2 Emphasis
While creating a text document, certain portions of text is emphasised
differently using the bold, italic and underline formatting options. Bold
makes characters darker and thicker. This is really helpful to make
certain parts of a document appear visually louder and prominent
compared to the rest of the document. Italic, on the other hand, is also
used to emphasise text but in a more subtle manner. Italic formatting
option makes characters slant, making them look different from the
35
36
gesture based text formatting (gbtf)
surrounding text. Finally, underline is another option, which can be
used to lay emphasis on certain parts of text. This formatting option
does not change the appearance of the text itself but only underlines
it. It is sometimes used to point out certain parts of text to a reader.
Figure 28: Gesture to achieve bold text.
The bold and the italic formatting options are used very often while
creating electronic documents. Therefore, they have been included as
gesture supported actions in Gesture Based Text Editor (discussed in
Chapter 6). The underline formatting option is not used very often
and hence has been included as one of the menu options. Figure 28
illustrates the gestures which can be used to perform bold. Making
text bold in appearance means making it thicker and darker (or in
other words, making color of the text more concentrated). In real life,
to concentrate the volume of a substance, it is usually compressed
or squeezed using force. As an equivalent to this action, Figure 28
illustrates the gesture used to perform bold formatting on text. After
selecting the required text to be formatted, all the fingers of a hand
are first placed on the MTT computer screen and then brought closer
to each other (as if squeezing a physical object). This results in the text
looking bold in appearance.
5.1 the concept
Similarly, in the real world, an object is made slant (or oblique)
by tilting it physically. In order to achieve similar action (and result)
while using GBTF, a user performs the gesture depicted in Figure 29.
Once the required text is selected, the user places two of her fingertips
on the MTT computer screen, approximately parallel to the vertical
axis of the application. Then, out of these two fingers, by keeping the
finger at the bottom steady, she makes a small approximate rightward
arc with the top finger, as if drawing an arc using a compass. This
rightward motion of the top finger (with respect to the bottom finger
and to the plane of the MTT application), makes the user feel as if she
is tilting text in the rightward direction. As a result of this gesture, the
selected text becomes slant instantly, transforming into italicized text.
Figure 29: Gesture to achieve italicised text.
5.1.3.3 Font Size
A text document with properly sized text makes a reader more comfortable and more pleased while reading it. In order to size the characters of a text document to appropriate dimensions and make the
document visually more legible to readers, the font size formatting
is used. Font size formatting is used to increase or decrease the size
37
38
gesture based text formatting (gbtf)
of the characters in a document. In the real world, an elastic object’s
(example: rubber-band, sponge etc,.) size can be increased by simply
pulling it apart, and it’s size can be decreased by pinching it.
Figure 30: Gesture to decrease text size.
In order to generate the same effect on the characters of a text
document on an MTT application, the gestures illustrated by Figure
30 and Figure 31 can be used. After selecting the required text to
be resized, by performing the pinch gesture results in the text’s size
becoming smaller and by performing the stretch gesture results in the
size of the text becoming bigger. Since this gesture is exactly what
the user would do, in order to resize real-world-objects, the entire
interaction is easy to remember and also natural to perform.
5.1 the concept
Figure 31: Gesture to increase text size.
5.1.4
Gestures Not Requiring Text Selection
not only enables user to format the appearance of text in an electronic document but also the appearance of the document as a whole.
Actions like alignment of paragraphs, indentation of paragraphs, using bullets and using page borders result in not just the appearance
of text but also of an entire page or the entire document. To perform
these actions using GBTF, text need not be preselected. The user simply
needs to perform the required gesture over desired areas.
GBTF
5.1.4.1 Page Borders
Page borders are used in many official documents and are used to
enhance the look of a document. On a traditional document editing
application, page borders can be inserted by clicking on the page
borders button present in the menus. On an MTT computer application,
using GBTF, a border can be inserted to a page by performing a simple
gesture. By just placing the palm sideways at one of the sides of the
document (parallel to that side), results in the insertion of a border on
that side of the page.
39
40
gesture based text formatting (gbtf)
5.1.4.2
Alignment
Text can be aligned in different ways in a electronic document. The
alignment is important because it helps in distinguishing various
elements of a document, for instance: part of a document (heading,
section etc.,), type of document (a report, a letter, a scientific paper
etc.,).
Figure 32: Gesture to achieve left aligned text.
Text can be aligned to the margin on the left hand side, to the margin
on the right hand side, to the center a page and as fully justified
(spanning from the left margin to the right margin). To perform this
task on a traditional computer application, the required text is first
highlighted and then the required alignment option is selected from
the menus.
5.1 the concept
Figure 33: Gesture to achieve right aligned text.
The task of alignment can be performed quickly and in an easier
manner on an MTT computer using gestures. Using GBTF, by performing the leftward or rightward swipe gestures on a paragraph (as seen
in Figure 32 and Figure 33), the paragraph is instantly aligned to the
left margin or to the right margin respectively.
41
42
gesture based text formatting (gbtf)
Figure 34: Gesture to achieve center aligned text.
Similarly, performing the pinch gesture or the zoom gesture on a
paragraph, as described in Figure 34 and Figure 35, the paragraph is
aligned to the center of the page or fully justified respectively. Since
all these gestures are exactly how a flexible physical object would be
manipulated, to achieve similar effect, they are easy to perform and
also remember, to end users.
5.1 the concept
Figure 35: Gesture to achieve justified text.
5.1.4.3 Indentation
Indentation of text is another important part of documents. It helps
in specifying various aspects of a document like: indicating sections
(and their subsections) by creating a hierarchy and for indicating the
beginning of paragraphs.
43
44
gesture based text formatting (gbtf)
Figure 36: Gesture to achieve right indented text.
To achieve indentation of required blocks of text, on a traditional
computer, the desired text is first selected and then rightward or
leftward indent is selected from the menus. On an MTT application,
this action can be realised using the gestures described in Figure 36
and Figure 37. By performing the leftward gesture on a paragraph, the
entire paragraph is indented to the left hand side and by performing
the rightward gesture, indents it to the right hand side. As the gestures
directly effect the paragraph on which they are performed, text need
not be selected before performing the gesture.
5.2 gbtf vs menu based formatting
Figure 37: Gesture to achieve left indented text.
5.2
gbtf vs menu based formatting
Many text editing and word processing applications created for multitouch computers still largely depend on menus to provide users
with an interface for interaction. Although using standard menus
and buttons is a more familiar environment for users, during their
initial few days of usage, gestures interaction is a more natural way
to interact with multitouch computers. However, performing gestures
is not suitable on all kinds of multitouch devices. On smaller devices
like mobile phones with relatively smaller multitouch screens, using
gestures can make the usage of the applications inconvenient, merely
due to the small sizes of the screens. Using hand gestures requires
a user to move her fingers and hands freely on the screen to form
some predefined shapes, which are interpreted as commands by the
application. On small devices, this process is somewhat restricted. For
these reasons, the usage of standard menus and buttons simplifies the
usage of application. This is one of the scenarios where buttons may
be preferred over gestures.
45
46
gesture based text formatting (gbtf)
However, the concept of buttons and menus were created, with
standard input devices like the mouse and the keyboard in mind. They
were created to mimic the physical electronic / mechanical devices,
which can be interacted with by pressing buttons. As time progressed,
even certain processes which do not require pressing buttons in real
life (like flipping pages of book and interacting with objects), were
implemented in applications which use buttons. These buttons were
sometimes physical buttons (as a part of the keyboard), or virtual ones
(as a part of the software application). However, since the last few
years of the previous century, there has been a major advancement
in the way computers and computer applications have been treated.
There has been a constant effort to combine the real and the digital
worlds, by performing tasks on computers using the same (or similar)
actions which one would use to perform the same task in real life. It
is this quest that has led to such huge advancements in the area of
multitouch technology.
An MTT computer has a large area which a user can work on. It
is usually as big as a standard desk used in the real world. While
performing non-computer-based work at a standard table, one is
seldom required to press buttons or select options from the menu.
While performing tasks like interacting with physical objects, writing
text, reading documents, discussing issues at a table, only gestures
are used in the real world. Therefore, in this context using gestures
on an MTT computer is a more natural way to perform tasks. Using
gestures brings this table-like computer very close to an actual table.
Also, in real life, most of the human to human communication is
performed largely using gestures, combined with speech and writing.
More importantly, many expressions can be expressed solely using
gestures. Due to this vast usage of gestures in real life, it is only natural
for users to use the same gestures to communicate with computers too.
This is the primary advantage of using gestures on MTT computers.
Further, using buttons and menus can be complicated on MTT computers due other reasons too. Some important issues which highlight
these aspects and also compares the usage of Gesture Based Text
Formatting (GBTF) with standard menu/button based text formatting:
• Hepatic feedback: When a user presses a real button, like a button
on a mouse or a keyboard or even a vending machine, the button
provides her a sensation on her fingertip of having pressed and
released it. Although a similar sensation has been mimicked on
smaller devices like mobile phones, using vibration, on an MTT
table this is not yet possible. Since an MTT table is intended to
be used by multiple users at the same time, providing a hepatic
feedback on every button press of every user, locally, only to
fingertips of particular users is not (yet) possible. Therefore using
gestures to perform tasks like text formatting provides a more
fluent experience to the user.
5.2 gbtf vs menu based formatting
• Cognitive load: When using buttons and menus to perform text
formatting, users are required to remember the location of various buttons, menus items and sub-menu items. In the case of
word processors (and text editors), the location of these buttons
can be changed and their appearances can be changed as well.
This is usually the case when one compares applications made
by different vendors and also when one compares different versions of the same application. Further, within a text editor or a
word processor, not all buttons and menu options are visible to
a user by default, when she installs the application. Some are
kept visible and some others are hidden by default. While using
these application, a user has to frequently hide/unhide buttons
by finding them within different menus. All these add to the
cognitive load on the user.
On the other hand, using gesture too can create cognitive load
on a user, because the user has to initially associate each gesture
with a certain action. However, once the user uses the application
for a few days and gets familiar with the gestures, she does not
require to think or find different options on the user interface
every time she performs a task. The process of using the application becomes as smooth as doing a real life task. The key
aspect here is to use gestures which the user does not require
to actually learn. In order to perform tasks, if a user can use the
same or similar gestures which she would use in real life, the
process of initial learning as well as later usage becomes even
more cherishing to the user.
• Ease of usage: On MTT computers, using buttons and menus can
be time consuming. This is because navigating through menus,
finding required buttons, selecting (tapping) the option are not
very easy to perform on a multitouch screen, especially on large
application which have several menus, sub-menus with several
buttons. Another reason for this unease is also the lack of hepatic
feedback, as just discussed. Using gestures alleviates all these
problems. Being a more natural way to interact, a user is not just
enabled to work faster but also very fluently, without any abrupt
pauses to find elements or options on the user interface. A user
is never required to navigate through menus, if an application
is completely implemented using gestures. On most occasions
however, buttons and menus are not completely eliminated from
gesture based applications created for MTT computers but are
used minimally alongside gestures. This same kind of approach
has been used in this thesis work.
All these discussed reasons clearly indicate that using gestures on
devices is a better approach. It not only makes the usage of the
MTT
47
48
gesture based text formatting (gbtf)
computer easier but also enables the user to work faster and more
efficiently.
MTT
6
I L L U S T R AT I O N O F T H E U S A G E O F G B T F
Using gestures to perform gesture formatting is a new approach to
creating text documents on MTT computers. It has several advantages
over the traditional menu based approach, as described in Section
5.2. In this chapter, this concept is illustrated using a Text Editor
application. This application has been implemented to support GBTF,
and it portrays the ease of usage of hand gestures for text formatting.
It also shows how the gesture based approach can be used effectively
for most frequent tasks.
6.1
system design
The system itself has been implemented in such a way that it closely
matches the behaviours of traditional Windows based applications
which enable the creation of documents with formatted text. However,
most these behaviours here are not the result of menu-button clicks
like in the case of traditional applications, but the result of short
hand gestures. A few not so commonly used text formatting options
have been implemented within a circular context menu, which is also
activated using a gesture. The system itself can be better understood
with the help of Figure 38.
Similar to traditional text editing and word processing applications,
which anticipate mouse clicks and respond to them by performing
the desired changes on required sections of text, this application
anticipates and responds to multitouch inputs generated via users’
hand-gestures. As seen in Figure 38, as soon as the system starts (or
is activated), it starts anticipating gesture inputs from the user. The
technique used to activate the GBTF system has been discussed in
Section 6.2. While the systems waits, if it encounters a multitouch
input (or contact information) from the user, it quickly gathers all this
information.
It is then examined to check whether the multitouch information
gathered from a user is meaningful and valid. In cases when the gather
information turns out to be invalid or meaningless to the system,
it simply ignores the input. This is because on multitouch devices,
users tend to perform unintended and accidental touch interactions.
However, if the information gathered by the GBTF system happens to
be valid and meaningful, it is passed on to be recognized. The process
of gesture recognition is performed by the gesture recognition engine,
which has been implemented as a core component of this application.
This gesture recognition engine can identify gestures performed by
49
50
illustration of the usage of gbtf
one or more fingers by users. Currently, it supports a limited number
of gestures.
Start
Wait for next
contact information
Analyse the validity
of information
No
Valid?
Yes
Identify the
gesture
Display results
to the user
Figure 38: Flowcharting depicting the design of the system.
However, since the entire gesture recognition engine has been implemented modularly, not being dependent on other parts of the
application, additional gesture recognition capabilities can be added
to it easily. Next, the gesture recognition engine tries to identify the
gesture by matching it with its collection of predefined set of possible
gestures. If it finds a match, it returns the identified gesture as the
result and if not, it informs that the gesture was not identified. If a
gesture is identified, the system immediately performs the resultant
action. This action pertains to altering the desired portion(s) of text
in the text document by formatting them with the desired formatting
option. The result of this action is displayed to the user on the screen
of the MTT computer. All these steps occur instantly and the user
provided with the experience of performing text formatting with some
really simple and quick gestures. Once an interaction is complete,
either by performing providing the user with results or by ignoring
invalid inputs and unidentified gestures, the system goes back to the
initial state, waiting for further multitouch inputs of the user. This
process occurs repeatedly.
6.2 implementation
6.2
implementation
The system, described in Section 6.1, has been implemented to reflect
the effectiveness of using GBTF on MTT computers. The application,
titled Gesture Based Text Editor, is a simple tool which enables the creation of text documents in Rich Text Format (or RTF). Text within these
documents can be formatted using some of the text formatting options
available on standard word processing applications like Microsoft
Word. Some other less commonly used text formatting options are
included within a specially designed menu, within this application.
6.2.1
Tools Used
The Gesture Based Text Editor application has been implemented using
Windows Presentation Foundation 4.0 (or WPF 4.0). It is implemented
on Windows 7 Operating System. Windows Presentation Foundation
is a software graphical subsystem for rendering user interfaces on Windows based applications [12]. WPF 4.0 is the fourth version of Windows
Presentation Foundation and it comes with several capabilities which
include an improved support for the usage of multitouch input within
applications. Alongside WPF 4.0, Microsoft Expression blend is also
used to design certain elements of this application. The application has
been built by taking advantage of the existing touch input capabilities
of Windows 7 therefore it is currently usable on computers running
the Windows 7 operating system.
6.2.2
Behaviour
The design of the system described in Section 6.1 using Figure 38
outlines the overall design of the Gesture Based Text Editor application.
Each state represented and each transition leading from one state to
another is comprised of multiple atomic tasks. These tasks are performed by the system is synchrony to achieve the goal of supporting
GBTF. These steps can be better understood with the help of the the
pseudocode illustrated by Algorithm 1. GBTF can be performed with
51
52
illustration of the usage of gbtf
the application by enabling the gesture mode. If the system finds this
mode enabled, it begins anticipating gestures by the user.
Data : User’s contact information
Result : Perform text formatting or display error
while Gesture Mode is Active do
while Waiting for user’s contact information do
if User’s contact information found then
Collect each finger’s contact information in separate
arrays;
if Collected contact information found valid then
Identify gesture;
if Gesture identified then
Perform text formatting;
Display results;
Provide details to the user;
else
Inform gesture to retry;
end
else
Inform gesture to retry;
end
end
end
end
Algorithm 1 : Pseudocode describing the behaviour of GBTF.
As described in Chapter 4, text formatting options have different
prerequisites. For example, options like increasing the font size and
changing the font color, of existing portions of text, require users to
highlight/select those portions of text before performing the formatting. On the other hand, options like changing text alignment, text
indent do not require pre-selection of text. When the system anticipates for gestures from a user, while being in the gesture mode, if
the user performs a gesture, it immediately captures the motion of
each finger of the user. This information is stored as an array of touch
points, which were recorded while a finger moved along the MTT surface. Each finger is assigned its own array while recording the overall
gesture. A gesture is assumed to be complete when the user removes
each placed finger, off the MTT’s surface.
Once a gesture is complete, the captured touch information, in the
form of one or more arrays, is first checked with some preliminary
tests, like the duration of the gesture. Very short duration gestures,
which span duration insufficient to perform the shortest possible
gesture in the predefined set, are ignored. On such occasions, the
user is informed through a message to try the gesture again. Once
6.2 implementation
the preliminary tests are passed by a gesture, the contact information
of the users fingers are passed on to the gesture recognition engine,
which performs the tasks of further analysing the input. The engine
compares the shapes formed by the motion of the users fingers against
the predefined set of gestures. If there is a match, it informs the
application which gesture was performed. If not it informs that the
gesture was not recognizable. The user is made aware of the usage of
specific gestures for specific formatting options, prior to her usage of
the application. This information is also available to the user within
the application, in the form of a reference page, which can be accessed
any time.
Upon receiving a response from the gesture recognition engine, the
application then decides how to proceed with the action. If a gesture
was identified by the engine, the application immediately performs
the associated text formatting on the desired portion of text. The
result of this action is displayed to the user on the MTT’s screen. If
the performed gesture is not recognized, the application informs the
user and asks her to try the gesture again. Such situations can occur
when the user does not perform a gesture accurately or when the user
performs an unintended/accidental contact on the screen. Once an
action is successfully completed, the user can proceed with typing
in more text by deactivating the gesture mode and can continue or at
a later time perform more text formatting using gestures, by simply
switching back to the gesture mode!.
The need to switch back and forth from gesture mode! is because the
gesture performance area is the same area which is used to type in the
text. Therefore the primary task of the gesture mode! is to block further
text input/highlighting within this area. This is important because a
dynamically changing nature of a document can interact and interfere
with the gesture capturing and recognition mechanism. Therefore the
separation is crucial to enable the application of being able to use a
text area for text input as well as of being able to use the same area to
perform gestures.
6.2.3
User Interface
The Gesture Based Text Editor application has been created to be simplistic and easy to use. The primary motive of the application is to
demonstrate the ease of usage of GBTF. Figure 39 shows how the application appears to an end user’s eyes. Although the application hugely
depends on user gestures to perform text formatting, there are a few
text formatting features which are performed using a special menu,
designed for this purpose. Further the application also has a standard
File menu on the top left corner, as seen in Figure 39. This menu
enables a user to save a document (in RTF format), open a document,
create a new document and exit the application.
53
54
illustration of the usage of gbtf
Figure 39: The Gesture Based Text Editor.
The primary area of the application is the big white area, where the
user can input text as well as gestures. To input text, the user uses
the standard virtual keyboard which is provided by Windows 7. This
keyboard pops up whenever a user taps in the text area (when the
gesture mode is not active. The gesture mode can be toggled to be ON or
OFF by using the button at the bottom on the right hand side, named
Gesture Mode. When active, the button turns green in color and when
not it turns light-red. Of course, gestures too are performed within
the same white area, as mentioned in this section, when the gesture
mode is active.
Further, for the reference of the user, an illustration of all the gestures
is available to the user within the application, which can be used
any time. The illustration is in the form of a large picture, depicting
the usage of each gesture. This picture can be opened by the user by
tapping on the Gesture Help button, also at the bottom at the right hand
side. On the left hand side, at the bottom of the user interface, is the
area where status information of the application is displayed to a user.
It tells user various information like, which task was performed as a
result of her previous gesture, error messages when the user preforms
a gesture inaccurately and whether the gesture mode! is turned on or
off. All this information acts as a constant feedback system to the user
and helps her better analyse situations and work faster.
6.2.3.1
The Circular Context Menu
Within the Gesture Based Text Editor application, the more commonly
text formatting options have been implemented as gestures. However,
some other of them have been included within a circular menu. As seen
in Figure 40, this menu includes formatting options like foreground
and background color of text, underlining the text, removing text
6.2 implementation
Figure 40: The Circular Menu.
formatting from formatted text, adding hyperlinks to text and selecting
all text. This menu can be accessed directly while typing in text input
(ie: when the gesture mode is turned off). To open this menu, the user
has to simply perform the hold gesture with one finger on the text
input area. For most of the options, excluding selecting all text, the user
first preselects the required portion of text and opens the menu by
performing the hold gesture directly on the highlighted text. By doing
so, the circular menu opens directly at that location, like a ring around
the finger. Then the user can choose any option from this menu and
the result is displayed on the screen instantly.
Figure 41: The Colour Selection Menu.
Some of the options like changing text color and setting hyperlinks
cannot be performed by a single click in the circular menu, because
they include the second step of selecting a color or entering the hyperlink. Therefore, for these options, second menus, seen in Figure
41 and Figure 42, follow the circular menu. When a user picks one
of these options from the circular menu, immediately closes and is
replaced by the second menu. Within the second menu, the user can
select color for the text foreground/background or enter a hyperlink.
Then once the user confirms her selection, the result is displayed on
the screen. Since the circular menu does not have a fixed location on
55
56
illustration of the usage of gbtf
the screen and pops up, whenever and where ever the user requires it,
it is convenient to use.
Figure 42: The Hyper Link Menu.
6.3
user study and the results
The Gesture Based Text Editor application discussed under Section 6.1
and under Section 6.2 was created to illustrate the usage of GBTF. On
completion, it was evaluated with the help of a group of users, who
used the application and provided feedback on their experience.
To perform the evaluation, a printed document containing one page
of formatted text was provided to each user. In this document, text
was formatted using various options like: Italicised font, Bold font,
center alignment and background color. The users were also provided
with an unformatted version of the same document in electronic form.
The users were asked to format the unformatted text using the Gesture
Based Text Editor and replicate the printed copy which they were
provided earlier. The users were briefed on how to use the application
and were also explained on what the had to achieve. On completion
of the formatting task, the users were asked to express their thoughts
and share their experience in writing, by answering a questionnaire
which was later provided to them. A total of 7 users participated in
the evaluation and all these users are students at the University of
Paderborn between the age group of 20 years and 30 years.
Through the questionnaire, the users were asked their opinions
on using gestures for text formatting, ease in learning and using the
gestures, effect of using gestures on the speed and fluency of text
formatting. They were also asked which gesture they found good
and which they found to be not apt. Finally they were asked how
their overall experience was, if the usage of gestures enriched their
experience and if there were any drawbacks they found which could
be worked upon.
Majority of the participants of the user study believed that using
gestures to perform text formatting on MTT was a good approach
and that it did make the process of document creation faster and
more fluent. Some of the participants found it difficult to remember
6.3 user study and the results
the associations between certain gestures and the corresponding text
formatting option. However, with a little practice, all the users were
able to perform text formatting comfortably. Also, although most users
liked all the implemented gestures, some of the participants found
certain gestures not apt for the text formatting option they represent.
Since all these gestures are only created to illustrate GBTF, there can be
a few of them which need to be further pondered about. Further, since
the gestures were implemented with right handed users in mind, one
of the users who was left handed found it difficult to perform certain
gestures. With the concepts of GBTF defined, it is always possible to
come up with better suited gestures and to simply add them to the
application.
However, overall all the users found the experience very enriching
compared to the standard menu based text formatting experience.
They were of the opinion that the usage of gestures has a positive
effect on text formatting and that with some adjustments with the
gestures used, the overall experience can be improved even further.
57
7
CONCLUSION
Using menu based interaction to perform text formatting on MTT
computers can adversely impact the efficiency and fluency of the
creation of an electronic document. This is because a user has to very
often suspend typing and search through menu and other buttons by
performing the tap gesture on them. While this task is easy to perform
using standard input hardware, the mouse and the keyboard, on MTT
computers, it can take a lot of time and can also get monotonous. The
thesis work presented in this document discussed all these issues and
also a solution to them.
7.1
summary
Text formatting is being used to enhance the look and the understandability of documents for millenniums now. Its importance in creating
text documents has remained as prominent in electronic documents as
it has remained with traditional paper based documents. This thesis
document shed light on this pivotal role which text formatting has
played for ages now. It also introduced the concepts behind multitouch computing. Further, the document also introduced the concepts
of gesture based interaction. The document then proceeds with the
discussion, by briefly describing related research work which has been
carried out for the past many years.
While using the standard menu based interaction to perform text
formatting on an MTT computer might seem simple and familiar to a
user at first, but as she starts using the system, it soon becomes apparent that it has a major set of drawbacks, which seriously hamper this
process. This document discussed various text formatting techniques
used in software applications today. The discussion was elaborated by
considering an example word processing application called Microsoft
Word and the features available in this application to perform text
formatting were also described. Also, since this application (and all
such word processing applications) depends on menu based input to
perform text formatting, the document also enlisted the problems one
could face while using the standard menu based approach on MTT
computers.
After introducing all these topics, the document then defines and
describes the concept of GBTF. This description is the primary focus of
this thesis work. The document also suggested how many of the actions performed using standard menu based approach can be replaced
using gestures. This definition is followed by a comparison of the
59
60
conclusion
standard menu based approach with GBTF. The comparison indicated
that in comparison to standard menu based text formatting, gesture
based text formatting can indeed be advantageous on MTT computers.
The definition of GBTF was further elaborated using a gesture based
text editing application, which was achieved by implementing a text
editor which can be used to create Rich Text documents and perform
many text formatting options within this application with the help of
gestures, in place of the standard menu based approach. The document
described this implementation by discussing how the states of the
system changed and through a pseudocode of the implementation.
Finally, a user study was performed at the University of Paderborn
to evaluate how much is the system to the liking of a layman. The
results of this study indicate that the concepts of GBTF were found
useful and well suited for MTTs by the participants. Although the users
found some imperfections in the gestures chosen, they believed that
with some adjustments in terms of the gestures, the experience can be
made even better. Overall, the users found the experience enriching.
7.2
future work
This document presented the concept of GBTF, illustrated the usefulness of the concept with an application which was implemented as
a part of this thesis work. It also described the usability evaluation
performed on the application on the basis of user-friendliness. With
the concept defined, there can be further evaluation performed in the
future on different other fronts, like cognitive effect, time consumption and so on. All such evaluations will serve as a basis for making
the concept of GBTF even stronger. They will also serve as a way of
finding any loopholes in the concept and also indicate how to fix such
shortcomings.
There is also a scope for future work on working out the best
suitable gestures for each scenario. Although gestures suggested in
this document were selected after analyzing different possibilities, they
might not be suitable for every scenario and every use-case. Since the
primary focus of the thesis work was to define and demonstrate the
concept of GBTF, much time and effort could not be spent on selecting
the perfect gesture for each case. A deeper analysis in this direction
will result in gesture sets which will be more apt and convenient for
different cases.
In the Gesture Based Text Editor application, which was implemented
to demonstrate the concept of GBTF, a user is enabled to perform
gestures only when the Gesture Mode is activated and the text editing
area is disabled for text input while the gesture is being performed.
The user needs to make this switch back and forth every time. This
limitation is not a loophole in the concept but a limitation with the
Richtextbox WPF! control used in the implementation. In future imple-
7.2 future work
mentations, this limitation can be overcome by either creating a custom
control which does not suffer from this effect or using a different touch
platform/technology altogether. Since the concepts defined here are
not platform specific, they can be easily used on other platforms and
with different technologies. This will result in applications where a
user can perform gestures seamlessly on the text area without having
to worry about unintentional alteration to the text. The user will then
no longer be required to disable the text input area and later enable it,
back and forth.
Subsequently, the concepts discussed in this document it can be
integrated into different MTT computer applications, not just in the
area of creating electronic documents but all such areas where users
need to perform text formatting. This will help the concept evolve and
spread to other domains. This thesis work primarily focuses on MTT
computers but in the future, the concepts can be further developed and
be made adaptable on other multitouch devices too. Since the need to
perform text formatting is not limited to electronic document creation
or MTT computers and since GBTF is scalable and the possibilities to
further develop and broaden this concept are endless.
61
BIBLIOGRAPHY
[1] Gesture, . URL http://en.wikipedia.org/wiki/Gesture.
[2] Gesture set, . URL http://www.gestureworks.com/.
[3] Renci multi-touch table development, . URL http://vis.renci.
org/multitouch/?p=164.
[4] The history of multitouch, . URL http://en.wikipedia.org/
wiki/Multi-touch#History_of_multi-touch.
[5] Rich notes text editor.
rich-notes/.
URL http://www.denvog.com/app/
[6] Existing multitouch technologies, . URL http://sethsandler.
com/multitouch/.
[7] Existing multitouch technologies, . URL http://wiki.nuigroup.
com.
[8] Typography in graphic design, . URL https://soma.sbcc.edu/
users/Russotti/111/read_mes/unit03_type_basics.html.
[9] What is typography? learn the basic rules and terms of
type!, .
URL http://www.creativebloq.com/typography/
what-is-typography-123652.
[10] Web typography, . URL http://en.wikipedia.org/wiki/Web_
typography.
[11] Tyype text editor. URL http://appvetica.com/tyype/.
[12] Windows presentation foundation. URL http://en.wikipedia.
org/wiki/Windows_Presentation_Foundation.
[13] Microsoft office is right at home, 2009.
URL http:
//www.microsoft.com/en-us/news/features/2009/jan09/
01-08cesofficeqaschultz.aspx.
[14] Herbert E Brekle. Das typographische prinzip. versuch einer
begriffsklaerung (in german), 1997. URL http://www.typeforum.
de/news_332.htm.
[15] Bill Buxton. Multi-touch systems that i have known and loved,
2007. URL http://www.billbuxton.com/multitouchOverview.
html.
[16] Jared M Diamond and Doug Ordunio. Guns, germs, and steel.
Norton New York, 1997.
63
64
bibliography
[17] Mary B Evans, Alicia A McBride, Matt Queen, Alex Thayer, and
Jan H Spyridakis. The effect of style and typography on perceptions of document tone. In Professional Communication Conference,
2004. IPCC 2004. Proceedings. International, pages 300–303. IEEE,
2004.
[18] WM Gribbons. Information design: a human factors approach
to a new typography. In Professional Communication Conference,
1993. IPCC 93 Proceedings.’The New Face of Technical Communication:
People, Processes, Products’, pages 17–23. IEEE, 1993.
[19] Yazmin Magallanes, Ariel Molina, and J Alfredo Sánchez. Combining gestures and graphical elements for collaboration using
multi-touch surfaces. In Electrical Communications and Computers
(CONIELECOMP), 2012 22nd International Conference on, pages
173–178. IEEE, 2012.
[20] Nimish Metha. A flexible machine interface. MA Sc. Thesis,
Department of Electrical Engineering, University of Toronto, 1982.
[21] ABDUL Rauf, S Batool, SU Rehman, and AA Shahid. Survey
based usability evaluation of ms word. In User Science and Engineering (i-USEr), 2010 International Conference on, pages 193–198.
IEEE, 2010.
[22] Benjamin Schwartz.
The phaistos disk, 1959.
http://www.jstor.org/discover/10.2307/543271?uid=
3737864&uid=2&uid=4&sid=21102708315857.
URL
[23] Malte Weiss, Julie Wagner, Yvonne Jansen, Roger Jennings, Ramsin Khoshabeh, James D Hollan, and Jan Borchers. Slap widgets:
bridging the gap between virtual and physical controls on tabletops. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems, pages 481–490. ACM, 2009.
D E C L A R AT I O N
I declare that the master thesis work titled:, which is presented is this
document is, to the best of my knowledge and belief, original and the
result of my own investigations and efforts, except as acknowledged,
and has not been submitted, either in part or whole, for a degree at
this or any other University.
Formulations and ideas taken from other sources are cited as such.
This work has not been published.
Paderborn, Germany
01.09.2013
Rishika Alva