Handheld Multimodal Interfaces
Technology, Markets, and
Opportunities
Jordan Cohen
CTO Voice Signal
February 2005
Baltimore, Maryland
Multimodal Communications Thru the Ages
Characteristics of Multimodal Interfaces
 Transparent
 Easy to Learn
 Easy to use
 Robust
 Flexible
 More than one input mode
 Mode Choice
 Adaptability over Time (Mobile Requirement)
 Efficient
 Accommodate Adverse Conditions
 Stable
 Combined Modes often Quicker
 Superior Error Handling
 Simplified Language
 Mode Switching Facilitates Error Recovery
 Mutual Disambiguation Possible for Simultaneous Inputs
– Sharon Oviatt, “Multimodal Interfaces”, 2002
The First Computer
The Babbage
Difference Engine
(1832)
25,000 parts
cost: £17,470
The Mainframe - IBM 3090

Famous Predictions
 “I think there is a world market for maybe five
computers”
(Thomas Watson, Chairman of IBM, 1943)
The Easy-To-Use PC
Down to the Pocket PC
The Shape of Multimodal Interfaces
Famous Predictions
 “This ‘telephone’ has too many shortcomings to
be seriously considered as a means of
communication. The device is inherently of no
value to us.”
Western Union internal memo, 1876.
The Early Car Phone
And There Were Cell Phones
 Motorola Made the Phone Portable!
 Martin Cooper, April 3, 1973
 First Public Cell Phone Call (non-car phone)
The Ubiquitous Computing Platform
Cell
Phone
Small
Signal RF
Digital Cellular Market
(Phones Shipped)
Power
RF
Power
Management
1996 1998 2000 2004*
Units
48M 162M 435M 600M*
Analog
Baseband
Digital Baseband
(DSP + MCU)
data from Texas Instruments
* - Projection
MP3 players – what are they?
 Name comes from type of file (MPEG), audio
layer three – MP3
 Key to popularity is file compression
 Some phones have MP3 play back capability
 Usually separate units
TODAY’S 3D MOBILE GAME HARDWARE
Things to Think About: Features
 Text (SMS)
 Wireless Internet (3G)
 Voice Dialing
 MP3
 Ringtones
 Push-to-talk
 Games
 Photo/Video
 Headsets (Bluetooth)
 Chargers
A Sign of the Times
“ I have always wished that my computer
would be as easy to use as my telephone.
My wish has come true. I no longer know
how to use my telephone.”
Bjarne Stroustrop (originator of C++)
Over 23,000 available Palm OS® titles!
 Enhanced Personal
Information Managers



Calendars
Address Books
Application Launchers
 Time management tools
 Advanced calculators
 Statistical tools
 Graphics design tools
 Audio design tools
 Thousands of ebooks
Business and Professional
 CRM
 Education
 Finance
 Government
 Health care
 Law
 Real estate
 Reference books
 SFA
 U.S. Navy
www.palmone.com/us/software
Killer app?
 Obvious strong sellers:
 Games, especially casual
 PIM and calendar tools
 Less obvious:
 Financial services
 Health care
 Emergency services
 Chat and dating services
 Average price of mobile applications:
 $16 on Handango (mostly PDA)
 $5 on carriers (mostly Java and BREW)
Strategy Analytics Survey (Kerr, 2004)
Evolution of Mobile Services
Messaging
Hi, I’ll see
you at 6
PM
Regards,
Bob
Basic text
messagin
g
Corporate access
Multimedia
messaging
with pictures
and audio
Multimedia
messaging
with video
Dial-up
connection
s
Browsing
Instant
connectivity,
colour
Black and white screen,
WAP Push
screens,
simple graphics
Always-on,
performance
and security
enhancements
Accelerated
connections,
voice and
data
multitasking
Downloading
xHTML for
WAP and web
integration,
animated
content
Video content
(MMS,
streaming)
Downloadable
applications,
Ringing tones, icons, MIDI sounds
screensavers,
business cards
Games
Mobile gaming revenue prediction [19], Play Away: The future of mobile entertainment (2002)
-
Other predictions to year 2006: $3,6 billion and $4,4 billion [21]
Mobile gamers in USA 7M in 2002 (IDC estimate), 2007 112,4M [21]
New mobile devices (big color displays, joystick, voices..)
Different run-time environments: Java key technology [20]
Licenced formats e.g. Codetoys ”Who Wants to be a Millionaire?”
Mobility enabling positioning based and other on-line interactive games
Entertainment in home, bus... Affordable device nearly all (in future)
Smart phones
 Smart phones as hybrid
products


Cell phone features (esp. voice)
PDA features (esp. PIM)
 Indirect market entry by
PDA industry
incumbents



Nokia
palmOne
Microsoft
 Industry ambiguity
 New suppliers, competitors,
customers
 Competition from separate
products
 Cross-industry alliances to
share capabilities
Smart phone
Cell phone
industry
PC, PDA, Cell Phone sales (20002002)
Millions of Units
410
390
Cell Phones
405
11
PDA's
2000
12.8
2001
11.7
2002
130
PC's
123
128
Source: Assinform / NetConsulting
Operating Systems
 Smartphone
 Windows Powered environment using familiar
Microsoft software
 Outlook capabilities, Windows Media Player,
infrared beaming, and MSN Messenger
 Programmable using Microsoft Mobile Development
Toolkit (incl. in The Smartphone Developer Kit)
 www.microsoft.com/mobile/smartphone/default.asp
Operating Systems
 Symbian OS Version 7.0s
Voice Activated Phonebook (VAP): Dialing with
Discrete Commands
•No user training: Unlike old voice dialing, VAP
works right out of the box, and voice-activates
every name in the phonebook, no matter how
large.
•Multimodal: User can dial by voice, or key
press. Dialogs are presented on the display,
and played through the earpiece for total
eyes free dialing
•Eyes-free, helpful voice guide: Intuitive
dialog is easy to use the very first time and
doesn’t require looking at the display.
•Listening indicators: Users can see visual (ear
icon) and hear (beep) cues when to speak.
•Dial from the entire phonebook: When
multiple numbers are stored, the system asks
which one to dial.
Connecting…
Sayathe
Name:
Which
Number?
Say
Command:
John
Smith
John
Smith
John
Smith
Digit
Dial
Mobile
Name Dial
555-1212
Quick Dial
Voice Memo
Home
Phone Book
Mobile
Browser
Work
Settings
Settings
Cancel
Help
Click to
Start VSuite
VSuite Core Feature Index
Comments from Time.com
"...Now there's something infinitely better: it's called
"speaker-independent" voice recognition, and the
best one out by far is from Voice Signal...this
million-dollar technology might already be in your
phone. Have a look. If not, trade it in, because this
stuff is awesome!..."
WILSON ROTHMAN
Time Magazine, “Gadget of the Week”, Feb. 4, 2004
Voice Activated Phonebook (VAP): Dialing with
Natural Commands
•Freedom, Flexibility, Convenience:
Natural Commands enhance the VSuite 1.x
experience by allowing users to accomplish
tasks with single, natural voice commands.
•What – Who – Where Paradigm
•What ?
Call, Send Message?
•To whom?
Person or Number?
•To which device?
Mobile? Email? Pager?
•Smart Dialog: When VSuite isn’t sure what
you said, it will ask, “Did you say?” allowing
for instant correction.
Connecting…
Say
a Command:
John
CallSmith
[number]
Mobile
Call [name]
555-1212
Go To [Application]
Voice Message [Name]
Lookup [Name]
Send [Message] to [ ]
Settings
Help
Cancel
Click to
Start VSuite
VSuite Upgrade Index
Voice Activated Message Setup
•Fastest and Simplest Method to Set Up
and Address Messages:
•Single key press
•Compatible with any message
client and protocol (SMS, EMS, MMS,
Email, Instant Messenger,)
•In just a few seconds, the messaging
client is open, addressed, and waiting for
text input.
•Any name or number in phonebook
can be addressed by voice
Say a Command:
John Smith
Call [number]
Call [name]
Go To [Application]
Voice Message [Name]
Lookup [Name]
Send [Message] to [ ]
Settings
Send
Help
ABC
Click to
Start VSuite
VSuite Upgrade Index
A Messaging Application
 Demonstration
Customers? - Nokia prognosis for
3G mobile communication
Monthly income pr. user in euro (1 euro = 1 USD ca.)
100
Location based services
90
Div. telecomm.
80
Commercials
Text messages
Entertainment
70
Information services
Payment transactions
Music and video
Internet surfing
Download from internet
Chat on internet
Multimedia messages
Photo messages
60
50
40
Vide conferencing
30
20
Normal speech
10
Fixed subscription
fees
0
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
Number of people online by language
spoken (http://www.glreach.com/globstats/)
British Student Cell Phone Penetration
 A quarter of 7 to 10-year-olds own mobile
phones, according to a survey of 2,000 children
and their parents by the market analysts Mintel.
The figures show that the proportion of primary
school pupils with mobiles has almost doubled
from 13% in 2001.

(Telegraph, 28 April 2004)
 There are as many cell phone in Britain as
there are people 10 years old or more.
 96% of 15 to 24-year-olds own a cellphone and
cannot function normally without one
Messaging
 Short Message Service (SMS)
 Up to 160 characters of text in length: words or
numbers or an alphanumeric combination
 Non-text based short messages (for example, in
binary format) are also supported. These are used
for ringtones and logos services for instance
 SMS is a store and forward service: short messages
are not sent directly from sender to recipient, but
always via an SMS Center (SMSC)
 Short messages can be sent and received
simultaneously with GSM voice, Data and Fax calls
Smileys – from the SMS world!
:) Original smiley
~#:-( Bad hair day
:-) Classic smiley
(-: Sender is left-handed
:-( Frown
{} No comment
:@ Ouch!
<:-) Dumb question
:-& Tongue-tied
#-) Partied all night
:-# My lips are sealed
:-$ Put your money where
your mouth is
Messaging
 Multimedia Message Service (MMS)
 The data speed in the networks must be at least
14.4 kbit/s for carrying MMS messages.
 MMS messages are currently between 30 kB and
100 kB in size
 They can contain still images (JPEG, GIF, WBMP),
text, voice (AMR voice), audioclips or videoclips
 MMS transport is done using WAP over GPRS (or
CSD) transport. WAP push features are used to
deliver the message from server to receiving phone.
British Commentary on Messaging:
 Texts overtook phone calls as the favoured
means of contacting someone by mobile in
2003. Research found more than eight out of
10 under-25s are more likely to text than call.
At New Year revellers smashed the record for
the most messages in a day, tapping out 111
million in the first 24 hours of 2004
 Nearly a quarter of British teenagers textmessage their friends more often than they talk
to their families according to a study of more
than 400 girls carried out on behalf of the
Guide Association. A fifth claim never to talk to
parents and siblings.
From www.didyouknow.cd/sms



If the universe did start with a
bang
when God loved and the angels
sang
one of the sparks that flew
chased time to become you
if kisses wer rain id send u
showers,
if fun was time id send u hrs,
if u needed a frnd id send u me!
As i lie on my bed and look at the
stars, and the moon, and the sky...
i wonder...
WHAT HAPPENED TO MY
ROOF?!?!?!?

here's my halo
made of glass
mess with me
and i'll kick ur ass

when God gave us friendships he
tried to be fair, but when i got you,
i got more than my share!

Ek dwaal doeloos verby Jupitor.
Waar is jy? Ek verdwaal iewers in
die Amasone maar waar is jy? In
my drome soek ek jou maar jy is
waar jy moet wees... in my hart
From Tech Central Station
 A survey of Italian children between the ages of
nine and ten found that, incredibly, 56 percent
owned cell phones. This was the finding of a
study published on 8 July 2003, coordinated by
Francesco Pira, a lecturer at the University of
Trieste.
 New research released by the research group,
mobileYouth has highlighted the growing
number of young children with mobile phones
in Britain . The study conducted across the 524 year old market highlights that there are
now 400,000 children in the UK under the age
of 10 owning a mobile phone, as against only
80,000 in 2000. (Cellular News, 12 Aug, 2003)
From IT Facts (www.itfacts.biz)
 Taiwan and Hong Kong lead the world in terms
of mobile penetration - there are 110 mobile
phones for every 100 inhabitants of Taiwan,
and 105.75 mobile phones for every 100
people in Hong Kong. Other countries ranked
lower in International Telecommunications
Union annual survey, with US reaching 57%
mobile penetration by the end of 2004

Dec 18, 04
 Gartner said that it expected 20 mln
smartphones to ship in 2006 compared to
about 13 mln PDAs
More on phone handsets
 By 2006, more ‘smart’ phones than




PCs.
80% of new phones will be color.
80% of new phones will have Internet.
74% handsets sold this year will be
GSM.
Tip: Use GSM handsets with shared
plans.
 Internationalization esp. imp. in mobile
market:


By 2006, Asia Pacific will have 45% of
mobile subscribers.
433 million handsets this year, 35% sold in
China alone.
Thank You
The End
Jordan Cohen
Jrc@voicesignal.com