Lecture 17:
Development of and for
Devices
Jake Wobbrock
05-830
Advanced User Interface Software
1
Not a Proper Toolkit Lecture



This lecture might have been about UI
toolkits for devices
There’s just one problem
There really aren’t any


At least, not like we think of toolkits for desktop
software
Instead we will talk about “things that help
you develop for devices.”

And other interesting device issues…
2
Overview

Introduce work in multiple semi-related areas of
software development of and for devices








Palm OS
Pocket PC / Windows CE
.NET Compact Framework
J2ME
Waba
BREW
WAP and WML
DirectX
3
Overview cont.

Highlight HCI research projects with and for
devices

Phidgets


Pebbles Project


Rapid prototyping for physical devices (physical
widgets)
Interoperation of handhelds and PCs
Handheld Web browsing (various efforts)

WebThumb, Power Browser, The Gateway, M-Links
4
Motivation: Handhelds

Cell phones and PDAs have seen huge growth in
use over recent years.



“Last year [2002] there was one cell phone sold for
every 15 human beings on the planet … Worldwide
mobile phone unit sales totaled 423.4M in 2002, a 6%
increase from 2001.”
“From 1999 to 2000 the number of PDAs sold nearly
doubled, from some 3.6M units to 6.9M… By 2004, the
number of units is expected to top 33.5M.”
Source: Biz Journals
5
Motivation cont.: Gaming





Nintendo GameBoy
NTT DoCoMo Inc. handhelds running Java
Others by Qualcomm, L.M. Ericsson,
Motorola, Siemens …
Convergence: Games for phones and PDAs,
not only custom handhelds
Source: CNN
6
The Punch Line



PDAs, phones, other handhelds experiencing
great growth in use
Development community needs to have
good tools and toolkits
What is available today?

Sadly, not much (at least compared to what’s
available for the desktop)
7
Palm OS

Visit http://www.palmsource.com/products/
Palm OS by PalmSource











Acer
AlphaSmart
Fossil
Garmin
Handspring
HandEra
Kyocera
Palm
Samsung
Sony
Symbol
8
Palm OS Facts

As of January 2002:



16M Palm-powered PDAs shipped (to date)
7000 applications
150,000 registered developers


Growing at 1000 per month
Source: Palm OS Programming, O’Reilly
9
Palm OS Development

Two main ways



CodeWarrior for Palm OS
PRC-Tools (GCC)
Other ways








Sun KVM and J2ME
IBM VisualAge Micro Edition (Windows or Linux)
Jump (use Java, compiles to Motorola 68K)
Waba (subset of Java, stripped-down VM)
PocketStudio (Pascal-based, aimed at Delphi users)
Pocket C (develop on the handheld for the handheld)
Assembler SDK (ASDK) – write directly for Motorola 68K
Pocket Smalltalk (includes garbage-collecting VM)
10
CodeWarrior for Palm OS


Special line of Metrowerks CodeWarrior
releases for Palm OS development
Three key components





Editor
Constructor
Palm OS Emulator (POSE)
Write code in C/C++
Downloaded to Palm via HotSync™
11
Editor
12
Constructor
13
Emulator (POSE)


Runs on the desktop
Emulates from a “very
low level”



Designed to simulate
speed despite running
on a desktop
Useful for debugging
Not useful for
interactions requiring
pen-dexterity
14
Technical Details


Palm OS apps center around event cascade
Catch events by defining handlers and
handling them



But not nearly as much framework as Visual
Basic or MFC… more like Win32
Often involves large switch statements
Single-threaded, single-process
15
The Event Cascade
16
CodeWarrior Summary



Generally the best option
Originally for Macintosh, then for Windows!
BUT…



Constructor is often not as WYSIWYG or Direct
Manipulation as you’d like (e.g., text fields, scroll
bars)
It lacks options for all programmatically-settable
options that affect visual appearance of controls
(e.g., button borders)
Emulator doesn’t work exactly like the device

Sometimes this matters, often this doesn’t
17
PRC-Tools (GCC)



Alternate means for Palm OS development
Developed before CodeWarrior ran on
Windows by the Free Software Foundation
Collection of tools for Unix or Windows





Gnu C Compiler (GCC) for Motorola 68K
Build-PRC (combine binaries into *.prc file)
Gnu Debugger (GDB)
PilRC (compiles resource files)
Falch IDE combines these with a UI
18
Pocket PC? Windows CE?

Pocket PC

Not considered (strictly-speaking) a PDA



Meant to be a full-fledged PC but pocket-sized
Refers to a device (platform)


Or at least not marketed that way
Also to a set of standard applications common on all Pocket PC
devices, esp. Pocket PC Shell
Windows CE (WinCE)



Refers to just the OS that runs on Pocket PCs
Has large number of capabilities (components) for the
Pocket PC developer to choose from
Current version is WinCE .NET
19
Pocket PCs vs. Palm OS Devices


“Basic PDAs allow you to store and
retrieve addresses and phone
numbers, maintain a calendar, and
create to-do lists and notes. More
sophisticated PDAs can run word
processing, spreadsheet, money
manager, games and electronic book
reading programs and also provide
email and Internet access.”
Source: Microsoft
20
Pocket PCs










Toshiba
Compaq iPaq
HP Jornada
Audiovox
T-Mobile
Dell Axim
ViewSonic
Razor Zayo
Cassio Cassiopeia
NEC MobilePro
21
Pocket PC Facts

Projected 2003 sales between 14M – 17M
devices



Sales hit 10M after just 10 months on the
market



Compare to PC sales of 135M – 138M
Pocket PC projected to catch up by 2008!
Introduced April 2000
In May 2001, held 26% of market. Rest held
by Palm OS devices.
Source: C|Net
22
The Shrunken Desktop






Pocket
Pocket
Pocket
Pocket
Pocket
… ??
Word
Excel
Outlook
IE
Windows Media Player
23
Pocket PC Development

eMbedded Visual Tools 3.0 (VC++, VB)

Largely just like Windows programming


Can even use MFC for VC++
Compiles for many different processors
24
Challenge

Problem: Portability


Not all Pocket PCs utilize the same WinCE components
So not all Pocket PC apps will run on all Pocket PCs


Example: The menu bar at the bottom is specific to
Pocket PCs, not WinCE


Despite same underlying OS!
Created and passed to the Pocket PC shell
Developers often have to change much UI code for
different Pocket PC platforms

Example: Handheld PC (H/PC) is wide and short, so tall
dialogs have to be re-laid-out
25
Resource Editor
26
Pocket PC Summary



Powerful palmtop computers
More complicated than Palm OS Devices
(e.g., can be multi-threaded)
Less integrated vertically than Palm OS
devices


Complicates development (e.g., UI components
not often supported on all Pocket PC devices)
With more power and capability comes more
complexity and complication

Affects the toolkits for development on such
devices
27
.NET Compact Framework




Subset of .NET framework for desktops
Not PDA-specific but targeted to any “smart
devices”: PDAs, mobile phones, set-top boxes,
automobiles, etc.
Delivered as extensions to Visual Studio .NET called
“Smart Device Programmability”
Tools and programming model are same as for
desktop .NET platform

Easy transition from .NET desktop development to .NET
compact framework development
28
.NET Compact Framework cont.

Argument is that there are currently too many
mobile CPUs and OSes, so development is difficult


.NET would allow integration of devices across
multiple platforms, networks, and programming
languages


True – but we know the Microsoft answer
A vision behind .NET in general
Compact Framework is a subset of all .NET
capabilities

Reduces footprint on resource-strained devices
29
J2ME – Java 2 Micro Edition

Uses subset of Java Virtual Machine for
smaller footprint (i.e., KVM)



Highly optimized runtime environment
Not PDA-specific: smart cards, pagers, mobile
phones, set-top boxes, vehicle telematics
systems (e.g., OnStar system)
Set of APIs defined by the Java Community
Process Program™

Includes user interface, security, networking
protocols, and more
30
J2ME Architecture

J2ME Architecture defines…

Configurations: A virtual machine and a minimal
set of class libraries


For Palm OS, configuration is “Connected Limited
Device Configuration” (CLDC)
Profiles: Higher-level APIs that further define the
application, user interface, and device-specific
properties

For Palm OS, profiles are PDA and “Mobile
Information Device Profile” (MIDP)
31
From http://java.sun.com/j2me/j2me-ds.pdf
J2ME Architecture cont.
32
Waba

Subset of Java




Stripped-down VM: small and fairly fast
No longs, doubles, exceptions, or threads
Open source: http://www.wabasoft.com/
Waba defines…




A language (strict subset of Java)
A virtual machine (Waba VM)
A class file format (strict subset of Java
bytecode)
A set of foundation classes
33
BREW

Qualcomm’s Binary Runtime Environment for
Wireless (BREW)


Like J2ME, provides application execution
environment
Goes farther! Provides business model for
certifying, downloading, and charging premium
content


With J2ME, this is left entirely to the marketers and
developers to figure out for themselves
Based on C++ (so it is fast, and small binaries)
34
WAP and WML

WAP (Wireless Application Protocol)




WML (Wireless Markup Language)



Provides “flat” Web content on handhelds
Requires constant connectivity by the device
Lacks a rich interactive experience for end-users
Specifies content and user interface for WAP-delivered
information
Based on XML: so describes data, not just presentation
of data
WMLScript


WAP:WML:WMLScript :: HTTP:HTML:JavaScript
Provides only minimal interactivity
35
WAP and WML cont.

Four things specified by WML





Text and image presentation and layout, including a
variety of formatting commands
Deck/card organizational metaphors for information
(similar to HyperCard)
Inter-card navigation and linking
Card parameterization and state management
The fate of WAP


Hasn’t been the success its advocates hoped
New push to make possible highly interactive content

J2ME, BREW, .netCF, Waba, but these all send a VM and increase
footprint
36
Game Development: DirectX

DirectX: Advanced suite of multimedia APIs



DirectDraw, Direct3D, DirectSound, DirectMusic,
DirectInput, DirectPlay, DirectShow…
Allows accessing of special hardware capabilities
(e.g., graphics and sound cards) without
requiring hardware-specific code
DirectX exists for Pocket PC


Handheld gaming projected to be a big new
market opportunity
Again, similar to programming for desktop
37
Transition from (Mainly)
Industrial Tools to Research
Initiatives for and with
Devices
38
Phidgets (Greenberg et al.)

Phidgets are “physical widgets”




Play the same role in physical UIs as widgets do
in graphical UIs
Package IO
Hide implementation details, provide abstraction
Have additional characteristics



Require a connection manager
A link between the software and hardware parts
A simulation mode in software only
39
Phidget Architecture

Physical Device


Wire Protocol


Generic interface which all phidgets have. End-programmers use it for basic identification and
capabilities.
Phidget-specific interface


Created by the Phidget Manager whenever a device is seen. They correspond 1:1 to physical
devices.
IGlabPhidget interface


COM object that has event-based API for end-programmers for connection management.
Phidget-specific COM objects


Communication protocol between device and host computer. Not visible to end-programmers.
Phidget Manager


Packaged physical unit to be used by the designer. Most phidgets are built around a circuit board.
Capabilities specific to a particular phidget. Extends the basic IGlabPhidget interface.
Phidget ActiveX controls

Wraps up the phidget code and provides an on-screen user interface and the ability to simulate
the phidget as an control.
40
Pebbles (Myers et al.)


Research project exploring the interoperation
of handhelds and PCs
Handhelds can be an aspect of the user, PCs
can be a fixture of the environment


Handhelds can control other devices too


Users may move from environment to
environment, PCs may stay behind
Personal Universal Controller
Handhelds can be used to control a PC by
people with motor impairments
41
A Current Research Issue


Handheld User Interfaces for Web Browsing
Challenges




Constrained screen-space
No mouse cursor (e.g., no link roll-overs)
Pages designed on a desktop for display on a
desktop
Scrolling more difficult

Graphic refresh can be slow, scrolling blurry
42
Handheld Web Browsing

The Gateway (MacKay)


WebThumb (Wobbrock et al.)


Exploration of interaction techniques for better handheld
browsing (UIST 2002).
M-Links (Trevor et al.)


Pages displayed as thumbnails expand as certain parts of
them are interacted with (CHI 2003).
Uses modes to separate action from navigation on
mobile phone Web pages (UIST 2001).
Power Browser (Buyukkokten et al.)

Generates summary views of handheld web content for
display (CHI 2000).
43
Conclusions





Toolkits for development for devices are far from
Amulet-level sophistication
Many processors and devices exist – complicates
development and toolkit design
Numerous standards and niche technologies
Minimizing footprint on resource-constrained
devices is a key constraint
HCI research can tackle issues beyond the specifics
of device hardware and software


e.g., interaction techniques for PC/PDA interoperability
e.g., interaction techniques for browsing
44