Supporting Multi-Fidelity Computations
in Mobile Interactive Applications
Dushyanth Narayanan
Mobile Interactive Applications
Wearable computing = mobility + interactivity
A mobile system has
 limited battery energy
 unpredictable, turbulent network environment
 other resources are scarce
An interactive system needs
 low latency (good interactive response)
 minimal user distraction
Goal: build a system to support mobile interactive apps
2
Some examples
Augmented reality
 an architect wants to remodel a warehouse.
 an aircraft technician wants to see hidden wiring.
 a soldier needs to be aware of enemy positions.
Search
 find restaurants that I can get to within five minutes.
 find images that look like this building’s façade.
 plan an alternate driving route before I miss my exit.
Speech recognition
Image processing
 surveillance: filter out uninteresting images, report
interesting ones.
3
Why are interactive apps special?
The challenge:
provide low latency, adequate battery lifetimes
in a turbulent, resource-poor environment.
The opportunity:
degrading the result is acceptable
since users are tolerant sinks of output.
 render 3-D objects at lower resolutions for augmented reality.
 simplify search query, or search a subset of the entire space.
 use a smaller language model for speech recognition.
4
Multi-fidelity computation
Traditional algorithms have a fixed output specification.
Resource consumption is the dependent variable.
Multi-fidelity computations can have a range of outputs.
Along with the output, the fidelity is indicated.
The computation provides knobs to set the fidelity.
Lets us bound resource usage by varying fidelity.
Dependence of variables is reversed.
5
Thesis statement
Multi-fidelity computation is a viable approach to
supporting mobile interactive applications.
 There are interesting multi-fidelity, mobile,interactive apps.
 Multi-fidelity computation improves their usability.
I plan to build multi-fidelity support in Odyssey.
 Design a programming model for multi-fidelity apps.
 Support the programming model in Odyssey.
 Demonstrate a small number of proof-of-concept apps.
 Evaluate the system.
6
Outline
 Thesis statement
 Fidelity and resources
 Programming model
 Research plan
 Related work
 Contributions
7
What is fidelity?
Fidelity of a computational result:
extent to which it matches a reference result.
Multi-dimensional fidelity:
A result might have multiple fidelity metrics.
A multi-fidelity computation provides knobs to set the
fidelity metrics.
At each execution, the system has to set these knobs.
8
Example of fidelity metrics
Radiosity colors a 3-D model prior to rendering
(might use in an augmented reality app).
Two fidelity metrics:
resolution (fraction of polygons retained)
continuous, from 0 to 1
choice of algorithm (hierarchical/progressive)
discrete, with two values
At each execution, system picks the fidelity
(e.g., resolution=0.4, method=progessive).
9
Radiosity application (unmodified)
10
Multi-fidelity radiosity app
11
Multi-fidelity radiosity app
12
Multi-fidelity radiosity app
13
Multi-fidelity radiosity app
14
Fidelity and resources
Setting the knobs affects the fidelity of the result;
it also affects the resource consumption.
The resources of interest to the user are:
 Latency (user wait time)
 Battery energy
 User attention
 Money
15
What is a resource?
“Resource” is an abstraction for anything required by a
computation from its environment .
Resources
Fidelity
Latency
User attention
User
visible Battery energy
Money
CPU
Memory
User
Network
invisible
Disk bw
Cache state
Quality
metrics
16
Cache state as a resource
Caching remote objects key to avoiding network access.
Cache (or cache state) is a critical resource:
 the computation makes demands on it.
 the system provides it at varying levels.
but not a resource in the traditional sense:
 value is not necessarily a number (set of objects).
 not shared or allocated in the usual way.
 not the same resource as cache space.
Example: user wants to search a set of documents.
“Better” cache state means better latency (or fidelity).
17
User attention as a resource
A wearable computer mediates user’s reality.
Interactive app make demands on user’s attention.
A multi-fidelity system has an impact on user attention:
 Different fidelities demand different levels of attention.
 System solicits user feedback to make good tradeoffs.
18
Resource usage dependencies
CPU usage is a function of fidelity, input data size
Battery energy and latency are functions of CPU,
network, disk usage.
Network
transmit
receive
Fidelity
Data size
Cache state
Disk bw
Money
Latency
Memory
CPU
local
remote
Battery
User attention
19
Outline
 Thesis statement
 Fidelity and resources
 Programming model
 Research plan
 Related work
 Contributions
20
Interactive computations
Interactive computation:
smallest unit of computation visible to the user.
We care about multi-fidelity interactive computations.
Each execution has a set of quality values, and a utility.
For each execution of such a computation, we need to
set the fidelity to maximize the utility.
21
Programming model
The application programmer
 identifies the multi-fidelity interactive computations.
 inserts hooks for system to set the fidelity.
 inserts hooks for system to monitor the execution.
The run-time system:
 selects the fidelity.
 monitors each execution and its resource consumption.
 uses the information to improve fidelity selection.
22
Research plan
Build a system to support this scenario:
User asks radiosity app to color some scene.
Radiosity app queries system for fidelity.
System makes a fidelity selection.
monitors/predicts resources available to app.
predict’s app’s resource consumption
(using dependency graph + resource availability).
maps user-visible resources + fidelity to user happiness
(using some utility function).
searches over the fidelity space to maximize utility.
23
Mapping fidelity to utility
Computation
Fidelity 1
Fidelity 2
…
Input data
System state
CPU availability
Memory availability
Network bandwidth
Disk bandwidth
Cache state
$ cost of network
Resource
consumption
functions
Quality
Fidelity 1
Resource
consumption
CPU
Memory
Network
Disk bw
Cache state
User attention
Fidelity 2
….
Latency
User attention
Utility
Utility
function
Money
Battery drain
Energy cost of CPU
Energy cost of network
Energy cost of disk
User’s distraction level
Resource consumption functions
24
Research plan
Develop an API
 Extend API to support cache state, user attention
Implement support in Odyssey
 Predicting resource availability
 Predicting resource consumption
 Determining the utility function
 Maximizing the utility function
Demonstrate proof-of-concept applications
 Radiosity application
 Augmented or virtual reality
 Search
 Engineering computation and/or 3-D visualization
Evaluate the system
25
Multi-fidelity API
On initialization
register_fidelity({f1:continuous[0..1],
f2:discrete[hier,prog]});
hint_resource(CPU, (f2 == hier) ?
1.4 nf1log(nf1) : 2 n2f12);
hint_constraint(LATENCY, 0.05 seconds);
When the user requests a coloring operation
begin_fidelity_op(1000 input data size in polygons);
returns {f1=0.4, f2=hier} values of fidelity metrics
… run the radiosity computation …
end_fidelity_op(); signals end of interactive computation
26
Cache state and user attention
To use cache state in making fidelity decisions:
 monitor cache state, staleness of objects.
 extend API to allow disclosure of accesses.
 predict #objects accessed, hit rate from past history.
To measure user attention:
 assign cost to each aspect of user interface
(windows, menu bars, audio messages).
 extend interface to allow app hints on user attention.
 do cost/benefit when soliciting user feedback on utility.
27
Predicting resource consumption (1)
Resource consumption is a function of
input data
fidelity
How can we know the resource consumption function?
Solution: a combination of
application programmer hints
learning from a history database
We represent input data by a single feature
(typically size).
28
Predicting resource consumption (2)
Split complex function into multiple simpler functions.
E.g. battery is an application-independent linear
function of CPU, network, and disk usage.
Network
transmit
receive
Fidelity
Data size
Cache state
Disk bw
Money
Latency
Memory
CPU
local
remote
Battery
User attention
29
Learning the utility function
Two ways of learning utility or user happiness:
Implicit (outside the scope of this thesis)
Explicit
application programmer hints
feedback from user
Hints will be in the form of constraints.
programmer-specified bounds on important quality metric(s)
Constraints are simplified utility functions.
assume utility is a step-function on constrained quality metric
30
Using user feedback
User feedback:
 slider bars/buttons/preference files to set user constraints
 query the user when we have a difficult tradeoff decision
 allow user to interrupt incremental-fidelity operations
More user feedback leads to better tradeoffs.
This is not free: we are consuming user attention.
There is a meta-tradeoff:
between goodness of tradeoff and user attention.
31
Finding sweet spots
Constraints are only an approximation:
assumption that utility is a step-function.
For non-real time apps, we can fudge the constraint.
Example: a search app has a 1s latency constraint.
In 1.2s we can search all documents in the cache.
After that, we need to go over the network.
There is a knee, or sweet spot at 1.2s.
I plan to automatically identify sweet spots.
32
Evaluation
Evaluating the idea:
 How much can we impact resource consumption by adapting
the fidelity?
 Does this adaptation lead to better usability?
 How easy/difficult is it to program to the multi-fidelity API?
Evaluating the system:
 How accurately do we predict resource consumption?
 How well do fidelity selections maximize utility?
 What is the overhead of computing fidelity selections?
 How accurately do we find sweet spots?
33
Timeline
design
implementation
cache state
user attention
radiosity app
other apps
evaluation
writing
Dec-99
Mar-00
Jun-00
Sep-00
Nov-00
Feb-01
May-01
34
Outline
 Thesis statement
 Fidelity and resources
 Programming model
 Research plan
 Related work
 Contributions
35
Related work
 Data fidelity
 Odyssey [Noble97], Dynamic Distillation [Fox96]
 Real-time QoS
 Multi-resource QoS [Lee99]
 Rialto [Jones95], Darwin [Chandra98].
 Multi-fidelity computations in the real world
 Approximation algorithms, probabilistic algorithms
 Online aggregation [Hellerstein97]
 Odyssey speech recognition
 User interfaces
 Coda user patience model [Mummert95]
 Mixed-initiative user interfaces [Horvitz99]
36
Contributions
Improving the usability of a mobile interactive system
 Broad abstraction of multi-fidelity computation
 Multi-fidelity API and system support
 Support for new resources (user attention, cache state)
 Applying learning to fidelity/resource tradeoff decisions
 Demonstration of mobile interactive multi-fidelity applications
37