The Endeavour Expedition:
Charting the Fluid Information Utility
Randy H. Katz, Principal Investigator
EECS Department
University of California, Berkeley
Berkeley, CA 94720-1776
1
Why “Endeavour”?
• DARPA BAA 99-07: Information Technology
Expeditions
• To strive or reach; a serious determined
effort (Webster’s 7th New Collegiate
Dictionary); British spelling
• Captain Cook’s ship from his first voyage of
exploration of the great unknown of his day:
the southern Pacific Ocean (1768-1771).
– These voyages brought brought more land and wealth to
the British Empire than any military campaign.
– Cook’s lasting contribution: comprehensive knowledge of
the people, customs, and ideas that lay across the sea
– “He left nothing to his successors other than to marvel at
the completeness of his work.”
2
Expedition Goals
• Enhancing human understanding through
information technology
– Dramatically more convenient for people to interact with
information, devices, and other people
– Supported by a “planetary-scale” Information Utility
» Stress tested by challenging applications in decision
making and learning
» New methodologies for design, construction, and
administration of systems of unprecedented scale and
complexity
– Figure of merit: how effectively we amplify and leverage
human intellect
• A pervasive Information Utility, based on
“fluid systems technology” to enable new
approaches for problem solving & learning
3
Expedition Assumptions
• Human time and attention, not processing or
storage, are the limiting factors
• Givens:
– Vast diversity of computing devices (PDAs, cameras,
displays, sensors, actuators, mobile robots, vehicles); No
such thing as an “average” device
– Unlimited storage: everything that can be captured,
digitized, and stored, will be
– Every computing device is connected in proportion to its
capacity
– Devices are predominately compatible rather than
incompatible (plug-and-play enabled by on-the-fly
translation/adaptation)
4
Expedition Challenges
• Personal Information Mgmt is the Killer App
– Not corporate processing but management, analysis,
aggregation, dissemination, filtering for the individual
• People Create Knowledge, not Data
– Not management/retrieval of explicitly entered
information, but automated extraction and organization of
daily activities
• Information Technology as a Utility
– Continuous service delivery, on a planetary-scale, on
top of a highly dynamic information base
• Beyond the Desktop
– Community computing: infer relationships among
information, delegate control, establish authority
5
Expedition Approach
• Information Devices
– Beyond extrapolated desktop devices to MEMSsensors/actuators plus capture/display to yield enhanced
activity spaces
• Information Utility
– “Fluid”, Network-Centric System Software
» Paths/Streams: process/store/manage information
» “Movable” Processing and Storage
» Partitioned/distributed functionality
Thin-Clients/Fat-Infrastructure
» Nomadic Data
» Negotiation-based Interfaces
» Always-Available Functionality
– Wide-area distributed coordination and control on scalable
servers
6
Expedition Approach
• Information Applications
– High Speed/Collaborative Decision Making, Learning
– Augmented “Smart” Spaces: Rooms, Vehicles
• Design Methodology
–
–
–
–
HW/SW Co-design
Formal Methods
Decomposable and Reusable Components
User-centered Design
7
High Speed
Learning
Decision Making
Classroom
Collaboration Spaces
E-Book
Vehicles
Info Appliances
Applications
Human Activity Capture
Event Modeling
Generalized UI Support
Transcoding, Filtering, Aggregating
Statistical Processing/Inference
Negotiated APIs
Interface Contracts
Proxy Agents
Self-Organizing Data
Wide-area Search & Index
Nomadic Data & Processing
Wide-Area Data & Processing
Movement & Positioning
Information
Utility
Automated Duplication
Distributed Cache Management
Stream- and Path-Oriented Processing & Data Mgmt
Non-Blocking RMI
PDA
Laptop
Soft-/Hard-State Partitioning
Wallmount Display
Camera
Handset Smartboard MEMS Sensor/Actuator/Locator
Information
Devices
8
Needed Expedition Expertise
•
•
•
•
•
•
•
•
•
•
•
MEMS and hardware devices
Scalable computing architectures
Networked-oriented operating systems
Distributed file systems
Data management systems
Security/privacy
User interfaces
Collaboration applications
Intelligent learning systems
Program verification
Methodologies for HW/SW design/evaluation
9
Interdisciplinary, TechnologyCentered Expedition Team
•
•
•
•
•
•
•
•
•
•
Alex Aiken, PL
Eric Brewer, OS
John Canny, AI
David Culler, OS/Arch
Joseph Hellerstein, DB
Michael Jordan, Learning
Anthony Joseph, OS
Randy Katz, Nets
John Kubiatowicz, Arch
James Landay, UI
• Jitendra Malik, Vision
• George Necula, PL
• Christos Papadimitriou,
Theory
• David Patterson, Arch
• Kris Pister, Mems
• Larry Rowe, MM
• Alberto SangiovanniVincentelli, CAD
• Doug Tygar, Security
• Robert Wilensky, DL/AI
10
Organization:
The Expedition
Cube
D
e
s
I
g
n
M
e
t
h
o
d
o
l
o
g
y
Applications
Rapid Decision Making, Learning,
Smart Spaces: Collaboration Rooms,
Classrooms, Vehicles
Information Utility
Fluid Software, Cooperating Components,
Diverse Device Support, Sensor-Centric
Data Mgmt, Always Available, Tacit
Information Exploitation (event modeling)
Information Devices
MEMS Sensors/Actuators, Smart Dust,
Radio Tags, Cameras, Displays,
Communicators, PDAs
Base Program
Option 1: Sys Arch for Diverse Devices
Option 2: Oceanic Data Utility
Option 3: Capture and Re-Use
Option 4: Negotiation Arch for Cooperation
Option 5: Tacit Knowledge Infrastructure
Option 6: Classroom Testbed
Option 7: Scalable Heterogeneous Component-Based Design
11
Base Program: Leader Katz
• Broad but necessarily shallow investigation
into all technologies/applications of interest
– Primary focus on Information Utility
» No new HW design: commercially available information
devices
» Only small-scale testbed in Soda Hall
– Fundamental enabling technologies for Fluid Software
» Partitioning and management of state between soft
and persistent state
» Data and processing placement and movement
» Component discovery and negotiation
» Flexible capture, self-organization, info re-use
– Limited Applications
– Methodology: Formal Methods & User-Centered Design
12
Option 1: “System Architecture for
Vastly Diverse Devices”
Leader Culler
• Distributed control & resource management:
data mvmt & transformation, not processing
– Path concept for information flow, not the thread
– Persistent state in the infrastructure, soft state in the
device
– Non-blocking system state, no application state in the
kernel
– Functionality not in device is accessible thru non-blocking
remote method invocation
• Extend the Ninja concepts (thin client/fat
infrastructure) beyond PDAs to MEMS
devices, cameras, displays, etc.
13
Option 2: Implementation & Deployment of Oceanic Data Info Utility
Leader Kubiatowicz
• Nomadic Data Access: serverless, homeless,
freely flowing thru infrastructure
– Opportunistic data distribution
– Support for: promiscuous caching; freedom from
administrative boundaries; high availability and disaster
recovery; application-specific data consistency; security
• Data Location and Consistency
– Overlapping, partially consistent indices
– Data freedom of movement
– Expanding search parties to find data, using applicationspecific hints (e.g., tacit information)
14
Option 3: Sensor-Centric Data
Management for Capture/Reuse
Leader Hellerstein
• Integration of embedded MEMS with
software that can extract, manage, analyze
streams of sensor-generated data
– Wide-area distributed path-based processing and storage
– Data reduction strategies for filtering/aggregation
– Distributed collection and processing
• New information management techniques
–
–
–
–
Managing infinite length strings
Application-specific filtering and aggregation
Optimizing for running results rather than final answers
Beyond data mining to “evidence accumulation” from
inherently noisy sensors
15
Option 4: Negotiation Architecture
for Cooperating Components
Leader Wilensky
• Cooperating Components
– Self-administration through auto-discovery and
configuration among confederated components
– Less brittle/more adaptive systems
• Negotiation Architecture
– Components announce their needs and services
– Service discovery and rendezvous mechanisms to initiate
confederations
– Negotiated/contractural APIs: contract designing agents
– Compliance monitoring and renegotiation
– Graceful degradation in response to environmental changes
16
Option 5: Tacit Knowledge Infrastructure/Rapid Decision Making
Leader Canny
• Exploit information about the flow of
information to improve collaborative work
– Capture, organize, and place tacit information for most
effective use
– Learning techniques: infer communications flow, indirect
relationships, and availability/participation to enhance
awareness and support opportunistic decision making
• New collaborative applications
– 3D “activity spaces” for representing decision-making
activities, people, & information sources
– Visual cues to denote strength of ties between agents,
awareness levels, activity tracking, & attention span
17
Option 6: Info Mgmt for
Intelligent Classrooms
Leader Joseph
• Electronic Problem-based Learning
– Collaborative learning enabled by information appliances
• Enhanced Physical and Virtual Learning Spaces
–
–
–
–
–
–
–
Wide-area, large-scale group collaboration
Capture interaction once for replay
Preference/task-driven information device selection
Service accessibility
Device connectivity
Wide-area support
Iterative evaluation
18
Option 7: Safe Component
Design and UI Design Tools
Leader Sangiovanni
• Information Appliances as an application of
hardware/software codesign
– Co-design Finite State Machines (CFSMs)
– Formal methods to verify safety from faults
– Safe partitioning of components into communicating
subcomponents placed into the wide-area
• Model-based User Interface Tools
– Information device user interfaces
– Multimodal interface design for variety of devices
19
Option 8: Scaled-up Field Trials
Leader Katz
• Testbed Rationale
– Study impact on larger/more diverse user community
– Higher usage levels to stress underlying architecture
– Make commitment to true utility functionality
• Increasing Scale of Testbeds
– Building-Scale
» Order 100s individuals
– Campus-Scale
» Order 1000s individuals
– City-Scale
» Order 100000 individuals
20
Putting It All Together
1. Diverse Devices
2. Data Utility
3. Capture/Reuse
4. Negotiation
5. Tacit Knowledge
6. Classroom
7. Design Methods
8. Scale-up
Devices
Component Discovery
& Negotiation
Fluid Software
Utility
Info Extract/Re-use
Self-Organization
Applications
Group Decision Making
Learning
21
Letters of Support
• AT&T Labs, Research: Dr. Hamid Ahmadi, Networking
and Distributed Systems Research Vice President
• Cadence: Dr. Patrick Scaglia, VP Research, Cadence
Laboratories
• Hewlett Packard: Dr. Steve Rosenberg, Manager,
External Research, HP Labs
• IBM: Dr. William Cody, Manager, Exploratory Database
Systems
• Intel: Dr. Richard Wirt, Director, Intel Microcomputer
Laboratory
• Lucent/Bell Labs: Dr. William M. Coughran, Jr., Bell
Labs Research Silicon Valley Vice President
22
Letters of Support
• Microsoft: Dr. Daniel Ling, Director, Microsoft
Research
• Motorola: Dr. John Barr, Director, System of Systems
Architecture, Personal Information Networking
Division
• Nortel Networks: Dr. Daniel Pitt, VP Technology and
Director Bay Architecture Lab
• Sprint: Dr. Frank Denap, Director, Advanced
Technology Labs
• Sun Microsystems: Dr. Greg Papadopoulos, Vice
President and Chief Technology Officer
• Xerox: Dr. Mark Weiser, Chief Technologist, Palo Alto
Research Center
23
Letters of Support
24
Discussion
25