Networks of Tiny Devices
Embedded in the Physical World
David Culler
Computer Science Division
U.C. Berkeley
www.cs.berkeley.edu/~culler
Intel Research
Berkeley
Technology Push
• Complete network embedded systems going
microscopic
Processing
Storage
Power
Sensing
Actuation
Communication
I SDQ SD
PLL baseband
filters
mixer
LNA
5/13/2002
TinyOS IPAM
2
Application Pull
• Complete NW embedded systems going microscopic
• Huge space of new applications
Monitoring & Managing Spaces
Disaster
Management
Habitat Monitoring
Circulatory Net
Condition-based
maintenance
Ubiquitous
computing
5/13/2002
TinyOS IPAM
3
Bridging the Technology-Application Gap
• Power-aware, communication-centric node
architecture
• Tiny Operating System for Range of HighlyConstrained Application-specific environments
• Network Architecture for vast, self-organized
collections
• Programming Environments for aggregate
applications in a noisy world
• Distributed Middleware Services (time, trigger,
routing, allocation)
• Techniques for Fine-grain distributed control
• Demonstration Applications
5/13/2002
TinyOS IPAM
4
Critical issues
• Highly constrained devices
– power, storage, bandwidth, energy, visibility
– primitive I/O hierarchy
• Observation and action inherently distributed
– many small nodes coordinate and cooperate on overall task
• The structure of the SYSTEM changes
• Devices ARE the infrastructure
– ad hoc, self-organized network of sensors
• Highly dynamic
–
–
–
–
–
passive vigilance most of the time
concurrency-intensive bursts
highly correlated behavior
variation in connectivity over time
failure is common
5/13/2002
TinyOS IPAM
5
A de facto platform for EmNets
• Developed a series of wireless sensor devices
•
•
•
•
•
TinyOS concurrency framework
Messaging Model
Networking stacks (RF and Serial)
Multihop routing
Several Key components
– sensing, logging, data filters, broadcast
• Simulation tools
• DARPA NEST OEP
5/13/2002
USC robomote
TinyOS IPAM
6
Many Research Groups using it
• UCB
–
–
–
–
–
–
–
•
•
•
•
•
•
NEST
SensorWeb
Blackout
Glaser structures
CBE
BFD
BRWC
UCLA
USC, ISI
Rutgers winlab
Intel
Bosch
Crossbow
5/13/2002
•
•
•
•
•
•
•
•
•
•
•
•
U Wash
Rutgers
UIUC
NCSA
U Virginia
Ohio State
UCSD
Dartmouth
MIT
UT Austin, ASU, Iowa
Accenture
and many more
TinyOS IPAM
7
The MICA architecture
51-Pin I/O Expansion Connector
8 Programming
Digital I/O 8 Analog I/O
Lines
• Atmel ATMEGA103
– 4 Mhz 8-bit CPU
– 128KB Instruction Memory
– 4KB RAM
DS2401 Unique ID
Atmega103 Microcontroller
– SPI interface
– 1-4 uj/bit r/w
Transmission
Power Control
• RFM TR1000 radio
Hardware
Accelerators
TR 1000 Radio Transceiver
– 50 kb/s – ASK
– Focused hardware acceleration
SPI Bus
• 4 Mbit flash (AT45DB041B)
Coprocessor
4Mbit External Flash
Power Regulation MAX1678 (3V)
• Network programming
• Rich Expansion connector
– i2c, SPI, GIO, 1-wire
– Analog compare + interrupts
• TinyOS tool chain
• sub microsecond RF
synchronization primitive
5/13/2002
TinyOS IPAM
2xAA form factor
8
Rich Sensor board
Mica PINS
PHOTO
Y Axis
X Axis
TEMP
Tone
Intr
Mic
Signal
SOUNDER
MICROPHONE
Gain Adjustment
MAGNETOMETER
ACCELEROMETER
ADC Signals (ADC1-ADC6)
On/Off Control
I2C Bus
Interrupt
Microphone
Sounder Magnetometer
1.25 in
Temperature
Sensor
5/13/2002
TinyOS IPAM
Light
Sensor
2.25 in
Accelerometer
9
More Sensors and Actuators
• Motor-Servo board interfaces any
combination of two motors, servos, and
solenoids to a toy car platform
• whisker board for obstacle detection
• digital accelerometer (ADXL202)
board for crude odometry
• GPS Board
• Weatherboard
• light, temp, humidity, barometric
pressure, occupancy (thermopile)
5/13/2002
TinyOS IPAM
10
Getting Ready for Outdoors
Acadia National Park
Mt. Desert Island, ME
Great Duck Island
Nature Conservancy
5/13/2002
TinyOS IPAM
11
A Operating System for Tiny Devices?
• Traditional approaches
– command processing loop (wait request, act, respond)
– monolithic event processing
– bring full thread/socket posix regime to platform
• Alternative
– provide framework for concurrency and modularity
– never poll, never block
– interleaving flows, events, energy management
=> allow appropriate abstractions to emerge
5/13/2002
TinyOS IPAM
12
Tiny OS Concepts
• Scheduler + Graph of Components
init
Commands,
Event Handlers
Frame (storage)
Tasks (concurrency)
Messaging Component
– frame per component, shared stack, no
heap
• Very lean multithreading
• Efficient Layering
5/13/2002
TinyOS IPAM
internal thread
Internal
State
TX_pack
et_done
(success
RX_pack
)et_done
(buffer)
• Constrained Storage Model
init
Power(mode)
TX_packet(buf)
–
–
–
–
msg_rec(type, data)
msg_sen
d_done)
• Component:
Events
send_msg
(addr,
type, data)
Commands
power(mode)
– constrained two-level scheduling model:
threads + events
13
application
Application = Graph of Components
Route map
router
sensor appln
packet
Radio byte
bit
Radio Packet
byte
Active Messages
RFM
5/13/2002
Serial Packet
UART
Temp
photo
SW
HW
ADC
clocks
TinyOS IPAM
Example: ad hoc, multi-hop
routing of photo sensor
readings
3450 B code
226 B data
Graph of cooperating
state machines
on shared stack
14
TOS Execution Model
• commands request action
– ack/nack at every boundary
– call cmd or post task
message-event driven
• events notify occurrence
• Tasks provide logical
concurrency
event-driven packet-pump
packet
HW intrpt at lowest level
may signal events
call cmds
post tasks
active message
5/13/2002
crc
Radio byte
encode/decode
event-driven bit-pump
bit
– preempted by events
• Migration of HW/SW
boundary
Radio Packet
event-driven byte-pump
byte
–
–
–
–
data processing
application comp
TinyOS IPAM
RFM
15
Dynamics of Events and Threads
bit event filtered
at byte layer
bit event =>
end of byte =>
end of packet =>
end of msg send
thread posted to start
send next message
radio takes clock events to detect recv
5/13/2002
TinyOS IPAM
16
Maintaining Scheduling Agility
• Need logical concurrency at many levels of the
graph
• While meeting hard timing constraints
– sample the radio in every bit window
Retain event-driven structure throughout
application
Tasks extend processing outside event window
All operations are non-blocking
lock-free scheduling queue
5/13/2002
TinyOS IPAM
17
Demonstration applications
• 29 Palms
• Cory Hall network
– ½ million packets over 3 weeks
•
•
•
•
•
•
•
Surge network and environment display
800 node ad hoc network
CBE
Glaser Shakes
Granlibakken retreat watcher
Robomote
Group response
=> continued application focus
• more real and long lived
• more dynamics
• extract architecture and create framework
5/13/2002
TinyOS IPAM
18
Example TinyOS study
• UAV drops 10 nodes along road,
– hot-water pipe insulation for package
•
•
•
•
•
•
•
•
Nodes self-configure into linear network
Synchronize (to 1/32 s)
Calibrate magnetometers
Each detects passing vehicle
Share filtered sensor data with 5 neighbors
Each calculates estimated direction & velocity
Share results
As plane passes by,
– joins network
– upload as much of missing dataset as possible from each node when
in range
• 7.5 KB of code!
• While servicing the radio in SW every 50 us!
5/13/2002
TinyOS IPAM
19
Structural performance due to multi-directional
ground motions (Glaser & CalTech)
.
Mote
Layout
1
3
1
54
6`
1
8
1
1
1
5
29
Mote infrastructure
Comparison of Results
Wiring for traditional
structural instrumentation
+ truckload of equipment
5/13/2002
TinyOS IPAM
20
Energy Monitoring/Mgmt System
•
•
•
•
50 nodes on 4th floor
5 level ad hoc net
30 sec sampling
250K samples to database
over 6 weeks
5/13/2002
TinyOS IPAM
21
Energy Monitoring Network Arch
20-ton
chiller
sensor net
GW
control net
GW
GW
802-11
PC
modbus
PC
MYSQL
telegraph
scada term
UCB power
monitor net
5/13/2002
Browser
TinyOS IPAM
22
Meeting Social Network
5/13/2002
TinyOS IPAM
23
Wealth of Research Challenges
• Large numbers of highly
constrained (energy &
capability), connected devices
service
• New family of issues across all
the layers
network
system
architecture
technology
5/13/2002
TinyOS IPAM
24
prog / data model
– operating in aggregate
application
algorithm / theory
– imperfect operation and reliability
mgmt / diag / debug
– able to be casually deployed in
infrastructure (existing or in design)
Node Communication Architecture
Classic Protocol
Direct Device
Processor
Control
Application Controller
Application Controller
Narrow, refined
Chip-to-Chip Interface
RF Transceiver
Protocol Processor
RF Transceiver
Hybrid Accelerator
Application Controller
Serialization Accelerator
Memory I/O BUS
Raw RF Interface
Timing Accelerator
Hardware Accelerators
RF Transceiver
5/13/2002
TinyOS IPAM
25
Novel Protocol Examples
• Low-power Listening
• Really Tight Application-level Time
Synchronization
• Localization
• Wake-up
• MACs
• Self-organization
5/13/2002
TinyOS IPAM
26
Low-Power Listening
• Costs about as much to listen as to xmit, even
when nothing is received
• Must turn radio off when there is nothing to hear.
• Time-slotting and rendezvous cost BW and
Latency
• Can turn radio on/of in <1 bit
Small sub-msg recv sampling
Trade small, infrequent
tx cost for freq. Rx savings
Low Power Listening
Preamble
Transmission
Start Symbol
Transmission
MAC Delay
Transmitting individual bits
Bit Modulations
Radio Samples
Slow, Periodic Sampling
Reception
5/13/2002
TinyOS IPAMStart Symbol Search
Receiving individual bits
Synchronization
Start Symbol Detection
27
Exposing Time Synchronization Up
• Many applications require correlated data
sampling
• Distributed time sync accuracy bounded by ½
the variance in RTT.
• Successful radio transmission requires sub-bit
synchronization
Provide accurate timestamping with msg
delivery
Jitter < 0.1us (propagation) + 0.25 us (edge
capture accuracy) + 0.625 us (clock synch)
5/13/2002
TinyOS IPAM
28
Localization
• Many applications need to
derive physical placement
from observations
– Spatial sampling, proximity,
context-awareness
• Radio is another sensor
• Sample baseband to
estimate distance
– Need a lot of statistical data
– Calibration and multipleobservations are key
Error
Error
Noise
Noise
• Acoustic time-of-flight
alternative
– Requires good time
synchronization
5/13/2002
TinyOS IPAM
29
Probability
Statistical Approach
Y
5/13/2002
Node A
X
Distance
TinyOS IPAM
Node B
30
Integrated Architecture
• Chip Area ~5 mm2
–
–
–
–
AVR core with protocol Accelerators
16 Kbytes on-chip ram
ADC
800-1GHz FSK transceiver, -90dBm receive sens’y
.5 mm2
4 mm2
.5 mm2
• Expected sleep current = 1 uW
– lifetime on a single AA = 400+ years
• Expected active (processing current)
– Processor @ 4 Mhz < 1 mW
– Radio: 1mW power consumption, 100Kbps
– ADC: 20 pJ/sample 10 Ksamps/second = .2 uW.
5/13/2002
TinyOS IPAM
31
Networking
• Hands-on Experience with Large
Networks of Tiny Network sensors
intense constraints, freedom of abstraction
• Re-explore entire range of networking
issues
–
–
–
–
–
–
–
–
encoding, framing, error handling
media access control, transmission rate control
discovery, multihop routing
broadcast, multicast, aggregation
active network capsule (reprogramming)
localization, time synchronization
security, network-wide protection
density independent wake-up and proximity est.
• Fundamentally new aspects in each
5/13/2002
TinyOS IPAM
32
The Nodes are the Infrastucture
• Simple Epidemic Algorithm Schema
if (new mcast) then
take local action
retransmit modified request
• Examples: Network wakeup, command
propagation
– Build spanning tree
» record parent
• Naturally adapts to available connectivity
• Minimal state and protocol overhead
=> surprising complexity in this simple mechanism
5/13/2002
TinyOS IPAM
33
Network Discovery: Radio Cells
5/13/2002
TinyOS IPAM
34
Network Discovery
5/13/2002
TinyOS IPAM
35
Controlled Empirical Study
• Experimental Setup
–
–
–
–
–
–
–
13x13 grid of nodes
separation 2ft
flat open surface
Identical length antennas, pointing vertically upwards.
Fresh batteries on all nodes
Identical orientation of all nodes
The region was clean of external noise sources.
• Range of signal strength settings
• Log many runs
5/13/2002
TinyOS IPAM
36
Example “epidemic” tree formation
5/13/2002
TinyOS IPAM
37
Final Tree
5/13/2002
TinyOS IPAM
38
Power Laws ?
1000
100
100
Links
Count
1000
10
10
1
1
1
10
100
1
10
100
Cluster Size
Cluster Size (1 + # children)
• Most nodes have very small degree (ave = .92)
• Some have degree = 15% of the population
• Few large clusters account for most of the edges
5/13/2002
TinyOS IPAM
39
Open Territory => Many Children
• Example: Level 1
5/13/2002
TinyOS IPAM
40
Open Territory => Many Children
• Example: Level 2 – variation in distance
5/13/2002
TinyOS IPAM
41
Open Territory => Many Children
• Example: Level 3 – long links
5/13/2002
TinyOS IPAM
42
Understanding Connectivity
• 16 transmit power settings
• For each transmit power setting,
each node transmits 20 packets.
• Receivers log successfully
received packets.
• Nodes transmit one after the other
in a token-ring fashion
• No collisions.
• Define “range”: radius where 75%
of enclosed nodes receive 75% of
packets
• Often good nodes at a distance
probability of reception from center
node vs xmit strength
5/13/2002
TinyOS IPAM
43
Importance of Asymmetric Links
• Asymmetric Link:
– >65% successful reception in one direction
– <25% successful reception in the other direction
• 10%-25% of links are asymmetric
• Many long links are asymmetric
– in large field it is likely that someone far away can hear you
– what does this mean for protocol design?
5/13/2002
TinyOS IPAM
44
Collisions are primary factor
• Nodes out of range may have overlapping cells
– hidden terminal effect
• Collisions => these nodes hear neither ‘parent’
– become stragglers
• As the tree propagates
– folds back on itself
– rebounds from the edge
– picking up these stragglers.
• Seen in many experiments
• Mathematically complex because behavior is not
independent beyond singe cell
5/13/2002
TinyOS IPAM
45
Stragglers
• significant fraction of links point ‘backwards’
5/13/2002
TinyOS IPAM
46
Minimal lessons learned
• Don’t think about wireless networks as bunch of
circles of radius r
– connectivity is a probability distribution
– falls off with distance, but not as simple fading law
– shape varies with time and context
• With large, dense arrays the low-probability
events are common
• Must strike a balance in exploiting structure and
adapting to observed behavior
• Want simple local rules that have predictable,
robust global behavior
5/13/2002
TinyOS IPAM
47
More typical routing for sensor nets
• Current applications dominated by data
acquisition
– route from many nodes to nearest gateway
– aggregate from many nodes
– routing determined by simple local rules
• Nodes listen to data transfers from neighbors
–
–
–
–
–
carries hop-count info
monitors link goodness of potential ‘parents’
dynamically selects best node is lesser hop count
includes hysterisis and continuous rediscovery
gateways emit null data with 0-hop
• Much to understand about how such algorithms
manage major change
5/13/2002
TinyOS IPAM
48
Self-propagating Programs?
Network Programming Rate
100%
Percent
Programmed
• TinyOS components
support class of applns.
• Tiny virtual machine
adds layer of
interpretation for specific
coordination
• Primitives for sensing
and communication
• Small capsules (24
bytes)
• Propagate themselves
through network
80%
60%
40%
20%
0%
0
5/13/2002
TinyOS IPAM
20
40
60
80
100 120 140 160 180 200 220 240
Tim e (seconds)
49
Multihop Bandwidth Management
• Should self-organize into fair, dynamic multihop net
• Hidden nodes between each pair of “levels”
– CSMA is not enough
• P[msg-to-base] drops with each hop
– Investment in packet increases with distance
– need to optimize for low-power fairness!
• RTS/CTS costly (power & BW)
 Local rate control to approx. fairness
 Priority to forwarding, adjust own data rate
 Additive increase, multiplicative decrease
 Listen for retransmission as ack
5/13/2002
TinyOS IPAM
50
Example: Multihop Adaptive Transmission
Control
Max rate: 4 samples/sec
- rate = 4p
Channel BW ~20 p/s
1
8
- cannot expect more than 1/3
thru parent
Monitor number of children (n)
1
5
1
4
1
6
1
7
B
a(n) ~ 1/n
b=½
p’ = p + a(n) on success (echo)
p’ = p * b
node
14
15
16
17
18
5/13/2002
mean
0.36
0.56
0.55
0.55
0.39
p/s (COV)
(64%)
(14%)
(11%)
(12%)
(11%)
without rate control,
success drops ~½ per hop
TinyOS IPAM
51
Key Experience
• Really good at building tinyOS subsystems
– non-blocking, split-phase event structures
• Internalized the “state of constant change”
paradigm
– ex: maintain routing tree by constantly rebuilding it
– soft state that is always suspect
– simple one-way protocols
• Operating in the aggregate
• Simple mechanisms to accomplish large goals
– MAC, ATC
• Out of the box on networking abstractions
– Low-power listen, wake-up, statistical sampling, weighted
aggregation
• Understanding of large scale dynamics
5/13/2002
TinyOS IPAM
52
Feeding experience back into simulation
5/13/2002
TinyOS IPAM
53
Rich set of additional challenges
• Efficient and robust security primitives
• Density independent wake-up, aggregation
– sensor => can use radio in ‘analog’ mode
• Resilient aggregators
• Programming support for systems of generalized
state machines
• Programming the unstructured aggregate
– SPMD, Data Parallel, Query Processing, Tuples
• Understanding how an extreme system is
behaving and what is its envelope
– adversarial simulation
• Self-configuring, self-correcting systems
5/13/2002
TinyOS IPAM
54
The “Law of Miniaturization”
Innovation
Log R
Personal Computer
Workstation
Server
Integration
Minicomputer
Mainframe
99
Time
• Each major generation is increasingly smaller, more deeply
interactive, arrives when previous is at its strength
• Vast majority of computing will be small, embedded, devices
connected to the physical world
– actually the case today, but...
– not connected to us, the web, or each other – this will change
5/13/2002
TinyOS IPAM
55
Where to go for more?
• http://www.tinyos.net/tos/
• Jason Hill, Robert Szewczyk, Alec Woo, Seth Hollar, David
Culler, Kristofer Pister. System architecture directions for
network sensors. ASPLOS 2000.
• David E. Culler, Jason Hill, Philip Buonadonna, Robert
Szewczyk, and Alec Woo. A Network-Centric Approach to
Embedded Software for Tiny Devices. EMSOFT 2001.
5/13/2002
TinyOS IPAM
56