Data-Rich Computing:
Where It’s At
Phillip B. Gibbons
Intel Research Pittsburgh
Data-Intensive Computing Symposium
March 26, 2008
Some slides are borrowed from Jason Campbell, Shimin Chen, Suman Nath,
and Steve Schlosser. Remaining slides are © Phillip B. Gibbons
Data-Intensive Computing Symposium
wiki
wiki
Wikipedia
(7000PB) ttp://www.inte
World Wide Web iki(10GB)
wiki w
tp://www.intel
1GB / person
p://www.intel.
(~1PB)
wiki
wi
100%
CAGR
200PB+ captured ://www.intel.c ki
200% CAGR
//www.intel.co i wiki wik
Human Genomics http://www.int
Particle Physics
Large Hadron
Collider
(15PB)
Annual Email
Traffic, no spam
(300PB+)
Internet Archive
(1PB+)
200 of London’s
Traffic Cams
2004 Walmart
Transaction DB
UPMC Hospitals
Imaging Data
MIT Babytalk
Speech
Experiment
(8TB/day)
(500TB/yr)
(500TB)
(1.4PB)
Estimated On-line
RAM in Google
Personal Digital
Photos
(8PB)
(1000PB+)
Typical Oil
Company
Merck Bio
Research DB
(350TB+)
100% CAGR
(1.5TB/qtr)
Terashake
One Day of
Earthquake Model Instant Messaging
of LA Basin
in 2002
(1PB)
(750GB)
Total digital data to be created this year 270,000PB
3
Phillip B. Gibbons, Data-Intensive Computing Symposium
(IDC)
Data-Rich Computing
Thriving in a World Awash with Data
Everyday Sensing &
Perception (ESP)
15MB today,
100s of GB soon
Sampling of
the projects
4
Cardiac CT
4GB per 3D scan,
1000s of scans/year
@ Intel
Research
Terashake Sims
~1 PB for LA basin
Object
Recognition
 GB today
TB needed
Phillip B. Gibbons, Data-Intensive Computing Symposium
Building ground
models of
Southern
California
SCEC
ground
model
Goal: Sample
entire region at
10m resolution
6x104 x 3x104 x
1x104 = 18x1012
sample points!
Image credit: Amit Chourasia, Visualization Services, SDSC
100 km
deep
~1 PB of data
uncompressed
Steve Schlosser, Michael Ryan, Dave O’Hallaron (IRP)
5
Phillip B. Gibbons, Data-Intensive Computing Symposium
Harvard ground model
50 8core blades
8GB memory
300GB disk
Time to Build:
SCEC model – ~1 day
Harvard model – ~6 hours
6
Phillip B. Gibbons, Data-Intensive Computing Symposium
Data-Rich Computing: Where It’s At
 Important, interesting, exciting
research area
I know where
it’s at, man!
 Cluster approach:
computing is co-located where the storage is at
Focus of this talk:
 Memory hierarchy issues:
where the (intermediate) data are at, over the course of
the computation
 Pervasive multimedia sensing:
processing & querying must be pushed out of the data center
to where the sensors are at
8
Phillip B. Gibbons, Data-Intensive Computing Symposium
Memory Hierarchy (I):
CMP Architecture
Processor Chip
P
P
P
L1
L1
L1
Interconnect
 Shared H/W
Resources
– On-chip cache
– Off-chip PIN
bandwidth
(Distributed) Shared L2 Cache
Longer latency
Lower bandwidth
Main Memory
9
Phillip B. Gibbons, Data-Intensive Computing Symposium
Memory
Memory Hierarchy (II):
CMPs, Memories & Disks on a LAN
 Cluster
Memory
Memory
SSD (Flash)
SSD (Flash)
and/or
and/or
Magnetic
Disk
Magnetic
Disk
– Orders of magnitude
differences in latency
& bandwidth among
the levels
– Differing access
characteristics:
– Quirks of disk
– Quirks of flash
– Quirks of cache
coherence
Moreover, can have WAN of such Clusters
10
Phillip B. Gibbons, Data-Intensive Computing Symposium
Hierarchy-Savvy Parallel Algorithm
Design (HI-SPADE) project
Goal: Support a hierarchy-savvy model of
computation for parallel algorithm design
 Hierarchy-savvy:
– Hide what can be hid
– Expose what must be exposed
– Sweet-spot between ignorant
and fully aware
 Support:
– Develop the compilers, runtime systems,
architectural features, etc. to realize the model
– Important component: fine-grain threading
11
Phillip B. Gibbons, Data-Intensive Computing Symposium
HI-SPADE project: Initial Progress
 Effectively Sharing a Cache among Threads
[Blelloch & Gibbons, SPAA’04]
– First thread scheduling policy (PDF) with provably-good
shared cache performance for any parallel computation
– W.r.t. sequential cache performance
– Hierarchy-savvy: automatically get good shared-cache
performance from good sequential cache performance
P
P
L2 Cache
Main Memory
12
P
P
P
With
PDF
Shared L2 Cache
Main Memory
Phillip B. Gibbons, Data-Intensive Computing Symposium
Example: Parallel Merging in Merge Sort
Work Stealing (WS):
P=8
Parallel Depth First (PDF):
Cache miss
Cache hit
Mixed
Shared cache = 0.5 *(src array size + dest array size).
13
Phillip B. Gibbons, Data-Intensive Computing Symposium
HI-SPADE: Initial Progress (II)
 Scheduling Threads for Constructive Cache
Sharing on CMPs [Chen et al, SPAA’07]
– Exposes differences between theory result & practice
– Provides an automatic tool to select task granularity
LU
Merge Sort
Hash Join
Work Stealing (ws) vs. Parallel Depth First (pdf); simulated CMPs
14
Phillip B. Gibbons, Data-Intensive Computing Symposium
HI-SPADE: Initial Progress (III)
 Provably Good Multicore Cache Performance
for Divide-and-Conquer Algorithms [Blelloch et al,
SODA’08]
– First model considering both shared & private caches
– Competing demands: share vs. don’t share
– Hierarchy-savvy: Thread scheduling policy achieves
provably-good private-cache & shared-cache performance,
for divide-and-conquer algorithms
P
P
P
P
P
L1
L1
L1
L1
L1
L2 Cache
Main Memory
15
Shared L2 Cache
Main Memory
Phillip B. Gibbons, Data-Intensive Computing Symposium
HI-SPADE: Initial Progress (IV)
 Online Maintenance of Very Large Random
Samples on Flash Storage [Nath & Gibbons, submitted]
Lexar CF card
– Flash-savvy algorithm (B-File) is 3 orders of magnitude faster
& more energy-efficient than previous approaches
– Well-known that random writes are slow on flash; we show a
subclass of “semi-random” writes are fast
Springboard for a more
general study of flash-savvy
algorithms based on
semi-random writes
(in progress)
Progress thus far is only the tip of the iceberg:
Still far from our HI-SPADE goal!
16
Phillip B. Gibbons, Data-Intensive Computing Symposium
Data-Rich Computing: Where It’s At
 Important, interesting, exciting
research area
I know where
it’s at, man!
 Cluster approach:
computing is co-located where the storage is at
 Memory hierarchy issues:
where the (intermediate) data are at, over the course of
the computation
 Pervasive multimedia sensing:
processing & querying must be pushed out of the
data center to where the sensors are at
17
Phillip B. Gibbons, Data-Intensive Computing Symposium
Pervasive Multimedia Sensing
 Rich collection of (cheap) sensors
– Cameras, Microphones, RFID readers, vibration sensors, etc
 Internet-connected.
Potentially Internet-scale
– Tens to millions of sensor feeds over wide-area
– Pervasive broadband (wired & wireless)
 Goal: Unified system for accessing, filtering,
processing, querying, & reacting to sensed data
– Programmed to provide useful sensing services
18
Phillip B. Gibbons, Data-Intensive Computing Symposium
Example Multimedia Sensing Services
Consumer services:
 Parking Space Finder
 Lost & Found / Lost pet
 Watch-my-child / Watch-myparent
 Congestion avoidance
19
Phillip B. Gibbons, Data-Intensive Computing Symposium
Example Multimedia Sensing Services
Health, Security, Commerce, and Science services:
• Epidemic Early Warning System
• Homeland Security
• Low Atmosphere Climate Monitoring
• Asset/Supply Chain Tracking
• Internet-scale Sensor Observatories
Our prototype
20
Phillip B. Gibbons, Data-Intensive Computing Symposium
Data & Query Scaling Challenges
 Data scaling
– Millions of sensors
– Globally-dispersed
– High volume feeds
– Historical data
NetRad: 100Mb/s
 Query scaling
– May want sophisticated data processing on all
sensor feeds
– May aggregate over large quantities of data,
use historical data, run continuously
– Want latest data, NOW
21
Phillip B. Gibbons, Data-Intensive Computing Symposium
IrisNet: Internet-scale Resourceintensive Sensor Network services
 General-purpose architecture for wide-area
sensor systems
– A worldwide sensor web
 Key Goal: Ease of service authorship
– Provides important functionality for all services
 Intel Research Pittsburgh + many CMU
collaborators
– First prototype in late 2002
– In ACM Multimedia, BaseNets, CVPR, DCOSS, Distributed
Computing, DSC, FAST, NSDI(2), Pervasive Computing,
PODC, SenSys, Sigmod(2), ToSN
22
Phillip B. Gibbons, Data-Intensive Computing Symposium
Data & Query Scaling in IrisNet
 Store sensor feeds locally
– Too much data to collect centrally
 Push data processing & filtering to sensor nodes
– Reduce the raw data to derived info, in parallel near source
 Push (distributed) queries to sensor nodes
– Data sampled » Data queried
– Tied to particular place: Queries often local
 Exploit logical hierarchy of sensor data
– Compute answers in-network
Processing & querying must be pushed out of
the data center to where the sensors are at
23
Phillip B. Gibbons, Data-Intensive Computing Symposium
IrisNet’s Two-Tier Architecture
Two components:
SAs: sensor feed processing
OAs: distributed database
Query
User
Web Server
for the url
...
OA
XML database
OA
XML database
SA
SA
senselet
senselet
senselet
senselet
Sensornet
24
...
Sensor
Sensor
OA
XML database
...
SA
senselet
senselet
Sensor
Phillip B. Gibbons, Data-Intensive Computing Symposium
Creating a New IrisNet Service
FFFFEFF
Only 500 lines of new
code
for Parking Space Finder
Send to OA
Updates DB
Image processing steps
vs. 30K lines of IrisNet code
Extended code
Senselet
Hierarchy (application-specific
(program to
aggregation)
filter sensor data) (XML schema)
SA
25
OAs
Front-end
Query with standard
DB language
SA
Research focus: Fault Tolerance
Phillip B. Gibbons, Data-Intensive Computing Symposium
Data-Rich Computing: Where It’s At
I know where
it’s at, man!
 Important, interesting, exciting
research area
 Cluster approach:
computing is co-located where the storage is at
 Memory hierarchy issues:
[HI-SPADE]
where the (intermediate) data are at, over the course of
the computation
 Pervasive multimedia sensing:
[IrisNet]
processing & querying must be pushed out of the data center
to where the sensors are at
26
Phillip B. Gibbons, Data-Intensive Computing Symposium
Backup Slides
27
Phillip B. Gibbons, Data-Intensive Computing Symposium
Techniques for Privacy Protection
 Cameras raise huge privacy concerns
– Use to it in London. Chicago protest
 Viewed by law enforcement vs. viewed by public
• IrisNet Goal: Exploit processing at the sensor node
to implement privacy policies
• Privileged senselet that
detects & masks faces
• All other senselets only
see masked version
Only tip of the iceberg
28
Phillip B. Gibbons, Data-Intensive Computing Symposium
Data Organized as Logical Hierarchy
<State id=“Pennysylvinia”>
<County id=“Allegheny”>
Example XML Hierarchy
<City id=“Pittsburgh”>
<Neighborhood id=“Oakland”>
<total-spaces>200</total-spaces>
<Block id=“1”>
<GPS>…</GPS>
<pSpace id=“1”>
<in-use>no</in-use>
<metered>yes</metered>
</pSpace>
<pSpace id=“2”>
…
</pSpace>
IrisNet automatically
</Block>
partitions the hierarchy
</Neighborhood>
among the OAs
<Neighborhood id=“Shadyside”>
…
…
29
Phillip B. Gibbons, Data-Intensive Computing Symposium
…
…
In-Network Query Processing:
Query Evaluate Gather (QEG)
/NE/PA/Allegheny/Pittsburgh/(Oakland | Shadyside) / rest of query
Q
Pittsburgh OA
1. Queries its XML DB
IrisNet’s
approach
2. Evaluate the result
Discovers Shadyside data
is cached, but not Oakland
Does DNS lookup to find
IP addr for Oakland
3. Gathers the missing data
by sending Q’ to Oakland OA
Q’
Combines results & returns
Oakland OA
QEG
Q’: /NE/PA/Allegheny/Pittsburgh/Oakland/ rest of query
30
Phillip B. Gibbons, Data-Intensive Computing Symposium