Big Data and SDN
Shinji Shimojo
NICT/Osaka U
2012.12
SDN
an application of SDN is overlay
PIAX overlay network platform
How does it help Big Data?
New Generation
Network Technology
• Network Virtualization
• Programability on Network
• ID/Locator Separation
• Contents Oriented Network
• In Network Processing
• Optical Networking
SDN=
Wireless
Networking
•
Software Defined Network
Software Defined
Network
Control
Plane
Software Defined
Network
Control
Plane
Software Defined
Network
C
control
Control
control
Control
Plane
• Vnode/Corelab
• Openflow
• NSI
path aggrigation for
maintenance
Control
Plane
OpenFlow Switching
OpenFlow Switch specification
OpenFlow Switch
sw Secure
Channel
hw
w
o
l
F
n
Ope ocol
t
Pro
SSL
Controller
PC
Flow
Table
http://www.openflowswitch.org/wp/documents/
The Stanford Clean Slate Program
cleanslate.stanford.edu
http://
Flow Table Entry
“Type 0” OpenFlow Switch
Rule
Action
Stats
Packet + byte counters
1. Forward packet to port(s)
2. Encapsulate and forward to controller
3. Drop packet
4. Send to normal processing pipeline
Switch MAC
Port
src
+ mask
MAC
dst
The Stanford Clean Slate Program
cleanslate.stanford.edu
Eth
type
VLAN
ID
IP
Src
http://
IP
Dst
IP
Prot
TCP
sport
TCP
dport
Example Experiment at the flow level
Mobility
The Stanford Clean Slate Program
cleanslate.stanford.edu
http://
Data Center network
Interop Tokyo 2012 Demonstra4on
Malicious Traffic
comes in
Sampling
Detector
(1)Alert
to
(4)DROP Controller
(2)COPY
RISE OF
Controller
DPI
(3)Detection by
Deep
Packet
Inspection
Demonstra*on Scenario:
•Malicious traffic comes in, and sampling detector finds suspicious traffic flow
•Detector sends alert to RISE OpenFlow Controller
•RISE Controller copies suspicious flow to DPI machine
•RISE Controller drops the flow
9
Evolution of router architecture
Vyatta, midokura, Junoshere
Control
Control
Control
Plane
Plane
Plane
Cisco
Control
Plane
nicira, cisco
Control
Control
Control
Plane
Plane
Plane
Flow
Control
Control
Control
Plane
Plane
Plane
Juniper
IP tunnel (STT)
Vnode
Openflow
Control
Plane
Ethernet
Control
Control
Control
Plane
Plane
Plane
Flow
Control
Control
Control
Plane
Plane
Plane
IP tunnel
Redirector
•
•
•
•
SDN makes network
design easy
SDN gives you
•
•
•
programability of network design
central control of all network device
network virtualization
Network Operation and configuration becomes centralized.
Effective sharing of resources brings reduction of CAPEX&OPEX.
Advanced network control such as
•
•
•
path aggregation for maintenance
selective use of network appliance
dynamic path addition for scale out
JGN-­‐X Network infrastructure Overview
New GeneraJon Network Layer
（Partnership Service R&D Environment)
DCN
DCN
DCN Plane
DCN
L2(VLAN)・L3(IP)NetworkLayer
OF
DCN
OF
Openflow PlaneOF
RISE
VN
OF
OF
Physical L1(OpJcal Testbed) Network Layer
VN
Virtual Node Plane
OF
Sapporo
VN
VN
VLAN Testbed Network
Example
StarBED3
1G
40G
10G
1G
DF
Sendai
OF
OF
OF
InternaJonal Circuit
Hiroshima
10G
Korea
OF
Fukuoka
DCN
10Gx2
10G
VN
Kanazawa
VN
Okayama
10Gx2
VN
VN
10G
DCN
Osaka
10G
40G
NICT 10G
Koganei
VN
10G
OF
OpJcal Testbed
DCN
40Gx2
40G
Tokyo
■Koganei-­‐Otemachi-­‐Hakusan
■Oteamchi〜Otemachi
OF
InternaJonal Circuit
VN
Nagoya
OF
OF
USA
Wireless Testbed
TH
SG
OF
HK
What is the StarBED (*BED) ?
• It's
an Internet/Ubiquitous
System Emulator/Simulator
• Re-configurable cluster supporting
various user requirements
• Large scale
• over 1000 PCs
16
Shinsuke Miwa
Our challenge
• Is SDN possible in wide area
• Technology
• Management
• Business model
RISE: OpenFlow/SDN Testbed
• A large-­‐scale OpenFlow/SDN testbed on JGN-­‐X
–for researchers, students, engineers, operators, …
• Challenges
–Ini4ally, demonstra4on of wide-­‐area deployment of OpenFlow technology (since 2009)
• Design of under-­‐lay network architecture (tunneling)
• Know-­‐hows in shoo4ng OpenFlow network troubles
–Now, development of OpenFlow testbed with …
• User isola4on that allows user-­‐defined controllers
• Interconnec4on of various testbed facili4es such as network emulator, wireless/op4cal network testbed, cloud infrastructure, …
2012/7/17
Eiji Kawai
18
RISE infrastructure (v1.0)
JGN-­‐X links
Q in Q trunk links
OpenFlow-­‐enabled vlan links
mgmt and sec-­‐chan seg. links
E-OFS
ksendai2
D-OFS
kkanazawa2
ksendai1
kote2
nfukuoka2
ksendai-­‐gs4k-­‐1
kkanazawa1
kote1
nfukuoka1
kote-­‐ex4200-­‐1
kkanazawa-­‐gs4k-­‐1
nfukuoka-­‐gs4k-­‐1
kote-­‐gs4k-­‐1
nokayama-­‐gs4k-­‐1
note-­‐gs4k-­‐1
ndojima-­‐gs4k-­‐1
nokayama1
nokayama2
ndojima1
ndojima2
nnagoya-­‐gs4k-­‐1
nnagoya1
nnagoya2
note1
note2
Smart X = Cyber Physical System with Big Data
•
IoT (Internet of Things)
•
•
•
•
billions of data sources, large scale data
mobility, charn
feature as a group or a set
a new form of security and privacy
Cloud
Wearable device
In house sensor
medical sensor
Mobile sensor
heterogeneous sensor network
Smart X = Cyber Physical System＋SDN
e-health
New Generation
ITS
Smart grid
Digital
Cynage
Big Data
Cloud
Multilayer Overlay Network by SDN
Customizable Overlay Network
Vital Sensor
Interior Sensors
Wearable Device
Mobile Sensors
Heterogeneous Sensor Network
Ongoing research project:CPS in MEXT
Platform for the real-‐‑‒world analysis
The aim of the project is to provide a pla^orm for real-­‐world analysis using various data sources such as sensors, contents of the social network services, etc.
Smart
Logis)cs
Marke)ng
Observa)ons
Environmental
Monitoring
New-­‐genera4on ICT Services
Crime
Preven)on
Health cares
Cloud
Disaster
Managements
Multilayer Network by SDN
Economic
Accident
CrimeOverlay Social
Status
Phenomenon Status
Status
Customizable Overlay Network
Traffic
Status
Natural
Disasters
Analyze status /
phenomenon in the realworld
Combine various kind of
information sources
Analyze Contents
Analyze Sensor Data
Collect Contents
Internet
Collect Sensor Data
22
PIAX:Overlay Agent Platform
・Fusion of overlay network and agent platform
・Middleware for highly distributed processing
・Open source http://www.piax.org/
Navigation
Env. monitoring
Smart City
Strea,omg
Home security
Flexibility and efficiency comes from agent mobility
Scalability comes from P2P mechanisms
Sensor
CGM
Device
Users
PIAX Device
PIAX Large scale data
management infrastructure
100,billion data processed in 1 million nodes
Distributed nods
finding nodes in structured P2P
network based on POI
Retrieving data from
the nodes
finding nodes in structured
P2P
POI by area
Weather sensor
User motion data
Distributed Data fusion
・Efficient data retrieval and analysis from overlay network
・Control of data granurality based on the interest
Macroscopic view of sensor data
Compare data with neighbors Delaunay overlay
Reducing no. of peer responding
using comparison
missing data
Actual
Randum sampling
(response peer：1000/10000)
Our Method
（Response peer：257/10000）
PIAX Testbed on JGN2plus
application of use
node/slice allocation
program distribution
peer deployment
gather execution results
Users
sapporo1
sapporo2
kyushu1
kyushu2
26
hiroshima1
hokuriku1
hiroshima2
kurashiki1 hokuriku2
kurashiki2
osaka1
osaka2
nagoya1
nagoya2
tohoku1
tohoku2
hakusan1
hakusan2
koganei1
koganei2
otemachi1
otemachi2
tokyo1
Layers of the PIAX testbed
- mobile agents
Programmable by
experimenters
- peers
- custom / plugin overlay network
-peers
-base overlay network / transport
(Id/Locator resolver)
- virtual nodes
- virtual network
- physical nodes
- physical network
12/03/12
Configured by testbed
operators
Virtual Cluster A
VM
VM
VM
VM
VM
VM
VM
VM
Virtual Cluster B
VM
VM
VM
VM
VM
GRE
AIST
Open
vSwitch
VM
VM
VM
VM
VM
VM
Osaka U
Open
vSwitch
Open
vSwitch
GRE
GRE
GRE
Open
vSwitch
Open
vSwitch
UCSD
VM
GRE
VM
Open
vSwitch
Open
vSwitch
Openflow Controller
Trema (Sliceable rou-ng switch)
GRE
Openflow network
Open
vSwitch
Open
vSwitch
Courtesy by Ichikawa
VM
VM
VM
Collaborating with TWAREN
Chicago
RISE
LA
JGN-X
Thailand
JGN-X
Singapore
JGN-X
KUAS
NCU
NCTU
NTUST
NCKU
NCHC
CHT-TL
Planning to connect each
SDN/OpenFlow testbed
through US academic
networks.
(As of Nov.2012 : Connected
via StarLight / PacificWave
with L2 path)
Global Testbed is the field
Thank you
Presented version is here!
https://sites.google.com/site/sshimojo/talks