Object-based Storage
Long Liu
2010-10-23
Outline
•
•
•
•
•
Why do we need object based storage?
What is object based storage?
How to take advantage of it?
What's the status of object based storage?
What can we do about it?
Outline
•
•
•
•
•
Why do we need object based storage?
What is object based storage?
How to take advantage of it?
What's the status of object based storage?
What can we do about it?
Background
• Existing enterprise storage infrastructures are
feeling the strain
• the volume of data generated by many
network-based applications continues to
escalate
Background
Two Technologies:
• Files:
– Flexible data sharing
– Secure
• Blocks:
– High performance
– Scalable
Files
Blocks
Comparison
Objects can be regarded as the convergence
of two technologies: files and blocks
Operations:
Operations:
Create object
Delete object
Read object
Write object
Read block
Write block
Addressing:
Addressing:
Block range
[object, byte range]
Block Based Disk
Object Based Disk
Comparison
Traditional Storage
Applications
Object-based Storage
Applications
File System
File System
Object Interface
Logical Block Interface
Hard Drive
Storage component
Object-based Storage Device (OSD)
Comparison
Two changes :
Object-based storage
offloads the storage
component to the
storage device
The device interface
changes from blocks to
objects
(a)
Traditional
model
CPU
Applications
System Call Interface
File System
User Component
File System
Storage Component
(b)
Object storage
model
CPU
Applications
System Call Interface
File System
User Component
Object Interface
Block Interface
Block I/O Manager
File System
Storage Component
Block I/O Manager
Storage Device
Storage Device
Motivation
• Improved device and data sharing
– Platform-dependent metadata moved to device
• Improved scalability & security
– Devices directly handle client requests
– Object security
• Improved performance
– Data types can be differentiated at the device
• Improved storage management
– Self-managed, policy-driven storage
– Storage devices become more autonomous
Objects
Outline
•
•
•
•
•
Why do we need object based storage?
What is object based storage?
How to take advantage of it?
What's the status of object based storage?
What can we do about it?
Object-based Storage
Object
OSD(Object-based Storage Device)
MDS(Metadata Server)
Object-based Storage
Object
OSD(Object-based Storage Device)
MDS(Metadata Server)
Object
•An object is a logical unit of storage
— ID (Identification)
— Application data
— Metadata which includes block allocation
and length
— Attributes that is accessible by users
•Objects have file-like methods
— open, close, read, write
Object
•The root object -- The OSD itself
•User object -- Created by SCSI commands from the application or client
•Collection object -- A group of user objects, such as all .mp3
•Partition object -- Containers that share common security and space
management characteristics
Object
P4
P3
P2
OSD
P1
Root Object
(one per device)
U1
User Data
Attributes
Partition Objects
Collection Objects
Metadata User Objects (for user data)
Object ID
Object-based Storage
Object
OSD(Object-based Storage Device)
MDS(Metadata Server)
Object Storage Devices
Intelligence with its own
CPU, Memory, Disk, Network
Redistribution with specific
strategy
Objects’ metadata
management
Object Storage Devices
Two changes :
Object-based storage
offloads the storage
component to the
storage device
The device interface
changes from blocks to
objects
(a)
Traditional
model
(b)
Object storage
model
CPU
CPU
Applications
System Call Interface
File System
User Component
File System
Storage Component
Applications
System Call Interface
File System
User Component
Object Interface
Interface
Block I/O Manager
File System
Storage Component
Block I/O Manager
Storage Device
Storage Device
Object Storage Devices
Expect wide variety of Object Storage Devices
Disk array subsystem
2 SATA disks – 240/500 GB
Highly integrated, single disk
4 Gbps per
shelf to cluster
Orchestrates system activity
Balances objects across OSDs
Stores up to 5 TBs per shelf
Object-based Storage
Object
OSD(Object-based Storage Device)
MDS(Metadata Server)
Metadata Server
Control the
interaction between
clients and OSDs
Dataflow of Metadata
Client
Applications
File
Client
System
Metadata Manager
Storage component
Metadata Servers (MDS)
Object Storage Devices (OSDs)
Object-based Storage Device (OSD)
Outline
•
•
•
•
•
Why do we need object based storage?
What is object based storage?
How to take advantage of it?
What's the status of object based storage?
What can we do about it?
Interface
T10 OSD Commands(face)
GET/SET ATTR
READ/WRITE
OSD Target
CREATE/REMOVE
OSD Client
10001110101
10000001110
11001110111
10001111000
..…
OSD Interface
Opaque
attributes
(stored only)
Shared
attributes
(stored &
processed)
User Data
Object
Attribute pages
OSD Commands(Interface)
• Basic Protocol
– READ
very basic
– WRITE
– CREATE
space mgmt
– REMOVE
– GET ATTR
attributes
– SET ATTR
• opaque
• internal
• shared
• Specialized
– APPEND
– CREATE & WRITE
– FLUSH
– LIST
• Security
– Authorization
– Integrity
– SET KEY
– SET MASTER KEY
shared secrets
• Groups
– CREATE COLLECTION
– REMOVE COLLECTION
– LIST COLLECTION
• Management
– CREATE PARTITION
– REMOVE PARTITION
– FLUSH PARTITION
– PERFORM SCSI COMMAND
– PERFORM TASK MGMT
Storage Technology Today
1
2
3
4
• Direct attached storage (DAS)
• Fabric Attached Storage (FAS)
-Network Attached Storage (NAS)
-Storage Area Networks (SAN)
Direct Attached Storage
LAN
Windows
UNIX
Windows
RAID
BACKUP SERVER
RAID
RAID
RAID
TAPE
A traditional Direct
Attached Storage
model
Fabric Attached Storage
C/S
Fabric Attached
Storage
SERVER
CLIENTS
DATA
Network Attached Storage
This figure illustrates
NAS being used to
share files among a
number
of clients.
Clients
File I/O
IP network
Storage area network
Block I/O
File server
Block storage
The files themselves
may be stored on a
fast SAN
Storage Area Networks
Clients
This figure
illustrates a SAN
file system
Data
The files
themselves are
stored on a fast
storage Area to
which the clients
are also attached.
Servers
Storage area network
Block-based storage devices
Security
CLIENTS
METADATA
SERVERS
Capability
Attribute
NETWORK
Capability
Attribute
OBJECT-BASED STORAGE DEVICE
Outline
•
•
•
•
•
Why do we need object based storage?
What is object based storage?
How to take advantage of it?
What's the status of object based storage?
What can we do about it?
Status
• Industrial
– Lustre
– Panasas
• Academic
Status
• Industrial
– Lustre
– Panasas
• Academic
Lustre
• First open sourced system with object storage
• High-performance parallel file system
• Consist of clients, MDS and OST(Object
Storage Targets)
Lustre
METADATA SERVER
CLUSTER
NETWORK
CLIENTS
Data
Management
OBJECT STORAGE TARGETS(OST)
Panasas
• High-performance file system
• Consist of OSD, Panasas File System, MDS
Panasas
Key Object Storage Features
Intelligent space management in storage layer
ßMedia geometry aware placement
ßData aware prefetching, caching & recovery
Encapsulation of data and attributes
ßNative object interface, good programming model
ßStorage interpreted attributes for per file
properties
Key Object Storage Advantages
ßRobust, shared access by many clients
ßScalable performance via an offloaded data path
ßStrong fine-grained end-to-end security
Panasas
• Clients are from Energy, Government, Finance,
Manufacturing and Higher Education
Status
• Industrial
– Lustre
– Panasas
• Academic
A Design of Metadata Server Cluster In
Large Distributed Object-based Storage
Motivation:
• Metadata server cluster maybe the bottleneck
• Frequent metadata access and movement
• Terrible load balance management
A Design of Metadata Server Cluster In
Large Distributed Object-based Storage
Application Server Cluster
VoD
Server
Web
Server
Database
Server
E-mail
Server
File
Server
Metadata
Data
Storage Network
(Fibre Channel)
MDS
Cluster
Security
Object-based Storage Device Cluster
Object-based Storage System Architecture
A Design of Metadata Server Cluster In
Large Distributed Object-based Storage
Application Servers
Hashing Partition
Application
File Hashing Manager
Mapping Manager
Metadata Server Cluster
Logical Partition Manager
Metadata Server Backend
Common Storage Space
Figure 3. Hashing Partition
Hashing Partition:
A total solution for
–File hashing
–Metadata partitioning
–Metadata storage
A Design of Metadata Server Cluster In
Large Distributed Object-based Storage
Pathname: /Dir1/Dir2/filename
1
①.Filename hashing
②.Selecting MDS through
Mapping Manager
③ .Accessing metadata by
pathname hashing result
④.Returning metadata to
application server
4
Pathname Hashing
Result (i)
Mapping Manager
Pathname
Metadata
&
etc
Pathname Hashing
Result (i+1)
2
Metadata Server Cluster
3
Pathname
Metadata
&
etc
Logical Partitions
Figure 4. Metadata Access Pattern
A Design of Metadata Server Cluster In
Large Distributed Object-based Storage
Hashing Partition
Mapping Manager
2
1
Metadata Server Cluster
3
Logical Partitions
4
Common Storage Space
Figure 5. MDS cluster failover procedure
Outline
•
•
•
•
•
Why do we need object based storage?
What is object based storage?
How to take advantage of it?
What's the status of object based storage?
What can we do about it?
From Our Perspective
The End
Thank you