Windows Azure Storage
Brad Calder
General Manager
Microsoft Corporation
1
2
BING REALTIME FACEBOOK/TWITTER SEARCH INGESTION ENGINE
Bing Ingestion Engine (Azure Service)
VM
VM
VM
VM
Windows Azure Blobs
Windows Azure Tables
3
Blobs, Drives, Tables and Queues
4
•
•
•
•
Blobs
Tables
Queues
Drives
5
6
Container
Blobs
https://<account>.blob.core.windows.net/<container>
Account
Table
Entities
https://<account>.table.core.windows.net/<table>
Queue
Messages
https://<account>.queue.core.windows.net/<queue>
7
http://<account>.blob.core.windows.net/<container>/<blobname>
Account
Container
images
Blob
PIC01.JPG
cohowinery
PIC02.JPG
videos
VID01.WMV
8
Cache-Control, Content-Encoding, Content-Type, etc
9
10
Signed URL
Signing Service
Read/Write
Read-only
Shared Access Signatures
(SAS) share with selected
few clients
Public containers
read-only access to anyone
24 Content Delivery
Network (CDN)
locations
Deliver content
efficiently around the
world
11
StorageCredentialsAccountAndKey credentials =
new StorageCredentialsAccountAndKey(accountName, key);
string baseUri = string.Format("http://{0}.blob.core.windows.net", accountName);
CloudBlobClient blobClient = new CloudBlobClient(baseUri, credentials);
// Create Container
CloudBlobContainer cloudContainer = blobClient.GetContainerReference(containerName);
bool hasCreated = cloudContainer.CreateIfNotExist();
// Access Blob in the Container
CloudBlob cloudBlob = cloudContainer.GetBlobReference(blobName);
//BlobRequestOptions has retry policy, timeout etc.
BlobRequestOptions options = new BlobRequestOptions();
//Upload the local file to Blob service
cloudBlob.UploadFile(uploadFileName, options);
//Download to local file name
cloudBlob.DownloadToFile(downloadFileName, options);
12
13
CloudStorageAccount account =
CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount");
//Initialize the local cache for drives mounted by this role instance
CloudDrive.InitializeCache(localCacheDir, cacheSizeInMB);
//Create a cloud drive (PageBlob)
CloudDrive drive = account.CreateCloudDrive(pageBlobUri);
drive.Create(1000 /* sizeInMB */);
//Mount the network attached drive on the local file system
string pathOnLocalFS = drive.Mount(cacheSizeInMB, DriveMountOptions.None);
//Use NTFS APIs to Read/Write files to drive
…
//Snapshot drive while mounted to create backups
Uri snapshotUri = drive.Snapshot();
//Unmount the drive
drive.Unmount();
14
15
http://<account>.table.core.windows.net/<table>(PartitionKey=“...”,RowKey=“...”)
Account
Tables
Entities
customer
Name=Joe
Address =…
cohowinery
order
Name=Tracy
Address =…
Name=Joe
Total=300.00
16
17
18
PartitionKey
PartitionKey
(Category)
(Category)
RowKey
RowKey
(Title)
(Title)
Timestamp
Timestamp
ReleaseDate
ReleaseDate
Action
Fast
Fast &
& Furious
Furious
…
…
2009
2009
Action
…
…
The Bourne Ultimatum
…
2007
…
…
…
…
…
…
Animation
Animation
Animation
Animation
Open Season 2
Open Season 2
The Ant Bully
The Ant Bully
…
…
…
…
2009
2009
2006
2006
PartitionKey
Comedy
(Category)
Office Space
RowKey
(Title)
…
Office Space
…Timestamp
1999
ReleaseDate
…
…
…
1999
X-Men Origins: Wolverine
…
……
2009
…
SciFi
…
X-Men
Origins: Wolverine
…
……
2009
…
…
War
War
…
Defiance
……
…
2008
Defiance
…
2008
…
Comedy
SciFi
…
19
20
[DataServiceKey("PartitionKey", "RowKey")]
public class Movie
{
/// Movie Category is the partition key
public string PartitionKey { get; set; }
/// Movie Title is the row key
public string RowKey { get; set; }
public DateTime Timestamp { get; set; }
public int ReleaseYear { get; set; }
public double Rating { get; set; }
public string Language { get; set; }
public bool Favorite { get; set; }
}
21
string baseUri = string.Format("http://{0}.table.core.windows.net", accountName);
CloudTableClient tableClient = new CloudTableClient(baseUri, credentials);
// Create Movie Table
string tableName = “Movies“;
tableClient.CreateTableIfNotExist(tableName);
TableServiceContext context = tableClient.GetDataServiceContext();
// Add movie
context.AddObject(tableName, new Movie("Action", “White Water Rapids Survival"));
context.SaveChangesWithRetries();
// Query movie
var q = (from movie in context.CreateQuery<Movie>(tableName)
where movie.PartitionKey == "Action" && movie.Rating > 4.0
select movie).AsTableServiceQuery<Movie>();
foreach (Movie movieToUpdate in q)
{
movieToUpdate.Favorite = true;
context.UpdateObject(movieToUpdate);
}
context.SaveChangesWithRetries( SaveChangesOptions.Batch );
22
http://<account>.queue.core.windows.net/<queuename>
Account
Queues
thumbnailjobs
Messages
data;http://…
cohowinery
data;http://…
orderprocess
Table=
PartitionKey=x…
23
Input Queue (Work Items)
Azure Queue
24
Input Queue (Work Items)
Azure Queue
25
26
UPDATE MESSAGE EXAMPLE
Periodically store progress information in message content
Extend visibility timeout with another 5 minutes
Get Message with 5 minutes visibility timeout
7:04
7:00
AM
7:09
7:07AM
Expires
Expires @
@ 7:05AM
7:09AM
Work items
Azure Queue
7:09
7:05
7:14
Retrieve progress from queue
message and resume
27
string baseUri = string.Format("http://{0}.queue.core.windows.net", accountName);
CloudQueueClient queueClient = new CloudQueueClient(baseUri, credentials);
//Create Queue
CloudQueue queue = queueClient.GetQueueReference(queueName);
queue.CreateIfNotExist();
//Add Message
CloudQueueMessage message = new CloudQueueMessage(“some content");
queue.AddMessage(message);
//Get Message
message = queue.GetMessage(TimeSpan.FromMinutes(5) /*Invisibility timeout*/);
// Process Message within the Invisibility Timeout
//Delete Message
queue.DeleteMessage(message);
28
MessageCount
29
30
Windows Azure Storage Internals and
Geo-Replication
31
Design Goals
•
“Windows Azure Storage: A Highly Available Cloud Storage Service with Strong
Consistency”
32
Access blob storage via the URL: http://<account>.blob.core.windows.net/
Storage
Location
Service
Data access
LB
LB
Front-Ends
Front-Ends
Partition Layer
Partition Layer
Stream Layer
Intra-stamp replication
Storage Stamp
Inter-stamp (Geo) replication
Stream Layer
Intra-stamp replication
Storage Stamp
33
North
Central US
North
Europe
Geo-replication
East Asia
South East
Asia
Geo-replication
Geo-replication
South
Central US
Europe
West
34
Microsoft Windows Azure Support
35
Data
FE
Committed State
Geo State
Primary Storage Stamp
Data
Partition
Server
Transaction Logs
FE
Committed State
Geo State
Geo-replication Log
Partition
Server
Secondary Storage Stamp
36
http://account.blob.core.windows.net/
Hostname
IP Address
account.blob.core.windows.net
North Central
Central US
US
South
Azure
DNS
Update DNS
DNS lookup
Data access
North Central US
Failover
South Central US
Geo-replication
37
Access blob storage via the URL: http://<account>.blob.core.windows.net/
Storage
Location
Service
Data access
LB
LB
Front-Ends
Front-Ends
Partition Layer
Partition Layer
Inter-stamp (Geo) replication
Stream Layer
Stream Layer
Intra-stamp replication
Intra-stamp replication
Storage Stamp
Storage Stamp
38
•
•
•
Append-only distributed file system
All data from the Partition Layer is stored into files (extents) in the Stream layer
An extent is replicated 3 times across different fault and upgrade domains
•
•
Checksum all stored data
•
•
•
With random selection for where to place replicas for fast MTTR
Verified on every client read
Scrubbed every few days
Re-replicate on disk/node/rack failure or checksum mismatch
Stream
Layer
(Distributed
File System)
M
M
Paxos
M
Extent Nodes (EN)
39
•
•
•
•
Provide transaction semantics and strong consistency for Blobs, Tables and Queues
Stores and reads the objects to/from extents in the Stream layer
Provides inter-stamp (geo) replication by shipping logs to other stamps
Scalable object index via partitioning
Partition
Master
Lock
Service
Partition Layer
Partition
Server
Partition
Server
Partition
Server
Partition
Server
M
Stream
Layer
M
Paxos
M
Extent Nodes (EN)
40
•
•
•
Front End
Layer
FE
FE
Stateless Servers
Authentication + authorization
Request routing
FE
FE
FE
Partition
Master
Lock
Service
Partition Layer
Partition
Server
Partition
Server
Partition
Server
Partition
Server
M
Stream
Layer
M
Paxos
M
Extent Nodes (EN)
41
Incoming Write Request
Ack
Front End
Layer
FE
FE
FE
FE
FE
Partition
Master
Lock
Service
Partition Layer
Partition
Server
Partition
Server
Partition
Server
Partition
Server
M
Stream
Layer
M
Paxos
M
Extent Nodes (EN)
42
Scalability Targets
43
1. Scalability targets of a single storage account
2. Scalability targets for Blobs, Table Entities and Queues within a storage account
44
•
•
Scalability targets of a single storage account
Account Scalability Targets
•
•
•
•
Capacity – Up to 100 TBs
Transactions – Up to 5000 entities per second
Bandwidth – Up to 3 gigabits per second
Partition data across storage accounts to go beyond these targets
45
•
Scalability targets for Blobs, Table Entities and Queues within a storage account
•
•
•
Single Blob – up to 60MBytes per second
Single PartitionKey in a Table – up to 500 entities per second
Single Queue - up to 500 messages per second
46
Windows Azure Storage Summary
• “Windows Azure Storage: A Highly Available Cloud Storage Service with
Strong Consistency”, ACM Symposium on Operating System Principals
(SOSP), Oct. 2011
http://blogs.msdn.com/windowsazurestorage/
47
http://www.microsoft.com/windowsazure/free-trial/
Windows Azure
Networking
48
Build a Hello World Application
http://www.microsoft.com/windowsazure/free-trial/
http://www.microsoft.com/windowsazure/sdk/
<?xml version="1.0" encoding="utf-8"?>
<ServiceConfiguration serviceName="MvcApplication1.Azure.ccproj"
xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="1" osVersion="*">
<Role name="MvcApplication1">
<Instances count="2" />
<ConfigurationSettings>
<Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString"
value="DefaultEndpointsProtocol=https;AccountName=MyAccount;AccountKey=key" />
</ConfigurationSettings>
</Role>
</ServiceConfiguration>
<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="MvcApplication1.Azure.ccproj" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" upgradeDomainCount="2">
<WebRole name="MvcApplication1" vmsize="Small">
<Sites>
<Site name="Web">
<Bindings>
<Binding name="Endpoint1" endpointName="Endpoint1" />
</Bindings>
</Site>
</Sites>
<Endpoints>
<InputEndpoint name="Endpoint1" protocol="http" port="80" />
</Endpoints>
<Imports>
<Import moduleName="Diagnostics" />
</Imports>
</WebRole>
</ServiceDefinition>
http://windows.azure.com
http://windows.azure.com