Windows Azure
Data Services
Scott Klueppel
Chief Cloud Evangelist
SOAlutions, Inc.
#gwab
@kloopdogg
Agenda
- Overview
- Management Portal / Tools
- Windows Azure Storage
- Tables, Blobs, Queues
- Windows Azure SQL Database
- Putting them together with
Cloud Design Pattern Examples
Windows Azure Storage
Overview
Management Portal
Storage Emulator / Tools
What is Windows Azure Storage?
- Storage as a service – Blobs, Tables, and Queues
- Lots of it (up to 200 TB) – pay as you go
- Highly Available
- Redundancy
- Highly Scalable
- Partitions
- Blazing Speed
- Performance “Targets”
Getting Started with Storage
- Create a storage account in the portal
- Connection string
Account Name
Account Key
Getting Started with Storage
- Use storage emulator for development
- Connection string
UseDevelopmentStorage=true
Windows Azure SDK
- SDK contains client libraries
- Committed to backwards compatibility
Windows Azure
SDK
2.2 Oct2013
Visual Studio
2013
Supported
Visual Studio
2012
Supported
Visual Studio
2010
Not Supported
Jul2013
Not Supported
Supported
Supported
2.0 Apr2013
Not Supported
Supported
Supported
2.1
Storage Client Library
- SCL 2.1 included with SDK 2.2
- SCL 3.0.x upgrade available with NuGet
Tools to interact with storage
Visual Studio Server Explorer
CloudXplorer/TableXplorer
AzureXplorer (VS Extension)
Demo
Management Portal
Storage
Tools
Windows Azure Storage
Tables
Blobs
Queues
Tables Concept Overview
Account
Table
racers
Entity
Name=Ricky
Status=Active
Name=Dale
Status=Active
gwabjax
results
Race=Jax 500
Date=3/29/14
Table Details
- Structured storage in the form of tables
-1MB max size
-252 properties
- Quick queries by clustered index (only index)
- 2,000 entities per second per partition
- Data-defined partition scheme
-Table name + PK
Table Details
- Tables store data more like key-value pairs
- No schema
First Name
Last Name
Date of Birth
Ricky
Bobby
1/1/1971
Dale
Earnhardt
2/2/1972
Cal
Naughton
March 3, 1973
Last Win
Daytona 500
- De-normalized structure optimized for performance
Table Entities
- Property Types
byte[]
bool
DateTime
double
Guid
Int32
Int64
String
Table Entities
- Required properties
PartitionKey, RowKey, Timestamp
- Operations on entities
Delete
Insert
Upsert (Insert + Merge/Replace)
Update (Merge or Replace)
Retrieve
Query
Using Tables
1. Get table reference
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("DefaultConnection"));
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("racers");
2. Execute operations on entities
TableOperation upsertOperation = TableOperation.InsertOrReplace(racer);
table.Execute(upsertOperation);
Using Tables with DTOs/POCOs
1. Get table reference (like before)
2. Execute operations on entities via EntityAdapter
With TableEntity – RacerEntity
var operation = TableOperation.InsertOrReplace(racerEntity);
table.Execute(operation);
With DTO – Racer
var adapter = new RacerAdapter(racer);
var operation = TableOperation.InsertOrReplace(adapter);
table.Execute(operation);
Table Best Practices
1. Avoid querying across partitions
2. Batch inserts (within partition)
- Order and details in same table/partition
3. Storing aggregate copies
- Ensures 1 query  1 partition
- Format data up front
4. Use intelligent PKs
Demo
Tables
Blobs
Queues
Blobs Concept Overview
Account
Container
Blob
DSC000256.jpg
photos
teamlogo.png
gwabjax
videos
1stPlace.avi
Blobs
- Types of Blobs
- Page
- Block
- Operations on blobs
Exists
List
Upload from Stream, ByteArray, Text, File
Download to Stream, ByteArray, Text, File
Delete
Using Blobs
1. Get blob container reference
CloudStorageAccount account = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("DefaultConnection"));
CloudBlobClient client = account.CreateCloudBlobClient();
CloudBlobContainer container = client.GetContainerReference("photos");
2. Perform operations with blobs
CloudBlockBlob blockBlob = container.GetBlockBlobReference(refId);
blockBlob.UploadFromStream(stream);
blockBlob.Metadata["FileName"] = fileName;
blockBlob.SetMetadata();
Demo
Tables
Blobs
Queues
Queues Concept Overview
Account
Queue
orders
Messages
Bulk
http://...
Small
http://...
gwabjax
emails-to-send
http://...
Queues
- Operations on queues
Add
Peek
Get
Update
Delete
- Queue naming
- All lowercase
- Alpha-numeric and hyphen “-”
- 3-63 characters
Queues in the wild
Client
Web Role
Browser
Web
App
Storage
Worker Role
Process
Task
Surveys
Queue
Get Survey
Complete
Survey
Post Results
Thank you
Add
Get
Delete
Process*
Queues in the wild
Client
Web Role
Browser
Web
App
Storage
Small
Surveys
Large
Surveys
Worker Role
Blob
Store
Big
Task
Small
Task
Get Survey
Complete
Survey
Small
Survey
Post Results
Thank you
Add
Get
Delete
Process*
Queues in the wild
Client
Web Role
Browser
Web
App
Storage
Small
Surveys
Queue
Large
Surveys
Queue
Worker Role
Blob/Table
Store
Large
Task
Small
Task
Get Survey
Complete
Survey
Large
Survey
Post Results
Thank you
Upload
Add
Get
Download
Delete
Process*
Using Queues
1. Get queue reference
CloudStorageAccount account = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("DefaultConnection"));
CloudQueueClient client = account.CreateCloudQueueClient();
CloudQueue queue = client.GetQueueReference("orders");
2. Perform operations with messages
CloudQueueMessage message = new CloudQueueMessage("ticket-order-17");
queue.AddMessage(message);
Demo
Tables
Blobs
Queues
Windows Azure
SQL Database
(formerly SQL Azure)
Management Portal
Data Access
Windows Azure SQL Database
Tools
- SSMS
- Visual Studio/SSDT
- Modeling tools
Code
- ADO.NET
- Enterprise Library
- ORMs (e.g. Entity Framework, nHibernate)
Key Features
- High Availability/Durability
- Three Replicas (1 Primary – 2 Secondary)
- 99.9% SLA
- Manageability
- Portal, PowerShell, SSMS, REST API
- Predictable Performance
- Scale-out*
What’s Different – Maybe Bad?
Missing
- Extended stored procedures
- SQL-CLR
- Service Broker
- Table partitioning
Important: All tables must have a clustered index
Data Types?
Data Type Category
Windows Azure SQL Database Support
Exact numerics
bigint, bit, decimal, int, money, numeric, smallint, smallmoney, tinyint
Approximate numerics
float, real
Date and time
date, datetime2, datetime, datetimeoffset, smalldatetime, time
Character strings
char, varchar, text
Unicode character strings nchar, nvarchar, ntext
Binary strings
binary, varbinary, image
Spatial data types
geography, geometry
Other data types
cursor, hierarchyid, sql_variant, table, timestamp, uniqueidentifier, xml
Development Story
- Local SQL 2012/Express database
- SSDT Database projects
- Deploy directly to Azure from Visual Studio
- Create deployment package
- Use Release Management
- Azure Connection String
<add name="DefaultConnection"
connectionString="Server=tcp:[server].database.windows.net,1433;
Database=[database]; User ID=[database]-host@[server]; Password=[password];
Encrypt=true;"
providerName="System.Data.SqlClient"/>
Demo
Management Portal
Create Server
Create Database
Firewall Rules
Patterns at Work
•
•
•
•
Cache-aside Pattern
Materialized View Pattern
Competing Consumers Pattern
Compensating Transaction Pattern
Cache-Aside Pattern
- Build cache on-demand
- Read cache first
- If not there
- Get from data store
- Store in cache
Materialized View Pattern
-
Improve performance in systems with difficult queries
Pre-populated views
Not updated by app, can be entirely rebuilt
Inherent delays
Doesn’t query well
Competing Consumers Pattern
- Protect against a large influx (burst) of requests
- Balanced workload
- Scalable
Compensating Transaction Pattern
- Eventual consistency
- Series of autonomous steps
- Intermediate steps appear
inconsistent
- Compensating Transaction
- Intelligent process to undo
succeeded steps
Compensating Transaction Pattern
- Example: Itinerary Creation
- Book 3 flights and 2 hotel rooms
- If one or more are unavailable, we start over
Compensating Transaction Pattern
- Example:
Itinerary Creation
What did we learn?
-
Azure data storage options/uses
Each has pros/cons/best fit
Management Portal and Tools
Design patterns
Questions?
Scott Klueppel
Chief Cloud Evangelist
SOAlutions, Inc.
#gwab
@kloopdogg
Thank you!
Scott Klueppel
Chief Cloud Evangelist
SOAlutions, Inc.
#gwab
@kloopdogg
References
• Cloud Design Patterns (P&P)
http://msdn.microsoft.com/en-us/library/dn568099.aspx
• Data Services – Storage
http://msdn.microsoft.com/en-us/library/windowsazure/gg433040.aspx
• Storage Differences – Emulator vs Cloud
http://msdn.microsoft.com/en-us/gg433135
• Data Services – SQL Database
http://msdn.microsoft.com/en-us/library/windowsazure/ee336279.aspx