INTEGRATION DAY
2015
Sam Vanhoutte
Azure Event Hubs, Stream Analytics
& Power BI
Nice to meet you
CTO
sam.vanhoutte@codit.eu
+32 474 849 993
@SamVanhoutte
7 year - BizTalk V-TSP
1st year - Integration MVP
be.linkedin.com/in/samvanhoutte/
Sam VANHOUTTE
2012 & 2013
2000 Belgium
2004 France
2013 Portugal
Partner of the Year
Award Finalist
Focused on integration solutions
> 60 Active integration customers
Application Integration
International Focus HQ in BE
e-news + SoMe
60 employees
> 50 consultants BizTalk certified
Special
M2M
Smart
purpose
communication
things
devices
The
The
Internet
Internet
of
ofEverything
Things
3
The IoT value chain
Machine Learning
Demo scenario
Speeding tickets
Ingest
Event Hubs
Traffic jam detection
Stream Analytics
Power BI
6
Power BI tools
• Power BI designer
(when no office license)
• PowerQuery >> Get data
• PowerPivot >> Prepare data
• PowerView >> Present data
7
Event Hubs
an Azure Service Bus service
(collection & ingestion)
Azure Service Bus
Relay
Queue
Topic
Notification Hub
Event Hub
Event Hub – IOT at Scale
Storage &
Analytics
Custom Code & 3rd
Party Services
Event Hub
Web/Mobile User
Interfaces
Event Sources
- Hyper Scale - Fully Managed - Interoperable - Secure - Cost Effective -
Integration
Services
Cloud Services
Introducing Azure Event Hubs
Event Producers
Event Hubs
an Azure Service Bus service
Producers
(sending data to Event Hubs)
Publishers
Very many publishers
Short-lived, low throughput: HTTPS
Long-lived, high throughput: AMQP
Long lived AMQP connections are billable,
HTTPS requests are not; AMQP connection
allowance included in tier
Publish to …
Event Producers
PartitionId
Direct
PartitionKey
PartitionKey selecting PartitionId
Publisher Policy (<eh>/publishers/<name>)
<name> overriding PartitionKey
15
Publishers
Very many publishers
Short-lived, low throughput: HTTPS
Long-lived, high throughput: AMQP
Long lived AMQP connections are billable,
Create Event Hubs
HTTPS requests are not; AMQP connection
allowance
included in tier
Send events to Event
Hub
Demo:
•
•
Publish to …
Event Producers
PartitionId
Direct
PartitionKey
PartitionKey selecting PartitionId
Publisher Policy (<eh>/publishers/<name>)
<name> overriding PartitionKey
16
Securing publishers
➔
SAS Policies are defined in the portal
➔
➔
Create Signatures per device
➔
➔
➔
Not linked to publishers and limited in numbers
Have a timespan
Are unique to a publisher
Revoking or blacklisting publishers
➔
Manage Rights needed for this
17
Securing publishers
➔
SAS Policies are defined in the portal
Demo:
Create Signatures
per device
• !Generating
SAS Signatures
Reminder
Sam,
increase
your instances
➔
➔
➔
➔
➔
Not linked to publishers and limited in numbers
Have a timespan
• Using Signatures by publishers
Are unique •to a Revoking
publisher publishers
Revoking or blacklisting publishers
➔
Manage Rights needed for this
18
Event Hubs
an Azure Service Bus service
Consumers
(reading data to Event Hubs)
Consumer groups
Receivers read from a consumer groups
Like
subscriptions
Checkpointing
per CG
{Default}
Create receivers
per partition
Maximum 20 CG
Consumer Grp
Consumers
Partitioned consumer model (not competing consumers!)
ID, Time, [Data]
ID, Time, [Data]
Like
subscriptions
ID, Time, [Data]
ID, Time, [Data]
ID, Time, [Data]
Each at their
own pace
EventProcessorHost
Out of the box Lease management
Nuget package
Managed
cursors by client
Uses storage for
state and CP
IEventProcessor
22
EventProcessorHost
Out of the box Lease management
Demo:
Nuget package
Uses storage for
state and CP
• Managed
Implementing EventProcessorHost
cursors by client
• Dashboarding demo
IEventProcessor
23
Securing consumers
➔
➔
Using SAS key / values or ACS with SBAZTool
Full SAS support for consumer groups will come
24
Event Hubs
an Azure Service Bus service
Pricing &
Tips + tricks
Pricing
Basic: Up to 100 connections, no extension
Standard: 1000 connections incl.
Throughput Unit Hour (Basic)
Throughput Unit Hour (Standard)
Ingress Events
Price
(US Dollars)
0.015 TU per hour
0.03 TU per hour
0.028 per 1,000,000 events
Cost Brokered Connections (1k-100k)
0.00004 connection/hour
Cost Brokered Connections (100k-500k)
0.00003 connection/hour
Cost Brokered Connections (500k+)
0.00002 connection/hour
Storage Overage >TUs*84GB
local-redundant Azure storage chargethrough
26
Tips & tricks
Multiple hubs per
namespace
Reuse senders &
factories
Use consumer
groups
27
Azure Stream Analytics
complex event processing
.
Introduction
(set up a basic ASA job)
Scenarios
Real time analytics
Millions events /
sec
Continuous
stream of data
.999%
SQL
High availability
SQL Syntax
Correlate
Fast time to
value
Easy test &
debug in portal
PowerBI !
30
End to end Architecture overview
Azure Stream Analytics
•
•
•
Temporal Semantics
Guaranteed delivery
Guaranteed up time
Event Inputs
- Event Hub
- Azure Blob
Outputs
Transform
-
Temporal joins
Filter
Aggregates
Projections
Windows
Etc.
-
SQL Azure
Azure Blobs
Event Hub
Table storage
PowerBI
Enrich
Correlate
Reference Data
- Azure Blob
Azure
Storage
Data Source
Collect
Process
Deliver
Consume
31
End to end Architecture overview
Azure Stream Analytics
•
•
•
Temporal Semantics
Guaranteed delivery
Guaranteed up time
Demo:
•
•
Event Inputs
Outputs
Transform
-
Temporal joins
Filter
Aggregates
Projections
Windows
Etc.
- SQL Azure
- Azure Blobs
- Event Hub
Set up Stream Analytics
The first job
- Event Hub
- Azure Blob
Enrich
Correlate
Reference Data
- Azure Blob
Azure
Storage
Data Source
Collect
Process
Deliver
Consume
32
Azure Stream Analytics
complex event processing
.
Query Syntax
(SQL like a pro)
Functions & supported types
Aggregate functions
Count, Min, Max, Avg, Sum
Scalar functions
Cast
Date and time: Datename, Datepart, Day, Month, Year, Datediff, Dateadd
String: Len, Concat, Charindex, Substring, Patindex
Types
Type
bigint
Description
Integers in the range -2^63 (-9,223,372,036,854,775,808) to 2^63-1 (9,223,372,036,854,775,807).
float
Floating point numbers in the range - 1.79E+308 to -2.23E-308, 0, and 2.23E-308 to 1.79E+308.
nvarchar(max)
Text values, comprised of Unicode characters. Note: A value other than max is not supported.
datetime
Defines a date that is combined with a time of day with fractional seconds that is based on a 24-hour clock and relative to UTC (time
zone offset 0).
Tumbling windows
Group events, based on timestamping
Tumbling window
Hopping window
Sliding window
Aggregate per time interval Schedule overlapping windows Windows constant re-evaluated
35
End to end Architecture overview
Azure Stream Analytics
•
•
•
Temporal Semantics
Guaranteed delivery
Guaranteed up time
Outputs
Important:
•
Event Inputs
Transform
-
Temporal joins
Filter
Aggregates
Projections
Windows
Etc.
- SQL Azure
- Azure Blobs
- Event Hub
For PowerBI you need
organizational account
- Event Hub
- Azure Blob
Enrich
Correlate
Reference Data
- Azure Blob
Azure
Storage
Data Source
Collect
Process
Deliver
Consume
36
End to end Architecture overview
Azure Stream Analytics
•
•
•
Temporal Semantics
Guaranteed delivery
Guaranteed up time
Demo:
•
•
•
Event Inputs
Outputs
Transform
-
Temporal joins
Filter
Aggregates
Projections
Windows
Etc.
- SQL Azure
- Azure Blobs
- Event Hub
Traffic management (speed tickets)
Traffic management (average speed)
Enrich
Traffic management (suspected cars)
- Event Hub
- Azure Blob
Correlate
Reference Data
- Azure Blob
Azure
Storage
Data Source
Collect
Process
Deliver
Consume
37
Tips & tricks
Use Timestamp by
Chose your
windowing logic
Consumer groups
Don’t delete blobs
while job runs
38