Financial Services Technology Expo
Microsoft StreamInsight
for Financial Services
A Microsoft Point of View
Presentation
Hilton New York Hotel
New York, NY
June 22-24, 2010
Microsoft StreamInsight
for Financial Services
Torsten Grabs
Lead Program Manager
Microsoft StreamInsight
email: torsteng@microsoft.com
2
Operational Intelligence Platform
Sources
Data Bus Caching
Processing
Distribution
Visualization
Refresh (Push)
Devices, Sensors
Reference
Data
Microsoft
StreamInsight
Automated
Decisions
Message Bus
Cache
Operational
Analytics
Operational Dashboard
(Ticking - Snapshot)
Refresh
(Push)
Web servers
Reporting Dashboard
(Refreshed)
In-memory
Database
ETL
Re-compute
(Pull)
Static Reports
Intra-Day
Cubes
Stock tickers &
News feeds
Service Broker
ETL
Historic
Cubes
Mining, Validation,
“What-If” Scenarios
Operational Analytics
Queries designed over offline/historical data
“Backtesting” of models
Ability to work over vast amounts of offline data
Queries deployed to online/live feeds
Tens of thousands of events/messages per sec
Continuous processing – standing queries
Timely results – low latency
Queries reason about time
Windows over time
Sequences of events
The Need for an Event-Driven Platform
Query results need to reflect important changes in
business reality immediately and enable responses to
them with minimal latency
Database Applications Event-driven Applications
Query
Paradigm
Ad-hoc queries or
requests
Continuous standing
queries
Latency
Seconds, hours, days
Milliseconds or less
Data Rate
Hundreds of events/sec
Tens of thousands of
events/sec or more
Query
Semantics
Declarative relational
analytics
Declarative relational and
temporal analytics
request
response
Event
input
stream
output
stream
5
Overview: Microsoft StreamInsight
.NET
C#
LINQ
Application
Development
Event sources
Devices, Sensors
Event
StreamInsight Engine
Pagers & Monitoring devices
Standing Queries
Event
Event
Event
Event
Event
Event
Event
Output Adapters
Input Adapters
Web servers
Event targets
Application at Runtime
`
KPI Dashboards,
SharePoint UI
Event
Trading stations
Event stores & Databases
C_ID
C_NAME
C_ZIP
Stock tickers & News feeds
Event stores & Databases
Static reference data
6
Event Types
StreamInsight events in use the .NET type
system
Events are structured and can have multiple fields
Fields are typed using the .NET framework types
StreamInsight engine provisioned timestamp
fields capture all the different temporal event
characteristics
Event sources populate time stamp fields
All calculations done based on “business time”
Timestamps Long
/Metadata
pumpID
…
…
String
Type
…
String
Location
…
Double
flow
…
Double
pressure
…
Event Streams & Adapters
A stream is a sequence of events
Defined over a .NET type
Possibly infinite
Stream characteristics:
Event/data arrival patterns (steady, bursty)
Out of order events: Order of arrival of events does
not match the order of their application timestamps
Adapters
Receive/get events from the data source
Enqueue events for processing in the engine
Insertions of new events
Changes to event durations
8
StreamInsight Query Features
Operators over streams
Calculations (PROJECT)
Correlation of streams from different data sources
(JOIN)
Check for absence of activity with a data source
(EXISTS)
Selection of events from streams (FILTER)
Stream partitioning (GROUP & APPLY)
Aggregation (SUM, COUNT, …)
Ranking and heavy hitters (TOP-K)
Temporal operations: hopping window, sliding window
Extensibility – to add new domain-specific
operators
LINQ Query Examples
LINQ Example – JOIN, PROJECT, FILTER:
from e1 in MyStream1
join e2 in MyStream2
on e1.ID equals e2.ID
where e1.f2 == “foo”
select new { e1.f1, e2.f4 };
Join
Filter
Project
LINQ Example – GROUP&APPLY, WINDOW:
from e3 in MyStream3
group e3 by e3.i into SubStream
from win in SubStream.HoppingWindow(
FiveMinutes,ThreeSeconds)
select new { i = SubStream.Key,
a = win.Avg(e => e.f) };
Grouping
Window
Project &
Aggregate
StreamInsight Licensing Guidance
StreamInsight is part of Microsoft SQL Server 2008 R2
• Standard & Enterprise edition for embedding in many ISV applications
• Standard & Enterprise edition for deployments on the edge
• Data Center edition for server deployments
Data Sources
Datacenter
•
•
Data rate: Tens of thousands of events every
second
Latency: Sub-second
Sensors
Web
servers
Stream
Insight
Devices
Stream
Insight
Aggregation &
Correlation
Stream
Insight
Stream
Insight
Feeds
Stream
Insight
Stream
Insight
Standard and Enterprise
•
•
Data rate: Thousands of events per
second
Latency: seconds
Actual mileage will vary
Stream
Insight
Stream
Insight
Stream
Insight
Complex Analytics
& Mining
Recap: Microsoft StreamInsight
Development experience with .NET,
C#, LINQ and Visual Studio 2008 or
2010
CEP Application
Development
Event sources
CEP platform from Microsoft to build eventdriven applications
StreamInsight
Engine are
Event-driven
applications
Standing Queries
fundamentally
different from
Event
Event
traditional database
Event
Event
applications:
queries
are
Event
continuous, consume and Event
produce streams,
and compute
Event
Event
results incrementally
Event
C_ID
C_NAME
C_ZIP
Static reference data
`
Output Adapters
Input Adapters
Flexible adapter SDK with
high performance to
connect to different event
sources and sinks
Event targets
The CEP platform does the
heavy lifting for you to deal
with temporal characteristics
of event stream data
12
For More Information
StreamInsight main page & download :
http://www.microsoft.com/sqlserver/2008/en/us/R2complex-event.aspx
StreamInsight blog:
http://blogs.msdn.com/streaminsight/
StreamInsight MSDN documentation:
http://msdn.microsoft.com/enus/library/ee362541(SQL.105).aspx
StreamInsight E-clinics on Microsoft e-learning
https://www.microsoftelearning.com/eLearning
More Microsoft Sessions
Tuesday June 22nd, 2:30 PM Madison Room: Cloud
Computing In Financial Services
Wednesday June 23rd, 8:00 AM Green Room – 4th Floor
Breakfast: Microsoft CRM for Financial Services
Wednesday June 23rd, 2:15 PM Petit Trianon Room – Real
Time Risk Management with Microsoft’s StreamInsight
Meet with our Cloud Computing, StreamInsight and HPC
Product teams at the Green Room, 4th Floor
Follow us on Twitter @FinServMSFT
Visit us at www.microsoft.com/financialservices
14
More Microsoft Sessions
Let us know about any topic you would like us to
follow up on!
Write the topic on the back of your business card
Drop it off in the bowl on your way out of the session
 You will be automatically entered to win a 16GB Zune HD!*
*Winner need not be present to win. Winners notified at the end
of the SIFMA conference.