Choosing an application
integration strategy
Anko Duizer
Class-A
Overview
•
•
•
•
•
•
•
•
•
Why integration?
What is integration?
Challenges
Common scenario’s
Which technologies does Microsoft offer?
Messaging
Design considerations
A different viewpoint: The Ring(s)
The Matrix
Why?
• Enterprises are typically comprised of
hundreds applications
– Custom built, acquired by a third party, legacy
systems
• Reasons
– Creating a single, big application to run a
complete business is impossible
– Flexibility to select “the best”
– Use a collaborative partner model
Those who can not remember the past are condemned to
repeat it!
The current situation
A
F
B
C
E
D
What?
• Connecting
– Computer systems
– Companies
– People
Application integration is the secure and
orchestrated sharing of processes and/ or
data between applications
Integration scenarios
•
•
•
•
•
•
Information portals
Data replication
Shared business functions
Service-oriented architectures
Distributed business processes
Business-to-business integration
Basic challenges
•
•
•
•
•
Networks are unreliable
Networks are slow
Any two applications are different
Change is inevitable
Organizational problems
Integration approaches
•
•
•
•
•
•
Simple association
Central hub
The big database approach
Message bus
Service integration
Hybrid
Simple association
Application
Application
BO
BO
Data
UI
BO
Endpoint
Data
UI
BO
Endpoint
Central hub
Application
Application
BO
BO
Data
UI
Data
UI
BO
BO
Endpoint
Endpoint
ERP System
The big database approach
Application
Application
BO
BO
UI
UI
BO
BO
Data
Message bus
Application
Application
BO
BO
Data
UI
Data
UI
BO
BO
Endpoint
Endpoint
Message bus
Service integration (1)
Application
Application
BO
BO
Data
UI
Data
UI
BO
BO
Endpoint
Endpoint
Service
Service integration (2)
Application
Endpoint
Service
Service
Service
Service
Endpoint
Endpoint
Endpoint
Application
Application
Application
Microsoft technology
•
•
•
•
•
•
BizTalk Server 2004
Web Services
MSMQ
DTS
Indigo (not available yet)
SQL Service Broker (not available yet)
BizTalk Server 2004
development
operation
management
orchestration services
orchestration
designer
action
administration
pipeline
designer
tracking
biztalk
editor
tracking
port
messaging services
monitoring
biztalk
mapper
config
Business
rules
messaging
management
applications
Web Services
proxy
listener
XML over HTTP
Application
Browser
Component
Internet
Host
Client
Page
HTTP request
.NET
COM
EJB
Perl
…
Host
proxy
MSMQ
Sender
Receiver
msg
msg
msg
msg
Queue
DTS
Source
Transforms
Destination
OLE DB
ODBC
Fixed field
ASCII delimited
ActiveX Script
Copy
Trim String
…
Custom
OLE DB
ODBC
Fixed field
ASCII delimited
Repl. publication
In
Out
DTS Data Pump
Indigo
Service
Port
Channel
Service
Message
Channel
Service
Channel
Service
SQL service broker
SQL Server object
Description
Message Type
Defines valid messages for exchange between services
Contract
Specifies type of messages and their direction in a
conversation, initiator or target
Queue
Stores messages before sending and after receipt as result set
Service Program
The part of a service broker application that reads messages
from a queue and processes them.
Service
Addressable endpoint for service communication
msg
msg
msg
msg
Queue
Message type
Contract
Service
Service
program
Messaging
•
•
•
•
Loosely coupled
Communicating asynchronously
Reliable
Makes the messaging responsible for
transferring data from one application to
another
Sync or async?
time
Process A
blocked
Process B
time
Process A
Process B
Basic messaging concepts
Endpoint
Application
Channel
Application
Endpoint
Message
Translation
Routing
Messaging patterns
Y
1)
3)
2)
• Request / Reply
• No Response required
• System makes request
and gets response
X
EAI
X
Y
• Needs ability to
cache response from
multiple requests to
respond to one
message
Y
• Fire and Forget
EAI
X
EAI
A B C
Y
A
EAI
X
EAI
X
B
C
4A/B)
XREF
• Multiple Applications
needed to satisfy request
• 4A: XREF Look-up
• 4B: XREF Update
X
EAI
5)
6A)
• Publish Subscribe
• Needs data from Z to be
• A, B and C subscribe to message
able to update Y
Z
The message is the message!
Main design considerations
• Timing/ Latency
• Identification
– Industry keys, Internal keys, Chained keys &
Matching algorithms
• Sharing entities, activities, processes and
utilities
• Designing endpoints
• Message transformation
Main design considerations
•
•
•
•
•
Routing
Security
System management
ACID Transactions
Service-oriented architectures
Shared data ownership
E-Commerce
Inventory
E-Commerce
Choosing
To EAI or to ETL?
Data (ETL)
Messages (EAI)
Data Transformation Services BizTalk Server 2004
Unit of
modeling
“Package” – an XML definition of
“workflow” and data flow
Schema that capture structure/
Contracts that capture behavior
Unit of
interaction
“Synchronous” DTS pipeline for
data movement
Asynchronous message passing
Instancing
DTS Service controls instances
Weak/opaque URI-based
references
Extensibility
Scripts, SQL, custom
components
Intermediaries and message
routing
Platform
Databases
Multiple
Evaluation criteria
Criterion
Description
Latency
How quickly is the data to be transferred?
Transformation
Complexity of the transformation
Scope
Application, organization or partner?
Volume
Quantity of exchange by a transfer
Intrusion
Effort
Degree of change to existing applications in
order to effect transfer
Effort required to build and maintain the solution
Organization
Which solution fits best with the organization?
The Ring(s)
1. Application
2. Enterprise
3. Collaborative partners
4. Everybody else
The Matrix
BizTalk
SSB
Near real
time
Near
Real time Near real
real time
time
Batch
Transformation High
High
N/A
N/A
High
Scope
Enterprise
& partners
App.
App.
App.
App. &
Enterprise
Volume
Medium
Medium
Low
Medium
High
Intrusion
Medium
Low
High
High
Low
Effort
High
Medium
High
Medium
Medium
Organization
tbd
tbd
tbd
tbd
tbd
Latency
Indigo
MSMQ
DTS
The Matrix - Reloaded
Indigo
Environment Any WS-*
Application
compliant
Any
distributed
application
MSMQ
SSB
Windows
SQL Server
2005 (both)
Asynchronous Database
application
Message
store
Type of
message
In-memory or
database
NT File
system
SQL Server
2005
Persistent &
Non-peristent
Transactional
only
Protocol
Various
Reliable,
Express,
Transaction
Various
TCP only
The Matrix - Revolutions
Ring
0
1
2
Communication
Synchronous Asynchronous
Asynchronous
Focus
Performance
Security
Integration
Security
Integration
Indigo
Yes
Maybe
No
BizTalk Server
No
Yes
Yes
WS-*
Maybe
Yes
Yes
MSMQ
No
Yes
No
SSB
Yes
No
No
3
Firewall
Conclusions
• Integration is hard!
• Integration requires good thinking!
• Different technologies available, use them
smartly!
• Start thinking asynchronously and
message based!
References
• Books
– Enterprise integration Patterns, Hohpe &
Wools
– Enterprise Integration Solutions, Spackman &
Speaker
• Web
– http://www.class-a.nl
– http://www.objectwatch.com
– http://msdn.microsoft.com/architecture
Questions
?
The Rings(s)
1. Synchronous, .NET or J2EE? ,
Performance, Indigo, SSB, Web Services
2. Asynchronous, MSMQ or MQSeries?,
DTS, Security, WS-*
3. Asynchronous, BizTalk Server, Security,
WS-*
4. Firewalls, for the rest you don’t care!