Connected Systems Consulting Ltd
Case Study
Integrating BizTalk & Lagan CRM
Contents
Introduction.............................................................................................................................................. 2
About Connected Systems Consulting ................................................................................................... 3
About the Authors ................................................................................................................................... 4
Michael Stephenson ............................................................................................................................ 4
Contributors ......................................................................................................................................... 4
Sedleigh Adams ............................................................................................................................... 4
The Project .............................................................................................................................................. 5
Batch Integration with Lagan ............................................................................................................... 5
Lagan Requests Data from other Applications .................................................................................... 5
Lagan sends updates to other applications ......................................................................................... 6
Project Use Cases ............................................................................................................................... 6
Project Architecture ............................................................................................................................. 6
Interesting Architecture Topics................................................................................................................ 7
Managing where the customer record came from ............................................................................... 7
Load Levelling ..................................................................................................................................... 8
Dynamic Error Handling ...................................................................................................................... 9
1
Case Study: Integrating BizTalk and Lagan CRM
Introduction
Lagan CRM is a software product owned by Kana and is very popular in the UK in the public sector
and social housing sector. Over the years we have come across Lagan on a few occasions and have
decided to write a small paper about how we helped a customer who for the purposes of this paper
we will call Acme to integrate Lagan with their internal line of business applications.
In this paper we will focus on the BizTalk and integration perspective of this project. There were a
number of other areas of this project such as telephony integration and user experience which are
interesting topics in their own right but are not areas we will be covering in this paper.
2
Case Study: Integrating BizTalk and Lagan CRM
About Connected Systems Consulting
Connected Systems Consulting Ltd is a small UK based consultancy specialising in integration
solutions based on the Microsoft technology platform. Connected Systems Consulting has worked
with customers to develop on premise integration solutions based around BizTalk and cloud to on
premise integration solutions using the Microsoft Windows Azure Service Bus.
For more information about Connected Systems Consulting please refer to our website:
http://connectedsystemsconsulting.sharepoint.com/
3
Case Study: Integrating BizTalk and Lagan CRM
About the Authors
Michael Stephenson
Michael Stephenson is the Managing Director of Connected Systems Consulting Ltd. Michael is also
Microsoft BizTalk MVP for a number of years and has been involved with the Microsoft Business
Platform Advisors programme including Windows Azure Advisors and Connected Systems Advisors.
Michael specialises in integration architecture and working with customers to help deliver effective
integration solutions based on the Microsoft platform.
Michael also specialises in ALM processes for integration projects and helping customers to have an
effective way of doing integration projects.
Michael is heavily involved with the UK Connected Systems User Group.
Michael please refer to the following links:
To find out more about
Blog: http://www.geekswithblogs.net/michaelstephenson
Linked In: http://www.linkedin.com/in/michaelstephensonuk1
Contributors
The following people have provided contribution to this case study.
Sedleigh Adams
Sedleigh Adams is an experienced system integration tester and support operator having extensive
experience with testing of complex systems such as British Airways reservation systems and airport
security processes. More recently Sedleigh has been working with Connected Systems Consulting to
help develop a managed service offering for BizTalk customers. Sedleigh has worked with customers
to ensure that they are effective at progressing their BizTalk projects through various parts of the
project life cycle and helping to provide the transition of projects into business as usual which may
involve hand over to the customers own support teams or providing support directly from Connected
Systems Consulting.
4
Case Study: Integrating BizTalk and Lagan CRM
The Project
The project making up this case study involved Acme implementing Lagan CRM as a part of their
application architecture with the specific purpose of using the case management functionality. Acme
were interested in a sub-set of their customers who’s accounts had some issues with them and Acme
wanted to provide a new call centre with the capability of working with their customers to resolve
account issues which would in turn have a significant impact on Acme’s cash flow.
To allow the advisors in this new call centre to have the right capability to case manage these
customers they would require the following core integration requirements with Lagan.
1. Batch Integration with Lagan to populate the case queue
2. Lagan requests data from other applications to help advisors to review the customer details
3. Lagan sends updates to other applications upon processing of a case
Acme was also undergoing a technology upgrade project and had previously chosen to use Microsoft
BizTalk Server as a core part of their integration platform which would help this CRM project but also
support on going integration requirements for future projects.
Acme had engaged with Connected Systems Consulting as a partner to help with their BizTalk
implementation.
Batch Integration with Lagan
Each week Acme would re-evaluate their case management position with a number of their back
office applications deciding which customers should be put into the case management process. The
back office applications would then produce a number of batches containing information about the
customers who needed case management and would submit them to BizTalk. The aim was that after
the batch process the back office applications and Lagan CRM would be synchronised and the start
of that week the call centre advisors would have prioritised lists of customers who they could contact
to help work through any account issues.
In the first phase of the batch integration there would be two batches of around 1000 records which
would be submitted for the pilot but overtime this would be rolled out across Acme’s other business
areas and by the end there would be up to 100 batches received within a very short time scale.
Lagan Requests Data from other Applications
When the call centre advisors were to begin work on Monday their queues would be full of customers
who needed case managing. As an agent in the call centre would begin working on a case they
would use Lagan to process the customer’s record and to record interactions such as outbound calls
to the customer.
Lagan initially would not store all of the information for the customer and in the case at Acme, Lagan
was not used as the master data source for customer information. The core customer information
was held in the back office applications and Lagan played the role of acting as a case management
front end to a lot of this data. Because Lagan would need to request additional information from other
back office applications BizTalk was used to facilitate this integration.
An example of this part of the process would be if the call centre agent needed to see some of the
customers historical interactions with Acme then Lagan would request this data and BizTalk would be
used to source this data from the application the case was originally loaded from.
5
Case Study: Integrating BizTalk and Lagan CRM
Lagan sends updates to other applications
In the same way that Lagan could obtain additional data from the back office applications a call centre
agent may also update some details about the customer and when this was committed then Lagan
would send this update to BizTalk and BizTalk would ensure that the right source application was
updated appropriately.
Project Use Cases
The following use case diagram illustrates the integration use case requirements for this project.
uc Primary Use Cases
System Boundary
Update Data
Back Office
Application 1
Lagan
Request Additional Data
«flow»
«flow»
Batch Case
Synchronisation
Back Office Application 2
Project Architecture
The below diagram illustrates the integration perspective on the project architecture.
6
Case Study: Integrating BizTalk and Lagan CRM
Call
Centre
Agent
Desktop
Lagan Web
Server
Lagan Database
Server
Basic Clustering
& Load Balancing
Back Office
Application A
Back Office
Application B
Interesting Architecture Topics
In the next section we will discuss some of the interesting architecture topics from this project.
Managing where the customer record came from
One of the areas of architectural interest concerns the routing between applications when Lagan
wants to query some additional data for a case. Sometimes you may consider a master data registry
solution but Acme do not currently have a master data management solution so the solution which
had been implemented involved telling Lagan which source application a case came from when cases
are originally created from the batch process. This data is then flown with the requests for additional
7
Case Study: Integrating BizTalk and Lagan CRM
data and BizTalk uses this data attribute to implement content based routing to go back to the original
source application to retrieve additional data.
This actually made the routing and publish subscribe pattern for the messaging between applications
a lot simpler as Lagan was able to act as the register of where the customers master data was held.
The below diagram illustrates how the source application is flown from Lagan and then used by
BizTalk to implement content based routing through property promotion.
Message submitted
containing source
application from Lagan
which is promoted to
BizTalk context
Filter for back office
application B
Eg. Source Application
= “Back Office
Application B”
Publish
Publish
Subscribe
Subscribe
MESSAGE BOX
Back Office
Application B
Lagan Web
Server
Back Office
Application A
Load Levelling
One of the key roles that BizTalk plays is during the batch processing features. In this area there will
be a queue of 1000’s of updates that need to be applied to cases in Lagan. In the Acme scenario
they are running an active/active instance of Lagan middle tier web services and the number of
updates which need to be applied could easily overload the web server and affect user activity if they
were sent to Lagan as fast as BizTalk would be capable of sending them.
This customer did not have the Lagan JMS capabilities available and all updates were to be applied
by the web service API. BizTalk has a number of ways of throttling the load and we were able to
control the load so that the through-put was throttled to a point that balances the duration to load the
batch with the speed the users need to work through the queues of resulting cases.
As the number of batches increases this throttling pattern simply means that the queue in BizTalk
gets a bit longer and we are feeding cases into Lagan at a constant rate that the Acme setup can
comfortably handle.
The below diagram illustrates the throttling implementation.
8
Case Study: Integrating BizTalk and Lagan CRM
The debatching
orchestrations will
handle many
concurrent batches
and queue up 1000's
of messages into the
BizTalk message box
waiting for processing
Debatching
Orchestration
Debatching
Orchestration
Publish
Publish
Subscribe
Subscribe
MESSAGE BOX
Throtting patterns can
be used to control how
many messages are
sent at once and the
rest are queued in the
BizTalk database
Adapter
Send to Lagan
Lagan Web
Server
Dynamic Error Handling
One of the interesting areas of the batch process was handling of different types of error. BizTalk
comes with a number of out of the box error handling features such as automatic retries and retry
intervals but one of the most useful in this scenario was the failed message routing capabilities.
When we are sending errors to Lagan in the batch process we implemented an orchestration which
would subscribe to failed messages from the batch send ports and this would allow us to create a
more complex handling routine. Out of the box the automatic retries do not look at what type of error
has occurred and simply implement one type of error handling regardless of the error type.
In our error handling orchestration we wanted to handle errors as follows:
Error
Lagan web server is down and returns a web
exception
Action
We would wait for a short period of time and then
send the message for retry. The retry count
would be incremented. If this error was persisting
and retries were exhausted we would suspend
the message and wait for an operator to resume
9
Case Study: Integrating BizTalk and Lagan CRM
it.
Lagan throws an error with Error Code 21
In this case we need to seek some assistance
from the Lagan support team so the message is
suspended and the BizTalk operator can query
with the Lagan team how to proceed. If the
message can be processed then the BizTalk
operator can resume the message and it will load
into Lagan.
Lagan throws an error with Error Code 12
As above
Lagan throws a SQL Deadlock error
In this case the error can usually be sent for retry
usually works the next time. In these cases we
often use diagnostics tools to inform the Lagan
team when instances such as these have
occurred so they can be investigated.
Lagan returns a case is already closed error
This case identified a defect in the process where
there was a mismatch between some status
codes. The case from the back office application
was identified as already being in Lagan and that
it could be closed but the status was already
closed. Initially our process would get these and
execute the default error handler but once we
identified this issue we were able to change the
error handling rules at runtime to clean up these
messages until the issue was fixed.
Default
Any other error which had not already been
matched would be suspended and the BizTalk
operator would be able to review the error and
decide which action to take and then send the
message for retry if required.
To implement this dynamic error handling capability in BizTalk we used the BizTalk Business Rules
Engine to develop a policy which would inspect the error details and the message which we were
attempting to process. The rules policy would then workout which action should be taken. The
actions that were supported include:
1. Increment the retry count and send the message for retry
2. Wait a period of time then increment the retry count and send the message for retry
3. Delete the message as it will never be able to be processed but raise an alert to inform the
operator
4. Delete the message as it cannot be processed and is a known issue that can be cleaned up
5. Suspend the message and raise an alert to the BizTalk operator who will decide if the error
can be sent for retry or not.
Conclusion
In this paper we have tried to provide a high level overview of a successful project involving BizTalk
and Lagan to deliver significant business value to Acme. Connected Systems Consulting were
approached by Acme to help when the project found some difficulties during the early stages and
10
Case Study: Integrating BizTalk and Lagan CRM
Connected Systems Consulting were able to help Acme to put the right team and ALM processes in
place alongside a simplified architecture which ensured the project was successful.
As a follow up to this case study we will also produce some guidance to help people with some of the
lessons learnt through integration with Lagan.
11
Case Study: Integrating BizTalk and Lagan CRM