How everyone should test
their BizTalk based
solutions
Darren Jefford
Solution Architect
Microsoft UK
darrenj@microsoft.com
http://blogs.msdn.com/darrenj
Objectives And Takeaways
Objective(s):
Some complex BizTalk solutions run into
problems
NONE should!
Raise awareness of key tools and
techniques to dramatically improve
customers experience
Based on real world experience
Drive the quality of enterprise-class BizTalk
solutions up
Administration is also key..
BizTalk, Testing, No Time!
We all know testing is important for every
solution
Customers rarely test well enough... but get
away with it
Solutions using BizTalk tend to be extremely
business critical and tolerate no downtime
Poor testing runs the risk of huge technical
and political problems
Extensive testing must always be
performed, no excuses!
Testing approach
“Test early, often and comprehensively”
Ensure that a demonstrator is created early
on to prove concepts, patterns and
indicative performance
Too many solutions are found to perform
badly in the last few weeks for any number
of reasons
Highlights sub-optimal design up-front
Identifies where the bottlenecks are likely to be
Increases confidence
We sometimes hear...
“it doesn’t scale”
“it doesn’t meet our performance
requirements”
“we were told it would do XX msgs/sec”
BizTalk is not
A universal panacea
Optimised for every customer scenario out-ofthe-box
Able to make bad code, good code
BizTalk is part of the overall end-to-end
solution
A great “case study”
“ [BizUnit,LoadGen,MSTest] provides the
ability to reliably and automatically execute
hundreds of test cases that otherwise would
have to be performed manually”
“improved turn around times by a factor of
upwards of 5. For a test pack containing
100 test cases which took up to 3 days to
complete manually we can now execute
in approx 3 hrs”
Laying the foundations
Critical to achieving representative
performance within your environment
Benchmark your environment
File Adapter -> File Adapter (3K) message should
get ~500msgs/sec
Don’t run perf tests on workstation hardware
Beware of virtual machines
Invest in SQL Server Hardware and Disks
Beware of UNC FILE Adapter locations
Message Tracking
Remember Virus Scanners
Foundation Checklist
 Databases and Logs separated onto separate drive letters and
therefore separate physical hard drives?
 Are the drives used for Databases and Logs configured on a fast diskarray and with RAID10/0+1?
 Next generation SAN technology tends to use huge caches so
RAID choice is becoming unimportant
 Default Database Autogrowth settings changes or AutoGrowth
disabled?
 SQL Server Agent Running?
 Have you created separate Receive, Orchestration and Send Hosts
and moved the appropriate Adapters and Orchestrations to the
appropriate host?
 Concurrent Web Service Calls configuration file setting changed?
<system.net>
<connectionManagement>
<add address="*” maxconnection="80"/>
</connectionManagement>
</system.net>
PerfMon Counters..... ??
A huge selection of performance counters
are made available by the BizTalk engine
It can be time consuming to identify the
counters you need especially when you
have multiple machines
Collection of counters is key
Perfmon Counters
Excel spreadsheet to define counter
collection for multiple machines
Visual Studio Profiler
New with Visual Studio Team Developer
Ideal for profiling custom code: custom
pipeline components, .NET class libraries
Profiler can be installed standalone for
servers..
vs_profiler.exe located in the
\vs\wcu\profiler directory of the VSTS
media
Must be driven via the command line to
attach to a service
Configuring the VS Profiler
VSPerfClrEnv /globalsampleon
VSPerfCmd /start:sample
/output:“c:\dir\profileroutput"
VSPerfCmd /attach:<PID>
VSPerfCmd /detach:<PID>
VSPerfCmd /shutdown
VSPerfClrEnv /globaloff
Code Coverage
Enables you to understand what code has
been executed during your test run
Identifies holes in your testing
Ideal for custom pipeline components, helper
assemblies, web services, etc.
vsinstr –coverage “YOURASSEMBLY”
vsperfmon –coverage –output
“filename.coverage”
vsperfcmd /shutdown
How do I test this?
BizTalk: DarrenJ Travel Services
Wiley Flight
Services
Booking
Web Service
Flight
Orchestration
Itinerary Message
[ASMX]
Itinerary
Orchestration
Hotel
Orchestration
Hire Car
Orchestration
Travel
Insurance
Orchestration
OrchestrationEventStream
BAMPrimaryImport
MSMQ
DirectEventStream
BizUnit
Test Cases are constructed from generic
and reusable test steps
Test cases described using an XML syntax
Driven from NUNIT or MSTest
Model every “use case” within your solution
Run a test
If you get a “green light” your solution functions
at a requirements level
Incredibly powerful, not just for the
development lifecycle but for changes, new
releases.....
Not just about BizTalk
Test Steps available “in box”
BAMDeploymentStep
CheckPop3MailStep
ContextManipulatorStep
CrossReferenceDataVerificationStep
DatabaseDeleteStep
DatabaseRowCountStep
DBQueryStep
DelayStep
DotNetObjectInvokerStep
EventLogCheckStep
EventLogClearStep
ExecuteCommandStep
FileCreateStep
FileMultiValidateStep
FilesExistStep
FilesMoveStep
FileDeleteMultipleStep
FileDeleteStep
FileMoveStep
FileValidateStep
HttpPostStep
HttpRequestResponseStep
MQSeriesClearQueueStep
MQSeriesGetStep
MQSeriesPutStep
MSMQCreateQueueStep
MSMQDeleteQueueStep
MSMQQueuePurgeStep
MSMQReadStep
MSMQWriteStep
OrchestrationConductorStep
ReceiveLocationEnabledStep
ReceivePortConductorStep
RenameDirectoryStep
RuleEngineStep
SMTPReadStep
SOAPHTTPRequestResponseStep
WaitOnFileStep
OutlookReadStep
Your own.....
BizUnit Test Case Format
TestSetup Stage
Test Step: ExecuteCommandStep
TestExecution Stage
Test Step: MSMQWriteStep
Test Step: MSMQReadStep
Validation Step:
XmlValidationStep
Test Step: DBQueryStep
TestCleanup Stage
Test Step: MSMQQueuePurgeStep
Setup
Initialise the platform
ready to perform the
test
Execution
The execution of the
test
Cleanup
Restore the platform
to its state prior to test
execution
Invoke a web service
<TestStep assemblyPath=""
typeName="SOAPHTTPRequestResponseStep">
<WebServiceWSDLURL>http://..</WebServiceWSDLURL>
<ServiceName>MyWebServiceName</ServiceName>
<WebMethod>SubmitItinerary</WebMethod>
<InputMessageTypeName>Itinerary</InputMessageTypeName>
<MessagePayload>ItineraryMessage.xml</MessagePayload>
<ContextLoaderStep assemblyPath="" typeName="XmlContextLoader">
<XPath contextKey="itineraryReference">
//ItineraryReceived/attribute::ItineraryReference
</XPath>
</ContextLoaderStep>
</TestStep>
Validate a MSMQ message
<TestStep assemblyPath="" typeName=“MSMQReadStep">
<QueuePath>.\Private$\HotelQ</QueuePath>
<Timeout>10000</Timeout>
<ValidationStep assemblyPath="“
typeName=“BinaryValidationStep">
<ComparisonDataPath>
HotelFragment.xml
</ComparisonDataPath>
<CompareAsUTF8>true</CompareAsUTF8>
</ValidationStep>
</TestStep>
Validate database rows
<TestStep assemblyPath="" typeName=“DBQueryStep">
<DelayBeforeCheck>25</DelayBeforeCheck>
<ConnectionString>...</ConnectionString>
<SQLQuery>
<RawSQLQuery>
SELECT * FROM dbo.bam_Itinerary_Completed
WHERE ActivityID='{0}‘
</RawSQLQuery>
<SQLQueryParams>
<SQLQueryParam takeFromCtx="itineraryReference“/>
</SQLQueryParams>
</SQLQuery>
<Rows><Columns>
<CustomerName>Darren Jefford</CustomerName>
</Columns></Rows>
</TestStep>
BizUnit
Writing a test for a end-to-end scenario
Orchestration Profiler
Code Coverage tools don’t really help with
Orchestrations
How do you understand where time is being
spend inside Orchestrations?
Enabling tracking and using HAT is the only real
way right now
No way to understand which Orchestrations
are not being fully tested
Edge Scenarios
Error paths
Orchestration Profiler
“Code Coverage” your orchestrations
Identify slow performing shapes and
orchestrations
LoadGen
BizTalk Load Generation Tool
New 2007 release provides WCF support
Generate messages to test how servers
handle normal and extreme workload
conditions
Data Generation
Load Throttling
Find the maximum sustainable rate of your rig,
automatically.
Multithreaded
“Throw away all those custom test
harnesses”, please... 
“in-box” Load Generators
FILE
Outputs files in a destination folder
HTTP
Supports both asynchronous and request-response messaging.
SOAP
Supports both asynchronous and request-response messaging
MSMQ
Supports both asynchronous and request-response messaging
MQSeries
Message can be local or remote, in-order or unordered, persistent or
non-persistent.
WSE
Supports both asynchronous and request-response messaging.
WSS
Windows® SharePoint® Services.
Custom
Base your own on the FileTransport sample
In-box Message Creators
CustomMC
Create new messages based on a template
Replace Items within the message with
differing data
SizeMC
Explodes a sample message to a certain size
(big batch files)
In-box Throttle Monitors
SQLSpool
Ensure the Spool depth remains within a
certain range
Rate
100 msgs/sec..
CPU
Keep the BizTalk servers within 80-90% CPU
utilisation
PerfMon
Monitor any PerfMon counter
LoadGen
Generating load into your BizTalk solutions
LoadGen and BizUnit
LoadGen
Fantastic load generation tool
Generate increasing load
Find the maximum sustainable point of your solution
BizUnit
Extensive library of Test Steps
Collecting PerfMon and EventViewer Logs
Storing Test Information
Validating Test Results
BizUnit can drive LoadGen
Automate all of your performance testing endto-end!!
LoadGen and BizUnit
Driving test cases using LoadGen and
BizUnit
•Shameless Plug Further Reading
In depth chapters covering Testing,
Performance & Scalability..
Summary
Get the environment right before any testing
Start testing early, and often.
A “demonstrator” is key for complex
scenarios
Use BizUnit for modelling “use cases”
Use LoadGen for generating load
Use Orchestration Profiler to ensure your
“touching” all parts of your solution during
testing
Use BizUnit and LoadGen together to
automate your tests
Tools
PerfMon Counter spreadsheet
Orchestration Profiler
BizTalk Documenter
LoadGen 2007
BizUnit
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.