Instrumentation Best Practices for High
Performance BizTalk Solutions
Authored by: Valery Mizonov (Microsoft Corporation)
Reviewed by: Mark Simms, Jayanthi Sampathkumar, Krish Srinivasan (Microsoft Corporation)
Copyright
The information contained in this document represents the current view of Microsoft Corporation on
the issues discussed as of the date of publication. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft
cannot guarantee the accuracy of any information presented after the date of publication.
This white paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights
under copyright, no part of this document may be reproduced, stored in, or introduced into a retrieval
system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or
otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property
rights covering subject matter in this document. Except as expressly provided in any written license
agreement from Microsoft, the furnishing of this document does not give you any license to these
patents, trademarks, copyrights, or other intellectual property.
Unless otherwise noted, the example companies, organizations, products, domain names, e-mail
addresses, logos, people, places, and events depicted herein are fictitious, and no association with any
real company, organization, product, domain name, e-mail address, logo, person, place, or event is
intended or should be inferred.
© 2010 Microsoft Corporation. All rights reserved.
Microsoft, MS-DOS, Windows, Windows Server, Windows Vista, Active Directory, BizTalk, Excel,
SharePoint, SQL Server, Visio, Visual C#, and Visual Studio are trademarks of the Microsoft group of
companies.
All other trademarks are property of their respective owners.
Contents
Background ................................................................................................................................................... 4
The Challenges .......................................................................................................................................... 4
The Solution .............................................................................................................................................. 6
Performance Considerations ...................................................................................................................... 10
Instrumentation Best Practices................................................................................................................... 11
Instrumentation of Custom Pipeline Components ................................................................................. 12
Instrumentation of BizTalk Maps ............................................................................................................ 13
Instrumentation of BizTalk Orchestrations ............................................................................................. 15
Instrumentation of Business Rules ......................................................................................................... 17
Instrumentation of Custom Components ............................................................................................... 19
Management of Instrumented Applications............................................................................................... 21
Event Trace Management Application Landscape.................................................................................. 22
Event Trace Management Tasks ............................................................................................................. 22
Starting Event Trace Sessions ............................................................................................................. 22
Monitoring Event Trace Sessions ........................................................................................................ 24
Stopping Event Trace Sessions ............................................................................................................ 24
Configuring Event Trace Sessions ....................................................................................................... 25
Combining Event Tracing With Real-Time Monitoring ....................................................................... 26
Conclusion ................................................................................................................................................... 27
Additional Resources/References ............................................................................................................... 28
Background
Application code instrumentation has always been crucially important to diagnosing and troubleshooting complex
software solutions. Rich tracing and logging enables the collection of detailed telemetry and diagnostic information
critical to understanding application behavior. This is especially important for distributed server-side applications such
as those deployed on the BizTalk Server platform.
There are many real-world examples where Microsoft® BizTalk® Server developers have added tracing and logging
capabilities to their applications using inefficient methods, often resulting in an adverse effect on the application’s
performance running under stress. The large number of tracing events emitted by the instrumented code may
become a factor limiting throughput or increasing latency. It is therefore becoming imperative to leverage more
optimal ways of enriching the application code with instrumentation without sacrificing performance.
This whitepaper discusses the alternative solution for enriching BizTalk solutions with high-performance
instrumentation, enabling BizTalk Server developers to develop fully instrumented applications and significantly
facilitate the diagnostic and troubleshooting aspects of their applications regardless of complexity. The whitepaper
highlights the benefits of the proposed solution and drills down into specific instrumentation scenarios across the
entire family of BizTalk solution artifacts.
The whitepaper is based on real-world lessons learned from customer engagements and reflect the learnings from
multiple customer projects led by Microsoft’s Windows Server® AppFabric Customer Advisory Team (AppFabric CAT).
The Challenges
Through a number of BizTalk projects and customer engagements, we observed that many developers typically
started adding instrumentation into their BizTalk applications by using a dependency on the System.Diagnostics.Trace
and System.Diagnostics.DefaultTraceListener components from the .NET Framework or Enterprise Library’s Logging
Application Block. Developers may also consider using third-party logging/tracing components such as log4net.
Generally, the use of the Win32 Debugging APIs such as those used by the DefaultTraceListener class and other
instrumentation packages may not deliver the desired level of agility and performance.
For instance, a dependency on the Win32 Debugging APIs for the purposes of code instrumentation may result in
higher than usual CPU utilization on the host machine while tracing data is being captured by the DebugView utility one of the most commonly used tools for intercepting Win32 debug events. In the example below, the CPU utilization
hits 85-90% on average when DebugView is running and collecting trace events from a BizTalk application running
under stress. The tool was consuming the vast majority of the CPU time due to a large number of events being
emitted by the instrumented BizTalk application and tracked by the DebugView utility into a trace log file.
.
Note that running DebugView during stress testing may significantly impact the application performance. The tool is
acting as a debugger and hooks on the OUTPUT_DEBUG_STRING_EVENT event. As a result, a BizTalk application
instrumented using the System.Diagnostics.Trace component configured with a default trace listener may be
experiencing performance degradation as the application threads will be suspended while the debug information is
written to the trace.
From a performance perspective, it is important to understand the throughput characteristics of the chosen
instrumentation package and determine the rate in which instrumentation events such as tracing method calls can be
logged in the output. For example, the benchmarked throughput of the most commonly used
System.Diagnostics.Trace component with a default trace listener can be anywhere between 2500 and 4000 trace
events per second when event capture is enabled in the DebugView tool. Although this number may appear to be
reasonably sufficient, it may turn out to become a factor slowing the application performance down, specifically in the
multi-threaded BizTalk applications where hundreds of events could be emitted concurrently from many worker
threads.
From an operational perspective, having a requirement to restart the BizTalk host instances so that any changes in the
tracing configuration such as enabling, disabling or changing the trace level can take effect, may be highly
undesirable or even unacceptable in the production environment. Some instrumentation packages such as Enterprise
Library solve this challenge by providing the support for real-time application configuration refresh, however, this
capability remains unavailable for other developers not leveraging the corresponding packages. For example, there is
no support for auto-refresh in the tracing configuration in the System.Diagnostics.Trace component unless a
developer specifically implements this functionality.
The above experiences and observations force to rethink many of the design and implementation decisions when it
comes to adding code-level instrumentation into BizTalk applications demanding high performance. To this end, we
have come forward with the following solution.
The Solution
We recommend the use of a high-performance instrumentation framework leveraging the Event Tracing for Windows
(ETW) infrastructure, enabling applications to take advantage of the general purpose, high-speed tracing
infrastructure supported directly by the operating system kernel.
ETW is a high-performance, low overhead, and highly scalable tracing facility provided by the Windows ® operating
system. Using an efficient buffering and logging mechanism implemented in the operating system, ETW provides a
fast transport for logging events raised by user-mode applications and kernel-mode device drivers. ETW was first
introduced on Windows 2000. Since then, various core operating system and infrastructure components have
adopted ETW as a first-class instrumentation and tracing technology.
In addition, the ETW event tracing infrastructure is being extensively used virtually by all major infrastructure
components inside the BizTalk runtime, including EPM, transport adapters, Message Agent, etc. The internal APIs in
the BizTalk runtime infrastructure provide a generic implementation of a custom ETW event provider along with
public helper classes that could be leveraged from any BizTalk application. One of the major components in the
BizTalk Server tracing APIs is TraceProvider which can be found in the Microsoft.BizTalk.Diagnostics namespace
provided by Microsoft.BizTalk.Tracing.dll assembly.
The Microsoft.BizTalk.Diagnostics.TraceProvider class enables submitting the user-defined tracing data to a
custom ETW provider and exposes the following key properties and methods:
namespace Microsoft.BizTalk.Diagnostics
{
[ComVisible(true)]
[Guid("748004CA-4959-409a-887C-6546438CF48E")]
public sealed class TraceProvider
{
public TraceProvider(string applicationName, Guid controlGuid);
public uint Flags { get; }
public bool IsEnabled { get; }
public void TraceMessage(uint traceFlags, object format);
public void TraceMessage(uint traceFlags, object format, object data1);
// + a few more overloads of TraceMessage accepting extra data items.
}
}
The custom instrumentation framework discussed in this whitepaper relies on the above component and provides a
rich set of tracing methods for various types of events such as informational, warnings, errors and exceptions. There is
also support for tracing the method calls and measuring execution durations using high-resolution timers.
The key component in the custom instrumentation framework is the ComponentTraceProvider class which is acting
as a wrapper for TraceProvider. Below is the class diagram depicting the core members of the instrumentation
framework:
The ComponentTraceProvider class delivers its core functionality through implementing the purposely defined
custom IComponentTraceProvider interface with the following methods:
// Writes an information message to the trace.
void TraceInfo(string format, params object[] parameters);
// Writes an information message to the trace. This method is provided for optimal
// performance when tracing simple messages which don't require a format string.
void TraceInfo(string message);
// Writes an information message to the trace. This method is intended to be used when
// the data that needs to be written to the trace is expensive to be fetched. The
// method represented by the Func<T> delegate will only be invoked if tracing is
// enabled.
void TraceInfo(Func<string> expensiveDataProvider);
// Writes a warning message to the trace.
void TraceWarning(string format, params object[] parameters);
// Writes a warning message to the trace. This method is provided for optimal
// performance when tracing simple messages which don't require a format string.
void TraceWarning(string message);
// Writes a message to the trace. This method can be used to trace detailed
// information which is only required in particular cases.
void TraceDetails(string format, params object[] parameters);
// Writes an error message to the trace.
void TraceError(string format, params object[] parameters);
// Writes an error message to the trace. This method is provided for optimal
// performance when tracing simple messages which don't require a format string.
void TraceError(string message);
// Writes the exception details to the trace.
void TraceError(Exception ex);
// Writes the exception details to the trace.
void TraceError(Exception ex, bool includeStackTrace);
// Writes the exception details to the trace.
void TraceError(Exception ex, bool includeStackTrace, Guid callToken);
// Writes an informational event into the trace log indicating that a method is
// invoked. This can be useful for tracing method calls to help analyze the code
// execution flow. The method will also write the same event into default
// System.Diagnostics trace listener, however this will only occur in the DEBUG code.
// A call to the TraceIn method would typically be at the very beginning of an
// instrumented code.
Guid TraceIn(params object[] inParameters);
// Writes an informational event into the trace log indicating that a method is
// invoked. This can be useful for tracing method calls to help analyze the code
// execution flow. The method will also write the same event into default
// System.Diagnostics trace listener, however this will only occur in the DEBUG code.
// A call to the TraceIn method would typically be at the very beginning of an
// instrumented code. This method is provided to ensure optimal performance when no
// parameters are required to be traced.
Guid TraceIn();
// Writes an informational event into the trace log indicating that a method is about
// to complete. This can be useful for tracing method calls to help analyze the code
// execution flow. The method will also write the same event into default
// System.Diagnostics trace listener, however this will only occur in the DEBUG code.
// A call to the TraceOut method would typically be at the very end of an instrumented
// code, before the code returns its result (if any).
void TraceOut(params object[] outParameters);
// Writes an informational event into the trace log indicating that a method is about
// to complete. This can be useful for tracing method calls to help analyze the code
// execution flow. The method will also write the same event into default
// System.Diagnostics trace listener, however this will only occur in the DEBUG code.
// A call to the TraceOut method would typically be at the very end of an instrumented
// code, before the code returns its result (if any). This method is provided to
// ensure optimal performance when no parameters are required to be traced.
void TraceOut();
// Writes an informational event into the trace log indicating that a method is about
// to complete. This can be useful for tracing method calls to help analyze the code
// execution flow. The method will also write the same event into default
// System.Diagnostics trace listener, however this will only occur in the DEBUG code.
// A call to the TraceOut method would typically be at the very end of an instrumented
// code, before the code returns its result (if any).
void TraceOut(Guid callToken, params object[] outParameters);
// Writes an informational event into the trace log indicating a start of a scope for
// which duration will be measured.
long TraceStartScope(string scope, params object[] parameters);
// Writes an informational event into the trace log indicating the start of a scope
// for which duration will be measured. This method is provided in order to ensure
// optimal performance when no parameters are available for tracing.
long TraceStartScope(string scope);
// Writes an informational event into the trace log indicating the start of a scope
// for which duration will be measured. This method is provided in order to ensure
// optimal performance when only 1 parameter of type Guid is available for tracing.
long TraceStartScope(string scope, Guid callToken);
// Writes an informational event into the trace log indicating the end of a scope for
// which duration will be measured.
void TraceEndScope(string scope, long startTicks);
// Writes an informational event into the trace log indicating the end of a scope for
// which duration will be measured.
void TraceEndScope(string scope, long startTicks, Guid callToken);
In addition, tracing can be enabled and disabled separately for individual component types such as custom pipeline
components, orchestrations, maps, rules and so on. The singleton TraceManager class exposes a static instance of an
object implementing the IComponentTraceProvider interface for each supported component type as follows:
// The main tracing component which is intended to be invoked from user code.
public static class TraceManager
{
// The trace provider for user code in the custom pipeline components.
public static IComponentTraceProvider PipelineComponent {/*Omitted for brevity*/}
// The trace provider for user code in workflows (such as expression shapes in the
// BizTalk orchestrations).
public static IComponentTraceProvider WorkflowComponent {/*Omitted for brevity*/}
// The trace provider for user code in the custom components responsible for data
// access operations.
public static IComponentTraceProvider DataAccessComponent {/*Omitted for brevity*/
}
// The trace provider for user code in the transformation components (such as
// scripting functoids in the BizTalk maps).
public static IComponentTraceProvider TransformComponent {/*Omitted for brevity*/}
// The trace provider for user code in the service components (such as Web
// Service, WCF Service or service proxy components).
public static IComponentTraceProvider ServiceComponent {/*Omitted for brevity*/}
// The trace provider for user code in the Business Rules components (such as
// custom fact retrievers, policy executors).
public static IComponentTraceProvider RulesComponent {/*Omitted for brevity*/}
// The trace provider for user code in the business activity tracking components
// (such as BAM activities).
public static IComponentTraceProvider TrackingComponent {/*Omitted for brevity*/}
// The trace provider for user code in any other custom components which don't
// fall into any of the standard categories such as Pipeline, Workflow,
// DataAccess, Transform, Service or Rules.
public static IComponentTraceProvider CustomComponent {/*Omitted for brevity*/}
}
At a glance, the above instrumentation framework delivers the following benefits:

Highest possible performance – the initial tests on a 3.2GHz Quad Core machine have demonstrated that
ETW-based tracing component is capable of writing about 1.4 million trace events per second comparing to
just 4000 events/second delivered by the Trace class from System.Diagnostics;




No high CPU utilization observed whenever trace events are persisted into a log file, even when running
under stress. ETW is using a buffering and logging mechanism implemented in the kernel. The logging
mechanism uses per-processor buffers that are written to disk by an asynchronous writer thread. This
significantly reduces the impact of the log write operations on the application and system performance;
Full operational flexibility enables switching the tracing on and off as well as changing the trace level
dynamically, making it easy to perform detailed tracing in production environments without requiring
machine reboot or application restart;
Tracing can be safely enabled to run continuously in a production environment using the circular logging
option which ensures that log files will not outgrow the available disk space;
Lightweight footprint on the instrumented application is achieved through minimizing the external
dependencies down to a single BizTalk assembly with no other supporting artifacts, configuration files, etc.
Performance Considerations
When compared to the other 3 popular rivals (System.Diagnostic.Trace in the .NET Framework, log4net and Enterprise
Library in their default configuration), the ETW-based instrumentation framework has demonstrated a significantly
better performance. In order to measure the true extent of performance benefits delivered by ETW, the following
scenario has been tested:



Test case: Execute 1 instrumented method 100,000 times writing 1 trace event in each iteration.
Pre-requisites: Configure all 4 benchmarked instrumentation frameworks to produce a text-based trace log
file containing the tracing data.
Test bed: A high-end desktop PC with 3.2GHz Quad Core CPU, 4GB RAM, 64-bit operating system, RAID-5
SATA II 7.2K disk array.
Below are the results from the benchmarking exercise:
The above results confirm that the ETW-based instrumentation framework outperforms all 3 rivals in the order of
magnitude. When visualized on a chart, the above performance numbers render a substantial gap between the
throughput of the ETW infrastructure and other 3 popular tracing frameworks:
The accompanying sample code includes a test tool which was used for benchmarking all 4 instrumentation
frameworks and producing the above results.
Instrumentation Best Practices
The examples below illustrate the usage pattern for the instrumentation framework discussed in this whitepaper. The
following code fragment demonstrates a simple use case whereby the instrumented method
(BeginAndCompleteActivity) captures two events indicating when the method was invoked (TraceIn) and when it was
completed (TraceOut).
public void BeginAndCompleteActivity(ActivityBase activity)
{
Guard.ArgumentNotNull(activity, "activity");
var callToken = TraceManager.TrackingComponent.TraceIn(activity.ActivityName,
activity.ActivityID);
this.eventStream.BeginActivity(activity.ActivityName, activity.ActivityID);
this.eventStream.UpdateActivity(activity.ActivityName, activity.ActivityID,
ActivityTrackingUtility.GetActivityData(activity));
this.eventStream.EndActivity(activity.ActivityName, activity.ActivityID);
TraceManager.TrackingComponent.TraceOut(callToken);
}
When enabled, the trace log will contain both events written with a timestamp along with a GUID-based method call
correlation token which enables to match the related TraceIn and TraceOut events together:
The following sections drill down into specific instrumentation scenarios across the entire family of artifacts in a
BizTalk solution and demonstrate how the custom instrumentation framework could be leveraged to add rich, highperformance tracing capabilities into the most common BizTalk solution components.
Instrumentation of Custom Pipeline Components
The custom pipeline components can be instrumented using TraceManager.PipelineComponent which is dedicated
to be used for this type of BizTalk artifacts. Some of the useful events to be captured through the code
instrumentation in the custom pipeline components may include:




Tracing calls to the core methods such as Execute, Disassemble, GetNext, etc. (using TraceIn and TraceOut);
Measuring duration of the above methods (using TraceStartScope and TraceEndScope);
Tracing the internal state of pipeline components which could in turn assist with troubleshooting (using
TraceInfo);
Capturing detailed information about runtime exceptions (using TraceError).
Below is an example of some of the techniques highlighted above:
Instrumentation of BizTalk Maps
The BizTalk maps can be instrumented using TransformTraceManager component which can be invoked from inside a
custom scripting functoid associated with an external assembly. Some useful events that can be captured from the
maps when they are being executed by the BizTalk runtime engine may include:


Tracing the invocation of XSLT templates implementing the BizTalk maps (using TraceIn and TraceOut);
Tracing the internal state of maps, e.g. the node values which are being transformed, which could in turn
assist with troubleshooting (using TraceInfo).
In the example below, the instrumented map is using the TraceInfo method to report the values of the GSxx nodes
found in the input document:
Note that BizTalk maps cannot invoke static methods from an external assembly by design. Consequently, there is a
separate non-static class (TransformTraceManager) that is responsible for relaying method calls between a map and
the statically initialized singleton TraceManager.TransformComponent object.
When adding instrumentation into BizTalk maps, it’s important to have the custom scripting functoids linked to
appropriate nodes in the output document, otherwise the scripting code will be omitted from the auto-generated
XSLT template. In the above example, all scripting functoids are connected to the root node in the output document
via a helper Logical OR functoid. This ensures that tracing takes place right before creating the root XML element and
results in generating the following XSLT code by the BizTalk mapper:
<!-- There is some XSLT code before this element -->
<xsl:template match="/s2:CSC_837D">
<xsl:variable name="var:v1"
select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS01/text()))" />
<xsl:variable name="var:v2"
select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS02/text()))" />
<xsl:variable name="var:v3"
select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS03/text()))" />
<xsl:variable name="var:v4"
select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS04/text()))" />
<xsl:variable name="var:v5"
select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS05/text()))" />
<xsl:variable name="var:v6"
select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS06/text()))" />
<xsl:variable name="var:v7"
select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS07/text()))" />
<xsl:variable name="var:v8"
select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0: GS08/text()))" />
<xsl:variable name="var:v9" select="userCSharp:LogicalOr(string($var:v1),
’true’, string($var:v2), string($var:v3), string($var:v4), string($var:v5),
string($var:v6), string($var:v7), string($var:v8))" />
<xsl:if test="$var:v9">
<ns0:W4880200-COMMON-EDIT-MED-CLAIM>
<!-- There is plenty of other XSLT code after this element -->
Instrumentation of BizTalk Orchestrations
As complexity of the BizTalk orchestrations evolves, the instrumentation is becoming a key factor in helping to
diagnose and troubleshoot behavioral problems, performance-related issues and other bottlenecks that were not
foreseen or manifested themselves during development.
A well-instrumented orchestration can be defined as follows:






The entry point into an orchestration is recorded either as the very first activity (for non-activated
orchestrations) or immediately after the top Receive shape (using TraceIn);
The internal state of orchestrations (e.g. variables, results from method calls, non-sensitive message payload)
is wisely traced (using TraceInfo);
Unexpected behavior is reported as soon as it is detected (using TraceWarning or TraceError);
The detailed information about all runtime exceptions is traced from inside the exception handling block
(using TraceError);
The duration of individual scopes as well as the entire orchestration is measured and traced (using
TraceStartScope and TraceEndScope);
The exit point from an orchestration is recorded (using TraceOut) either right before the Terminate shape or
at the very last step in the orchestration.
Below are some examples of the instrumentation techniques highlighted above. In these examples, the BizTalk
orchestrations are instrumented using TraceManager.WorkflowComponent that is being invoked from within
Expression shapes.
1.
First, an entry point into the orchestration is traced inside the Trace Entry expression shape:
2.
Secondly, the internal state of the Listen shape is traced for the purposes of facilitating troubleshooting:
3.
Next, the exception handling scope is instrumented with a TraceError event in order to provide detailed
information about the exception:
4.
In addition, the orchestration contains instrumented scopes for which duration is measured by calling the
TraceStartScope at the beginning of the scope and completing the scope with a call to the TraceEndScope
method:
5.
And lastly, the exit event is traced right before the orchestration’s termination point:
Instrumentation of Business Rules
The instrumentation of business rules can be extremely valuable when there is a requirement to understand how rule
conditions are being evaluated, what rules are being fired as well as how long it takes to execute a rule set (policy).
A well-instrumented business rule policy can be defined as follows:




All calls to helper classes responsible for invoking the Business Rules Engine (BRE) APIs (such as Policy
objects) are traced (using TraceIn and TraceOut);
The duration of policy execution is recorded (using TraceStartScope and TraceEndScope);
The BRE policies are invoked passing an instance of a class implementing the IRuleSetTrackingInterceptor
interface - this class should report on all significant events which occur during policy execution;
Each rule in the policy informs the instrumentation framework indicating that the rule was fired by the BRE
engine (using TraceInfo, TraceWarning or TraceError).
The business rules instrumentation is provided by 3 components available in the custom instrumentation framework:


TraceManager.RulesComponent provides general instrumentation methods for all helper .NET components
accessing the BRE APIs;
RuleTraceManager residing in the Microsoft.BizTalk.CAT.BestPractices.Framework.RulesEngine namespace
implements a static class which is intended to be used on the Action pane inside the Business Rule
Composer. Its main purpose is to expose a set of tracing methods with a fixed parameter list. Internally, the
RuleTraceManager is relaying all calls to TraceManager.RulesComponent. A requirement for a separate

class has arisen from the fact that the Business Rules Engine doesn’t currently support the invocation of
methods with a variable number of parameters (also known as parameter arrays);
TracingRuleTrackingInterceptor which is available in the Microsoft.BizTalk.CAT.BestPractices.
Framework.RulesEngine namespace provides a custom implementation of the IRuleSetTrackingInterceptor
interface. This tracking interceptor captures all events occurring during BRE policy execution and produces
respective trace events using the TraceManager.RulesComponent class.
The following recommendations would apply in the context of business rules instrumentation:


When calling any static methods from inside the business rules, make sure that StaticSupport parameter is
present either in the application’s configuration file (e.g. BTSNTSvc.exe.config) or system registry. Otherwise,
the following behavior may be encountered;
Consider adding the TraceInfo instrumentation event into each individual rule to facilitate the analysis of rule
execution without having to analyze all events reported by the Rules Engine through
TracingRuleTrackingInterceptor.
Below are some examples of the instrumentation techniques highlighted above.
1.
First, the individual rules contain an instrumentation event recording the fact of the rule being invoked by
BRE:
2.
Secondly, any unexpected control flow such as execution of rules not intended to have been fired for a given
parameter set is being traced using the TraceError event:
3.
And lastly, the custom component responsible for invocation of the BRE policies is instrumented with scope
duration tracing and other events which can provide valuable diagnostic information and assist with
troubleshooting:
Instrumentation of Custom Components
Custom .NET components can be instrumented using TraceManager.CustomComponent which enables:




Method-level instrumentation whereby calls to individual methods are recorded in the trace log with zero or
more parameters (using TraceIn and TraceOut);
Measuring durations to be able to capture accurate execution timings (using TraceStartScope and
TraceEndScope);
Tracing the internal state of custom components which could in turn assist with troubleshooting (using
TraceInfo or TraceWarning);
Writing detailed information about runtime exceptions (using TraceError).
Below is an example showing some of the techniques highlighted above:
In addition, there may be requirements for enabling individual custom components to emit instrumentation events
and be able to capture these in isolation from events produced by other components. This approach opens the
opportunity for collecting the detailed behavioral and telemetry data related to a specific component whilst ensuring
that any events emitted by other custom components will not be captured in order to reduce the “noise” level.
To enable this scenario and add instrumentation at the individual component level, the following steps are to be
followed.
1.
First, every .NET component which required to be individually instrumented needs to be decorated with a
GUID attribute available in the System.Runtime.InteropServices namespace. This GUID value will be used to
uniquely identify a component which is going to provide events to the ETW infrastructure. This value is also
known as Provider ID in the ETW terminology.
2.
Next, a new protected readonly static field is to be added into each non-sealed instrumented .NET
component. For sealed classes, the private readonly static modifier needs to be used. This class member is
statically initialized with an instance of the framework component implementing the
IComponentTraceProvider interface which provides a set of tracing and instrumentation methods:
3.
Lastly, all important aspects of the custom component’s behavior need to be instrumented by calling the
relevant tracing methods provided by the CustomTraceManager static member shown above:
The benefit of the above technique is that it enables to collect only those events which are truly necessary to perform
troubleshooting of a specific component in the BizTalk solution, eliminate “noisy” events which may come from other
components, reduce the amount of events in the trace log and greatly reduce the time required to analyze the trace
log data.
Management of Instrumented Applications
After deployed and launched, an instrumented BizTalk application will be ready to emit a number of events related to
the application’s behavior, internal state, execution scope durations, runtime exceptions and everything else that the
application developers decided to report on through the code instrumentation.
The next section drills down into most common administrative tasks involved in managing the BizTalk solutions
instrumented using the ETW event trace infrastructure. Until now, the emphasis has been on a developer enriching
the application code with instrumentation and tracing. It’s time to look at the code instrumentation from an IT Pro
perspective and walk through the common management scenarios such as:



Extracting data out of the instrumented BizTalk applications into traditional human-readable trace log files;
Monitoring the event trace sessions, enabling and disabling event tracing;
Configuring the ETW event collectors.
Event Trace Management Application Landscape
The Event Tracing for Windows (ETW) infrastructure provides the ability to start and stop event tracing sessions,
monitor their status, configure a variety of logging settings such as buffer size, flush interval, stop conditions and
more. The majority of the administrative tasks are available from both the GUI and the command-line interface to suit
the needs and preferences of different IT persona types.
In addition to using the Reliability and Performance Monitor, application developers and technical teams have
access to other system tools and scripts that automate many aspects of event trace management:





Logman creates and manages event trace sessions and performance logs and supports many functions of
Performance Monitor from the command line;
WEvtUtil enables the retrieval of information about event logs and publishers, installs and uninstalls event
manifests, runs queries, exports/archives/clears logs;
TraceLog starts, stops, or enables trace logging;
TraceFmt formats and displays trace messages from an event trace log file or a real-time trace session;
TraceView configures and controls trace sessions and displays formatted trace messages from real-time
trace sessions and trace logs.
The custom instrumentation framework discussed in this whitepaper simplifies the use of the above tools by
providing 2 easy-to-use scripts with a minimum of command-line parameters. These scripts can be found in the
TracingTools folder and are intended to address the following requirements:


StartTrace.cmd provides the ability to start an ETW event trace session either for the entire instrumented
BizTalk solution, its selected component types or individually instrumented classes. This script acts as a
wrapper for the Tracelog tool from the Windows Resource Kit.
StopTrace.cmd is responsible for stopping the ETW event trace sessions and converting a binary trace log
file into human-readable format. This script is dependent on Tracelog and Tracefmt tools to flush trace data,
stop tracing and apply binary-to-text conversion.
The next subsections are intended to clarify how the above administrative scripts map to common management
scenarios.
Event Trace Management Tasks
The typical administrative tasks involved into managing the events produced by instrumented application code can
be summarized as follows.
Starting Event Trace Sessions

To start an event tracing session for the entire instrumented BizTalk application, run StartTrace.cmd using
one of following commands depending on the trace level required (All, High, Medium or Low). Note that the
log file name specified after -log parameter can be any name but should not contain whitespaces.
: All trace events (Info, Details, Warning, Error, In, Out, Start Scope, End Scope)
StartTrace -log BtsAppAllEvents -level all
: Limited trace events (Info, Warning, Error)
StartTrace -log BtsAppCoreEvents -level high
: Trace events indicating unexpected behavior (Warning, Error)
StartTrace -log BtsAppUnexpectedEvents -level medium
: Trace events related to runtime exceptions (Error)
StartTrace -log BtsAppExceptions -level low

To start an event tracing session for a specific type of application components, run StartTrace.cmd using
one of following commands depending on the component type and trace level required:
: Start trace for all instrumented pipeline components
StartTrace -log PipelineComponentsAllEvents -level all -component Pipeline
: Start trace for all instrumented orchestrations
StartTrace -log OrchestrationsAllEvents -level all -component Workflow
: Start trace for all instrumented data access components
StartTrace -log DataAccessComponentsAllEvents -level all -component DataAccess
: Start trace for all instrumented maps or custom transform code
StartTrace -log MapsAllEvents -level all -component Transform
: Start trace for all instrumented Web/WCF services or service proxies
StartTrace -log WCFServicesAllEvents -level all -component Service
: Start trace for all instrumented business rules
StartTrace -log BusinessRulesAllEvents -level all -component Rules
: Start trace for all instrumented BAM activities
StartTrace -log BAMActivitiesAllEvents -level all -component Tracking
: Start trace for all instrumented custom .NET components
StartTrace -log CustomComponentsAllEvents -level all -component Custom

To start an event tracing session for individually instrumented application components, the component’s
GUID attribute value must be provided in the command line as per the following examples:
: Start trace to capture all events for instrumented HL7Disassembler component
StartTrace -log HL7DasmFullTrace -level all -component Custom -guid 0EC4A54D-6B9747C1-9118-A2BF8B4E7595
: Start trace to capture errors in instrumented HL7Disassembler component
StartTrace -log HL7DasmErrorTrace -level low -component Custom -guid 0EC4A54D-6B9747C1-9118-A2BF8B4E7595
We recommend you consolidate multiple calls to StartTrace and StopTrace scripts into a single parameter-less batch
file in order to make it easier to start and stop tracing.
Monitoring Event Trace Sessions
After an event tracing session is started, it will remain running until it is manually stopped, a stop condition is
encountered or host is rebooted. To query the present status of the event tracing sessions, one of the following
approaches can be used.

To find out the event tracing session status from the command line, run the Logman utility using the
following syntax:
: Lists all event tracing sessions configured on the local machine
Logman query -ets
: Lists all event tracing sessions with a name matching the specified pattern
Logman query -ets | find "AllEvents"

To find out the event tracing session status from the GUI, open the Reliability and Performance Monitor, and
then navigate to the Event Trace Sessions section.
Stopping Event Trace Sessions
To stop a running event trace session, the target management tools differ depending on whether or not the trace
session must be temporarily suspended or terminated with no intent to resume. Consequently, one of the following
approaches can be used.

To completely stop an event tracing session from the command line, run StopTrace.cmd passing the
original name of the trace log without file extension:
: Stop trace for all instrumented pipeline components
StopTrace -log PipelineComponentsAllEvents
: Stop trace for all instrumented orchestrations
StopTrace -log OrchestrationsAllEvents
: Stop trace for all instrumented data access components
StopTrace -log DataAccessComponentsAllEvents
: Stop trace for all instrumented maps or custom transform code
StopTrace -log MapsAllEvents
: Stop trace for all instrumented Web/WCF services or service proxies
StopTrace -log WCFServicesAllEvents
: Stop trace for all instrumented business rules
StopTrace -log BusinessRulesAllEvents
: Stop trace for all instrumented BAM activities
StopTrace -log BAMActivitiesAllEvents
: Stop trace for all instrumented custom .NET components
StopTrace -log CustomComponentsAllEvents
Note that StopTrace.cmd will attempt to automatically convert the binary trace log into text-based format
using the Tracefmt tool. Depending on the log size and disk IO performance, this operation can take several
minutes.

To temporarily suspend an event tracing session from the command line without producing a text-based log
file, run the Logman utility using the following syntax:
: Stop event tracing sessions all instrumented pipeline components
: Do not convert log into text file at this moment in time
Logman stop PipelineComponentsAllEvents -ets

To temporarily suspend the event tracing session from the GUI, open the Reliability and Performance
Monitor, navigate to the Event Trace Sessions section, right-click on the target event tracing session, and
then select Stop from the context menu:
Configuring Event Trace Sessions
The default configuration of the ETW event trace sessions created by StartTrace.cmd can be attributed as follows:





Buffer Size: 128K
Maximum Buffers: 100
Log Mode: Circular
Maximum Log Size: 1000MB
Flush Time: Not Set



Clock Type: Performance
Stream Mode: File
Pre-allocate File Space: No
Should the default configuration be found insufficient and need modifications, the script can be updated to include
the desired configuration settings. The new settings must be specified in the following line in StartTrace.cmd:
"%TraceLogTool%" -cir 1000 -b 128 -max 100 -start %TraceLogName% -flags %TraceLevel% -f
%TraceLogFileName% -guid #%TraceComponentGUID%
Below are some of the command line parameters supported by the Tracelog tool that may be useful when
customizing the event tracing configuration:
-b
<n>
-min <n>
-max <n>
-f <name>
-append
-prealloc
-seq <n>
-cir <n>
-newfile <n>
-ft <n>
-paged
-rt
-kd
Sets buffer size to <n> Kbytes
Sets minimum buffers
Sets maximum buffers
Log to file <name>
Append to file
Pre-allocate
Sequential logfile of up to n Mbytes
Circular logfile of n Mbytes
Log to a new file after every n Mbytes
Set flush timer to n seconds
Use pageable memory for buffers
Enable tracing in real time mode
Enable tracing in kernel debugger
Combining Event Tracing With Real-Time Monitoring
Traditionally, BizTalk developers needed the ability to monitor the application's behaviour in real-time. Whether it’s
ad-hoc debugging or functional testing, having the instrumented code producing events that can be analysed
instantly is crucial.
The previous sections were focused on collecting tracing data mostly for offline analysis, meaning that the trace
events are captured silently into a trace log and are not visible until the event trace session is stopped and the trace
log file is converted into human-readable format. This scenario is well suited for production environments or use
cases when tracing data analysis is deferred until later stage.
Fortunately, the ETW tracing infrastructure and supporting tools provide full flexibility and help address the real-time
monitoring requirements by allowing projecting the trace events through a number of ETW consumers. Specifically,
the TraceFmt utility is capable of streaming trace events via Win32 Debugging APIs as events are being emitted by
the instrumented code. This makes it possible to leverage the familiar tools such as DebugView to monitor ETWinstrumented applications in real-time, provided it’s strictly for debugging or functional testing purposes.
In order to enable real-time monitoring, we recommend taking advantage of the BizTalk CAT Instrumentation
Framework Controller tool. The tool helps eliminate the need for dealing with multiple command-line utilities and
memorizing their parameters. It also saves time on starting, stopping and customizing tracing sessions. It can quickly
enable real-time tracing to DebugView (or other Win32 debug event listeners), to a trace log file or to both at the
same time for additional flexibility.
The Controller can be used both on development machines and production servers.
In the development environments, the tool addresses the needs of many BizTalk developers who are accustomed to
using the diagnostic APIs in .NET Framework. In combination with the DebugView tool, it provides the ability to see
diagnostic messages in real-time.
In the production environments, the Controller allows to easily manage tracing sessions by enabling tracing on a
production server with only a negligible impact on performance (when tracing to a file), disabling active tracing
sessions, adjusting filter options and auto-converting binary trace logs into text files.
Behind the scene, the Controller simply invokes the respective command-line tools with relevant parameters. As a
result, it helps reduce the amount of time spent on figuring out and dealing with the command-line syntax.
Conclusion
The traditional ways of instrumenting BizTalk solutions may not always be the most effective from a performance
standpoint. The commonly used instrumentation and tracing components leveraging the Win32 Debugging APIs may
introduce a potential bottleneck and become responsible for performance degradations in multi-threaded BizTalk
applications running under stress.
From the other side, source code instrumentation delivers a great degree of visibility into the application behavior
and helps reduce the overall troubleshooting efforts. Consequently, a fundamentally new approach to instrumenting
high performance BizTalk solutions has become crucially important to enable collecting the rich and detailed
diagnostic information in a non-intrusive manner with virtually no overhead and no impact on the application
performance.
The Windows Server AppFabric Customer Advisory Team at Microsoft aimed to provide the community with validated
best practices to help BizTalk developers enrich their solutions with the high-performance instrumentation internally
adopted by many Microsoft products. These best practices have been reflected in the form of a reusable framework
which BizTalk developers can easily plug in and adopt in their own implementations.
The source code containing the instrumentation framework discussed in this whitepaper can be found on the MSDN
Code Gallery via the following hyperlink:
Download: BizTalk Solution Instrumentation Framework.zip
Note that all source code files are governed by the Microsoft Public License (Ms-PL) as explained in the
corresponding legal notices.
Additional Resources/References
For more information on the related topic, please visit the following resources:







“Improve Debugging And Performance Tuning With ETW” article in the MSDN Magazine.
“Intro to Event Tracing For Windows” post on Matt Pietrek’s blog.
“How To Use Event Tracing For Windows For Performance Analysis” presentation available from the MSDN
Download Center.
“Logman Utility Command Line Reference” on TechNet.
“Tracelog Utility Command Line Reference” article in the MSDN Library.
“Controlling Event Tracing Sessions” article in the MSDN Library.
“Tools for Software Tracing” article in the MSDN Library.