Instrumentation
Software Profiling
Software Profiling
●
●
Instrumentation of a program so that data related to
runtime performance (e.g execution time, memory
usage) is gathered for one or more pieces of the code
during the program's execution
Used to analyze and improve a program's
performance
Software Profiling
●
Typically questions answered using profiling
➔
➔
➔
➔
What lines of code are responsible for the bulk of
execution time?
How many times is this looping construct executed?
How many times a method is called and what % of
total program run time it consumes?
Which approach to coding a block of logic is more
efficient?
Eclipse TPTP
●
TPTP = “Test and Performance Tools Platform” (http://www.eclipse.org/tptp)
●
Part of the open-source Eclipse project
– Project created in August 2004
– Successor of the Eclipse “Hyades” tools sub-project (2002)
– Involves several participants: Compuware, IBM, Intel, OC Systems,
SAP, Scapa Technologies
●
Offers tool to create tools to automate software Quality Control
●
Four major components / sub-projects:
●
●
Platform
●
Monitoring tools
●
Testing tools
●
Tracing and Profiling tools
Most recent release: TPTP 4.6.1 (Sept. 2009)
TPTP Objectives
●
Bring software test and performance tools into the
Eclipse environment
●
●
●
By providing a platform and a set of components to
develop testing, profiling and monitoring tools for
applications.
In a consistent way that maximizes integration with tools
used in the other processes of the software lifecycle.
Addresses the entire test and performance life
cycle, from early testing to production application
monitoring, including test editing and execution,
monitoring, tracing and profiling, and log analysis
capabilities
TPTP Objectives
●
●
●
Reduce the cost and complexity of
implementing effective automated software
quality control processes
Support a broad spectrum of computing
systems including embedded, standalone,
enterprise, and high-performance
Share data through an OMG-defined test
profile model, Common Base Event (CBE)
Model and trace model implemented via the
Eclipse Modeling Framework (EMF)
TPTP Project Structure
●
TPTP Platform Project
●
●
●
Core framework upon which the development of the
monitoring, testing and tracing and profiling tools relies.
Common user interface, standard data models, data
collection and communications control, as well as remote
execution environments
TPTP Monitoring Tools Project
●
●
Monitoring and logging phases of the application lifecycle,
collecting and analyzing system and application resources.
– Correlate disparate logs from multiple points across an
environment.
Monitoring of application servers and system performance
(e.g. CPU and memory utilization).
TPTP Project Structure
●
TPTP Testing Tools Project
●
●
●
●
Addresses testing phase of the application
lifecycle.
Contains testing editors, deployment and execution
of tests, execution environments and associated
execution history analysis and reporting.
Includes exemplary tools for JUnit based
component testing tool, Web application
performance testing tool, and a manual testing tool.
Future TTCN-3 support ?
TPTP Project Structure
●
TPTP Tracing and Profiling Tools Project
●
●
●
Addresses the tracing and profiling phases of the
application lifecycle.
Includes exemplary profiling tools for both singlesystem and distributed Java applications through a
JVMPI monitoring agent that collects trace and
profile data.
Includes a generic tool kit for probe insertion.
TPTP Architecture
Presentation System
Target System
TPTP Workbench
Controller
Interface
Controller
Interface
Test
Trace
Monitor
Platform
Distributed
Control
Framework
Agent Controller
Application
Standard Widgets
Trace Collection
EMF Data Models
Log Collection
Data Collection
Interface
Data Loader
Test
Log
Trace
Statistical
Eclipse Platform
Distributed
Data Collection
Framework
Data Collection
Reference Perspectives
Execution
Environment
JVMPI
JVMTI
Performance
Monitor
Test Execution
TPTP Architectural Components
(1)
●
On the controller system:
●
User interface: the graphical view from which test execution is
directed.
●
Data models: based on the Eclipse Modeling Framework (EMF)
– Includes models for:
●
test execution data
●
logs
●
traces
●
statistics
●
Eclipse platform: used for windows, editors, etc.
●
Interfaces to the agent controller, and data collection engine.
TPTP Architectural Components
(2)
●
On the target system:
●
the application to be tested
●
the application’s execution environment:
–
●
●
includes the Java virtual machine and additional
monitoring via virtual machine tool interfaces
the test engine, to start the application and its
environment
data collection
Models implemented by TPTP (1)
●
●
Includes a reference implementation of the UML2
test profile (MOF).
Test profile: definition model for the creation of:
●
test suites
●
test cases
●
data pools
●
invocation
●
loops
●
synchronization
Models implemented by TPTP (2)
●
●
Definition model for the definition, creation, management and
persisting of test executions over time including:
●
Tests to be executed.
●
Deployments.
●
Locations.
●
Verdicts.
●
Messages and console output from the test execution.
A collection of test execution traces and results: the
test log.
Agent Controller
●
●
●
A daemon process that resides on each deployment host and enables
client applications to launch host processes and interact with agents
that coexist within host processes
Contains a server that has the capability to launch and manage local
or remote applications from a local TPTP workbench
The Java profiler is a type of agent managed by AC that runs in the
Java Virtual Machine process and receives notifications of JVM
events.
●
The Java profiling engine included uses the Java Virtual Machine
Tool Interface to profile local or remote Java applications from a
local TPTP workbench
●
●
Launch and profile local or remote Java applications and import
local or remote logs
Platform support : Windows, Linux, Solaris, HP-UX, AIX, Z/OS,
OS/400
Agent Controller
Eclipse
Workbench
Agent
Controller
Java
Profiling
Agent
Host
process
Agent Controller Architecture
Image © 2006 IBM, made available under the EPL
Agent Controller Terminology
●
●
●
●
Host process
●
The process that contains the application under test
Agent
●
A reusable binary file that provides services to the host process, and more
importantly, provides a portal by which application data can be forwarded to
attached clients
Client
●
A local or remote application (e.g. Eclipse Workbench) that is the terminal
destination of host process data that is externalized by an agent
The Agent Controller
●
A daemon process that resides on each deployment host and provides the
mechanism by which client applications can either launch new host processes, or
attach to agents that coexist within existing host processes. The client can reside
on the same host as the Agent Controller, or it can be remote. The Agent Controller
can only interact with processes on the same node, and it is required on the same
machine the targeted JVM is
Virtual Machine (VM) Interface
• JVMTI, the Java Virtual Machine Tool Interface
●
●
●
http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/index.html
Provides interface for tools that monitor or control the virtual machine:
debuggers, profilers, thread monitors, coverage analysis
Capabilities: observation / control of
●
memory
●
threads
●
stack frame
●
memory heap
●
local variables
●
break points
●
classes and object instances; methods
●
timers
Versions of VM interfaces and
the Agent Controller
●
In versions of Java up to 1.4.2, there were two
interfaces called JVMPI and JVMDI, where
P = “profiling” and D = “debugging”
●
●
These were deprecated in Java 5 and removed in
Java 6.
Both JVMPI and JVMDI have been replaced
by JVMTI.
TPTP Java Profiling Tool
●
●
●
●
●
Broadly useful for performance analysis and for gaining a deeper
understanding of a Java program
Consists of the Profiling and Logging Perspective and a number of
graphical and tabular views
Enables to profile and interact with applications, to work with profiling
resources, and to examine applications for performance and memory
usage problems
Help to visualize and understand program execution, pinpoint the
operations that take the most resources, as well as to explore patterns
of program behavior
Enables to test application's performance early in the programming
development cycle for improvements
TPTP Java Profiling Tool
●
●
A library that attaches to a JVM to capture and record the Java application's
behavior
Is a type of agent managed by the Agent Controller
●
Runs in the JVM (Java Virtual Machine) process and receives notifications of
JVM events, based on the JVMTI (Java Virtual Machine Tool Interface).
Best used to identify performance details such as classes or methods
responsible for the poor execution performance, also be used to analyze
application heap and find memory leaks
●
Output from the profiling agent is in the form of XML fragments (XML4Profiling)
●
●
●
Can be launched from the TPTP workbench, Applications can be in
workbench's workspace, binaries that are on the file system or in a server
Can also be invoked using the -Xrun JVM option in command line
Profiling and Logging Perspective
●
●
The profiling tools available in the Profiling and Logging perspective
provide comprehensive information about the performance of an
application
The profiling tool (in this scenario i.e. – Java Profiling) provides
information pertaining to
●
JVM performance
●
Object allocations and references
●
Garbage collection
●
Object methods performance
●
Object->Object interactions
●
Thread interactions
to name a few
Monitor and Navigator View
Profiling Views
Monitor and Navigation View
Java process
associated
with a host
Time of
profile run
Monitors
Monitor and Navigation View
●
●
Profiling and Logging perspective provides resources to administering
and managing profiling
Profiling resources
●
Project: Make a project of your profiling effort
●
Monitor: Aggregate different processes and agents
●
Host: The host you are profiling
●
Process: The executing program
●
●
Agent : Provides services to a process, a mechanism by which
process data can be sent to (attached) clients
Profiling Type: Group profile data collection
Monitor and Navigation View
●
●
●
●
Project:
●
Container of the profiling resources
Monitor:
●
A logical container for the profiling information that is collected from a
group of agents. The views at the monitor level show data from these
agents. Monitors are useful for aggregating processes and agents from a
distributed application
Host:
●
Owns the processes that are profiled. A host runs processes. You can
specify a host either by its name or by its IP address
Process & Agent:
●
In the Agent Controller architecture model, an agent is a binary file that
provides services to the host process by which application data can be
transferred to attached clients
Profiling Views
●
Number of views to visualize and organize profiling
data
●
Memory Statistic view
●
Execution Statistic view
●
Coverage Statistic view
●
Object Reference view
●
Execution Flow view and table
●
Method Invocation view and table
●
UML2 Trace Interactions view
Memory Statistics View
Shows:
Total instances: total number of instances of each class.
●
Live instances: The number of instances that are alive (i.e. instances
that were not collected by the garbage collector).
●
Total Size (bytes): The size of an instances associated with a specific
class
●
Active Size (bytes): The size of an instances associated with a specific
class
●
Execution Statistics View
Execution Statistics View
●
Shows base time, average time, cumulative, and number
of method calls:
●
●
●
●
Base Time: The amount of time (in seconds) a method has
taken to execute. Not including the execution time of any other
methods called from this method.
Average base time: The average base time required to execute
a method once.
Cumulative base time: The amount of time (in seconds) a
method took to execute. Including the execution time of any
other methods called from this method.
Calls: The number of times a method was invoked.
UML2 Execution Flow
Thread Analysis
Thread Analysis
Probe kit Instrumentation
●
●
●
●
Tool in the Test and Performance Tools Platform project
Scriptable Byte Code Instrumentation (BCI) engine for Java
instrumentation.
Probes contain fragments of Java code to be called from classes and
at times specified by the programmer including:
●
Method entry
●
Method exit
●
Exception handling
●
Class loading
Probes can be used to collect runtime data, including:
●
Method arguments and the “this” object
●
Exception objects
●
Return values
Probe kit Instrumentation
●
Runtime data collected by probes can be used for :
●
Profiling
–
–
●
Debugging
–
●
Send any numeric data for graphing: heap size, CPU load,
anything-per-second rates
Report when a method takes more than X amount of time to
run.
target program runs at nearly full speed even while evaluating
conditions, unlike running under the debugger.
Code analysis
–
Send any kind of call-and-return, parent/child information
Create Probe Source
Fragment Types
●
entry
●
●
exit
●
●
run upon method entry. Will not run for methods that were
inserted into the class by Probekit.
run upon method exit: a normal exit, when the method throws
an exception, or when a thrown exception propagates out of
the method. Will not run for methods that were inserted into
the class by Probekit.
catch
●
run at the beginning of a catch clause in the method, or at
the beginning of a finally clause that runs as the result of an
exception.
Fragment Types
●
staticInitializer
●
●
executableUnit
●
●
run before every executable unit of code in methods that match the probe's
target and filter specification, and for which source code is available. If the
method does not have source line information, it will appear to have a single
executable unit, numbered zero..
beforecall
●
●
run inside the class initializer of every probed class. If the class does not already
have a static initializer, one will be created.
run in the calling method immediately before the target method is called. Not
valid for method probes.
aftercall
●
run in the calling method immediately after the target method exits: either a
normal exit, or when the target method throws an exception. Not valid for
method probes.
Creating Probes
Apply Probe
Probe results