Eclipse* TPTP Project
Apache* Harmony Project
A Harmonious Combo :
Experiences Profiling
with Eclipse* TPTP
*
Chris Elford -- Intel Corporation
Sergey Kuksenko -- Intel Corporation
Confidential | Date | Other Information, if necessary
Harmony logo source: http://harmony.apache.org/
© 2008 Intel Corporation; made available under the EPL v1.0 | March 2008
© 2002 IBM Corporation
Eclipse* TPTP Project
Apache* Harmony Project
Outline
 Overall project goals
 Introduction to Apache* Harmony
 The Eclipse* TPTP profiler
 First experiences using TPTP
 Analysis of advanced workload with TPTP
 Looking to the future
2
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
© 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Goals and Methodology of our Experiments
 Goals
 Identify Harmony performance opportunities using TPTP
 Integrate TPTP into our performance methodology
 Approach
 Start simple…
 Verify that we can recognize known performance issues w/ TPTP
 Understand how opportunities manifest in TPTP
 Understand current limitations of TPTP and Harmony JVMTI
 Fix critical issues along the way
 Grow…
 Move forward looking for new opportunities to optimize Harmony
 Use TPTP more regularly
 Suggest TPTP & Harmony JVMTI enhancements for our use case
Most of what we show today is in the “Start Simple” category
3
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
© 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Apache* Harmony Overview
Primary Goal of Harmony
 “a large and healthy community of those interested in runtime
platforms”
Creation of
 “A compatible, independent implementation of the Java SE 5
JDK under the Apache License v2”
 “A community-developed modular runtime (VM and class
library) architecture.”
Why?
 Enables innovation/research and adoption
 w/o license restrictions that limit consumption
4
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
© 2008 Intel Corporation; made available under the EPL v1.0
Source: http://harmony.apache.org/
Eclipse* TPTP Project
Apache* Harmony Project
Apache* Harmony Overview (continued)
 Why J2SE 5?
 “Starting with Java SE 5, as that is the first version of Java SE for
which the licensing allows an open source implementation”
 “Continue with Java SE 6 and any subsequent versions that follow.”
 Who all is using Apache* Harmony technology?
 Since its an open license, we don’t know for sure… but to name a few
 http://mail-archives.apache.org/mod_mbox/harmonydev/200712.mbox/%3c4759E2CD.5010308@gmail.com%3e
 http://mail-archives.apache.org/mod_mbox/harmonydev/200711.mbox/%3c8084e12c0711201546m7e3cc028kf5df2763df708fd3@
mail.gmail.com%3e
Harmony is a clean room implementation of J2SE spec
distributed under Apache* V2 license
5
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
Source: http://harmony.apache.org/faq
© 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
A Brief History of Apache* Harmony
May 2005: Accepted by the ASF and started in Apache* Incubator
May 2006: Harmony has VM and classlib functional to run Eclipse*
3.0
May 2006: JavaOne - demonstrated functional classlib with
Swing/AWT
July 2006: First JRE Snapshot Published
Oct 2006: Became ASF Top Level Project
May 2007: First Milestone Build Released (M1)
May 2007: JavaOne – “In Harmony with Eclipse” bundle CDs
distributed
…
Dec 2007: Fourth Milestone Build Released (M4)
Feb 2008: Fifth Milestone Build Released (M5)
Note: We plan to run Eclipse on Harmony in the demo later in talk
6
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
© 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Apache* Harmony: What’s Inside?
 ~2.3 million LOC
 ~1.6 million Java* code, ~ 0.7 million C/C++
 Serious testing and engineering infrastructure
 Components
 API: ~99% JDK5, ~90% JDK6
 Project VMs: DRLVM (and others)
 Tools: javac, javah, javap, jarsigner, keytool, appletviewer
 Platform Support
 Systems with x86 processors supporting Intel® SSE or greater
 Systems using processors compatible with Intel® EM64T
 Intel® Itanium®-based systems
 Operating System Support
 Several versions of Windows* and Linux
7
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
© 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Apache* Harmony and Eclipse*: Synergy
 The Eclipse* platform was the first large application
enabled on Harmony back in ’06
 Harmony test base includes >20 Eclipse* scenarios
 Harmony performance has been tuned for Eclipse* platform
 Harmony adopted EUT for daily execution
 Harmony JVMTI has proven to work with TPTP
 Harmony adopted key TPTP JVMTI tests for daily execution
 “In Harmony with Eclipse” bundles updated regularly
 http://sourceforge.net/projects/eclipse-harmony/
8
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
© 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Eclipse* TPTP Profiling Tool Overview
 Broadly useful for performance analysis and for gaining a
deeper understanding of a Java* program
 Consists of the Profiling and Logging Perspective and a
number of graphical and tabular views
 Visualize program execution and threading behavior;
Pinpoints operations taking most resources; Explore patterns
of program behavior
 Enables you to test your application's performance early in
the programming development cycle for improvements
 Option to run application with agent at near full speed
(enabled mode) and attach later at certain application phase.
 Assorted filtering functionality which can help to localize
problem and reduce overhead for large long run applications
9
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
© 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Obligatory Eclipse* TPTP Architecture Diagram
Presentation System
Target System
Distributed
Control
Framework
Agent
Control Interface
Artifact
Management
Test Creation
and Execution
Trace Analysis
And Profiling
Runtime
Monitor / log
Agent
Control Interface
Eclipse TPTP GUI
Reference Perspectives
And Workflow
EMF Data Models
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
10 © 2008 Intel Corporation; made available under the EPL v1.0
Trace Collection
Log Collection
Execution
Environment
JVMPI Monitor
Correlation
Eclipse Platform
Application
Injection
Distributed
Data Collection
Framework
Data Collection
Interface
Data Loader
Statistical
Log
Test
XMI
Trace
Real Time
Export
Testability Interface
Data Collection
Standard Widgets and
Core Plug-ins
Test Engine
System Performance
Monitor
JVMTI Monitor
Eclipse* TPTP Project
Apache* Harmony Project
The New Java* Profiler (JVMTI) vs. JVMPI
 Java* 5.0 introduced new standards for profiling
 Prior to Java* 5.0 (Java* 1.4-) the standards and the interfaces
to support profiling (JVMPI) were experimental
 Java* J2SE 6 eliminated JVMPI interface
 The new standard (JVMTI): an innovative solution to profiling;
enables precise control over application parts to profile
 Eclipse* TPTP has embraced this innovation and the new Java
profiler is based on this new standard
 In Eclipse* TPTP old Java profiler (JVMPI) still exists
 Only for backward compatibility with Java 1.4
 Because of JVMPI/JVMTI interface differences there are some
differences in behavior between old and new profiler
 Eclipse* TPTP will eventually remove Old Java* profiler
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
11 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Eclipse* TPTP Profiler Updates for Ganymede
 Improve overall usability via assorted fixes
 Using feedback from users
 Reduce usability diffs between JVMPI and JVMTI…
 e.g., SSL for secure communications
 e.g., Reduce overuse of dynamic attach/detach
 Enhancements
 e.g., Better multithread analysis
 Improved visualization
 Contended lock analysis
 Track more thread states (e.g., join points)
 e.g., Efficient binary transfer format
 e.g., Better Java* 6 support
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
12 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Motivation of using TPTP on Harmony
When you have 1.6 million lines of Java* code…
Using TPTP for Harmony performance analysis:
startup optimizations.
 Team already uses profiling tools extensively
Intel® VTune™ tools heavily used
Correlation of architectural events to assembly code
VM and JIT code quality analysis and great optimization
Mainly use batch mode (execute then perform offline analysis)
Want to also do higher level Java* language centered analysis
Rich Java* centered analysis of call graph
Package and Class and Method summarization
Java* Thread and Heap analysis
Ideally continue with batch mode analysis model
 Why TPTP Profiler?
Leverage open source to create open source
Maximize collaboration potential w/ engineer next door
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
13 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Harmony analysis with TPTP: Startup performance
 Consider simplest workloads
 Hello World
 Startup phase of a GUI application
 Goal
 Understand detailed end to end behavior
 Find inefficiencies in class libraries for optimization
 Approach
 Disable default filters in TPTP
 Collect trace
 Analyze
 Discoveries from our first use of TPTP
 Even for small apps, data volume can be huge w/o filters
 Several TPTP usability bugs; fixes targeted for Ganymede
 Xml4profiling lacks command line tool for tsv/csv dumps
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
14 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
A few words on filtering in TPTP
 Collection Time Filtering
 Pro: Reduce data volume and instrumentation overhead (critical)
 Con: Data for excluded classes/method can never be seen
 Model Import Filtering
 Pro: Reduce workbench memory footprint (critical on 32bit OSes)
 Con: Can only see data that is in the model
 View Level Filtering
 Pro: Remove items from being visualized in the workbench
 Con: Does not reduce data volume or model footprint
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
15 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Harmony performance analysis w/ TPTP: HWA startup.
Sort by number
of calls
write(int) is at the
top
Callgraph:
Too
much time is spent when reading the manifestSort
ofby
signed
base jar file...
java.util.jar.InitManifest:.read()->java.io.ByteArrayOutputStream:.write(int)
java.util.jar.IninManifest:.read()
at the top
time
5% boost expected from moving key signed jar to bootclasspath.
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
http://issues.apache.org/jira/browse/HARMONY-5277
16 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Harmony perf analysis w/ TPTP: EIOffice startup.
See http://sourceforge.net/projects/eio-harmony/
Java.lang.reflect.Method.getName()
is at the top
Sort by number
of calls
further investigation shows that Method:.getName() is used in
linear search for finding methods of class
from java.lang.Class.getMethod(String)
4% boost expected by replacing linear search w/ hashed search in lookup
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
http://issues.apache.org/jira/browse/HARMONY-5284
17 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
More advanced study: SPECjbb* 2005
 Analyze more complex, long running, workload
 Snapshot performance during workload steady-state
 We use SPECjbb* 2005 workload for some of our performance tests
 Goal
 Understand application behavior
 Find inefficiencies in class libraries for optimization
 Ideal Approach
 Disable default filters in TPTP, Collect trace, Analyze
 Would prefer to collect trcxml from our batch scripts then do offline analysis
 Discoveries from our use of TPTP
 Can’t get sufficient detail with filtering; w/o filtering; too much data
 Pause/resume helps… But doesn’t help allow batch mode execution
 Hope for richer control with Application mode in Ganymede
 Avoid attach mode
 Instrumentation during run perturbs performance
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
18 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Performance study: SPECjbb* 2005
Overall Execution statistics
spec.org
class
3rd party library
classes
Apply filter
• SPECjbb* 2005 source cannot be modified
• 3rd party library
classes
hard to influence
(view
level filter)
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
19 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Performance study: SPECjbb* 2005
Sort by Base Time
Candidates for
investigation
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
20 © 2008 Intel Corporation; made available under the EPL v1.0
Cumulative Time
Eclipse* TPTP Project
Apache* Harmony Project
Performance study: SPECjbb* 2005
First candidate
class
Hot methods
• Next() and nextInt() are rather small.
• Typically disappear in JIT inlining
• JIT behavior invisible to TPTP
• Class instrumentation impacts behavior
• Message: be careful with getters/setters/etc
• TPTP can be set to automatically filter them
• We need to see them so we can double check JIT
• No opportunity for tuning... Move along.
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
21 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Performance study: SPECjbb* 2005
TreeMap iteration is
expensive
next() at the top
Look into details
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
22 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Performance study: SPECjbb* 2005
next() is hot
next() invokes get()
pretty much each time
Problem!
• Single generalized iterator impl. (e.g.,
KeyIterator, ValueIterator,
EntryIterator)
Also…
• Ignores
common cases for which
• Creates isexcess
“getter” objects.
optimization
possible
• Inlining complicated due to
interface call
5% boost from implementing three specialized iterators for
common use cases
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
http://issues.apache.org/jira/browse/HARMONY-5232
23 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Performance study: SPECjbb* 2005
Next hot classes:
java.math.*
Choose hot method
for details
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
24 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Performance study: SPECjbb* 2005 – java.math.* (~10%)
Short Quiz:
Each BigDecimal.add() invokes BigInteger.add()
java.math.* took ~10%?
How did we get more then 30%?
Do we
really need toiscreate
separate BigInteger
Only 1 object
(BigDecimal)
created
for each
BigDecimalBigInteger,
value?
• Instead ofobject
3isobjects
(BigDecimal,
int[])
A BigDecimal
a BigInteger
plus a scaling
factor
Let’s incorporate small values (<19 digits)
• Less GC• stress!
directly into BigDecimal.
Result: More then a 30% boost!!!
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
http://issues.apache.org/jira/browse/HARMONY-551
25 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
What to do next to analyze Harmony with TPTP?
So now that we’ve tried TPTP, what is next?
 Use Eclipse TPTP profiler often to improve Harmony
 Work closely with Eclipse* TPTP developers
 So they better understand how we use TPTP
 Use heap profiler
 Harmony’s memory footprint is fairly large
 Target smaller heaps (e.g., for J2SE on smaller devices)
 Already touch tested it
 Use thread profiler
 Particularly excited about optimizing multithreaded Java for
multicore platforms
 Already touch tested it
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
26 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Summarizing wish lists for TPTP and Harmony
Features we would like to see in TPTP in future
 Better handling of huge datasets (multi-GB)
 Simpler and richer batch (scripted) collection
 Command line data reduction utility (offline analysis)
 Interface for filtering “easily” inlinable getter/setter methods at
collection time
Features we would like to see in Harmony in future
 Fuller JVMTI implementation
 e.g., RedefineClasses - needed for TPTP dynamic attach/detach
 Better memory footprint on large Eclipse* scenarios
 Further tuned Eclipse* workbench startup time
 Java* J2SE6 support
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
27 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Conclusion
Our perspective
 Apache* Harmony making great progress
 Eclipse* TPTP allows great Java* code analysis
 Both projects have opportunities for further enhancement
 Look forward to future collaboration between Harmony and TPTP
What we would like from you
 Go try Harmony (w/ Eclipse* platform)
 Tell us about your experience (and feel free to join)
 Go try Eclipse* TPTP profiler
 Tell us about your experience (and feel free to join)
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
28 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
TPTP Resources (profiling and beyond)
 Learn and Try
http://www.eclipse.org/tptp/home/documents/conferences/eclipseCon2007
http://www.eclipse.org/tptp/home/documents/conferences/eclipseCon2007
/%283669%29%20Profiling%20Java%20applications%20using%20Eclips
e%20TPTP%20v1_1.htm
 Webs and Wikis
http://eclipse.org/TPTP
http://wiki.eclipse.org/TPTP
 Downloads and Updates
http://www.eclipse.org/tptp/home/downloads
http://www.eclipse.org/tptp/home/downloads/updateManager.php
 News and Mail
http://www.eclipse.org/tptp/home/project_info/general/mailnews.php
 Use and Participate
http://wiki.eclipse.org/TPTP_User_Experiences_Profiling
http://www.eclipse.org/tptp/home/project_info/general
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
29 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Harmony Resources
 Learn and Try
http://harmony.apache.org/hdk.html
http://harmony.apache.org/subcomponents/drlvm/index.html
 Webs and Wikis
http://harmony.apache.org/
http://wiki.apache.org/harmony/
 Downloads and Updates
http://harmony.apache.org/download.cgi
http://sourceforge.net/projects/eclipse-harmony/
 Use and Participate
http://harmony.apache.org/quickhelp_contributors.html
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
30 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Intel, the Intel logo, Itanium and VTune are trademarks or
registered trademarks of Intel Corporation or its subsidiaries
in the United States and other countries.
* Other names and brands may be claimed as the property of
others.
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
31 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Questions?
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
32 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
BACKUP
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
33 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Eclipse* TPTP -- The New Java* Profiler
 A set of libraries that attach to a JVM for recording Java application's
behavior
 An extensible framework, consisting of a core runtime component
(Martini), an agent managed by the Agent Controller (JPIAgent), and a
set of data collection libraries built on top of the Martini runtime.
 Can be used to identify performance details such as classes or methods
responsible for execution bottlenecks, analyze application heap to find
memory leaks and visualize threading behavior.
 Output in the form of XML fragments (XML4Profiling)
 Can be launched from the Eclipse IDE or as a standalone program using
Java command-line options
 Applications under test can reside in Eclipse workspace, binaries on file
system, or hosted in a J2EE application server.
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
34 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Apache* Harmony Status: Overall
 API completeness: ~99%
 Remaining 1% is in Swing as well as spread over all code
 Non conformant – Sun* is silent on TCK issue.
 Compatibility is proven by a number of free and commercial apps
running on Harmony
 VM completeness: ~100% (according to the spec)
 Interpreter, 2 JITs, DPGO
 Optimized JIT is based on StarJIT – initially tuned to show best
performance on Intel® Itanium® processors – has a good large enough
set of HLOs
 Parallel, generational GC with innovative algorithms.
 Two commercial-quality VMs are working with Harmony classlibs:
 IBM* J9*
 BEA* JRockit*
 Performance is good on server benchmarks, client still needs
some attention.
 Large set of test suites: Unit, VTS (conformance), reliability, stress,
performance, etc.
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
35 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
How we collected a snapshot w/o filters
1. Configure profiling to save to a file

Destination tab in workbench
2. Configure aggregated stats with 3600 second refresh

Avoid auto refresh… Only manual
3. Start the run and immediately pause collection

4.
5.
6.
7.
8.
9.
Pause button… Could not find a way to start paused…
Wait until steady state reached
Resume collection
Wait a few minutes
Pause collection
Wait for data to come across
Terminate

Do not save
10. Restart Eclipse

Ensures maximum memory available for file import
11. Import from where you saved file in (1).
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
36 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Heap profile: a first try
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
37 © 2008 Intel Corporation; made available under the EPL v1.0
Eclipse* TPTP Project
Apache* Harmony Project
Thread profile: a first try
A Harmonious Combo: Experiences Profiling Apache* Harmony with Eclipse* TPTP
38 © 2008 Intel Corporation; made available under the EPL v1.0