Memory issues in
production systems
Production system


Restricted access

Application, DB, Application server, log files

Debugging, monitoring
Investigation should not affect users

Reproducing memory issue is not the best idea
Memory leaks


What is memory leak

java.lang.OutOfMemory: heap size

other symptoms - extensive garbage collection
Hard to discover


Hard to investigate


Users don't recognize it
Few information
Sometimes hard to fix

Design issues
Investigation of memory issues

Log files


Monitoring behavior of running application


application logs, access logs, gc logs
JConsole
Heap dump Analysis

JHAT, MAT, Netbeans
Log files


Motivation

find issues

find patterns leading to issues
Problems



combining different logs (access log, application
log)
bad visibility of root cause – not visible or hidden in
big amount of minor issues
memory problems trigger only few types of
messages
Investigating log files

Application logs


Access logs


Log free memory once per minute
Analysis of traffic and business use cases
GC logs

Show problems with GC (stop the world)
Monitoring via JMX

Tools



JConsole
Motivation

More information than in logs

Monitoring tests
Problems


Not always applicable on production system
(security reasons)
Test might not cover all scenarios
Heap dump analysis

Tools


Motivation



JMAP, JHAT, MAT, Netbeans
Offline investigation of behavior during specific time
frame
Comparison of different heap dumps
Problems

Access to heap dumps (security)
Creating Heap dump


jmap -dump:live,format=b,file=heap.dump <pid>

Part of JDK

Used to create heap dump on demand
JVM parameters



-XX:+HeapDumpOnOutOfMemoryError XX:HeapDumpPath=/directory/subdirectory
Produces file java_pid<pid>.hprof
Used for production system, creates heap dump when
OutOfMemory is thrown
Tools for heap dump analysis


JHAT

Part of JDK 6

Running as web application
IDE


Netbeans, MAT (Eclipse plugin)
Commercial tools
Heap dump analysis - JHAT

Heap Histogram


List of classes with number of instances and used
memory
Referrers vs Referees

Navigating in object graph

Example: Form → String → [c
Heap dump analysis – Netbeans,
MAT

Dominator tree


Shallow vs Retained heap




Object x dominates object y if every path in object
graph from root to y goes through x
Shallow heap is memory used by object
Retained set of X is set of objects which would be
removed by GC if X is removed by GC
Retained heap is sum of shallow heap in retained
set
Other queries

Leak suspects
Heap dump Analysis - OQL

Object query Language

Different dialect for each tool

JHAT



Simple SQL combined with javascript
callback functions
MAT


More SQL like (Subquery, union, distinct)
Predefined functions (dominator)
OQL Examples

JHAT


select
heap.findClass('org.apache.catalina.util.ServerInfo')
.statics.serverNumber.toString()
MAT

SELECT serverNumber.toString() FROM OBJECTS
org.apache.catalina.util.ServerInfo
Experiences



Bugs
Incorrect usage of external libraries or
frameworks
Design issues
Bugs
Map <String, String> redirectUrlCache = new LinkedHashMap<String, String>() {
@Override
protected boolean removeEldestEntry(Map.Entry eldest) {
return size() > 10000;
}
}
String getRedirectUrl(String url) {
if(redirectUrlCache.containsKey(url)) {
return redirectUrlCache.get(url);
}
String result = computeRedirectUrl(url);
redirectUrlCache.put(url, result);
return result;
}
Usage of frameworks


Hibernate

Short time usage of big amount of memory

Too big first level cache
Good understanding of framework is a must
Design Issues

Overused session scope

Big pages


Big buffer in page context

Page context referenced by tag handlers

Heavy load causes big memory usage

Fix might require changing use cases
Reusing functionality
Interesting links

http://blogs.sun.com/sundararajan/entry/querying_java_heap_with_oql


http://www.eclipse.org/mat/downloads.php


OQL help for JHAT
Standalone MAT
www.google.com