Tuesday, January 26. 2016
The past week I was involved in an issue with a customer that runs a JavaEE application inside a WebSphere Application Server (WAS). Quickly I realized that there was a high memory usage (the app server needed to be restarted every day, at night hours). That high consumption was not explained by the application and I recommended to obtain a memory dump to analyze what components were using the memory. I usually use Memory Analyzer Tool (MAT) to parse and study a dump (it has very good features like the dominator tree, which is incredibly useful to understand what the hell is happening inside the JVM heap). The problem in this customer is that they are using the IBM JVM (not common Sun / Oracle / OpenJDK virtual machine) and even the memory dump is different (Sun derived JVM uses typical HPROF file while the WAS administrators sent me a PHD -Portable Heap Dump- file, which I did not even known the extension name before). Obviously that format is not recognized by MAT by default. This entry is an easy guide to make MAT to understand IBM Java dumps.
The first important difference with my typical use of MAT is that I needed the eclipse plugin (MAT application is a eclipse plugin but it can also be used as a standalone desktop program which is much better, take in mind you usually need a lot of memory to parse a big memory dump). In order to understand the IBM JVM dumps the DTFJ (Diagnostic Tool Framework for Java) is also compulsory and I have only found it as a eclipse update (I suppose you can somehow add the JAR libraries to the desktop application but I do not care about it, I will only need this thing from time to time, so using eclipse was enough for me).
The procedure to install all the components is exampled below.
Install the eclipse software and start the IDE (mars in my case).
tar zxvf Desktop/eclipse-jee-mars-1-linux-gtk-x86_64.tar.gz cd eclipse ./eclipse
Once the IDE is up and running the MAT extension should be installed as suggested in the MAT download page. For that, click in the Help tab and Install New Software... In the window add the MAT update site (http://download.eclipse.org/mat/1.5/update-site/) and click Add. Select the first component which is the plugin for the eclipse IDE (Memory Analizer for Eclipse IDE). Click Next.
The eclipse window shows a summary of the installation. Click Next.
In the third window of the wizard accept the license and click Finish.
The eclipse IDE downloads and installs the components and requests for a restart.
The same steps should be followed to install the IBM DTJF as commented here. Again in Help → Install new Software... Select now the DTJF update site (http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/runtimes/tools/dtfj/), click the Add button and then select the only component that appears (IBM Monitorig and Diagnostics Tools). Finally click Next.
Click Next in the summary window.
Click to accept the terms of the license agreement and Finish.
Again after the download/installation of the libraries the IDE asks for a restart to activate the new plugin.
Now the eclipse IDE has the MAT application extension inside the IDE and the capability to open and parse IBM memory dump formats (both features are given by each plugin installation). So in this point we are able to open any memory dump produced by an IBM JVM but there is a common problem here. Usually the dumps are quite big (for example my dump is almost 1GB and represents a dump of a 8GB JVM heap) so it is absolutely necessary to tune the memory given to the eclipse IDE (that is why I prefer to use the separate desktop application). So please modify the eclipse.ini file and add a proper memory configuration for your dump (in my case -Xms1g -Xmx8g).
cat eclipse.ini -startup plugins/org.eclipse.equinox.launcher_1.3.100.v20150511-1540.jar --launcher.library plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.300.v20150602-1417 -product org.eclipse.epp.package.jee.product --launcher.defaultAction openFile -showsplash org.eclipse.platform --launcher.XXMaxPermSize 256m --launcher.defaultAction openFile --launcher.appendVmargs -vmargs -Dosgi.requiredJavaVersion=1.7 -XX:MaxPermSize=256m -Xms1g -Xmx8g
Close the eclipse IDE, modify the eclipse.ini file accordingly and restart the IDE.
Now the IDE is ready to open my PHD file, just click the File menu tab and then Open File... Select the PHD dump and open it. Now the MAT starts to analyze the dump (it can be a quite long process, be patient in general with MAT) and then presents a wizard about the report to generate (I usually click Cancel here). That is all, the MAT has just parsed an IBM JVM memory dump.
Luckily, in my problem, the classes guilty of the high memory consumption were inside WAS packages (they seem related to Dynamic Replication Services -DRS- and Memory to Memory replication -MTM-) and not with the application (which is good for me and bad for WAS administrators).
As you see this entry is a very simple howto to add MAT and IBM DTFJ as eclipse plugins in order to parse and analyzing memory dumps from the IBM JVM. It is an extraordinary simple howto but I prefer to have my own procedure to be sure I do not forget or loose it.