Saturday, May 23. 2015
Monitoring and Graphing a JavaEE application using SNMP
The previous month I was teaching a course about several Java technologies and, between them, JMX (Java Management Extensions) was explained in detail. In the training I prepared an activity about an adapter that transformed a JMX MBean to SNMP. The SNMP (Simple Network Management Protocol) is a very used protocol in monitoring, it started for network elements but now it is ubiquitous and it is used for any hardware or software. All the monitoring systems support it out of the box to graph performance information or send alarms (examples are applications like nagios or cacti). This protocol uses a hierarchical namespace containing Object IDentifiers (OID) which are represented like a list of numbers (typically 1.3.6.13.6.1...). Each OID identifies a variable that can be read and/or written using the protocol and represents a real magnitude (for example in a network device the speed negotiated by a port or if this port is connected). Standardized OIDs are mapped to meaningful names using a MIB (Management information base) which describes each OID (for example this is the definition MIB for the internal JVM SNMP which just exposes some general JVM information). The activity for the training was very simple and the solution had three main drawbacks:
The integration used JoeSNMP that does not support SNMPv3.
The integration did not work for all the MBean types (model in particular) and in all situations.
The solution worked at application level, and not at container level, which is the ideal solution.
Today's entry is going to explain how to perform a better SNMP integration for JMX monitoring. The solution is based in the snmp4j implementation and Tomcat 8 is used as the container. In the following sections we will see that the integration starts an SNMP server at container level (only one SNMP server in the JVM) and then each application can register its MBeans in both servers, the JMX and the SNMP one. In this example the MBean attributes are mapped to an SNMP OID and type. The solution should be interpreted just as a PoC (Proof of Concept) and some parts, mainly the SNMP agent server, need some love to be in a real production level.
The SNMP server
The snmp4j library is a very detailed SNMP implementation and, for JMX integration, three different libraries are needed:
snmp4j.jar: core library.
snmp4j-agent.jar: agents that can be used to start an SNMP server.
snmp4j-agent-jmx.jar: library that implements the adapter to bridge SNMP and JMX.
The snmp4j project implements an SNMP server just extending the BaseAgent class defined in the snmp4j-agent library. This class lets the developer to configure the server at all levels (communities, views, users and all SNMP stuff) but programmatically. It seems that snmp4j can load and save configuration but using serialization which, in my opinion, is not very useful. The snmp4j-agent-jmx library provides a sample agent to run an SNMP server in a JMX environment (JMXTestAgent class) but I preferred to construct my own agent just to try to understand how snmp4j agents work. My agent class is es.rickyepoderi.snmpjmxagent.SnmpAgent, it supports SNMP version 2 and 3 and only one user (testuser with the same password) is registered with full access. Everything is hardcoded and that is the reason why I said before that there is room for enhancement in this part (you know, a configuration file would be very nice ).
The mapping class
The snmp4j implementation is very elaborated in the JMX mapping (how to map a JMX MBean attribute to an SNMP OID). As my PoC just wants to map JMX attributes to SNMP I decided to implement a simple mapping class called ScalarAttributeDescription with the following properties:
oid: The oid I want to assign to this JMX attribute (for example 1.3.6.1.2.1.1.1).
attributeName: The name of the attribute in the JMX MBean.
attributeType: The class of the attribute in the JMX MBean (String, Long or whatever).
access: The type of access in SNMP (snmp4j defines some simple access like ACCESS_READ_WRITE or ACCESS_READ_ONLY).
variable: The type of data in SNMP (OctetString, Counter64 or whatever). The attributeType and the variable should match.
With a list of descriptions of this kind, developers can map every MBean attribute to an OID, and later assign them to the SNMP agent previously created.
The singleton SNMP module
As I explained in the introduction only one SNMP agent is going to run in the container and the different applications deployed inside it should be able to register their MBeans in the agent. For that reason I decided to implement a singleton which let us register and unregister an MBean inside the SNMP server. This singleton called SnmpJmxModule has two main methods:
public MOGroup registerBean(MBeanServer server, ObjectName name, ScalarAttributeDescription[] descs);
This method receives the JMX server, the name of the MBean inside the previous server and the array of descriptions for the bean's attributes that want to be populated in the SNMP server. The method uses the snmp4j API to construct the proper mapping objects and register them inside the SNMP agent. The returned MOGroup is the group of objects that have been registered in the server, the group can be used later to unregister them from the server.
public void unregisterBean(MOGroup mog);
The method to unregister from the SNMP server a previously registered group of OIDs.
With this singleton, each application can have a different namespace (for example app1 is under 1.3.6.1.2.1.1 and app2 under 1.3.6.1.2.1.2) and below it they can map as many MBeans and attributes as they want. This way a lot of applications can be running and exposing their JMX Beans through SNMP in different OIDs (exactly as I wanted) in the same container.
Example Application
Finally I prepared a little servlet application (the typical HelloWorld) that uses a Hello MBean to manage the greetings. The MBean maintain some counters about the number of hellos processed (there are two types: world hellos, which are the ones given to the world when no name is specified; and named hellos, the ones sent to a specified name) and let us change the greeting message. The servlet (which is deployed to the generic pattern "/*") uses the MBean to say the hello and, this way, the counters are updated and the greeting message used. Here it is the the simple interface that the standard MBean uses.
public interface HelloMBean {
public String getMessage();
public void setMessage(String message);
public long getHellos();
public long getWorldHellos();
public long getNamedHellos();
public String sayHello(String name);
public void reset();
}
But the interesting part is performed in the listener, when the context is initialized by the container it registers the Hello MBean in the JMX server, constructs the mappings for it and registers them in the SNMP agent. The code is not very complicated.
public void contextInitialized(ServletContextEvent sce) {
try {
System.err.println("Registering the bean...");
ServletContext sc = sce.getServletContext();
// register the Hello MBean in the JMX server, the bean is stored
// in the context using the attribute JMX_BEAN
Hello mbean = new Hello();
String name = "es.sample.jmx:type=HelloMBean,name=" + sc.getContextPath();
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName objectName = new ObjectName(name);
mbs.registerMBean(mbean, objectName);
sc.setAttribute(JMX_BEAN, mbean);
// register the Bean also in the SNMP server
// for that the four attributes are described to get a proper mapping
// the MOGroup is stored also in the context to unregister at destroy
// This number is just to deploy the app two times and have another namespace
int number = Math.abs(name.hashCode());
ScalarAttributeDescription[] descs = new ScalarAttributeDescription[]{
new ScalarAttributeDescription(new OID(new int[]{1, 3, 6, 1, 2, 1, number, 1}),
"Message", String.class, MOAccessImpl.ACCESS_READ_WRITE, new OctetString()),
new ScalarAttributeDescription(new OID(new int[]{1, 3, 6, 1, 2, 1, number, 2}),
"Hellos", Long.class, MOAccessImpl.ACCESS_READ_ONLY, new Counter64()),
new ScalarAttributeDescription(new OID(new int[]{1, 3, 6, 1, 2, 1, number, 3}),
"WorldHellos", Long.class, MOAccessImpl.ACCESS_READ_ONLY, new Counter64()),
new ScalarAttributeDescription(new OID(new int[]{1, 3, 6, 1, 2, 1, number, 4}),
"NamedHellos", Long.class, MOAccessImpl.ACCESS_READ_ONLY, new Counter64())
};
SnmpJmxModule sjm = SnmpJmxModule.getInstance();
MOGroup mog = sjm.registerBean(mbs, objectName, descs);
sc.setAttribute(SNMP_MO_GROUP, mog);
} catch (Exception e) {
System.err.println("Error creating and registering the bean...");
e.printStackTrace();
}
}
As you see the ScalarAttributeDescription array defines just the four attributes that the Hello MBean uses and maps then to a numeric OID. The hello application uses the namespace 1.3.6.1.2.1.{number}, the variable number is just a hash of the context path to be able to deploy the same application several times (instead of using two apps with different namespaces I am going to use the same but this last number makes the OIDs unique). It is also clear that the MBean itself and the SNMP MOGroup (the group of objects registered inside the SNMP agent) are stored inside the context as attributes. This way the servlet can recover the MBean to perform the hello responses and the contextDestroyed method can unregister them.
public void contextDestroyed(ServletContextEvent sce) {
try {
System.err.println("Unregistering the bean...");
ServletContext sc = sce.getServletContext();
// unregister the mbean from JMX server
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName name = new ObjectName("es.sample.jmx:type=HelloMBean,name=" + sc.getContextPath());
mbs.unregisterMBean(name);
// unregister from SNMP agent
MOGroup mog = (MOGroup) sc.getAttribute(SNMP_MO_GROUP);
SnmpJmxModule sjm = SnmpJmxModule.getInstance();
sjm.unregisterBean(mog);
} catch (Exception e) {
System.err.println("Error unregistering the bean...");
e.printStackTrace();
}
}
Deployment
Finally both projects should be deployed in the Tomcat container. Some libraries should be placed at container level (${TOMCAT_HOME}/lib folder):
cp snmp4j-agent-jmx-2.1.0.jar snmp4j-agent-2.3.1.jar snmp4j-2.3.3.jar SnmpJmxAgent.jar ${TOMCAT_HOME}/lib
All the snmp4j libraries and my little wrapper are copied to the general lib folder (the wrapper library contains the SnmpAgent server, the ScalarAttributeDescription mapping definition and the singleton SnmpJmxModule). This way all the applications inside the container can use snmp4j API and my registration classes.
Finally the servlet demo application is deployed normally. It can be deployed twice in order to emulate two different applications (remember that a hash number make them use different OID namespaces).
cp JmxServlet.war ${TOMCAT_HOME}/webapps/JmxServlet.war
cp JmxServlet.war ${TOMCAT_HOME}/webapps/JmxServlet2.war
And that is all! With these simple steps the Hello MBean is registered and unregistered in both servers (JMX and SNMP) without problem. All the objectives are fulfilled: snmp4j-agent-jmx has no problem with any MBean type; SNMP server runs at container level; any application can register its MBeans in the SNMP agent using a different namespace tree; snmp4j supports SNMPv3.
Demo
Here I present a video where the sample hello application is already deployed once, so if I access to the browser I receive my named hello. Therefore if I launch jconsole I can inspect my MBean and even change my greeting message. But interestingly I can also perform a snmpwalk (v2c and v3) retrieving the same counters. Finally the application is deployed again with a different name and the new MBean is crawled using SNMP without any problem.
You can download both NetBeans projects from the following links: SnmpJmxAgent (the wrapper library installed at container level) and JmxServlet (the sample hello servlet application that exposes the Hello MBean).
Summary
Today's entry is very interesting. Java gives JMX as the monitoring framework inside the JVM. A lot of applications and frameworks (all the containers, like tomcat or glassfish, and all the important frameworks, like Spring or Hibernate) expose their respective MBeans in order to manage and monitor their behavior. Obviously your projects can also take advantage of this technology and custom MBeans can be deployed to monitor or even configure your own Java applications (thing that I see much less than I would desire). JMX technology fits perfectly inside Java but not so well inside the typical monitoring solutions (software like cacti or nagios). Usually if you want to integrate your application inside a monitoring system some script or little program is needed (do you remember a previous entry about this question here in the blog?). Nevertheless SNMP is the perfect protocol for monitoring systems. All of them understand and retrieve SNMP objects out of the box. Besides JMX takes into account the translation of itself to other protocols like SNMP (JMX uses adapters for this task). So, why not to integrate them? This entry uses the snmp4j implementation to start an SNMP server inside Tomcat and expose some Mbean attributes through this protocol, this way, that information can be collected by any monitoring system.
Really cool! Isn't it?
Comments