couchbase-manager

Going back with my couchbase-manager this week the new version 0.2 has been released. That was long ago when the first version of the manager was made public but I have been waited how several issues evolved. I have to admit that there has not been a lot of movement but I think it is a good time for the second version.

Let's start with the new features in the version:

• Two new properties can be used inside the manager. The persistTo and replicateTo properties are two new features of the last versions of the couchbase Java client. In this new version you can configure the manager in order to assure that the session data is stored in disk or replicated to a defined number of nodes.

  Remember that the couchbase-manager is asynchronous in nature so the properties are used in such a way. For example a session saved with persistTo=ONE assures that the next operation will wait for the set/cas operation which, in turn, will wait the operation to be saved in the disk of one node.

• The special transcoder that the manager uses (it has to handle with the special ObjectStreams that glassfish uses) has been moved from being used in the operations (just another parameter) to be incorporated in a CouchbaseConnectionFactory. That was because of the previous new feature, both the transcoder and persist/replication options cannot be passed as the same time as arguments in all the operations.

• Previous version has several problems shutting down the manager. Mainly it was because glassfish is quite restrictive about what managers can be configured (a bug was opened with this issue but there has been no progress), and the coherence-web name was reused (this manager is a memory manager for the so-named Oracle product).

  In order to avoid the annoying ClassNotFoundException for com.tangosol.net.CacheFactory class, I have created an empty class with the same name and a shutdown method. Besides some other minor fixes were added to stop the manager flawlessly. Now the manager can be stopped and started as many times as it was needed (any application that uses the couchbase-manager can be undeployed without problems).

• The main feature of this version is that it is now mavenized. Previously it was a common Netbeans project but the new version uses a maven project (it is easier to compile and the dependencies are more clear).

  The main problem here is that glassfish, although it has a maven repository, it seems to be abandoned. Until now (current version 3.1.2.2) it is still usable but I am not very confident about the new versions (maven repository is not updated since May 2011).

  Besides I have been added some tests which uses the manager-test web services application. Those tests just check normal operations (creation, update, refresh and deletion of a session) and the four performance tests I used to execute to measure performance. Obviously you need to install the couchbase-manager and the application in some glassfish to test the project.

• Finally the manager has been updated and tested with the newer dependencies:

  • GlassFish Server Open Source Edition 3.1.2.2
  • Couchbase community edition 2.0.1
  • spymemcached-2.8.12.jar
  • couchbase-client-1.1.5.jar
  • commons-codec-1.6.jar
  • netty-3.5.5.Final.jar
  • httpcore-4.1.1.jar
  • httpcore-nio-4.1.1.jar

  The new versions of the couchbase client library have added two new dependencies (httpcore and httpcore-nio). In my humble opinion too many dependencies for the library (maybe the new Java 7 nio implementation could be used instead all this mess but I really do not know).

In the period between the versions there was an attempt to use a different serializer technique. The manager now uses common Java Serializer and it is known that other implementations like Kryo give more performance. Some little tests in order to add Kryo were done but I finally gave up because a lot of internal classes of glassfish require special treatment (objects from cdi/weld, EJBs and so on). Objects of those types need to use the Java Serializer again and I simply thought that the change was not worthy.

The bug about the LDAP realm (the couchbase-manager cannot recover a principal which is stored inside an LDAP realm) was at least updated. Another guy experienced the same problem (but in a completely another way) and the bug was marked to be fixed in version 4.0.1. So I know this is a bug and I will not change the way the manager recovers the principal user from the store when logged.

And finally an annoying behavior still persists when using the non-sticky configuration in the manager. As I have commented several times the deletion of the session cannot be done if the session object had been previously locked (it returns an error). For that reason the only way to delete a session is performing an unlock first and then sending the deletion. Obviously if the session is unlocked weird things could happen and, in my tests, they really happen. I will try to insist on this to the couchbase guys, it should be a way to delete a locked object.

So now you already know, the version 0.2 of the couchbase-manager is out. I encourage all of you to test this new version and comment me or fill a bug in github. I need your comments to check if the manager is minimally usable.

Thanks!

EDIT: I have just discovered that the new glassfish maven repo can be found here. Nice! It is a pity the version 2.0 is just released.

After the announce for the couchbase-manager-0.1 this post is going to show how it deals with some JavaEE sample applications. Before the releasing I made it work with some of them in order to make the manager compatible with part of the typical JavaEE elements (EJB, CDI, JSF Managed Beans, JavaEE security and so on). As it is said in the release notes for 0.1 version the manager is still not fully tested but, at least, some applications work.

JSF Beans, CDI and EJBs.

JSF and EJB are two important technologies in the JavaEE world and I recovered the old CertSecurityCustom project for the occasion. If you remember that project lets us login using a certificate or a typical username/password form and checks the user against an LDAP repository. The project mainly consists in a EJB and a session JSF bean. After making all the objects to implement the Serializable interface and fixing some issues with the manager (mainly the memcached transcoder must use a special glassfish Input/Output object stream) it worked like a champ.

Here I realized another thing, an application can get and attribute from the session, change it and not set it back (it is supposed that it is the same object and no set is necessary). So in 0.1 version the manager marks as dirty any session when any attribute is got (that means mainly all sessions are always set or cas, for the moment touch is only used when the attribute retrieved is a simple object -Integer, String and some other classes that cannot be changed internally-).

Besides I inject the JSF bean using CDI (@Named) and typical bean (@ManagedBean), and same with the ldap ejb (@EJB and @Inject). The four combinations worked after the commented changes.

JavaEE security

JavaEE security is another hard issue I wanted to be functional in the first realease. The application that I chose was the CertSecurityJavaEE project (the second example in the certificate login series which uses again an LDAP server but doing standard JavaEE security).

More problems here were found. Glassfish declares the principal property of the session as transient (keyword that means the property has to not be serialized and therefore saved). Looking at the replicated glassfish implementation, I saw that only the username property is kept replicated. If the session is used in another server (in which the user did not logged in), the replicated implementation logs the used silently using a special method in the realm (createFailOveredPrincipal). I did the same trick. Now the manager session has a username property that is saved in couchbase with the other data, when the session is filled that property is checked and if it is different from the principal in the session (null mainly) the principal is recovered from the realm calling the same method.

That was very nice but it didn't work with the LDAP realm. I opened a bug with the issue but silence is the only answer til the moment (some collaboration please!). Changing the realm from ldap implementation to file (users and groups are stored in a plain file) my solution works as expected. The user logins to one application server and if it changes (non-sticky load balancing or a failure in sticky configuration) the user is logged in the background. Take in mind that this happens the first time the user access a new server (the manager sees the principal is null but the username is filled and it uses the realm to reassemble the principals of the logged user).

With all those changes my little application started working using the couchbase manager.

Java Pet Store 2.0EA

Finally I tested the typical Pet Store application. This application was for along time a kind of a reference application for JavaEE, now it is abandoned and its last version 2.0EA is only JavaEE4 compliant (not 5). Nevertheless I think it is a good test cos it mixes several technologies of the JavaEE specification.

With the pet store application I faced with a glassfish issue when using mod_jk and mod_deflate in the apache. But after fixing that issue the manager started to work smoothly. All the session beans were transformed into serializable objects (in version 0.1 the manager uses standard java serialization to convert the session into a byte array, that means all the objects that are going to be saved inside the session should implement the Serializable interface).

And here it is the video. First the custom ldap login application is shown (the application refused to change me from one server to the other but finally it did), you can see how the couchbase console shows one session and, as soon as I log out, the session disappeared. Then the standard JavaEE security application is accessed. Both servers maintain my logged session using the file realm. Finally I enter Pet Store application and click over some of the pages which work smoothly.

Your browser doesn't support the video tag. See this entry for more information about how to see the videos in this blog. You can download the file instead.

Well, the couchbase-manager-0.1 is working with some (more or less) complicated JavaEE applications. I am conscious that there are more tests to do, and that is the reason to release the first version, let more people testing it. If anyone of you is interested in the manager please install it and test it with your applications (use github or this blog to report issues). Of course I guarantee nothing about the fix.

ciao!

All the last entries in the blog are dealing with my new idea about the couchbase-manager whose first version was released some days ago. I think the current status is good enough to release a first testing version. The main problem is that I have tested it only with a few real applications and that makes me extremely cautious about the quality. For the moment the last weekend I did another test I was eager to do: running the web services test application with an auto-failover three node couchbase cluster.

The demo environment is very similar to the commented in the full HA glassfish setup entry:

• My wheezy laptop with another two KVM debian testing boxes. Each virtual box contains an Apache with mod_jk module and a glassfish 3.1.2 installation. The glassfish setup consists in the admin server and two clustered instances (one in each virtual machine). The Apache in the first box balances the two cluster instances non sticky, the second one does the same but sticky.

• The three node couchbase cluster is also setup, one server in each virtual wheezy and the remaining one in my laptop. A two-node cluster seems not to be sufficient to activate the auto-failover feature which I want to test. The version I am using is 2.0.0 developer preview 4 (locking operations did not work in 1.8.0).

• Inside the cluster the manager-test application is deployed to use the couchbase-manager in non-sticky configuration. The manager is configured to use the three couchbase servers.

So the idea is simple. I run a long multi-thread test that uses some sessions and, while everything is running, the couchbase server installed in my laptop is stopped. Theoretically some of the client threads will begin to fail (the ones which were using the crashed server) but, in 30 seconds, couchbase will fail over the downed node and re-balance the situation with the two remaining servers. All the clients should recover their sessions and run again without any problem. Let's see the video.

Your browser doesn't support the video tag. See this entry for more information about how to see the videos in this blog. You can download the file instead.

As you see the couchbase console shows a three server cluster. And immediately the client program is executed with the following options:

$ time java -cp . es.rickyepoderi.managertest.client.Test \
  -b "http://192.168.122.21/manager-test/SessionTest?wsdl" \
  -t 8 -ci 100 -ts 2000 -d

That means that eight threads (clients) will create a session and perform a update or a refresh every two seconds for a hundred times, at the end the session will be invalidated (that gives me 200 seconds to do the video ). Once all the threads are doing the operations the couchbase deployed in my laptop is stopped. In that moment some clients start to fail (for example child thread 18 or 19). The console marks the server as down and the 30 seconds auto-failover timeout is triggered. While the affected clients keep failing, the failover for the crashed server is automatically performed (it is marked in the console with the alarm icon). From that moment the failed clients recover the session from another server and they continue smoothly (the session is not created again, the same session is recovered!). The done! button in the console is clicked to acknowledge the situation and then the downed server is re-started. The console immediately informs about the new situation and, clicking manually the add back button, the cluster is reestablished. Once the three nodes are again back in the cluster the sessions are manually re-balanced inside the new cluster topology (rebalance button). In this point the only weird thing happens, some clients fail again (it is like in the re-balancing process some errors could happen). The exception given by the manager is the following:

java.lang.IllegalStateException: EXCEPTION
        at es.rickyepoderi.couchbasemanager.session.CouchbaseManager.doSessionLoad(CouchbaseManager.java:715)
        at es.rickyepoderi.couchbasemanager.session.CouchbaseWrapperSession.doLoad(CouchbaseWrapperSession.java:383)
        at es.rickyepoderi.couchbasemanager.session.CouchbaseWrapperSession.lockForeground(CouchbaseWrapperSession.java:478)
        at org.apache.catalina.connector.Request.lockSession(Request.java:4165)
        at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:312)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
        at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
        at com.sun.grizzly.http.ajp.AjpProcessorTask.invokeAdapter(AjpProcessorTask.java:125)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
        at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
        at java.lang.Thread.run(Thread.java:679)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Cancelled
        at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:103)
        at es.rickyepoderi.couchbasemanager.couchbase.ClientResult.createClientResultObject(ClientResult.java:103)
        at es.rickyepoderi.couchbasemanager.couchbase.ClientRequest.waitForCompletion(ClientRequest.java:259)
        at es.rickyepoderi.couchbasemanager.session.CouchbaseManager.doSessionLoad(CouchbaseManager.java:696)
        ... 22 more
Caused by: java.lang.RuntimeException: Cancelled
        ... 26 more

So the operation was canceled. Maybe some re-tries would be necessary (in case of a cancel) but that kind of things complicate so much the implementation that, for the moment, I am not going to do it (besides the re-try could also be performed by the couchbase/spymemcached client ). But it could be said that it is working and, more important, it is working well.

I love it when a plan comes together!