Saturday, September 29. 2012
Couchbase Manager for Glassfish: JavaEE examples
After the announce for the couchbase-manager-0.1 this post is going to show how it deals with some JavaEE sample applications. Before the releasing I made it work with some of them in order to make the manager compatible with part of the typical JavaEE elements (EJB, CDI, JSF Managed Beans, JavaEE security and so on). As it is said in the release notes for 0.1 version the manager is still not fully tested but, at least, some applications work.
JSF Beans, CDI and EJBs.
JSF and EJB are two important technologies in the JavaEE world and I recovered the old CertSecurityCustom project for the occasion. If you remember that project lets us login using a certificate or a typical username/password form and checks the user against an LDAP repository. The project mainly consists in a EJB and a session JSF bean. After making all the objects to implement the Serializable interface and fixing some issues with the manager (mainly the memcached transcoder must use a special glassfish Input/Output object stream) it worked like a champ.
Here I realized another thing, an application can get and attribute from the session, change it and not set it back (it is supposed that it is the same object and no set is necessary). So in 0.1 version the manager marks as dirty any session when any attribute is got (that means mainly all sessions are always set or cas, for the moment touch is only used when the attribute retrieved is a simple object -Integer, String and some other classes that cannot be changed internally-).
Besides I inject the JSF bean using CDI (@Named) and typical bean (@ManagedBean), and same with the ldap ejb (@EJB and @Inject). The four combinations worked after the commented changes.
JavaEE security
JavaEE security is another hard issue I wanted to be functional in the first realease. The application that I chose was the CertSecurityJavaEE project (the second example in the certificate login series which uses again an LDAP server but doing standard JavaEE security).
More problems here were found. Glassfish declares the principal property of the session as transient (keyword that means the property has to not be serialized and therefore saved). Looking at the replicated glassfish implementation, I saw that only the username property is kept replicated. If the session is used in another server (in which the user did not logged in), the replicated implementation logs the used silently using a special method in the realm (createFailOveredPrincipal). I did the same trick. Now the manager session has a username property that is saved in couchbase with the other data, when the session is filled that property is checked and if it is different from the principal in the session (null mainly) the principal is recovered from the realm calling the same method.
That was very nice but it didn't work with the LDAP realm. I opened a bug with the issue but silence is the only answer til the moment (some collaboration please!). Changing the realm from ldap implementation to file (users and groups are stored in a plain file) my solution works as expected. The user logins to one application server and if it changes (non-sticky load balancing or a failure in sticky configuration) the user is logged in the background. Take in mind that this happens the first time the user access a new server (the manager sees the principal is null but the username is filled and it uses the realm to reassemble the principals of the logged user).
With all those changes my little application started working using the couchbase manager.
Java Pet Store 2.0EA
Finally I tested the typical Pet Store application. This application was for along time a kind of a reference application for JavaEE, now it is abandoned and its last version 2.0EA is only JavaEE4 compliant (not 5). Nevertheless I think it is a good test cos it mixes several technologies of the JavaEE specification.
With the pet store application I faced with a glassfish issue when using mod_jk and mod_deflate in the apache. But after fixing that issue the manager started to work smoothly. All the session beans were transformed into serializable objects (in version 0.1 the manager uses standard java serialization to convert the session into a byte array, that means all the objects that are going to be saved inside the session should implement the Serializable interface).
And here it is the video. First the custom ldap login application is shown (the application refused to change me from one server to the other but finally it did), you can see how the couchbase console shows one session and, as soon as I log out, the session disappeared. Then the standard JavaEE security application is accessed. Both servers maintain my logged session using the file realm. Finally I enter Pet Store application and click over some of the pages which work smoothly.
Well, the couchbase-manager-0.1 is working with some (more or less) complicated JavaEE applications. I am conscious that there are more tests to do, and that is the reason to release the first version, let more people testing it. If anyone of you is interested in the manager please install it and test it with your applications (use github or this blog to report issues). Of course I guarantee nothing about the fix.
ciao!
Saturday, August 11. 2012
Couchbase Manager for Glassfish: In Action
All the last entries in the blog are dealing with my new idea about the couchbase-manager whose first version was released some days ago. I think the current status is good enough to release a first testing version. The main problem is that I have tested it only with a few real applications and that makes me extremely cautious about the quality. For the moment the last weekend I did another test I was eager to do: running the web services test application with an auto-failover three node couchbase cluster.
The demo environment is very similar to the commented in the full HA glassfish setup entry:
My wheezy laptop with another two KVM debian testing boxes. Each virtual box contains an Apache with mod_jk module and a glassfish 3.1.2 installation. The glassfish setup consists in the admin server and two clustered instances (one in each virtual machine). The Apache in the first box balances the two cluster instances non sticky, the second one does the same but sticky.
The three node couchbase cluster is also setup, one server in each virtual wheezy and the remaining one in my laptop. A two-node cluster seems not to be sufficient to activate the auto-failover feature which I want to test. The version I am using is 2.0.0 developer preview 4 (locking operations did not work in 1.8.0).
Inside the cluster the manager-test application is deployed to use the couchbase-manager in non-sticky configuration. The manager is configured to use the three couchbase servers.
So the idea is simple. I run a long multi-thread test that uses some sessions and, while everything is running, the couchbase server installed in my laptop is stopped. Theoretically some of the client threads will begin to fail (the ones which were using the crashed server) but, in 30 seconds, couchbase will fail over the downed node and re-balance the situation with the two remaining servers. All the clients should recover their sessions and run again without any problem. Let's see the video.
As you see the couchbase console shows a three server cluster. And immediately the client program is executed with the following options:
$ time java -cp . es.rickyepoderi.managertest.client.Test \ -b "http://192.168.122.21/manager-test/SessionTest?wsdl" \ -t 8 -ci 100 -ts 2000 -d
That means that eight threads (clients) will create a session and perform a update or a refresh every two seconds for a hundred times, at the end the session will be invalidated (that gives me 200 seconds to do the video ). Once all the threads are doing the operations the couchbase deployed in my laptop is stopped. In that moment some clients start to fail (for example child thread 18 or 19). The console marks the server as down and the 30 seconds auto-failover timeout is triggered. While the affected clients keep failing, the failover for the crashed server is automatically performed (it is marked in the console with the alarm icon). From that moment the failed clients recover the session from another server and they continue smoothly (the session is not created again, the same session is recovered!). The done! button in the console is clicked to acknowledge the situation and then the downed server is re-started. The console immediately informs about the new situation and, clicking manually the add back button, the cluster is reestablished. Once the three nodes are again back in the cluster the sessions are manually re-balanced inside the new cluster topology (rebalance button). In this point the only weird thing happens, some clients fail again (it is like in the re-balancing process some errors could happen). The exception given by the manager is the following:
java.lang.IllegalStateException: EXCEPTION at es.rickyepoderi.couchbasemanager.session.CouchbaseManager.doSessionLoad(CouchbaseManager.java:715) at es.rickyepoderi.couchbasemanager.session.CouchbaseWrapperSession.doLoad(CouchbaseWrapperSession.java:383) at es.rickyepoderi.couchbasemanager.session.CouchbaseWrapperSession.lockForeground(CouchbaseWrapperSession.java:478) at org.apache.catalina.connector.Request.lockSession(Request.java:4165) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:312) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231) at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195) at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849) at com.sun.grizzly.http.ajp.AjpProcessorTask.invokeAdapter(AjpProcessorTask.java:125) at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746) at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045) at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228) at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90) at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79) at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54) at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59) at com.sun.grizzly.ContextTask.run(ContextTask.java:71) at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532) at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513) at java.lang.Thread.run(Thread.java:679) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Cancelled at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:103) at es.rickyepoderi.couchbasemanager.couchbase.ClientResult.createClientResultObject(ClientResult.java:103) at es.rickyepoderi.couchbasemanager.couchbase.ClientRequest.waitForCompletion(ClientRequest.java:259) at es.rickyepoderi.couchbasemanager.session.CouchbaseManager.doSessionLoad(CouchbaseManager.java:696) ... 22 more Caused by: java.lang.RuntimeException: Cancelled ... 26 more
So the operation was canceled. Maybe some re-tries would be necessary (in case of a cancel) but that kind of things complicate so much the implementation that, for the moment, I am not going to do it (besides the re-try could also be performed by the couchbase/spymemcached client ). But it could be said that it is working and, more important, it is working well.
I love it when a plan comes together!
Sunday, July 22. 2012
Couchbase Manager for Glassfish: Asynchronous Operations
Yesterday the version 0.1 of the couchbase-manager was released (only for testing pusposes). I decided to invest some time in order to improve the performance using asynchronous operations. Until now all the operations were synchronous, the manager waited patiently for the couchbase server to return the result. If you think about it the last operation against the memory manager can be done asynchronously, in that way the session is ordered to be saved (or touched) but the operation is just initiated, the control is returned immediately to the manager and the session is processed against the memory repository in the background. Normally the couchbase operation will finish before the next user request reaches the application server. Using background saving sticky configuration has almost none operation against couchbase and non-sticky always saves the last access (actually the operation is done, but in the background).
The modification was quite hard and it made me to add a lot of code to the project. I implemented a totally new way of access the couchbase server but, I think, the new code is much more tidy. A ClientRequest is always created for any couchbase operation and any request produces a ClientResult. Although the request always use the async couchbase/spymemcached method it can be executed synchronously or asynchronously. Synchronization is done just waiting the operation to finish calling the waitForCompletion method in the request. Async execution is always done (just not calling the waiting method) but it is also possible to execute some code after the operation finishes (execOnCompletion method). This method will be used to clean some data in the session once it is saved or touched and it is implemented using a different thread that waits for the operation to finish. Finally there is another method, waitForExecution, this one waits a previous async execution to finish. The waitForCompletion and waitForExecution should not be mistaken. The first one waits the operation to finish, the second waits the thread. The first one is used to perform a synchronous execution, the second one to wait an async execution to finish. The main target for the waitForExecution is waiting when a new operation should be executed but a background one is still being processing (imagine a session is being set in the background but, before terminating, another request access the app server, before being locked it should wait the previous operation to finish).
Besides the async feature, correct exception handling, managing the couchbase errors, has been added to the manager. The main idea is any access to the couchbase server that returns an exception or an incorrect state (for example not found when unlocking) marks the session in ERROR condition and throws an IllegalStateException (a RuntimeException). Of course the error could be retrieved in a background operation, in those cases the session is just marked. Any session in ERROR should be re-read from couchbase in the next request. This last part can be better understood using an sticky example. A new request from a client comes to the app server, as the manager is configured sticky it does not read the session from couchbase and just locked it internally, after the application manages the session it is saved in the background. Background set operation fails. Because of the decoupling, the session can only be marked in error state and the next request is forced to re-read the session from couchbase (no matter if sticky or not, when session is in error state it is always re-read).
As I said async calling is only used in the last part of the request life-cycle (set/cas if the session was modified, touch/unlock if only accessed and unlock/delete if invalidated). Besides if the operation needs two calls (you know some operations needs two couchbase calls, no touchAndUnlock and no unlockAndDelete), only the last one is done in background mode (so non-sticky scenario is again penalized by the two operation issue). Obviously asynchronous processing could also be used in the first locking part but I think in this stage is not worthy (session should be accessed in almost any request, and manager should wait for the operation anyways). In summary, first part of the request (lock in non-sticky configuration) is synchronous and the last part (save, touch or delete in both scenarios, sticky and non-sticky) is asynchronous.
For all that the new features were complicated and I spent some weeks implementing the changes (remember I do all those things in my spare time). I think that now it is completed (and I hope that I have done it correctly). First I show the graphics comparing previous synchronous execution to current async access. I also link the sheet with the meassured times.
Checking the new times I have the following conclusions:
Now sticky configuration is as fast as the non-replicated (normal) glassfish setup. It is only a bit slower in session creation. That is clear cos now no operation against the couchbase server is done in the foreground (stickyness saves the first getAndLock and asynchronous calls save the set/touch final operation). This point I think is incredible.
Non-sticky configuration is still a bit slower than the other setups but, comparing the times between the two non-sticky scenarios, they are very very close, especially in update and refresh operations. As I said before with a more complete set of couchbase lock methods (touchAndUnlock and unlockAndDelete) the times would be a little better but non-sticky configuration always adds some penalty in performance.
There have been several changes since these tests (so maybe the times are different in the 0.1 version) but I wanted to comment the new way of working. You can test the 0.1 version just following the installation instructions on the wiki. Please report any issue or question here in the blog or via github.
An asynchronous goodbye!
Comments