Saturday, February 19. 2011
RGraph Scatter Plot using WebSockets
Today entry is the last one (at east for the moment) about the Scatter Graph problem. If you remember this series started as a HTML5 migration study of an old applet scatter graph application, the first post tried to implement a XY client-side graph using Canvas element and the second one added Web Services real samples (prototypejs + JSON + JAX-RS). Now I am going to replace the Web Services for a new HTML5 feature: WebSockets. I had no experience at all with them and I think this is a good example where this new technique fits smoothly.
WebSockets are a new protocol coordinated by the IETC (The Internet Engineering Task Force) that tries to complement our old friend HTTP which falls short in some aspects of the Web 2.0. A WebSocket is a bidirectional (communication can be started by server or client), full-duplex (both ends can send information at the same time), minimal overhead (only two bytes are used in each frame), long lived (it remains opened until an end explicitly closes it) communication channel over a single TCP socket. Standardized by the W3C the JavaScript WebSockets API is completely asynchronous and very easy to use. The protocol also defines new ws:// and wss:// prefixes which indicate a WebSocket and a WebSocket Secure connection respectively. This technique is clearly defined for avoiding all the pain associated with AJAX/Web2.0 frameworks (overcoming HTTP limitations, more simplicity, better performance,...).
The technology has been hardly pushed by Google since the beginning, obviously this company is very interested in simplifying and improving web 2.0 applications. Therefore Chrome supports WebSockets since its version 4.0 and Apple joined later with Safari 5. Until some months ago the adoption was going smoothly with firefox 4 and Opera 11 announcing WebSockets as one of their new features. Nevertheless they turned out wrong when a protocol security flaw was discovered (standard is still in draft version 76). Adam Barth showed how a malicious code can be injected in the browser using a technique called cache poisoning (exploiting a problem in the initial handshake between client and server when a web proxy is involved). Some days after the bug was reported firefox announced the new feature would be disabled by default in its new beta and same did Opera. Microsoft was always more reluctant to WebSockets, even before the security flaw was pointed out they had not said a word about this protocol and IE9. In my opinion WebSockets is a perfect example of the current anxiety about the web (all browsers trying to support any new feature before competitors, no matter if the technique is mature or not, at the same time they put any stumbling block in another's path) but the technology behind them is clearly a good idea that just needs more time. Here it is a good and more detailed report of these events.
So now my scatter graph application is going to obtain the samples using a WebSocket instead of the prototype/JAX-RS web service. In order to use the new protocol both sides need to support it, for the client part default debian Chromium 6 will be used and Glassfish 3.1 (RC2 - b41) will be our Application Server (Glassfish uses Grizzly as its Web Server component which is the piece of software that really supports WebSockets). In the implementation side almost all remain the same but the following changes:
- The JAX-RS resource class is replaced by a WebSocketApplication (Grizzly WebSocket 1.9 API). PlotApplication.java is the real class and JAXB is still being used for marshalling and unmarshalling stuff (the same PlotData object is used but JAXB is now directly called). Other difference is that JSON is now used in both ways (communication from Server/Java to browser/JavaScript and vice-versa).
- The WebSockets technology in Grizzly needs a WebSocketsServlet.java to integrate the previous application into the WAR module. GlassFish Web Sockets Sample and Web Sockets and HTML5 in Glassfish are two good entries about how to implement a WebSocket Application using Glassfish/Grizzly.
- The WebSocket application uses the same JPA entities to access the database but now a simple EJB PlotQueryEjb.java has been implemented (adding @Stateless annotation to the WebSocket application threw an exception). CDI is used for injection as usual.
- The JavaScript in the server side also needs to be changed, basically prototype.js is replaced by the WebSocket API. So now the TestRGraphChart.js (JavaScript object) and TestRGraphChart.xhtml (Facelets page) are slightly modified in the sending/receiving part.
For all these reasons the video this time is similar and less interesting. I first execute a test.html page that shows the JSON data exchanged, only WAVE type is requested so the last 24 hours of wave highs are returned, the WebSocket is then closed. After that the same actions of the previous scatter graph video are performed (first showing sea level magnitude, one sample per minute, and then waves, one per hour). The WebSocket scatter graph has been tested with Chromium 6 and firefox4 b11 obtaining a good and smooth usability (FF4 needs to override the security block for WebSockets).
In short this entry is a presentation of the WebSockets, another HTML5 improvement. WebSockets are not just a new feature they are really a new protocol definition standardized to make an easier and better Web 2.0. Its adoption is being full of obstacles and many browsers do not support them by default (only Chrome and Safari right now). So current entry has to be intended as a training application and not a production ready example. You can download the complete NetBeans project used for the video from here.
Stay tuned for more about the Web Wars!
Friday, January 21. 2011
Improving RGraph Plot with Real Data via Web Services
In a previous entry I talked about how to implement a scatter graph using javascript and the new HTML5 canvas element. This post was just the first step towards a more robust solution. If you remember my intention was studying the feasibility of a migration from an old applet application to a new HTML5 one. Today I will try to continue the PoC plotting real data inside the graph instead of the fixed sine wave (javascript generated) RGraph drew before.
First of all the real samples must be retrieved from somewhere and, obviously, a relational database is the easiest solution (besides the current application uses a database too). It is important to remember that the scatter plot represents a physical magnitude (wind strength, sea level, high of waves,...) in time. For the PoC some JPA (Java Persistence API) entities were created, these classes are very simple. Every sample only contains the timestamp and the measured value. It is clear that in a real scenario tables would usually have more columns (current devices measure several magnitudes at the same time) but I think this is enough for this example. Three classes are presented:
- PlotType.java: Enumeration that handles all the plots the application can draw (my idea is simple, if a new plot is needed it has to be added here). SEA_LEVEL and WAVE are the current plot types (Sea Level is measured one sample per minute and the High of Waves one per hour).
- PlotSample.java: Abstract entity class that represents any sample (as I said, a timestamp and the measured value).
- SeaLevel.java and Wave.java: Two examples of real plots, both extend PlotSample and define a <PLOT_TYPE_NAME>-findSamplesBetweenDates query which is always used to query for the samples between two specified dates.
Once the data is organized a web service is needed. It is quite clear this technique is perfect to retrieve the data for plotting. Nowadays there are a lot of JavaScript libraries to easily call a server-side Web Service (prototypejs, jquery,...) and managing JSON (JavaScript Object Notation). For this part I decided to use JAX-RS (Java API for RESTful Web Services) to implement the service and JAXB (Java Architecture for XML Binding) for generating the JSON data. JAX-RS (which I had rarely used) is incredible easy and very good services are implemented in a little time. Besides JSON can be specified as the result to be produced and this, using JAXB at the same time, lets you return a Java object avoiding all JSON generating stuff. Only two classes are presented here:
- PlotData.java: This is the JAXB element that represents the data to be plotted by the browser. The data returned when a scatter plot is going to be drawn is just a simple class with some properties (start and end timestamps, plot type name, description,...) and a map with the samples (value keyed by the timestamp).
- PlotResource.java: The JAX-RS resource which receives the name of the plot type to graph, start and end timestamp. With this information it queries the database and creates the PlotData to be returned.
Two important tips here. First, JavaScript (in my opinion) handles the timezone in dates not very good. The date object is created using the timezone of the browser/system (GMT+1 in my case) but it is quite difficult to convert from one timezone to another. The date object has a getTimezoneOffset method that returns the offset in minutes between your zone and GMT, but your code needs to add and subtract offsets all the time. For this reason I decided to manage timestamps (milliseconds since standard epoch of 1/1/1970) instead of string dates. A timestamp will be represented in JSON with a String containing the big number of milliseconds prefixed by a character "T" (for example the epoch 1/1/1970 would be the string "T0").
Second, I wanted the JSON response was (more or less) the following:
{ "type":"WAVE", "start":"T1295095138172", "end":"T1295181538172", "description":"High of the waves", "legend":"Meters (m)", ... "data": { "T1295096400000":"3.4", "T1295100000000":"3.16", "T1295103600000":"3.52", ... } }
The JSON data part (the list/map of samples) should be the pair timestamp and value, which is the natural way to represent a map in JSON. Nevertheless JAXB has some problems with maps and they are not represented in this natural way (see RFE JERSEY-551 for more info). So I added the JsonMapAdapter.java class this enhancement manages in order to obtain the JSON representation I showed before.
So now the browser can query via the restful web service in order to obtain the data to plot. The final part is performing some little changes in the JavaScript developed for the previous post, now the array of samples has to be retrieved calling the Web Service. The presentation part are the following files:
- PlotTypeBean.java: A extremely simple request scoped JSF bean only used to list the plot types defined in the application (the combo box at the right) and select one of them.
- TestRGraphChart.xhtml: XHTML page very similar to the one used in the first entry but using facelets/JSF in order to call the previous bean and so on.
- TestRGraphChart.js: The JavaScript file that uses prototype.js and JavascriptToolbox data.js to call the web service and parse/manage JSON and timestamps.
So the PoC is ready, RGraph plots a scatter XY graph of which data is retrieved via Web Service. The application was deployed in a Java EE 6 compatible application server (Glassfish v3) using its default database engine (derby/javadb). I personally think this is an elegant solution for my problem, all the plotting part is done by the browser and the server only sends the samples to draw. The solution works quite well in browsers with decent canvas performance (which are all the current browsers except IE8, check the previous post). In the video I first call restful web service with WAVE plot type (high of waves), I use no parameters so it returns 24 hours of data (WAVE plot type has one sample per hour). I change to the graph page in order to display the whole day of the sea level plot (one sample per minute). I also move the graph forward and reverse in time. Then I increase the range of time to display a whole week of data (less than 10,000 points cos I did not insert in my database the whole week of samples). And then I choose the time range to plot selecting directly over the canvas element. Finally I select the other plot type, high of waves again. As you can see it works very well in my chromium 6 browser. The application is also very smooth in iceweasel 3.5 and epiphany 2.30 (the other two browsers I have installed in my debian box).
At the moment the JavaScript always requests all the samples to plot. It is clear that some data can be reused (for example when the plot is moved forward or reverse) and the JavaScript could be improved to only request the missing samples and after join all of them. You can improve that part if you need and want it.
As a summary this entry goes one step further in the scatter graph problem. Now the solution plots real data. Data which is read from a database and requested by the browser using a RESTful Web Service and JSON. Here I present the complete NetBeans project (without any jar file as usual). As I said in the first entry of this series the PoC is currently unusable (or at least without a second plan for IE8), but the time for triggering HTML5 features is getting closer day by day...
Please plot a smile in your face!
Tuesday, November 23. 2010
Canvas Scatter Plot
These weeks I have been working in an application that shows some sea physical data (sea level, waves altitude, water temperature, wind strength,...) using scatter plots. Now the application uses an applet to display the XY graphs which, I must say, is incredibly fast. But, as I commented before in the digital signature entries, applets are now a quite obsoleted solution and sometimes are not very good in terms of compatibility (users need to have a Java Virtual Machine installed, problems with different versions, many linux distributions use by default openjdk/icedtea/gcjwebplugin which crashes sometimes and many other issues which usually generate a lot of complaints). I decided to study a direct browser solution using new HTML5 and javascript features.
Spending some time searching over the web I saw there are a lot of libraries to directly draw charts in javascript. But looking deeper I discovered the majority of them use the new HTML5 canvas tag. The canvas is a new component to render inside its area 2D shapes and images using javascript.
The first library I used was JSCharts (I did not think too much, it was at the first position in the previous link). It is very powerful but does not fit completely for my problem. The plots of the application were usually a magnitude (meters, degrees, velocity,... in the Y axis) throughout the time (X axis). In the applet version some magnitudes show a very large number of points (around ten thousand). JSCharts cannot handle such a number of points, the resulting graphs are very beautiful but too much complex for this large amount of data. There are some features that explains this fact: points are circles, a specific scatter plot does not exist and a line chart (with invisible line) must be used, time labels must be assigned separately,... All of them make JSCharts not fast enough to draw the 10,000 points in a reasonable time.
After this first disappointment I decided to just test with a direct canvas implementation (i.e. developing my own javascript that plots a very simple XY graph). But searching information about canvas I discovered a very straight forward graph library which also uses the canvas tag, RGraph. This implementation is much simpler than JSCharts and it has a direct scatter plot graph type. Developing a initial time/magnitude plot the times became very very competitive (for example RGraph scatter plot draws 10,000 points in less than a second in many browsers).
Finally a study about browser compatibility must be done. And one more time, same thing happened in the HTML5 video announcement, all Internet Explorers (included current IE8) do not support the canvas tag. Right now there is a ExCanvas project that emulates the canvas tag using VML. The solution (as you can see later) is incredibly slow and there are initiatives for doing the same using flash , silverlight or a ActiveX component. IE9, which is in beta state, is going to support canvas with many other HTML5 features. For this reason I prepared a little example of a XY plot to test usability with different numbers of points. The sample code draws a sine wave (javascript generated) in a period of time and let us change the number of points, showing finally the drawing time in a box (the part for generating the wave is not taken into account). At the end of the entry I present the little example inside an iframe (I know it is not very beautiful but this could be easily improved).
When I played with the resulting graph using different browsers numbers were the following (I took them not very precisely):
100 points | 1,000 points | 10,000 points | |
iceweasel 3.5.15 (linux) | 26ms | 80ms | 712ms |
chromium 6.0.472.63 (linux) | 10ms | 18ms | 98ms |
firefox 3.6.12 (Windows) | 32ms | 130ms | 1,018ms |
firefox 4.0 Beta 7 (Windows) | 32ms | 120ms | 970ms |
chrome 7.0.517.44 (Windows) | 11ms | 32ms | 240ms |
IE8+ExCanvas (Windows) | 800ms | 3,850ms | 70,000ms+ |
IE9 Preview 7 (Windows) | 17ms | 65ms | 500ms |
Opera 10.63 (Windows) | 25ms | 49ms | 305ms |
My conclusion is a HTML5 plot solution results usable in any browser that supports canvas, but IE8 kills us again. Any canvas graph is fateful for many windows users. I decided some time ago to trigger all HTML5 new features when debian testing will distribute iceweasel 3.6 (squeeze must be released before) and chromium 6.0 (it is already) and IE9 will be released. As you can see in the numbers, times are very good in the last versions of the browsers and even better in the future ones. You already know but it is absolutely necessary to keep our browsers in their last versions. Now there are a lot of options (free, open source, distributed with the OS,...), so there is no excuse. Make web developers life easier...
Please, upgrade your browser!!!
Comments