Sunday, November 30. 2014
Compiling Servlet (part III)
This is the third and the last entry of the in-line compilation PoC series. The first entry started the development of a compiling servlet that loads, compiles and executes another servlet source file. The second post extended the previous servlet adding JSP management. Both entries were based on the idea of having my own parser and compiler libraries (the ECJ Java compiler is used for compilation and the Jasper 2 engine for parsing JSP). This third entry is dedicated to another JavaEE technology that can be integrated inside the same idea (template hot replacement): Java Server Faces (JSF). In this case we will see that no compilation is needed, not even a servlet is necessary but, nevertheless, I decided to not change the title of the series.
JSF was born like a component driven user interface technology which used internally JSP to render HTML pages. The technology offers a lot of JSP components to construct the pages and, besides, it gives an easy access to Java code though JSF beans (little java objects that links presentation and logic layers). JSF technology is based on the Model View Controller (MVC) architecture for separating logic from presentation. Since JSF 2.0 the JSP dependency was optional by using Facelets instead. Facelets is a new template technology which is much simpler and dynamic and, in my opinion, much better suited for HTML generation. Finally it is important to consider that around JSF there are a lot of different technologies which are more or less related: CDI (Context Dependency Injection), EL (Expression Language), AJAX,...
Going back to the in-line compiling issue, JSF (as I commented in the previous paragraph) gives us two different approaches: JSP or Facelets. In the case of JSP the solution would have been more or less similar to the previous post (at the end JSP technology is there and it follows the same rules). But with Facelets the story is very different, because Facelets integrates seamlessly in the HTML templating system without any compilation. So in this entry the idea is loading the XHTML facelet templates dynamically without re-deploying the JavaEE application.
With JSF / Facelets the standard itself provides ways of extending the solution in order to create your own resource resolvers. The current JavaEE standard recommends to use a ResourceHandlerWrapper in order to provide your own resource handler implementation (a resource handler is a class to inject into the Facelets system your own templates or resources). Any JSF resource handler is intimate related with the URL class because they return resources as an URL. So, in order to implement your own resource handler you usually have to create in turn your own URL connection. For that purpose I followed this documentation and invented my memory protocol (syntax mem:/some/resource.xhtml). This memory protocol is very simple, the URL path will be read (as in the previous entries) from the application context as a resource (remember that this is a PoC, this protocol in a real scenario should read the templates from some external repository). The classes needed to create your own URL protocol are the following (see the previous link for further details):
sun.net.www.protocol.mem.Handler: The handler that manages my memory URL, it should extend the URLStreamHandler and, in this case, only overrides the openConnection method. That method locates the resource from the context of the application. The package and name of the class are imposed by the JavaSE standard.
es.rickyepoderi.jsf.MemoryURLConnection: The previous class returns a MemoryURLConnection, this class extends the URLConnection class and is extremely simple for this PoC. It is just a wrapper of the URL and the resource located and stored as an InputStream.
es.rickyepoderi.jsf.MemoryStreamHandlerFactory: This factory is used to manage memory URLs and should extend the URLStreamHandlerFactory class. The factory must be added to the JVM someway. In common JavaSE a user factory is added using URL.setURLStreamHandlerFactory but Tomcat already uses some custom factory so the container gives the developer the possibility of adding more factories using the method TomcatURLStreamHandlerFactory.addUserFactory.
With the memory URL implemented (a fake URL which just loads a resource from the context of the application), two new classes were developed in order to have my own JSF resource handler:
es.rickyepoderi.jsf.MemoryResourceHandler: The wrapper resource handler that locates the facelet templates from the context. JavaEE gives us a ResourceHandlerWrapper which wraps the default JSF handler and let us extend its functionality. In the PoC the handler just overrides the method createViewResource (the method for loading XHTML templates). The method first tries to locate the template inside the wrapped / default JSF handler but, if null is returned, a MemoryViewResource is created.
es.rickyepoderi.jsf.MemoryViewResource: The memory view resource is an extension of the ViewResource class that acts as a link between the Facelets system and the memory URL.
And that is all. The process is really cumbersome but the idea behind it is really simple. A new resource handler is implemented which has the mission of locating resources for Facelets templating system. In my example only the view resources are managed (the XHTML templates), but any other resource (images, CSS, JS files,...) can be retrieved overriding the other methods of the wrapper. As resource handlers return resources as URL objects, a new memory protocol was also created to represent my resources. Using a new URL protocol in Java is a bit crazy because you have to implement several classes and integrate the new protocol via the URL factory.
Adding the handler to the faces-config.xml file is the last step to activate it inside the application:
<application> <resource-handler>es.rickyepoderi.jsf.MemoryResourceHandler</resource-handler> </application>
With all the previous classes and the handler correctly specified in the JSF configuration file a JSF page can be requested and it is being server without having it in the common directory. The PoC handler tries to locate the page inside the context if it is not found by the default JSF handler. I am going to say it another time, the final target would be retrieving the XHTML template from an external repository, the PoC just retrieves it from another non common directory as an example.
Here it is a little video that shows the three scenarios presented in the series. First the Compiler Servlet is accessed and another one (CompiledServlet) is compiled and executed. Then two different JSP files are requested, both are located inside the application bundle, parsed, compiled and executed. And finally two facelets pages are rendered, the subject of this entry, using the customized handler the XHTML files are recovered from a non standard directory. The final Netbeans project CompileTest (without libs) is uploaded here, it contains all the classes presented throughout the series (it is attached in this last post because now you have understood all the scenarios). Remember that the application only works in tomcat 8 and JSP processing needs some weird steps (check the previous entry).
This last entry is much more interesting than the previous two. JSF / Facelets technology already takes into account the fact of implementing your own resource handler. Therefore the final solution is not as weird as the previous ones. With JSF and Facelets the XHTML templates can be obtained from another place easily, opening the possibilities of storing them inside any external repository. Using Facelets no compiling is necessary (a JSP needs to be compiled, this fact complicated the solution a lot). I think that JSF / Facelets is the best suited javaEE technology for the hot-replacement feature I was investigating. As I commented at the beginning of this series I wanted to explore a hot-replacement technique for a new idea I have in mind. JSF / Facelets is the winner, the template is dynamically loaded in a standard way and perfectly integrated inside the framework.
Best Regards!
Saturday, November 22. 2014
Script that adds Mode Lines to my TV
Another entry motivated by an oversight. My old desktop computer died some months ago and I decided to replace it with my old laptop (it was just packed in the box room). The desktop PC was connected directly to the TV and so is the laptop now. But my TV has a very nasty problem (thanks samsung), the EDID is incorrectly or not interchanged at all. In the desktop box I had to configure the mode lines manually using the xorg.conf file. I remember that I had to boot windows, install some weird application in order to get the current mode lines and add them to the file. When I switched to the laptop the situation was more or less the same, but in this case, I used the xrandr command. Besides I realized that another command, cvt, shows standard mode lines in VESA (that avoided me to repeat the horrible process with Windows). Finally I did a simple but working script that configured the new mode line, set it to the TV output and finally switch off the integrated screen. It was executed as soon as the laptop was switched on.
The other day I was cleaning the computer's hard drive (it was almost full with things that are now inside my new laptop and properly backed up), when I deleted the script (the fact that it was called lala.sh did not help much). So I had to re-write it again. It was just some minutes but I hate when things like that happen. This one will not happen again. Here it is my little script:
#!/bin/bash # For more details https://wiki.archlinux.org/index.php/xrandr #cvt 1360 768 # 1360x768 59.80 Hz (CVT) hsync: 47.72 kHz; pclk: 84.75 MHz #Modeline "1360x768_60.00" 84.75 1360 1432 1568 1776 768 771 781 798 -hsync +vsync # add the 1360x768 VESA mode (cvt) xrandr --newmode "1360x768_60.00" 84.75 1360 1432 1568 1776 768 771 781 798 -hsync +vsync # assign the mode to the TV xrandr --addmode VGA1 "1360x768_60.00" # set the mode, make TV primary and switch laptop screen off xrandr --output VGA1 --mode "1360x768_60.00" --primary --output LVDS1 --off
Now I will have to rethink the script no more for sure. Obviously I will lose other things but not this one. I am really a disaster and I feel like a fucking idiot when I have to do the same thing again and again... Which is quite common I must say.
Never again!
Saturday, November 15. 2014
Compiling Servlet (part II)
In the previous post I started the implementation of a little PoC about in-line compiling in a Java application. The first entry showed how a Servlet compiled another Servlet from source and used the resulting class to instantiate a new object that actually served the request. This idea is useful for hot replacement of certain classes without re-deploying the whole application bundle. Nevertheless using a Servlet java file as the replacement unit is quite radical, when I thought about this hot-replacement I was thinking in Java Server Pages (JSP) or any other modern Java technology.
A JSP is a way of generating dynamic HTML pages embedding java code inside an HTML template. This is an old standard in the JavaEE specification and I am sure that all of you know about them. Obviously storing JSP pages outside the WAR file for a later use is much simpler and easier to maintain. The JSP could be stored in another repository (DDBB, NoSQL, Cache system,...) and the application would use that JSP if it has been modified. As I said with JSP the in-line compilation idea is less cumbersome and easier to understand.
If you are familiar with JavaEE you already know that any Servlet container parses JSP files into a Java Servlet source file and then that file is compiled and executed. So, more or less, the container already performs the idea I am trying to explore. The main difference is that, by default, the container always expects the JSP files located in the File System (after de-archiving the deployment file). Usually they use temporary or working directories where intermediate Java source and class files are generated. Besides all the containers check for modifications in those JSP files and, if the requested JSP is modified, it compiles the file again before processing the request (this behavior can be disabled and usually it is not recommended for production environments, compiling is a heavy task).
At this point I managed two ideas:
Just relay in common container behavior and put something (a filter for instance) that previously reads the JSP from the repository and updates the current JSP in the file system if necessary. Doing that the container then would detect that the JSP had been modified and it would compile the file into a Servlet again.
Follow the idea presented in the previous entry, do all the job by myself.
The first point was the easiest solution but it has two main drawbacks: first, the container should be configured to detect changes (usually the container is only configured in development mode for that task) and, second, this idea manages files in the file system (thing that I tried to avoid).
So finally I decided to explore the solution of parsing and compiling the JSP by my own. The second part is already done (see the previous post) but the first part is a new one (parsing the JSP into a Servlet). Following the idea started in the previous entry, I have tried to use the jasper 2 JSP engine, the engine that tomcat container uses. Looking the code of the jasper implementation, the main ideas are the following:
Tomcat uses the JDTCompiler which is a implementation that internally uses the JDT / ECJ (the same Java compiler that I used in the previous entry).
The JDTCompiler extends the abstract Compiler class. The latter class performs the parsing phase (from the JSP to the Java Servlet Source) in the generateJava method, the second phase (compiling Java to class) is not implemented, the method generateClass is declared abstract (it is the JDTCompiler class the one that implements it using the ECJ compiler).
So I tried to implement another Compiler that extended the abstract class and overrided both methods (generateJava for generating the Java in memory, and generateClass using the SimpleCompiler developed in the previous entry). Then my class would substitute the default JDTCompiler. The main problem is that the generateJava method cannot be directly reused because of the fact that it uses a private method setupContextWriter to create the file writers. If that method could have been overridden the writer could be replaced by a memory one without more modifications. Instead of copying the whole method and replacing the writer inside it, I decided to compile my own version of the jasper.jar with that method declared as protected (I know it is a big shoddy job but this is just a PoC and it is much easier to code).
Once I could override that method I developed my MemoryCompiler that implements a jasper compiler but overrides both methods: the now protected setupContextWriter which construct a Writer from a ByteArrayOutputStream (in memory) and the generateClass which uses the SimpleCompiler to insert the resulting java class into memory.
I also needed to implement a MemoryJspCompilationContext which extends the default JspCompilationContext. It is very similar to the default one but there are minor differences to obtain the JSP source from a simple string instead from a file.
Finally I created a DefaultCompilerOptions which is a bunch of options that the jasper compiler uses but defaulted to the values needed by my compilations. The important method is the getCompilerClassName which returns the new and bright MemoryCompiler class which I implemented.
With all this changes my old CompilerServlet was extended in order to generate a Servlet class from a JSP using this method:
protected void processCompileJsp(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { try { String jspSource = readJspFile(request.getParameter("name")); DefaultCompilerOptions opts = new DefaultCompilerOptions(this.getServletContext()); JspRuntimeContext rtctx = new JspRuntimeContext(this.getServletContext(), opts); JspCompilationContext clctxt = new MemoryJspCompilationContext( "/" + request.getParameter("name") + ".jsp", jspSource, opts, this.getServletContext(), null, rtctx); org.apache.jasper.compiler.Compiler clc = clctxt.createCompiler(); clc.compile(true, true); Class clazz = ((MemoryCompiler) clc).getGeneratedClass(); if (javax.servlet.jsp.JspFactory.getDefaultFactory() == null) { // initialize in case Tomcat has not done yet javax.servlet.jsp.JspFactory.setDefaultFactory( new org.apache.jasper.runtime.JspFactoryImpl()); } HttpServlet servlet = (HttpServlet) clazz.newInstance(); servlet.init(this.getServletConfig()); servlet.service(request, response); } catch (Exception e) { throw new ServletException(e); } }
The idea is almost the same which I used to compile the Servlet source, now the JSP is read (again it is retrieved from the WAR file) and a compilation context is created using my MemoryJspCompilationContext. Besides the DefaultCompilerOptions sets the compiler to my MemoryCompiler class. This way the JSP is parsed and compiled using my own classes and only using memory arrays for both processes.
I tested the JSP compilation inside a tomcat 8 container and I needed to put some packages inside the general lib folder. The reason is that the internal classes of the Tomcat infrastructure also uses the Jasper engine, so there were conflicts between the both levels (application and tomcat system level). Finally I solved this problem placing some classes in the system level (not in the application /WEB-INF/lib directory), all the package es.rickyepoderi.compiler (the compiler classes used in the previous entry and the new created for this post) had to be moved from the application to a JAR in the general tomcat lib folder. Remember also to replace the original jasper.jar with my modified version (the one with the setupContextWriter method defined as protected)
In summary this entry is again a Frankenstein implementation which makes a Servlet to dynamically load, parse, compile and execute a JSP file (in the entry the file was obtained from the same WAR file but they could be retrieved from a database, a cache or any other repository). The solution is very botched cos the jasper JSP engine was modified in order to extend the setupContextWriter method and not generate any file during JSP processing. Besides the solution conflicted with Tomcat internal jasper library and some application classes should have been moved to a JAR in the main lib directory. But finally the CompilerServlet can execute both, Servlet source files or JSP, re-compiling them on the fly and with no intermediary files in the File System. Take in mind that this PoC is very bound to the Tomcat container (changes would be needed in order to work in any other JavaEE server).
Again (and this time it makes even more sense), don't try this at home!
Comments