Blogs de Nologin

We are the time we left: Novag Citrine under Debian GNU/Linux

nospam@example.com (Aitor Acedo) — dom, 29 ene 2012 17:05:31 +0000

This is my first blog entry and I'd like to explain the steps that I followed to play chess with my Novag Citrine connected to my Debian GNU/Linux wheezy box.
The main reason to connect Citrine with the PC is able to store my games against the machine and this way I can analyze all movements and learn what I did wrong. Moreover you could choose other engines installed in your PC (GNU Chess 4/5, Crafty, Sjeng, ...) to play your games using citrine board as a simple interface, even play on Free Internet Chess Server with the board.

Really it's not very difficult achieve that Citrine 'talk' with GNU/Linux, mainly because there is a user space driver developed by Maximiliano Pin here that works very well, so in this entry I only want to collect instructions, step by step, followed in my system.

1. Download novagdrv driver from here:

$ wget http://download.savannah.gnu.org/releases/novagdrv/novagdrv-0.3.tgz

$ wget http://download.savannah.gnu.org/releases/novagdrv/novagdrv-0.3.tgz.sig

2. In order to verify program which we just downloaded, we must download the project keyring from here
File downloaded novagdrv-keyring.gpg

3. Import project keyring (public keys from project members) into your public key store.

$ gpg --import novagdrv-keyring.gpg

gpg: key 6002A0BC: public key "Maximiliano Pin " imported

gpg: Total number processed: 1

gpg:               imported: 1  (RSA: 1)

4. Verify downloaded file:

$ gpg --verify novagdrv-0.3.tgz.sig novagdrv-0.3.tgz

gpg: Good signature from ...

5. After verify successfully this file we are going to decompress and compile the driver.

$ tar xvzf novagdrv-0.3.tgz

$ cd novagdrv-0.3

$ make

g++ -g   -c -o main.o main.cc

g++ -g   -c -o novag.o novag.cc

g++ -g   -c -o serial.o serial.cc

g++ -g   -c -o game.o game.cccode>

g++ -g   -c -o position.o position.cc

g++ -g -o novagdrv main.o novag.o serial.o game.o position.o

6. When we take a look in current directory we discover that novagdrv binary file has been created, and we move binary file into a /usr/local/bin, for example (any place inside our PATH environment variable):
# mv novagdrv /usr/local/bin

7. The driver uses /dev/ttyS0 as default COM device, so if we connect Citrine to other device we previously must create a text file $HOME/.novagdrvrc with this content:
device=/dev/ttyUSB0
We could use a USB to serial RS232 adapter if your computer hasn't a COM device.

8. In order to test the driver you can connect Citrine and run the binary file and if all it's OK you'll see the movements made in the board written on the terminal.

$ novagdrv &

e4-e5...

Once driver works we need applications to analyze and play games. In the first place we are going to install ChessDB.

CHESSDB
First of all we download sources from here.
We should get MD5 and SHA1 fingerprints from its download page.

MD5: 6aee2e4eca26576cbe6e63dc6b8d55db

SHA1: 456ff217ebdb0f6231a507ce737747c964f77f6e

1. Again we check downloaded file integrity.

$ echo "6aee2e4eca26576cbe6e63dc6b8d55db  ChessDB-3.6.18.tar.gz" > chessdb.md5

$ md5sum -c chessdb.md5

ChessDB-3.6.18.tar.gz: OK

Also we can check with SHA1 checksum, although obviously it's not necessary verify twice downloaded file.

$ echo "456ff217ebdb0f6231a507ce737747c964f77f6e  ChessDB-3.6.18.tar.gz" > chessdb.sha1

$ sha1sum -c chessdb.sha1

ChessDB-3.6.18.tar.gz: OK

2. Decompress and apply patch found with novag user space driver:

$ tar xvzf ChessDB-3.6.18.tar.gz

$ patch -p1  ../../driver/novagdrv-0.3/ChessDB-3.6.18-novagdrv.patch

patching file Makefile.conf

patching file Makefile.mingw

patching file tcl/file.tcl

patching file tcl/lang/english-basic.tcl

patching file tcl/menus.tcl

patching file tcl/start.tcl

patching file tcl/tools/novagdrv.tcl

Be sure that our system satisfies all software requirements to be able to compile sources files, in my case I had to install two packages:
# aptitude install tcl8.5-dev tk8.5-dev
If you want to test your Tcl/Tk version you can run:

$ tclsh

% info patchlevel

8.5.11

3. Edit configure script to add a new location for X11 libraries (I had forgotten to say that my Debian is 64 bits version):

set x11Path {

        /usr/lib/x86_64-linux-gnu

        ...

}

$ ./configure

configure: Makefile configuration program for ChessDB

    Renaming "Makefile" to "Makefile.bak"

    Tcl/Tk version: 8.5

    Your operating system is: Linux 3.1.0-1-amd64

    Location of "tcl.h": /usr/include/tcl8.5

    Location of "tk.h": /usr/include/tcl8.5

    Location of Tcl 8.5 library: /usr/lib

    Location of Tk 8.5 library: /usr/lib

    Location of X11 library: /usr/lib/x86_64-linux-gnu

The Makefile configured for your system was written.

Now just type "make" to compile ChessDB.

# make install

And that's all to ChessDB Debian installation
$ chessdb

Tools -> Novag Chess Computer.

EBOARD
Installation procedure to Eboard is very similar to the previous one so we're going to comment only some steps.
Download source code from here

1. Apply novag patch:

$ tar xvjf eboard-1.1.1.tar.bz2

$ cd eboard-1.1.1

$ patch -p1  ../../driver/novagdrv-0.3/eboard-1.1.1-novagdrv.patch

2. Prepare to compilation process:

$ ./configure

configuring eboard 1.1.1...

checking sanity of install... ok

testing C++ compiler...

  trying g++ ... it works

header verification:

  all headers found.

header verification:

  sys/audioio.h       : no

checking for IPPROTO_TCP in netinet/in.h... yes

checking for TCP_NODELAY in netinet/in.h... no

checking for SOL_TCP in netinet/in.h... no

checking for IPPROTO_TCP in netinet/tcp.h... no

checking for TCP_NODELAY in netinet/tcp.h... yes

checking for SOL_TCP in netinet/tcp.h... yes

  net options: netinet/tcp.h required, IPPROTO_TCP present.

looking for pkg-config... /usr/bin/pkg-config

looking for GTK+ version... 2.24.8, ok

looking for libpng... 1.2.46, ok

header verification:

  all headers found.

wrote config.h

wrote config.make

writing Makefile... ok

creating eboard-config... ok

creating eboard.spec... ok



Summary:

eboard version 1.1.1

binaries  will be installed to    /usr/local/bin

man pages will be installed under /usr/local/man

datafiles will be installed to    /usr/local/share/eboard

NLS support: yes

done.

I found a problem compiling eboard patched sources, message printed out for g++ compiler was:

Invalid conversion from `const char*' to `char*'

3. My dirty solution at this point to fix the problem was to remove 'const' restriction in append method editing two files ntext.h and ntext.cc:

$ vim ntext.h

//void append(const char *text, int len, int color);

void append(char *text, int len, int color);

$ vim ntext.cc

//void NText::append(const char *text, int len, int color) {

void NText::append(char *text, int len, int color) {

# make clean && make install

$ eboard -novagport /dev/ttyUSB0

Now it's time to enjoy your Citrine!

Ricky's Hodgepodge: Moving Forward to OpenJDK

nospam@example.com (rickyepoderi) — s�b, 28 ene 2012 13:35:00 +0000

If you do not know, Oracle removed the DLJ license from its closed-source JDK implementation some months ago (Java7 was born without the license and Java6 removed it in U29). The Operating System Distributor License for Java (DLJ) was a Sun initiative to facilitate the distribution of the JDK/JRE with operating systems based on OpenSolaris or Linux, in general that license let the distros to repackage, distribute and use the JDK.

The reasons why Oracle removed that license were controversial. Some people thought that Oracle was definitely pushing forward the OpenJDK adoption and others saw dark maneuvers in this change. For one reason or another the truth is Debian (and all the rest of linux distributions) is starting the transition from sun-java6-* to openjdk-6-* packages. In my humble opinion OpenJDK is a very good virtual machine and both implementations, OpenJDK and Oracle closed-source JDK, have a lot in common (Joe Darcy talked about this issue last summer in the OSCON convention). As you already know the only big difference for end-users and developers is the Java Web Plugin, I wrote before an entry about this particular issue, commenting the story and the problems of OpenJDK related to the web-plugin implementation.

I am on vacation right now and I was requested to encrypt my working laptop long time ago (because of security reasons and company procedures), so today I have re-installed my box. Now dual boot linux/windows is not allowed and, in general, the recommended linux setup was so different to the one I had that I have preferred to wipe it out. Actually this is the first time I have an only debian installation. As a collateral effect I have decided to move forward with java (I think it is the correct time cos I have been finishing all my current projects), openjdk-6-jdk (with icedtea plugin of course) is now my default java implementation. I also think that it is a quite early to start with openjdk-7-jdk, I will wait till U2 or U4 arrive to wheezy.

I hope I do not need to install Oracle JDK back again, I really hate installing tar bundles in my Debian boxes.

OpenJDK is here to stay!

Ricky's Hodgepodge: Simple but Full Glassfish HA Using Debian

nospam@example.com (rickyepoderi) — s�b, 21 ene 2012 12:27:00 +0000

During the past months I has been working in a project of which environment is mainly Windows (that is the reason of the previous entry about windows mod_proxy_html compilation ). Talking with the customer I realized that Windows 2008 provides a simple and out of the box Load Balancing / High Availability mechanism for TCP/IP client/server infrastructures called Network Load Balancing Services (NLBS). The customer complained about the lack of a similar solution in Linux (remember this customer is pro-windows). During my working life I have always used hardware load balancers in those situations, so I have never investigated how to implement a simple LB/HA solution which only deals with the OS. If you remember I presented an entry about HA in application servers and using an in-memory repository for session management some months ago. Today entry is less ambitious and more practical, I just want to setup a full Glassfish HA setup using only Debian (full means with no single point of failure).

The solution

Looking for information about the problem in the internet, I found lots of LB/HA sample solutions for TCP/IP enviroments in Linux, most of the times they involve two products (see for example this entry):

keepalived. This package provides a VRRP (Virtual Router Redundancy Protocol) implementation stack and a versatile health checking system. So this software can commute a virtual IP between two or more hosts, the health checking system is used to test the servers and decide which one is the master and which one is the backup server. In short keepalived provides High Availability for an IP. In linux there are other more complicated HA techniques to get a full and typical cluster (shared storage, typical virtual -eth0:1 style- IPs,...) but I really do not know much about it.
haproxy. A TCP/HTTP software load balancer that, obviously, adds the Load Balancing part to the solution. In internet examples haproxy is commonly used cos it is generic (it can distribute the load for any TCP/IP protocol: LDAP, HTTP, SMTP, IMAP,...).

But in this entry a particular case is shown, a Java Application Server (Glassfish), and it is clear that there are better techniques for load balancing two Java Servers. For this reason my solution will use two other packages instead of haproxy:

mod_jk. Finally I decided to use the Apache Tomcat Connector as my software load balancer. Glassfish is a very nice piece of software and it has several connectors that can be added to an Apache server to perform this function:
- mod_proxy_http: General HTTP reverse proxy module for Apache which supports load balancing.
- mod_proxy_ajp: Similar to mod_proxy_http but it manages Apache JServ Protocol version 1.3 (AJP13) to talk with the App Server (Glassfish supports AJP13).
- mod_jk: The chosen module also uses AJP13 to connect Apache and the App Server but while mod_proxy_ajp is developed by the httpd group mod_jk is developed by tomcat guys. Usually it is more configurable and better in terms of failure detection.
- HTTP Glassfish Load Balancer Plug-in. This plugin is the official Oracle Glassfish load balancer module which is not available in the Open Source Edition. For this reason it was not selected for the demo.

Apache. The famous HTTP web server which is necessary to install mod_jk inside it.

So the solution is quite clear now. First of all keepalived gives a virtual IP which commutes between two Apache/mod_jk boxes (only one httpd server works actively, the other one acts as a backup). The package uses scripts to check if the web server is working fine. The httpd box which has the active IP receives the requests and apache/mod_jk pair redistributes them between two Glassfish servers. So the first layer provides only HA and the second layer LB/HA (take into account that the work in the first layer is lighter, the hard process is performed by the App Server).

My demo environment is the simplest one. Two debian wheezy boxes were virtualized (KVM) inside my laptop: debian1 (192.168.122.21) and debian2 (192.168.122.22). The Linux distribution was installed with the minimal number of packages. Each box will have an Apache/mod_jk and a Glassfish server. Cos the Glassfish 3.1.1 is configured as a cluster debian1 runs the Domain Administrator Server (DAS) too. Replicated session is used (session is sent from one instance to the other when it is changed) but mod_jk uses sticky distribution (the first request is distributed but all the rest from the same source are sent to the same Glassfish server). The keepalived software manages a virtual IP (192.168.122.23) which is assigned to the master host (host with the highest priority). I present a little diagram of my demo solution.

OpenJDK Installation

A Debian solution deserves OpenJDK (the open source JVM):

# apt-get install openjdk-6-jdk

Glassfish version 3.1.1 is going to be used and it is known that there are problems with old versions of JavaSE6. That was the reason to choose wheezy (OpenJDK 6b24) instead of squezze (6b18). JDK was installed in both machines.

Glassfish Installation

The Glassfish installation, although it is not very complicated, requires several steps (I have followed this guide).

I prefer to run glassfish with its own user. So first of all a system user and group were created.
```
# groupadd glassfish
# useradd -c "Glassfish User" -d /opt/glassfish -g glassfish -m -s /bin/bash glassfish
# passwd glassfish
```
This step was done in both machines.
First the linux distribution for Glassfish OpenSource Edition 3.1.1 was downloaded and installed.
```
$ wget http://download.java.net/glassfish/3.1.1/release/glassfish-3.1.1-unix.sh
$ bash glassfish-3.1.1-unix.sh
```
Please choose to only install the software (do not configure it). I do not like to put installation screens (I am sure that all of you are intelligent enough to install it). I remark only two questions, where to install it (/opt/glassfish/glassfish3.1.1) and what java to use (/usr/lib/jvm/java-6-openjdk-amd64). I completed the installation in both machines.

A domain (domain1) was then created in debian1 (all asadmin commands were launch from /opt/glassfish/glassfish3.1.1/bin).

$ ./asadmin create-domain --savemasterpassword=true --savelogin=true domain1
Enter admin user name [Enter to accept default "admin" / no password]> admin
Enter the admin password [Enter to accept default of no password]> 
Enter the admin password again> 
Enter the master password [Enter to accept default password "changeit"]> 
Enter the master password again> 
Using default port 4848 for Admin.
Using default port 8080 for HTTP Instance.
Using default port 7676 for JMS.
Using default port 3700 for IIOP.
Using default port 8181 for HTTP_SSL.
Using default port 3820 for IIOP_SSL.
Using default port 3920 for IIOP_MUTUALAUTH.
Using default port 8686 for JMX_ADMIN.
Using default port 6666 for OSGI_SHELL.
Using default port 9009 for JAVA_DEBUGGER.
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=debian1.demo.kvm,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US]
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=debian1.demo.kvm-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US]
No domain initializers found, bypassing customization step
Domain domain1 created.
Domain domain1 admin port is 4848.
Domain domain1 admin user is "admin".
Login information relevant to admin user name [admin]
for this domain [domain1] stored at
[/opt/glassfish/.asadminpass] successfully.
Make sure that this file remains protected.
Information stored in this file will be used by
asadmin commands to manage this domain.
Command create-domain executed successfully.

In v3 there are no profile types, all domains are equal. After creation domain1 was secured (console uses https instead of plain communication).

$ ./asadmin enable-secure-admin 
Command enable-secure-admin executed successfully.

As Glassfish is going to use two working instances inside a cluster I usually delete listeners and servers which are not used by the web console.

$ ./asadmin delete-http-listener http-listener-1
Command delete-http-listener executed successfully.

$ ./asadmin delete-http-listener http-listener-2
Command delete-http-listener executed successfully.

$ ./asadmin delete-virtual-server server
Command delete-virtual-server executed successfully.

Now Glassfish v3 uses SSH to communicate with remote instances. So debian2 has to be added to the domain (public key is exchanged for future commands):

$ ./asadmin setup-ssh --generatekey=true debian2.demo.kvm
Enter SSH password for glassfish@debian2.demo.kvm> 
Created directory /opt/glassfish/.ssh
/usr/bin/ssh-keygen successfully generated the identification /opt/glassfish/.ssh/id_rsa
Copied keyfile /opt/glassfish/.ssh/id_rsa.pub to glassfish@debian2.demo.kvm
Successfully connected to glassfish@debian2.demo.kvm using keyfile /opt/glassfish/.ssh/id_rsa
Command setup-ssh executed successfully.

And the new node (debian2) was installed:

$ ./asadmin install-node debian2.demo.kvm
Created installation zip /opt/glassfish/glassfish3.1.1/bin/glassfish298897142999171245.zip
Successfully connected to glassfish@debian2.demo.kvm using keyfile /opt/glassfish/.ssh/id_rsa
GlassFish is already installed on debian2.demo.kvm under /opt/glassfish/glassfish3.1.1.
Command install-node executed successfully.

Cos glassfish was installed in both machines this command does nothing.

As I said the remote instances are managed via SSH, for this reason there is a new concept called SSH node. These nodes are very different from previous node-agents, they are not a Java daemon at all, they are just remote hosts which run instances of the domain (so they are a concept and not a real daemon or process). Two nodes, called debian1-ssh and debian2-ssh, were created.
```
$ ./asadmin create-node-ssh --nodehost localhost debian1-ssh
Command create-node-ssh executed successfully.

$ ./asadmin create-node-ssh --nodehost debian2.demo.kvm debian2-ssh
Command create-node-ssh executed successfully.
```

Then the cluster (cluster1) and the two instances (debian1-gf and debian2-gf) were also created (each instance was registered in the previous nodes and inside the cluster):

$ ./asadmin create-cluster --systemproperties HTTP_SSL_LISTENER_PORT=8181:HTTP_LISTENER_PORT=8080 cluster1
Command create-cluster executed successfully.

$ ./asadmin create-instance --cluster cluster1 --node debian1-ssh debian1-gf
Command _create-instance-filesystem executed successfully.
Port Assignments for server instance debian1-gf: 
JMX_SYSTEM_CONNECTOR_PORT=18686
JMS_PROVIDER_PORT=17676
ASADMIN_LISTENER_PORT=14848
HTTP_LISTENER_PORT=8080
JAVA_DEBUGGER_PORT=19009
IIOP_SSL_LISTENER_PORT=13820
IIOP_LISTENER_PORT=13700
OSGI_SHELL_TELNET_PORT=16666
HTTP_SSL_LISTENER_PORT=8181
IIOP_SSL_MUTUALAUTH_PORT=13920
The instance, debian1-gf, was created on host localhost
Command create-instance executed successfully.

$ ./asadmin create-instance --cluster cluster1 --node debian2-ssh debian2-gf
Command _create-instance-filesystem executed successfully.
Port Assignments for server instance debian2-gf: 
JMX_SYSTEM_CONNECTOR_PORT=18686
JMS_PROVIDER_PORT=17676
ASADMIN_LISTENER_PORT=14848
HTTP_LISTENER_PORT=8080
JAVA_DEBUGGER_PORT=19009
IIOP_SSL_LISTENER_PORT=13820
IIOP_LISTENER_PORT=13700
OSGI_SHELL_TELNET_PORT=16666
HTTP_SSL_LISTENER_PORT=8181
IIOP_SSL_MUTUALAUTH_PORT=13920
The instance, debian2-gf, was created on host debian2.demo.kvm
Command create-instance executed successfully.

Once the instances were running (start-local-instance executed in each node), a new http-listener was registered to listen for the AJP13 protocol (this port will be used later inside mod_jk workers configuration). More details about this configuration in this link:

$ ./asadmin create-http-listener --listenerport 8009 --listeneraddress 0.0.0.0 --defaultvs server --target cluster1 jk-connector
Command create-http-listener executed successfully.

$ ./asadmin set configs.config.cluster1-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true
debian1-gf:
configs.config.cluster1-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true

debian2-gf:
configs.config.cluster1-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true

configs.config.cluster1-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true
Command set executed successfully.

In order to set the jvmRoute some java and system properties were added (mod_jk uses this property to perform sticky load balancing, in the first request the value of the jvmRoute is appended to the JSESSIONID cookie and the module will use this value to redirect the following requests to the same server).
```
$ ./asadmin create-jvm-options --target cluster1 "-DjvmRoute=\${AJP_INSTANCE_NAME}"
debian1-gf:
Created 1 option(s)

debian2-gf:
Created 1 option(s)

Created 1 option(s)
Command create-jvm-options executed successfully.

$ ./asadmin create-system-properties --target debian1-gf AJP_INSTANCE_NAME=debian1
Command create-system-properties executed successfully.

$ ./asadmin create-system-properties --target debian2-gf AJP_INSTANCE_NAME=debian2
Command create-system-properties executed successfully.
```
In glassfish all instances under a cluster share the same configuration, for this reason different values cannot be set to the same java property in each instance. So the trick is to set a different system property (these props can be defined by instance) and assign the java property referencing the system prop.
Finally the famous clusterjsp was deloyed with high availability enabled (by default glassfish uses session replication). It seems that clusterjsp application is not part of the glassfish 3.1.1 distribution (at least in the linux no multi-language bundle I used), so I created this simple clusterjsp.war (from a previous ear I have from v2 bundle).
```
$ ./asadmin deploy --availabilityenabled=true --target cluster1 ~/clusterjsp.war 
Application deployed with name clusterjsp.
```

At that time both instances could only be managed with *-local-instance commands (launching the commands from its own host). I realized that that was because the master password (password for JKS keystores) was not locally saved (although I swear that I had added savemasterpassword in the domain creation command). To solve that I changed the password in both hosts (I put the same old password cos I just wanted it to be saved in the ~/.asadmintruststore file):

$ ./asadmin change-master-password --savemasterpassword true debian1-ssh
Enter the old master password>
Enter the new master password> 
Enter the new master password again> 
Command change-master-password executed successfully.

$ ./asadmin change-master-password --savemasterpassword true debian2-ssh
Enter the old master password>
Enter the new master password> 
Enter the new master password again> 
Command change-master-password executed successfully.

Finally to not use any password in commands at debian2 machine I saved the user and password information as well in this second host (the user and password is saved in ~/.asadminpass file):

$ ./asadmin --host debian1.demo.kvm --port 4848 login
Enter admin user name [default: admin]> admin
Enter admin password> 
Login information relevant to admin user name [admin]
for host [debian1.demo.kvm] and admin port [4848] stored at
[/opt/glassfish/.asadminpass] successfully.
Make sure that this file remains protected.
Information stored in this file will be used by
asadmin commands to manage the associated domain.
Command login executed successfully.

Finally I created a simple startup script which needs to be placed in /etc/init.d/glassfish-inst. This script launches a single instance (DAS instance is not needed to be started at boot time, so only the cluster instance will be started in each machine) and it needs asadmin command to not prompt for anything (see the previous step to save the passwords). After the file is correctly placed and the needed variables changed it must be registered in debian.
```
# chmod 755 /etc/init.d/glassfish-inst
# update-rc.d glassfish-inst defaults
update-rc.d: using dependency based boot sequencing
```

So now we have a clustered (session replicated) application. Besides each cluster instance listens for AJP13 protocol in port 8009. Everything is ready for Apache/mod_jk configuration. In this demo I chose Glassfish but the main concepts (maybe changing some pieces) are the same for any application server. And as a final comment for this part, you already know that I personally recommend not to replicate sessions, this process has a big impact if the sessions are too many, too big and/or too volatile (but also be aware that in that case a server lost means a session lost).

Apache/mod_jk Installation

Both packages are part of the main Debian repository, so installing them is just a command like this:

# apt-get install apache2 libapache2-mod-jk

Configuration for JK module is placed in /etc/apache2/mods-available/jk.conf. I just used the debian default but commenting /jk-status and /jk-manager locations (I decided to define them inside the site). This config file points to /etc/libapache2-mod-jk/workers.properties as the workers configuration file. The important lines in the second file are the following:

worker.list=loadbalancer,jk-status
worker.debian1.port=8009
worker.debian1.host=debian1.demo.kvm
worker.debian1.type=ajp13
worker.debian1.lbfactor=1
worker.debian2.port=8009
worker.debian2.host=debian2.demo.kvm
worker.debian2.type=ajp13
worker.debian2.lbfactor=1
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=debian1,debian2
worker.loadbalancer.sticky_session=1
worker.jk-status.type=status

There are two workers defined, debian1 and debian2 (the name of the worker has to be the same defined in the jvmRoute property), using AJP13 protocol on both virtual hosts with port 8009. Another worker loadbalancer is used to distribute requests between the two previously defined workers. The distribution is configured sticky. Finally the status worker (a page that informs about worker status) is defined with the name jk-status. No special tuning (timeouts and more) was done.

Once mod_jk is configured the default site defined in /etc/apache2/sites-available/default adds two new locations to proxy the clusterjsp app and the status worker:

    <Location /jk-status>
        # Inside Location we can omit the URL in JkMount
        JkMount jk-status
        #Order deny,allow
        #Deny from all
        #Allow from 127.0.0.1
    </Location>

    # mount clusterjsp
    JkMount /clusterjsp loadbalancer
    JkMount /clusterjsp/* loadbalancer

Finally the mod_jk is enabled and apache2 restarted:

# a2enmod jk
# /etc/init.d/apache2 restart

These steps have to be done in both hosts (debian1 and debian2). And, at this moment, both Apache servers are balancing the two glassfish clustered instances. As mod_jk detects application server failures the LB/HA of glassfish layer is achieved.

Keepalived Installation

The final package, keepalived, is installed following the debian way:

# apt-get install keepalived

This software has just a little configuration located in /etc/keepalived/keepalived.conf. This file just contains the following:

vrrp_script chk_http_port {   # Requires keepalived-1.1.13
  script "wget -q -T 1.0 -t 2 --delete-after -O /tmp/test.wget http://localhost:80/index.html"
  interval 5            # check every 5 seconds
  weight 2              # add 2 points of prio if OK
}

vrrp_instance VI_1 {
  interface eth0
  state MASTER          # MASTER debian1, BACKUP debian2
  virtual_router_id 51  # same id in both hosts
  priority 101          # 101 on master, 100 on backup
  virtual_ipaddress {
    192.168.122.23      # virtual IP
  }
  track_script {
    chk_http_port
  }
}

A VRRP instance is defined in eth0 interface. This instance manages the IP 192.168.122.23. Host debian1 is declared MASTER with an initial priority of 101, debian2 is the backup instance with a priority of 100. A script chk_http_port is executed in both hosts and it assigns two points more. This way if debian1 Apache fails it only has 101 points while debian2 has 102 (100 + 2 added by the check) and the backup server is transitioned to master (the virtual IP moves from debian1 to debian2). As soon as debian1 Apache runs again (103 points are now assigned to debian1) this host become the master again. If whole debian1 fails (a hardware failure for example), the multicast packets, which are supposed to be sent by the master server, stop being received by the backup server and therefore its status will be risen to master (check VRRP protocol specification in RFC 2338). Take in mind that keepalived does not use typical virtual IPs (eth0:1 and so on), it does HA using VRRP which manages multicast and ARP. VRRP only works inside the same network (both machines have to be inside the same net). With keepalived the first layer is configured in HA (only one apache is working, the other is just awaiting as a backup server).

The checking script is very simple, using wget a test page from the Apache is retrieved (timeout of one second but with two tries). If wget returns 0, the page was got successfully, in case of any other return code, the page was not returned and the check fails. Never test a glassfish page (doing that you are mixing layers and the results can be very weird). Here you have master configuration for debian1 and backup for debian2.

Conclusion

As a conclusion I am going to present a video that shows my demo installation. By default debian1 is the master vrrp server and if clusterjsp page is requested, its apache/mod_jk redirects to any of the glassfish servers (debian1 in the video). Cos the module is configured sticky I can set some attributes in the session and debian1 is always my working glassfish. Now debian1-gf glassfish instance is stopped and clusterjsp is redirected to debian2 (now debian1 apache and debian2 glassfish are working). But then debian1 apache is stopped, debian2 transitions to master state (a tail for /var/log/messages displays keepalived information) and application is still working (now debian2 is doing all the job, apache and glassfish). After restarting debian1 Apache, this host becomes again the master. Finally the virtual machine of debian1 is suddenly stopped, debian2 transitions again to master. Because clusterjsp application was deployed with session replication the session is never lost in the video.

Your browser doesn't support the video tag. See this entry for more information about how to see the videos in this blog. You can download the file instead.

For me it is clear that linux has also a good and simple LB/HA mechanism, keepalived is usually more than enough in typical TCP/IP client/server solutions. With this configuration a virtual IP that commutes between nodes is setup (only HA at the first layer). This layer consists in a software load balancer and, as I commented, usually generic haproxy is used. Nevertheless apache/mod_jk is a much proper solution for AJP13 capable Java Application Servers. The second layer (glassfish in this case) has LB/HA features, cos the load balancer (mod_jk) distributes the load between the two instances at the same time it checks the server status. So the second layer processes requests in parallel. The problem for my customer was that their linux boxes were RedHat6 and keepalived is not distributed by default in the distro (it seems that cluster extra software is needed in order to get this package).

That's why Debian rules!

Ricky's Hodgepodge: Compiling mod_proxy_html in Linux and Windows

nospam@example.com (rickyepoderi) — vie, 06 ene 2012 12:02:00 +0000

These weeks I am finishing a long project which uses Apache (more precisely OHS / Oracle HTTP Server) in a reverse proxy configuration. Usually default apache mod_proxy modules are more than enough to configure a good reverse proxy but, sometimes, a special module called mod_proxy_html is necessary. When the pages served by the backend server manage absolute links (the ones that start by / or by the complete protocol://host:port uri) typical mod_proxy configuration falls short, because those mods never parse or change the HTML code (just the headers). Obviously mod_proxy_html does exactly that, parsing and replacing the conflicting links in the html page. It is important to remark that this behavior is not recommended, take in mind that the cost of parsing every HTML is not small.

My initial idea was only using the module in those applications which were problematic, there were no other solution (like specific application server plugin or a smart proxy uri that fits with the final backend server location) and the customer did not want to modify. But the problem is that mod_proxy_html is not distributed with the default Apache source bundle (it seems that it will be integrated in forthcoming Apache 2.4 cos it was donated by its creator to the foundation but currently it should be installed separately). All linux distros distribute the module as a separate package (because, as I explained, it is quite important in some reverse proxy configurations) but this is not the case of OHS. So my only chance was compiling the module by myself.

Although custom modules are not supported, OHS provides the apxs command to add them to the server at customers own risk and I desperately needed a plan B just in case an app was problematic. But the other painful point was that my OHS server is running in a Windows 2008 host. Cos I have no experience at all compiling in Windows I decided to start smoothly: compiling mod_html_proxy in an Apache/debian installation, then in a Linux OHS and finally in a Windows OHS. I compiled 3.0.1 version of the module and not current 3.1.2 for several reasons: new version uses two modules (I did not want to compile two times), my first try with 3.1.2 did not work as expected (I spent short time with the problem) and it is the current version in debian (you already know my total confidence in this distribution).

Adding mod_proxy_html to Debian/Apache

Although debian has a libapache2-mod-proxy-html package I compiled it by myself downloading the debian source package (remember I was training to compile it in OHS later). In order to do that I needed first some development packages: apache and libxml (this module uses libxml to parse the HTML pages and perform the replacements):

# apt-get install apache2-prefork-dev libxml2-dev

Then the module was compiled and installed:

# apxs2 -c -I /usr/include/libxml2 -I . -i mod_proxy_html.c 
/usr/share/apr-1.0/build/libtool --silent --mode=compile --tag=disable-static x86_64-linux-gnu-gcc -prefer-pic -DLINUX=2 -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_REENTRANT -I/usr/include/apr-1.0 -I/usr/include/openssl -I/usr/include/xmltok -pthread     -I/usr/include/apache2  -I/usr/include/apr-1.0   -I/usr/include/apr-1.0  -I/usr/include/libxml2 -I.  -c -o mod_proxy_html.lo mod_proxy_html.c && touch mod_proxy_html.slo
/usr/share/apr-1.0/build/libtool --silent --mode=link --tag=disable-static x86_64-linux-gnu-gcc -o mod_proxy_html.la  -rpath /usr/lib/apache2/modules -module -avoid-version    mod_proxy_html.lo
/usr/share/apache2/build/instdso.sh SH_LIBTOOL='/usr/share/apr-1.0/build/libtool' mod_proxy_html.la /usr/lib/apache2/modules
/usr/share/apr-1.0/build/libtool --mode=install cp mod_proxy_html.la /usr/lib/apache2/modules/
libtool: install: cp .libs/mod_proxy_html.so /usr/lib/apache2/modules/mod_proxy_html.so
libtool: install: cp .libs/mod_proxy_html.lai /usr/lib/apache2/modules/mod_proxy_html.la
libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin" ldconfig -n /usr/lib/apache2/modules
----------------------------------------------------------------------
Libraries have been installed in:
   /usr/lib/apache2/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
chmod 644 /usr/lib/apache2/modules/mod_proxy_html.so

Some configuration files were created to include the custom module in a2enmod/a2dismod debian commands. So I included the /etc/apache2/mods-available/proxy_html.load and /etc/apache2/mods-available/proxy_html.conf (load the module and default configuration). Once the module was integrated in debian scripts I enabled all the needed ones to perform reverse proxying:

# a2enmod proxy proxy_connect proxy_http proxy_ftp proxy_html

Finally I setup a Location directive which performed a reverse proxy from /proxy-test/ to a Tomcat running in my laptop (I added it to the default site, /etc/apache2/sites-enabled/000-default).

<Location /proxy-test/>
    ProxyPass http://magneto:8080/
    ProxyPassReverse http://magneto:8080/
    SetOutputFilter proxy-html
    ProxyHTMLURLMap http://magneto:8080/ /proxy-test/
    ProxyHTMLURLMap / /proxy-test/
</Location>

The location proxifies (pass and reverse) all requests from the /proxy-test/ uri to my tomcat installation but with a filter, the proxy-html one. This filter searches and replaces the two annoying absolute links with our location uri (this is the goal of the ProxyHTMLURLMap directive). If you need more examples about the configuration please mod_proxy_html#Example_5-_mod_proxy_html">mod_proxy_html#Example_5-_mod_proxy_html">mod_proxy_html#Example_5-_mod_proxy_html">check this page.

And that was all! The Apache worked as a reverse proxy perfectly. I also prepared a simple html test page with some conflicting links to test.

Adding mod_proxy_html to Linux/OHS

The second step was doing the same but with OHS in a Linux box. I installed a new Linux KVM virtual box, OHS 11.1.1.5.0 binaries and perform the same actions. This time oracle system user is used for compiling and installing, and some parameters are different (take into account that I am using the libxml provided by OHS and not the system one):

$ export ORACLE_HOME=/opt/oracle/middleware/Oracle_WT1
$ export ORACLE_INSTANCE=$ORACLE_HOME/instances/instance1
$ export CONFIG_FILE_PATH=$ORACLE_INSTANCE/config/OHS/ohs1
$ export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/ohs/lib:$LD_LIBRARY_PATH

$ /opt/oracle/middleware/Oracle_WT1/ohs/bin/apxs -I /usr/include/libxml2  -I . -L/opt/oracle/middleware/Oracle_WT1/ohs/lib -lxml2 -c -o mod_proxy_html.so -i mod_proxy_html.c
/opt/oracle/middleware/Oracle_WT1/ohs/build/libtool --tag=CC --mode=compile cc -O    -DNO_RC2 -DNO_RC5 -DNO_IDEA -DBSAFE -fPIC  -DLINUX=260 -DMOD_SSL=206104 -DMOD_PERL -DUSE_PERL_SSI -I/include -DEAPI  -D_LARGEFILE64_SOURCE -DUSE_EXPAT -I../lib/expat-lite -I/opt/oracle/middleware/Oracle_WT1/ohs/include  -I/opt/oracle/middleware/Oracle_WT1/ohs/include   -I/opt/oracle/middleware/Oracle_WT1/ohs/include  -I/usr/include/libxml2 -I.  -c -o mod_proxy_html.lo mod_proxy_html.c && touch mod_proxy_html.slo
 cc -O -DNO_RC2 -DNO_RC5 -DNO_IDEA -DBSAFE -fPIC -DLINUX=260 -DMOD_SSL=206104 -DMOD_PERL -DUSE_PERL_SSI -I/include -DEAPI -D_LARGEFILE64_SOURCE -DUSE_EXPAT -I../lib/expat-lite -I/opt/oracle/middleware/Oracle_WT1/ohs/include -I/opt/oracle/middleware/Oracle_WT1/ohs/include -I/opt/oracle/middleware/Oracle_WT1/ohs/include -I/usr/include/libxml2 -I. -c mod_proxy_html.c  -fPIC -DPIC -o .libs/mod_proxy_html.o
 cc -O -DNO_RC2 -DNO_RC5 -DNO_IDEA -DBSAFE -fPIC -DLINUX=260 -DMOD_SSL=206104 -DMOD_PERL -DUSE_PERL_SSI -I/include -DEAPI -D_LARGEFILE64_SOURCE -DUSE_EXPAT -I../lib/expat-lite -I/opt/oracle/middleware/Oracle_WT1/ohs/include -I/opt/oracle/middleware/Oracle_WT1/ohs/include -I/opt/oracle/middleware/Oracle_WT1/ohs/include -I/usr/include/libxml2 -I. -c mod_proxy_html.c -o mod_proxy_html.o >/dev/null 2>&1
/opt/oracle/middleware/Oracle_WT1/ohs/build/libtool --tag=CC --mode=link cc -O    -DNO_RC2 -DNO_RC5 -DNO_IDEA -DBSAFE -fPIC  -o mod_proxy_html.la  -L/opt/oracle/middleware/Oracle_WT1/ohs/lib -lxml2 -rpath /opt/oracle/middleware/Oracle_WT1/ohs/modules -module -avoid-version    mod_proxy_html.lo
rm -fr  .libs/mod_proxy_html.a .libs/mod_proxy_html.la .libs/mod_proxy_html.lai .libs/mod_proxy_html.so
/usr/bin/gcc -shared  .libs/mod_proxy_html.o  -L/opt/oracle/middleware/Oracle_WT1/ohs/lib -lxml2  -Wl,-soname -Wl,mod_proxy_html.so -o .libs/mod_proxy_html.so
ar cru .libs/mod_proxy_html.a  mod_proxy_html.o
ranlib .libs/mod_proxy_html.a
creating mod_proxy_html.la
(cd .libs && rm -f mod_proxy_html.la && ln -s ../mod_proxy_html.la mod_proxy_html.la)
/opt/oracle/middleware/Oracle_WT1/ohs/build/instdso.sh SH_LIBTOOL='/opt/oracle/middleware/Oracle_WT1/ohs/build/libtool' mod_proxy_html.la /opt/oracle/middleware/Oracle_WT1/ohs/modules
/opt/oracle/middleware/Oracle_WT1/ohs/build/libtool --mode=install cp -f mod_proxy_html.la /opt/oracle/middleware/Oracle_WT1/ohs/modules/
cp -f .libs/mod_proxy_html.so /opt/oracle/middleware/Oracle_WT1/ohs/modules/mod_proxy_html.so
cp -f .libs/mod_proxy_html.lai /opt/oracle/middleware/Oracle_WT1/ohs/modules/mod_proxy_html.la
cp -f .libs/mod_proxy_html.a /opt/oracle/middleware/Oracle_WT1/ohs/modules/mod_proxy_html.a
ranlib /opt/oracle/middleware/Oracle_WT1/ohs/modules/mod_proxy_html.a
chmod 644 /opt/oracle/middleware/Oracle_WT1/ohs/modules/mod_proxy_html.a
PATH="$PATH:/sbin" ldconfig -n /opt/oracle/middleware/Oracle_WT1/ohs/modules
----------------------------------------------------------------------
Libraries have been installed in:
   /opt/oracle/middleware/Oracle_WT1/ohs/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,--rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
chmod 755 /opt/oracle/middleware/Oracle_WT1/ohs/modules/mod_proxy_html.so

OHS provides the includes for Apache but not for libxml (they are not part of the distribution). I checked with this simple test.c that the version is a 2.7.x so I just compiled against system headers which were of the same version.

OHS does not have the beautiful organization of the configuration files that debian uses, so I added the lines directly in the httpd.conf. They are exactly the same changes I presented before but in raw mode .

And it worked again! So this step was done very quickly.

Adding mod_proxy_html to Windows/OHS

This was my final goal but I was sure it was going to be painfully done. I will try to explain all the steps I did but maybe I forget any of them (I did so many things that I am not sure which were necessary and which were useless).

The first point was installing a 2008r2 (evaluation licensed) and the OHS 11.1.1.5.0 (64 bit installation). This was the easy part .
Then the compiler suite was needed. For that I installed the Visual C++ 2008 Express Edition with SP1.
The problem with that is this edition only works for win32 compilations (and not for the win64 which I needed). But I read this great forum post about this issue and I successfully installed Windows SDK for Windows Server 2008 and .NET Framework 3.5 and performed the changes explained in the forum.
I created a new project (Win32 Project / DLL) and I included the mod_proxy_html.cpp which is exactly the same used in Linux but with the following include at the beginning (it adds to the project all the needed windows headers):
```
#include "stdafx.h"
```
Starting the compilation of the mod_proxy_html 3.0.1 file I understood that libraries work different in Windows. DLL files are not enough and you need a LIB file which (I think) define all the symbols of the external library (functions, vars,...). The mod_proxy_html depends on four external libraries: libxml, libapr-1, libaprutil-1 and libhttpd (libxml is used to parse HTML pages and the other are typical Apache libraries used in modules). OHS provides all of them but only as DLL files (in %ORACLE_HOME%\ohs\bin). I suppose that this situation is quite common with third-party software (but do not trust in me, I am not a Windows specialist). Luckily this blog explains how to create a LIB file from the DLL and this forum entry how to add external libraries to a project (includes for compiling and libs for linking).
As OHS does not provide the libxml headers (same issue than in Linux) I added an additional directory with my Linux headers. With them the compilation complained about iconv, as libiconv.dll is not part of the OHS distribution I supposed that the libxml provided for Windows is not compiled with iconv support (quite normal in Windows I guess) so I changed xmlversion.h header to disable iconv support (I changed the 1 in the #if for a 0):
```
#if 0
#define LIBXML_ICONV_ENABLED
#endif
```
One particular problem was that sockaddr_in6 structure was not found (Apache uses IPV6 and IPV4) at compiling time. After a lot of reading I found that this structure is defined in ws2tcpip.h and I needed to change the windows.h provided by the SDK. I commented in the file C:\Program Files\Microsoft SDKs\Windows\v6.0A\Include\Windows.h the following include:
```
#include <winsock.h>
```
and replaced it by this one:
```
#include <ws2tcpip.h>
```
I suppose that the first one is IPV4 only and the second one is for both (but I really do not know). Besides I commented these lines in project header stdafx.h (I think these defines hide some include which I needed):
```
//#define WIN32_LEAN_AND_MEAN  // Exclude rarely-used stuff from Windows headers
//#define _WINSOCKAPI_
```

With all the previous steps done the module compilation still gave a lot of errors. All those errors were only casts and I fixed all of them one by one . Finally the module compiled and linked, a beautiful DLL was generated. But it did not work. When the web server was started it gave the following error:

Syntax error on line 248 of C:\\Oracle\\Middleware\\Oracle_WT1\\instances\\instance1\\config\\OHS\\ohs1/httpd.conf: Can't locate API module structure `proxy_html_module' in file C:/Oracle/Middleware/Oracle_WT1/ohs/modules/mod_proxy_html.dll: No error

The Apache server did not find the variable of the module cos the DLL was not generated like the server wanted (the DLL did not expose the module variable). After a lot of time I realized that all my problems commented in this point (cast errors and the module variable) were generated by wrong compiling and linking options. I changed a lot of them and I am not sure which of them are the important ones. For this reason all my changes and the complete command line for compiling and linking are going to be presented (the following table shows all modified -non default- options of the project and the commands, besides here it is the Visual project file):

C/C++
General:	Additional Directories:	C:\Oracle\Middleware\Oracle_WT1\ohs\include C:\User\Administrator\Documents\Visual Studio 2008\Projects\Project1\mod_proxy_html\libxml2
	Debug Information Format:	Disabled
	Warning Level:	Level 3 (/W3)
Optimization:

Optimization: Maximize Speed (/O2) Code Generation: Enable Minimal Rebuild: Yes (/Gm) Smaller Type Check: No Basic Runtime Checks: Default Runtime Library: Multi-threaded Debug DLL (/MDd) Precompiled Headers: Create/Use Precompiled Header: Use Precompiled Header (/Yu) Advanced: Compile As: Compile as C Code (TC) Show Includes: Yes (/showIncludes) Command Line: /O2 /I "C:\Oracle\Middleware\Oracle_WT1\ohs\include" /I "C:\Users\Administrator\Documents\Visual Studio 2008\Projects\Project1\test\libxml2" /D "WIN32" /D "_DEBUG" /D "_WINDOWS" /D "_USRDLL" /D "MOD_PROXY_HTML_EXPORTS" /D "_WINDLL" /D "_UNICODE" /D "UNICODE" /Gm /EHsc /MDd /Yu"stdafx.h" /Fp"Debug\mod_proxy_html.pch" /Fo"Debug\\" /Fd"Debug\vc90.pdb" /W3 /nologo /c /TC /showIncludes /errorReport:prompt Linker: General: Enable Incremental Linking: No (/INCREMENTAL:NO) Additional Library Directories: C:\Oracle\Middleware\Oracle_WT1\ohs\bin Input: Additional Dependencies: libxml2.lib libapr-1.lib libaprutil-1.lib libhttpd.lib Debugging: Generate Debug Info: Yes (/DEBUG) System: SubSystem: WINDOWS (/SUBSYSTEM:WINDOWS) Optimization: References: Eliminate Unreferenced Data (/OPT:REF) Advanced: Randomized Base Address: Disable Image Randomization (/DYNAMICBASE:NO) Fixed Base Address: Image must be loaded at a fixed address (/FIXED) Target Machine: MachineX64 (/MACHINE:X64) Command Line: /OUT:"C:\Users\Administrator\Documents\Visual Studio 2008\Projects\Project1\mod_proxy_html\Debug\mod_proxy_html.dll" /INCREMENTAL:NO /NOLOGO /LIBPATH:"C:\Oracle\Middleware\Oracle_WT1\ohs\bin" /DLL /MANIFEST /MANIFESTFILE:"Debug\mod_proxy_html.dll.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"c:\Users\Administrator\Documents\Visual Studio 2008\Projects\Project1\mod_proxy_html\Debug\mod_proxy_html.pdb" /SUBSYSTEM:WINDOWS /OPT:REF /DYNAMICBASE:NO /FIXED /NXCOMPAT /MACHINE:X64 /ERRORREPORT:PROMPT libxml2.lib libapr-1.lib libaprutil-1.lib libhttpd.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib

After all that hell I finally got a mod_proxy_html.dll valid for OHS 11.1.1.5.0 (win64) on Windows 2008r2. Again I did the same modifications in the httpd.conf file and the reverse proxy worked fine. Now a video is presented in which I first access directly to my tomcat installation and request the test page. There it is clear that some links are absolute. Then I change to my windows virtual box using the proxy location. Same tomcat page is shown and now the test HTML have the links modified to point to the correct URI (mod_proxy_html is in action!). Finally I request the server info page, the Apache is a OHS Windows X64 with my mod_proxy_html.cpp perfectly loaded.

Your browser doesn't support the video tag. See this entry for more information about how to see the videos in this blog. You can download the file instead.

This entry summarizes how to add mod_proxy_html (a proxy module that modifies the links inside the HTML sent by the backend in order to fix them) to Apache and OHS. The entry shows how to compile the module in Debian/Apache, Linux/OHS and Windows/OHS. My final goal was adding the module to an OHS (64 bits bundle) running in a Windows 2008r2. I usually never work with Windows and I spent so much time doing that that I wanted to preserve the information here. The next time someone tells me how easy Windows is I am going to ask him to compile something, an Apache module for example.

May the force be with you!

Ricky's Hodgepodge: Client Certificates In PHP / Oracle Iplanet Web Server

nospam@example.com (rickyepoderi) — s�b, 17 dic 2011 14:48:00 +0000

These days I have been working in a customer with the same issue which was discussing in the last series of this blog: client certificate login (custom solution). The big difference here was that this customer is working with PHP (last version 5.3) and Oracle Iplanet Web Server 7.0 (formerly Sun Java System Web Server, formerly Sun ONE Web Server, formerly Iplanet Web Server and now included in Oracle Fusion Middleware WebTier). This web server is the last version of the Sun web server which comes from the time of Sun and Netscape alliance. It is curious how just when I decided to publish the series this topic has come out to me again and again.

PHP when is integrated with Apache (the most common situation) uses some environment variables (SSL_CLIENT_CERT, SSL_CLIENT_S_DN and so on) that mod_ssl (module for Apache Web Server to get SSL communication -HTTPS- which relies on OpenSSL) provides to PHP in $_SERVER array. These vars contain some info about the SSL communication in general and the client certificate in particular, some of them are different parts or fields of the cert and SSL_CLIENT_CERT is the whole certificate, PEM format (see StdEnvVars option in the documentation for further info). As always the developer team has its way environment, windows + Apache 2.2 + PHP 5.2, while the pre and production environment of the customer is Solaris 10 + Iplanet 7.0 + PHP 5.3. Obviously the application did not work when it was tested in pre-production (what a strange thing!). The variables that mod_ssl should inject were not there, which was really confusing for the developers, of course the fact that the web server was an Iplanet and not an Apache did not matter much to them.

When PHP is compiled against the Iplanet Web Server it uses old NSAPI (Netscape Server Application Programming Interface) to interact with the http daemon. Typical PHP compilation (very short in modules) for this environment is like this:

./configure --prefix=/opt/php-5.3 --with-nsapi=/opt/SUNWwbsvr7 \
    --enable-ftp --with-pear --enable-pdo --with-openssl

The /opt/SUNWwbsvr7 directory is where the Iplanet Web Server is installed, the installtion needs sample applications component because include files are in there. Then you need to configure some files to make the web server process PHP files (see this wiki page for compilation and configuration details).

If you check the NSAPI developer guide, the client certificate is placed as a request variable named auth-cert. This var contains the certificate in DER format but encoded into text with BASE64. The nsapi.c file in PHP source registers this variable in its own environment with another name CLIENT_CERT. So you can access the certificate (DER + BASE64) like this $_SERVER['CLIENT_CERT']. Then the certificate can be parsed using some PHP/SSL methods. This sample PHP file performs exactly what I said.

<?php
// header to plain/text
header("content-type: text/plain");
// print server vars
print("SERVER vars:\n");
print_r($_SERVER);
// get the headers
print("\n\nHEADERS:\n");
print_r(getallheaders());
// print certificate parsed by SSL
$pem = "-----BEGIN CERTIFICATE-----\n"
    . $_SERVER['CLIENT_CERT']
    . "\n-----END CERTIFICATE-----\n";
$cert = openssl_x509_parse($pem);
print("\n\nCertificate parsed by OpenSSL:\n");
print_r($cert);
// print some variables from the cert
print("\n\nGetting some data from the parsed cert:\n");
print("name: " . $cert['name'] . "\n");
print("subject->CN: " . $cert['subject']['CN'] . "\n");
print("validFrom: " . $cert['validFrom'] . "\n");
?>

As you see the server variable is converted into SSL PEM format (as the certificate is already in BASE64, setting the header and footer line is the only step to transform it into PEM) and then it is parsed and stored into an array by openssl_x509_parse function. With this array any field of the certificate can be retrieved easily.

When finally the application went to production I realized that the PHP web server was behind a proxy (another difference, this time an architectural one). In this case the certificate is not received in the commented way, in this architecture all the SSL stuff (certificate exchange process included) is managed by the proxy and the client certificate is usually forwarded to the backend server using a header (proxy-auth-cert in case of iPlanet Web Proxy Server). In this solution getallheaders PHP function should be used to get the certificate from the specified header and construct the PEM variable exactly in the same way I did in the previous example.

So with PHP and Oracle Iplanet Web Server (any NSAPI server in general although I do not know anyone else) you obviously cannot use mod_ssl/Apache environment variables, there is another one called CLIENT_CERT which is the certificate itself in DER + BASE64 (behind a proxy the certificate is usually placed in a header). Once this fact is known it is easy to find a way of getting any certificate field with PHP code (I used a PHP/SSL function to parse the certificate into an array of variables). Please if you are a developer, do not start coding before all the versions, software stack and architecture are perfectly defined, you usually will work twice or even more (and you will drive a lot of people crazy too, people like me).

Never lose hope!

Ricky's Hodgepodge: Zuma's Revenge!

nospam@example.com (rickyepoderi) — dom, 11 dic 2011 19:02:29 +0000

This week I was on vacation. I was on the mood of doing nothing, so I decided to play some PC game. I had very good references of the Zuma Blitz that people play in Facebook. Besides I had played before Plants vs Zombies of the same company (PopCap games). This one was fantastic. So finally I decided to play Zuma's Revenge! (PC version of the Facebook one).

It has been horrible, sometimes I forget how obsessive I am. I quickly got the last level (level 60), just in one game but I died in that stage. The problem is that, when you die, you do not start in the same stage but five levels before (and with only two lifes!). Maybe I am too clumsy, but it was absolutely impossible for me. I played long hours and the best I did was reaching level 60, I did not even fight against the final boss. I realized I had a problem when I dreamt with the damned balls. So today I had only one mission, finishing the game at all cost (I mean, of course, cheating). Finally I got a cheat to slow down the balls (I would have preferred infinite lifes but I just needed to fool myself). Finally I beat the final boss.

So now it is over. The situation makes me not to trust anymore on PopCap games, I want smooth games, never like this one in which you have to be the fucking hawkeye to reach the final.

You are advised!

PS: Of course both games (Zuma and Plants vs Zombies) work in Linux under wine.

Ricky's Hodgepodge: Gnome-Shell Extensions

nospam@example.com (rickyepoderi) — s�b, 19 nov 2011 10:42:00 +0000

Some weeks ago I realized that gnome-shell and all gnome3 packages were ready to be installed in my wheezy box. Cos I were (and I still am) very busy with a project I am involved, I decided not to upgrade my system at that moment. I had heard such bad comments about this new version that I felt afraid about the change (I was very comfortable with my old desktop). But I have always been a loyal gnome user so the previous weekend my new gnome3/gnome-shell environment was installed, tested and customized.

Obviously the bad comments about the new version are justified in the big differences between both versions. In my opinion plain gnome 3.0 (current version in wheezy), though very eye-candy, is very very restrictive and unproductive in its use. Lots of clicks are wasted to do the same thing you previously did in just one. This annoying fact is very related to the lost of many applets I usually managed (virtual desktop management, menus, launchers, window selector, weather,...), which made your life easier. But finally I discovered that gnome-shell introduces a new idea: gnome shell extensions. The gnome-shell uses extensions in a very similar way than firefox does but, right now, there is not a trusted gnome extension repository or a extension manager application (easy installation, auto-update and so on). But it is clear for me that it is planned for the near future.

I spent short time studying how extensions work and finally I realized they are a very good idea. From a technical view a extension consists in some JSON (configuration), JavaScript (code) and CSS (aspect) files (all HTML standards which is a good thing, at least for me). I am sure that you will can easily replace anything you did in gnome2 with a suitable extension in gnome3 but, as always, nowadays all this stuff is quite immature. One important aspect is that these extensions can also be installed in your ~/.local directory (you do not need packages from your linux distribution, although it is sure that some basic extensions will be installed by default in all distros, check this debian bug for example).

In order to configure gnome-shell and its extensions you better install gnome-tweak-tool application. But you always can do all configuration using typical gsettings command or dconf-editor application (utilities to manage the gnome configuration in general). So once I had understood new gnome-shell stuff I installed the following extensions to be comfortable with my new desktop:

Standard gnome-shell-extensions

Gnome itself has a collection of extensions (these are the ones that I think, at least, are going to be in any linux distro by default). I installed two of them:

dock: Shows a dock-style task switcher on the right side of the screen (it replaces my typical launchers and window selectors). Next version 3.2 adds configuration properties for changing the default position and auto-hiding.
user-theme: Loads a shell theme from ~/.themes/<name>/gnome-shell (I like to customize my desktop ).

The installation process is the following:

Get the source from GIT:

$ git clone git://git.gnome.org/gnome-shell-extensions

Go to tag gnome 3.0.2 (version of my current wheezy gnome-shell):
```
$ cd gnome-shell-extensions/
$ git checkout 3.0.2
```

Autogen, make and install them (only the two extensions):

$ ./autogen.sh --prefix=$HOME/.local --enable-extensions="user-theme dock"
$ make
$ make install

Right now there is a bug in glib-2.0 that it does not take into account extended schemas you have in your ~/.local/share/glib-2.0/schemas directory. So you have to copy the new schemas from there to the global directory and compile them:

$ cd ~/.local/share/glib-2.0/schemas/
$ sudo cp org.gnome.shell.extensions.dock.gschema.xml \
  org.gnome.shell.extensions.user-theme.gschema.xml /usr/share/glib-2.0/schemas/
$ sudo glib-compile-schemas /usr/share/glib-2.0/schemas/

All the themes for gnome3 (GTK, window manager or gnome-shell itself) can be download from gnome-look.org. They are installed as always inside ~/.themes and ~/.icons directories. Use gnome-tweak-tool to change any theme.

gnome-shell-weather-extension

Another applet I used to add to my gnome2 panel was the weather applet. An equivalent extension is the gnome-shell-weather-extension done by Simon Legner but it nowadays does not have several locations like the old applet . The installation is more or less the same:

Clone the repository with git (branch 3.0):

$ git clone -b gnome3.0 https://github.com/simon04/gnome-shell-extension-weather

Install:

$ cd gnome-shell-extension-weather/
$ ./autogen.sh --prefix=$HOME/.local
$ make
$ make install

Because of the commented glib2 bug you need to include the schemas globally:

$ cd ~/.local/share/glib-2.0/schemas
$ sudo cp org.gnome.shell.extensions.weather.gschema.xml /usr/share/glib-2.0/schemas/
$ sudo glib-compile-schemas /usr/share/glib-2.0/schemas/

Then you need to set the city for the weather, in my case Madrid. I did it via gsettings (you can also use dconf-editor):
```
$ gsettings set org.gnome.shell.extensions.weather woeid "'766273'"
```

gnome-shell-workspace-indicator

Finally workspaces in default gnome3 needs too many clicks for me (I change working workspace very often to waste time doing that). So I installed another extension that adds workspace selection in gnome-shell toolbar. This extension has been incorporated to default gnome-shell extensions in latest version 3.2 (but it does not exist in the 3.0.2/wheezy version I installed in the first point).

The installation is pretty the same:

Clone the sources:

$ git clone git://github.com/erick2red/shell-extensions.git

Generate and install:

$ cd shell-extensions/
$ ./autogen.sh --prefix=$HOME/.local --enable-extensions="workspace-indicator"
$ make
$ make install

Installation of the schemas:

$ cd schemas/
$ sudo cp org.gnome.shell.extensions.workspace-indicator.gschema.xml \
  /usr/share/glib-2.0/schemas/
$ sudo glib-compile-schemas /usr/share/glib-2.0/schemas/

I prefer wide indicator (with wide indicator you see Workspace 1 instead of the concise 1 you get with default configuration):
```
$ gettings set org.gnome.shell.extensions.workspace-indicator \
  wide-indicator true
```

I tried to be comfortable but with the least number of extensions. Think you have to re-add them every time you upgrade gnome-shell version (until there was a better solution: a gnome extension manager infraestructure, distro provided extensions or whatever). My desktop currently has the following aspect.

New theme for my Gnome-Shell (Adwaita-White)

Dock extension provides a right menu to launch and bring to front apps

Current weather extension and forecast (Madrid)

Workspace extension to easily change current workspace

I have written this entry because, as I commented before, I will need to update my extensions when a new version of the gnome-shell arrives (3.2 for example). I hope that shortly some extension management will come to gnome3 and all this stuff will be automatically done (as firefox does), but right now extensions need to be installed manually. There are a lot of them scattered throughout the internet and there are also several links which comment the must-have ones (for example this, this or this). Installation can be independent of the linux distribution (although some basic extensions will come by default for sure). I have only installed four extensions in my desktop: user-theme support, dock, weather and workspace. You can enable or disable extensions and customize them via gnome-tweak-tool (coarse-grained changes) and gsettings or dconf-editor (fine-grained configuration).

Finally I got the conclusion that gnome3 and its extensions are a brilliant idea. I think that, with a proper management and repository, its future is awesome but right now gnome3 falls a bit short. In my opinion that is the reason for all the bad comments and the negative impression of many users.

Long life gnome3!

Pon un sol en tu vida: Adobe Flash: The misunderstood ugly guy

nospam@example.com (Sergio Lopez) — dom, 13 nov 2011 09:29:00 +0000

Almost everyone hates Flash, and the discontinuation of it's mobile version was seen by many as a triumph.

Being a GNU/Linux user myself, I also suffered all those crappy versions, late releases and crazy development cycles. But, looking back and trying to be fair, I must acknowledge that the availability of a Flash player for the major operating systems has lowered the barrier for multiplatform development for many people, specially for independent companies. And I'm not talking only about embedded videos on websites, but also videogames (like Machinarium or VVVVVV) and applications. In some sense, Flash was more successful than Java at building bridges between platforms.

That said, the failure of Flash on mobile platforms is not surprising at all. Pride seems to be the favorite sin for Adobe. They were barely able to produce decent quality versions for non-mobile platforms, so it was pretty clear their development work force wasn't strong enough to provide properly optimized versions for the more complex Android ecosystem.

Adobe rejected the option of freeing its Flash Player, while keeping Adobe Flash (the content creator) as a commercial product. Apparently, they feared publishing the source code of its player would make easier for its competitors to provide alternative editors. But competition is not a bad thing, as it usually enriches the ecosystem and increases the popularity of a product. And opening the player would have extended Flash to newer platforms, improved the quality of non-Windows versions, and reduced total development cost, giving them more chances to success on mobile platforms. Perhaps they would even won the favor of some other companies (namely Google), which would in turn embed support for Flash on their own products by themselves.

Adobe lost their big chance. Not even them understood the truly potential of their own product.

Ricky's Hodgepodge: Certificate Security in JavaEE - Standard Solution (Glassfish)

nospam@example.com (rickyepoderi) — mi�, 09 nov 2011 10:33:00 +0000

Today I am going to present the last entry of the series about certificate security in a JavaEE application. If you remember the first post dealt with the environment setup (not very interesting in terms of development but a nice guide about a quick CA and secure server installation) and the second one commented custom security using client certificates (the application itself mapped the certificates and real users and groups). This last entry will present client certificates authentication but using standard JavaEE security.

JavaEE uses security based on roles, each user will have some roles, and depending these roles he can or cannot perform the specific application actions. Talking about what can be protected JavaEE adds security to two types of resources in a direct way: any web resource (pages, servlets,...) and Enterprise Java Beans (EJB) (the whole bean and/or specific methods of the bean). The security can be mapped via the deployment descriptor, web.xml in case of the web resources (security-constraint directive) and ejb-jar.xml in case of EJBs (method-permission and so on), or using annotations (@DeclareRoles, @RolesAllowed, @PermitAll or @DenyAll). The JavaEE API also offers methods to know the roles of a user and perform a programmatic security (isUserInRole in HttpServletRequest and isCallerInRole in EJBContext). So the idea is pretty clear, based on the roles of the user the application has to manage its own security which can be declarative (assembler files or annotations), programmatic (API) or both.

So now that JavaEE security is understood we need to know how the roles are assigned to a specific user. Basically the roles are defined for the application (again assembler file descriptors or annotations) and then they are mapped against users and groups inside some repository. JavaEE manages the concept of realms, which are stores for users and groups associated to the container. The containers perform the mapping between them and the application roles via a proprietary file descriptor (glassfish-web.xml, weblogic.xml,...) and/or using the container management console (case of IBM WebSphere). Each application server manages the realms in its own way, they can be more or less powerful (for example weblogic has clearly a much more customizable realms than glassfish or tomcat).

The last part of theory which needs to be commented is the login issue. Once the JavaEE application has added role restrictions it needs to establish a login method. The standard configures the login inside the login-config part of the web.xml. It permits the following methods for authentication: NONE, BASIC, DIGEST, FORM or CLIENT-CERT. This login is also associated to a realm in the container. So now it is clear, when an application establishes JavaEE security, it defines a realm, a login method and different roles; the users who try to access the application will need to login first and, in this process, they get all the groups from the selected realm (in JavaEE jargon the user and groups obtained from the realm are called principals); these groups are mapped against the app roles; finally the roles are used to implement security (declarative or programmatic) inside the application. As you see the circle is now closed.

JavaEE is sometimes quite overelaborated, but be sure that the ideas behind it are very very simple. You can read the JavaEE tutorial (current version 6) for further information about its security model.

Going back to our particular case of client certificates, it is clear that CLIENT-CERT authentication login method has to be selected. And, as it is a standard defined login method, everything should work smoothly, but Glassfish is not the case. What I want you to have in mind is that now all this stuff is in the hands of your container, we are talking about JavaEE, therefore the way login and mapping are handled does not depend on the application but on the container itself.

I chose glassfish v3 as my application server for this series and it has a special certificate realm, this realm is always used when CLIENT-CERT login is specified for an application (this part is not configurable). By default it only gets the subject of the certificate and sets it as the user principal, but nothing else, so forget about ldap, groups or whatever. Nevertheless glassfish joins realm implementations to a JAAS (Java Authentication and Authorization Service) login module context. A JAAS context can specify several (and custom) login modules, this way a custom certificate login module can be implemented to get the user from the certificate and match him to a user in the company repository. This way I defined a new login module inside the domains/domain1/config/login.conf:

certLdapRealm {
       sample.loginmodule.CertLdapLoginModule required realm=ldap;
};

I created a CertLdapLoginModule.java which extends AppservCertificateLoginModule, an abstract class that glassfish explicitly provides to implement custom certificate login modules. My new class only has to implement the authenticateUser method, in here, my idea is clear: getting the user from the subject (same as in custom security, the user login is the CN part of the certificate subject) and querying the ldap with it in order to obtain the user and groups.

I first thought reusing the glassfish LDAPRealm for all the ldap stuff but this realm has all methods declared as private and I could not find a way of searching the user. The class is declared final so it cannot be extended either. Finally I copied the class code and a new LDAPRealm.java was created. It is almost the same code except the method findAndBind, which now receives a new boolean parameter checkPassword, if it is false no bind is performed and the user and groups are retrieved without password.

You have noticed that the certLdapRealm JAAS definition defines a realm option parameter, this parameter marks the realm which implements the new LDAPRealm class. The code inside my custom login class (autheticateUser method) is the following:

protected void authenticateUser() throws LoginException {
    String realm = (String) _options.get(REALM_OPTION);
    if (realm == null) {
        throw new LoginException(
            "LoginModule has not 'realm' option to get the LDAP realm!");
    }
    try {
        // get the user from the certificate
        LdapName name = new LdapName(this.getX500Principal().getName());
        String certName = name.getRdn(name.size() - 1).getValue().toString();
        // get the realm
        LDAPRealm ldap = (LDAPRealm) Realm.getInstance(realm);
        if (ldap == null) {
            throw new LoginException("Realm'" + realm + "' does not exists!");
        }
        String[] grpList = ldap.findAndBind(certName, null, false);
        // add the short user as a user principal
        // LoginContextDriver sets the cert subject as the name of the user
        // this let the user to set roles based on user (uid) names
        getSubject().getPrincipals().add(new PrincipalImpl(certName));
        // add groups => commit sets them as group principals
        commitUserAuthentication(grpList);
    } catch (Exception e) {
        throw new LoginException(e.getMessage());
    }
}

The method gets the LDAPRealm using the specified option. The CN part of the certificate subject is read in order to query the ldap. The custom findAndBind method is used with false argument (do not bind the user). If everything goes well the groups that have been returned are passed to be transformed into principals. Check that the user himself is added as a principal before, this is done cos the certificate realm uses the complete DN as username (now both ways, complete DN or just the UID part can be used in the role mapping file).

Finally the certificate realm inside glassfish has to be configured to use the new JAAS module. This was done directly in the domain.xml file (asadmin can also be used for this configuration):

<auth-realm name="certificate"
    classname="com.sun.enterprise.security.auth.realm.certificate.CertificateRealm">
        <property name="jaas-context" value="certLdapRealm"></property>
        <property name="assign-groups" value="allusers"></property>
</auth-realm>

The jaas-context property specifies the JAAS context to be used (it was set to certLdapRealm). The other property assign-groups are groups that are directly assigned to any user who successfully logs in this realm (they are used in web.xml for specifying any authenticated user).

Finally the LDAPRealm (used inside the custom login module) has to be configured in glassfish too:

<auth-realm name="ldap" classname="sample.realm.LDAPRealm">
    <property name="directory" value="ldap://debian.demo.kvm:389"></property>
    <property name="base-dn" value="dc=demo,dc=kvm"></property>
    <property name="jaas-context" value="myLdapRealm"></property>
    <property name="search-bind-dn" value="cn=admin,dc=demo,dc=kvm"></property>
    <property name="search-bind-password" value="****"></property>
    <property name="assign-groups" value="allusers"></property>
</auth-realm>

This is a normal configuration for a glassfish LDAPRealm (except that my class is used instead of the glassfish default). Please check that the realm name is ldap, exactly the same name which is passed to my certLdapRealm JAAS context in the login.conf file. The extra realm and login module classes were packaged into a jar and placed inside domains/domain1/lib directory (see this article for more information about custom realms and login modules in glassfish).

I know that all of this is quite confusing, so I am going to repeat the idea. The default certificate realm now depends on a custom certLdapRealm JAAS context. This context executes a custom CertLdapLoginModule which uses another realm ldap (custom LDAPRealm but with minimal differences from the glassfish default) to search the user and his groups in the LDAP repository. In my opinion glassfish realms and login modules are too dependent on each other, they always generate cross references, and that generates some mess.

So now everything is correctly configured to deploy a JavaEE security application which uses client certificates. I deployed the same application that was used in the custom security entry, but now it does not need all ldap stuff so it is almost an empty app (only some facelets pages and little helper Java classes). The interesting part of this application is inside the descriptor (web.xml) file. It defines the following constraint:

    <security-constraint>
        <display-name>CertSecurityJavaEE constraint</display-name>
        <web-resource-collection>
            <web-resource-name>Servlet Application</web-resource-name>
            <url-pattern>/faces/*</url-pattern>
        </web-resource-collection>
        <auth-constraint>
            <role-name>allusers</role-name>
        </auth-constraint>
        <user-data-constraint>
           <transport-guarantee>CONFIDENTIAL</transport-guarantee>
        </user-data-constraint>
    </security-constraint>

All faces pages (pattern /faces/*) are declared inside the constraint. The role allusers is declared as needed to access the application (any user authenticated in the realm). Besides the constraint declares the data as confidential, this means that the application has to be used securely (over HTTPS). The login part is as follows:

    <login-config>
        <auth-method>CLIENT-CERT</auth-method>
        <realm-name>ldap</realm-name>
        <form-login-config>
            <form-login-page>/faces/login.xhtml</form-login-page>
            <form-error-page>/faces/login.xhtml</form-error-page>
        </form-login-config>
    </login-config>

I declared CLIENT-CERT which means the user needs to present a certificate to get access (as I said glassfish uses the certificate realm for this configuration, and all the previous configuration comes into stage). And finally some roles are defined for the application:

    <security-role>
        <description>Users Role</description>
        <role-name>allusers</role-name>
    </security-role>
    <security-role>
        <description>StaticGroup1 Role</description>
        <role-name>StaticGroup1</role-name>
    </security-role>
    <security-role>
        <description>DynGroupPreSales Role</description>
        <role-name>DynGroupPreSales</role-name>
    </security-role>
    <security-role>
        <description>SampleClient1 Role</description>
        <role-name>SampleClient1</role-name>
    </security-role>

The application manages four roles:

allusers: All the users of the application (any authenticated user).
StaticGroup1: The static group of the ldap (I used it in the previous entry).
DynGroupPreSales: The dynamic group of the ldap (I also used it before).
SampleClient1: This is an example user role, only the SampleClient1 is part of this role. I added this role to check if my login module handles user short login name (UID in LDAP) correctly.

As I said glassfish used a specific glassfish-web.xml descriptor file to map the roles against the users and groups in the realm, here it is my mapping:

  <security-role-mapping>
    <role-name>allusers</role-name>
    <group-name>allusers</group-name>
  </security-role-mapping>
  <security-role-mapping>
    <role-name>StaticGroup1</role-name>
    <group-name>StaticGroup1</group-name>
  </security-role-mapping>
  <security-role-mapping>
    <role-name>DynGroupPreSales</role-name>
    <group-name>DynGroupPreSales</group-name>
  </security-role-mapping>
  <security-role-mapping>
    <role-name>SampleClient1</role-name>
    <principal-name>SampleClient1</principal-name>
  </security-role-mapping>

Basically I mapped all roles to the so-called LDAP group (remember allusers is a nonexistent ldap group, which is automatically added by the realm when a user is authenticated, it was defined in the assign-groups property of both realms) except SampleClient1 which is mapped against the same username (I used the short CN name, the one that is added by my custom login module).

Now the application is ready, but please think about it, how does it work? When the user access to the application he needs to present a certificate (it is compulsory) and there is no other choice. As you can imagine this is too much restrictive in common situations. In other JavaEE containers a fallback authentication method can be configured (CLIENT-CERT,FORM for example in weblogic) but not in glassfish. So I decided to deploy the same application twice, one with CLIENT-CERT authentication (/CertSecurityJavaEE-cert context) and the other with FORM (/CertSecurityJavaEE context). If you check the previously presented login part of the web.xml the ldap realm is specified (remember that glassfish uses certificate realm for CLIENT-CERT authorization no matter the configuration, so this ldap realm is only used when FORM authentication is selected). In order to use my custom LDAPRealm class here I needed to implement a custom LDAPLoginModule.java that extends glassfish default (because the damned cross references), but you can ignore this, it is too confusing even for me who implemented the solution.

The CLIENT-CERT option makes unnecessary the client authentication configuration we did in the secure listener of glassfish (check the first entry of this series). If CLIENT-CERT authentication method is established for an application the app server automatically requests a client certificate for its context in the secure port. So I unchecked this option and now a user without certificate is able to access the FORM defined application through the same secure port (the other /CertSecurityJavaEE context). Besides an HTML error page for 401 error code (Unauthorized) was configured for the application. What does it mean? Now a user who presents a certificate in /CertSecurityJavaEE-cert but has no mapped user in the LDAP server will receive a custom 401 error page, this page will show a link against the FORM authenticated application. This is configured (standard way) in the web.xml file.

    <error-page>
        <error-code>401</error-code>
        <location>/resources/errorpages/401.html</location>
    </error-page>

Here it is the video. Now when I access to the certificate application (/CertSecurityJavaEE-cert) a certificate is requested. Same as before if I select the revoked certificate (03) firefox requests me to choose another one because it is revoked, if I pick up the valid one (02) I just enter as SampleClient1 (see all the roles are granted and the long certificate subject is used as username). Now I cannot logout cos once you have selected a certificate for a site the browser uses always it for this site (it is the same case than in custom security). Then I clear my sessions in firefox (this logs me out to try again) and I try to enter with a new certificate, created for SampleClient3 with serial number 05 but without a mapped real user, so this time my login module fails to match a user and the 401 page is reached. There I can click the presented link and I access the non-certificate (/CertSecurityJavaEE) application. Finally I log in using SampleClient2 (remember this user has a revoked certificate but he is a valid user in the LDAP server). He has no role assigned as expected and now the user name is the short one.

Your browser doesn't support the video tag. See this entry for more information about how to see the videos in this blog. You can download the file instead.

The moral for standard JavaEE certificate login is that it depends on the container implementation, the application just needs to configure CLIENT-CERT authentication. As you see glassfish realms (and the certificate one particularly) are quite confusing, there is a lot of mess between realms and login modules and it is specially painful using or extending its default classes (I do not know why but they have a lot of private methods and they are usually declared as final). The best I could do was implementing a custom login module which was associated to the certificate realm, this login class performs the mapping between certificate and user (groups included). Besides the application was deployed twice, one requests client certificates and the other uses typical username and password login, this way users without a certificate have a less secure but functional way to login. For this entry two projects were developed, the library project with realms and login modules and the Web Application project. My personal conclusion is that all of this is so particular for glassfish that a different entry like this one is needed for every JavaEE container.

You win some, you lose some. Cheerio!

Ricky's Hodgepodge: Certificate Security in JavaEE - Custom Solution

nospam@example.com (rickyepoderi) — s�b, 22 oct 2011 14:38:00 +0000

This entry is the second chapter of the JavaEE certificate security series. The previous post explained how all the environment was configured (OpenSSL demo CA and glassfish application server). Now it is time to deploy an application in the container and start using the certificates. This entry will deal with custom security, custom means that the application will use a repository for users and groups by its own, implementing the login classes and not JavaEE standard.

Before commenting the application let's explain how a web application usually works with certificates. If you remember the first entry, the client certificate is validated by the application server (or the web server in general) before any code of the application is executed (in the video the certificate validation is performed even no application was already deployed in glassfish). Therefore it is concluded that when the request reaches the application the certificate is valid. Here the only problem that remains is who is the user which corresponds to this valid certificate. You have to think that usually a company is going to trust in some CA certs (for example, the national eID, DNIe in case of Spain), but the registered users will be stored in a company repository (LDAP, DDBB or whatever). At this point the application needs someway to map certificates and users, usually some data in the issued certificate is used to match the user: some part of the certificate subject (usually the CN or the Email part), an attribute extension (some extra data included in the cert),... It does not matter what field, but the idea is some information stored in the cert will be used to find the real user in the repository (in the previous example of Spainsh eID the ID number, DNI number, would be a good candidate for matching users, obviously the DNI should be a compulsory field at customer/user registration process).

The other question is when to perform this match between the certificate and the real user in the store, but this problem is solved as in any other login. In a web application with custom security any request should be intercepted to check if the user is logged in, if the request has no user information the user is redirected to the login page. In Java the user information is always stored in a session attribute (the application checks that the username and any other relevant information are inside the session) and the interceptor technique depends on the framework used (a typical servlet filter can be the simplest one). If our application is going to use certificates, the interceptor code will also be responsible of matching the user, in case of a successful match the user will be automatically logged in without password prompting.

In summary custom security needs to map the certificate against a real user inside the typical login interceptor code. My sample application uses an OpenLDAP repository (LDAP is the most common repository for users and groups) and it is implemented using JSF (Java Server Faces) and CDI (Context Dependency Injection). Instead of a filter it will use a JSF phase listener, if there is a solution in the chosen framework for a specified problem it will always be the recommended way to go (do not mix technologies). The main files in the solution are the following.

The LoginBean is a JSF session bean which is used to store the logged user information. As I said Java applications usually use the session to save the information associated with the logged user, in case of JSF, a session Bean is the direct solution.

This bean has some important methods:
- login: The login action method performs typical login. The user introduces username and password in a login page and this page calls this method which finds and binds the user against the LDAP repository (the user is found and then the password is checked). This is the method executed in normal (non-certificate) login. I have used the ldapbp library (typical Sun, now Oracle, LDAP library) in order to get the user and the groups easily. All the LDAP stuff is executed inside a Singleton annotated EJB (Enterprise JavaBean) but forget this, it is just a way to acess the sample repository.
```
public String login() {
    userEntry = ldap.findUser(username, password.toCharArray());
    groupsCache.clear();
    if (userEntry != null) {
        return "index";
    } else {
        FacesUtils.errorMessage("loginForm:username", 
            "login_invalid_username_or_password");
        return "login";
    }
}
```
- trustedLogin: Other method of the bean which performs the match between certificates and users. It receives the X509 certificate of the client and with the CN part of the Subject (I have implemented a very simple match condition) it searches the LDAP repository for a matching user. If a valid user is found the method automatically fulfills all the variables in the session bean. This method only executes the login process if the user is not previously logged in (avoiding useless searches and validations).
```
public String trustedLogin(X509Certificate cert) {
    try {
        // only check if user not logged in
        if (!isLoggedIn()) {
            // get CN part => it's the last one
            LdapName name = new LdapName(cert.getSubjectDN().getName());
            String certName = name.getRdn(name.size() - 1).getValue().toString();
            // validate via PKIX if configured
            boolean certOK = !validatePKIX || validatePKIX(cert);
            if (certOK) {
                // login without password
                LdapEntry entry = ldap.findUser(certName, null);
                if (entry != null) {
                    username = certName;
                    userEntry = entry;
                    groupsCache.clear();
                }
            }
        }
    } catch (Exception e) {
        logger.error("Error performing the login", e);
    }
    return username;
}
```
- isLoggedIn: Method that returns if there is an active session of some user. It only checks that the username and userEntry variables are filled (both vars are initialized by the two previous methods).
```
public boolean isLoggedIn() {
    return username != null && userEntry != null;
}
```
- validatePKIX: This method validates the certificate using OCSP and/or CRL programmatically. You know that in my demo environment this is already done by the container, but I decided to include this code here just in case of an hypothetical container which does not support these validations. The bean can be configured to perform this extra validation in previous trustedLogin method, this way the container would do normal validation (trusted CA and valid dates) and the Java application would perform OCSP or CRL (this method uses exactly the same APIs than JSSE/glassfish).
- isUserInGroup: The bean performs custom security, this is an equivalent method to standard isUserInRole. It returns true if the user belongs to the specified LDAP group (static and dynamic groups can be passed because ldapbp is used). Group membership is cached inside the LoginBean.
```
public boolean isUserInGroup(String groupname) {
    // check cache
    String key = groupname.toLowerCase();
    if (groupsCache.containsKey(key)) {
        return groupsCache.get(key);
    } else {
        // read from ldap and set into cache
        boolean res = ldap.isUserInGroup(this.userEntry.getDn(), groupname);
        groupsCache.put(key, res);
        return res;
    }
}
```
- logout: Simple method to delete the session associated to this user.
```
public String logout() {
    HttpSession session = (HttpSession) FacesContext.getCurrentInstance(
        ).getExternalContext().getSession(false);
    if (session != null) {
        session.invalidate();
    }
    return "logout";
}
```
When using CDI for JSF Beans the @ManagedProperty annotation cannot be used (and I used it a lot to initialize the beans before). In this sample application I tried to figure out how to initialize beans with Injection but I realized there is no direct solution, finally I used a properties file as it is commented in this post.
The CertLoginPhaseListener interceptor. As I commented previously, using JSF the recommended solution for the login interceptor stuff is a JSF Phase Listener (see the link I presented before for further information). These listeners are always executed associated with a JSF request/response phase and therefore can be used as filters. The listener performs the following steps:
- It recovers the certificate from the request (this is the normal way of getting the associated certificate in a Java container):
```
X509Certificate[] certs = (X509Certificate[]) 
    req.getAttribute("javax.servlet.request.X509Certificate");
```
- If there is at least one certificate (the client is accessing via the secure https port) the listener calls trustedLogin to automatically log the user in if it is the case. To do that programmatic EL (Expression Language) is used.
```
if (certs != null && certs.length > 0) {
    try {
        // login silently without password via certificate, the trustedLogin
        // performs the login if there is no previous session, certificate is valid
        // and the user is mapped against the ldap
        ExpressionFactory ef = facesContext.getApplication().getExpressionFactory();
        MethodExpression me = ef.createMethodExpression(facesContext.getELContext(),
                "#{loginBean.trustedLogin}", null, new Class[]{X509Certificate.class});
        me.invoke(facesContext.getELContext(), new Object[]{certs[0]});
    } catch (Exception e) {
        logger.error("Error checking certificate", e);
    }
}
```
- After that the listener calls isUserLoggedIn just to check if there is an active session, again EL is used.
```
ExpressionFactory ef = facesContext.getApplication().getExpressionFactory();
ValueExpression ve = ef.createValueExpression(facesContext.getELContext(), 
    "#{loginBean.loggedIn}", Boolean.class);
boolean isLoggedIn = (Boolean) ve.getValue(facesContext.getELContext());
```
- Finally this boolean variable can be true (the user previously logged in or he has been automatically logged in by the certificate method) or false (the user has never logged in and there is no certificate or it matched no user). Based on this variable the listener permits the user to continue or redirects him to the login page. It is quite interesting the case of a registered user who tries to access the login page, if this case happens the user is also redirected, but this time to main application page (avoiding any login). Besides if the client has a valid certificate but it did not match any user login page will be presented through the secure port.
```
if (!loginPage && !isLoggedIn) {
    // drive the user to the login page
    facesContext.getApplication().getNavigationHandler(
        ).handleNavigation(facesContext, null, "login");
} else if (loginPage && isLoggedIn) {
    // already login => go to main page
    facesContext.getApplication().getNavigationHandler(
        ).handleNavigation(facesContext, null, "index");
}
```
The login.xhtml and index.xhtml are the facelets pages for the login and the main page. Login is the typical username and password page and the main one simply shows the user logged in and if it belongs to an LDAP static and a dynamic group. These groups were previously loaded into the OpenLDAP repo (SampleClient1 belongs to both groups and SampleClient2 to none of them).

So today the video exemplifies all possible login methods. If the user accesses the application using non-secure port (8080) the listener detects no certificate and no logged user and it presents a common login page. In this page I log in using SampleClient2 (although his certificate is revoked he is a valid user in the LDAP repository, so he can access the application using normal login and password way). But if I access the https port (8181) and I use the revoked certificate glassfish denies the access. The second time I try with SampleClient1, now the listener detects my certificate, matches it against the LDAP user (the Subject of this cert is CN=SampleClient1,O=demo.kvm,C=ES and the CN part is used to find the associated user) and it fulfills the LoginBean data to silently log me in. Finally the listener redirects the request to the index main page, SampleClient1 user and his groups are shown in the page. As I am using certificate login I cannot logout (the certificate is always detected and the user is logged when he has not an active session). In order to delete my login I clean my cookies and active logins in the browser and I try with another certificate, I created a new certificate for SampleClient3 but this user does not exists in the LDAP repository, this time the listener does not match any user to this new Subject CN and, therefore, login page is presented in the secure port. I enter again as SampleClient2.

Your browser doesn't support the video tag. See this entry for more information about how to see the videos in this blog. You can download the file instead.

In summary using certificates as a login method in a custom security application is not very complicated. Mapping the certificate to a real application user is the main part, and, in a custom security application, it is done in the same place in which login interception is done. My sample application NetBeans project can be downloaded and it just performs all of this with JSF and a phase listener, the users and groups are placed in an OpenLDAP server and the ldapbp library is used to retrieve group membership easily, the application maps the certificate via the CN part of the subject. Although the application is very simple I think it exemplifies perfectly a certificate login using custom security. Some days ago (when I had already implemented this demo application but not published the entry) I faced a customer who had tried to do the same in a very very weird way (I better do not comment this), that reassured me in publishing this series. Take into account that, although the entry implements the certificate security in Java, it rules in any web application (PHP, Rails, Python or whatever).

Remember to be smart!

Ricky's Hodgepodge: Certificate Security in JavaEE - Demo Setup

nospam@example.com (rickyepoderi) — s�b, 08 oct 2011 13:27:00 +0000

Some days ago I was discussing with a colleague what would be the best way to get certificate security in a JavaEE application. We did not reach any final conclusion so I decided to start a demo environment to test some simple cases. In this post the environment is going to be setup, the CA and the Application Server.

In order to get a complete infrastructure a CA (Certificate Authority) with CRL (Certificate Revokation List) publication and/or OCSP (Online Certificate Status Protocol) checking is needed. As I explained in the first post of the signing applet series a CA is a trusted company which issues certificates. The other two concepts are mechanisms for validating certificates or, more precisely, to know if a certificate has been revoked. A CRL is a list of revoked certificates and this list is usually accessed via internet. OCSP is a standard protocol to check certificates online. It is clear if some application is going to trust in user certificates, there should be someway to deal with revocations.

I decided to use openssl as a simple but complete CA infrastructure. The famous security library also has some CA capabilities which are perfect for a demo environment. As the JavaEE container I setup Glassfish 3.1.1 (but I think the concepts are valid for any other app server). Of course Debian Squeeze is the chosen OS (installed as a KVM virtual box). The steps for creating the complete infrastructure are the following.

Creation and configuration of the OpenSSL CA. For this point I followed this helpful web page.

First of all, openssl package is installed:
```
# apt-get install openssl
```
After that the file /etc/ssl/openssl.cnf has to be modified. Here it is my modified file but basically only two options were added to [usr_cert] section:
```
authorityInfoAccess = OCSP;URI:http://debian.demo.kvm:3456/
crlDistributionPoints = URI:http://debian.demo.kvm/my.crl
```
This options add the OCSP and CRL information to the certificate. Both techniques have to be explicitly added to the generated certificates, this way, when something (a browser, app server or whatever) reads the certificate it can check its validity against the specified URLs. As you see my debian demo box is called debian.demo.kvm.

Some directories needed by the CA configuration are created now (CA was located inside /etc/ssl/demoCA directory which is also specified in openssl.cnf):
```
# cd /etc/ssl
# mkdir -p demoCA/newcerts
```
Some files are also initialized:
```
# touch /etc/ssl/demoCA/index.txt
# echo 01 > /etc/ssl/demoCA/serial
# echo 01 > /etc/ssl/demoCA/crlnumber
```

Creation of the CA certificate. In this second step the CA certificate is generated (directory is always /etc/ssl if not specified). The certificate is requested:

# cd /etc/ssl
# openssl req -subj "/C=ES/O=demo.kvm/CN=ca.demo.kvm" -new -newkey rsa:2048 \
  -keyout private/cakey.pem -out careq.pem
Generating a 2048 bit RSA private key
...............................................................+++
.........................+++
writing new private key to 'private/cakey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

Now the certificate is self-signed against the CA. Please check the extensions v3_ca option, this is the CA cert and it should not contain OCSP or CRL info (this option makes the command to use the other v3_ca section which does not contain OCSP or CRL info).

# openssl ca -out cacert.pem -days 365 -keyfile private/cakey.pem -selfsign \
  -extensions v3_ca -infiles careq.pem 
Using configuration from /usr/lib/ssl/openssl.cnf
Enter pass phrase for private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 1 (0x1)
        Validity
            Not Before: Sep 16 16:38:41 2011 GMT
            Not After : Sep 15 16:38:41 2012 GMT
        Subject:
            countryName               = ES
            organizationName          = demo.kvm
            commonName                = ca.demo.kvm
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                BC:39:85:FF:DB:6E:46:FF:08:52:32:6F:97:65:91:F6:F4:A9:24:E7
            X509v3 Authority Key Identifier: 
                keyid:BC:39:85:FF:DB:6E:46:FF:08:52:32:6F:97:65:91:F6:F4:A9:24:E7
                DirName:/C=ES/O=demo.kvm/CN=ca.demo.kvm
                serial:01

            X509v3 Basic Constraints: 
                CA:TRUE
Certificate is to be certified until Sep 15 16:38:41 2012 GMT (365 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

Finally the CA cert is exported in DER format (DER format will be better for JKS stores in glassfish) and in PKCS#12 format (a p12 file contains the keys and the certificate, so it can be considered as a backup).

# openssl x509 -in cacert.pem -outform DER -out cacert.der
# openssl pkcs12 -export -out cacert.p12 -in cacert.pem -inkey private/cakey.pem 
Enter pass phrase for private/cakey.pem:
Enter Export Password:
Verifying - Enter Export Password:

The first certificate (the CA one) is ready. This certificate will be used to sign all the others (users and glassfish server).

Creation of two client certificates. Once the CA cert is generated two client certs are going to be issued, the procedure is very similar but, when signed, the CA certificate is used (instead of self-signing) and now no extensions option is specified (now user_cert section will be used, with OCSP and CRL info). Again in this point I followed another tutorial, in this case the rsa keys were generated first and then the certificate was requested, created and transformed to DER and PKCS#12.

# openssl genrsa -out private/client1.key 1024
# openssl req -subj "/C=ES/O=demo.kvm/CN=SampleClient1" -key private/client1.key \
  -new -out client1.req
# openssl ca -in client1.req -cert cacert.pem -keyfile private/cakey.pem -out client1.pem
Using configuration from /usr/lib/ssl/openssl.cnf
Enter pass phrase for private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 2 (0x2)
        Validity
            Not Before: Sep 16 16:42:31 2011 GMT
            Not After : Sep 15 16:42:31 2012 GMT
        Subject:
            countryName               = ES
            organizationName          = demo.kvm
            commonName                = SampleClient1
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Comment: 
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier: 
                36:24:6E:0E:9B:A4:30:FA:85:7B:59:0D:35:DE:09:F0:C1:89:4B:D6
            X509v3 Authority Key Identifier: 
                keyid:BC:39:85:FF:DB:6E:46:FF:08:52:32:6F:97:65:91:F6:F4:A9:24:E7

            Authority Information Access: 
                OCSP - URI:http://debian.demo.kvm:3456/

            X509v3 CRL Distribution Points: 
                URI:http://debian.demo.kvm/my.crl

Certificate is to be certified until Sep 15 16:42:31 2012 GMT (365 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

# openssl x509 -in client1.pem -outform DER -out client1.der
# openssl pkcs12 -export -out client1.p12 -in client1.pem -inkey private/client1.key
Enter pass phrase for private/client1.pem:
Enter Export Password:
Verifying - Enter Export Password:

Doing the same steps a second certificate (client2) was generated. But, in this case, the certificate was then revoked (this way we have a valid and an invalid user certificate for testing).

# openssl ca -revoke client2.pem -keyfile private/cakey.pem -cert cacert.pem
Using configuration from /usr/lib/ssl/openssl.cnf
Enter pass phrase for private/cakey.pem:
Revoking Certificate 3.
Data Base Updated

Installing a certificate for glassfish. Another necessary certificate is the one for the application server ssl socket, working with user certificates always needs a server-side https connection. New version v3 of glassfish only uses JKS java store (so forget about the NSS certificate store which I commented in a previous entry about 2.1.1 version).

The first thing to do is installing java JDK. In debian you need to add the non-free debian repository in the /etc/apt/sources.list and install the oracle (closed-source) package:

# apt-get install sun-java6-jdk

If you prefer OpenJDK (open-source) implementation you need to install another package instead (I did not test it but I am sure it would also work smoothly):

# apt-get install openjdk-6-jdk

Glassfish 3.1.1 was downloaded, the full platform ZIP file, and installed (uncompressing the file is the only step). In the virtual debian box glassfish was installed in /opt directory:

# cd /opt
# unzip ${DOWNLOAD_DIR}/glassfish-3.1.1.zip
# cd /opt/glassfish3/bin
# ./asadmin start-domain

As you see ZIP installation contains a local domain1 domain already configured. The certificate stores are placed in the glassfish/domains/domain1/config directory. The keystore.jks file is the store for server certificates and cacerts.jks the store for trusted CA certificates. Now it is the time to request a new certificate for the https port (glassfish always configures a secure port, port 8181 by default, with a self-signed s1as certificate, which is going to be replaced).

The certificate (called server) is created without password (RETURN was pressed):

# keytool -genkey -alias server -keyalg RSA -keysize 1024 -keystore keystore.jks \
  -dname "CN=debian.demo.kvm,OU=GlassFish,O=demo.kvm,C=ES"
Enter keystore password:  
Enter key password for <server>
	(RETURN if same as keystore password):

Then the request file is generated:

# keytool -certreq -alias server -keyalg RSA -file /etc/ssl/server.req -keystore keystore.jks
Enter keystore password:  
Enter key password for <server>

Now it is time to sign the request with our CA. Again user_cert section is used, so OCSP and CRL information will be also added to this certificate (this is not necessary but it is quite nice). The server has to be transformed to DER format to be used in JKS.

# cd /etc/ssl
# openssl ca -in server.req -cert cacert.pem -keyfile private/cakey.pem -out server.pem
Using configuration from /usr/lib/ssl/openssl.cnf
Enter pass phrase for private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4 (0x4)
        Validity
            Not Before: Sep 16 16:56:14 2011 GMT
            Not After : Sep 15 16:56:14 2012 GMT
        Subject:
            countryName               = ES
            organizationName          = demo.kvm
            organizationalUnitName    = GlassFish
            commonName                = debian.demo.kvm
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Comment: 
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier: 
                36:33:F0:39:B6:3C:71:29:E3:0D:25:76:6F:70:46:EE:E5:73:97:C6
            X509v3 Authority Key Identifier: 
                keyid:BC:39:85:FF:DB:6E:46:FF:08:52:32:6F:97:65:91:F6:F4:A9:24:E7

            Authority Information Access: 
                OCSP - URI:http://debian.demo.kvm:3456/

            X509v3 CRL Distribution Points: 
                URI:http://debian.demo.kvm/my.crl

Certificate is to be certified until Sep 15 16:56:14 2012 GMT (365 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

# openssl x509 -in server.pem -outform DER -out server.der

And finally the certificate is imported into the keystore.jks glassfish store. Cos the certificate is signed by an unknown CA the cacert is also imported in both (user and trusted CA files). So now three commands are executed: import CA file in cacerts.jks, import CA file into keystore.jks and finally import the new server certificate in keystore.jks.

# cd /opt/glassfish3/glassfish/domains/domain1/config
# keytool -import -trustcacerts -alias myca -file /etc/ssl/cacert.der -keystore cacerts.jks
Enter keystore password:  
Owner: CN=ca.demo.kvm,O=demo.kvm,C=ES
Issuer: CN=ca.demo.kvm,O=demo.kvm,C=ES
...
Trust this certificate? [no]:  yes
Certificate was added to keystore

# keytool -import -trustcacerts -alias myca -file /etc/ssl/cacert.der -keystore keystore.jks
Enter keystore password:  
Owner: CN=ca.demo.kvm,O=demo.kvm,C=ES
Issuer: CN=ca.demo.kvm,O=demo.kvm,C=ES
...
Trust this certificate? [no]:  yes
Certificate was added to keystore

# keytool -import -alias server -file /etc/ssl/server.der -keystore keystore.jks
Enter keystore password:  
Certificate reply was installed in keystore

At this point we have a valid certificate with the alias server ready to be used in the https listener. In the glassfish console (http://debian.demo.kvm:4848) we select Configuration → server config → Network Config → http-listener-2. Here we see that security is enable in this listener, clicking the SSL tab the Client Authentication checkbox is selected and in the Certificate Nickname current auto-generated certificate s1as is changed for the new server alias.

This change does not need a reboot of the server. At this moment the 8181 port is using the new certificate (server alias) but it is also demanding a user certificate (so you only access the welcome page if a user certificate is installed in the browser, it will be done in the last step).

Publication of the CRL and setup of the OCSP server. The main idea of this demo is that the Application Server (glassfish) was able to validate user certficates. To do that both OCSP and CRL techniques are going to be available.

Openssl has a minimal (only recommended for testing) OCSP server, we can start the process directly from the command line. As we have specified port 3456 in the certificate URI the daemon is started as follows (the -text option dumps communication to the terminal):
```
# cd /etc/ssl
# openssl ocsp -index demoCA/index.txt -CA cacert.pem -rsigner cacert.pem \
  -rkey private/cakey.pem -port 3456 -text
Enter pass phrase for private/cakey.pem:
Waiting for OCSP client connections...
```
This way the client certificates can now be checked. The validation process uses the serial number of the certificate (a kind of identifier the CA assigns for every certificate). In the first step a serial file was initialized to 0x1, this way the CA has been assigning a consecutive number to each generated certificate (0x1 for cacert, 0x2 for client1, 0x3 for client2 and 0x4 for server). The client1 certificate (0x2) is a good one:
```
# openssl ocsp -issuer cacert.pem -nonce -CAfile cacert.pem \
  -url http://localhost:3456 -serial "0x2"
Response verify OK
0x2: good
	This Update: Sep 16 16:51:49 2011 GMT
```
But the client2 certificate (0x3) is revoked:
```
# openssl ocsp -issuer cacert.pem -nonce -CAfile cacert.pem \
  -url http://localhost:3456 -serial "0x3"
Response verify OK
0x3: revoked
	This Update: Sep 16 16:51:51 2011 GMT
	Revocation Time: Sep 16 16:50:19 2011 GMT
```
CRL is a complementary way of validating certificates, it can be intended as a secondary method to use in case of an OCSP error. The CRL (revocation certificate list) is downloaded using standard http by the checkers (app server, browser,...). This way I decided to install an apache server in the box and then copy the generated file inside document root directory.
```
# apt-get install apache2
```
After that the CRL list is generated using openssl:
```
# cd /etc/ssl
# openssl ca -gencrl -keyfile private/cakey.pem -cert cacert.pem -out my.crl
```
And it is copied to the apache2 root directory in debian:
```
# cp /etc/ssl/my.crl /var/www/my.crl
```
If you check the previous URL set in the certificate was http://debian.demo.kvm/my.crl which is exactly where my CRL is now.
CRL and OCSP checking in the glassfish server. By default glassfish server only checks the CA, it must be a trusted one (that is why cacert was imported to cacerts.jks store) and dates, but it does not validate revocations (neither OCSP nor CRL). This entry explains how to do it in glassfish v2 but it also works in v3 (the solution consists in activating the checking in JSSE implementation which I suppose works in any server that uses this ssl socket layer).

In the glassfish console Configurations → server-config → JVM Settings → JVM Options tab two new options have to be set (-Dcom.sun.net.ssl.checkRevocation=true and -Dcom.sun.security.enableCRLDP=true). Restart is required when JVM options are modified.

But this options only activate CRL checking and not OCSP. In order to finally have all the validation methods the /usr/lib/jvm/java-6-sun/jre/lib/security/java.security has to be modified uncommenting the following line (please think that this path varies depending where your JDK is installed, this is the default path for debian Oracle JDK):
```
ocsp.enable=true
```
If you check in this file there are more options to customize ocsp validation (I did not spend more time on this).

Finally glassfish is ready to receive and validate client certificates using both techniques. Java implementation first tries OCSP protocol and, if it fails, the CRL is downloaded (check commented post for further info).
Setting up client certificates. Now all the server side is ready but our browser needs some certificates, client1 and client2. If you remember both were exported in PKCS#12 format and both p12 files can be directly imported to firefox. Of course remember to also import the cacert.der as a trusted CA.

In firefox (iceweasel) 5.0 Edit → Preferences → Advanced Tab → Encryption subtab → View Certificates. In Your Certificates tab click the Import button and select client1.p12 and client2.p12 (the password that you inserted when the certificates were exported is requested at this point).

The cacert.der is imported in the same window but in Authorities tab. There click again Import. Check all the trust settings (web sites, email users and software) when requested.

Finally all the environment (server and client) is ready to access the glassfish secure port. In the following video I first show that my two certificates are imported and the CA is trusted. Then I access the 8181 port of my debian box, as client certificate was configured the browser asks me which one to use, but I select the revoked certificate (see it has the 03 serial). So the glassfish server does not permit the access (it is revoked!). Trying again I select the other certificate (02 serial) and, this time, welcome glassfish page is shown.

Your browser doesn't support the video tag. See this entry for more information about how to see the videos in this blog. You can download the file instead.

If you recheck this entry I have not done anything yet. Only the environment is ready and no application is deployed inside glassfish. At first moment I was not going to post the setup but it needed so many steps and particular options that I finally decided to publish this entry. Entries in this blog are many times posted just for me, because I want to have a clear guide of what I did. This is one of these times, but of course it is also here for anyone who finds it useful. I have two more entries in mind. In the first one I will try to deploy a custom security application (not using JavaEE security but programmatic). And in the second JavaEE security will be incorporated. But you know, it will depend on my mood and my time.

Don't change the channel!

Ricky's Hodgepodge: Tomcat 7 and the Invoker Servlet

nospam@example.com (rickyepoderi) — dom, 25 sep 2011 14:24:00 +0000

Some weeks ago I was on vacation and spent some days in my home town. One afternoon I was with an ex-colleague who requested my help to migrate a Tomcat application (you know, with friends like that...). In summary the application was a huge (lots of classes) and old (I think it runs right now in tomcat 4.1) servlet and JSP web app. He was trying to migrate it to a newer version of tomcat (6 or 7) but he is not a specialist in this matter.

In summary the main problem was that this application uses the Invoker Servlet which I did not even known before. It is a strange Servlet used in previous versions of tomcat which is now deprecated in version 6 and totally removed in 7. It seems that the Invoker is a dynamic servlet which allows run-time loading of other servlets based on class name. Currently it is considered evil and that is the reason for its deprecation and clean up. I thought this Servlet needed to be removed but my mate explained me that the application had dozens and dozens of servlets (and references in JPSs too) and it would be a total nightmare. So, focusing in version 6 and after some configurations changes, the application started successfully in tomcat 6 and it begun to work using the Invoker Servlet. My friend still had (and has) a lot of work to do but he needed this little push to start on. Then he kindly paid the beers I deserved.

But turning the issue over in my mind later I decided to find a direct way to replace the Invoker Servlet in tomcat 7. Obviously you can workaround the problem just getting the servlet code and putting it inside your project but, you know, this is not my way. I am going to try the following two ideas:

In order to not change any reference to the servlets, all of them need to be mapped following the exact way Invoker does (Invoker uses the path /servlet/servlet.class.fully.qualified.name). Obviously keeping the class name in the request is not recommended in terms of security but, at least, no dynamic re-thrown and common servlet mapping is used.
The other snag is that the application can potentially have hundreds of servlets (I know, this is crazy but I am also sure that many of you have seen an application like this at least once in your life). Besides my mate is a bit lazy, so I am sure he is not going to check all the links to find which are the servlets to map in the configuration file. This way my idea is using new Servlet 3.0 programmatic registration to search, create and map them. Only servlets under some specified packages will be checked.

It is important to understand that this is not a good solution (it is still evil although a bit less). Defining and mapping all your servlets in the web.xml is the best solution and never use the qualified class name as the map name. This workaround needs to be understand as a temporary step that let you use last versions of tomcat with Invoker Servlet applications (huge ones, in little ones just map your servlets).

TOMCAT6

Once everything is explained I am going to deploy an Invoker Servlet application inside tomcat 6 and then move it to tomcat 7. In order to setup the Invoker inside version 6 you need to uncomment invoker definition and mapping in ${TOMCAT6_DIR}/conf/web.xml:

    <servlet>
        <servlet-name>invoker</servlet-name>
        <servlet-class>
          org.apache.catalina.servlets.InvokerServlet
        </servlet-class>
        <init-param>
            <param-name>debug</param-name>
            <param-value>0</param-value>
        </init-param>
        <load-on-startup>2</load-on-startup>
    </servlet>

    <servlet-mapping>
        <servlet-name>invoker</servlet-name>
        <url-pattern>/servlet/*</url-pattern>
    </servlet-mapping>

The context needs to be defined as privileged (I think tomcat guys do that in order to make clear that this is evil) inside the ${TOMCAT6_DIR}/conf/context.xml. You just need to add privileged="true" to the context tag:

<Context privileged="true">

Finally a simple application was deployed (here it is the netbeans project for version 6). This application has two servlets: InfoServlet.java (typical servlet that shows some request variables like url, path, cookies, headers, parameters and so on) and HelloServlet.java (a servlet that forwards to a JSP and just says hello). Besides I copied both classes in three different packages (sample.invoker.servlet, sample.invoker.other and sample.invoker.another), I like to test Invoker against several packages and servlet classes.

TOMCAT7

Tomcat 7 supports new Servlet standard version 3.0 and the main idea is to programmatically register the servlets found inside some pre-defined packages. All this code has been placed in a context listener (see the servlet 3.0 link I presented before). The listener needs to be added in the web.xml configuration file:

    <listener>
        <listener-class>
            sample.invoker.ctxlistener.InvokerLoadListener
        </listener-class>
    </listener>

In order to search for classes that extends HttpServlet (and implements ContainerServlet, an interface which is needed by the Invoker Servlet, I try to be as restrictive as possible) I followed the idea presented by vtatai in this post. But I extended it to search through files and jars. Basically the code gets the resources with the specified package using the ClassLoader and then only those which are file: or jar: are inspected, reflection is used to check if the class is a servlet. Using an init parameter in the web.xml you can specify one or more package names to take into account (only servlets under these packages will be registered).

    <context-param>
        <param-name>invoker.packages</param-name>
        <param-value>
            sample.invoker.servlet,
            sample.invoker.other,
            sample.invoker.another
        </param-value>
        <description>List of packages to check for servlets (comma separated)</description>
    </context-param>

The final part performs the dynamic registration. The complete InvokerLoadListener.java and netbeans project for tomcat 7 can be downloaded.

Here you see my video. I start tomcat 6 and execute InfoServlet and HelloServlet (different packages). Then I stop version 6 and start the version 7 (I did not change ports to have both tomcats running). Executing the same servlets you can check that both results are exactly the same.

Your browser doesn't support the video tag. See this entry for more information about how to see the videos in this blog. You can download the file instead.

Today entry tries to explain how to workaround the Tomcat Invoker Servlet in new version 7 (this last version has removed the servlet because of its evilness). The solution uses Servlet 3.0 API, class loaders and reflection to dynamically register all the servlets in the application. It is a bit less insecure (there is no redirection, standard mapping and only servlets under specified packages are included) and, more important, it works inside Tomcat 7. Of course the context listener that searches for servlets can take some time to find all of them, but if your servlets are located in a few packages it is not so much, besides think this code is only executed once at startup.

David, you owe me another beer!

Ricky's Hodgepodge: Adding Sharing Buttons to the Entries

nospam@example.com (rickyepoderi) — mar, 13 sep 2011 15:53:00 +0000

The other day I read some recommendations about how you can improve your blog impact (I am sorry but I cannot find the link again and besides I think it was in Spanish). One of the main advice was adding the typical sharing buttons to your blog entries (you know, the buttons to share via digg, reddit, facebook like or tweet this, I am sure you have seen these buttons dozens of times). I am not a fun of social networks (no twitter, my facebook account was requested to be closed,...), I usually read some bookmark pages (digg, slashdot or Spanish menéame) but always anonymously, and the number of visits to the blog is merely a matter of pride. So, although it really does not matter very much, I am completely agree with that advice, anything that let you spread your entries it is clearly a good idea.

So from now on when you enter to see a single entry some links to share it will appear just after the post information and before trackbacks and comments. I have just added an AddThis toolbar after reading this blog entry. I tested nothing, as I said I have no account in any of these sites, so if you detect any problem sharing or bookmarking an entry please comment below.

Thank you all!

Ricky's Hodgepodge: Testing WebGL

nospam@example.com (rickyepoderi) — jue, 08 sep 2011 09:14:00 +0000

One of the common topics of this blog is the HTML5 standard and its multiple new features. Today I am going to test WebGL, which is a standard specification to add 3D graphics in a browser canvas element without using any plug-in or external component (so it is not part of HTML5 itself but depends on it). Obviously this feature is deeply connected to graphic hardware acceleration because any WebGL application needs to be GPU accelerated in order to run smoothly. WebGL is a breaking new feature in the web world and another element of great controversy among browsers. If you remember, the canvas element was already commented in a previous series of this blog, but there only 2D graphics were shown.

WebGL standard is being driven by the Khronos Group and all players in the web are involved (Apple, Google, Mozilla and Opera) except Microsoft. The corporation headquartered in Redmond rejects this standard and has declared several times that IE will not support it, they consider WebGL harmful in terms of security and they are not totally wrong.

The summary of the current situation is more or less the following. Chrome (initial webGL support in version 9) and firefox (version 4 also added WebGL) support WebGL in Windows (and I think Mac OS) and it is enabled by default. Opera does not support it although there is a WebGL preview version for its version 11.50 only for windows. On Mac OS Safari was adding the support in their WebKit nightly builds but maybe in Lion WebGL is already supported (sorry but I am not a Mac OS user). There are some summaries of the supportability matrix, for example in the Khronos page or in the first chapter of this tutorial, but this is a constantly changing subject so please always re-check. Nevertheless the main problem comes in the Linux world. Here there are a lot of problems related to the implementation of many graphic drivers, firefox reported issues with Linux GPU drivers and both chrome and firefox added a gpu blacklist to disable some features (WebGL mainly) depending the driver and the OS involved. Firefox 6 and 7 are supposed to remove more and more drivers from the blacklist, same behavior is expected in the next versions of chrome, as soon as linux driver implementations became more reliable. In summary current browsers in Linux hardly support the nvidia binary blob as the only out of the box WebGL driver (at least in my case my two boxes are blacklisted, my desktop uses gallium R300 driver and my laptop classic intel stack).

After this little introduction I am going to present a quick demo of WebGL. I am very, very interested in porting my PFC project to a web version. It is incredible to me how a 12 years old project (which had to be run inside a SGI server) can now be run inside a browser. If you remember what I told in that entry, I lost my code and all the data (mesh and textures) because of a broken CD, so this implementation is only a simple demo with new and fixed data (there is a lot of room for improvement). The demo has the following features:

A fixed height mesh is retrieved using a static json file. This file contains the starting latitude and longitude of the mesh, the number of points (rows and columns), the length or step of each side, the matrix data and a lot of configuration parameters. This mesh is not drawn completely. In OpenGL and WebGL (they are very similar) the idea is simple, all the figures are sent to be drawn and depending the perspective, camera and other elements the engine draws what is needed. But in my PFC the mesh was so big that only a part of it was sent to be drawn. I have followed the same idea in this demo.
Basic camera management. The camera is implemented using its own axis, this way the camera can turn and move following natural keystrokes and mouse movements.
Tile representation and painting. In the real application every satellite image was the texture for a tile mesh (50x50 points). This way the part of the mesh that corresponded to an image was painted independently (the application drew each square of mesh with the associated texture). I have implemented the same here but, in the demo, there is only one tile (a 8x8 tile) which is repeated all the time horizontally and vertically (11 times in each direction).
Easy texture loading. A simple FIFO queue has been implemented to load and unload images. Actually this is not used in the demo cos only one image is used (there is only one tile which is repeated, so only one texture is needed and the queue is never filled).
Basic ambient and directional light for emulating the sun (in my PFC there were more environmental effects like sky, fog,...).
Management of the configuration parameters. I have used the new input type number which is only supported in some browsers. HTML5 is everywhere.

The whole demo is below, integrated in the entry using an iframe. It is a pity that I cannot have at my disposal the real data from my PFC (damn CD). As I said a simple pattern (only 8x8 points) is repeated all the time (11x11 tiles). This way I do not upload a lot of images to the server. How to navigate using keyboard or mouse is explained in the right part of the page.

Remember what I have said before, in order to test this example you need a modern browser if you use Windows (firefox, chrome or the Opera 11 preview but not IE9). In Linux you need firefox or chromium but usually you need to disable the black list. Debian testing packages have just been upgraded to Chromium 13 and iceweasel 5 respectively and both support WebGL. In chromium the blacklist has to be ignored starting the browser with the following option:

$ chromium --ignore-gpu-blacklist

In Firefox I have changed some webgl settings (in the about:config page):

webgl.disabled	false (default)
webgl.force-enabled	true
webgl.force_osmesa	false (default)
webgl.osmesalib	/usr/lib/x86_64-linux-gnu/libOSMesa.so.6
webgl.prefer-native-gl	true
webgl.shader_validator	true (default)
webgl.verbose	true

And the browser has to be executed with the following environment variable:

$ MOZ_GLX_IGNORE_BLACKLIST=1 iceweasel

This entry is doubtless the one that I have spent more time working on it. I was developing the JavaScript code long hours and I have to admit that I started it some time ago (last months I have been quite busy and I was never in the mood to finish it). Documentation about WebGL is very poor and thanks to this good tutorial I could start the development, but when you need something special, custom or a mix of things you usually wasted much time just trying to identify how to do it. In summary current WebGL support is quite a mess (much more in the Linux world) but it is an essential feature to move gaming to the web. When I saw that quake was available inside a browser I was totally shocked. I personally think this technology is unstoppable, no matter how buggy or insecure it was. Needless to say, flash is the current option for all of this, so WebGL should be very buggy and very insecure to be unworthy.

Long life to the web wars!

Pon un sol en tu vida: How Intel/AMD (inadvertently) fixed GNU Hurd

nospam@example.com (Sergio Lopez) — dom, 04 sep 2011 08:29:00 +0000

Last Friday, on Hurd's IRC channel at freenode, we've accidentally noticed some machines were able to run certain operations on Hurd as KVM guest up to 10x faster than others. As antrik correctly guessed, this is the effect of Intel's Extended Page Tables, which allow the guest operating system to deal with it's own page faults. I suppose AMD's Rapid Virtualization Indexing could have a similar effect, but I don't have the hardware to be able to test it (please write to the mailing lists if you've been able to check it).

Since most people (including myself) are running Hurd in a virtualized environment, having the ability of taking advantage of this circumstance by moving to hardware with this technology supposes a great improvement, heavily reducing compilation times and increasing the interactivity of the entire system.

Let's see an small example of the difference between running with and without EPT.

Running without EPT (modprove kvm-intel ept=0):

root@debian:~# dd if=/dev/zero of=/dev/null bs=256k count=1000

1000+0 records in

1000+0 records out

262144000 bytes (262 MB) copied, 2.08 s, 126 MB/s

And with EPT:

root@debian:~# dd if=/dev/zero of=/dev/null bs=256k count=1000

1000+0 records in

1000+0 records out

262144000 bytes (262 MB) copied, 0.23 s, 1.1 GB/s

I think there will be interesting times for GNU Hurd.

Ricky's Hodgepodge: Videos Are About to Change (Second Time Lucky)

nospam@example.com (rickyepoderi) — mar, 23 ago 2011 15:41:00 +0000

Long time ago the way videos are presented in this blog were about to change. I was going to replace my useful flash flowplayer with the new HTML5 video tag. I changed my mind because video were not supported in any stable IE and there was some warfare about the supported codecs in the rest of the browsers.

At the same time I published that entry google made public the WEBM video+audio format, they created a webm project and since then the new codec has been quickly adopted by browsers. Only IE and Safari do not support WEBM by default, but this time google solved the issue announcing plugins for both OS integrated browsers (they are in a very early stage). Microsoft IE9 was released to the public in march of this year, with a lot of the new HTML5 features (included video and audio support). Iceweasel testing package was upgraded to 5.0 last week in Debian (the browser was still at 3.5 version before). Therefore all my requisites to trigger HTML5 solutions are accomplished and there is no reason to not change my videos.

From now on the videos are going to be presented using the video tag and only in WEBM format. Just like this:

<video src="out.webm" autobuffer controls>
  <img src="error.png" alt="Error!" />
  Your browser does not support the video tag. See this 
  <a href="http://blogs.nologin.es/rickyepoderi/index.php?/archives/41-...">entry</a> 
  for more information about how to see the videos in this blog. 
  You can <a href="out.webm">download the file</a> instead.
</video>

And this is my last video in the new format just as an example (video used in Backing Up Data Into an Encrypted External Disk entry):

Your browser doesn't support the video tag. See this entry for more information about how to see the videos in this blog. You can download the file instead.

So maybe you have reached this post because you cannot see my videos. At this moment browser status about video and WEBM is the following:

Firefox: Firefox supports both video and WEBM since 4.0 version.
Chrome: Both are supported since 6.0.
Opera: Video supported since 10.50, WEBM since 10.60.
IE: IE9 supports video tag but WEBM is not a valid format out of the box. You need to install a plugin offered by the webm project. Remember that IE9 cannot be installed in Windows XP.
Safari: Video is supported since 3.1 but again WEBM needs an external codec. Here there are two options, the plugin from google and perian, an open source QuickTime component that adds native support for many popular video formats (it supports webm since 1.2.3). Some friends who have tested both solutions recommend the perian option (it seems perian is very common in the Mac world and google plugin gives some problems right now, sorry but I am not a mac guy).
Others: If you are using any other browser you are a geek, so I hope that you know what you are doing and how to see an HTML5 video with WEBM format.

As a summary, in most cases if you cannot see the videos you only need to upgrade your browser to the last version. Besides if you are using Windows integrated IE9 or Mac OS integrated Safari you need to manually install a WEBM codec plugin (see previous links). Finally if you are using IE8 (or previous) in a Windows XP box I strongly advise you to change your browser for another one (please do not get stuck).

HTML5 is here! At least for my videos.

PS: Thanks to Luis who made some changes in the blog system installation to make this work (adding webm mimetype and IE9 compatibility meta tag). Thanks also to Jaime, Mariano, Nacho, Ramón and Victor (both) who tested the options in Mac OS / Safari.

Ricky's Hodgepodge: Backing Up Data Into an Encrypted External Disk

nospam@example.com (rickyepoderi) — mar, 16 ago 2011 17:09:00 +0000

Today entry deals with a common and usually annoying task, backing up your laptop, so this time the post is going to be more useful for me than for anybody else. Last week my company gave me a 2TB USB 3.0 disk to make backups of my laptop data. But there is a requirement, the backup File System should be encrypted. As always there is an absolutely clear procedure to make all the task in Windows but not in Linux or Mac. Besides the procedure involves a closed source program (a McAfee application) to encrypt the external disk. In my opinion the requirements have never to mention particular applications and only guarantee the final goal. I can understand that a lot of custom solutions would be a mess for the company (to recover data when a employee leaves the company or whatever) but, please, at least think about all the OSes and provide a step by step procedure for all of them (Windows, Mac and Linux).

The mate that gave me the disk finally told me that, as I use Linux, I did not have to encrypt the disk. But searching for information over the Internet (I knew nothing about disk encryption before) I realized that this task would not be very difficult. I found two good pages: 7 Steps To An Encrypted Partition (local or removable disk) and Setting up an encrypted volume on an external hard drive on CentOS. I also found a video of how to install Debian on a encrypted root file system (this task has nothing to do with the backup but I think it will be the next requirement for my laptop). So I finally decided to encrypt the disk anyways. This entry summarizes the setup for the two tasks using Debian, encrypting the disk and performing the backup.

Step 1. The first step is configuring the partitions inside the disk. I will use two partitions, one encrypted (for the backup and confidential or customer data) and other not (for general use). The first time I plugged the disk my Debian box recognized it as /dev/sdb, so from here the /dev/sdb1 will be the encrypted and /dev/sdb2 the normal partition.

# fdisk /dev/sdb

I deleted the FAT32 partition the disk came with and created two new ones. First partition (/dev/sdb1) of 1TB:

Partition number (1-4, default 1): 
Using default value 1
First sector (2048-3907029167, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-3907029167, default 3907029167): +1024G

And the second one (/dev/sdb2) with the rest of the disk:

Partition number (1-4, default 2): 
Using default value 2
First sector (2147485696-3907029167, default 2147485696): 
Using default value 2147485696
Last sector, +sectors or +size{K,M,G} (2147485696-3907029167, default 3907029167): 
Using default value 3907029167

After saving the changes the disk presents the following two partitions:

# fdisk -l /dev/sdb

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xee7a7a06

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  2147485695  1073741824   83  Linux
/dev/sdb2      2147485696  3907029167   879771736   83  Linux

Step 2. It seems that for encrypted devices it is compulsory to write some random data in the partition (it is easier to decrypt a blank initialized disk). For this step there are various options (see the links I presented before) but I finally executed the badblocks command:

# badblocks -c 10240 -s -t random -v -w /dev/sdb1

Take care, in my 1TB partition this step took hours (my disk is giving around 26MB/s write rate).

Step 3. Now it is time to create the encrypted file system in the /dev/sdb1 disk partition. The method used in Linux to encrypt File Systems uses the dm-crypt kernel module and creates a virtual device below /dev/mapper (it is a kind of wrapper over the real /dev/sdb1 but encrypting the data before writing on disk).

First of all the package for the management command needs to be installed, the module has to be loaded in the kernel and added to the /etc/modules file (modules in this file are loaded at boot time).

# apt-get install cryptsetup
# modprobe dm-crypt
# echo "dm-crypt" >> /etc/modules

Once the module is in place the new encrypted mapper is created above /dev/sdb1. I have used default values and normal password setup (it is also possible to use a key file instead of a password, see this archlinux page for more information about this).

# cryptsetup luksFormat -v -y /dev/sdb1 

WARNING!
========
This will overwrite data on /dev/sdb1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: 
Verify passphrase: 
Command successful.

The default cypher is aes-cbc-essiv:sha256, sha1 for password hashing and 256 for key size (all this values can be changed using -c, -h and -s options of the command respectively). This is the summary for my device.

# cryptsetup luksDump /dev/sdb1 
LUKS header information for /dev/sdb1

Version:       	1
Cipher name:   	aes
Cipher mode:   	cbc-essiv:sha256
Hash spec:     	sha1
Payload offset:	4096
MK bits:       	256
UUID:          	ff088328-138a-4ca7-aa0a-5cbdd2064ce8

Key Slot 0: ENABLED
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Finally the new encrypted device is opened with backup name (the password is requested at this point).

# cryptsetup luksOpen /dev/sdb1 backup
Enter passphrase for /dev/sdb1:

After the luksOpen a /dev/mapper/backup device is ready to be initialized and used.

Step 4. I have decided to use the new btrfs file system. The reason is that this next generation file system manages some nice features like on the fly compression and snapshots, encryption and some synchronization integration are also planned (see the project ideas). Obviously compression will save some space in the disk and snapshots let us create some backup copies with a minimal cost in terms of time and space (my idea is backing up data and maintaining some snapshots of the backups).

The btrfs module is currently marked as experimental in my 3.0.0 Debian kernel but its web page claims that since 2.6.31 only forward compatible disk format changes are planned. In order to create the FS the tools package has to be installed before.

# apt-get install btrfs-tools

Now the FS is made with BACKUP name.

# mkfs.btrfs -L BACKUP /dev/mapper/backup

WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

fs created label BACKUP on /dev/mapper/backup
	nodesize 4096 leafsize 4096 sectorsize 4096 size 1024.00GB
Btrfs Btrfs v0.19

The new FS is mounted for testing and its owner changed to my common user.

# mount /dev/mapper/backup /mnt
# df -h /mnt
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/backup   1008G  200M  957G   1% /mnt
# chown ricky:ricky /mnt

At this point the FS can be unmounted and closed. I will never mount the partition again manually, the idea is that gnome mounts it automatically (password is requested as soon as LUKS FS is detected).

# umount /mnt
# cryptsetup luksClose /dev/mapper/backup

Step 5. Now it is time to think about the backup. The easiest way to backup some directories in Linux is using the rsync utility. This application is a powerful, fast and versatile file copying tool which supports local and remote (ssh for example) copies. The idea for the backup is the following:

When the external USB disk is plugged gnome mounts them (both partitions) automatically. For the encrypted file system the password is requested. Inside the root mount point a backup script is located, just clicking on it and running it the backup is started. Gnome automounter mounts the FS with default options. That is the reason I finally decided not to use the compression option of btrfs (UDEV is used for the automatic mount, and perhaps a new UDEV rule can be used to change the default behavior but I preferred not to complicate the solution).
This script called backup-system-gksudo.sh is a simple script that just opens a new gnome-terminal executing gksudo (gnome graphical sudo) over the real backup script backup-system.sh (btrfs command needs root access).
The backup-system.sh script reads a local and hidden configuration file called .backup-config. This file has some general variables like the directory to maintain in sync (usually it would be / directory but, in my case, /home is where all my data are placed), number of snapshots to maintain, directory for host backups, file for excluded dirs,...
First the script creates a new btrfs sub-volume for the specified box (this way the same disk can hold backups for several boxes). The volume is placed in host-backup/$(hostname) directory (the first directory name is customizable via the configuration file). If the volume already exists this step is just skipped.
After that the script continues executing the rsync command. The command is called with the following options:
```
rsync -a --delete -v --stats --progress \
  --exclude-from="${dirName}/${EXCLUDED_DIRS_FILE}" \
  "${SOURCE_BACKUP_DIR}" "${hostDirName}"
```
With this execution the SOURCE_BACKUP_DIR is in sync against the external and encrypted btrfs volume. The -a options is for archive mode, --delete option removes files deleted in the source directory, and -v, --stats and --progress options are for showing some information while copying. The --exclude-from option let us exclude some directories from the copy, the pattern file is defined inside the configuration (in my case this var points to a .exclude-dirs file that contains entries like videos or comics which are huge directories I do not like to backup). This step can take hours the first time but it is quite fast when only changes are sent.
When the rsync command finishes a btrfs snapshot is taken. This snapshot is located at the same level of the host sub-volume but labeled with the current date (the snapshot name is like this $(hostname)-YYYYMMDDHHMMSS).
Finally old snapshots are deleted (in the configuration file there is a MAX_SNAPSHOTS variable, if the snapshots are more than this number the old ones are removed).
At the end of the script a message is shown to the user. Zenity command is used to tell the user that the backup has been successfully done or the script has terminated with errors.

Here a video is presented to show the backup execution. It is at accelerated speed, the real backup took around two minutes. First the disk is inserted and gnome requests the password for the encrypted partition. When the device is automatically mounted the graphical script is executed (first I show that only one snapshot is in the hosts directory). This script launches a new gnome-terminal, after the root password is also requested the backup (rsync) and snapshot are performed. Once the script finishes a successful message is shown. Finally I enter in the host directory and show the just created snapshot directory.

In summary this entry comments how to implement a backup utility over an encrypted external disk in Linux (Debian). I have written it just to remember what I did (I had never worked with this utilities, cryptsetup, dm-crypt, btrfs or rsync). As you have seen my idea is keeping it as simple as possible. Only the chosen File System (btrfs) is a bit strange, but thinking about current and announced features it seems to be the proper one (maybe in the near future btrfs alone can do all the stuff explained here, sync and encryption included, avoiding the use of any other program). Please reuse and adapt it if you think it is useful for you.

Remember to backup your ideas too!

Ricky's Hodgepodge: Changing Again the Way I Listen to Music

nospam@example.com (rickyepoderi) — s�b, 16 jul 2011 21:11:00 +0000

I have just requested my Spotify free account to be deleted (it is done via email). I had this account since nearly the beginning cos I was moved from LastFM. It seems the story repeats again and, much to my sorrow, it will not be my last time. I was a loyal user of LastFM for years (besides there was a plugin for rhythmbox which I loved) but in 2009 they decided I needed a subscription. I changed to Spotify which impressed me. It was not a radio, it was your personal music device but with almost all the songs of the world. Of course the snag was that there was no native Linux application, but wine worked well enough for me. Sadly Spotify changed free account access some months ago, since then I could only listen to music ten hours per month, and they are very few! So I needed to change again. I decided to use Grooveshark. It is just a web page where you can also select the songs you like to listen. When I discovered you can drag some songs, turn on the radio and it keeps playing forever choosing similar songs, I decided it would be my first choice. Here the problems are that the page uses the annoying flash player and there are no songs of The Chemical Brothers (not even The Test, a glaring error as my brother says ). But, again, it is good enough for me (dammit! I just want to listen to music!). I am going to use the chromium application shortcut feature, I prefer it in a separate and clear window.

This week I found that Grooveshark sometimes stops playing and asks if you are still there (obviously it happens when you let it playing for a long time). It is really a pain in the ass, you have to stop what you are doing and click that yes, I am fucking here. Please take this into account. I am not sure what I hate more, this or Spotify ads (they were annoying but the music was never stopped).

Well, Grooveshark is my new way to listen to music... How long will it last this time?

Pon un sol en tu vida: Sometimes, swap space still matters

nospam@example.com (Sergio Lopez) — vie, 15 jul 2011 17:19:07 +0000

Most modern, general purpose Operating Systems (OS), come with a full-fledged Virtual Memory (VM) system that generates the illusion of having more memory than the real amount installed in the machine. Whether this virtual memory is backed by real RAM, disk swap or it just isn't backed by any physical device, is something up to the OS.

Continue reading "Sometimes, swap space still matters"

Nolife at Nologin: Grandes frases

nospam@example.com (bufon) — vie, 15 jul 2011 12:40:40 +0000

Empiezo una sección de grandes frases dichas en el trabajo, en esta nolife.

"El hombre invisible no es un voyeur"

Gersius

Ricky's Hodgepodge: Testing the IcedTea-Web Java Plugin

nospam@example.com (rickyepoderi) — s�b, 09 jul 2011 17:44:00 +0000

Some days ago I realized that debian packages related to java plugin in OpenJDK/IcedTea had been upgraded in testing to the new IcedTea-Web infrastructure. The IcedTea-Web is a redesign of the old gcjwebplugin and Netx JavaWS implementations. This entry talks a bit about the complex story of java web-plugin in OpenJDK and then a little testing of the new plugin is presented.

If you have been interested in the process of open-sourcing the Sun Java Virtual Machine (called OpenJDK) you already know that the Java web-plugin and Java Web Start (JavaWS) were never incorporated. The first component is the browser plugin that let HTML pages to integrate Java Applets inside them. If you remember I have talked about applets two times in this blog, first in the signature applet series and second in the scatter plot trilogy. JavaWS is a similar technology to Applets but not as tightly coupled with the browser, here Java applications run outside it but interaction is also possible. JavaWS is also known as Java Network Launching Protocol (JNLP) because of the definition files and the protocol used to launch the applications.

When Sun announced the possibility of open-sourcing the JDK in the JavaOne conference of 2006 there were some parts of the code which were covered by third party licenses (some GUI classes, the java audio system, some cryptography components and many more). For that reason the first versions of OpenJDK, released under GPLv2 license, were not complete and they had encumbrances. These parts had to be downloaded apart, as a binary blob, in order to get a full Java Virtual Machine. The plan was to gradually replace these proprietary components with alternative implementations. Nowadays OpenJDK has no more encumbered code or components and the sources for versions 6 and 7 can be downloaded from its web page (initially only JDK7 was to be open-sourced but finally JDK6 was included in a complex merge process). Nevertheless the web plugin and the JavaWS were not included in the free code neither in the encumbered parts (both technologies, although related to J2SE, are not part of it).

Update 10 of the non-free Sun JDK6 (not OpenJDK) replaced the Java plugin with a new and bright implementation. In the mids of 2009 Joe Darcy, at that moment the release manager of OpenJDK6, announced this new implementation was to be included in the OpenJDK tree. But some updates after and with no signs of the new plugin Joe himself confirmed that circumstances had changed. For one reason or another both technologies (Applets and JavaWS) were never moved from non-free implementation to OpenJDK.

Separately (but very close to OpenJDK) the IcedTea project is the effort of RedHat of making the OpenJDK usable without any encumbered bits as soon as possible. This project quickly added to OpenJDK the gcjwebplugin, a plugin implementation started as a Savannah project but later moved to the GNU classpath, and Netx, an alternative implementation of the JNLP protocol. The problem with these two projects was that both were old implementations and quite buggy. If you remember one of my reasons to start the HTML5 scatter graph study was the gcjwebplugin, default plugin for all linux distros.

I do not know the real reasons (maybe Sun announcement of plugin integration, maybe both technologies are not commonly used,...) but the real thing is that the plugin progress has been slow. IcedTea was born in 2007 using old implementations, in 2009 Deepak Hole announced the new IcedTea plugin in an enormous step forward, in 2010 it was integrated in IcedTea for the first time and, finally, in 2011 IcedTea-Web was released as a separated sub-project of the general IcedTea project.

Current version of IcedTea-Web is 1.1 and it is the version which has recently arrived to testing. Some days ago these packages were upgraded in my desktop box (I have installed OpenJDK there to see the progress of this implementation and not the non-free packages I usually use in my working laptop) and I decided to test how good the status of this new plugin is. I have to mention my previous experiences were very disappointing (when plugins were a little complex they never work). In order to do that I have checked the two most complex applets I have access to.

The scatter plot Applet I commented in the canvas entries. This applet is used in Puertos del Estado web page (a governmental organization that studies sea and harbor issues in Spain). This application uses Google Web Toolkit (GWT) and the applet is presented always a magnitude is plotted (several applets in the same page, lots of sample points,...). When I was working there one of my mates used an ubuntu box with OpenJDK and the applet crashes sometimes (usually when you were displaying several graphs, adding or removing them).
The Signature Applet I presented in the security series. Of course I am going to test the crytocard version using my Spanish eID (the most complex version, signed jar, JNI to the opensc-pkcs11 library and so on).

The conclusions are not very promising. It seems the new IcedTea-Web is evolving but it is not ready yet. With the scatter plot Applet I added and removed several graphs and two issues were found. When the plots are added to the page (a new div element is added) they are not properly re-painted, it is like the plot was not there (maybe this issue happens because the plots are added dynamically -javascript- to the page). Then I discovered the Sea Level plot hung the plugin. This magnitude plot (as I commented in its own entry) displays a lot of samples (more than 10,000) and it seemed IcedTea-Web could not manage this high number of samples. Using jconsole and looking at the code I found a loop that leaves much to be desired. Keeping my habits I developed a patch and reported a bug. I did not do anything with the first issue, it is quite related to the way applets are displayed and I do not have the source code of the applet.

In the Signature application more problems came out. First the PKCS11 provider did not load the certificates from the card. The KeyStore.getInstance(keyStoreType) method call does not work in OpenJDK/IcedTea-Web. I changed the call to KeyStore.getInstance(keyStoreType, provider) and it magically started to work. The second problem, when the signature was computed mail/activation libraries gave the following error javax.activation.UnsupportedDataTypeException: no object DCH for MIME type application/pkcs7-mime. It only worked when I copied Bouncy Castle libraries to the ext/lib OpenJDK directory. I do not really know what this issues are about but they seem related to security concerns, although this applet is self-signed (none of them happened in a simple java execution using OpenJDK -out of the applet- or in the non-free JDK plugin). Obviously the second Applet does not work (cos you cannot trust in a user copying files to lib/etx directory) but, as I commented in the signature entry, this Applet is just a PoC (Proof of Concept) and it does some strange things. I consider a real signature Applet (a one that just signs and/or encrypts) maybe would work.

It is important to comment that these two applets, although complex, work smoothly in the closed-source plugin. Finally I want to add that now the IcedTea-Web has a itweb-settings configuration command which emulates typical ControlPanel of non-free JDK (here there is also room for improvement but, you know, it is more than nothing).

I present a video where, using IcedTea-Web plugin inside iceweasel, I connect to the internet Plotter Applet and add three different graphs. As you see, applets are not re-painted (they need a click to force the rendering). But at least with my patch Sea Level plots are working. Then the Signature Applet is accessed (local application), once the libraries are in the lib/ext folder the mail is signed and sent without a problem.

It seems the IcedTea-Web is finally improving but nowadays it is still not the Java plugin that OpenJDK deserves. I did not test any JavaWS example (I have no one and it would be great if any of you could test some custom application) but I ran two applets which I know were problematic in OpenJDK before. Now both of them work better but some issues were still discovered. I know two tests are not many but they are meaningful to me. The best point is the plugin is now an independent project in IcedTea which I think guarantees better maintenance and improvement. Despite all this I am sure that if I have to develop a new applet or JavaWS application I will take this plugin implementation into account.

Keep on pushing IcedTea guys!

Ricky's Hodgepodge: Usign OpenMQ with Tomcat

nospam@example.com (rickyepoderi) — s�b, 18 jun 2011 11:41:41 +0000

Some weeks ago a company mate asked me some questions about a JavaEE application that needs a JMS queue for a special process. The customer wanted to install the software inside Tomcat and I warned her that Tomcat is just a Servlet container and not a fully JavaEE application server, therefore she needed another JMS server software. Although I have never worked with open source JMS servers I advised to use ActiveMQ but the problem was the customer already used OpenMQ. I tried to find some examples about this configuration but it was impossible, so I finally could not help very much. But yesterday I had some free time and I tried it by myself.

OpenMQ is the Oracle (formerly Sun) open source JMS server used in Glassfish. It used to be tightly coupled with the application server but the last versions are trying to split the two platforms. Nevertheless I was unable to find any example of how to integrate OpenMQ inside Tomcat, so this shows the division is just underway. I decided to follow a very good guide for setting ActiveMQ in Tomcat and then perform the needed changes.

OpenMQ can be downloaded from its web page and there is a simple ZIP distribution file. Just uncompressing the file a MessageQueue4_5 directory is popped out with all the necessary bits. From here this directory will be called {OPENMQ_DIR}.

$ unzip openmq4_5-binary-Linux_X86.zip

After installation the broker (the JMS java daemon) can be just started.

{OPENMQ_DIR}/mq/bin$ ./imqbrokerd &
[17/Jun/2011:17:14:46 CEST] 
==============================================
Open Message Queue 4.5
Oracle
Version:  4.5  (Build 29-b)
Compile:  Wed Feb  9 22:53:30 PST 2011

Copyright (c) 2010, Oracle and/or its affiliates.  All rights reserved.
===============================================
Java Runtime: 1.6.0_26 Sun Microsystems Inc. /usr/lib/jvm/java-6-sun-1.6.0.26/jre
[17/Jun/2011:17:14:46 CEST]    IMQ_HOME=/home/blog/MessageQueue4_5/mq
[17/Jun/2011:17:14:46 CEST] IMQ_VARHOME=/home/blog/MessageQueue4_5/var/mq
[17/Jun/2011:17:14:46 CEST] Linux 2.6.38-2-amd64 amd64 magneto (4 cpu) ricky
[17/Jun/2011:17:14:46 CEST] Java Heap Size: max=188416k, current=188416k
[17/Jun/2011:17:14:46 CEST] Arguments: 
[17/Jun/2011:17:14:46 CEST] [B1060]: Loading persistent data...
[17/Jun/2011:17:14:46 CEST] Using built-in file-based persistent store: 
/home/blog/MessageQueue4_5/var/mq/instances/imqbroker/
[17/Jun/2011:17:14:46 CEST] WARNING [B3168]: Invalid broker address for this broker 
to run in cluster: Loopback IP address is not allowed in broker address 
mq://127.0.1.1:7676/?instName=imqbroker&brokerSessionUID=6784163344567163648
for cluster
[17/Jun/2011:17:14:46 CEST] WARNING [B1137]: Cluster initialization failed. 
Disabling the cluster service.
[17/Jun/2011:17:14:46 CEST] [B1270]: Processing messages from transaction log file...
[17/Jun/2011:17:14:46 CEST] [B1039]: Broker "imqbroker@magneto:7676" ready.

Of course the software can be configured in more detail (OpenMQ supports high availability, different backends to store messages and so on) but it was not the goal of this entry. With this nonexistent configuration the broker uses the directory {OPENMQ_DIR}/instances/imqbroker, inside it there are several things: the configuration properties file, the directory where messages are stored by default,...

Once the broker is up and ready a queue called sampleQueue is created.

{OPENMQ_DIR}/mq/bin$ ./imqcmd create dst -n sampleQueue -t q -u admin
Password: 
Creating a destination with the following attributes:

Destination Name    sampleQueue
Destination Type    Queue

On the broker specified by:

-------------------------
Host         Primary Port
-------------------------
localhost    7676

Successfully created the destination.

OpenMQ admin user also has admin as the default password (although it can be easily changed using imqusermgr update -u admin -p newpassword). With an OpenMQ queue ready to be used the next step is configuring Tomcat resources. But, obviously, we need first to include client libraries in Tomcat classpath. I just copied the jars in the lib directory. At this point I am assuming a Tomcat 6 server is installed in {TOMCAT_DIR} directory.

$ cp {OPENMQ_DIR}/mq/lib/imq.jar {TOMCAT_DIR}/lib
$ cp {OPENMQ_DIR}/mq/lib/jms.jar {TOMCAT_DIR}/lib

And now the hard part comes, cos now I need to define the resources in Tomcat to be populated in the JNDI tree, and this was the part I could not find documentation anywhere. Looking OpenMQ source code (that's the way I wanna Open Source) I discover that OpenMQ has some JNDI ObjectFactory classes (an ObjectFactory is the way Tomcat creates and loads any kind of object in the JNDI tree). All of them are placed in the com.sun.messaging.naming package. After some tries (I will never admit how many) I discovered the factory and the queue have to be defined in the GlobalNamingResources of the server.xml file this way.

  <GlobalNamingResources>
    ...
    <Resource name="jms/sampleFactory" 
              auth="Container" 
              type="com.sun.messaging.QueueConnectionFactory" 
              description="OpenMQ Queue Connection Factory"
              factory="com.sun.messaging.naming.QCFObjectFactory" 
              securityPort="7676"
              parm="--"
              subnet="0"
              host="-s localhost"
              ackTimeout="-t 30000"
              version="1.1" />
    <Resource name="jms/sampleQueue" 
              auth="Container" 
              type="com.sun.messaging.Queue" 
              description="OpenMQ Queue"
              factory="com.sun.messaging.naming.QObjectFactory" 
              destName="sampleQueue"
              version="1.1" />
  </GlobalNamingResources>

I really do not know why the references are so exotic but, as I said before, it seems the reason is the high coupling with former Sun application server (if you see some values are like arguments to start the broker daemon). With this part solved the rest of the configuration is just like ActiveMQ or any other JMS software. We need to add the references in the default context (context.xml file).

<Context>
    ...
    <ResourceLink global="jms/sampleFactory" name="jms/sampleFactory" 
        type="javax.jms.ConnectionFactory"/>
    <ResourceLink global="jms/sampleQueue" name="jms/sampleQueue" 
        type="javax.jms.Queue"/>
</Context>

And finally a little web application is necessary to test the setup. I quickly developed a servlet with a send and receive (consume) method, the complete Netbeans project can be downloaded here. In short the web.xml file has to define again the references.

    <resource-ref>
        <res-ref-name>jms/sampleFactory</res-ref-name>
        <res-type>javax.jms.ConnectionFactory</res-type>
        <res-auth>Container</res-auth>
        <res-sharing-scope>Shareable</res-sharing-scope>
    </resource-ref>	
    <resource-ref>
        <res-ref-name>jms/sampleQueue</res-ref-name>
        <res-type>javax.jms.Queue</res-type>
        <res-auth>Container</res-auth>
        <res-sharing-scope>Shareable</res-sharing-scope>
    </resource-ref>

And here it is the doSend method (nothing special was done).

    private void doSend(String txt) throws ServletException {
        QueueConnection connection = null;
        QueueSession queueSession = null;
        try {
            System.out.println("Getting connection factory from JNDI...");
            QueueConnectionFactory connFactoryObj = (QueueConnectionFactory) 
                initialContext.lookup("java:/comp/env/jms/sampleFactory");
            System.out.println("Creating connection...");
            connection = connFactoryObj.createQueueConnection("admin", "admin");
            queueSession = connection.createQueueSession(true, 
                Session.SESSION_TRANSACTED);
            System.out.println("Getting queue from JNDI...");
            Queue queue = (Queue) initialContext.lookup("java:/comp/env/jms/sampleQueue");
            System.out.println("Creating the session");
            QueueSender queueSender = queueSession.createSender(queue);
            TextMessage message = queueSession.createTextMessage();
            message.setText(txt);
            System.out.println("Sending the message: " + message);
            queueSender.send(message);
            queueSession.commit();
        } catch (Exception e) {
            if (queueSession != null) {
                try {queueSession.rollback();} catch(Exception ex) {}
            }
            e.printStackTrace();
            throw new ServletException(e);
        } finally {
            if (queueSession != null) {
                try {queueSession.close();} catch(Exception e) {}
            }
            if (connection != null) {
                try {connection.close();} catch(Exception e) {}
            }
        }
    }

Finally I decided to extend or improve this solution. In the resource configuration only host and port are specified, this means a lot of the JMS software features are wasted. OpenMQ right now does not recommend the use of imqBrokerHostName or imqBrokerHostPort properties, now it uses a new imqAddressList which defines one or more broker instances (HA setup). So I created a new ObjectFactory called CFReflectionObjectFactory.java which simply lists all the connection configuration properties of OpenMQ (using reflection over the class ConnectionConfiguration) and set the property if it is defined in the Tomcat resource definition. This way any property can be passed to OpenMQ connection. Using this little improvement (I also copied the one class jar library inside the Tomcat lib directory) the Resource for the factory was as follows.

    <Resource name="jms/sampleFactory" 
              auth="Container" 
              type="com.sun.messaging.QueueConnectionFactory" 
              description="OpenMQ Queue Connection Factory"
              factory="com.sun.messaging.naming.CFReflectionObjectFactory" 
              imqAddressList="mq://localhost:7676/jms"
              imqReconnectEnabled="true"
              imqReconnectInterval="60000"
              imqReconnectAttempts="5" />

If you see I have set the host using recommended imqAddressList property and some other reconnection properties just as example (because of reflection any property can be set, any property which exists in the ConnectionConfiguration OpenMQ class of course). With this final setup I present a video where the demo servlet is used to send three text messages to the queue and then they are retrieved.

I hope that I am helping my mate and maybe someone else (possibly it is a bit late, I am sorry). It is incredible there was no information about a Tomcat/OpenMQ setup. In summary there are some ObjectFactories ready to be used in the OpenMQ distribution but I fear that they are a bit old and obsolete, and that is the reason I created a new one which can define any configuration property.

Be smart!

Ricky's Hodgepodge: Sound Issue in Linux Adobe Flash (Better Solution)

nospam@example.com (rickyepoderi) — vie, 03 jun 2011 06:21:00 +0000

Yesterday I spent some time reading the fedora bug about the flash issue commented in the previous entry. There are very interesting comments in it, but more important, there are two attachments to replace by yourself the memcpy calls with memmove (one is a bash script and the other is a C++ program).

I tried the C++ source code program and it seems to work quite well:

# wget -O replace.c https://bugzilla.redhat.com/attachment.cgi?id=487982
# g++ -o replace replace.c
# cd /usr/lib64/flashplugin-nonfree/
# cp libflashplayer.so libflashplayer.so.ORIG
# ~/replace libflashplayer.so
Found memmove at symbol table index 516
Found and fixed 1 reloc entries

Of course this solution is a much better approach, because it depends directly on you, you can understand what you are doing and it will work for future plugin releases (if they are still broken). But remember I was desperate at those times. Another thing I did not comment before is that this bug only applies to amd64 linux platforms (not i386, 32 bit installations). And paying more attention to glib package upgrade (I also upgraded my desktop debian box yesterday), I realized there is a warning screen about this memcpy issue you need to accept. I really do not know if I just accepted it without reading in the laptop upgrade or it is new now... I bet for the first option.

Cheerio!

9€®$!µ$: Por qué me gusta tanto el VGA Planets

nospam@example.com (gersius) — jue, 26 may 2011 13:45:23 +0000

Realmente no es tanto el juego como los jugadores. Llevo algunos años jugando con una panda de españoles y argentinos, y las partidas son divertidísimas. Para muestra, el siguiente mensaje en la lista de correo de la partida:

Manga de cagones trolos chupatrozos, en el turno 10, cunpliendo la consigna, yo ataque a Peron, de acuerdo a lo que todos decian. Le tome la unica puta nave que vi en el horizonte 2D de nuestro universo... y el resto? nada... manga de travestidos disfrazados de moyano... ya no les creo nada... ahora en el turno 20, Peron puede transitar libremente entre mis planetas (no en ellos) y fuera de mis campos minados, joder!

Y las hostilidades aún no han comenzado... pronto empezarán las explosiones

Ricky's Hodgepodge: Sound Issue in Linux Adobe Flash

nospam@example.com (rickyepoderi) — mar, 24 may 2011 20:31:45 +0000

Another quick entry this time. Some months ago I started to watch dexter via streaming, but the past week I noticed the sound was very garbled and strange. My streaming site uses (in my linux user opinion) the annoying adobe flash plugin. Being very busy these days I just found the debian associated bug but, at that moment, it solved nothing. Today I have rechecked it and a user comment links to another adobe bug. It explains that the plugin is using memcpy instead of memmove. And if man pages are consulted it is clear what is going on:

The memcpy() function copies n bytes from memory area src to memory area dest. The memory areas should not overlap. Use memmove(3) if the memory areas do overlap.

Last glibc upgrade has got this sound issue out (you know I have a thing about upgrading my debian box). If you follow the previous adobe link there is a solution from a Russian page. It seems the memcpy function has been binary replaced by memmove (I did not know it was even possible). In my debian box the solution is very similar to the explained by the Russian mate. Please be aware of the fact that you are trusting in a binary change done by this guy (you know, desperate times call for desperate measures).

# wget http://catap.ru/patches/flash64/memcpy-10.3.162.29.bsdiff
# cp /usr/lib64/flashplugin-nonfree/libflashplayer.so \
  /usr/lib64/flashplugin-nonfree/libflashplayer.so.old
# bspatch /usr/lib64/flashplugin-nonfree/libflashplayer.so.old \
  /usr/lib64/flashplugin-nonfree/libflashplayer.so \
  memcpy-10.3.162.29.bsdiff

Adobe flash plugin is a pain in the ass of any linux user. So please I encourage everybody to change to new HTML5 features, especially pages like grooveshark, megavideo or youtube (google is already doing that) which are part of my life.

Thank you!

Ricky's Hodgepodge: JAX-WS and Multiple Endpoints

nospam@example.com (rickyepoderi) — s�b, 30 abr 2011 10:49:00 +0000

These last weeks I have been working in an identity project that deals with Web Services. The customer wants the Identity Solution to integrate several applications via Web Services. Therefore the agreement of a WSDL between all parts was the first issue to work with. The WSDL (Web Services Description Language) is an XML language to describe and model Web Services, using simpler words, it is a way to define what methods to implement and what the data model exchanged is (without any doubt). Finishing the introduction JAX-WS (Java Api for XML Web Services) is the API that deals with Web Services as part of the Java EE platform specification (it gives annotations, WSDL parsing and many other features to simplify the development of Web Services in the Java platform).

It is important to explain that in this case my part (the identity side) would be the client whilst the different applications would be the Web Services server side. This way the Identity Software would invoke the methods of the agreement in order to provision a new account, to change some attributes or to delete a user in the integrated application. If a WSDL is defined previously no compatibility problems are expected, no matter the server software or the implementation language. Besides, as all methods and exchange objects had been deliberately defined generic, only one client would have been needed in the identity side for all applications. In fact the agreement document version 1.0 (with the first version of the WSDL inside too) was sent to customer and to the different application teams some months ago.

But people sometimes do not understand what a WSDL is and, even worse, developers think some methods with similar functionality are more than sufficient (which is basically true but it converts all the previous work in a waste of time and the scheduled timetable in a real bullshit). Of course these little modifications are considered not important enough to even advise the other parts involved. To sum the first application to integrate had developed their way Web Services. Besides the application team used a custom and abominable Web Services framework which did not generate the artifacts, so the XML data had to be manually parsed (oh my gosh!). Another of its features was every operation was bound to a different endpoint (a different URL). Needless to say the WSDL was not automatically generated and it had also to be written manually. So you can image the situation. After some crisis times I decided to tune their WSDL and implement the same file in a modern application server, mainly my idea was testing with a confident implementation and reporting errors to them as fast as I can.

But developing a multi-endpoint web service with JAX-WS was not as easy as I expected and this is the reason for this entry. I decided to use Metro implementation which is the JAX-WS reference and open-source implementation done by Oracle (formerly Sun). Actually metro is the complete Web Services stack for glassfish (and other application servers) and covers many other components like JAX-RS (see my ScatterPlot entry for more information for RESTful Web Services), JAXB, WSIT or SAAJ inside the Java EE. There are more implementations like Apache Axis 2, Apache CXF or Codehouse XFire but I have more experience with this one and I suppose basics are the same for all of them. I spent some time to create the Web Service and I want to save this information here.

For obvious reasons I cannot present the real WSDL file so I created a very simple user CRUD service for a supposed application, here it is the basic-crud.wsdl file. This WSDL is not generic but it is easier and better for learning purposes. The definition has four endpoints with one operation inside each (Create, Read, Update and Delete). If you download the JAX-WS Metro bundle there are some sample applications inside, and one of them explicitly shows how to create a jax-ws service from a given WSDL file (fromwsdl inside samples directory). Following the instructions the wsdl must be imported in order to create the interfaces and exchange objects:

$ wsimport -keep -Xnocompile -verbose -d ../../../src/java/ basic-crud.wsdl 
parsing WSDL...

generating code...

sample/user/crud/basic/CreatePort.java
sample/user/crud/basic/CreateRequest.java
sample/user/crud/basic/CreateResponse.java
sample/user/crud/basic/DeletePort.java
sample/user/crud/basic/DeleteRequest.java
sample/user/crud/basic/DeleteResponse.java
sample/user/crud/basic/ObjectFactory.java
sample/user/crud/basic/ReadPort.java
sample/user/crud/basic/ReadRequest.java
sample/user/crud/basic/ReadResponse.java
sample/user/crud/basic/SampleUser.java
sample/user/crud/basic/Status.java
sample/user/crud/basic/StatusCode.java
sample/user/crud/basic/UpdatePort.java
sample/user/crud/basic/UpdateRequest.java
sample/user/crud/basic/UpdateResponse.java
sample/user/crud/basic/WsCRUDUserService.java
sample/user/crud/basic/package-info.java

The wsimport command parses the WSDL file and generates the Java portable artifacts (keep option does not delete the Java files and they are saved inside src directory). With this command all Java classes and the Web Service Interfaces are generated for all the ports. After that the different ports need to be coded and, following the fromwsdl example, the @WebService annotation must only have the endpointInterface property set in each. In my example all the ports just manage the users into memory (a memory example repository). The code for read operation is presented below.

package sample.user.crud.basic.implementation;

import javax.jws.WebService;
import sample.user.crud.basic.ReadRequest;
import sample.user.crud.basic.ReadResponse;
import sample.user.crud.basic.SampleUser;
import sample.user.crud.basic.Status;
import sample.user.crud.basic.StatusCode;

@WebService(endpointInterface = "sample.user.crud.basic.ReadPort")
public class ReadPort {

    private UserMapSingleton map = UserMapSingleton.getSingleton();

    public ReadResponse read(ReadRequest request) {
        ReadResponse response = new ReadResponse();
        Status status = new Status();
        response.setStatus(status);
        try {
            if (request.getId() == null) {
                status.setStatus(StatusCode.ERROR);
                status.setMessage("No ID passed!");
            } else {
                SampleUser user = map.getMap().get(request.getId());
                response.setUser(user);
                status.setStatus(StatusCode.OK);
                if (user == null) {
                    status.setMessage("User '" + request.getId() + "' not found!");
                }
            }
            return response;
        } catch (Throwable t) {
            status.setStatus(StatusCode.ERROR);
            status.setMessage(t.getMessage());
            return response;
        }
    }
}

When the Web Services are created from a defined WSDL all the ports have to be added to the sun-jaxws.xml file manually (as I said previously only the endpointInterface attribute annotation is present in the Java class and all the configuration is defined via the sun-jaxws file). This file contains all the endpoints with the implementation and interface part. You have to think this is exactly the opposite of what is done when the WSDL is not previously defined (everything is annotated and nothing is set in the sun-jaxws.xml). Here it is the file for the four CRUD endpoints.

<endpoints 
    xmlns="http://java.sun.com/xml/ns/jax-ws/ri/runtime"
    version="2.0">

    <endpoint
        name="wsCRUDUserServicePortTypeBindingCreate"
        implementation="sample.user.crud.basic.implementation.CreatePort"
        interface="sample.user.crud.basic.CreatePort"
        wsdl="WEB-INF/wsdl/basic-crud.wsdl"
        service="{http://basic.crud.user.sample}wsCRUDUserService"
        port="{http://basic.crud.user.sample}wsCRUDUserServicePortTypeBindingCreate"
        url-pattern="/wsCRUDUserServicePortTypeBindingCreate" />

    <endpoint
        name="wsCRUDUserServicePortTypeBindingRead"
        implementation="sample.user.crud.basic.implementation.ReadPort"
        interface="sample.user.crud.basic.ReadPort"
        wsdl="WEB-INF/wsdl/basic-crud.wsdl"
        service="{http://basic.crud.user.sample}wsCRUDUserService"
        port="{http://basic.crud.user.sample}wsCRUDUserServicePortTypeBindingRead"
        url-pattern="/wsCRUDUserServicePortTypeBindingRead" />

    <endpoint 
        name="wsCRUDUserServicePortTypeBindingUpdate"
        implementation="sample.user.crud.basic.implementation.UpdatePort"
        interface="sample.user.crud.basic.UpdatePort"
        wsdl="WEB-INF/wsdl/basic-crud.wsdl"
        service="{http://basic.crud.user.sample}wsCRUDUserService"
        port="{http://basic.crud.user.sample}wsCRUDUserServicePortTypeBindingUpdate"
        url-pattern="/wsCRUDUserServicePortTypeBindingUpdate" />

    <endpoint 
        name="wsCRUDUserServicePortTypeBindingDelete"
        implementation="sample.user.crud.basic.implementation.DeletePort"
        interface="sample.user.crud.basic.DeletePort"
        wsdl="WEB-INF/wsdl/basic-crud.wsdl"
        service="{http://basic.crud.user.sample}wsCRUDUserService"
        port="{http://basic.crud.user.sample}wsCRUDUserServicePortTypeBindingDelete"
        url-pattern="/wsCRUDUserServicePortTypeBindingDelete" />

</endpoints>

Finally the web.xml is modified to add JAX-WS listener and servlet and the servlet is mapped against all the URLs specified in the previous file.

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee 
  http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">
    <listener>
        <listener-class>
            com.sun.xml.ws.transport.http.servlet.WSServletContextListener
        </listener-class>
    </listener>
    <servlet>
        <display-name>wsCRUDUserService</display-name>
        <servlet-name>wsCRUDUserService</servlet-name>
        <servlet-class>com.sun.xml.ws.transport.http.servlet.WSServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <servlet-mapping>
        <servlet-name>wsCRUDUserService</servlet-name>
        <url-pattern>/wsCRUDUserServicePortTypeBindingCreate</url-pattern>
        <url-pattern>/wsCRUDUserServicePortTypeBindingRead</url-pattern>
        <url-pattern>/wsCRUDUserServicePortTypeBindingUpdate</url-pattern>
        <url-pattern>/wsCRUDUserServicePortTypeBindingDelete</url-pattern>
    </servlet-mapping>
    <session-config>
        <session-timeout>30</session-timeoutv
    </session-config>
    <welcome-file-list>
        <welcome-file>index.jsp</welcome-file>
    </welcome-file-list>
</web-app>

In theory at this point everything is set and the project is ready to be used. I deployed the war file inside a standard glassfish 2.1.1 application server. If you access to any of the four mapped endpoints Metro lists all the endpoints defined in the application (see screenshot below). If the WSDL is requested to any of the four endpoints the same XML file is returned (cos four endpoints are coded from the same WSDL file).

Finally I implemented a quick JUnit test case class which checks the four endpoints work as expected (it creates, updates and deletes a user doing some reads between modify operations to test the functionality).

Today's entry explains how to create a JAX-WS Web Service with several endpoints from a pre-defined WSDL definition file using reference Metro implementation. As you have stated it is not very difficult and you can directly follow fromwsdl sample instructions. But I wanted to summarize the process here to have a quick guide for the next time (I do not usually work with Web Services this way and I spent some time to achieve it). In my defense, it seems incredibly simpler once you have written the full instructions, these things always make me think I am really stupid . The complete NetBeans project can be download from here.

God is in the detail. Cheerio!

Ricky's Hodgepodge: Glassfish Enterprise Profile on Linux amd64

nospam@example.com (rickyepoderi) — dom, 10 abr 2011 14:01:00 +0000

The previous week an ex co-worker, and a good friend, asked me if I had tested Glassfish 2.1.1 on a x64 linux box. The question surprised me cos my new laptop is an amd64 Linux installation, all my glassfish tests are done in it and I had not noticed any problem. He clarified me that he could not setup an enterprise profile. I remember that Glassfish 2.1.1 has no difference between open source and supported edition (it was based on the same source code), so this entry is about how to setup a Glassfish enterprise profile in Linux x64 using its community version.

Glassfish v2 can be installed using three different profiles: developer, cluster or enterprise. Developer profile is a single application server instance. Cluster is a multi-instance setup with central management for all servers and cluster/Load Balancing features enabled. Enterprise profile is exactly like cluster but using NSS (Netscape Security Services) instead of JKS (Java KeyStore) for the certificate store and the possibility of using HADB for session management (a kind of multinode database that Sun used to integrate with Glassfish to get the third type of session-aware cluster I talked about in a previous post). My friend explained to me that the problem was with NSS libraries (supported Glassfish edition includes them), the libraries provided by the Linux distribution were ELF 32-bit and, obviously, they do not work with a 64 bit JVM and system.

I am going to explain the steps I followed testing this issue. This is usually how I like to show my entries, because I think this way is more useful and it is also better for me to remember.

As the Glassfish documentation comments, in open source edition NSS and NSPR libraries have to be installed independently from glassfish. In my debian box this step was easily done.
```
$ apt-get install libnss3-1d libnss3-tools libnspr4-0d libnspr4-dev libnss3-dev
```
Developer packages were not needed at this time but following points are going to use them.
Glassfish installer for open source edition was downloaded from glassfish site. For linux only one package is available. Cluster setup was installed using glassfish instructions.
```
$ java -Xmx256m -jar glassfish-installer-v2.1.1-b31g-linux.jar
$ cd glassfish
$ lib/ant/bin/ant -f setup-cluster.xml
```
Glassfish community edition is installed in the directory where the installer is launched, inside a new glassfish folder (this installation directory will be denoted ${GLASSFISH_DIR} since now). By default the previous process creates a domain (domain1) using cluster profile (not enterprise). So this domain was deleted.
```
$ cd ${GLASSFISH_DIR}/bin
$ ./asadmin delete-domain domain1
```
The configuration file ${GLASSFISH_DIR}/config/asenv.conf was modified to include where NSS are installed (debian installs main NSS and NSPR libs in /usr/lib). So the following properties were modified:
```
AS_NSS="/usr/lib"
AS_NSS_BIN="/usr/bin"
```
If you try to create an enterprise profile domain just like this the issue explained by my colleague happens. It seems a internal libasnss.so is also used and glassfish only provides it in 32 bit mode (as you see in the download page there is no Linux x64 installer). But this is open source so, not wasting any time, I took a look to glassfish 2.1.1 subversion and I found a nssstore.c file. This is a Java Native Interface (JNI) wrapper above NSS library (Java can call native libraries using JNI and this file is a wrapper to make the interaction easier). Although there are some links for completely building glassfish (one for glassfish v3 and another outdated one for v2), I quickly gave up because the process seems to be huge. So I decided to only compile the problematic file.

First the directory that contains this file was checked out:
```
$ cd /tmp
$ svn checkout https://svn.java.net/svn/glassfish~v2/trunk/appserv-native-ee
$ cd appserv-native-ee/src/cpp/nssutil/
```
After that the header file has to be made. In JNI this is done reading the class and executing javah command on it (the problematic class, source of the exception, was com.sun.enterprise.ee.security.NssStore).
```
$ javah -classpath ${GLASSFISH_DIR}/lib/appserv-se.jar \
  -o com_sun_enterprise_ee_security_NssStore.h com.sun.enterprise.ee.security.NssStore
```
With this header the C file can be compiled against NSS/NSPR debian 64 bits libraries:
```
$ gcc -fpic -I/usr/lib/jvm/java-6-sun-1.6.0.24/include/ \
  -I/usr/lib/jvm/java-6-sun-1.6.0.24/include/linux/ \
  -I/usr/include/nss/ -I/usr/include/nspr/ nssstore.c -c
$ gcc -shared -Wl,-rpath,${GLASSFISH_DIR}/lib -o libasnss.so nssstore.o -lc \
  -L/usr/lib -lnss3 -lnspr4 -lnssutil3 -lsmime3
```
The new libasnss.so library is copied to glassfish lib directory (a backup is previously done).
```
$ cp ${GLASSFISH_DIR}/lib/libasnss.so ${GLASSFISH_DIR}/lib/libasnss.so.ORIG
$ cp libasnss.so ${GLASSFISH_DIR}/lib/libasnss.so
```
After the previous step the domain is successfully created but it fails to start. It complains about library /usr/lib/amd64/libsoftokn3.so does not exist. Looking again the code of the class EESecuritySupportImpl.java (thrower of the new exception) it tries to initialize the PKCS11 provider (method initNSS) with the libsoftokn3 NSS library compounding its path as follows: the AS_NSS path from asenv.conf (in our case /usr/lib), the architecture if system is 64 bits (amd64 in our case) and finally libsoftokn3.so if system is not windows. So the result is /usr/lib/amd64/libsoftokn3.so but, in case of debian, this library is located in /usr/lib/nss.

I did not think too much and, after becoming root, I created a link.
```
# cd /usr/lib
# ln -s nss amd64
```

Finally the enterprise domain was created and started successfully.

$ cd ${GLASSFISH_DIR}/bin
$ ./asadmin create-domain --profile enterprise --user admin --adminport 9898 \
  --savelogin=true --savemasterpassword=true domain1
$ ./asadmin start-domain

Here it is a video where the new domain is created and started (as you see my architecture is amd64, the certificate store is NSS and glassfish works perfect).

As a conclusion the glassfish installer for Linux is only provided in 32 bits. It works in x64 Linux system using developer and cluster profile cos a complete Java stack is used (no JNI library is needed) but it fails in enterprise. The enterprise profile uses NSS as the certificate store and a little libasnss.so (JNI library) is provided for easier integration, but it is only in 32 bit. In this entry the library was re-compiled in my amd64 box. Of course another problems could come out cos I did not test all the features and maybe more libraries are used (there are more native libraries in the lib glassfish directory and all of them are 32 bit).

That's the Way I Wanna Rock 'n' Roll!

Ricky's Hodgepodge: High Availability in Application Servers (sequel)

nospam@example.com (rickyepoderi) — mar, 05 abr 2011 06:40:00 +0000

In the previous entry a study about High Availability (HA) in Application Server was done. The main reason for this study was checking how proper an in-memory repository (like membase) is for session management. I found memcached-session-manager (MSM), a session manager for tomcat and memcached, and, for that, I performed a benchmark of the three typical HA solutions using tomcat. In the comments section Martin Grotzke (the main developer of MSM) pointed out that, if mod_jk had been configured sticky, the proper behavior of MSM was also to be sticky. I want to explain that my understanding of the third scenario (a external repository is used to save sessions) consists in that the new backend (JDBC, in-memory or whatever) has to be a complete replacement for session management. This way it should be accesseded twice in every request (first to get the session and then, if session is modified, to save changes). This is the reason I configured MSM non-sticky, but obviously this is the most demanding configuration and quite weird (sticky mod_jk but non-sticky MSM is something clearly useless in a real deployment). Finally I have included the complete sticky benchmark to compare all results fairly.

The MSM project page explains the differences between sticky and non-sticky configuration. When tomcat servers are accessed in a sticky way MSM can assume that, if it has an active session, the session is up to date (cos no other server can modify it). Therefore sticky MSM configuration saves all gets (except in the first request, when there is no active session it has to check if it was previously created in a failed server). Besides MSM seems to use only one object in sticky configuration and two in non-sticky (session itself is obviously always managed but a kind of validity object is also managed in non-sticky implementation), so this fact saves some gets and sets too. In summary, there is a lot less work to do in sticky than in non-sticky configuration. Do not forget a get/set also represents a serialization/de-serialization process.

The MSM/membase sticky solution presents the following numbers. I also show the same charts and graphs of the previous entry:

Average and 90% Line time (four configurations).
New graphic results from JMeter for sticky scenario.
Membase statistics page for this new benchmark.
CPU utilization in both virtual boxes (all scenarios).

Average response time and 90% Line time

JMeter graph results for sticky membase scenario

Membase statistics for default bucket

CPU used in debian box 1

CPU used in debian box 2

If you see the numbers of sticky configuration (8ms average request time and 14ms for 90% line time) are very similar to the replication cluster, and half a way between simple load balancing and SMS non-sticky. It is quite logical, more or less, half of the work is not necessary. Looking at CPU percentage the new sticky architecture is around 16% (tomcat process consumes a bit less than in replication scenario but membase adds another 4%). So, again, everything is as expected and it is clear that when this solution was mature enough it would be very very competitive with replication cluster but more scalable. All the options have been benchmarcked and studied, as I said before, now it is fairer.

Cheerio!

Ricky's Hodgepodge: High Availability in Application Servers

nospam@example.com (rickyepoderi) — s�b, 02 abr 2011 10:33:00 +0000

Today entry, despite it is not a typical topic for me, is about an architectural decision: High Availability (HA) in Application Servers. Actually this entry comes out because I have been seeing these days several solutions which involve a in-memory repository in order to get a clustered session-aware Application Server infrastructure (for example this glassfish + coherence presentation). So I decided to study this new solution more deeply. The entry is divided in three different sections: presentation of some HA theory in Application Server world, preparation of a demo and some benchmark results.

CONCEPTS

As always some concepts are needed before the entry goes into the substance. Mainly Application Servers have three different clustered solutions.

Load Balancing + Stickyness

The first and more easy clustered solution is just a setup with two or more applications servers which receive clients in a sticky way. When I say stickyness I refer to the fact that when a user request has first sent towards a specified server (and the java session has been created in this server) the load balancing element always sends his future requests to the same server. This way the session is maintained in only one server but everything works as expected.

In this scenario there is no special session treatment and, therefore, if one application server dies all its sessions (clients) are lost. The balancer will detect the server has fallen and all new requests will be redirected against another server but, as sessions are stored individually, the user will have to re-login and previous work (in a shopping cart application for instance) could be lost.

As you see this solution is very very simple. There are lots of load balancers solutions (software or hardware) and almost all of them support stickyness (using jsessionid tracking cookie) and server status checks. Stickyness can also be used in the other two solutions and, in fact, it is usually recommended.

Load Balancing + Session Replication

The second solution tries to solve the session lost problem of the previous scenario. Many application servers (all famous ones at least) implement a in-memory session replication. This way when a request modifies any attribute in the session these changes are sent to the rest of servers and, consequently, session is always up to date in all of them.

This solution solves the main problem but it is not absolutely safe. Some problems are the following: all sessions are in all serves (this is the main one, think about a ten server cluster with thousands and thousands of sessions), replication is not immediately done, performance drops when a lot of sessions are involved or they are very big and, if all servers die unexpectedly, they are also lost.

Load Balancing + HA Storage

The final solution is to use another persistent element to store sessions (Application Server can save sessions only in the repository or in both sites, its own heap and in the new repository). This solution has two main problems, the High Availability feature is moved from Application Servers to the external repository and the performance penalty of storing and retrieving any session may be major.

DEMO

The third solution presented before used to be not very common, mainly because the repository for storing sessions traditionally was a database, which represents a severe performance impact and a clustered database was also needed (you know, the box was worth more than its contents). So usually AppServer HA solutions were reduced to the first (when session lost was not a decisive penalty) or the second scenario (when a session-aware cluster was really needed). Nevertheless, after memcached, the in-memory backends are getting more and more popular and this kind of external repositories can solve, in theory, all the typical problems of JDBC stores. When I heard about this idea (as I commented in the beginning of the entry some commercial Application Servers are starting to offer this solution) I checked if there was any open source implementation for doing the same and I found the memcached-session-manager or MSM project. MSM honors its name and it is exactly that, a session manager for tomcat (6 and 7) and memcached. So the rest of this chapter I am going to explain how to setup the previous three scenarios using tomcat.

Obviously a distributable application is also needed for testing. For that I am going to merge two Web Modules (WAR files). On the one hand, the typical glassfish example clusterjsp application, some JSPs which are very useful to check if Load Balancing and Clustering is working. On the other, the ServletBenchmarck, a quite well thought application to bechmark Servlet engines. I have added the JSPs of clusterjsp inside ServletBenchmark war file so I can test individually and benchmark at the same time (here it is the war file I deployed inside tomcats).

Tomcat + Apache2/mod_jk

For the first scenario typical mod_jk balancing is going to be used. The mod_jk is a module for Apache2 that interacts with tomcat servers using AJP 1.3 protocol. It supports load balancing, server checks and stickyness. I installed two debian Squeeze boxes using KVM virtualization. Then I setup two apaches and two tomcats with stickyness enabled, debian packages have been used for everything (there are a lot of installation guides over the internet, for example this one). Here it is my workers.properties if you want to check it out.

With this configuration the following video is presented. When I access apache the mod_jk module redirects my request against one of the tomcats. Using clusterjsp page some attributes are added to the session, as configuration is sticky, same tomcat is always accessed. To test Load Balancing I kill the server which is being used. As you see the other tomcat takes the control but session is empty as expected.

Tomcat + Apache2/mod_jk + Session Replication

The second scenario can be easily implemented because of cluster tomcat feature. This configuration makes each tomcat Server to broadcasts topology and session changes to other servers in the cluster (it seems tomcat uses multicast for spreading changes). In order to configure tomcat inside a cluster only some minor changes in server.xml are needed.

With this setup now the video is better. After I kill the first accessed tomcat and mod_jk redirects my request to the other one, my session is still alive and all the attributes are still displayed. Replication is doing the job!

Tomcat + Apache2/mod_jk + memcached-session-manager

This is a more interesting and newer scenario. For this configuration I decided not to use memcached (like MSM expects) but membase. Membase is a next generation memcached with several improvements (mainly because it is not a cache, it is a real NoSQL database which stores data in disk, has replication features and manages topology changes) but keeps all goodness (simplicity, elasticity, very fast,...).

The main reason for this change is how I understand this solution. For me the external repository must handle HA independently and memcached is just a cache system, it does not cover replication or even a server fault (for this reason MSM stores a second backup session when more than one memcached is configured). But membase uses the idea of buckets, which are isolated virtual containers for data and they are currently of two types: memcached (same old characteristics) or membase type (new features of persistence, replication and rebalancing). Actually membase does not perform an automatic rebalancing, if a server fails an administrator must rebalance the cluster manually via the web console or via script (semi-automatic rebalancing). In my opinion for session storage the persistence part (saving data in disk) is not necessary, although sessions can survive through membase reboots I think the cost is greater than the benefits. So if membase had a bucket with only replication and rebalancing I think it would be the best configuration for this goal. Think about that, a membase without disk storage is the perfect extension of the second scenario, session is replicated only the configured times (in a membase cluster you can set how many copies are saved) and space is not wasted in all tomcats. Of course there are some disadvantages like tomcat accesses to membase every request (but membase is supposed to be very fast) and partial replication is not possible (membase is a general purpose repository so it does not know anything about sessions or attributes). But, in general, I think a in-memory replicated repository could be the best solution for an architecture with a lot of servers, a lot of sessions or very big ones.

As I said MSM is thought to be configured against, at least, two memcached servers (one for main session storage and the second as a backup if the first one fails) but I will only configure one (membase replicates information instead of MSM). Clients for membase can be of two types (see membase types of clients) and MSM uses the spymemcached library which is of type 1 (typical memcached client which is not aware of cluster topology). For this situations membase deploys a gateway process (called moxi) which is the responsible of redirect the requests to the correct membase server (for more information about moxi gateway see membase documentation). So finally I setup a two node membase cluster (one in each debian box), with default membase type bucket (one extra copy configured) and each tomcat instance accesses the server-side moxi. The membase software was installed using community 64bit debian package downloaded from couchbase site. MSM was configured following its beautiful and simple guide, I used non-sticky configuration (although I use sticky mod_jk workers I wanted that sessions were always retrieved and stored) and kryo recommended serialization.

As usual my setup fell into a MSM bug, a lot of objects remained in membase when sessions were invalidated (they are not removed). The problem seemed to be that the code always tried to delete the backup session object although it had not been created previously (cos only one memcached is configured), it thrown an exception and other object was not removed (MSM stores two objects per session). I fixed the problem and, as usual, I try to be a friendly neighbor and I sent an email to MSM main developer explaining my case (Martin Grotzke kindly opened an issue for me).

After the bug was fixed a membase session storage system was correctly working. Now the sessions are always retrieved and then saved from/to membase on every client request to tomcat. So now in the video when the server is killed the other one successfully retrieves the session from membase and the attributes are still there. But even better, if I kill both Tomcats and I start the first one killed my session is still active (cos membase stores it outside tomcat). In the membase statistics page for default bucket the new session objects has been created.

I also wanted to test a JDBC solution (in order to have a comparison with previous SQL backends) and I tried to configure tomcat JDBC Store and the Persistence Manager with MySQL (I tried to follow this guide). But it did not work for me. Although sessions survived through tomcat restarts they were almost never persisted when tomcat was killed. It seemed like sessions were not always stored inside the table and they were just inserted based on a time scheduling. In short I was not able to make sessions to be inserted immediately in the database (if someone is smarter than me please comments will be appreciated). And I have to admit this point is a real pity for the next point.

BENCHMARK

Finally I set up a benchmark to test the three scenarios. As I commented the ServletBenchmark is a smart application that simulates a common web navigation (check the application link previously presented for details). In this case it was specially interesting cos this benchmark manages session with three different types of users:

Short size (10 bytes) and short lived session (one request).
Medium size (up to 150 bytes) and medium lived session (five requests).
Long size (up to 2100 bytes) and long lived session (20 requests).

I prepared this JMeter test plan which launches 60 concurrent threads looping over the three ServletBenchmark cases for half an hour. In each loop the thread has 20% of probabilities to execute the short scenario, 30% the medium and 50% the long one. When the scenario is completed the thread sleeps for 1 second and then invalidates the session. Architecture is also very important for understanding any benchmark. And, as I explained before, my demo site is a crap: all inside my laptop; two KVM virtualized Debian Squeeze linux (apache2, tomcat and membase installed in each); the JMeter benchmark tool in my real Debian laptop pointing to the first apache. So it is clear numbers are not very important, just the difference between them.

You can download the complete results for the three scenarios (Load balancing, replication cluster and membase solution). And here they are the numbers via charts and graphics:

Average response time and 90% Line time (response time in which the 90% of requests were completed) for the three solutions.
JMeter graph results for the three scenarios.
Membase statistics (screenshot of the default bucket statistics).
CPU consumed (top used) by involved processes (in the third scenario tomcat and membase percents have been added). Two graphs, one for each debian box.

Average response time and 90% Line time

JMeter graph results for Load Balancing scenario

JMeter graph results for Replication Cluster scenario

JMeter graph results for Membase scenario

Membase statistics for default bucket

CPU used in debian box 1

CPU used in debian box 2

For me the results are exactly what I had in mind before the benchmark. First and important, there is no error in any of the tests, so the three solutions are very reliable. A thing that surprise me is that numbers are quite low (6, 7 and 11 ms for average response time respectively). This means KVM works really well and, maybe, as everything is deployed in the same physical box, communication latencies are nearly zero. If we look at the response times things work as expected. Each solution gives more availability than the previous one with a cost in performance. But this cost seems reasonable, only 1 ms in replication over simple load balancing and 4 ms in membase solution over replication cluster. And in the 90% line, distances remain the same (which means three solutions work quickly most of the times). When checking the CPU used, load balancing uses around 5-7% of the CPU time, replication over 12-14% and membase (tomcat + membase daemons) over 20-23%. They are also expected except that membase is a little more of what I thought, tomcat process itself is a bit more CPU hungry than in replicated scenario (cost of serialize the whole session) and membase processes (moxi, memcached and beam) add around 6-7% more.

Final comments, the benchmark has not stressed the system enough (I think I would have needed more threads, lower sleeping times or a more intense application but I also think that this is more than enough for what I wanted to show). The membase scenario is a really good competitor, cos a 4-5ms penalty is almost nothing in my opinion (think in a real scenario where requests do some real processing and can last for 100ms or more). And remember these numbers can be even lower if membase do not save sessions in disk. It was a real pity that the numbers of a real JDBC solution cannot be compared (seeing what is the real distance between new membase NoSQL solution and a traditional MySQL/PostgreSQL one would have been great). Of course this membase solution is not in a production ready state (there are some snags, mainly with membase HA) but the goal of this entry was testing if this kind of in-memory softwares are convenient as session stores (numbers have been presented) and, at the same time, checking some real open-source scenario.

I have invest some time in this benchmark but it was a thing I really wanted to do. I hope it was worthy for someone else. See you next time!

Ricky's Hodgepodge: Friday Night Fever (or How a Computer Guy Spends the Friday Night)

nospam@example.com (rickyepoderi) — s�b, 19 mar 2011 14:59:57 +0000

For some strange reasons this week I needed to use my Spanish eID (DNIe) again and, sometimes these things happen, firefox hanged like crazy. Finally I had to reboot my system to use the platform pre-installed windows to complete the formality. Yesterday night I decided to find out what the hell was going on here.

As always I put opensc in debug mode (setting debug to 10 in /etc/opensc/opensc.conf) and what I saw was really amazing. Starting firefox (iceweasel) inside a terminal the debug dumped like crazy but it sometimes hanged for exactly 60 seconds. So it was not totally frozen, it worked quite well but sometimes it was waiting for something one minute. The problem is opensc calls a lot to the card and firefox hangs at least two or three times every startup (which means more than two or three minutes waiting firefox to come up).

First thing to always check in these situations is what has been changed (I have a thing about upgrading my Debian laptop). Since the time I wrote the DNIe entry the opensc packages (the project that deals with PKCS#11 and PKCS#15) remain the same, PC/SC lite (the project for communicating with smartcards and readers) has been upgraded from 1.5.5 to 1.6.6 and libccid (the handler library for my reader) from 1.3.11 to 1.4.2. I decided to come back to original packages (uninstalling and installing the old ones) to see what happened and, voilà, it started to work as before, no watings.

At that moment I was not totally happy cos I do not feel comfortable when I do not know what really happens and, even less, holding packages. So I started compiling by myself the complete software stack, first PC/SC lite 1.7.0, then libccid 1.4.2 and then opensc 0.12.1. But with opensc I changed my mind and I compiled it twice. First the old source from DNIe official page (opensc-dnie), the external library I talked about in the old entry, and opendnie (see this mail and this spanish blog), the effort of a Spanish guy to re-implement all the stuff to include Spanish eID into opensc without legal issues (maybe this work deserves a new entry). I had been thinking about opendnie for some time but I had never had the the time to test it. The old source code for opensc-dnie now does not compile, the problem is libassuan (the little window the library shows when you are about to sign something) has changed its api. But checking opensc patch the modification of dialog.c was easy (here it is the patch if you want to use it). Opendnie compiled with no issues.

Testing and debugging I realized the problem was not in any of these packages but in libusb (the spurious hang also happened in my compiled versions and it did not matter you use opendnie or opensc-dnie). Now libccid 1.4.x is linked against libusb-1.0 (thing that did not happen with 1.3.x) and this is the reason the bug came out after the upgrade. So I also compiled libusb 1.0.8 and I saw the problem was inside libusb_control_transfer function. This function sends the transfer to the usb device and then waits for the response with a 60 seconds timeout. Finally I changed the fixed 60 seconds and tried again. And it worked, both solutions (opendnie and opensc-dnie) hanged no more. My patch was very crappy and straight forward but I opened a bug against libusb. At that time it was around 2 am at night and I had not even dinner. You know... it was a computering Friday night fever!

Today in the morning Ludovic Rousseau (the PC/SC and CCID guy) answered me saying this bug was known. He opened a ticket six months ago with a resolution patch inside but it is not closed yet (and Ludovic seems not very happy with that). I sent another post saying my issue can therefore be closed and then I tested Ludovic's patch. The patched libusb also worked correctly so I sent a final comment to Ludovic's ticket saying his patch fixed my issue (at this moment both posts are not displayed cos libusb moderates the comments). And that is all. I do not know how other jobs are but we are masochists and (worst of all) we enjoy it.

See you next time!

Ricky's Hodgepodge: RGraph Scatter Plot using WebSockets

nospam@example.com (rickyepoderi) — s�b, 19 feb 2011 11:54:00 +0000

Today entry is the last one (at east for the moment) about the Scatter Graph problem. If you remember this series started as a HTML5 migration study of an old applet scatter graph application, the first post tried to implement a XY client-side graph using Canvas element and the second one added Web Services real samples (prototypejs + JSON + JAX-RS). Now I am going to replace the Web Services for a new HTML5 feature: WebSockets. I had no experience at all with them and I think this is a good example where this new technique fits smoothly.

WebSockets are a new protocol coordinated by the IETC (The Internet Engineering Task Force) that tries to complement our old friend HTTP which falls short in some aspects of the Web 2.0. A WebSocket is a bidirectional (communication can be started by server or client), full-duplex (both ends can send information at the same time), minimal overhead (only two bytes are used in each frame), long lived (it remains opened until an end explicitly closes it) communication channel over a single TCP socket. Standardized by the W3C the JavaScript WebSockets API is completely asynchronous and very easy to use. The protocol also defines new ws:// and wss:// prefixes which indicate a WebSocket and a WebSocket Secure connection respectively. This technique is clearly defined for avoiding all the pain associated with AJAX/Web2.0 frameworks (overcoming HTTP limitations, more simplicity, better performance,...).

The technology has been hardly pushed by Google since the beginning, obviously this company is very interested in simplifying and improving web 2.0 applications. Therefore Chrome supports WebSockets since its version 4.0 and Apple joined later with Safari 5. Until some months ago the adoption was going smoothly with firefox 4 and Opera 11 announcing WebSockets as one of their new features. Nevertheless they turned out wrong when a protocol security flaw was discovered (standard is still in draft version 76). Adam Barth showed how a malicious code can be injected in the browser using a technique called cache poisoning (exploiting a problem in the initial handshake between client and server when a web proxy is involved). Some days after the bug was reported firefox announced the new feature would be disabled by default in its new beta and same did Opera. Microsoft was always more reluctant to WebSockets, even before the security flaw was pointed out they had not said a word about this protocol and IE9. In my opinion WebSockets is a perfect example of the current anxiety about the web (all browsers trying to support any new feature before competitors, no matter if the technique is mature or not, at the same time they put any stumbling block in another's path) but the technology behind them is clearly a good idea that just needs more time. Here it is a good and more detailed report of these events.

So now my scatter graph application is going to obtain the samples using a WebSocket instead of the prototype/JAX-RS web service. In order to use the new protocol both sides need to support it, for the client part default debian Chromium 6 will be used and Glassfish 3.1 (RC2 - b41) will be our Application Server (Glassfish uses Grizzly as its Web Server component which is the piece of software that really supports WebSockets). In the implementation side almost all remain the same but the following changes:

The JAX-RS resource class is replaced by a WebSocketApplication (Grizzly WebSocket 1.9 API). PlotApplication.java is the real class and JAXB is still being used for marshalling and unmarshalling stuff (the same PlotData object is used but JAXB is now directly called). Other difference is that JSON is now used in both ways (communication from Server/Java to browser/JavaScript and vice-versa).
The WebSockets technology in Grizzly needs a WebSocketsServlet.java to integrate the previous application into the WAR module. GlassFish Web Sockets Sample and Web Sockets and HTML5 in Glassfish are two good entries about how to implement a WebSocket Application using Glassfish/Grizzly.
The WebSocket application uses the same JPA entities to access the database but now a simple EJB PlotQueryEjb.java has been implemented (adding @Stateless annotation to the WebSocket application threw an exception). CDI is used for injection as usual.
The JavaScript in the server side also needs to be changed, basically prototype.js is replaced by the WebSocket API. So now the TestRGraphChart.js (JavaScript object) and TestRGraphChart.xhtml (Facelets page) are slightly modified in the sending/receiving part.

For all these reasons the video this time is similar and less interesting. I first execute a test.html page that shows the JSON data exchanged, only WAVE type is requested so the last 24 hours of wave highs are returned, the WebSocket is then closed. After that the same actions of the previous scatter graph video are performed (first showing sea level magnitude, one sample per minute, and then waves, one per hour). The WebSocket scatter graph has been tested with Chromium 6 and firefox4 b11 obtaining a good and smooth usability (FF4 needs to override the security block for WebSockets).

In short this entry is a presentation of the WebSockets, another HTML5 improvement. WebSockets are not just a new feature they are really a new protocol definition standardized to make an easier and better Web 2.0. Its adoption is being full of obstacles and many browsers do not support them by default (only Chrome and Safari right now). So current entry has to be intended as a training application and not a production ready example. You can download the complete NetBeans project used for the video from here.

Stay tuned for more about the Web Wars!

Pon un sol en tu vida: How device size affects disk performance in Linux (II)

nospam@example.com (Sergio Lopez) — mi�, 16 feb 2011 09:10:44 +0000

I've just noticed that in a virtualized environment, where roundtrips are more expensive, read speed differences between a page-aligned partition and a non page-aligned one are even more noticeable:

Non-aligned partition (size in sectors: (41929649-63)+1=41929587):

none:~# fdisk -l -u /dev/sdb



Disk /dev/sdb: 21.4 GB, 21474836480 bytes

255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors

Units = sectors of 1 x 512 = 512 bytes

Disk identifier: 0x93dbf2be



   Device Boot      Start         End      Blocks   Id  System

/dev/sdb1              63    41929649    20964793+  83  Linux



none:~# dd if=/dev/sdb1 of=/dev/null bs=1M count=100 skip=0

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 3.80593 s, 27.6 MB/s



none:~# dd if=/dev/sdb1 of=/dev/null bs=1M count=100 skip=100

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 3.82304 s, 27.4 MB/s



none:~# dd if=/dev/sdb1 of=/dev/null bs=1M count=100 skip=200

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 3.83713 s, 27.3 MB/s

Page-aligned partition (size in sectors: (41929654-63)+1=41929592) :

none:~# fdisk -l -u /dev/sdb



Disk /dev/sdb: 21.4 GB, 21474836480 bytes

255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors

Units = sectors of 1 x 512 = 512 bytes

Disk identifier: 0x93dbf2be



   Device Boot      Start         End      Blocks   Id  System

/dev/sdb1              63    41929654    20964796   83  Linux





none:~# dd if=/dev/sdb1 of=/dev/null bs=1M count=100 skip=0

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 0.618494 s, 170 MB/s



none:~# dd if=/dev/sdb1 of=/dev/null bs=1M count=100 skip=100

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 0.582423 s, 180 MB/s



none:~# dd if=/dev/sdb1 of=/dev/null bs=1M count=100 skip=200

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 0.589081 s, 178 MB/s

NOTE: In both tests, the data is cached in the host (so there is no real I/O to disks but traffic through virtualization pipes), but clean in the guest.

Pon un sol en tu vida: How device size affects disk performance in Linux

nospam@example.com (Sergio Lopez) — mi�, 09 feb 2011 08:04:48 +0000

While running some tests in a client's environment, we've noticed reading from a partition of a multipath device was considerably slower than reading from its parent node:

[root@none]# dd if=mpath4 of=/dev/null bs=1M count=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 8.92711 seconds, 120 MB/s



[root@none]# dd if=mpath4p1 of=/dev/null bs=1M count=1024 skip=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 17.5965 seconds, 61.0 MB/s

We asked to client support of a well-known GNU+Linux vendor, and they indicated that this behavior was "expected", since this kind of partitions were created by stacking a dm-linear device over the original multipath node. I wasn't satisfied by this answer, since AFAIK dm-linear only did a simple transposition of the original request over an specified offset (the beginning of the partition), so I decided to investigate a bit further on my own.

Continue reading "How device size affects disk performance in Linux"

Ricky's Hodgepodge: Debian 6.0 released!

nospam@example.com (rickyepoderi) — dom, 06 feb 2011 09:53:58 +0000

This time a quick entry. As everybody who usually reads this blog knows I use debian testing as my common linux distribution (working laptop and general use desktop). Today I want to present one of the reasons why I love debian. The next version of this operating system is released when it is ready, and when is it ready? When critical bugs are counted down to zero. And squeeze was ready this morning:

It is incredible how an open community, where anybody opens, follows and patches bugs, can work in such an efficient way.

Congratulations to everyone involved! I admire you guys! And welcome wheezy!

Nolife at Nologin: El Libro sobre el Origen de Nologin

nospam@example.com (bufon) — mar, 25 ene 2011 14:49:00 +0000

Nuevo libro publicado por Nologin Publishing:

Porque, a pesar de lo que dicen en algunos sitios, Nologin tiene más de 10 años.

¿Quieres un adelanto del contenido? Este es el índice:

Continua leyendo "El Libro sobre el Origen de Nologin"

Ricky's Hodgepodge: Improving RGraph Plot with Real Data via Web Services

nospam@example.com (rickyepoderi) — vie, 21 ene 2011 22:29:00 +0000

In a previous entry I talked about how to implement a scatter graph using javascript and the new HTML5 canvas element. This post was just the first step towards a more robust solution. If you remember my intention was studying the feasibility of a migration from an old applet application to a new HTML5 one. Today I will try to continue the PoC plotting real data inside the graph instead of the fixed sine wave (javascript generated) RGraph drew before.

First of all the real samples must be retrieved from somewhere and, obviously, a relational database is the easiest solution (besides the current application uses a database too). It is important to remember that the scatter plot represents a physical magnitude (wind strength, sea level, high of waves,...) in time. For the PoC some JPA (Java Persistence API) entities were created, these classes are very simple. Every sample only contains the timestamp and the measured value. It is clear that in a real scenario tables would usually have more columns (current devices measure several magnitudes at the same time) but I think this is enough for this example. Three classes are presented:

PlotType.java: Enumeration that handles all the plots the application can draw (my idea is simple, if a new plot is needed it has to be added here). SEA_LEVEL and WAVE are the current plot types (Sea Level is measured one sample per minute and the High of Waves one per hour).
PlotSample.java: Abstract entity class that represents any sample (as I said, a timestamp and the measured value).
SeaLevel.java and Wave.java: Two examples of real plots, both extend PlotSample and define a <PLOT_TYPE_NAME>-findSamplesBetweenDates query which is always used to query for the samples between two specified dates.

Once the data is organized a web service is needed. It is quite clear this technique is perfect to retrieve the data for plotting. Nowadays there are a lot of JavaScript libraries to easily call a server-side Web Service (prototypejs, jquery,...) and managing JSON (JavaScript Object Notation). For this part I decided to use JAX-RS (Java API for RESTful Web Services) to implement the service and JAXB (Java Architecture for XML Binding) for generating the JSON data. JAX-RS (which I had rarely used) is incredible easy and very good services are implemented in a little time. Besides JSON can be specified as the result to be produced and this, using JAXB at the same time, lets you return a Java object avoiding all JSON generating stuff. Only two classes are presented here:

PlotData.java: This is the JAXB element that represents the data to be plotted by the browser. The data returned when a scatter plot is going to be drawn is just a simple class with some properties (start and end timestamps, plot type name, description,...) and a map with the samples (value keyed by the timestamp).
PlotResource.java: The JAX-RS resource which receives the name of the plot type to graph, start and end timestamp. With this information it queries the database and creates the PlotData to be returned.

Two important tips here. First, JavaScript (in my opinion) handles the timezone in dates not very good. The date object is created using the timezone of the browser/system (GMT+1 in my case) but it is quite difficult to convert from one timezone to another. The date object has a getTimezoneOffset method that returns the offset in minutes between your zone and GMT, but your code needs to add and subtract offsets all the time. For this reason I decided to manage timestamps (milliseconds since standard epoch of 1/1/1970) instead of string dates. A timestamp will be represented in JSON with a String containing the big number of milliseconds prefixed by a character "T" (for example the epoch 1/1/1970 would be the string "T0").

Second, I wanted the JSON response was (more or less) the following:

{
  "type":"WAVE",
  "start":"T1295095138172",
  "end":"T1295181538172",
  "description":"High of the waves",
  "legend":"Meters (m)",
  ...
  "data":
  {
    "T1295096400000":"3.4",
    "T1295100000000":"3.16",
    "T1295103600000":"3.52",
    ...
  }
}

The JSON data part (the list/map of samples) should be the pair timestamp and value, which is the natural way to represent a map in JSON. Nevertheless JAXB has some problems with maps and they are not represented in this natural way (see RFE JERSEY-551 for more info). So I added the JsonMapAdapter.java class this enhancement manages in order to obtain the JSON representation I showed before.

So now the browser can query via the restful web service in order to obtain the data to plot. The final part is performing some little changes in the JavaScript developed for the previous post, now the array of samples has to be retrieved calling the Web Service. The presentation part are the following files:

PlotTypeBean.java: A extremely simple request scoped JSF bean only used to list the plot types defined in the application (the combo box at the right) and select one of them.
TestRGraphChart.xhtml: XHTML page very similar to the one used in the first entry but using facelets/JSF in order to call the previous bean and so on.
TestRGraphChart.js: The JavaScript file that uses prototype.js and JavascriptToolbox data.js to call the web service and parse/manage JSON and timestamps.

So the PoC is ready, RGraph plots a scatter XY graph of which data is retrieved via Web Service. The application was deployed in a Java EE 6 compatible application server (Glassfish v3) using its default database engine (derby/javadb). I personally think this is an elegant solution for my problem, all the plotting part is done by the browser and the server only sends the samples to draw. The solution works quite well in browsers with decent canvas performance (which are all the current browsers except IE8, check the previous post). In the video I first call restful web service with WAVE plot type (high of waves), I use no parameters so it returns 24 hours of data (WAVE plot type has one sample per hour). I change to the graph page in order to display the whole day of the sea level plot (one sample per minute). I also move the graph forward and reverse in time. Then I increase the range of time to display a whole week of data (less than 10,000 points cos I did not insert in my database the whole week of samples). And then I choose the time range to plot selecting directly over the canvas element. Finally I select the other plot type, high of waves again. As you can see it works very well in my chromium 6 browser. The application is also very smooth in iceweasel 3.5 and epiphany 2.30 (the other two browsers I have installed in my debian box).

At the moment the JavaScript always requests all the samples to plot. It is clear that some data can be reused (for example when the plot is moved forward or reverse) and the JavaScript could be improved to only request the missing samples and after join all of them. You can improve that part if you need and want it.

As a summary this entry goes one step further in the scatter graph problem. Now the solution plots real data. Data which is read from a database and requested by the browser using a RESTful Web Service and JSON. Here I present the complete NetBeans project (without any jar file as usual). As I said in the first entry of this series the PoC is currently unusable (or at least without a second plan for IE8), but the time for triggering HTML5 features is getting closer day by day...

Please plot a smile in your face!

Ricky's Hodgepodge: Decision Taken

nospam@example.com (rickyepoderi) — mar, 04 ene 2011 07:45:00 +0000

Finally I decided to leave nologin. It has been a fabulous nine years here but I feel now it is time for a change. Our paths have been diverging for some time and now I think it is the correct time to walk separately. I will really miss you all but we will surely keep in touch.

Now I have to realize I am not here and I am working at other place. I am putting my new helmet on...

“Maybe you can finally understand that it is possible to be true to your goal and still pursue your own personal happiness. The two are not mutually exclusive.”
- Magneto to Professor Charles Xavier in Uncanny X-Men Vol 1 #309 (Scott Lobdell - John Romita Jr)

I think nologin let me continue this blog right here. But stay calm if entries are less frequent, my life is changing in these moments and maybe I need some time to get used to it.

Ricky's Hodgepodge: CalDAV and the Whitepages Portlet

nospam@example.com (rickyepoderi) — jue, 30 dic 2010 08:02:00 +0000

In the past weeks the blog received some comments about the old caldav introduction series. If you remember this series consists of two entries, the first one introduces caldav and presents some http/webdav methods and the second post shows some real caldav requests. At that time I decided to not attach the complete project but linking every java file when it goes into stage. I thought this way would result in a better understanding but, obviously, I was wrong. For this reason I finally changed my mind and I am going to present the whole project and a real PoC example.

For some months I have been thinking in improving my little whitepages portlet (shown in both VoIP entries, Skype internet solution and Pidgin/OpenFire custom intranet project). It would be great if the whitepages, besides VoIP presence and interaction capabilities, could show the events or free/busy information for a specified company mate. This way any colleague can see what events has any other mate arranged for today and act accordingly.

It is quite clear that CalDAV is perfect for accessing the company calendar solution. In this case I have not installed any Calendar software (like I did with open sourced Calendar Server in the CalDAV entries) and the whitepages portlet will work against internet google and Yahoo! calendars. Finally I configured Yahoo! as the default calendar cos google does not support free-busy-query in its CalDAV implementation.

Before implementing the whitepages extension I worked with the old CalDAV project. Some jackrabbit libraries have been upgraded to new 2.2.0 version and another report has been implemented: free-busy-query. This report queries for the free/busy information of a user in a specified time. Its response only shows when the user has some events arranged, is busy, and when is free, but not specifying any detail of any event. If you check the libraries, jackrabbit-webdav depends on the old Commons HttpClient 3.1 which was superseded long time ago by a new HttpClient in the HttpComponents project (actual GA version 4.0.3). But its API has been deeply changed and jackrabbit needs some time to upgrade (see bug JCR-2406 for more details). This is an important issue to consider before choosing jackrabbit-webdav and I did not comment it before. As I promised here it is the complete CalDAV NetBeans project (libraries are missing because of space).

As I commented in previous VoIP entries the whitepages is a JSF 2.0 portlet application which uses OpenLDAP as the user repository. Every user entry has several attributes, and some of them are simply displayed (name, telephone,...) and others are treated specially. For example the jpegPhoto attribute is fetched and transformed into an image tag, the mail one is converted into a common email href link and skype/pidgin attribute is interpreted to show the presence and interaction links (start chat, add to my contacts,...).

For calendar integration a new attribute is used. This attribute is going to have the complete URI for the CalDAV Event collection of the user, in case of Yahoo! the attribute will be https://caldav.calendar.yahoo.com/dav/<yahoo-email>/Calendar/<calendar-name>/. But like the other special attributes, the calendar one will not be just presented, the URI will be used to access and query for the today user events. These events will be retrieved and presented to the end user. In order to fetch the events of a user calendar it must be shared with whitepages internal user or be totally public (i.e. any other google or Yahoo! user can see our events). In Yahoo! the calendar accessibility is set in Options → Share Calendar → Other Ways to Share → Public Link and in google in Settings → Calendar settings → Calendars → Click the calendar you want to share → Share this Calendar → Make this calendar public. In both sites there is an option to only share free/busy information and not make the calendar absolutely public (details are hidden and only availability is shown) but, also in both, this option does not work. Google does not support free-busy-query (CalDAV cannot access free/busy information) and in Yahoo! this option is disabled (with a beautiful coming soon message). In a custom calendar solution the whitepages could also be configured to use a superuser with special grants to access any other user calendar. But, as I commented, finally I set Yahoo! URI in the LDAP attribute of my entry and therefore my calendar must be public. The portlet tries first to obtain all the events arranged for today using calendar-query report (this needs all events to be public) and, if this first option does not work, it tries a second free-busy-query report to only show when the person is occupied (only free/busy access is needed).

Here it is a new video. My hypothetic boss wants to see the whitepages improvements before the department meeting at the afternoon and he searches for me in the intranet portal. After checking I have one hour free in the morning and I am online in Skype he decides to chat with me and arrange a meeting. I accept the offer and update my Yahoo! calendar. The update is immediately shown in the portlet.

I hope CalDAV entries were clearer now. This entry has attached the complete caldav project (see above for the link) and shows whitepages as a real example (very simple but a real one). I like the way my little whitepages application is being improved.

Happy new year!

Nolife at Nologin: Conspiraciones

nospam@example.com (lfabiani) — jue, 09 dic 2010 22:52:00 +0000

No soy dado a escribir sobre temas serios para evitar repercusiones, pero hoy voy a romper esta norma no escrita. Primero de todo, dejar claro lo siguiente:

Si creyera que el mundo es perfecto, no creería en las conspiraciones ni en las cortinas de humo.
Si no creyera en las conspiraciones, creería que todo el mundo es bueno.
Si creyera que todo el mundo es bueno, ninguna coincidencia me parecería sospechosa.

Y, tras el salto, una serie de extrañas coincidencias.
Continua leyendo "Conspiraciones"

Ricky's Hodgepodge: Canvas Scatter Plot

nospam@example.com (rickyepoderi) — lun, 22 nov 2010 23:37:00 +0000

.canvas-table { margin-left: auto; margin-right: auto; text-align: center; } .canvas-table td { padding-left:4px; }

These weeks I have been working in an application that shows some sea physical data (sea level, waves altitude, water temperature, wind strength,...) using scatter plots. Now the application uses an applet to display the XY graphs which, I must say, is incredibly fast. But, as I commented before in the digital signature entries, applets are now a quite obsoleted solution and sometimes are not very good in terms of compatibility (users need to have a Java Virtual Machine installed, problems with different versions, many linux distributions use by default openjdk/icedtea/gcjwebplugin which crashes sometimes and many other issues which usually generate a lot of complaints). I decided to study a direct browser solution using new HTML5 and javascript features.

Spending some time searching over the web I saw there are a lot of libraries to directly draw charts in javascript. But looking deeper I discovered the majority of them use the new HTML5 canvas tag. The canvas is a new component to render inside its area 2D shapes and images using javascript.

The first library I used was JSCharts (I did not think too much, it was at the first position in the previous link). It is very powerful but does not fit completely for my problem. The plots of the application were usually a magnitude (meters, degrees, velocity,... in the Y axis) throughout the time (X axis). In the applet version some magnitudes show a very large number of points (around ten thousand). JSCharts cannot handle such a number of points, the resulting graphs are very beautiful but too much complex for this large amount of data. There are some features that explains this fact: points are circles, a specific scatter plot does not exist and a line chart (with invisible line) must be used, time labels must be assigned separately,... All of them make JSCharts not fast enough to draw the 10,000 points in a reasonable time.

After this first disappointment I decided to just test with a direct canvas implementation (i.e. developing my own javascript that plots a very simple XY graph). But searching information about canvas I discovered a very straight forward graph library which also uses the canvas tag, RGraph. This implementation is much simpler than JSCharts and it has a direct scatter plot graph type. Developing a initial time/magnitude plot the times became very very competitive (for example RGraph scatter plot draws 10,000 points in less than a second in many browsers).

Finally a study about browser compatibility must be done. And one more time, same thing happened in the HTML5 video announcement, all Internet Explorers (included current IE8) do not support the canvas tag. Right now there is a ExCanvas project that emulates the canvas tag using VML. The solution (as you can see later) is incredibly slow and there are initiatives for doing the same using flash , silverlight or a ActiveX component. IE9, which is in beta state, is going to support canvas with many other HTML5 features. For this reason I prepared a little example of a XY plot to test usability with different numbers of points. The sample code draws a sine wave (javascript generated) in a period of time and let us change the number of points, showing finally the drawing time in a box (the part for generating the wave is not taken into account). At the end of the entry I present the little example inside an iframe (I know it is not very beautiful but this could be easily improved).

When I played with the resulting graph using different browsers numbers were the following (I took them not very precisely):

	100 points	1,000 points	10,000 points
iceweasel 3.5.15 (linux)	26ms	80ms	712ms
chromium 6.0.472.63 (linux)	10ms	18ms	98ms
firefox 3.6.12 (Windows)	32ms	130ms	1,018ms
firefox 4.0 Beta 7 (Windows)	32ms	120ms	970ms
chrome 7.0.517.44 (Windows)	11ms	32ms	240ms
IE8+ExCanvas (Windows)	800ms	3,850ms	70,000ms+
IE9 Preview 7 (Windows)	17ms	65ms	500ms
Opera 10.63 (Windows)	25ms	49ms	305ms

My conclusion is a HTML5 plot solution results usable in any browser that supports canvas, but IE8 kills us again. Any canvas graph is fateful for many windows users. I decided some time ago to trigger all HTML5 new features when debian testing will distribute iceweasel 3.6 (squeeze must be released before) and chromium 6.0 (it is already) and IE9 will be released. As you can see in the numbers, times are very good in the last versions of the browsers and even better in the future ones. You already know but it is absolutely necessary to keep our browsers in their last versions. Now there are a lot of options (free, open source, distributed with the OS,...), so there is no excuse. Make web developers life easier...

Please, upgrade your browser!!!

Pon un sol en tu vida: DRM y Software Privativo van de la mano

nospam@example.com (Sergio Lopez) — mar, 09 nov 2010 21:24:30 +0000

Navegando por reddit, me he encontrado con esta entrada en la que un usuario se queja de que no puede reproducir un Blue-Ray legítimamente adquirido, ya que está utilizando como salida de vídeo una TV conectada al puerto HDMI de su portátil (un Macbook Pro).

En dicha entrada se comenta cómo los fabricantes se perjudican a sí mismos con este tipo de prácticas. Ciertamente es un aspecto a destacar, pero también es importante tener en cuenta que dichas prácticas no serían posibles sin que existiese cierto Software Privativo que haga efectivas las limitaciones artificiales impuestas por los fabricantes. Y donde hoy es impedir que se reproduzca un contenido si se visualiza en una TV sin HDCP, mañana podría ser cualquier otra prohibición igualmente ridícula y arbitraria.

Y es que este es uno de los principales motivos por los que Richard M. Stallman inició el proyecto GNU hace 25 años: impedir que los fabricantes tuvieran la capacidad de decidir sobre lo que el usuario puede o no hacer con su computadora. El Software Libre, como su propio nombre indica (al menos en su denominación castellana, más precisa que la anglosajona), trata sobre la libertad, y no sobre la gratuitidad, matización que con frecuencia se suele pasar por alto.

Si en atención a nuestra comodidad entramos en el jardín vallado que nos presentan y damos por buenas este tipo de prácticas (y entre ellas podemos incluir también modelos de distribución de software como el AppStore), le estamos dando vía libre a los fabricantes para que tomen el control de nuestros dipositivos y decidan por nosotros lo que podemos hacer con ellos. Y los peligros que esto entraña en una sociedad donde las TI están presentes a todos los niveles (y cada días más), son impredecibles.

Nolife at Nologin: El final de la familia Zapatero

nospam@example.com (lfabiani) — jue, 04 nov 2010 07:56:00 +0000

Hoy llego al trabajo indignado. El congreso ha aprobado la nueva Ley del Registro Civil. Entre otras cosas, establece que cuando no haya acuerdo entre los padres, los apellidos se pondrán en orden alfabético. Hasta ahora, primaba el apellido del padre por motivos tradicionales. Vamos, el final de la familia Zapatero.

Es cuestión matemática: Zapatero es uno de los últimos apellidos alfabéticamente, luego cualquier Zapatero que se case (hombre o mujer) tiene más de un 90% de posibilidades de que su pareja tenga un apellido prevaleciente. Luego quedará en segundo lugar, no siendo heredable (salvo, lo dicho, acuerdo entre los padres). La siguiente generación se repetiría el proceso. En 2 o 3 generaciones no quedarán Zapateros.

Hay que tener en cuenta que el acuerdo entre los padres para modificar la norma general, pese a ser una posibilidad, no es lo más extendido, ni con esta ni con ninguna norma, y que de facto la tradición se fundamenta en normas que se han convertido en hábito. Así que esta ley nos pinta un mundo con apellidos que empiecen sólo por las primeras letras del abecedario. Como indican en El País:

Patricia López Peláez, profesora titular de Derecho Civil de la Uned, explica que ese sistema alfabético puede provocar que, en unos años, terminen prevaleciendo los apellidos de la primera parte del alfabeto. "Pero es cierto que se ha buscado un criterio objetivo para no discriminar ni favorecer a ninguno de los miembros de la pareja", explica.

Se me ocurren muchos otros titulares maravillosos:

Zapatero legisla para que Aznar perdure pero él se extinga.

Los Villaarriba y los Villabajo se unen para manifestarse contra el fin de sus linajes.

En el futuro, en España todos nos apellidaremos Abel.

Personalmente, me parece una barbaridad lingüistica. En aras de la igualdad, se comete una estupidez cuya consecuencia será que en unos años nuestro sistema de apellidos perderá en riqueza. Los Zapata, Zaragoza, Zurueta, Zurbarán, Zafón, Zumárraga y demás serán historia y quedarán relegados a las páginas, expulsados de la realidad.

La clave está en que se ha perdido el componente de aleatoriedad en la elección del nombre de los apellidos. Hasta ahora el orden en España dependía de una variable aleatoria como era el sexo de los progenitores (quizá en el futuro deje de ser aleatorio, pero ahora no lo es). Los legisladores dicen en el mismo artículo del País:

(...) se ha estudiado la legislación de otros países en los que se puede elegir el orden de los apellidos. En Alemania, por ejemplo, se lanza una moneda al aire para tomar la decisión, en caso de que los padres no se pongan de acuerdo.

Se ha estudiado pero NO se ha comprendido. En Alemania garantizan que todos los apellidos tengan posibilidades de continuar existiendo. Aquí no. Prefiero mil veces una discriminación positiva que cambie la tortilla radicalmente, poniendo delante el nombre de la madre (después de todo, las madres legan más componente genético que los padres gracias a las mitocondrias) que esta estupidez que va a terminar con parte de nuestro legado cultural.

PD: Busco un logo para una campaña: "Los estupidos legislan: Zapatero mata su apellido". ¿Alguna sugerencia? Vamos a darle visibilidad a esta estupidez.

Ricky's Hodgepodge: Using Cassandra with EJB Technology

nospam@example.com (rickyepoderi) — lun, 25 oct 2010 18:38:00 +0000

The past week Eric, a french reader of the blog, contacted me because of my old Cassandra entry. He was trying to get some help in how to implement an Enterprise Java Bean for the NoSQL database. As I explained in the Terminal Services posts the portlet used Thrift and Cassandra to store the application tree which was shown in the series. I am going to present the same solution but replacing the portlet with an EJB 3.1 / JSF 2.0 web application. The idea of this post is explaining the whole process.

First of all the Cassandra data model was designed. If you remember, the Terminal Services portlet is a tree very similar to a file system. Any item can be a folder or an application. Folders can contain any number of other folders or applications, and applications are leafs which contain the RDP definition file (the file which is downloaded to launch the Terminal Services client program). Following the ideas presented in the old Cassandra post only two simple columns were needed.

Applications: The applications column family will contain all the applications and folders. The key will be the complete and uppercased path of the application (for example /APPLICATIONS/SKYPE) and the values will represent different data. More precisely the columns will be the following:

name: application name to show (Skype).
description: application description.
icon: 16x16 PNG icon.
type: app or folder.
rdp: rdp file to startup this application (only for apps).

Applications: { // ColumnFamily
  /INTERNET: { // key
    name: Internet
    description: Internet applications
    icon: 0x19EA14...
    type: folder
  },
  /INTERNET/SKYPE: { // key
    name: Skype
    description: VoIP client application (www.skype.com)
    icon: 0x56FA12...
    type: app
    rdp: 0xA4326A...
  }
  ...
}

ApplicationStructure: This other column family just cover the folder structure which is missing in the other one. This column is quite simple, the key continues to be the path of the folder in uppercase and the values are its children (path and name of every child). This column family gives an easy way to access all the children of a specified folder.

ApplicationStructure: { // ColumnFamily
  /APPLICATIONS: { // key
    /APPLICATIONS/INTERNET: Internet
    /APPLICATIONS/PAINT: Paint
  },
  /APPLICATIONS/INTERNET: { // key
    /APPLICATIONS/INTERNET/PIDGIN: Pidgin
    /APPLICATIONS/INTERNET/SKYPE: Skype
  }
  ...
}

In order to create the EJB which manages applications (and folders) the idea is just creating a bean which handles basic CRUD operations for applications. This class is called CassandraManagerEJB. In the previous posts this class was a POJO (liferay was deployed inside a tomcat and EJB technology was unavailable) that has been now converted into a Singleton Enterprise Bean. Another class called Application represents any application or folder with typical getters and setters. Obviously the CassandraManagerEJB receives objects of this class and transforms them into thrift mutations and columns for writing and vice-versa for reading. There is a special folder keyed by "/" for the root path. The EJB uses commons-pool to treat Thrift connections inside a pool. The three main methods in the EJB are the following:

readApplication: Reads any application from the Cassandra repository. The application or folder path is passed to the method. The attributes to read can be also specified in order to not read all of them. But it also receives a parameter to specify how the children is retrieved:

NONE: No children are returned, the column family Applications is the only one read.
ONLY_NAMES: The application is also read from the ApplicationStructure column family and only its data is filled in the children list (path and name).
FULL: After reading the children from ApplicationStructure all of them are also got from the Applications column family. So now all the specified attributes are filled in the children list.
RECURSIVE: The method is called recursively to retrieve all hierarchy with all attributes specified, from the path passed to the bottom.

createApplication: Creates or updates an application or folder. This method do not let create folders with children (for folders is just like mkdir command), application names cannot contain '/' character (only root can do that) and the specified parent must exists previously in Cassandra repository. If the application object passed does not follow these rules an Exception is thrown.

deleteApplication: Deletes an application or folder. If the application does not exist or it is a folder with children the method throws an exception (again it can be considered like a rmdir command in case of a folder).

I tested the web module deploying it inside a J2EE6 compatible glassfish v3 application server. So the EJB is annotated to be a Singleton and an application scoped named object, this way the EJB can be injected using Context Dependency Injection (CDI). Remember the beans.xml file must be created to make WELD (CDI implementation) start to work. I recommend to read this post to know more about the differences between CDI and EJB injecting techniques.

The JSF part will use the created enterprise bean to display all the tree. There is only one managed bean which is called ApplicationTreeBean but it is accompanied by some helper classes. This JSF bean uses injection to get the Singleton EJB and is also annotated as named to be injected into index.xhtml facelet like a session scoped bean. The JSF part can be more or less complicated but it just presents the application tree we saw in the Terminal Server entries (see image below).

As all of you can see this way of integrating Cassandra into EJB 3.1 technology is very very straight forward but, not being an expert in Cassandra, I think it is the most suitable nowadays. Remember all Cassandra stuff presented here uses the inflexible Thrift client (I chose it cos I wanted to learn Cassandra internals), there are some more java clients out there and surely they are friendlier. Here it is the whole Netbeans project just in case (libraries have been deleted to save space).

When I talked all this stuff to Eric, he advised me there is a project called Kundera (I have just realized it is listed as Cassandra client in the previous link) which is trying to implement a JPA provider for Cassandra (see the following post). At first moment I think it makes no sense, the Java Persistence API is tightly coupled with SQL (it can be said it was designed to cover SQL databases) and, therefore, not very suitable for a NoSQL repository like Cassandra. The weekend I spent some time trying to figure out how, for instance, the queries are performed. Kundera mixes Cassandra repository with Lucene indexing engine and, this way, some little queries can be performed against the indexes. Looking into relations (many to many, one to many, one to one,...) I saw some of them are still not implemented, but this point is less problematic cos developers could (more or less) follow the same ideas I commented in my previous entry about Cassandra. So, despite of my initial skepticism, the project has at least some sense. I still believe the idea is a bit farfetched but I have to admit they are implementing the same solution I would have used in a supposed Cassandra project. If you remember I already talked about mixing Cassandra and some indexing engine like solr (Lucene based) to get a robust result.

Good job guys! Continue surprising me!

Ricky's Hodgepodge: PFC

nospam@example.com (rickyepoderi) — dom, 26 sep 2010 21:22:00 +0000

The other day I was helping my mother to tidy my parent's junk room when I found all the documentation of my PFC (a kind of dissertation needed in Spain to get your bachelor degree). The whole stuff was a non-working CD, a VHS tape and the book itself.

At that time I was quite interested in computer graphics and I tried to do my project in the graphics department of my University. This department was (and still is) a group called GIGA, Spanish acronym for Advanced Computing Graphics Group. After some initial doubts I finally decided to continue an unfinished project of the group. The GIGA had collaborated with CHE (a governmental company that deals with water issues in my home region, its name comes from the river that crosses Zaragoza, the Ebro river) and the group had achieved from them some valuable information:

An altitude map file. The file was a 40 megabyte file with the altitudes of the Ebro basin terrain measured every 100 meters (the file was a altitude mesh of squares which sides are lengthened by 100m).
Satellite images of the same basin (6 CDs). The quality of the images was more or less a point of color every 5 meters.

I know that now, in the google era, this information is a real shit but 15 years ago all of this was a data of enormous worth. The GIGA group had developed a quick application to make a video for a CHE presentation in which you virtually cruised throughout some places of the basin. The idea of my tutor (Francisco José Serón) were mainly three points:

Developing, using some interpolation methods, a software that let them to create a more dense mesh.
Improving the flying application because it had been done just for the presentation. I suppose my other tutor (Juan Antonio Magallón) spent long nights to finish the job.
Making some study for finding water (rivers, lakes,..) in the satellite images. Finally this point was not done (only some comments and not a thorough job).

Of course only the second point was truly interesting to me and the reason why I wanted to work with GIGA but it was an all or nothing. The flying application used GTK for window management (menus, buttons,...) and OpenGL for drawing. The mesh was painted and the satellite images were used as textures. I switched from plain C to a C++ application and I tried to make a more structured code. During developing another collaboration came up, GIGA was in touch with a woman who was making a thesis about fire spreading. Of course the idea of a video or images about a fire spreading over some place of the basin quickly emerged.

And that is the reason for this entry. I have already lost my project, because all of it (code, altitude mesh, some texture images -the small ones-,...) was in the broken CD (I am trying to bring it back to life but I think it is a waste of time) but I want to preserve the video, some photos I still have and a poster I did. As I have already said in some occasions this blog is the perfect place to put things I do not want to lose.

Here I present some of the photos I still has, only a few cos the majority of them are lost inside the CD. The poster I configured about my project is also here.

Wire image with a long step in the mesh

Wire and texturized image with a long step in the mesh

Texturized image with a long step wire mesh

Wire image with a more dense mesh integrated

Wire and texturized image

Texturized image

Poster of my PFC

Fire spreading photo

Fire spreading photo 2

Fire spreading texture for tile 23/77

Fire spreading texture for tile 23/78

Finally the video, which was done later in a second collaboration with the fire spreading thesis. My application generated the main scenes but the script and all the making of the film was done by Diego Gutierrez, another guy of the GIGA group. Thanks to Edu for digitizing the VHS tape.

It is amazing how things change in less than 15 years. Enjoy!

PS: I sent an email to one of my tutors cos I do not know for sure if I can publish this information (video and images) or not. I have received no answer so I decided to continue. Please use comments or contact me if I am really not allowed to share these data.

Nolife at Nologin: No-Lata

nospam@example.com (bufon) — lun, 20 sep 2010 14:34:57 +0000

En este mes de septiembre se cumple el décimo aniversario de Nologin.

Según la Wikipedia, los 10 años de casados son las "Bodas de Lata".

Aplicando la lógica ilógica de Nolife:

¡¡Feliz Aniversario de Lata!!

Nolife at Nologin: El ataque de los caracoles asesinos

nospam@example.com (bufon) — mar, 14 sep 2010 15:00:00 +0000

Esta samana pasada perdimos varios compañeros. En la oficina acabamos de atravesar un incidente realmente traumático al que apenas sobreviví.

Todo comenzó de la manera más inocente. Uno de los jefes tuvo la idea de entrenar un ejército de caracoles que nos ayudaran en nuestro propósito de controlar el mundo (que viene a ser el propósito de cualquier empresa mediana o grande con aspiraciones). Las largas jornadas de instrucción iban bien, hasta que los caracoles se rebelaron.

Se me ponen los pelos como sus cuernos sólo de verlo...

No dudaron en atacarnos, a pesar de todo lo que les habíamos dado. Nos cortaron la retirada, haciéndose con la puerta. Se movían sigilosamente, con paciencia, rodeándonos. Un círculo perfecto, y cerrándose. Aún recuerdo cómo se lanzaron lentamente sobre algunos de mis compañeros, sus cuerpos recubiertos de esas asquerosas babas. Recuerdo sus gritos.

Después se retiraron. Nos dejaron vivir a algunos. Ahora están sueltos. ¿Qué pretenden? Oh, señor, ¿qué fuerza hemos liberado en nuestra inconsciencia? ¿Cuál será el precio que tendremos que pagar?

Cuando veáis a varios caracoles, arrastrándose en formación hacia vosotros, ¡HUID!

NOTA: Como hemos aprendido la lección, estamos enseñando técnicas ninja a gavilanes caracoleros para que persigan y acaben con los caracoles. Aprenden deprisa.

Ricky's Hodgepodge: DNIe and the Signature Applet

nospam@example.com (rickyepoderi) — dom, 05 sep 2010 20:51:00 +0000

Previously I talked about how to deal with web security and certificates in a two part series of entries. In the first post I generally talked about security concerns and I presented a little solution based on Java applets. The second one extended the PoC using the same certificates that some browsers do (IE and firefox). I already said that a real security device (cryptocard, cryptousb,...) was the missing point and I told to all of you the time will come with my Spanish electronic identity (eID) card renewal. The time is here!

DNIe (this is how the new Spanish eID card is known) has two valid certificates (two years of expiration time), one for digital signature and the other for authentication, two fingerprints (right and left index finger), the written signature and a photo. First of all I want to say the integration of the DNIe in linux is quite a hell. Linux implementation is based on the OpenSC project. This project is an old friend in the linux world and it implements a PKCS#15 backend (PKCS#15 is the RSA standard that defines how the data is organized in a card) and gives a PKCS#11 library (RSA standard API to interact with crypto-devices) to be used inside firefox and many other applications. DNIe distributes an extra library not included in common opensc bits (the library is called libopensc-dnie.so and it is inside opensc-dnie package) which is configured in the opensc configuration file as an external shared library. At first sight I thought these bits are only distributed as a closed blob but then I realized that there is also a source zip file. So I am going to explain both methods of installation (following exactly the same steps I performed).

To make the DNIe blob work in my debian testing box I needed to download the Debian_Lenny_opensc-dnie_1.4.6-2_i386.deb.tar from DNIe official page. The problem here is the blob only works with the distributed 0.11.7 version of opensc (actual version is 0.11.13) and you must exactly install these packages:

# dpkg -i libltdl3_1.5.26-4+lenny1_i386.deb libopensc2_0.11.7-7_i386.deb opensc_0.11.7-7_i386.deb opensc-dnie_1.4.6-2_i386.deb

The first one, libltdl3, is a package of Lenny (Stable) which have disappeared in testing but it is compulsory for opensc/DNIe and not included in the distribution tar file. Of course you also need to hold new versions of debian opensc packages (if not any upgrade will break your DNIe again).

# echo "opensc hold" | dpkg --set-selections
# echo "libopensc2 hold" | dpkg --set-selections

This integration is so crappy I had decided to not use DNIe in debian (I would have preferred to use a Windows virtual box or similar and not to dirty my system this way). But then, as I commented, I found a source distribution in the same download page. The source is not completed (it seems some certificate and key variables has been deleted from the source and one file does not compile) but this thread in kriptopolis (a spanish security web site) explains how to make it work. The thread is quite long but it was worth the time. After some nm and objdump commands I finally compiled the sources. Now I have my libopensc-dnie.so which links perfectly against OpenSC 0.11.13, debian testing version, and it works without a problem using the same opensc.conf configuration file that blob DNIe packages provide. The new library was tested inside firefox, thunderbird and my Java Signature Applet. I also want to comment that DNIe has a test page which has been incredibly useful to check out all this stuff.

So finally it can be said that DNIe binary packages only work for the distributed 0.11.7 version but compiling the sources you can make the custom DNIe library work linked against current OpenSC version. This is not a perfect solution (any new change in OpenSC can broke again DNIe) but I have to admit that publishing the sources is an awesome step to open source integration. The source zip from DNIe claims to be GPLv3 licensed and I only hope OpenSC can integrate this code in a future release (avoiding all this pain). OpenSC guys already have a open ticket about this issue. I submitted some information on it a few days ago and waiting is the only thing we can do right now.

After DNIe is fully functional in my box we can extend our Signature Java Applet PoC in order to integrate DNIe as a valid device. If you have read the two previous entries you already know that the only thing to do is finding a proper JCE provider for Spanish eID card. It is quite clear that DNIe/OpenSC gives a PKCS#11 library and, obviously, the PKCS#11 provider is the correct bet in linux. If you remember I developed a multi-purpose PKCS11Signer.java (used in the second entry for NSS) which is perfect for OpenSC. Now the applet is instantiated this way.

<applet id="signApplet" codebase="resources/applet/" code="sample.applet.SignApplet" archive="SSignApplet.jar,Sbcmail-jdk16-145.jar,Sbcprov-jdk16-145.jar,Smail-1.4.3.jar" width="350" height="200">
  <param name="clazz" value="sample.applet.PKCS11Signer"/>
  <param name="param0" value="sample.applet.pkcs11Name###DNIe"/>
  <param name="param1" value="sample.applet.keyAcessNeeded###false"/>
  <param name="param2" value="library###/usr/lib/opensc-pkcs11.so"/>
  <param name="param3" value="slot###0"/>
</applet>

With DNIe (and any common PKCS#11 library but Mozilla NSS) only the shared library (/usr/lib/opensc-pkcs11.so in debian distribution) and the slot (0 in my configuration) are specified. As in the previous examples DNIe only has a master pin so password prompt for keys is disabled (sample.applet.keyAcessNeeded is set to false). Any further information about the JCE PKCS#11 provider can be achieved in the J2SE documentation.

Finally I present a video where the mail signer application is used again. Using Chromium the email sheet is filled and the applet is started. There the two certificates inside DNIe are displayed and, using the signature one, the email is signed and sent. Now icedove/thunderbird application is started (the startup is quite slow cos the ID card is plugged and OpenSC initialization is not very fast). In this case when the email arrives to me it shows a question mark, that is a consequence of Spanish id certificates does not have any email specified inside them. At the end the certificates of the DNIe are shown again but using thunderbird itself, so DNIe is successfully working on both, the applet and the email application.

As a conclusion it can be said that right now the status of DNIe in the open source world is a bit confusing. The distribution of the source code under GPLv3 license can clarify the situation and it would be wonderful if the code will be integrated someway inside OpenSC project. Talking about the applet signature PoC, DNIe is the perfect example of how a new crypto-device is smoothly integrated just using JCE and PKCS#11 standard.

It is really great when you make the things work! Never give up!

Nolife at Nologin: Entrada no-patrocinada: Platos Vacíos Sin Vergüenza

nospam@example.com (bufon) — jue, 26 ago 2010 10:56:39 +0000

Suele ser bastante común que algunos blogs tengan entradas patrocinadas, en las que hacen publicidad descarada de productos a cambio de algún tipo de beneficio. En Nolife llegamos más allá, y tendremos no-patrocinación para hablar de no-cosas.

Hoy vamos a hablar sobre la ONG a la que pertenece uno de nuestros no-compañeros (por no decir fundador), llamada "Platos Vacíos Sin Vergüenza".

Continua leyendo "Entrada no-patrocinada: Platos Vacíos Sin Vergüenza"

Ricky's Hodgepodge: Integrating Terminal Services into a Portal (Linux Version)

nospam@example.com (rickyepoderi) — s�b, 21 ago 2010 13:56:00 +0000

In a previous entry Remote Desktop Protocol (RDP) was discussed and a custom portlet was presented. The portlet shows some applications (that were previously registered in Microsoft Remote Desktop Services) and it lets the user click any of them to download a RDP connection settings file that silently starts client-side RDP application. The solution was tested on a Windows XP box. Windows and Mac have free Microsoft RDC client application, but Linux is a complete different story.

Rdesktop is the traditional, and I think the only one, open source RDP solution. Until the time I faced that pre-sale I was not aware of the poor status of this project. Last rdesktop 1.6.0 version only supports version 5.2 of the remote desktop protocol and its releases are becoming more and more infrequent (1.6.0 was released in May 2008, 1.5.0 in September 2006, 1.4.1 in May 2005 and so on). The request to support RDP protocol 6.0 was open in the summer of 2007 (a year after 6.0 version was released with Windows Vista) and it is still not assigned. But, you know, this is open source, do not complain and do not wait others to do your job, fix it yourself! Now talking seriously, if somebody has enough time and skills and wants to participate in the never-ending war against Microsoft this is a good project that needs some love.

Coming back to the RDP portlet, Linux only has rdesktop and the portlet must work with it. The main restriction is that this program cannot use TS gateway, so only typical plain RDP (3389 TCP port) communication is possible (remember gateway can use the more secure, easier to integrate HTTPS protocol). This limitation makes absolutely compulsory the use of a VPN in order to establish a secure channel between an internet user and the server. And, of course, 5.2 protocol seems to be more bandwidth-hungry than 6.0/7.0 but nothing can be done with this issue.

Nevertheless seamless remote application was introduced in 1.5.0 rdesktop version. I am not really sure if this feature uses RDP 6.0 protocol or it was done ad hoc but, after all, rdesktop has the -A option to run a remote application. SeamlessRDP extension was a contribution of Cendio to rdesktop in another example of how open source works. This feature needs to install a server side windows program called seamlessrdpshell which is invoked by rdesktop as the alternate shell with the remote application to execute as an argument. For example:

$ rdesktop -A -s "seamlessrdpshell.exe c:\Windows\System32\mspaint.exe"

In my windows 2008 server installation this program was installed under c:\Windows\System32\ directory (this way it is always found in the execution path). Besides the executable was published as a valid TS Remote Aplication with command line parameters enabled (TS Server lets it run with arguments).

Other issue is rdesktop does not understand rdp files (it is just a common linux command with several execution options). Some other graphical applications (like tsclient) can parse rdp files and call rdesktop accordingly but none of them (at least I could not find anyone) understands the seamless rdp integration (-A option). So I had to code a little python example that reads the rdp file and transforms the file tags into rdesktop options (plaease do not blame me, I already know there is room for improvement). The script needs two minor changes in the generated rdp file (now the alternate shell is specified with the full path to the application and the username is also supplied) but it still works for Microsoft client. Here it is a wordpad.rdp example. Finally the password to rdesktop invocation is hardcoded inside the script (I have commented the script is improvable, haven't I?).

With all the commented limitations the portlet solution is ready to be executed in linux. Now when the application link is clicked the downloaded rdp file will be executed by the python script, which will run rdesktop with the seamless application option. The video is very similar to the presented in the previous entry but with Debian as OS and firefox/iceweasel as browser. If you see when rdesktop is launched the usual windows login page is presented, automatically fulfilled, and then it disappears, the reason is rdesktop does not have Network Level Authentication (new feature of 6.1 version of the protocol).

In summary, this time linux solution is less practical (internet access needs a VPN, a better RDP file parser would be necessary in a real deployment and some bandwidth improvements are lost) but thinking in a global solition, I mean, a solution for all type of clients, is always a good thing and, this way, nobody will say I did not try it.

Keep the faith!

Nolife at Nologin: La esencia de la publicidad

nospam@example.com (bufon) — lun, 16 ago 2010 22:00:00 +0000

La publicidad muchas veces intenta convencer de que el cielo es rojo, de que los perfumes dan la felicidad o las galletas caseras arreglan familias rotas. Por suerte, en Nologin no tenemos que recurrir a trucos como el que se ve en este video, tan..., representativo de la realidad.

Ricky's Hodgepodge: Personal Entries

nospam@example.com (rickyepoderi) — vie, 13 ago 2010 11:18:00 +0000

Today another entry about changes in the blog. First I have decided to add some information about myself in the left part. The previous weekend I requested my facebook account to be deleted, I did not use it. So I will try to post some personal entries from time to time and maybe make the blog a place where I can contact some distant friends (but of course this does not depend on me). This decision does not affect the usual topics which I want to keep on publishing. Second, this blog installation does not send any email to me (maybe I can discuss this issue with my company mates but I am afraid it is for some security reasons), so I have also added the last comments plugin in the left frame. This way I quickly know when someone has made one.

Cheers!

Nolife at Nologin: La esencia del comercial

nospam@example.com (bufon) — lun, 09 ago 2010 22:00:00 +0000

Hay dos maneras de hacer negocios: resolviendo problemas o necesidades, y creando problemas o necesidades. A veces la separación entre resolver y crear es realmente estrecha porque la mayoría de los usuarios no tienen conciencia del problema al que se han acostumbrado. Los usuarios no saben lo bien que les vendría tener dientes nuevos porque se han acostumbrado a los viejos. Y allí entra un buen comercial, uno mejor que Coco, por supuesto:

Ricky's Hodgepodge: Integrating Terminal Services into a Portal

nospam@example.com (rickyepoderi) — vie, 06 ago 2010 14:01:00 +0000

At the end of 2009 I participated in a pre-sale involving a portal migration in a governmental environment. The portal had some content management, portlets and an uncommon feature: citrix. This customer worked in a quite special way, their employees usually were out of their own installations but they needed regular access to some internal applications. The problem was part of these programs were very old dos/windows desktop applications which could not be easily moved to the web (customer had upgraded some to new web technologies but a lot of them still worked following this old-fashioned way). The portal used an old version of Citrix XenApp to place these old windows applications the customer really needed inside the portal. Besides that, typical requirements in a portal developing were demanded (sso, cms, eye-candy,...). An one more thing, customer wanted to change citrix for other solution and themselves also talked about the new features of windows Terminal Services.

Remote Desktop Protocol or RDP is a proprietary protocol developed by Microsoft to provide graphical interface access from one computer to another remote one. This technology is also known as Terminal Services because of the former name of Microsoft RDP server software (now called Remote Desktop Services or RDS). Windows 2008R2 introduced the new 7.0 version of the protocol at the end of the last year but the main improvements were done in the jump between 5.2 and 6.0/6.1 versions. Let's summarize the changes:

Version 6.0 (November 2006 - Windows Vista):
- Terminal Services Remote Applications. Remote Programs are a feature of Windows Server Terminal Services that lets client computers connect to a remote computer and execute programs that are installed on it instead of the complete windows desktop. The experience is the same as running a program that is installed on the local computer. An administrator must first publish the programs for end-users to access them.
- Terminal Services Gateway servers. TS Gateway server is a type of gateway that enables authorized users to connect to remote computers on a corporate network. TS Gateway uses RDP together with HTTPS protocol to help create a more secure, encrypted connection. A TS Gateway server uses port 443 and transmits data through a Secure Sockets Layer (SSL) tunnel. TS Gateway opens usual RDP intranet solution up to the world wide web.
- Improved bandwidth tuning for RDP clients.
Version 6.1 (February 2008 - Windows Server 2008):
- Terminal Services Web Access. TS Web Access is a service that makes Windows Server programs (TS RemoteApp) available to users from a Web browser. TS Web Acces is a kind of application inside IIS that lets user start remote applications from a web browser. This feature is not a protocol improvement but a new feature of the windows server and client.
- Network level server authentication. Technology that requires the user to authenticate himself before a session is established with the server. Originally the server loaded the login screen for the remote user, this used up resources on the server, and was a potential area for denial of service attacks.
Version 7.0 (October 2009 - Windows Server 2008R2 and Windows 7):
- Web Single Sign-On (SSO) and Web forms-based authentication. Web SSO makes sure that after a user is logged on, no additional passwords are required for RD Gateway, RD Web Access, RD Session Host servers and RemoteApp programs.
- Multiple improvements in bandwidth (aero glass composed desktop support, smooth fonts, audio and video redirection).
- Multiple features to Virtual Desktop Infrastructure or VDI integration (virtual desktops and pools).

So now RDP alone accomplishes all common client requirements. Remote Applications gives to the user a seamless desktop integration, TS Gateway moves RDP and Remote Applications to the internet, Web Access moves again RDP to the web and finally it seems there have been several bandwidth improvements to make a better experience (Microsoft published a good whitepaper about RDP bandwidth when 6.1 version was released).

Now the main problem is how to integrate RDP inside a portal. The easiest way is just using the TS Web Access inside an iframe (the same technique I used in the sudoku entry to show you the html design). Liferay for example has an IFrame portlet that lets you point the portlet region to another page (TS Web Access server in our case), integrating TS web application easily in your portal. But, in my opinion, TS Web Access is not the best option because it works using an ActiveX element which directly bans any OS different than Windows, any web browser different than internet explorer, any RDP client different than windows Remote Desktop Connection (RDC).

As always my solution is maybe more complex but I am sure it is more open and funny. Based on the fact that RDP connection settings can be saved in a RDP file (this way these files can be easily edited, copied, and distributed), my idea is developing a little portlet which shows to the user the remote applications configured and, when clicked, a generated rdp file is downloaded to be executed by local remote desktop client. This way the solution works in any system that has a RDP compatible application. Microsoft has free Remote Desktop Connection 7.0 for windows OSes and Microsoft Remote Desktop Connection 2.0.1 Client for Mac (it supports RDP 6.0). In a similar way I did with Skype, browser can be configured to open rdp files (application/x-rdp mime type) with the selected client-side application. Linux and other *nix OSes are another question and I will try to face this problem in a future post.

The solution is a simple JSF 2.0 portlet (deployed on liferay 6.0.2) that uses cassandra as repository to store the application data (remember I already commented this to you in a previous entry about cassandra database). Portlet implementation saves the complete rdp file (the same that is generated and exported from TS Remote Administration application) in the database. When you click the application link the rdp file is fetched from cassandra and downloaded to the client.

This time the video shows the following: on the right a Windows 2008 60-day evaluation version configured as TS Server; on the left my Windows XP with RDC 7.0 installed; using IE8 I access to liferay and search for Notepad application; after clicking on it the rdp file is downloaded from my portlet and the browser asks to open it; accepting the offer RDC client remotely opens Notepad; when a simple text file is saved on administrator desktop it can be shown on windows 2008 server. Take into account I could not install 2008R2 (cos Microsoft only distributes 64 bit version and my laptop is i386), so the server only supports protocol version 6.1. This version lacks new SSO features and TS client application does not ask for user and password cos I locally saved my credentials before and it is reusing them.

Clearly notepad is a useless program but integrating this kind of remote applications can be very very important in some environments (legacy applications, programs with expensive licenses, binaries with local limited access,...). RDP protocol and its new features open up a lot of new possibilities and portlet integration makes its management easier for end users.

Finally I want to point out the use of cassandra as repository and portletfaces bridge as portlet/JSF 2.0 joint. I already stated the worth of the first one in the commented cassandra entry but I had never talked about the second piece of software. Neil Griffin heads this java project to create a portlet bridge for the new JSF 2.0 specification. I used alpha 1 version in my skype/pidgin VoIP solution and alpha 3 in this TS portlet, and I have to mention the outstanding progress. Good job Neil!

Nolife at Nologin: La esencia del usuario

nospam@example.com (bufon) — lun, 02 ago 2010 22:00:00 +0000

Cuando uno es usuario o cliente de lo que sea (desde un restaurante o una tienda a una ventanilla pública o una consultoría), se siente demasiadas veces como el comensal del chiste de Coco...

En Nologin sabemos lo que es eso y por eso nos adaptamos a las necesidades de nuestros clientes, para que ellos no se tengan que adaptar a nosotros que bastante tienen con lo suyo. Y eso hemos intentado transmitir con nuestra nueva página web, en la que confiamos que encontréis lo que necesitáis. Si no es así, no dudéis en poneros en contacto con nosotros.

Pon un sol en tu vida: Webs de administración y puertos redirigidos

nospam@example.com (Sergio Lopez) — jue, 29 jul 2010 13:41:32 +0000

Si trabajas administrando sistemas, es posible que te hayas encontrado en la tesitura de tener que acceder a la consola de administración de algún dispositivo a través de un túnel SSH, redirigiendo algún puerto de la máquina de destino a otro local. Al hacer esto, puedes encontrarte con la circunstancia de que la aplicación a la que estás accediendo tiene referencias estáticas a su IP real, lo cual provoca que no puedas acceder correctamente a ella.

Si este es tu caso, y estás utilizando GNU/Linux, puedes hacer uso de iptables para salvar el escollo. Por ejemplo con

iptables -t nat -A OUTPUT -p tcp --dport 9000 -j DNAT --to 127.0.0.1:9000

estaríamos redigiriendo todo el tráfico saliente hacia el puerto 9000, al mismo puerto en nuestra máquina local.

Nolife at Nologin: La esencia de la consultoría informática

nospam@example.com (bufon) — lun, 26 jul 2010 22:32:36 +0000

Hay algunas características de la relación del consultor con el cliente que ponen de los nervios.

Hay unas características de la relación del informático con el cliente que sacan de quicio.

Cuando uno es consultor informático, se siente la Tía Pepa...

NOTA: Empiezo una serie de recopilatorios de videos "esenciales" con las que confío alegraros un poco el verano y creáis que estoy trabajando.

Ricky's Hodgepodge: Introducing CalDAV (Part II)

nospam@example.com (rickyepoderi) — s�b, 24 jul 2010 09:22:00 +0000

In the previous entry the CalDAV protocol was introduced. This protocol is a standard way of managing calendar and scheduling iCalendar data files as WebDAV collections. If you have read the first part the final goal was testing some CalDAV methods against three different providers (Google Calendar, Yahoo! Calendar and Calendar Server -open source version of Apple iCal Calendar Server-). The tester class was developed using jackrabbit-webdav library for WebDAV protocol and iCal4j for iCalendar data processing. In the previous post OPTIONS and PROPFIND methods were presented. The first one returns via headers if CalDAV is supported over an URI. The second method returns the properties of the same calendar CalDAV URI.

This entry continues testing CalDAV with the following methods:

Creation of a calendar VEVENT via PUT method.
Retrieving the just created VEVENT using calendar-multiget REPORT.
Querying the VEVENTs using calendar-query REPORT.
Deletion of the VEVENT.

PUT

The PUT method uploads a resource to a web server. Defined in the HTTP specification PUT sends any file inside body part, in case of CalDAV the only requirement is content was of type iCalendar (text/calendar in Content-Type header). Because a new event is going to be created I have added a If-None-Match conditional header to not perform the PUT if any (*) resource already exists in the same URI.

HttpClient client = ...
PutMethod put = null;
try {
  UUID uuid = UUID.randomUUID();
  CalendarBuilder builder = new CalendarBuilder();
  net.fortuna.ical4j.model.Calendar c = new net.fortuna.ical4j.model.Calendar();
  c.getProperties().add(new ProdId("-//Ben Fortuna//iCal4j 1.0//EN"));
  c.getProperties().add(Version.VERSION_2_0);
  c.getProperties().add(CalScale.GREGORIAN);
  TimeZoneRegistry registry = builder.getRegistry();
  VTimeZone tz = registry.getTimeZone("Europe/Madrid").getVTimeZone();
  c.getComponents().add(tz);
  VEvent vevent = new VEvent(new net.fortuna.ical4j.model.Date(),
    new Dur(0, 1, 0, 0), "test");
  vevent.getProperties().add(new Uid(uuid.toString()));
  c.getComponents().add(vevent);
  String href = uri + uuid.toString() + ".ics";
  put = new PutMethod(href);
  put.addRequestHeader("If-None-Match", "*");
  put.setRequestEntity(new StringRequestEntity(c.toString(), "text/calendar", "UTF-8"));
  client.executeMethod(put);
  return href;
} finally {
  if (put != null) {
    put.releaseConnection();
  }
}

This code creates a new iCal4j calendar and assigns timezone and a event to it. After that a PUT method is constructed and the previously iCalendar object is set as its request. In WebDAV (and CalDAV) the client side is the responsible of choosing a name (the URI part) to the resource, here I have used a UUID to generate a random and different href URI.

As PUT method is almost the same in the three server implementations only Google Calendar trace is presented.

Google Calendar:

PUT /calendar/dav/rickyepoderi@gmail.com/events/fca0402c-f00a-493c-9b48-8cec2a203728.ics HTTP/1.1
If-None-Match: *
User-Agent: Jakarta Commons-HttpClient/3.1
Content-Length: 3041
Content-Type: text/calendar; charset=UTF-8
Authorization: Basic XXXXXXXXXX
Host: www.google.com

BEGIN:VCALENDAR
PRODID:-//Ben Fortuna//iCal4j 1.0//EN
VERSION:2.0
CALSCALE:GREGORIAN
BEGIN:VTIMEZONE
TZID:Europe/Madrid
TZURL:http://tzurl.org/zoneinfo/Europe/Madrid
X-LIC-LOCATION:Europe/Madrid
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:19810329T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:19961027T030000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
END:STANDARD
BEGIN:STANDARD
TZOFFSETFROM:-001444
TZOFFSETTO:+0000
TZNAME:WET
DTSTART:19010101T011444
RDATE:19010101T011444
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:+0000
TZOFFSETTO:+0100
TZNAME:WEST
DTSTART:19170506T000000
RDATE:19170506T000000
RDATE:19180416T000000
RDATE:19190406T000000
RDATE:19240417T000000
RDATE:19260418T000000
RDATE:19270410T000000
RDATE:19280415T000000
RDATE:19290421T000000
RDATE:19370523T000000
RDATE:19380323T000000
RDATE:19390416T000000
RDATE:19400317T000000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0100
TZOFFSETTO:+0000
TZNAME:WET
DTSTART:19171007T010000
RDATE:19171007T010000
RDATE:19181007T010000
RDATE:19191007T010000
RDATE:19241005T010000
RDATE:19261003T010000
RDATE:19271002T010000
RDATE:19281007T010000
RDATE:19291006T010000
RDATE:19371003T010000
RDATE:19381002T010000
RDATE:19391008T010000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:WEMT
DTSTART:19420503T000000
RDATE:19420503T000000
RDATE:19430418T000000
RDATE:19440416T000000
RDATE:19450415T000000
RDATE:19460414T000000
END:DAYLIGHT
BEGIN:DAYLIGHT
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:WEST
DTSTART:19420902T010000
RDATE:19420902T010000
RDATE:19431004T010000
RDATE:19441011T010000
RDATE:19450930T020000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:19460930T000000
RDATE:19460930T000000
RDATE:19490930T010000
RDATE:19741006T010000
RDATE:19751005T010000
RDATE:19760926T010000
RDATE:19770925T010000
RDATE:19781001T010000
RDATE:19790930T030000
RDATE:19800928T030000
RDATE:19810927T030000
RDATE:19820926T030000
RDATE:19830925T030000
RDATE:19840930T030000
RDATE:19850929T030000
RDATE:19860928T030000
RDATE:19870927T030000
RDATE:19880925T030000
RDATE:19890924T030000
RDATE:19900930T030000
RDATE:19910929T030000
RDATE:19920927T030000
RDATE:19930926T030000
RDATE:19940925T030000
RDATE:19950924T030000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:19490430T230000
RDATE:19490430T230000
RDATE:19740413T230000
RDATE:19750419T230000
RDATE:19760327T230000
RDATE:19770402T230000
RDATE:19780402T230000
RDATE:19790401T020000
RDATE:19800406T020000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0100
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:19790101T000000
RDATE:19790101T000000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20100709T080233Z
DTSTART;VALUE=DATE:20100709
DURATION:PT1H
SUMMARY:test
UID:fca0402c-f00a-493c-9b48-8cec2a203728
END:VEVENT
END:VCALENDAR

HTTP/1.1 201 Created
DAV: 1, calendar-access, calendar-schedule, calendar-proxy
Content-Type: text/calendar; component=vevent; charset=UTF-8
Date: Fri, 09 Jul 2010 08:02:35 GMT
Expires: Fri, 09 Jul 2010 08:02:35 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Content-Length: 0
Server: GSE

CALENDAR-MULTIGET REPORT

The first real CalDAV method. A REPORT is an extensible mechanism for obtaining information about resources defined in RFC-3253. The value of a report can depend on additional information specified in the REPORT request body and in the REPORT request headers. CalDAV specification defines some of them and calendar-multiget is the first one to be tested. It returns the specified data (any CalDAV report lets us define the iCalendar information that is going to be returned) for a list of resources (a list of URIs that represents iCalendar resources). As any other WebDAV method calendar-multiget has a standard XML request (defined via DTD) that is sent to the server. The response (like many other reports) is a multi-status response (a list of resource responses that match the report criteria with the properties requested attached).

So, first of all, I developed some classes that implement all the XML stuff (properties and report itself). All this simple classes are inside a sample.caldav.xml package and they use the W3C DOM XML parser (jackrabbit-webdav already uses it). Besides jackrabbit-webdav has a ReportMethod class to execute any report against a WebDAV server, this class uses another one (ReportInfo) that defines the report to execute. The problem here is jackrabbit assumes all properties to return are specified using only names (simple XML tags like <getetag/> or <getctag/>) and CalDAV properties are more complicated (see the XML classes of the package presented before). For this reason my only chance was extending ReportInfo with a CalendarMultiGetReportInfo.java that constructs the correct XML elements for this report (toXml method). One thing I did not understand completely was why jackrabbit needs all the reports to be registered inside ReportType class, this registration uses a Report class which I did not see where is used. For this reason I created and registered an empty report CalendarMultiGetReport.java. A class for constants called CalDavConstants.java is also used in a similar way jackrabbit uses DavConstants.java. The final code to execute the report that retrieves the previously created event is the following.

HttpClient client = ...
ReportMethod report = null;
try {
  // append props GETTETAG
  DavPropertyNameSet props = new DavPropertyNameSet();
  props.add(DavPropertyName.GETETAG);
  // append calendar-data
  RequestCalendarData calendarData = new RequestCalendarData();
  Comp vcalendar = new Comp("VCALENDAR");
  vcalendar.getProp().add(new Prop("VERSION"));
  vcalendar.getComp().add(new Comp("VEVENT"));
  vcalendar.getComp().get(0).getProp().add(new Prop("SUMMARY"));
  vcalendar.getComp().get(0).getProp().add(new Prop("UID"));
  vcalendar.getComp().get(0).getProp().add(new Prop("DTSTART"));
  vcalendar.getComp().get(0).getProp().add(new Prop("DTEND"));
  vcalendar.getComp().get(0).getProp().add(new Prop("DURATION"));
  vcalendar.getComp().get(0).getProp().add(new Prop("RRULE"));
  vcalendar.getComp().get(0).getProp().add(new Prop("RDATE"));
  vcalendar.getComp().get(0).getProp().add(new Prop("EXRULE"));
  vcalendar.getComp().get(0).getProp().add(new Prop("EXDATE"));
  vcalendar.getComp().get(0).getProp().add(new Prop("RECURRENCE-ID"));
  vcalendar.getComp().add(new Comp("VTIMEZONE"));
  calendarData.setComp(vcalendar);
  // create the report
  ReportInfo reportInfo = new CalendarMultiGetReportInfo(props,
    calendarData, new String[]{createdHref});
  report = new ReportMethod(uri, reportInfo);
  client.executeMethod(report);
  MultiStatus multiStatus = report.getResponseBodyAsMultiStatus();
  for (int i = 0; i multiStatus.getResponses().length; i++) {
    MultiStatusResponse multiRes = multiStatus.getResponses()[i];
    String href = multiRes.DavPropertySet propSet = multiRes.getProperties(
      DavServletResponse.SC_OK);
    DavProperty prop = (DavProperty) propSet.get(
      CalDavConstants.CALDAV_XML_CALENDAR_DATA, CalDavConstants.CALDAV_NAMESPACE);
    System.err.println("HREF: " + href);
    CalendarBuilder builder = new CalendarBuilder();
    net.fortuna.ical4j.model.Calendar c = builder.build(
      new StringReader(prop.getValue()));
    System.err.println("calendar-data: " + c.toString());
  }
} finally {
  if (report != null) {
    report.releaseConnection();
  }
}

As you see the example requests for VCALENDAR information (only property VERSION) and, inside it, the VEVENT (with only some properties) and the VTIMEZONE (with all properties). The HREF part must be the one returned in the creation part. I added a ETag (Entity Tag) property as a non CalDAV property to be retrieved. Studying the below communication traces there are some curious results. First open source Calendar Server follows the guidelines and only returns the requested props but google and yahoo! always return all calendar data. And second only Calendar Server returns exactly the same data put before, google and yahoo! have modified the iCalendar resource (some kind of pre-processing before storing it).

Google Calendar:

REPORT /calendar/dav/rickyepoderi@gmail.com/events/ HTTP/1.1
Depth: 0
User-Agent: Jakarta Commons-HttpClient/3.1
Content-Length: 673
Content-Type: text/xml; charset=UTF-8
Authorization: Basic XXXXXXXX
Host: www.google.com

<?xml version="1.0" encoding="UTF-8"?>
<C:calendar-multiget xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:D="DAV:">
  <D:prop>
    <D:getetag/>
    <C:calendar-data>
      <C:comp name="VCALENDAR">
        <C:prop name="VERSION"/>
          <C:comp name="VEVENT">
            <C:prop name="SUMMARY"/>
            <C:prop name="UID"/>
            <C:prop name="DTSTART"/>
            <C:prop name="DTEND"/>
            <C:prop name="DURATION"/>
            <C:prop name="RRULE"/>
            <C:prop name="RDATE"/>
            <C:prop name="EXRULE"/>
            <C:prop name="EXDATE"/>
            <C:prop name="RECURRENCE-ID"/>
          </C:comp>
        <C:comp name="VTIMEZONE"/>
      </C:comp>
    </C:calendar-data>
  </D:prop>
<D:href>https://www.google.com/calendar/dav/rickyepoderi@gmail.com/events/fca0402c-f00a-493c-9b48-8cec2a203728.ics</D:href>
</C:calendar-multiget>

HTTP/1.1 207 Multi-Status
DAV: 1, calendar-access, calendar-schedule, calendar-proxy
Content-Type: application/xml; charset=UTF-8
Date: Fri, 09 Jul 2010 08:02:36 GMT
Expires: Fri, 09 Jul 2010 08:02:36 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Content-Length: 1059
Server: GSE

<?xml version="1.0" encoding="UTF-8"?>
<D:multistatus xmlns:D="DAV:">
  <D:response>
    <D:href>https://www.google.com/calendar/dav/rickyepoderi@gmail.com/events/fca0402c-f00a-493c-9b48-8cec2a203728.ics</D:href>
    <D:propstat>
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop>
        <D:getetag>"63414345755"</D:getetag>
        <C:calendar-data xmlns:C="urn:ietf:params:xml:ns:caldav">BEGIN:VCALENDAR
PRODID:-//Google Inc//Google Calendar 70.9054//EN
VERSION:2.0
CALSCALE:GREGORIAN
X-WR-CALNAME:Ricardo Martin
X-WR-TIMEZONE:Europe/Paris
BEGIN:VEVENT
DTSTART:20100708T220000Z
DTEND:20100708T230000Z
DTSTAMP:20100709T080235Z
UID:fca0402c-f00a-493c-9b48-8cec2a203728
CREATED:20100709T080235Z
DESCRIPTION:
LAST-MODIFIED:20100709T080235Z
LOCATION:
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:test
TRANSP:OPAQUE
BEGIN:VALARM
ACTION:DISPLAY
DESCRIPTION:This is an event reminder
TRIGGER:-P0DT0H10M0S
END:VALARM
END:VEVENT
END:VCALENDAR</C:calendar-data>
      </D:prop>
    </D:propstat>
  </D:response>
</D:multistatus>

Yahoo! Calendar:

REPORT /dav/rickyepoderi@yahoo.es/Calendar/Ricardo_Martin/ HTTP/1.1
Depth: 0[\n]
User-Agent: Jakarta Commons-HttpClient/3.1
Content-Length: 691
Content-Type: text/xml; charset=UTF-8
Authorization: Basic XXXXXXXXXX
Host: caldav.calendar.yahoo.com

<?xml version="1.0" encoding="UTF-8"?>
<C:calendar-multiget xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:D="DAV:">
  <D:prop>
    <D:getetag/>
      <C:calendar-data>
      <C:comp name="VCALENDAR">
        <C:prop name="VERSION"/>
        <C:comp name="VEVENT">
          <C:prop name="SUMMARY"/>
          <C:prop name="UID"/>
          <C:prop name="DTSTART"/>
          <C:prop name="DTEND"/>
          <C:prop name="DURATION"/>
          <C:prop name="RRULE"/>
          <C:prop name="RDATE"/>
          <C:prop name="EXRULE"/>
          <C:prop name="EXDATE"/>
          <C:prop name="RECURRENCE-ID"/>
        </C:comp>
        <C:comp name="VTIMEZONE"/>
      </C:comp>
    </C:calendar-data>
  </D:prop>
  <D:href>https://caldav.calendar.yahoo.com/dav/rickyepoderi@yahoo.es/Calendar/Ricardo_Martin/b1e1683b-08d5-4ca3-b537-ed6a0c5867b9.ics</D:href>
</C:calendar-multiget>

HTTP/1.1 207 Multi Status
Date: Fri, 09 Jul 2010 09:39:46 GMT
P3P: policyref="http://info.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV"
DAV: 1, 2, 3, access-control, calendar-access, calendar-schedule
DAV: version-control, addressbook, extended-mkcol, calendar-proxy
DAV: calendarserver-principal-property-search
Content-Type: text/xml; charset=utf-8
X-Cache: MISS from store138.c104.cal.gq1.yahoo.com
Age: 0
Transfer-Encoding: chunked
Connection: keep-alive
Server: YTS/1.17.23

<?xml version="1.0" encoding="UTF-8"?>
<D:multistatus xmlns:D="DAV:">
  <D:response>
    <D:href>/dav/rickyepoderi@yahoo.es/Calendar/Ricardo_Martin/b1e1683b-08d5-4ca3-b537-ed6a0c5867b9.ics</D:href>
      <D:propstat>
        <D:status>HTTP/1.1 200 OK</D:status>
        <D:prop>
          <D:getetag>"400-400"</D:getetag>
          <C:calendar-data xmlns:C="urn:ietf:params:xml:ns:caldav">BEGIN:VCALENDAR
VERSION:2.0
PRODID:Zimbra-Calendar-Provider
BEGIN:VEVENT
UID:b1e1683b-08d5-4ca3-b537-ed6a0c5867b9
SUMMARY:test
DTSTART;VALUE=DATE:20100709
DURATION:PT1H
STATUS:CONFIRMED
CLASS:PUBLIC
X-MICROSOFT-CDO-ALLDAYEVENT:TRUE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
TRANSP:OPAQUE
X-MICROSOFT-DISALLOW-COUNTER:TRUE
DTSTAMP:20100709T093944Z
SEQUENCE:0
END:VEVENT
END:VCALENDAR</C:calendar-data>
      </D:prop>
    </D:propstat>
  </D:response>
</D:multistatus>

Calendar Server:

REPORT /calendars/users/admin/calendar/ HTTP/1.1
Depth: 0
User-Agent: Jakarta Commons-HttpClient/3.1
Content-Length: 660
Content-Type: text/xml; charset=UTF-8
Authorization: Digest username="admin", realm="Test Realm", nonce="64064457517812303241909094429", uri="/calendars/users/admin/calendar/", response="3fc6f79c4b2c8ef0c60e415429fc864e", algorithm="md5"
Host: localhost:8008

<?xml version="1.0" encoding="UTF-8"?>
<C:calendar-multiget xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:D="DAV:">
  <D:prop>
    <D:getetag/>
    <C:calendar-data>
      <C:comp name="VCALENDAR">
        <C:prop name="VERSION"/>
        <C:comp name="VEVENT">
          <C:prop name="SUMMARY"/>
          <C:prop name="UID"/>
          <C:prop name="DTSTART"/>
          <C:prop name="DTEND"/>
          <C:prop name="DURATION"/>
          <C:prop name="RRULE"/>
          <C:prop name="RDATE"/>
          <C:prop name="EXRULE"/>
          <C:prop name="EXDATE"/>
          <C:prop name="RECURRENCE-ID"/>
        </C:comp>
        <C:comp name="VTIMEZONE"/>
      </C:comp>
    </C:calendar-data>
  </D:prop>
  <D:href>http://localhost:8008/calendars/users/admin/calendar/8fbfb266-6f6b-4201-b51a-a75416640a25.ics</D:href>
</C:calendar-multiget>

HTTP/1.1 207 Multi-Status
Content-Length: 3457
Accept-Ranges: bytes
Server: CalendarServer/trunk(r5842M) Twisted/10.0.0 TwistedWeb/9.0.0+r5842
Last-Modified: Fri, 09 Jul 2010 09:49:08 GMT
DAV: 1, access-control, calendar-access, calendar-schedule, calendar-auto-schedule, calendar-availability, inbox-availability, calendar-proxy, calendarserver-private-events, calendarserver-private-comments, calendarserver-sharing, addressbook, calendarserver-principal-property-search
ETag: W/"285DD2-1000-4C36F094
Date: Fri, 09 Jul 2010 09:49:08 GMT
Content-Type: text/xml

<?xml version='1.0' encoding='UTF-8'?>
<multistatus xmlns='DAV:'>
  <response>
    <href>/calendars/users/admin/calendar/8fbfb266-6f6b-4201-b51a-a75416640a25.ics</href>
    <propstat>
      <prop>
        <getetag>"234cd0ad246d14829d189feca8c0eead"</getetag>
        <calendar-data xmlns='urn:ietf:params:xml:ns:caldav'><![CDATA[BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//PYVOBJECT//NONSGML Version 1//EN
BEGIN:VTIMEZONE
TZID:Europe/Madrid
TZURL:http://tzurl.org/zoneinfo/Europe/Madrid
BEGIN:STANDARD
DTSTART:19961027T030000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:STANDARD
DTSTART:19010101T011444
RDATE:19010101T011444
TZNAME:WET
TZOFFSETFROM:-001444
TZOFFSETTO:+0000
END:STANDARD
BEGIN:STANDARD
DTSTART:19171007T010000
RDATE:19171007T010000
RDATE:19181007T010000
RDATE:19191007T010000
RDATE:19241005T010000
RDATE:19261003T010000
RDATE:19271002T010000
RDATE:19281007T010000
RDATE:19291006T010000
RDATE:19371003T010000
RDATE:19381002T010000
RDATE:19391008T010000
TZNAME:WET
TZOFFSETFROM:+0100
TZOFFSETTO:+0000
END:STANDARD
BEGIN:STANDARD
DTSTART:19460930T000000
RDATE:19460930T000000
RDATE:19490930T010000
RDATE:19741006T010000
RDATE:19751005T010000
RDATE:19760926T010000
RDATE:19770925T010000
RDATE:19781001T010000
RDATE:19790930T030000
RDATE:19800928T030000
RDATE:19810927T030000
RDATE:19820926T030000
RDATE:19830925T030000
RDATE:19840930T030000
RDATE:19850929T030000
RDATE:19860928T030000
RDATE:19870927T030000
RDATE:19880925T030000
RDATE:19890924T030000
RDATE:19900930T030000
RDATE:19910929T030000
RDATE:19920927T030000
RDATE:19930926T030000
RDATE:19940925T030000
RDATE:19950924T030000
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:STANDARD
DTSTART:19790101T000000
RDATE:19790101T000000
TZNAME:CET
TZOFFSETFROM:+0100
TZOFFSETTO:+0100
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:19810329T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:19170506T000000
RDATE:19170506T000000
RDATE:19180416T000000
RDATE:19190406T000000
RDATE:19240417T000000
RDATE:19260418T000000
RDATE:19270410T000000
RDATE:19280415T000000
RDATE:19290421T000000
RDATE:19370523T000000
RDATE:19380323T000000
RDATE:19390416T000000
RDATE:19400317T000000
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:19420503T000000
RDATE:19420503T000000
RDATE:19430418T000000
RDATE:19440416T000000
RDATE:19450415T000000
RDATE:19460414T000000
TZNAME:WEMT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:19420902T010000
RDATE:19420902T010000
RDATE:19431004T010000
RDATE:19441011T010000
RDATE:19450930T020000
TZNAME:WEST
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:19490430T230000
RDATE:19490430T230000
RDATE:19740413T230000
RDATE:19750419T230000
RDATE:19760327T230000
RDATE:19770402T230000
RDATE:19780402T230000
RDATE:19790401T020000
RDATE:19800406T020000
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
X-LIC-LOCATION:Europe/Madrid
END:VTIMEZONE
BEGIN:VEVENT
UID:8fbfb266-6f6b-4201-b51a-a75416640a25
DTSTART;VALUE=DATE:20100709
DURATION:PT1H
SUMMARY:test
END:VEVENT
END:VCALENDAR
]]></calendar-data>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
</multistatus>

CALENDAR-QUERY REPORT

Calendar-query is other report defined in CalDAV to request calendar entries that fulfill a specified filter (entries between dates, iCalendar data with a specified value in a field and so on). In this case the same package presented before manages the XML stuff and, following the same idea, a CalendarQueryReportInfo.java extension and CalendarQueryReport.java was coded. First class is the ReportInfo for this new report and the second one does the registration part.

HttpClient client = ...
ReportMethod calendarQuery = null;
try {
  RequestCalendarData calendarData = new RequestCalendarData();
  Filter filter = new Filter("VCALENDAR");
  filter.getCompFilter().add(new Filter("VEVENT"));
  Calendar start = Calendar.getInstance();
  start.add(Calendar.MONTH, -1);
  Calendar end = Calendar.getInstance();
  filter.getCompFilter().get(0).setTimeRange(
    new TimeRange(start.getTime(), end.getTime()));
  ReportInfo reportInfo = new CalendarQueryReportInfo(calendarData, filter);
  calendarQuery = new ReportMethod(uri, reportInfo);
  client.executeMethod(calendarQuery);
  MultiStatus multiStatus = calendarQuery.getResponseBodyAsMultiStatus();
  for (int i = 0; i multiStatus.getResponses().length; i++) {
    MultiStatusResponse multiRes = multiStatus.getResponses()[i];
    String href = multiRes.getHref();
    DavPropertySet propSet = multiRes.getProperties(DavServletResponse.SC_OK);
    DavProperty prop = (DavProperty) propSet.get(
      CalDavConstants.CALDAV_XML_CALENDAR_DATA, CalDavConstants.CALDAV_NAMESPACE);
    System.err.println("HREF: " + href);
    CalendarBuilder builder = new CalendarBuilder();
    net.fortuna.ical4j.model.Calendar c = builder.build(
      new StringReader(prop.getValue()));
    System.err.println("calendar-data: " + c.toString());
  }
} finally {
  if (calendarQuery != null) {
    calendarQuery.releaseConnection();
  }
}

The sample report performs a query with a filter that returns all the events in a month. In this case all calendar data is requested.

Google Calendar:

REPORT /calendar/dav/rickyepoderi@gmail.com/events/ HTTP/1.1
Depth: 1
User-Agent: Jakarta Commons-HttpClient/3.1
Content-Length: 342
Content-Type: text/xml; charset=UTF-8
Authorization: Basic XXXXXXXXXX
Host: www.google.com

<?xml version="1.0" encoding="UTF-8"?>
<C:calendar-query xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:D="DAV:">
  <D:prop>
    <C:calendar-data/>
  </D:prop>
  <C:filter>
    <C:comp-filter name="VCALENDAR">
      <C:comp-filter name="VEVENT">
        <C:time-range end="20100709T101456Z" start="20100609T101456Z"/>
      </C:comp-filter>
    </C:comp-filter>
  </C:filter>
</C:calendar-query>

HTTP/1.1 207 Multi-Status
DAV: 1, calendar-access, calendar-schedule, calendar-proxy
Content-Type: application/xml; charset=UTF-8
Date: Fri, 09 Jul 2010 10:14:57 GMT
Expires: Fri, 09 Jul 2010 10:14:57 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Content-Length: 993
Server: GSE

<?xml version="1.0" encoding="UTF-8"?>
<D:multistatus xmlns:D="DAV:">
  <D:response>
    <D:href>/calendar/dav/rickyepoderi%40gmail.com/events/3fb8047e-4a51-4207-90a9-629e8bed1866.ics</D:href>
    <D:propstat>
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop>
        <C:calendar-data xmlns:C="urn:ietf:params:xml:ns:caldav">BEGIN:VCALENDAR
PRODID:-//Google Inc//Google Calendar 70.9054//EN
VERSION:2.0
CALSCALE:GREGORIAN
X-WR-CALNAME:Ricardo Martin
X-WR-TIMEZONE:Europe/Paris
BEGIN:VEVENT
DTSTART:20100708T220000Z
DTEND:20100708T230000Z
DTSTAMP:20100709T101456Z
UID:3fb8047e-4a51-4207-90a9-629e8bed1866
CREATED:20100709T101456Z
DESCRIPTION:
LAST-MODIFIED:20100709T101456Z
LOCATION:
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:test
RANSP:OPAQUE
BEGIN:VALARM
ACTION:DISPLAY
DESCRIPTION:This is an event reminder
TRIGGER:-P0DT0H10M0S
END:VALARM
END:VEVENT
END:VCALENDAR</C:calendar-data>
      </D:prop>
    </D:propstat>
  </D:response>
</D:multistatus>

Yahoo! Calendar:

REPORT /dav/rickyepoderi@yahoo.es/Calendar/Ricardo_Martin/ HTTP/1.1
Depth: 1
User-Agent: Jakarta Commons-HttpClient/3.1
Content-Length: 342
Content-Type: text/xml; charset=UTF-8
Authorization: Basic XXXXXXXXXXXXX
Host: caldav.calendar.yahoo.com

<?xml version="1.0" encoding="UTF-8"?>
<C:calendar-query xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:D="DAV:">
  <D:prop>
    <C:calendar-data/>
  </D:prop>
  <C:filter>
    <C:comp-filter name="VCALENDAR">
      <C:comp-filter name="VEVENT">
        <C:time-range end="20100709T102151Z" start="20100609T102151Z"/>
      </C:comp-filter>
    </C:comp-filter>
  </C:filter>
</C:calendar-query>

HTTP/1.1 207 Multi Status
Date: Fri, 09 Jul 2010 10:21:52 GMT
P3P: policyref="http://info.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV"
DAV: 1, 2, 3, access-control, calendar-access, calendar-schedule
DAV: version-control, addressbook, extended-mkcol, calendar-proxy
DAV: calendarserver-principal-property-search
Content-Type: text/xml; charset=utf-8
X-Cache: MISS from store138.c104.cal.gq1.yahoo.com
Age: 0
Transfer-Encoding: chunked
Connection: keep-alive
Server: YTS/1.17.23

<?xml version="1.0" encoding="UTF-8"?>
<D:multistatus xmlns:D="DAV:">
  <D:response>
    <D:href>/dav/rickyepoderi@yahoo.es/Calendar/Ricardo_Martin/00584253-ab0d-41c0-b1fe-ee391eab6f78.ics</D:href>
    <D:propstat>[\n]
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop>
        <C:calendar-data xmlns:C="urn:ietf:params:xml:ns:caldav">BEGIN:VCALENDAR
VERSION:2.0
PRODID:Zimbra-Calendar-Provider
BEGIN:VEVENT
UID:00584253-ab0d-41c0-b1fe-ee391eab6f78
SUMMARY:test
DTSTART;VALUE=DATE:20100709
DURATION:PT1H
STATUS:CONFIRMED
CLASS:PUBLIC
X-MICROSOFT-CDO-ALLDAYEVENT:TRUE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
TRANSP:OPAQUE
X-MICROSOFT-DISALLOW-COUNTER:TRUE
DTSTAMP:20100709T102149Z
SEQUENCE:0
END:VEVENT
END:VCALENDAR</C:calendar-data>
      </D:prop>
    </D:propstat>
  </D:response>
</D:multistatus>

Calendar Server:

REPORT /calendars/users/admin/calendar/ HTTP/1.1
Depth: 1
User-Agent: Jakarta Commons-HttpClient/3.1
Content-Length: 342
Content-Type: text/xml; charset=UTF-8
Authorization: Digest username="admin", realm="Test Realm", nonce="11647135610574514491121002269", uri="/calendars/users/admin/calendar/", response="9fc57dc0becf313811f762083aaefe7c", algorithm="md5"
Host: localhost:8008

<?xml version="1.0" encoding="UTF-8"?>
<C:calendar-query xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:D="DAV:">
  <D:prop>
    <C:calendar-data/>
  </D:prop>
  <C:filter>
    <C:comp-filter name="VCALENDAR">
      <C:comp-filter name="VEVENT">
        <C:time-range end="20100709T103056Z" start="20100609T103056Z"/>
      </C:comp-filter>
    </C:comp-filter>
  </C:filter>
</C:calendar-query>

HTTP/1.1 207 Multi-Status
Content-Length: 8512
Accept-Ranges: bytes
Server: CalendarServer/trunk(r5842M) Twisted/10.0.0 TwistedWeb/9.0.0+r5842
Last-Modified: Fri, 09 Jul 2010 10:30:55 GMT
DAV: 1, access-control, calendar-access, calendar-schedule, calendar-auto-schedule, calendar-availability, inbox-availability, calendar-proxy, calendarserver-private-events, calendarserver-private-comments, calendarserver-sharing, addressbook, calendarserver-principal-property-search
ETag: "285DD2-1000-4C36FA5F"
Date: Fri, 09 Jul 2010 10:30:56 GMT
Content-Type: text/xml[\r][\n]

<"?xml version='1.0' encoding='UTF-8'?>
<multistatus xmlns='DAV:'>
  <response>
    <href>/calendars/users/admin/calendar/2b1785aa-0893-49b6-be52-ede2290bdc4a.ics</href>
    <propstat>
      <prop>
        <calendar-data xmlns='urn:ietf:params:xml:ns:caldav'><![CDATA[BEGIN:VCALENDAR
VERSION:2.0
CALSCALE:GREGORIAN
PRODID:-//Ben Fortuna//iCal4j 1.0//EN
BEGIN:VTIMEZONE
TZID:Europe/Madrid
TZURL:http://tzurl.org/zoneinfo/Europe/Madrid
BEGIN:STANDARD
DTSTART:19961027T030000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:STANDARD
DTSTART:19010101T011444
RDATE:19010101T011444
TZNAME:WET
TZOFFSETFROM:-001444
TZOFFSETTO:+0000
END:STANDARD
BEGIN:STANDARD
DTSTART:19171007T010000
RDATE:19171007T010000
RDATE:19181007T010000
RDATE:19191007T010000
RDATE:19241005T010000
RDATE:19261003T010000
RDATE:19271002T010000
RDATE:19281007T010000
RDATE:19291006T010000
RDATE:19371003T010000
RDATE:19381002T010000
RDATE:19391008T010000
TZNAME:WET
TZOFFSETFROM:+0100
TZOFFSETTO:+0000
END:STANDARD
BEGIN:STANDARD
DTSTART:19460930T000000
RDATE:19460930T000000
RDATE:19490930T010000
RDATE:19741006T010000
RDATE:19751005T010000
RDATE:19760926T010000
RDATE:19770925T010000
RDATE:19781001T010000
RDATE:19790930T030000
RDATE:19800928T030000
RDATE:19810927T030000
RDATE:19820926T030000
RDATE:19830925T030000
RDATE:19840930T030000
RDATE:19850929T030000
RDATE:19860928T030000
RDATE:19870927T030000
RDATE:19880925T030000
RDATE:19890924T030000
RDATE:19900930T030000
RDATE:19910929T030000
RDATE:19920927T030000
RDATE:19930926T030000
RDATE:19940925T030000
RDATE:19950924T030000
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:STANDARD
DTSTART:19790101T000000
RDATE:19790101T000000
TZNAME:CET
TZOFFSETFROM:+0100
TZOFFSETTO:+0100
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:19810329T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:19170506T000000
RDATE:19170506T000000
RDATE:19180416T000000
RDATE:19190406T000000
RDATE:19240417T000000
RDATE:19260418T000000
RDATE:19270410T000000
RDATE:19280415T000000
RDATE:19290421T000000
RDATE:19370523T000000
RDATE:19380323T000000
RDATE:19390416T000000
RDATE:19400317T000000
TZNAME:WEST
TZOFFSETFROM:+0000
TZOFFSETTO:+0100
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:19420503T000000
RDATE:19420503T000000
RDATE:19430418T000000
RDATE:19440416T000000
RDATE:19450415T000000
RDATE:19460414T000000
TZNAME:WEMT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:19420902T010000
RDATE:19420902T010000
RDATE:19431004T010000
RDATE:19441011T010000
RDATE:19450930T020000
TZNAME:WEST
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:DAYLIGHT
BEGIN:DAYLIGHT
DTSTART:19490430T230000
RDATE:19490430T230000
RDATE:19740413T230000
RDATE:19750419T230000
RDATE:19760327T230000
RDATE:19770402T230000
RDATE:19780402T230000
RDATE:19790401T020000
RDATE:19800406T020000
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
X-LIC-LOCATION:Europe/Madrid
END:VTIMEZONE
BEGIN:VEVENT
UID:2b1785aa-0893-49b6-be52-ede2290bdc4a
DTSTART;VALUE=DATE:20100709
DURATION:PT1H
DTSTAMP:20100709T103055Z
SUMMARY:test
END:VEVENT
END:VCALENDAR
]]></calendar-data>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
</multistatus>

DELETE

Finally a usual HTTP DELETE method is sent. With it the resource previously created is removed. As in the PUT method the three servers act more or less the same way and only google calendar trace is shown.

HttpClient client = ...
DeleteMethod delete = null;
try {
  delete = new DeleteMethod(createdHref);
  client.executeMethod(delete);
} finally {
  if (delete != null) {
    delete.releaseConnection();
  }
}

A DELETE method returns No Content (HTTP code 204) on success.

Google Calendar:

DELETE /calendar/dav/rickyepoderi@gmail.com/events/cf3bdc67-df96-46bf-a0a8-86cd72133097.ics HTTP/1.1
User-Agent: Jakarta Commons-HttpClient/3.1
Content-Length: 0Authorization: Basic XXXXXXXXXX
Host: www.google.com

HTTP/1.1 204 No Content
DAV: 1, calendar-access, calendar-schedule, calendar-proxy
Date: Fri, 09 Jul 2010 10:49:33 GMT
Server: GSE

This little Main.java is the client class I have used to obtain the communication traces presented in this entries. And, as a conclusion, jackrabbit-webdav and iCal4j have done a good job. Although caldav4j is a more high level CalDAV API and I am sure it can produce faster results, the chosen libraries let us see some very interesting communication and implementation details. I hope the goal of this two-part series has been achieved, getting some little knowledge about CalDAV access programming.

Have a nice weekend!