Sunday, April 30. 2017
Loading PEM CRLs is slow in java
The previous week I was working with CRLs files and a very big PEM CRL bundle was taking several minutes to being parsed using Java. The complete file was several megabytes in size but the fact did not justify such a long time. A Certificate Revocation List (CRL) is a bunch of certificates issued by a Certificate Authority (CA) that have been revoked (the certificate is not valid anymore). Usually a CRL is kilobytes in size, when a revoked certificate expires it is usually removed from the CRL (it is invalid anyway), and this practice makes the CRL size be moderate. The revocation info is just the certificate serial number and the date, so thousand and thousands of them are needed to have such a big file.
Looking at the code the method to parse the CRLs is generateCRLs. In my case the file was a list of PEM encoded CRLs (PEM is base64 DER format and enclosed between a header -----BEGIN X509 CRL----- and a footer -----END X509 CRL-----, DER format is a binary format using ASN1). The X509Certificate implementation accepted both formats (DER and PEM) but it seems to do something very weird to deal with the PEM format (you can see the code in OpenJDK). The DER format is read directly, but PEM is first moved into a memory buffer, the base64 data is decoded and the resulting DER is parsed. The buffer is initially 2KB in size and it grows by chunks of 1K (the Arrays.copyOf method is used, so a new array is allocated and the data copied again and again). This is not a bad idea if the CRL is several kylobytes in size but, if it has several megas, the process is extremely slow.
Obviously our decision was transforming our CRL file from PEM to DER format. The load took just a few seconds instead of the several minutes it was taking before. Nevertheless I spent some time trying to improve the performance of the java method. Now java8 has standard Base64 encoder and decoder and, more important, it has a method to directly read from an InputStream. This way the extra buffer can be avoided and it always read from the file directly. I tried my idea and it works. I opened a bug against the java bug database and it is public now. My patch is there if you need it.
With the modification an enormous CRL file of 30MB (around one million revocations) gives the following times:
- DER: 2 seconds.
- PEM (patched): 14 seconds
- PEM (before): 379 seconds
So, the DER file is always faster (there is no Base64 decoding) but, at least, now the PEM process time is in the same scale. If you need to manage CRLs files in java, please, never use the PEM format, always manage DER native format. I do not know if my patch will be finally included but, it does not matter, it just improves the situation, DER will always be faster and with a smaller footprint.
Regards!
Sunday, April 9. 2017
ircstatus: A pidgin plugin to change your nick in IRC
I have moved to another job again (I know, I can not stand still) and in my new company the old IRC is used. Besides all the people inform their status changing their nicks (for example if you go to lunch, your nick username would be modified to username|lunch or something similar). I have to say that it is a mess but... When in Rome do as the Romans do. Pidgin is my favorite program for chatting. It is a very mature program and it saved my ass once in the past so I configured it to join to the different rooms I have to be present. Some plugins were indeed interesting and I also configured them to make my job easier. Nevertheless I did not find anyone to change my nick when I change my general pidgin status.
The ircaway plugin does a similar job, it changes your nick to usernameAway when the IRC status is changed to away. Indeed it would have been enough, but it is not exactly what I wanted. I had defined several statuses in pidgin (for example cofee, lunch, brb, afk, meeting,...) and, when my status is changed to one of them, my nick should be changed accordingly. Looking to some plugins source code it should not be very difficult, so I started to develop one and here it is the result: ircstatus.
The plugin does exactly what I want:
It should be configured with one IRC account (I thought in configuring more than one but I changed my mind, it is not necessary). That account is the one that will receive the nick changes. You can also configure the separator string (by default is "|" but you can configure another one, "_" for example).
The plugin listens for status changes (signal savedstatus-changed) and performs the change of the nick in the IRC account specified in the preferences. Obviously the plugin checks that the IRC account exists and is connected. The new nick is constructed by the concatenation of the initial username, the separator and the title of the status. The title is sanitized (non-alphanumeric characters are replaced by an underscore "_").
And that is all. The plugin is very simple but it works exactly like I want. Besides I can combine it with another plugin away-on-lock which changes automatically your status if the screen-saver is activated. This way my nick is always modified even when I forget to change my status manually, which is quite often because of my bad memory (away-on-lock changes the status to one of the away status, you can configure which one, and in turn my ircstatus changes the nick accordingly). I suppose that such a bothering plugin will not be needed by a lot of people, but you can clone or download it from my github anyway.
Cherio!
Comments