Ricky's Hodgepodge

Some weeks ago I was hardly affected by an old issue about Java Server Faces (JSF) in the Sun/Oracle mojarra implementation which Wildfly/JBoss incorporates. The bug indeed is quite obvious, when you mix JSF and the JavaServer pages Standard Tag Library (JSTL) some components can be duplicated at refreshing time. The issue has an easy test-case and it was explained in detail in this mojarra bug. The JSF framework maintains a tree of components that represents the page/view (the tree can be stored at the server side or the client side). The components are refreshed constantly when the user interacts with the page. A composite is a reusable facelet component that is composed by more components (the composite is defined and then reused in different pages). On the other side JSTL is the old tag library already used in the JSP technology (common tags like <c:if>, <c:forEach>, <c:set>,...) and, obviously, both technologies act a different level. The JSTL runs when the view is build (for example if there is a <c:if> that resolves to false, that part of the view is not created), then JSF/facelets runs over the generated view to create the tree. Simplifying it, JSTL runs first from top to bottom and then hands over the result to JSF which in turn runs from top to bottom again.

So, when you mix both technologies you have to be specially cautious because of this special interaction. The bug was caused by using the same composite several times in the tree with a JSTL component modifying the number of its inclusions (for example a <c:if> which makes a second instance of a composite to be rendered or a <c:forEach> that adds the same composite different number of times). The special interaction between the two technologies complicated the resolution for a long time. Because of the fact that developers have to be cautious mixing both, the issue was always considered by mojarra maintainers like a developer error and never like a bug. During several years the problem remained there with bugs filed and rejected, questions in several threads and blog entries about it. Besides JSF is now an old and disappearing technology, fact that complicates even more the resolution (remember that now the presentation layer has moved entirely to the browser side, using javascript frameworks).

In my new job I discovered that several JBoss cases were very similar, same problem reported and always answered by the same mantra: try not to mix JSTL and JSF cos it gives problems. At first I just repeated the mantra but, seeing two or three of them with a similar description and not having a real reason to not use JSTL and JSF together, I decided to investigate further. Checking the test-case in previous mojarra bug I got fully convinced that this was a real bug and not an improper use of IDs by the developer. And I finally deep dived into the code to try to understand the real problem. I have to say that JSF is a really complicated framework and I spent several days hacking the code (getting tired, desperate sometimes, going in circles, trying a lot of things that did not work, doing another things to make me relax and think different, trying again,...) but finally I understood the bottom problem. As always, the final reason was not very complicated and it could be fixed with a few lines of code. I gave a detailed explanation in the same bug if you are interested in the details.

Nevertheless, mojarra is very convoluted, and my little patch could affect somewhere else, I needed to test it a bit better. The mojarra implementation has some good automated test suite, so I decided to pass the suite and, if no new error was reported, comment in the bug and present my patch. Following this guide I installed again a glassfish server to run the tests. Things now are a bit different and some tricks were needed to make the automatic tests run properly but, after setting everything in place, all the tests passed without any problem, and there are a lot, so finally I was confident that my patch went in the correct direction. Using the same bug report I added two comments in it, the first one explaining the issue just as I understood it, and the second one with the patch. After some days of silence the patch moved on (my team internally pressed a lot to try to reach Oracle) and now the patch is applied in all the 2.2 and 2.3 branches (even an automated test has been added to the suite for this issue). I hope they also move the test to master because it would be a real pity that future versions were released with this insidious bug still present.

Yesterday I realized that new mojarra versions 2.3.1 and 2.2.8-22 had been released, and the patch is incorporated in both. So I suppose that silently all depending projects will be updating to them (JBoss included). That is the reason for this entry. It was a nice challenge to make it work, I am a newbie in my company, and I was in the middle of a problem in a library that was implemented by a another company (a company I worked for before). So it was a nice experience, learning how things work at this level. But you know how I think...

Never compromise, not even in the face of Armageddon. (Rorschach - Watchmen)

I was advised by Albert Mestre, with a comment in one of my previous entries, that the DNIe is not working for the last version of chrome. I usually do not use that browser so I was not aware of it (last time I checked it worked, but previously I had a debian box and the chromium browser was an old version). This weekend I spent some time trying to know what was happening.

First I installed the library like I explained in a previous entry and (at command level) it is working ok. The module is in place and the certificates can be read.

• The module is installed in the database:

  modutil -dbdir sql:.pki/nssdb/ -list
  
  Listing of PKCS #11 Modules
  -----------------------------------------------------------
    1. NSS Internal PKCS #11 Module
  	 slots: 2 slots attached
  	status: loaded
  
  	 slot: NSS Internal Cryptographic Services
  	token: NSS Generic Crypto Services
  
  	 slot: NSS User Private Key and Certificate Services
  	token: NSS Certificate DB
  
    2. opensc
  	library name: /home/rmartinc/apps/opensc/lib/opensc-pkcs11.so
  	 slots: 1 slot attached
  	status: loaded
  
  	 slot: Gemalto PC Twin Reader 00 00
  	token: PIN1 (DNI electrónico)
  -----------------------------------------------------------
  

• The certificates can be listed:

  certutil -d sql:$HOME/.pki/nssdb -L -h "PIN1 (DNI electrónico)"
  
  Certificate Nickname                                         Trust Attributes
                                                               SSL,S/MIME,JAR/XPI
  
  Enter Password or Pin for "PIN1 (DNI electrónico)":
  PIN1 (DNI electrónico):CertAutenticacion                    u,u,u
  PIN1 (DNI electrónico):CertCAIntermediaDGP                  ,,   
  PIN1 (DNI electrónico):CertFirmaDigital                     u,u,u
  

• And any of them (for example the authentication one) can be parsed:

  certutil -d sql:$HOME/.pki/nssdb -L -h "PIN1 (DNI electrónico)" -n "PIN1 (DNI electrónico):CertAutenticacion"
  Enter Password or Pin for "PIN1 (DNI electrónico)":
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number:
              XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
          Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
          Issuer: "CN=AC DNIE 001,OU=DNIE,O=DIRECCION GENERAL DE LA POLICIA,C=ES"
          Validity:
              Not Before: Mon Oct 10 06:58:43 2016
              Not After : Wed Aug 19 22:00:00 2020
          ...
  

So, in general, the library is working at command level. But when you start the browser and try to use the card a coredump is generated. The trace is something like this:

(gdb) bt
#0  0x00007f794760dda9 in BN_num_bits () at /usr/lib64/chromium-browser/./libboringssl.so
#1  0x00007f794760dde9 in BN_num_bytes () at /usr/lib64/chromium-browser/./libboringssl.so
#2  0x00007f794765480d in rsa_default_size () at /usr/lib64/chromium-browser/./libboringssl.so
#3  0x00007f7947652605 in RSA_size () at /usr/lib64/chromium-browser/./libboringssl.so
#4  0x00007f79476401cf in pkey_rsa_verify () at /usr/lib64/chromium-browser/./libboringssl.so
#5  0x00007f794763d012 in EVP_DigestVerifyFinal () at /usr/lib64/chromium-browser/./libboringssl.so
#6  0x00007f7947659d45 in ASN1_item_verify () at /usr/lib64/chromium-browser/./libboringssl.so
#7  0x00007f78f670b02b in cwa_verify_icc_certificates (icc_cert=0x1b5a5b4e40f0, sub_ca_cert=0x1b5a5b605a50, provider=
    0x1b5a5a3cac40, card=0x1b5a5a213000) at cwa14890.c:354
#8  0x00007f78f670b02b in cwa_create_secure_channel (card=card@entry=0x1b5a5a213000, provider=0x1b5a5a3cac40, flag=flag@entry=1)
    at cwa14890.c:1114
#9  0x00007f78f6709bb3 in dnie_pin_verify (card=card@entry=0x1b5a5a213000, data=data@entry=0x7f791560b2f0, tries_left=tries_left@entry=0x1b5a5a8368e4) at card-dnie.c:2181
...

It seems that chrome has forked the openssl and now they have their own implementation called boringssl. And it crashes with the DNIe. In some part of the implementation the driver checks that the certificates returned by the card are valid and it uses openssl API for that (method X509_verify). As chrome loads its own libraries the pkcs#11 also has to use them and everything finishes in big core dump. I tried to pre-load the openssl libraries but then it is the chromium itself the one that does not work. So openssl and boringssl seem to be incompatible. The same day I decided to re-test the module against openssl and libressl (another fork done by the BSD guys) and the DNIe works well with both of them.

I am not going to spend more time on this. It seems that something weird happens with the boringssl implementation (besides the problem appears just checking the validity of the intermediate CA certificate, which is read from the card). My tests were done with the chromium 57 that comes with fedora 25, Albert reported the issue with version 58. I do not like that google decided to use its own implementation of such an important library if they are going to make them incompatible. In summary, for the moment DNIe driver for OpenSC is not working with chrome (and I do not think the situation is going to change for the moment). I do not know if other OpenSC drivers (several of them uses openssl) are also affected.

Sorry for the bad news!

EDIT: It works in my old debian laptop with chromium 57 and 58 (after updating to last current package). Debian decided to link boringssl statically and then the issue is avoided. So I think is more a bug in fedora than a general problem. But if you see that the boringssl so file is in your system you have to worry. At the end it is not a very big problem, I am going to change the title.

The previous week I was working with CRLs files and a very big PEM CRL bundle was taking several minutes to being parsed using Java. The complete file was several megabytes in size but the fact did not justify such a long time. A Certificate Revocation List (CRL) is a bunch of certificates issued by a Certificate Authority (CA) that have been revoked (the certificate is not valid anymore). Usually a CRL is kilobytes in size, when a revoked certificate expires it is usually removed from the CRL (it is invalid anyway), and this practice makes the CRL size be moderate. The revocation info is just the certificate serial number and the date, so thousand and thousands of them are needed to have such a big file.

Looking at the code the method to parse the CRLs is generateCRLs. In my case the file was a list of PEM encoded CRLs (PEM is base64 DER format and enclosed between a header -----BEGIN X509 CRL----- and a footer -----END X509 CRL-----, DER format is a binary format using ASN1). The X509Certificate implementation accepted both formats (DER and PEM) but it seems to do something very weird to deal with the PEM format (you can see the code in OpenJDK). The DER format is read directly, but PEM is first moved into a memory buffer, the base64 data is decoded and the resulting DER is parsed. The buffer is initially 2KB in size and it grows by chunks of 1K (the Arrays.copyOf method is used, so a new array is allocated and the data copied again and again). This is not a bad idea if the CRL is several kylobytes in size but, if it has several megas, the process is extremely slow.

Obviously our decision was transforming our CRL file from PEM to DER format. The load took just a few seconds instead of the several minutes it was taking before. Nevertheless I spent some time trying to improve the performance of the java method. Now java8 has standard Base64 encoder and decoder and, more important, it has a method to directly read from an InputStream. This way the extra buffer can be avoided and it always read from the file directly. I tried my idea and it works. I opened a bug against the java bug database and it is public now. My patch is there if you need it.

With the modification an enormous CRL file of 30MB (around one million revocations) gives the following times:

• DER: 2 seconds.
• PEM (patched): 14 seconds
• PEM (before): 379 seconds

So, the DER file is always faster (there is no Base64 decoding) but, at least, now the PEM process time is in the same scale. If you need to manage CRLs files in java, please, never use the PEM format, always manage DER native format. I do not know if my patch will be finally included but, it does not matter, it just improves the situation, DER will always be faster and with a smaller footprint.

Regards!