Java

As you have already seen in the previous entry, Debian 7.0 is now public and therefore I am using Jessie in testing. Openjdk 7 package is now update 21 (icedtea 2.3.9) and, as I promised, I finally upgraded from openjdk-6 to 7. Taking advantage of the situation I tested the previous OCSP issue I commented in this blog but with the new version.

In 7u21 the issue is exactly the same but presenting different error messages. In the error cases the message presented is the following Signature length not correct: got 256 but was expecting 128. Much more cryptic I have to say than the previous messages (Error verifying OCSP Responder's signature or Responder's certificate not valid for signing OCSP responses).

Checking again the code the problem is still the same. Just the issuer certificate or the configured certificate is passed to the OCSPResponse, so the cases 1 and 3 cannot be mixed with the case 2. But there is a good new, since the following commit the OCSP and OCSPResponse classes accept a list of responder certificates (previously it was only one) and therefore now the fix is even easier. It just consists in adding both certs (issuer cert and configured responder if it is the case) to the list. so now the fix is a simpler patch that only changes the OCSPChecker class.

Talking about the bug I received an answer from Oracle telling me that the BUG has been accepted (it was around two months ago) but I have no more news since then. There is not a public link in the java database either. I am going to try to contact with icedtea guys to see what they think about that.

Keep on trying!

I have received some comments about the old entry that installed a full glassfish HA solution using debian, people complain that the post did not deal with Stateful EJBs. A Stateful EJB is a enterprise bean that acts as a server-side extension of the client maintaining its state during calls (the bean object maintains its properties). Obviously this type of EJB has to also be replicated between the application server instances in case of an HA architecture. The reason for that oversight is clear, I am not a fan of remote EJBs, I always recommend to use local ones (which are deployed and called inside the same JVM and their performance is much better) or Web Services.

Stateful EJBs should work smoothly in glassfish and today I am going to present a little example (and, as you will see later, I faced a very nasty issue). The first thing I did was installing the solution with glassfish 3.1.2.2, I repeated the same steps explained in the previous entry one by one. When the (in)famous clusterjsp was running I started to develop and deploy a simple stateful EJB following Markus Eisele's blog entry.

1. The remote interface was created. The sample bean is just a counter with the count as the state to maintain/replicate:

   @Remote
   public interface Sample {
       
       public String increment();
       
       public String reset();
       
       public String stop();
   }
   

2. The real EJB implementation is coded. The increment method returns the instance name which is processing the call and the current value. The stop method finishes the count and removes the EJB (the @Remove annotation means that, when the client calls this method, the EJB can be destroyed).

   @Stateful(name="SampleBean")
   @Remote(Sample.class)
   public class SampleBean implements Sample, Serializable {
   
       private int count = 0;
        
       @Override
       public String increment() {
           return new StringBuilder(System.getProperty("com.sun.aas.instanceName"))
                   .append(": ")
                   .append(++count)
                   .toString();
       }
   
       @Override
       public String reset() {
           count = 0;
           return new StringBuilder(System.getProperty("com.sun.aas.instanceName"))
                   .append(": reset to ")
                   .append(count)
                   .toString();
       }
   
       @Override
       @Remove
       public String stop() {
           return new StringBuilder(System.getProperty("com.sun.aas.instanceName"))
                   .append(": releasing count on ")
                   .append(count)
                   .toString();
       }   
   }
   

3. The EJB is packed in a JAR and the JAR inside an EAR, then it is deployed with availability enabled (exactly as a web WAR application file).

   $ ./asadmin deploy --availabilityenabled=true --target cluster1 ~/clusterejb.ear 
   Application deployed with name clusterejb.
   Command deploy executed successfully.
   

   At this point the JNDI name for the EJB should be available (the relative path depends on the names used in ear and ejb-jar files):

   $ ./asadmin list-jndi-entries --context "java:global/clusterejb/clusterejb-ejb" cluster1
   debian1-gf:
   SampleBean__3_x_Internal_RemoteBusinessHome__: javax.naming.Reference
   SampleBean: javax.naming.Reference
   SampleBean!es.rickyepoderi.sample.Sample: javax.naming.Reference
   
   debian2-gf:
   SampleBean__3_x_Internal_RemoteBusinessHome__: javax.naming.Reference
   SampleBean: javax.naming.Reference
   SampleBean!es.rickyepoderi.sample.Sample: javax.naming.Reference
   

4. Finally I created a simple test client that connects to the remote EJB and calls the increment method. The client performs two iterations, the internal one increments the EJB and the outer loop just creates another EJB (so you can repeat the counting loop as many times as you want).

   public class Test {
       static public void main(String[] args) throws Exception {
           InitialContext ic = null;
           Sample sample = null;
           Properties env = new Properties();
           String ans;
           BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
           do {
               try {
                   ic = new InitialContext(env);
                   String lookup = "java:global/clusterejb/clusterejb-ejb/" +
                       "SampleBean!es.rickyepoderi.sample.Sample";
                   System.out.println("Retrieving " + lookup);
                   sample = (Sample) ic.lookup(lookup);
                   do {
                       System.out.println(sample.increment());
                       System.out.print("More (Y/n): ");
                       ans = in.readLine();
                   } while (!ans.equalsIgnoreCase("n"));
               } finally {
                   if (sample != null) {
                       System.out.println("Stoping count!!!");
                       try {System.out.println(sample.stop());} catch (Exception e) {}
                   }
                   if (ic != null) {
                       System.out.println("Closing context!!!");
                       try {ic.close();} catch (Exception e) {}
                   }                
               }
               System.out.print("New EJB (Y/n): ");
               ans = in.readLine();
           } while (!ans.equalsIgnoreCase("n"));
       }
   }
   

5. The client is executed in the following way:

   $ java -Dcom.sun.appserv.iiop.endpoints=debian1.demo.kvm:23700,debian2.demo.kvm:23700 \
     -cp .:/home/ricky/glassfish3/glassfish/lib/gf-client.jar es.rickyepoderi.client.Test
   

   In theory the glassfish provider gets the specified property (com.sun.appserv.iiop.endpoints) and constructs a specific environment for the JNDI lookup similar to the following (that environment can be used directly in the code too):

   Properties env = new Properties();
   env.put("java.naming.factory.initial", 
     "com.sun.enterprise.naming.impl.SerialInitContextFactory");
   env.put("java.naming.factory.state", 
     "com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl");
   env.put("com.sun.appserv.ee.iiop.endpointslist",
     "corbaloc:iiop:1.2@debian1.demo.kvm:23700,iiop:1.2@debian2.demo.kvm:23700");
   

   It uses the SerialInitContextFactory and constructs the corbaloc URL using both servers. Besides it randomizes the order (sometimes debian1 will be the primary and debian2 the alternative and sometimes the other way around) so the solution gets some balancing over the servers (take in mind the order is assigned at context creation).

   At this point the client should just work but I faced a very weird problem. When the test application was connected to the debian1-gf instance everything worked as expected, but when the client connected to the debian2-gf one no failover was done. The client application hung trying to always connect to the second instance even thought that server was down. I spent all the morning trying to find what the hell was happening and finally I understood the situation.

   In RMI/IIOP it seems that the server architecture is managed during conversation, I mean, although I set both servers in the property (and in turn in the corbaloc URL), when the client contacts with the primary server the servers are defined again using the IOR field (Interoperable Object Reference). And my problem was that debian1 returned both (debian1 and debian2) in the reference but debian2 returned itself twice (no debian1 was answered as a valid alternative server). Finally (and with a big headache as a result) the problem was that glassfish uses nodes as the corba servers and, if you re-check how they were created, I created debian1-ssh node as localhost and debian2-ssh with the real hostname (I thought the nodes were only used for management purposes). So debian1 used localhost and debian2 (debian1 and debian2) and debian2 localhost and debian2 (debian2 and debian2), that produced the annoying issue explained before. So I changed the node directly in the domain.xml configuration file:

   <node node-host="debian1.demo.kvm" name="debian1-ssh" 
    type="SSH" install-dir="${com.sun.aas.productRoot}">
     <ssh-connector>
       <ssh-auth></ssh-auth>
     </ssh-connector>
   </node>
   

   But obviously the correct solution would have been creating the debian1-ssh node using the real hostname:

   $ ./asadmin create-node-ssh --nodehost debian1.demo.kvm debian1-ssh
   

   I confess that I do not know if those names can be forced somehow (maybe some configuration in glassfish let use specific names and not the names of the nodes, but it does not matter much right now). As usual it was much more complicated realizing what was going on than fixing the issue.

6. After that I successfully tested the EJB. Iterating over the first loop the server changes (randomly the client provider change the primary server from debian1 to debian2). When I was being served by a specific server (debian1) and that server was shutting down, the alternate (debian2) continued the counting. The count is preserved without problem cos replication is maintaining the state of the EJB. Here it is the video that shows the previous explanation.

Today's entry complements a previous one of the blog (simple but full glassfish HA using debian), that entry tested only the web part but some people have commented to me what happened with a Stateful EJB. Usually that type of EJB should work in the HA scenario smoothly but a weird problem happened to me, using localhost instead of the real hostname for node creation complicated my solution. Once that issue was fixed the solution worked smoothly and EJB HA was assured by the glassfish cluster. Here I upload the clusterejb project I used in the entry. The moral is glassfish replication also works with stateful EJBs.

Replicated regards!

Today's entry is going to explain an issue I had some weeks ago with an OCSP (Online Certificate Status Protocol) responder. I had to use a specific responder which worked in such a way that I was not able to make Java work against it. I did not understand what was happening and I decided to test the problem more deeply in this entry.

Let's start explaining how an OCSP responder works: the server receives requests from clients which want to check if a specific certificate is revoked; the responder checks the status of the certificate and answers if it is good, revoked or unknown; cos the protocol is affected by man in the middle attacks the response is signed; usually the response is signed by the same CA that issued the certificate being checked. And here my issue appears, my specific OCSP server checked certificates issued by different CAs, and to do that it followed two rules:

• If the certificate to be checked is issued by one of their CAs, the response was signed by the same CA (common situation).

• But if the certificate was issued by a partner authority it was signed by a special certificate issued by themselves.

I could not configure Java to work against that OCSP responder. So I decided to read what RFC2560 (the standard that defines OCSP protocol) says, and this is the important part:

All definitive response messages SHALL be digitally signed. The key used to sign the response MUST belong to one of the following:

1. the CA who issued the certificate in question
2. a Trusted Responder whose public key is trusted by the requester
3. a CA Designated Responder (Authorized Responder) who holds a specially marked certificate issued directly by the CA, indicating that the responder may issue OCSP responses for that CA

There are three possibilities or cases, which I will try to explain a bit more:

• Case 1: The certificate used to sign the response is the same certificate which issued the certificate being checked. This is the most common situation.

• Case 2: The response is signed by a trusted certificate for the client. That means the requester should know before that certificate and configure it someway to trust in the OCSP responses.

• Case 3: The response is signed by a certificate which has a special extension (OCSPSigning) to inform that it is able to sign OCSP responses, besides it must be issued by the same CA which issued the certificate being checked. This case is used to delegate OCSP responders to a partner CA.

If you recheck what my responder was doing, it was using case 1 for its own certificates (common case) and case 2 for foreign certificates (the one in which the client should know the certificate used in order to trust in the responses). Therefore I understand that the OCSP responder was working well, its behavior is covered by the standard.

OCSP in Java is part of its Public-Key Infrastructure (PKIX) implementation. It has several properties defined in the Security configuration for controlling the checking process. Let's summarize the most important:

1. ocsp.enable. Setting this property to true the OCSP checking is used for validating certificates (CRL is always enabled but not OCSP).

2. ocsp.responderURL. Property that fixes the use of an OCSP server instead the one provided by the certificate extended properties.

3. ocsp.responderCertSubjectName. Property to fix the certificate used by the responder to sing responses (there are other properties for doing the same thing). This option is used for the case 2 defined in the standard.

At this point I decided to create an openssl ocsp environment which tested the three possibilities, for that I need a client certificate and three certificates for the OCSP responder (I followed similar commands to the ones I used in this previous entry):

• cakey.pem/cacert.pem: RSA key and certificate of the demo CA (self-signed). If it is used in the OCSP responder the case 1 is tested (password Kiosko_00 to be used).

• ocsp-trusted.key/ocsp-trusted.pem: The certificate is signed by the CA but with no special OCSPSigning extension. The pair will be used as the responder certificate for the case 2, the client should be configured to trust in that certificate.

• ocsp-signing.key/ocsp-signing.pem: The pair used for case 3, the certificate is signed by the CA and has the OCSPSigning extension (see this blog entry to see how I did it). Cos it is signed by the same CA it works without any configuration at client side.

• client1.pem: The certificate to be checked (signed by the same demo CA).

This zip file contains the demoCA directory with all the openssl configuration to launch the ocsp responder. I developed a simple Java to test PKIX validation in the three scenarios. The program sets to true ocsp.enable and uses the fourth and fifth argument to set the ocsp.responderURL and the ocsp.responderCertSubjectName (this argument is optional). Previous arguments are the client certificate to check, the cacerts keystore to use and the password of the keystore. I tested the three possible scenarios:

1. Case 1: The program checks client1 certificate against openssl OCSP launched with demo CA as the response signer.

   $ openssl ocsp -index demoCA/index.txt -CA demoCA/cert/cacert.pem \
     -rsigner demoCA/cert/cacert.pem -rkey demoCA/private/cakey.pem -port 3456 -text
   

   In order to work in this case the Java program should be launched with no certificate fixed as the responder one:

   $ java -cp . CertificateChecker client1.pem cacerts.demo changeme http://localhost:3456
   Loading certificate...
   Loading cacerts...
   Performing PKIX validation...
   Result: OK
   

   The demo CA should be imported in the cacerts.demo keystore (our demo CA must be trusted for the Java program).

2. Case 2: Now the ocsp server is started using the ocsp-trusted key and certificate, remember this one has no special extension.

   $ openssl ocsp -index demoCA/index.txt -CA demoCA/cert/cacert.pem \
     -rsigner demoCA/cert/ocsp-trusted.pem -rkey demoCA/private/ocsp-trusted.key \
     -port 3456 -text
   

   In the second scenario Java should be configured in such a way that the client knows the certificate used in the responses. The ocsp-trusted certificate was added to the keystore and the fifth argument was passed to mark that this specified subject is used to sign the responses.

   $ java -cp . CertificateChecker client1.pem cacerts.demo changeme \
     http://localhost:3456 "CN=OCSP-TRUSTED, O=demo.kvm, ST=demo, C=ES"
   Loading certificate...
   Loading cacerts...
   Performing PKIX validation...
   Result: OK
   

   So it works, if the subject of the responder certificate is specified the Java program trusts in its responses.

3. Case 3: The ocsp server is launched using the ocsp-signing key and certificate, that certificate is signed with the OCSPSigning extension.

   $ openssl ocsp -index demoCA/index.txt -CA demoCA/cert/cacert.pem \
     -rsigner demoCA/cert/ocsp-signing.pem -rkey demoCA/private/ocsp-signing.key \
     -port 3456 -text
   

   Now in order to work the client program it should be started again but without any specific certificate set as an argument.

   $ java -cp . CertificateChecker client1.pem cacerts.demo changeme http://localhost:3456 
   Loading certificate...
   Loading cacerts...
   Performing PKIX validation...
   Result: OK
   

   It works again. Java PKIX implementation checks that the certificate used has the OCSPSigning extension and the issuer of that certificate is the same that the issuer of client1 certificate.

I know, now you are saying but what the hell, Java supports all the possibilities. Where is the problem? A different configuration is needed to accomplish the three cases. Case 2 only works if the property ocsp.responderCertSubjectName is set (or any other option that fixes the certificate used for signing responses) but case 1 and 3 only works with that property not set. There is no way to configure the Java Security in a way that handles the three scenarios at the same time. What happens if there is an OCSP server whose responses use mixed cases? You are in a big problem if case 2 is involved. If you remember the responder I commented in the introduction, it works using cases 1 and 2, I would have never got it working no matter the time I had spent. It was absolutely impossible.

First question, does the standard support that an OCSP responder uses all the scenarios at the same time? Or does it force the responder to not mix case 2 with the other two? I have not read anything against the first sentence and Java supports cases 1 and 3 at the same time, so why not case 2. Second question, is mixing case 2 with any of the other cases useful for a responder? I think it is, it is quite reasonable that the responder uses case 1 (the common method) for their own certificates and case 2 for certificates issued by other authorities (case 3 is also valid but you need the partner authority to sign a certificate for you). Remember that case 1 does not need client configuration and case 2 does need it, so it is logical to minimize the use of case 2. I think this is a BUG in the Java implementation and I will try to report it. If any of you is an expert in OCSP please let me know your thoughts.

As you know I try to be a good neighbor and I checked why Java worked that way. Looking the code for openjdk6 b27 (last bundle released for version 6 at the moment) I checked that class sun.security.provider.certpath.OCSPChecker chooses the responder certificate from the properties defined in the Security configuration or from the issuer of the certificate that is going to be validated. It is one or the other. Then the class sun.security.provider.certpath.OCSPResponse receives that certificate as a parameter and validates the sign. OCSPResponse covers cases 1 and 3, the certificate used for signing can be the one passed as an argument or another certificate which has the OCSPSigning extension and was issued by the argument cert. That is the reason Java/PKIX isolates case 2 from the other two scenarios.

Because I spent so much time guessing all of this I decided to make a patch. The fix involves the two classes commented in the previous paragraph and another one which is an intermediary class between them (OCSP class in the same package). In my solution OCSPResponse class receives two arguments: the issuer cert (the certificate which issues the client certificate to be checked) and the responder cert (the certificate defined in the Security properties, it can be null). With both certs the class can check the three cases at the same OCSP response, including the reviled scenario 2. Here it is my patch. With the new classes no matter how the openssl ocsp responder is started (any case) the same configuration for the Java program works:

$ java -Xbootclasspath/p:/home/ricky/Desktop/jdk-patch/src/share/classes/ \
  -Djava.security.debug=certpath -cp . CertificateChecker client1.pem cacerts.demo \
  changeme http://localhost:3456 "CN=OCSP-TRUSTED, O=demo.kvm, ST=demo, C=ES"

The previous command works for the three certificates (demoCA case 1, ocsp-trusted case 2 and ocsp-signing case 3) cos I am prepending my modified classes to the boot classpath. The same arguments, the same configuration works no matter which case the responder uses. Cos explanation for this issue is so long and complex I posted the entry before reporting the BUG (I will link this entry from the BUG explanation). I opened another BUG before but I had not understood the problem completely and people at Java team did not say anything to me (I was half wrong so it is fair).

Let's see what happens now!