Ricky's Hodgepodge

The mod-auth-openidc is a module to provide OIDC authentication to the apache web server. In this entry I will try to configure the apache module in order to work with a keycloak server. The idea is that two clients will be configured: the first one will be a normal client (confidential) that will provide normal code-to-token redirect flow; the second one will be a bearer-only endpoint client (the application just validates the token that should be sent using a bearer authentication). The blog entry is just a summary of the steps I did to configure it. I hope that it helps to someone else. A debian box (called debian.sample.com) was used to perform the tests and configuration.

1. First the jdk and keycloak is installed and started.

   
   apt-get install openjdk-11-jdk
   wget https://downloads.jboss.org/keycloak/6.0.1/keycloak-6.0.1.zip
   unzip keycloak-6.0.1.zip
   cd keycloak-6.0.1/bin/
   ./standalone.sh
   

   The admin users are created for the EAP and the keycloak itself.

   
   ./add-user.sh -u admin -p XXXXX
   ./add-user-keycloak.sh -u admin -p XXXXX
   

   And finally the interfaces are changed to serve through the correct IPs (not only localhost).

   
   ./jboss-cli.sh --connect
   /interface=public:write-attribute(name=inet-address, value="${jboss.bind.address:192.168.100.20}")
   /interface=management:write-attribute(name=inet-address, value="${jboss.bind.address:0.0.0.0}")
   

2. Any OIDC installation uses certificates so I created a CA and a certificate for the host (same one is valid for both keycloak and the apache server because they are placed in the same host).

   First the CA certificate is created:

   
   cd /etc/ssl
   mkdir -p demoCA/newcerts
   touch /etc/ssl/demoCA/index.txt
   echo 01 > /etc/ssl/demoCA/serial
   echo 01 > /etc/ssl/demoCA/crlnumber
   openssl req -subj "/C=ES/O=sample.com/CN=ca.sample.com" -new -newkey rsa:2048 -keyout private/cakey.pem -out careq.pem
   openssl ca -out cacert.pem -days 10000 -keyfile private/cakey.pem -selfsign -extensions v3_ca -infiles careq.pem
   openssl x509 -in cacert.pem -outform DER -out cacert.der
   openssl pkcs12 -export -out cacert.p12 -in cacert.pem -inkey private/cakey.pem
   

   And then the certificate for the server:

   
   openssl genrsa -out private/debian.sample.com.key 2048
   openssl req -subj "/C=ES/O=sample.com/CN=debian.sample.com" -key private/debian.sample.com.key -new -out debian.sample.com.csr
   

   The alternate names for my host are added in the /etc/ssl/openssl.cnf to have a correct certificate:

   
   [ v3_req ]
   # Extensions to add to a certificate request
   basicConstraints = CA:FALSE
   keyUsage = nonRepudiation, digitalSignature, keyEncipherment
   subjectAltName = @alt_names
   
   [alt_names]
   DNS.1 = debian.sample.com
   IP.1 = 192.168.100.20
   

   Then the CA is used to sign the certificate and it is generated:

   
   openssl ca -in debian.sample.com.csr -cert cacert.pem -keyfile private/cakey.pem -out debian.sample.com.crt -extensions v3_req 
   openssl pkcs12 -export -out debian.sample.com.p12 -in debian.sample.com.crt -inkey private/debian.sample.com.key
   keytool -importkeystore -srckeystore debian.sample.com.p12 -srcstoretype pkcs12  -srcalias 1 -destkeystore debian.sample.com.jks -deststoretype jks -destalias debian.sample.com
   cat debian.sample.com.crt cacert.pem >debian.sample.com.all.pem
   keytool -import -alias debian.sample.com -file debian.sample.com.all.pem -keystore debian.sample.com.jks
   

   Finally copy the CA certificate to the trusted ones and update the debian certificates.

   
   cp cacert.pem /usr/share/ca-certificates/cacert.crt
   echo "cacert.crt" >> /etc/ca-certificates.conf
   update-ca-certificates
   

3. Now the certificate is added to the keycloak. It is copied to the configuration directory:

   
   cp debian.sample.com.jks /home/java/keycloak-6.0.1/standalone/configuration/
   

   And the elytron subsystem is configured to use it:

   
   /subsystem=elytron/key-store=debian:add(type=jks, relative-to=jboss.server.config.dir, path=debian.sample.com.jks, credential-reference={clear-text=XXXXX})
   /subsystem=elytron/key-manager=debian-manager:add(key-store=debian, credential-reference={clear-text=XXXXX})
   /subsystem=elytron/server-ssl-context=debian-context:add(key-manager=debian-manager, protocols=["TLSv1.2"])
   batch
   /subsystem=undertow/server=default-server/https-listener=https:undefine-attribute(name=security-realm)
   /subsystem=undertow/server=default-server/https-listener=https:write-attribute(name=ssl-context, value=debian-context)
   run-batch
   

4. One client mod-auth-openidc is created with confidential type.

   

   Remember to take note of the client's secret (in the Credentials tab) to later configure the module.

5. And now the bearer only client mod-auth-oauth20 is created, the access type is changed to bearer-only:

   

   The credential will be also needed because the token will be verified using the introspect endpoint.

6. Now the apache and all the needed modules are installed into the debian box:

   
   apt-get install apache2 libapache2-mod-php libapache2-mod-auth-openidc
   

7. The certificates are configured, so first they are copied into the directories:

   
   cp debian.sample.com.crt  /etc/ssl/
   cp debian.sample.com.key /etc/ssl/private/
   

   Then the /etc/apache2/sites-available/default-ssl.conf file is modified to use those certificates:

   
           <VirtualHost *:443>
                   SSLCertificateFile      /etc/ssl/debian.sample.com.crt
                   SSLCertificateKeyFile /etc/ssl/private/debian.sample.com.key
                   SSLCertificateChainFile /etc/ssl/certs/cacert.pem
                   SSLCACertificatePath /etc/ssl/certs/
   

   Enable all the needed modules, the ssl site and restart the apache service:

   
   a2enmod ssl
   a2enmod php7.3
   a2enmod auth_openidc
   a2ensite default-ssl
   systemctl restart apache2
   

8. Now the openidc module is configured inside one location of the previous ssl host configuration file:

   
   OIDCProviderMetadataURL https://debian.sample.com:8443/auth/realms/master/.well-known/openid-configuration
   OIDCRedirectURI https://debian.sample.com/mod-auth-openidc/oauth2callback
   OIDCCryptoPassphrase 0123456789
   OIDCClientID mod-auth-openidc
   OIDCClientSecret 950225ad-3980-4a22-a14c-5ceebd366328
   OIDCProviderTokenEndpointAuth client_secret_basic
   OIDCSessionInactivityTimeout 1800
   OIDCSessionMaxDuration 28800
   #OIDCUserInfoRefreshInterval 60
   OIDCRefreshAccessTokenBeforeExpiry 10
   OIDCRemoteUserClaim preferred_username
   OIDCScope openid
   OIDCPassIDTokenAs claims payload
   OIDCProviderCheckSessionIFrame "https://debian.sample.com:8443/auth/realms/master/protocol/openid-connect/login-status-iframe.html"
   OIDCDefaultLoggedOutURL "https://debian.sample.com"
   
   <Location /mod-auth-openidc>
       AuthType openid-connect
       Require valid-user
       LogLevel debug
   </Location>
   

   A little show.php page is prepared to show all the OIDC variables injected (it should be copied inside the location /var/www/html/mod-auth-openidc):

   
   <html>
   <body>
       <h1>OIDC Variables</h1>
       <ul>
   <?php
       foreach($_SERVER as $key => $value) {
           if (strlen($key) > 4 && substr($key, 0, 5) === "OIDC_") {
               echo "<li><strong>" . $key . "</strong>: " . $value . "</li>";
           }
       }
   ?>
       </ul>
       <p><a href="oauth2callback?logout=https%3A%2F%2Fdebian.sample.com">logout</a>
       <iframe title='empty' style='visibility: hidden;' width='0' height='0' tabindex='-1'
               id='openidc-op' src='oauth2callback?session=iframe_op' >
       </iframe>
       <iframe title='empty' style='visibility: hidden;' width='0' height='0'
               tabindex='-1' id='openidc-rp' src='oauth2callback?session=iframe_rp&poll=5000'>
       </iframe></p>
   </body>
   </html> 
   

   The /mod-auth-openidc location will be protected using OIDC and the configuration is prepared to use the same settings than in keycloak (same timeouts). With this configuration the module redirects the user to the keycloak login page and, once it is logged in, the code-to-token flow finishes the process. The apache module injects a lot of variables that are shown in the PHP page. Besides the session management is configured with the keycloak iframe. This way if we log to another application (for example the account keycloak page) ane performs a logout, the iframe automatically detects the change in the cookie and the user is logged out (the browser is redirected to the default apache index page, because it was configured as the OIDCDefaultLoggedOutURL). The following video shows the module working.

   In my tests the only problem I have seen is that the mod-auth-openidc, although it is configured to refresh the token before expiration (OIDCRefreshAccessTokenBeforeExpiry is set to 10 seconds, so the access token is automatically refreshed when it is near to the expiration), if the refresh fails the session is maintained. In keycloak the session can be deleted (for example removed by an admin or just because it has reached its max life) and in the apache module it would be not detected. In my tests only the timeout (OIDCSessionInactivityTimeout set to 30 minutes) detects this, the local session in the apache is removed because of inactivity and this performs a new redirect/code-to-token flow that fails, and the user should log again into the system.

9. The mod-auth-openidc can be configured to also use plain OAUTH 2.0. This is the typical configuration for REST endpoints which just consume a bearer token and a full OIDC (code-to-token) is not needed. The module just checks the bearer token sent in the authentication and returns a OK (200) or an unauthorized error (401). In my configuration I decided to use the keycloak introspection endpoint to validate the token (it can also be configured locally, but this way is simpler).

   
   OIDCOAuthClientID mod-auth-oauth20
   OIDCOAuthClientSecret a78de633-5eb2-4ba9-abc2-ec33b86afe83
   OIDCOAuthIntrospectionEndpoint "https://debian.sample.com:8443/auth/realms/master/protocol/openid-connect/token/introspect"
   OIDCOAuthRemoteUserClaim preferred_username
   
   <Location /mod-auth-oauth20>
       AuthType oauth20
       Require valid-user
       LogLevel debug
   </Location>
   

   So another location /mod-auth-oauth20 is used to setup an oauth20 application. Here a simple hello world file is added, hello-world.php inside directory /var/www/html/mod-auth-oauth20:

   
   <?php
   header("Content-Type: text/plain");
   echo "Hello " . $_SERVER["REMOTE_USER"] . "!";
   ?>
   

10. Finally the idea is that the first location (which is using OIDC and has access to an access token that is refreshed automatically by the module 10 seconds before its expiration) can call to the bearer-only application. The call.php does exactly that and is located again in the first location /var/www/html/mod-auth-openidc:

    
    <?php
    // Get CURL resource
    $curl = curl_init();
    curl_setopt_array($curl, [
        CURLOPT_RETURNTRANSFER => 1,
        CURLOPT_URL => 'https://debian.sample.com/mod-auth-oauth20/hello-world.php',
        CURLOPT_HTTPHEADER => array('Authorization: Bearer ' . $_SERVER['OIDC_access_token'])
    ]);
    // Send the request & save response to $resp
    $resp = curl_exec($curl);
    // Close request to clear up some resources
    curl_close($curl);
    
    header("Content-Type: text/plain");
    echo $resp;
    ?>
    

    And this works, the following video shows that the endpoint returns a 401 error if called directly, but if we login into the OIDC location and access to the call page the hello world application is executed correctly. So everything is working as expected.

    In this case, the only problem is that the token in keycloak is very big (around 1KB) and it's not saved in the default cache (it seems there is a key limit of 512B), so the introspection call is performed always. In a real production scenario I would try to do a local validation or even using another cache. But it works OK for my testing setup.

And that is all. My idea was showing that the mod-auth-openidc and keycloak can work together quite nicely. My only particular snag is that the refresh of the token does not logout on error. If the access token is configured to automatically perform a refresh, if the refresh fails the session in the apache keeps maintained OK and used. That will bring issues for sure (for example the call to the hello-world endpoint will fail, because the token is expired) and I think it would be nice if the module can be configured to logout on a refresh error. I am trying to improve this so I asked into the google groups list, let's see if this goes to something fruitful.

Best regards!

One of the things that took me more time to understand when starting to use Wildfly was how caching worked for authentication. Besides the functionality is quite different comparing it to the old JBoss-AS so I think that it is good to summarize my findings here. I know how I am, and I will forget part of the idea if I do not use it again in the next months, and I will have doubts that will make me test it again. So better if I spend some time writing the current entry.

In the pre-wildfly times the web part was based in the jbossweb subsystem, which was mainly Tomcat. In Tomcat (at least in its default functionality, I do not know if you can change that default behavior) when a login happens because of a security-constraint the information is stored in the session. Therefore, once the session is authenticated (no matter the mechanism used to do it -BASIC, FORM, KERBEROS,...-) there are no more login calls. The principal associated with the request is retrieved from the session and no more authentication processes are executed. This behavior has the consequence that the requests that use no session (for example static files that does not create a session) are authenticated in every request until the session is created and used as a cache for the authentication information (take this on mind).

But when Wildfly entered in the game, the web part was renewed with the undertow project and, to my surprise, the previous behavior was completely changed (and with no documentation about it). Now the basic idea is that undertow always, and always means always, performs a verification of the account. For that it uses the interface IdentityManager that is called every time to refresh the account. For example looking into the BasicAuthenticationMechanism, the class that manages the BASIC authentication, this line performs the verification of the header. Does it mean that now Wildfly always logs the user in at every request? Bombarding the LDAP or Database to check the password and obtaining the roles all the time? Well, in theory yes, but in the end there are other caches to avoid this.

I am going to focus the entry in the old security subsystem (although I think that elytron is more or less the same). If the final authentication uses a security-domain there is a JAAS stack of login modules that performs the real authentication of the user (LDAP, files, database or any module that is finally authenticating the user). The security domain has a cache-type property, there is a default value that has a static cache of entries using LIRS strategy (there is also a infinispan type that lets configure more specific cache settings). So, using this cache, although undertow performs the verification in every request, when the process finally enters in the security domain code this cache is in place and no real authentication is performed. But, important to remember, if the cache-type is removed in the security domain configuration, every request is authenticated again and again.

At this point the things seem to be clear, now Wildfly authenticates always the request (if security is needed) but there is a cache that can be configured to avoid flooding the user store. Using a more specific cache like infinispan the application can even realize about changes in the final user store (password changed or new roles associated). But I have been talking about Basic authentication until now, using Basic the username and password are sent by the browser in every request, but... What happens with other mechanisms? For example, when using a Form to login, the username and password are only available when the login page is displayed to the user. What does Wildfly do in those cases?

Looking the code undertow performs another trick for those mechanisms. The FormAuthenticationMechanism registers the account when the login is really executed and it is available from there for the subsequent requests (the last parameter true has the effect to cache the account). Once the account is registered by the form mechanism another CachedAuthenticatedSessionMechanism enters in the game and performs the verification of the previous account. So, in summary, if the application is using a mechanism that (by design) has no access to the account in every request (for example Basic can do that but not Form), the mechanism implementation itself should register the account (as the Form mechanism does) and the cached mechanism is the one in charge of getting the cached account and doing its verification. But, important again, even in Form the account is validated in every request (the previously commented cache in the security domain should be configured to avoid this).

So, in the end, the summary is just the following three points:

1. Wildfly starts the authentication process for every request that should be authenticated (information is not stored in session like in the pre-wildfly versions).

2. Even mechanisms that cannot naturally do that (because the username/password pair is not available in every request, Form for example) starts the login in every request.

3. In order to avoid this constant re-login you need to configure cache in the security domain (default or infinispan type).

As I said, the new elytron has not been considered, I do not have all the details but I think that it is more or less the same. You can check it if you are curious about how the new security subsystem works. In case of a security domain the cache is usually configured to the default type and, therefore, there is usually no problem (but better if we understand what is happening under the hood). Hope it helps to somebody else. I think that it is incredible that this kind of changes are not reflected in any documentation or specification paper.

Regards!

Just a very quick entry today. The previous weekend I wasted a lot of time trying to upgrade my phone to LineageOS 16. In general I followed the upgrading guide for my phone model but when I started the adb sideload it always failed with the error: "Error applying update: 7(ErrorCode:: kInstallDeviceOpenError)". I tried Friday evening with no success, I was searching for information a couple of hours at Saturday and I finally got it working on Sunday. I think summarizing what was needed in the blog would be better, maybe it is useful for someone else.

In general the reason seems to be that tissot (my phone model, Xiaomi Mi A1) needs something special in the TWRP boot loader. Finally I got it working after reading this beautiful issue. The problem is explained by the maintainer himself in this thread. My model, unlike most of the usual A/B devices, has a weird partitioning where most partitions are not slotted (except of boot, system, modem). By default, TWRP does not expect to have any non-slotted partition in the A/B OTA package. That is why the installation fails on all TWRP packages except the one provided by the maintainer. So, if having this model, you need to download that exact TWRP image in order to upgrade the system.

After using the correct TWRP version everything went successfully and I have LineageOS 16 in my phone. The main problem was finding the issue, I needed to google a lot until I found that page which described my exact problem.

Regards!