HOWTO - Build a Keycloak/Ubuntu/MariaDB Cluster Without Multicast UDP
I’ve been trying to learn more about Keycloak lately but two things kept frustrating me; a lot of the information available online doesn’t work and cloud providers blocking multicast UDP
. I lost my notes once too many and decided to document the whole process here for future reference. I used jboss-cli.sh to edit standalone-ha.xml to make it easier to automate with configuration managers. So lets begin.
Create two or more servers in your favorite cloud provider, make sure to set up a private network. For the purpose of this HOWTO, we have the following:
- Keycloak 14.0.0, which is the latest version as of the time of this writing
- Two servers; keycloak1 and keycloak2. Running Ubuntu 20.04. Any other OS should work with minor modifications.
- MariaDB 10.3, running on keycloak1
- haproxy, running on keycloak1
- The private IP for keycloak1 is 172.16.0.2
- The private IP for keycloak2 is 172.16.0.3
Server preparation
Upgrade Ubuntu, then reboot
1apt update
2apt full-upgrade
3apt autoremove
4reboot
Add the following entries to /etc/hosts on both servers. This enables the servers to communicate with each other over the private LAN using names instead of IPs. This is optional but I find it makes configuration files a bit easier to comprehend.
1172.16.0.2 keycloak1
2172.16.0.3 keycloak2
On keycloak1, install MariaDB, make sure to edit /etc/mysql/mariadb.conf.d/50-server.cnf and set bind-address to 172.16.0.2
1apt install mariadb-server
Create the keycloak database, set the character set and collation because Keycloak will fail to startup without
1CREATE DATABASE keycloak CHARACTER SET latin1 COLLATE latin1_swedish_ci;
2GRANT ALL ON keycloak.* TO keycloak@'172.16.0.%' identified by 'keycloak';
If you don’t set the character set and collation, you will run into the following error
1Change Set META-INF/jpa-changelog-1.9.1.xml::1.9.1::keycloak failed. Error: Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual
Install OpenJDK 8 and download Keycloak to /opt
1apt install openjdk-8-jdk-headless
2
3cd /opt
4wget https://github.com/keycloak/keycloak/releases/download/14.0.0/keycloak-14.0.0.tar.gz
5tar zxvf keycloak-14.0.0.tar.gz
Keycloak preparation
Download and set up the MySQL connector
1mkdir -p /opt/keycloak-14.0.0/modules/system/layers/keycloak/com/mysql/main
2
3tar zxvf mysql-connector-java-8.0.26.tar.gz
4cp mysql-connector-java-8.0.26/mysql-connector-java-8.0.26.jar /opt/keycloak-14.0.0/modules/system/layers/keycloak/com/mysql/main
Create /opt/keycloak-14.0.0/modules/system/layers/keycloak/com/mysql/main/module.xml
1<?xml version="1.0" ?>
2<module xmlns="urn:jboss:module:1.3" name="com.mysql">
3 <resources>
4 <resource-root path="mysql-connector-java-8.0.26.jar" />
5 </resources>
6 <dependencies>
7 <module name="javax.api"/>
8 <module name="javax.transaction.api"/>
9 </dependencies>
10</module>
Now that the MySQL driver is ready, we need to tell Keycloak to load it. Create driver.cli
1embed-server --server-config=standalone-ha.xml -c
2
3# Add mysql driver if it doesn't already exist
4if (outcome != success) of /subsystem=datasources/jdbc-driver=mysql:read-resource
5 /subsystem=datasources/jdbc-driver=mysql:add(driver-name=mysql,\
6 driver-module-name=com.mysql,\
7 driver-class-name=com.mysql.cj.jdbc.Driver,\
8 driver-xa-datasource-class-name=com.mysql.cj.jdbc.MysqlXADataSource)
9end-if
10
11quit
Load the MySQL driver.
1/opt/keycloak-14.0.0/bin/jboss-cli.sh --file=driver.cli
Now to define the datasource. Create datasource.cli
1embed-server --server-config=standalone-ha.xml -c
2
3# Remove old database connection if it exists
4if (outcome == success) of /subsystem=datasources/data-source=KeycloakDS:read-resource
5 data-source remove --name=KeycloakDS
6end-if
7
8# Add new database connection if it does not exist
9if (outcome != success) of /subsystem=datasources/xa-data-source=KeycloakDS:read-resource
10 xa-data-source add \
11 --name=KeycloakDS \
12 --driver-name=mysql \
13 --jndi-name=java:jboss/datasources/KeycloakDS \
14 --user-name=keycloak \
15 --password="keycloak" \
16 --valid-connection-checker-class-name=org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLValidConnectionChecker \
17 --exception-sorter-class-name=org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter
18
19 /subsystem=datasources/xa-data-source=KeycloakDS/xa-datasource-properties=ServerName:add(value=keycloak1)
20 /subsystem=datasources/xa-data-source=KeycloakDS/xa-datasource-properties=DatabaseName:add(value=keycloak)
21end-if
22
23quit
Load the datasource.
1/opt/keycloak-14.0.0/bin/jboss-cli.sh --file=datasource.cli
Create /opt/jboss.properties. This file will allow us to define variables in standalone-ha.xml. The values below are for keycloak1, substitute keycloak1 with keycloak2 and 172.16.0.2 with 172.16.0.3 for keycloak2.
1jboss.server.name=keycloak
2jboss.node.name=keycloak1
3jboss.bind.address=172.16.0.2
4jboss.bind.address.private=172.16.0.2
Now lets run keycloak on keycloak1 and keycloak2 independently to see if it starts. When Keycloak runs for the first time it creates the required tables in MySQL.
1/opt/keycloak-14.0.0/bin/standalone.sh --server-config=standalone-ha.xml --properties=/opt/jboss.properties
If you see the below then you are good to proceed. Otherwise go back and see what is wrong
1INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
Add an admin user to be able to log in to the master realm
1/opt/keycloak-14.0.0/bin/add-user-keycloak.sh -u admin
Cluster configuration
Now that we have Keycloak working, it’s time to configure clustering. Keycloak uses multicast UDP by default which is blocked by cloud providers. I will be explaining three options here, only one of them needs to be implemented.
- TCPPING, where you define all hosts statically in standalone-ha.xml. Can easily be automated with your configuration manager. This is my personal preference.
- DNS_PING, uses an A or SRV record to point to the private IPs of the cluster members. I haven’t looked at how it would work with SRV and will explain how to use the A record. This would usually be used with Kubernetes.
- JDBC_PING, uses a database table to keep track of cluster nodes, not my favorite method but keeping it here as an option. I have run into cases where nodes would not properly clean up the table when shutting down. Bringing the cluster back up would involve shutting down all nodes, truncating the table and then starting all nodes back up.
TCPPING
Create tcpping.cli
1embed-server --server-config=standalone-ha.xml
2
3if (outcome == success) of /subsystem=jgroups/stack=tcpping:read-resource
4 /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
5 /subsystem=jgroups/stack=tcpping:remove()
6end-if
7
8/subsystem=jgroups/stack=tcpping:add
9/subsystem=jgroups/stack=tcpping/transport=TCP:add(socket-binding=jgroups-tcp)
10/subsystem=jgroups/stack=tcpping/protocol=TCPPING:add
11/subsystem=jgroups/stack=tcpping/protocol=TCPPING/property=initial_hosts:add(value=${initial.hosts:127.0.0.1[7600]})
12/subsystem=jgroups/stack=tcpping/protocol=TCPPING/property=port_range:add(value=0)
13/subsystem=jgroups/stack=tcpping/protocol=MERGE3:add
14/subsystem=jgroups/stack=tcpping/protocol=FD_SOCK:add(socket-binding=jgroups-tcp-fd)
15/subsystem=jgroups/stack=tcpping/protocol=FD:add
16/subsystem=jgroups/stack=tcpping/protocol=VERIFY_SUSPECT:add
17/subsystem=jgroups/stack=tcpping/protocol=pbcast.NAKACK2:add
18/subsystem=jgroups/stack=tcpping/protocol=UNICAST3:add
19/subsystem=jgroups/stack=tcpping/protocol=pbcast.STABLE:add
20/subsystem=jgroups/stack=tcpping/protocol=pbcast.GMS:add
21/subsystem=jgroups/stack=tcpping/protocol=MFC:add
22/subsystem=jgroups/stack=tcpping/protocol=FRAG2:add
23/subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcpping)
24quit
Load the TCPPING configuration.
1/opt/keycloak-14.0.0/bin/jboss-cli.sh --file=tcpping.cli
Add the line below to /opt/jboss.properties
1initial.hosts=keycloak1[7600],keycloak2[7600]
DNSPING
Create dnsping.cli
1embed-server --server-config=standalone-ha.xml
2
3if (outcome == success) of /subsystem=jgroups/stack=dnsping:read-resource
4 /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
5 /subsystem=jgroups/stack=dnsping:remove()
6end-if
7
8if (outcome == success) of /socket-binding-group=standard-sockets/socket-binding=jgroups-dnsping:read-resource
9 /socket-binding-group=standard-sockets/socket-binding=jgroups-dnsping:remove()
10end-if
11/socket-binding-group=standard-sockets/socket-binding=jgroups-dnsping:add(interface="private")
12
13/subsystem=jgroups/stack=dnsping:add
14/subsystem=jgroups/stack=dnsping/transport=TCP:add(socket-binding=jgroups-tcp)
15/subsystem=jgroups/stack=dnsping/protocol=dns.DNS_PING:add(socket-binding=jgroups-dnsping)
16/subsystem=jgroups/stack=dnsping/protocol=dns.DNS_PING/property=dns_query:add(value=${dns.query:127.0.0.1})
17/subsystem=jgroups/stack=dnsping/protocol=MERGE3:add
18/subsystem=jgroups/stack=dnsping/protocol=FD_SOCK:add(socket-binding=jgroups-tcp-fd)
19/subsystem=jgroups/stack=dnsping/protocol=FD:add
20/subsystem=jgroups/stack=dnsping/protocol=VERIFY_SUSPECT:add
21/subsystem=jgroups/stack=dnsping/protocol=pbcast.NAKACK2:add
22/subsystem=jgroups/stack=dnsping/protocol=UNICAST3:add
23/subsystem=jgroups/stack=dnsping/protocol=pbcast.STABLE:add
24/subsystem=jgroups/stack=dnsping/protocol=pbcast.GMS:add
25/subsystem=jgroups/stack=dnsping/protocol=MFC:add
26/subsystem=jgroups/stack=dnsping/protocol=FRAG2:add
27/subsystem=jgroups/channel=ee:write-attribute(name=stack,value=dnsping)
28quit
Load the DNSPING configuration.
1/opt/keycloak-14.0.0/bin/jboss-cli.sh --file=dnsping.cli
Add the line below to /opt/jboss.properties
1dns.query=HOSTNAME
HOSTNAME needs to point to the private IPs of all your Keycloak servers, assuming keycloak.example.com
1dig keycloak.example.com
2
3; <<>> DiG 9.16.1-Ubuntu <<>> keycloak.example.com
4;; global options: +cmd
5;; Got answer:
6;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41774
7;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
8
9;; OPT PSEUDOSECTION:
10; EDNS: version: 0, flags:; udp: 65494
11;; QUESTION SECTION:
12;keycloak.example.com. IN A
13
14;; ANSWER SECTION:
15keycloak.example.com. 42 IN A 172.16.0.2
16keycloak.example.com. 42 IN A 172.16.0.3
17
18;; Query time: 0 msec
19;; SERVER: 127.0.0.53#53(127.0.0.53)
20;; WHEN: Sun Jul 25 05:13:48 CEST 2021
21;; MSG SIZE rcvd: 87
You can either set up a public hostname or configure a private DNS server to serve the required record. I will not go into details of the process here.
JDBC_PING
create jdbcping.cli
1embed-server --server-config=standalone-ha.xml
2
3if (outcome == success) of /subsystem=jgroups/stack=jdbcping:read-resource
4 /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
5 /subsystem=jgroups/stack=jdbcping:remove()
6end-if
7
8/subsystem=jgroups/stack=jdbcping:add
9/subsystem=jgroups/stack=jdbcping/transport=TCP:add(socket-binding=jgroups-tcp)
10/subsystem=jgroups/stack=jdbcping/protocol=JDBC_PING:add(data-source="KeycloakDS")
11
12/subsystem=jgroups/stack=jdbcping/protocol=JDBC_PING/property=datasource_jndi_name:add(value="java:jboss/datasources/KeycloakDS")
13
14/subsystem=jgroups/stack=jdbcping/protocol=JDBC_PING/property=initialize_sql:add(value="CREATE TABLE JGROUPSPING (own_addr varchar(200) NOT NULL, bind_addr varchar(200) NOT NULL, created timestamp NOT NULL, cluster_name varchar(200) NOT NULL, ping_data blob, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name))")
15/subsystem=jgroups/stack=jdbcping/protocol=JDBC_PING/property=insert_single_sql:add(value="INSERT INTO JGROUPSPING (own_addr, bind_addr, created, cluster_name, ping_data) values (?,'${jgroups.bind.address:127.0.0.1}',NOW(), ?, ?)")
16/subsystem=jgroups/stack=jdbcping/protocol=JDBC_PING/property=delete_single_sql:add(value="DELETE FROM JGROUPSPING WHERE own_addr=? AND cluster_name=?")
17/subsystem=jgroups/stack=jdbcping/protocol=JDBC_PING/property=select_all_pingdata_sql:add(value="SELECT ping_data FROM JGROUPSPING WHERE cluster_name=?")
18/subsystem=jgroups/stack=jdbcping/protocol=MERGE3:add
19/subsystem=jgroups/stack=jdbcping/protocol=FD_SOCK:add(socket-binding=jgroups-tcp-fd)
20/subsystem=jgroups/stack=jdbcping/protocol=FD:add
21/subsystem=jgroups/stack=jdbcping/protocol=VERIFY_SUSPECT:add
22/subsystem=jgroups/stack=jdbcping/protocol=pbcast.NAKACK2:add
23/subsystem=jgroups/stack=jdbcping/protocol=UNICAST3:add
24/subsystem=jgroups/stack=jdbcping/protocol=pbcast.STABLE:add
25/subsystem=jgroups/stack=jdbcping/protocol=pbcast.GMS:add
26/subsystem=jgroups/stack=jdbcping/protocol=MFC:add
27/subsystem=jgroups/stack=jdbcping/protocol=FRAG2:add
28/subsystem=jgroups/channel=ee:write-attribute(name=stack,value=jdbcping)
29quit
Testing
Once you are done with either method above, start Keycloak on both nodes
1/opt/keycloak-14.0.0/bin/standalone.sh --server-config=standalone-ha.xml --properties=/opt/jboss.properties
Verify that you see the below in the log
1INFO [org.infinispan.CLUSTER] (thread-5,ejb,keycloak1) ISPN000094: Received new cluster view for channel ejb: [keycloak1|1] (2) [keycloak1, keycloak2]
2INFO [org.infinispan.CLUSTER] (thread-14,ejb,keycloak2) [Context=authenticationSessions] ISPN100010: Finished rebalance with members [keycloak2, keycloak1], topology id 5
Session Management
Now that the cluster is up, it’s time to set up load balancing. According to the Keycloak 14.0 Documentation , Keycloak uses Inifispan to handle logged in sessions, so for performance reasons it’s better to enable sticky sessions on the load balancer. One drawback to the default configuration is if one node goes down all sessions handled by that node will be lost.
To solve this we need to increase the number of session owners. Create session.cli
1embed-server --server-config=standalone-ha.xml
2
3/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions:write-attribute(name=owners,value=${owner.count:1})
4/subsystem=infinispan/cache-container=keycloak/distributed-cache=authenticationSessions:write-attribute(name=owners,value=${owner.count:1})
5/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions:write-attribute(name=owners,value=${owner.count:1})
6/subsystem=infinispan/cache-container=keycloak/distributed-cache=clientSessions:write-attribute(name=owners,value=${owner.count:1})
7/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineClientSessions:write-attribute(name=owners,value=${owner.count:1})
8/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures:write-attribute(name=owners,value=${owner.count:1})
9/subsystem=infinispan/cache-container=keycloak/distributed-cache=actionTokens:write-attribute(name=owners,value=${owner.count:1})
10
11quit
Load the new session configuration.
1/opt/keycloak-14.0.0/bin/jboss-cli.sh --file=session.cli
Add the line below to /opt/jboss.properties. Note that I kept the default session owner to 1.
1owner.count=2
We also need to configure Keycloak to work behind a reverse proxy. Create listener.cli
1embed-server --server-config=standalone-ha.xml
2
3/subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=proxy-address-forwarding, value=true)
4
5quit
Update the http-listener configuration.
1/opt/keycloak-14.0.0/bin/jboss-cli.sh --file=listener.cli
HAProxy configuration
1apt install haproxy
Edit /etc/haproxy/haproxy.cfg. Check out this article
to see how I set up certificates.
1global
2 log /dev/log local0
3 log /dev/log local1 notice
4 chroot /var/lib/haproxy
5 user haproxy
6 group haproxy
7 daemon
8
9 # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
10 ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
11 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
12 ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
13
14 # Generated using `openssl dhparam -out /etc/haproxy/dhparams.pem 2048`
15 ssl-dh-param-file /etc/haproxy/dhparams.pem
16
17defaults
18 log global
19 mode http
20 option httplog
21 option dontlognull
22 timeout connect 500
23 timeout client 5000
24 timeout server 5000
25
26frontend terminator
27 bind KEYCLOAK1_PUBLIC_IP:80
28 bind KEYCLOAK1_PUBLIC_IP:443 ssl crt-list /etc/haproxy/certs alpn h2,http/1.1
29
30 # LetsEncrypt
31 acl acme path_dir /.well-known/acme-challenge
32
33 http-response set-header Strict-Transport-Security max-age=15768000 if { ssl_fc }
34 redirect scheme https code 301 if !{ ssl_fc } host_https !acme
35 http-request set-header X-Forwarded-Proto https if { ssl_fc }
36
37 use_backend acme if acme
38
39 default_backend keycloak
40
41backend keycloak
42 # Add the SERVERID cookie to allow sticky sessions
43 cookie SERVERID insert indirect
44
45 # LIGHT and DARK are just identifiers so the internal structure is not exposed. You can use different values
46 server keycloak1 172.16.0.2:8080 check cookie LIGHT
47 server keycloak2 172.16.0.3:8080 check cookie DARK
48
49backend acme
50 server acmetool 127.0.0.1:402
Reload HAProxy, Ubuntu starts it after installation by default
1systemctl reload haproxy
Startup script
Only thing left is to configure Keycloak as a service
1[Unit]
2Description=Keycloak Application Server
3After=remote-fs.target syslog.target network.target
4
5[Install]
6WantedBy=multi-user.target
7
8[Service]
9User=jboss
10Group=jboss
11ExecStart=/opt/keycloak-14.0.0/bin/standalone.sh --server-config=standalone-ha.xml --properties=/opt/jboss.properties
12
13Restart=on-failure
Now you can use this article to generate tokens.
About Me
Dev gone Ops gone DevOps. Any views expressed on this blog are mine alone and do not necessarily reflect the views of my employer.
Recent Posts