Galera Security, TLS, Rolling Maintenance, Upgrades, Failure Injection, and Recovery

Learning outcomes

A three-node cluster that survives failures but sends replication and State Snapshot Transfer data in clear text, runs untested rolling upgrades, or has no full-outage recovery drill is not production-ready. This lesson turns the previous four mechanisms into an operating procedure: secure every traffic plane, change one node at a time, verify provider/server compatibility, inject failures, and recover only from proven authoritative state.

1. Start with a clean cluster and record the compatibility packet

services:  n1:    image: mariadb:12.3.2    container_name: mdb15-n1    environment:      MARIADB_ROOT_PASSWORD: labroot    command:      - --bind-address=0.0.0.0      - --binlog-format=ROW      - --default-storage-engine=InnoDB      - --innodb-autoinc-lock-mode=2      - --wsrep-on=ON      - --wsrep-provider=/usr/lib/galera/libgalera_smm.so      - --wsrep-cluster-name=servicehub15      - --wsrep-cluster-address=gcomm://n2,n3      - --wsrep-node-name=n1      - --wsrep-sst-method=rsync      - --wsrep-new-cluster    ports:      - "33151:3306"  n2:    image: mariadb:12.3.2    container_name: mdb15-n2    environment:      MARIADB_ROOT_PASSWORD: labroot    command:      - --bind-address=0.0.0.0      - --binlog-format=ROW      - --default-storage-engine=InnoDB      - --innodb-autoinc-lock-mode=2      - --wsrep-on=ON      - --wsrep-provider=/usr/lib/galera/libgalera_smm.so      - --wsrep-cluster-name=servicehub15      - --wsrep-cluster-address=gcomm://n1      - --wsrep-node-name=n2      - --wsrep-sst-method=rsync    ports:      - "33152:3306"  n3:    image: mariadb:12.3.2    container_name: mdb15-n3    environment:      MARIADB_ROOT_PASSWORD: labroot    command:      - --bind-address=0.0.0.0      - --binlog-format=ROW      - --default-storage-engine=InnoDB      - --innodb-autoinc-lock-mode=2      - --wsrep-on=ON      - --wsrep-provider=/usr/lib/galera/libgalera_smm.so      - --wsrep-cluster-name=servicehub15      - --wsrep-cluster-address=gcomm://n1      - --wsrep-node-name=n3      - --wsrep-sst-method=rsync    ports:      - "33153:3306"

docker compose down -v --remove-orphans# Bootstrap exactly one node for a brand-new disposable cluster.docker compose up -d n1# Wait until n1 reports wsrep_ready=ON, then join the other nodes.docker compose up -d n2 n3# Verify all three nodes before creating application data.docker exec mdb15-n1 mariadb -uroot -plabroot -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'; SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';"docker exec mdb15-n2 mariadb -uroot -plabroot -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';"docker exec mdb15-n3 mariadb -uroot -plabroot -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';"

SELECT VERSION() AS server_version;SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_provider_name','wsrep_provider_version','wsrep_protocol_version','wsrep_cluster_state_uuid','wsrep_cluster_status','wsrep_local_state_comment');SHOW GLOBAL VARIABLES WHERE Variable_name IN ('wsrep_provider','wsrep_sst_method','wsrep_cluster_name');

Store this packet with the maintenance change. “Same MariaDB major version” is not enough when Galera provider, SST tool, package source, operating system, or schema-upgrade path can differ.

2. Three traffic planes need three security stories

Encrypting client connections does not encrypt inter-node write sets. Encrypting wsrep replication does not automatically secure an rsync/mariabackup SST. Treat each plane independently and verify it.

3. Provider-level TLS pattern for Community Galera

Galera provider TLS uses wsrep_provider_options such as socket.ssl_ca, socket.ssl_cert, and socket.ssl_key. The certificate authority (CA) establishes trust; each node should have a private key protected by operating-system permissions and a certificate appropriate for the identities you validate.

[mariadb]ssl_ca=/etc/mysql/tls/ca.pemssl_cert=/etc/mysql/tls/node.pemssl_key=/etc/mysql/tls/node-key.pemwsrep_provider_options="socket.ssl=true;socket.ssl_ca=/etc/mysql/tls/ca.pem;socket.ssl_cert=/etc/mysql/tls/node.pem;socket.ssl_key=/etc/mysql/tls/node-key.pem"

These provider options are not dynamic in the ordinary case, so plan controlled restarts. Some versions/products support rolling TLS-transition mechanisms; verify exact documentation instead of assuming a mixed encrypted/unencrypted cluster will interoperate safely.

4. Secure SST explicitly

The lab used rsync for simplicity. MariaDB documents method-specific SST security: mariabackup can use TLS settings and needs donor-side backup authentication/privileges in Community-style manual setups; rsync commonly relies on an external secure transport such as stunnel when encryption is required.

Before production, answer four questions: Which SST method is used on every node? What credentials/privileges does it need? Is transfer encrypted and authenticated? What happens to donor availability during the transfer?

SHOW GLOBAL VARIABLES WHERE Variable_name IN ('wsrep_sst_method','wsrep_sst_auth','wsrep_sst_donor');

Never print real wsrep_sst_auth secrets into tickets or screenshots. Prefer secret files/socket authentication/automatic mechanisms only when your exact supported product version documents them.

5. Rolling maintenance runbook

One-node-at-a-time maintenance preserves quorum in a three-node cluster, but only if the remaining two nodes are healthy and sized for the workload.

Repeat for the next node only after the previous one is fully returned. Major-version upgrades require explicit release documentation; SST methods can have cross-major backup/redo compatibility constraints. A “rolling restart worked in staging” does not prove a rolling major upgrade is supported.

6. Safe node-failure injection

Kill n3 abruptly rather than stopping it gracefully. n1+n2 should retain quorum. Continue a small write, then restart n3 and require state convergence.

docker kill mdb15-n3docker exec mdb15-n1 mariadb -uroot -plabroot -e "SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_cluster_status','wsrep_cluster_size');"

INSERT INTO servicehub_galera_lab.tickets(customer_id,state,priority) VALUES (801,'open',3);

docker start mdb15-n3docker logs --since 2m mdb15-n3 2>&1 | grep -Ei "IST|SST|Synced|ready" || true

SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_cluster_status','wsrep_cluster_size','wsrep_ready','wsrep_local_state_comment');SELECT COUNT(*) FROM servicehub_galera_lab.tickets WHERE customer_id=801;

7. Full-cluster failure: this is where runbooks matter

To practice safely, first ensure no external clients use the lab, then stop all nodes. After graceful shutdown, inspect grastate.dat and choose the node marked safe to bootstrap. After a hard crash, use --wsrep-recover to compare recovered positions.

docker stop mdb15-n3 mdb15-n2 mdb15-n1docker exec mdb15-n1 cat /var/lib/mysql/grastate.datdocker exec mdb15-n2 cat /var/lib/mysql/grastate.datdocker exec mdb15-n3 cat /var/lib/mysql/grastate.dat

8. Wrong recovery: bootstrap stale state, then let newer nodes join

This shortcut can discard acknowledged history. Galera will treat the newly bootstrapped component as authoritative; a later node does not arrive as a competing truth source that automatically merges missing commits. The repair for an uncertain split/divergence is an incident: stop writes, identify accepted histories, restore/reconcile from backups/binlogs/business records as appropriate, and rebuild clean nodes.

This is also why Chapter 13 remains mandatory. Galera reduces some node-loss scenarios; it does not provide historical recovery from operator error, malware, logical corruption, or a bad bootstrap decision.

9. Security and maintenance evidence checklist

• Server version, package source, wsrep_provider_version, protocol version, SST tool/version recorded.
• Private cluster network/firewall restricts 4567/4568/4444 to intended members; wsrep_allowlist considered where supported.
• Client TLS and connector identity verification tested independently from inter-node TLS.
• Provider/SST certificates have ownership, rotation dates, monitoring, and rollback procedure.
• Backups are encrypted/off-site as required and restore-tested; cluster nodes are not counted as backup copies.
• Rolling change rehearsed with one node out and remaining quorum/capacity measured.
• Full outage bootstrap runbook names evidence/decision owner, not a permanently privileged “bootstrap node.”

10. Production judgment and bridge to Chapter 16

Operate Galera as a distributed system: compatibility, certificates, membership, flow control, state transfer, backups, and client routing all have owners and observable acceptance criteria. A safe maintenance window is not merely “restart nodes one at a time”; it is drain → prove quorum → change → rejoin → validate → return.

Chapter 16 moves from cluster availability to large-table lifecycle. Partitioning and online DDL can still create cluster-wide consequences: metadata operations, write-set volume, SST risk, and maintenance windows must be evaluated against the Galera topology you now understand.

docker compose down -v --remove-orphans