Bootstrap, Join, State Snapshot Transfer, Incremental State Transfer, and Donor Selection

Learning outcomes

A Galera node that restarts is not useful merely because mariadbd started. It must recover the exact cluster state it missed. If a donor still has the missing write sets in its GCache (Galera write-set cache), the node can perform an Incremental State Transfer (IST). If the gap is not recoverable from GCache—or the joiner has no compatible state—it needs a full State Snapshot Transfer (SST). Choosing the wrong node to bootstrap after a complete outage is more dangerous than either transfer: it can intentionally declare stale data authoritative.

1. Bootstrap, join, and rejoin are three different operations

2. Build the disposable cluster

services:  n1:    image: mariadb:12.3.2    container_name: mdb15-n1    environment:      MARIADB_ROOT_PASSWORD: labroot    command:      - --bind-address=0.0.0.0      - --binlog-format=ROW      - --default-storage-engine=InnoDB      - --innodb-autoinc-lock-mode=2      - --wsrep-on=ON      - --wsrep-provider=/usr/lib/galera/libgalera_smm.so      - --wsrep-cluster-name=servicehub15      - --wsrep-cluster-address=gcomm://n2,n3      - --wsrep-node-name=n1      - --wsrep-sst-method=rsync      - --wsrep-new-cluster    ports:      - "33151:3306"  n2:    image: mariadb:12.3.2    container_name: mdb15-n2    environment:      MARIADB_ROOT_PASSWORD: labroot    command:      - --bind-address=0.0.0.0      - --binlog-format=ROW      - --default-storage-engine=InnoDB      - --innodb-autoinc-lock-mode=2      - --wsrep-on=ON      - --wsrep-provider=/usr/lib/galera/libgalera_smm.so      - --wsrep-cluster-name=servicehub15      - --wsrep-cluster-address=gcomm://n1      - --wsrep-node-name=n2      - --wsrep-sst-method=rsync    ports:      - "33152:3306"  n3:    image: mariadb:12.3.2    container_name: mdb15-n3    environment:      MARIADB_ROOT_PASSWORD: labroot    command:      - --bind-address=0.0.0.0      - --binlog-format=ROW      - --default-storage-engine=InnoDB      - --innodb-autoinc-lock-mode=2      - --wsrep-on=ON      - --wsrep-provider=/usr/lib/galera/libgalera_smm.so      - --wsrep-cluster-name=servicehub15      - --wsrep-cluster-address=gcomm://n1      - --wsrep-node-name=n3      - --wsrep-sst-method=rsync    ports:      - "33153:3306"

docker compose down -v --remove-orphans# Bootstrap exactly one node for a brand-new disposable cluster.docker compose up -d n1# Wait until n1 reports wsrep_ready=ON, then join the other nodes.docker compose up -d n2 n3# Verify all three nodes before creating application data.docker exec mdb15-n1 mariadb -uroot -plabroot -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'; SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';"docker exec mdb15-n2 mariadb -uroot -plabroot -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';"docker exec mdb15-n3 mariadb -uroot -plabroot -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';"

CREATE DATABASE IF NOT EXISTS servicehub_galera_lab;CREATE TABLE IF NOT EXISTS servicehub_galera_lab.tickets (  ticket_id BIGINT PRIMARY KEY AUTO_INCREMENT,  customer_id BIGINT NOT NULL,  state VARCHAR(24) NOT NULL,  priority INT NOT NULL DEFAULT 0,  updated_at TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6)             ON UPDATE CURRENT_TIMESTAMP(6)) ENGINE=InnoDB;CREATE TABLE IF NOT EXISTS servicehub_galera_lab.counters (  counter_name VARCHAR(64) PRIMARY KEY,  counter_value BIGINT NOT NULL) ENGINE=InnoDB;INSERT INTO servicehub_galera_lab.counters(counter_name,counter_value)VALUES ('dispatch',0)ON DUPLICATE KEY UPDATE counter_value=VALUES(counter_value);INSERT INTO servicehub_galera_lab.tickets(customer_id,state,priority) VALUES (301,'open',1),(302,'open',2);

SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_local_state_comment','wsrep_local_cached_downto','wsrep_last_committed','wsrep_cluster_size');

wsrep_local_cached_downto is the lowest sequence number the local GCache can currently serve for IST. GCache sizing is workload/time-window dependent; do not convert a default size into a universal “hours of outage” promise.

3. Short outage: observe IST

Stop n3 but keep its container filesystem. Advance the cluster from n1 with a modest number of transactions so the donor's GCache still contains the gap.

docker stop mdb15-n3

INSERT INTO servicehub_galera_lab.tickets(customer_id,state,priority)SELECT 400+n, 'queued', MOD(n,5)FROM (SELECT 1 n UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5) AS x;SHOW GLOBAL STATUS LIKE 'wsrep_last_committed';

docker start mdb15-n3docker logs --since 2m mdb15-n3 2>&1 | grep -Ei "IST|SST|state transfer|ready" || true

SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_local_state_comment','wsrep_ready','wsrep_last_committed');SELECT COUNT(*) AS ticket_count FROM servicehub_galera_lab.tickets;

The expected path is IST if the missing write sets are still in a donor GCache and the cluster state UUID is compatible. The exact log wording is version/provider dependent; the acceptance criterion is that n3 returns to Synced with the same application state.

4. Empty joiner: force the need for SST

An empty node has no prior state to incrementally extend. Recreate n3 without its old container filesystem and start it normally. Galera must obtain a full state before it can become Synced.

docker rm -f mdb15-n3docker compose up -d n3docker logs --since 3m mdb15-n3 2>&1 | grep -Ei "SST|state transfer|ready" || true

SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_local_state_comment','wsrep_ready','wsrep_cluster_size');SELECT COUNT(*) AS ticket_count FROM servicehub_galera_lab.tickets;

With rsync, donor availability behavior differs from mariabackup. Production selection must account for database size, donor blocking, transfer bandwidth, encryption, backup-tool version compatibility, and the possibility that a joiner restart triggers a full transfer during peak load.

5. Donor selection is a capacity decision

Galera can choose a donor automatically. wsrep_sst_donor lets a joiner express preferred node names or lists. A preferred donor is not automatically the safest donor: it may be slow, already serving heavy traffic, behind in local apply, or separated by a weak network path.

SHOW GLOBAL VARIABLES WHERE Variable_name IN ('wsrep_node_name','wsrep_sst_method','wsrep_sst_donor');SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_local_state_comment','wsrep_ready','wsrep_local_recv_queue_avg');

When using mariabackup SST, current MariaDB documentation requires local donor authentication with backup privileges unless the exact product/version provides an alternative automatic mechanism. Community labs should not silently copy Enterprise-only account-management behavior.

6. GCache: outage window, not backup

GCache stores recent write sets so a returning node can request only what it missed. It is operational acceleration, not durable business recovery. It can be too small for a long outage, can be lost or become unavailable, and does not protect against logical mistakes replicated to every node.

7. Full-cluster outage: choose authority from state evidence

After a graceful full shutdown, Galera writes saved state in grastate.dat. The node marked safe_to_bootstrap: 1 is the intended starting point. After a hard crash, all nodes can show unsafe/unknown positions. Then run mariadbd --wsrep-recover on each stopped node and compare recovered sequence positions before selecting the most advanced node.

docker stop mdb15-n1 mdb15-n2 mdb15-n3docker exec mdb15-n1 sh -lc "cat /var/lib/mysql/grastate.dat"docker exec mdb15-n2 sh -lc "cat /var/lib/mysql/grastate.dat"docker exec mdb15-n3 sh -lc "cat /var/lib/mysql/grastate.dat"

If you intentionally practice a full restart, bootstrap only the chosen most-advanced node, then start the remaining nodes normally. Never edit safe_to_bootstrap merely to silence an error without proving why that node is authoritative.

8. Production judgment

• Provision enough GCache for your measured write-set rate and expected maintenance/outage window, then monitor actual cached range.
• Prefer IST for routine short outages; plan SST as a normal, resource-heavy fallback rather than an exceptional surprise.
• Select SST method against exact version/engine/encryption requirements; mariabackup SST has backup-tool compatibility and credential considerations.
• Document bootstrap authority and rehearse full-cluster recovery. The safest node is determined from state, not hostname conventions such as “node1.”
• Keep tested Chapter 13 backups even with Galera. A cluster synchronizes state; it does not preserve every historical state you may need.

Next, stress the cluster with conflicting writers and slower apply so you can see certification failures, auto-increment spacing, and flow-control backpressure rather than memorizing them.

docker compose down -v --remove-orphans