Load Balancing, Primary Component, Split Brain, Fencing, and Application Failover

Learning outcomes

ServiceHub's cluster is healthy, but the application still has three hostnames. A TCP load balancer that sends traffic to “any port 3306 that accepts a socket” can route writes to a node that is Non-Primary, still Joining, or donating a blocking SST. High availability therefore needs two cooperating control planes: Galera decides cluster membership/quorum; an external routing/fencing layer decides which database endpoints applications may use.

1. Start the lab and define an application readiness gate

services:  n1:    image: mariadb:12.3.2    container_name: mdb15-n1    environment:      MARIADB_ROOT_PASSWORD: labroot    command:      - --bind-address=0.0.0.0      - --binlog-format=ROW      - --default-storage-engine=InnoDB      - --innodb-autoinc-lock-mode=2      - --wsrep-on=ON      - --wsrep-provider=/usr/lib/galera/libgalera_smm.so      - --wsrep-cluster-name=servicehub15      - --wsrep-cluster-address=gcomm://n2,n3      - --wsrep-node-name=n1      - --wsrep-sst-method=rsync      - --wsrep-new-cluster    ports:      - "33151:3306"  n2:    image: mariadb:12.3.2    container_name: mdb15-n2    environment:      MARIADB_ROOT_PASSWORD: labroot    command:      - --bind-address=0.0.0.0      - --binlog-format=ROW      - --default-storage-engine=InnoDB      - --innodb-autoinc-lock-mode=2      - --wsrep-on=ON      - --wsrep-provider=/usr/lib/galera/libgalera_smm.so      - --wsrep-cluster-name=servicehub15      - --wsrep-cluster-address=gcomm://n1      - --wsrep-node-name=n2      - --wsrep-sst-method=rsync    ports:      - "33152:3306"  n3:    image: mariadb:12.3.2    container_name: mdb15-n3    environment:      MARIADB_ROOT_PASSWORD: labroot    command:      - --bind-address=0.0.0.0      - --binlog-format=ROW      - --default-storage-engine=InnoDB      - --innodb-autoinc-lock-mode=2      - --wsrep-on=ON      - --wsrep-provider=/usr/lib/galera/libgalera_smm.so      - --wsrep-cluster-name=servicehub15      - --wsrep-cluster-address=gcomm://n1      - --wsrep-node-name=n3      - --wsrep-sst-method=rsync    ports:      - "33153:3306"

docker compose down -v --remove-orphans# Bootstrap exactly one node for a brand-new disposable cluster.docker compose up -d n1# Wait until n1 reports wsrep_ready=ON, then join the other nodes.docker compose up -d n2 n3# Verify all three nodes before creating application data.docker exec mdb15-n1 mariadb -uroot -plabroot -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'; SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';"docker exec mdb15-n2 mariadb -uroot -plabroot -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';"docker exec mdb15-n3 mariadb -uroot -plabroot -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';"

CREATE DATABASE IF NOT EXISTS servicehub_galera_lab;CREATE TABLE IF NOT EXISTS servicehub_galera_lab.tickets (  ticket_id BIGINT PRIMARY KEY AUTO_INCREMENT,  customer_id BIGINT NOT NULL,  state VARCHAR(24) NOT NULL,  priority INT NOT NULL DEFAULT 0,  updated_at TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6)             ON UPDATE CURRENT_TIMESTAMP(6)) ENGINE=InnoDB;CREATE TABLE IF NOT EXISTS servicehub_galera_lab.counters (  counter_name VARCHAR(64) PRIMARY KEY,  counter_value BIGINT NOT NULL) ENGINE=InnoDB;INSERT INTO servicehub_galera_lab.counters(counter_name,counter_value)VALUES ('dispatch',0)ON DUPLICATE KEY UPDATE counter_value=VALUES(counter_value);INSERT INTO servicehub_galera_lab.tickets(customer_id,state,priority) VALUES (701,'open',1);

A conservative generic routing gate for normal transactional traffic is: component is Primary, node is connected, node is ready, and local state is Synced. Your production gate may additionally exclude desynced nodes, donors, maintenance nodes, or nodes whose local capacity/latency is unhealthy.

SELECT  @@hostname AS node,  MAX(CASE WHEN Variable_name='wsrep_cluster_status' THEN Value END) AS cluster_status,  MAX(CASE WHEN Variable_name='wsrep_connected' THEN Value END) AS connected,  MAX(CASE WHEN Variable_name='wsrep_ready' THEN Value END) AS ready,  MAX(CASE WHEN Variable_name='wsrep_local_state_comment' THEN Value END) AS local_stateFROM information_schema.GLOBAL_STATUSWHERE Variable_name IN ('wsrep_cluster_status','wsrep_connected','wsrep_ready','wsrep_local_state_comment');

2. Why TCP-only load balancing is unsafe

A naive HAProxy/NGINX/ELB check might declare a backend healthy whenever TCP 3306 opens. During state transfer or a membership transition, mariadbd can be running while wsrep is not ready. A proxy that ignores database state can oscillate clients onto nodes that reject writes or have not finished synchronization.

Free/open options such as HAProxy or ProxySQL can be used with appropriate health logic. MariaDB MaxScale/Enterprise tooling can provide integrated capabilities depending on product/version, but they are external components and are not required by this lesson.

3. Partition drill: isolate one node

Disconnect n3 from the custom Docker network. The remaining n1+n2 majority should retain a Primary Component. The isolated n3 should lose normal cluster readiness. This is a disposable local failure injection; do not reproduce it on production interfaces.

docker network disconnect $(docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}}{{end}}' mdb15-n3) mdb15-n3 || true

If your Compose network name is not easily derived, run docker network ls and disconnect n3 from the project network explicitly. Then inspect n1 and n2.

SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_cluster_status','wsrep_cluster_size','wsrep_ready','wsrep_local_state_comment');

n1/n2 should converge on a two-member Primary Component. n3 may become unreachable from the host depending on Docker networking; if you can still reach its published port, its wsrep state—not TCP—is the acceptance criterion. Never override a minority partition to Primary merely to keep it writable.

4. Quorum prevents divergent writes only if operators respect it

Galera's safety behavior can be defeated operationally. If an isolated minority is manually bootstrapped with --wsrep-new-cluster or pc.bootstrap=YES while the majority still accepts writes, you have created two authoritative components. When connectivity returns, there is no magical row-by-row conflict merger that restores a single correct history.

5. Fencing: prevent the wrong node from being a writer

Fencing is any mechanism that reliably prevents a failed/old authority from accepting authoritative writes. In Galera, loss of Primary Component is an important database-level fence, but production designs reinforce it with routing removal, network/security controls, orchestration, and credential ownership.

• Remove the node from the load-balancer pool before planned maintenance.
• Do not use a wildcard DNS record that keeps resolving to a known-bad node.
• Use health checks that fail closed on uncertain wsrep state.
• For failover automation, ensure there cannot be two independent control planes both declaring different nodes/clusters healthy.
• Record exactly which component is authoritative during an incident.

6. Reconnect the node and require convergence before routing

Reconnect n3 to the project network and let it rejoin. It may use IST or SST depending on how long it was isolated and what GCache remains.

docker network connect NETWORK mdb15-n3docker logs --since 2m mdb15-n3 2>&1 | grep -Ei "IST|SST|Synced|ready" || true

SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_cluster_status','wsrep_cluster_size','wsrep_connected','wsrep_ready','wsrep_local_state_comment');

Only after the node is Primary/Synced/ready and passes application checks should routing be restored.

7. Application failover means reconnect plus transaction semantics

When a database node disappears, existing TCP sessions fail. A load balancer cannot teleport an in-flight transaction to another session. The application needs bounded reconnect/retry logic and must understand whether the last transaction committed before the connection broke.

8. Read/write routing choices

Galera allows reads and writes on any Synced Primary node, but “possible” is not always “best.” A preferred-writer policy can reduce cross-node conflicts and make connection pools simpler. Distributing read traffic can improve capacity, but long/heavy reads on one member can indirectly hurt the whole cluster if apply falls behind and triggers flow control.

Critical read-after-write paths may use Galera causal-read mechanisms such as wsrep_sync_wait on the session, but that adds wait latency. Define consistency requirements explicitly instead of routing every read randomly and assuming “synchronous means no timing difference anywhere.”

9. Production judgment

A production Galera HA design needs at least: quorum-aware node health, routing ownership, fencing, connection retry semantics, capacity to operate after a node loss, and a tested return-to-service gate. Put those rules in automation/runbooks and test partitions deliberately.

Next, harden the cluster transport, rotate certificates/secrets, remove nodes safely for maintenance, and practice the most dangerous Galera procedure: recovering after a full cluster outage without bootstrapping stale state.

docker compose down -v --remove-orphans