Failure Injection, Split-Brain Avoidance, Maintenance, Upgrades, and HA Acceptance Tests

Learning outcomes

High availability cannot be accepted from a diagram. The ServiceHub team needs evidence for what happens when one member dies, the primary dies, Router restarts, a network partition removes quorum, or maintenance changes versions/configuration. This lesson turns those scenarios into an acceptance matrix with explicit safety gates and stop conditions.

Define acceptance before injecting failure

A failure test without acceptance criteria becomes theater: the operator kills a process, sees something reconnect, and declares success. Instead define what must remain true. For ServiceHub, the most important invariant is that each committed work-order command has exactly one durable business outcome and there is never more than one writable authority after partition/failover.

Acceptance marker: a business invariant you can verify everywhere

CREATE DATABASE IF NOT EXISTS servicehub_ha_acceptance;CREATE TABLE IF NOT EXISTS servicehub_ha_acceptance.commands (  command_id CHAR(36) PRIMARY KEY,  command_type VARCHAR(40) NOT NULL,  payload_hash CHAR(64) NOT NULL,  created_at TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),  UNIQUE KEY uq_payload(command_type,payload_hash)) ENGINE=InnoDB;INSERT INTO servicehub_ha_acceptance.commands(command_id,command_type,payload_hash)VALUES(UUID(),'HA_BASELINE',SHA2('servicehub-ha-baseline',256));SELECT COUNT(*) AS baseline_rows,       COUNT(DISTINCT command_id) AS distinct_commandsFROM servicehub_ha_acceptance.commands;

The primary key and business uniqueness constraint make duplicate acceptance visible. During failover tests, use a new UUID and deterministic payload hash so a retry can distinguish “already committed” from “never committed.” This is much stronger than checking only whether SELECT 1 succeeds.

Test 1: single secondary loss

docker stop servicehub-ha-db3# MySQL Shell on a surviving member:cluster.status({extended:2})# Verify writes still work through Router RW endpoint, then recover:docker start servicehub-ha-db3cluster.rejoinInstance('icadmin@127.0.0.1:33153')cluster.status({extended:2})

Acceptance is not merely “the INSERT worked.” The cluster must show a two-member majority during the failure, the recovered member must return ONLINE, and business rows/GTID history must converge. Record how long the cluster ran with reduced fault tolerance; alerting should make that degraded window operationally visible.

Test 2: unexpected primary loss

First identify the current primary. Open a Router client, insert a uniquely keyed command, then stop the primary container. Watch Group Replication elect a replacement and Router route a new connection. Do not reuse the broken session as proof of transparent failover.

cluster.status({extended:2})

SET @cmd = UUID();INSERT INTO servicehub_ha_acceptance.commands(command_id,command_type,payload_hash)VALUES (@cmd,'PRIMARY_FAILOVER',SHA2(CONCAT('before-',@cmd),256));SELECT @cmd AS committed_command_id, @@hostname AS backend;

# Example only; choose the container that cluster.status() reports as primary.docker stop servicehub-ha-db1# Then inspect election from a surviving member.cluster.status({extended:2})

Reconnect through Router and query the recorded command. If the write committed before failure, it must exist exactly once. If the connection disappeared during commit in a different test, use the command ID to resolve the ambiguity before retrying.

Test 3: Router loss must not become database loss

If you deploy only one Router, the database can be healthy while the application endpoint is down. That is a middleware single point of failure. Stop the disposable Router process and verify that direct cluster health remains OK. In production, applications normally need multiple Router instances or a service layer that can reach another Router.

# Stop the disposable Router process/service.# Exact command depends on how the bootstrap directory/service was started.# Then verify cluster status independently in MySQL Shell:cluster.status({extended:2})# Restart Router and reconnect through its configured read/write endpoint.

Test 4: network isolation and split-brain avoidance

A partition that leaves one member isolated from the other two is different from a process crash: all machines may still be alive. The majority side can continue; the isolated minority cannot safely act as an independent group. A more severe partition may leave only a minority reachable to the application site, in which case the safe service can become unavailable.

Planned maintenance: preserve quorum and reversibility

Maintenance is the best time to practice disciplined failover because the failure is scheduled. Work one member at a time. Before touching a node, check cluster status, application error budget, transaction queues, backup/recovery readiness, and version compatibility. After each step, wait for the member to return ONLINE and catch up before touching the next member.

cluster.status({extended:3})cluster.options({all:true})// Also record server versions and Group Replication member versions from SQL.

SELECT MEMBER_HOST,MEMBER_PORT,MEMBER_STATE,MEMBER_ROLE,MEMBER_VERSIONFROM performance_schema.replication_group_members;SELECT @@version, @@version_comment, @@global.gtid_executed;

Mixed-version operation during a rolling upgrade is subject to version-compatibility rules. Do not infer that “newer can always join older” or vice versa. Check the current release's Group Replication upgrade documentation immediately before maintenance.

Compatibility/lock-risk failure: detect before changing the cluster

The prompt requires a failure caught during preflight/canary validation. A realistic example is a schema change that needs a long metadata lock while a production transaction is holding the table. Instead of launching the DDL cluster-wide during maintenance, test on a canary workload, inspect lock behavior, and use an expand/contract migration if the blocking window violates the maintenance objective.

START TRANSACTION;SELECT * FROM servicehub_ha_acceptance.commandsWHERE command_type='HA_BASELINE'FOR UPDATE;-- Keep open briefly for the canary test.

-- In the disposable lab only:ALTER TABLE servicehub_ha_acceptance.commands  ADD COLUMN operator_note VARCHAR(100) NULL;-- If this waits beyond the canary budget, cancel/rollback the test rather than-- forcing production through the same blocking window.

A reversible alternative is expand/contract: add a nullable/backward-compatible structure during one deployment, update application writers/readers gradually, backfill in bounded work, verify, then remove the old contract later. The exact method depends on the schema change; the principle is to avoid coupling “upgrade” to one irreversible blocking event.

Complete outage is not quorum loss

If Group Replication has stopped on all members after a major outage, the recovery problem is different from one surviving minority partition. MySQL Shell provides dba.rebootClusterFromCompleteOutage(). The operator should connect to the most appropriate up-to-date member, compare GTID/state where possible, and avoid the force option unless the documented risk is understood. An outage runbook should never treat “pick any node and bootstrap it” as acceptable.

// Only after a real complete outage and state review:var cluster = dba.rebootClusterFromCompleteOutage('ServiceHubHA')cluster.status({extended:2})

Final HA acceptance matrix

Production judgment and bridge to Chapter 16

High availability is a system of mechanisms and operating decisions: Group Replication protects distributed write ownership with quorum/certification; InnoDB Cluster provides a supported lifecycle control plane; Router provides metadata-aware client routing; the application owns reconnect, retry, idempotency, and user-facing behavior; backup/PITR protects against failures that replication faithfully copies.

The next chapter makes these mechanisms observable in normal operation. Chapter 16 moves into Performance Schema, the sys schema, logs, metrics, and dashboards so operators can detect the precursor signals—queue growth, connection pressure, waits, I/O, memory, and errors—before every problem becomes a failover drill.

Authoritative references

• Group Replication network partition and quorum handling
• MySQL Shell — Restoring and rebooting InnoDB Cluster
• Rejoining an InnoDB Cluster instance
• MySQL Router bootstrapping
• Group Replication upgrades