Chapter 15 · Group Replication, InnoDB Cluster, Router, and High Availability
Consensus-Oriented Group Replication Concepts, Membership, Quorum, and Failure Modes
Build the high-availability mental model before touching failover commands: membership views, majority quorum, Group Replication states, certification, and why a minority partition must stop rather than invent a second writable truth.
Learning outcomes
Chapter 14 showed how one source can asynchronously feed a replica. That design helps with read scaling and recovery, but it leaves a hard question: who is allowed to become the writer when a server or network segment fails? Group Replication addresses that problem with a membership service, distributed transaction certification, and majority-based decisions. InnoDB Cluster then gives you a supported administrative layer around Group Replication, while MySQL Router gives applications a stable endpoint.
Explain Group Replication membership, member states, certification, majority quorum, and why a minority partition must stop making progress.
Build a disposable three-member InnoDB Cluster with free MySQL Community Server and MySQL Shell tooling.
Use Performance Schema and AdminAPI status as evidence instead of inferring cluster health from a single reachable TCP port.
Distinguish one-member failure from quorum loss and describe the safety reason behind each response.
Write the decision gates required before any force-quorum recovery so a partition cannot become split brain.
High availability is a correctness problem first
A database is not highly available merely because another process is listening on another port. During a failure, two requirements compete: keep serving traffic, and never create two independent writable histories that both claim to be authoritative. The second failure mode is split brain. Once two partitions accept conflicting writes independently, reconnecting them is not ordinary replication catch-up; you have created a data-reconciliation incident.
Group Replication therefore chooses safety when it cannot prove a majority. Members exchange group communication messages and maintain a consistent membership view. Transactions that need to be accepted by the group go through a certification process. When a majority cannot be reached, the surviving minority cannot safely decide whether the missing servers are dead or merely isolated on the other side of a partition. The safe behavior is to stop progress that requires consensus.
| Term | Mental model | Evidence |
|---|---|---|
| member | one MySQL Server participating in the group | replication_group_members |
| view | the membership generation agreed by the group | member-state and view-change information |
| quorum | a majority able to make safe distributed decisions | reachable ONLINE majority, not a percentage guessed by the operator |
| certification | conflict check applied to transactions before group-wide commit acceptance | transaction outcomes plus group member statistics |
| ONLINE | member fully synchronized and participating | MEMBER_STATE = ONLINE |
| split brain | two isolated sides both behave as writable authority | prevented by quorum/fencing; not repaired by simply reconnecting |
Why three members are the smallest useful HA teaching topology
With three members, a majority is two. If one server fails, two remain and the group can still form a majority. If two fail, the one remaining member does not have quorum and should not simply promote itself. This makes the safety rule observable with a small local lab. A two-member group cannot lose either member and still have a majority, which is why “two copies” is not the same as a failure-tolerant voting topology.
A quorum-based cluster can protect write ownership during many failures, but it can replicate logical mistakes, dropped rows, or bad application writes to every member. Chapter 13 backup/PITR remains necessary.
Mandatory labs target MySQL Community Server 8.4.10 LTS, MySQL Shell 8.4.10 LTS, and—where routing is required—MySQL Router 8.4.10 LTS. The examples use three disposable members because a three-member group can retain majority after one member fails. All data and credentials are lab-only.
| Component | Lab endpoint | Role at start | Why it exists |
|---|---|---|---|
| db1 | 127.0.0.1:33151 | PRIMARY candidate | first cluster seed and normal starting primary |
| db2 | 127.0.0.1:33152 | SECONDARY | quorum member and failover candidate |
| db3 | 127.0.0.1:33153 | SECONDARY | third vote so one member can fail without losing majority |
| MySQL Shell | local client | AdminAPI control plane | creates/changes metadata-managed topology |
| MySQL Router | local middleware | application endpoint | routes clients using cluster metadata rather than hard-coded server addresses |
Build the disposable cluster
Use three local native instances, VMs, or the following container server layer. The containers advertise host-reachable SQL endpoints. On Linux engines that do not automatically define host.docker.internal, the Compose host-gateway mapping provides it inside the containers; if your host itself cannot resolve that name, add a local hosts-file mapping to 127.0.0.1 or use equivalent native instances.
services: db1: image: mysql:8.4.10 container_name: servicehub-ha-db1 environment: MYSQL_ROOT_PASSWORD: LabRootOnly_2026! MYSQL_ROOT_HOST: "%" ports: ["33151:3306"] extra_hosts: ["host.docker.internal:host-gateway"] command: - --server-id=151 - --report-host=host.docker.internal - --report-port=33151 - --gtid-mode=ON - --enforce-gtid-consistency=ON - --binlog-format=ROW - --log-replica-updates=ON volumes: ["ha1:/var/lib/mysql"] db2: image: mysql:8.4.10 container_name: servicehub-ha-db2 environment: MYSQL_ROOT_PASSWORD: LabRootOnly_2026! MYSQL_ROOT_HOST: "%" ports: ["33152:3306"] extra_hosts: ["host.docker.internal:host-gateway"] command: - --server-id=152 - --report-host=host.docker.internal - --report-port=33152 - --gtid-mode=ON - --enforce-gtid-consistency=ON - --binlog-format=ROW - --log-replica-updates=ON volumes: ["ha2:/var/lib/mysql"] db3: image: mysql:8.4.10 container_name: servicehub-ha-db3 environment: MYSQL_ROOT_PASSWORD: LabRootOnly_2026! MYSQL_ROOT_HOST: "%" ports: ["33153:3306"] extra_hosts: ["host.docker.internal:host-gateway"] command: - --server-id=153 - --report-host=host.docker.internal - --report-port=33153 - --gtid-mode=ON - --enforce-gtid-consistency=ON - --binlog-format=ROW - --log-replica-updates=ON volumes: ["ha3:/var/lib/mysql"]volumes: ha1: ha2: ha3:# Bash / macOS / Linux / Git Bash / PowerShell# Save the previous YAML as compose.yaml.docker compose up -ddocker compose ps# Verify the published SQL endpoints from the host.mysql -h 127.0.0.1 -P 33151 -uroot -p -e "SELECT @@version,@@server_id,@@report_host,@@report_port"mysql -h 127.0.0.1 -P 33152 -uroot -p -e "SELECT @@version,@@server_id,@@report_host,@@report_port"mysql -h 127.0.0.1 -P 33153 -uroot -p -e "SELECT @@version,@@server_id,@@report_host,@@report_port"AdminAPI should be allowed to validate requirements instead of copying a memorized Group Replication option file. Requirements include InnoDB for replicated tables, primary-key-equivalent row identity, binary logging, ROW format, GTIDs, and compatible member settings. dba.configureInstance() reports what it must change and whether a restart is required.
// Start: mysqlsh root@127.0.0.1:33151 --js// Repeat configureInstance for all three disposable instances.dba.configureInstance('root@127.0.0.1:33151', { clusterAdmin: 'icadmin', clusterAdminPassword: 'LabIcAdminOnly_2026!'})dba.configureInstance('root@127.0.0.1:33152', { clusterAdmin: 'icadmin', clusterAdminPassword: 'LabIcAdminOnly_2026!'})dba.configureInstance('root@127.0.0.1:33153', { clusterAdmin: 'icadmin', clusterAdminPassword: 'LabIcAdminOnly_2026!'})// Reconnect as the dedicated cluster administrator.\connect icadmin@127.0.0.1:33151var cluster = dba.createCluster('ServiceHubHA')cluster.addInstance('icadmin@127.0.0.1:33152', {recoveryMethod:'clone'})cluster.addInstance('icadmin@127.0.0.1:33153', {recoveryMethod:'clone'})cluster.status({extended:1})The recoveryMethod:"clone" examples intentionally use empty disposable db2/db3 instances. Clone provisioning replaces the target state. Never point this lab step at a server containing data you need.
Observe membership from both control planes
SELECT MEMBER_ID, MEMBER_HOST, MEMBER_PORT, MEMBER_STATE, MEMBER_ROLE, MEMBER_VERSIONFROM performance_schema.replication_group_membersORDER BY MEMBER_ROLE, MEMBER_HOST, MEMBER_PORT;SELECT *FROM performance_schema.replication_group_member_stats\GAdminAPI cluster.status() is the supported cluster-management view. Performance Schema is the server-level evidence. Healthy single-primary output should show three ONLINE members, one PRIMARY, and two SECONDARY roles. The exact primary depends on the cluster creation/election history, so do not hard-code “db1 must always be primary” into an application.
cluster.status({extended:2})cluster.describe()cluster.options({all:true})Failure 1: lose one member, keep quorum
Stop a secondary container, not the primary, and observe the remaining membership. The group still has two of three votes. Availability can continue, but fault tolerance is reduced: another member failure would remove the majority.
docker stop servicehub-ha-db3# In MySQL Shell, still connected to a surviving member:cluster.status({extended:1})# Restore it after observing the degraded state.docker start servicehub-ha-db3cluster.rejoinInstance('icadmin@127.0.0.1:33153')cluster.status({extended:1})Failure 2: lose quorum, do not improvise a new truth
If only one of three members is reachable, that server cannot know whether the other two are truly down or alive together on another network partition. The correct default is therefore a blocked/unavailable group, not automatic unilateral writing. This can feel “less available,” but it is exactly what protects the database from two writable histories.
cluster.forceQuorumUsingPartitionOf(instance) can redefine the group around a surviving partition. It can create split brain if excluded members are still alive and accepting traffic. The mandatory lab stops before executing force quorum: first document fencing, GTID/state checks, and which partition is authoritative.
| Gate before force quorum | Required evidence |
|---|---|
| Fence excluded side | network/load balancer/process controls prove old writers cannot accept application writes |
| Identify authoritative partition | business owner/incident lead chooses the side whose committed history is retained |
| Inspect reachable member state | AdminAPI status, Group Replication state, GTID sets, error logs |
| Record data-loss risk | acknowledge transactions that may exist only on the excluded side |
| Plan rejoin | excluded members will be reprovisioned/rejoined; they do not simply reconnect themselves |
Production judgment
Use Group Replication/InnoDB Cluster when the application requires automated writer election and the team is prepared to operate quorum, member lifecycle, routing, backups, and failure testing together. Do not use it merely because “three servers sounds safer.” Every extra member adds network communication, recovery, upgrade, and capacity considerations. Monitor member state, queue/apply behavior, transaction conflicts, recovery progress, and Router health, but define alerts around service objectives rather than universal numeric thresholds.
Next, we turn from who may write to how many members may write. Single-primary is the default and the simpler operating model; multi-primary introduces distributed write conflicts and stricter workload constraints.
Knowledge check
- Why can a three-member group tolerate one member failure but a two-member group cannot while preserving majority?
- Does MEMBER_STATE=ONLINE on one server prove that the entire cluster has quorum?
- Why does a minority partition block instead of electing itself?
- What must be proven before forcing quorum around a surviving partition?
- Does InnoDB Cluster replace backup/PITR?
Reveal answers
- Three members retain a 2-of-3 majority after one failure; one surviving member of a two-member group is only half, not a majority.
- No. A member state is evidence from a particular membership view; inspect the full AdminAPI/Performance Schema topology and reachability.
- It cannot distinguish “the others are dead” from “I am isolated”; unilateral progress could produce split brain.
- The excluded side is fenced, the authoritative partition is chosen, current transaction/GTID state is recorded, data-loss risk is accepted, and a rejoin/reprovision plan exists.
- No. A cluster can replicate accidental or malicious changes to all members, so independent tested backups remain required.