MySQL Shell AdminAPI, InnoDB Cluster Lifecycle, Recovery, and Rejoin Operations

Learning outcomes

Raw Group Replication commands are useful for understanding the mechanism, but once a topology is an InnoDB Cluster, MySQL Shell AdminAPI is the supported administrative boundary. AdminAPI keeps cluster metadata, recovery accounts, member configuration, and topology operations coherent. This lesson treats that metadata as part of the system—not a decorative wrapper you can bypass whenever a raw server command seems faster.

AdminAPI is the topology control plane

MySQL Shell is an administration client. AdminAPI is the API inside Shell that manages InnoDB Cluster, InnoDB ClusterSet, and InnoDB ReplicaSet. An InnoDB Cluster uses Group Replication underneath, but AdminAPI adds metadata and operations that understand the topology as a whole.

Preflight every candidate member

dba.configureInstance() validates whether an instance meets InnoDB Cluster requirements and can persist required changes where supported. It reports incompatible variables instead of asking you to memorize a version-specific checklist. A restart may still be required for particular settings, and the generated report tells you that.

dba.configureInstance('root@127.0.0.1:33151', {  clusterAdmin:'icadmin',  clusterAdminPassword:'LabIcAdminOnly_2026!'})dba.configureInstance('root@127.0.0.1:33152', {  clusterAdmin:'icadmin',  clusterAdminPassword:'LabIcAdminOnly_2026!'})dba.configureInstance('root@127.0.0.1:33153', {  clusterAdmin:'icadmin',  clusterAdminPassword:'LabIcAdminOnly_2026!'})

Use a dedicated administration identity rather than operating the cluster permanently as root. Chapter 11's separation-of-duties principles still apply to HA tooling.

Create or retrieve the cluster safely

\connect icadmin@127.0.0.1:33151var cluster = dba.getCluster('ServiceHubHA')// If this is a brand-new reset lab instead:// var cluster = dba.createCluster('ServiceHubHA')cluster.status({extended:2})cluster.describe()cluster.options({all:true})

status() tells you current runtime health; describe() describes topology metadata; options() exposes configured policy. These answer different questions. A server can still be reachable by TCP while the cluster reports a degraded or unreachable member.

AddInstance: recovery method is a data decision

A joining member must obtain the cluster's state. Clone takes a physical snapshot from a donor and replaces the target instance's state. Incremental recovery transfers missing transactions when the target's GTID set and retained history make that possible. The safe choice depends on whether the target contains data to preserve, whether required binary-log history still exists, and whether GTID ancestry is compatible.

cluster.addInstance('icadmin@127.0.0.1:33152', {recoveryMethod:'clone'})cluster.addInstance('icadmin@127.0.0.1:33153', {recoveryMethod:'clone'})cluster.status({extended:2})

Failure lab: a member leaves and cannot automatically rejoin

Stop Group Replication on one disposable member to simulate a member that has left the cluster. This is failure injection, not a recommended normal administration path. Then return to AdminAPI for recovery.

STOP GROUP_REPLICATION;SELECT MEMBER_HOST, MEMBER_PORT, MEMBER_STATE, MEMBER_ROLEFROM performance_schema.replication_group_members;

cluster.status({extended:2})cluster.rejoinInstance('icadmin@127.0.0.1:33153')cluster.status({extended:2})

rejoinInstance() validates the member and attempts to return it to the cluster. If its transaction history or identity has changed incompatibly, rejoin may not be safe; removal and reprovisioning can be the better path.

Remove versus rejoin versus reprovision

Planned primary change is not the same as failure recovery

In single-primary mode, AdminAPI can elect a specific healthy member with setPrimaryInstance(). Use this for controlled maintenance or topology tests. It is not the same as forcing a partition after quorum loss.

cluster.status({extended:1})cluster.setPrimaryInstance('icadmin@127.0.0.1:33152', {  runningTransactionsTimeout: 30})cluster.status({extended:1})

The timeout controls how long the operation waits for in-flight transactions before allowing the topology change to complete. Do not copy a universal timeout into production; choose it from transaction behavior and maintenance objectives.

Tempting but unsafe: fix AdminAPI errors by hand-editing Group Replication

After an instance belongs to InnoDB Cluster, Oracle explicitly recommends administering it with Shell/AdminAPI. Manually changing critical Group Replication variables, membership, UUID-related state, or recovery channels can make cluster metadata disagree with server reality. A raw command may make one symptom disappear while creating a harder topology-recovery problem.

Cluster metadata is operational state, not just documentation

AdminAPI stores metadata describing the cluster, instances, Router relationships, and supported topology. That metadata lets Shell reason about an operation before changing multiple servers. This is why an instance can be reachable but still be an invalid cluster member: its local MySQL process may be healthy while its server identity, GTID history, recovery state, or metadata relationship no longer matches the managed topology.

When troubleshooting, preserve both views. First use cluster.status({extended:2}) or extended:3 to see what AdminAPI believes. Then inspect server-level evidence such as replication_group_members, error logs, GTID state, and relevant replication channels. Do not “repair” the discrepancy before recording it; the disagreement itself may explain the incident.

Recovery method selection: ask three questions

Before adding or rejoining an instance, answer three questions in order. First, may the target's current data be destroyed? If not, clone is not an acceptable shortcut. Second, is the target's GTID history compatible with the cluster? An errant transaction or divergent history can make incremental recovery unsafe. Third, does an ONLINE donor still retain every transaction needed for incremental recovery? Purged binary logs can remove the incremental path even when histories otherwise align.

These questions convert “Clone or Incremental?” from a prompt-selection habit into a recovery decision. In production, record the answer and evidence in the change ticket/runbook. A destructive recovery method should never be chosen because it is the default highlighted option in an interactive prompt.

Complete outage and quorum loss are different incidents

A complete outage means Group Replication has stopped across the cluster. Quorum loss means a group or partition remains but cannot form a majority. The recovery commands are therefore different. dba.rebootClusterFromCompleteOutage() reconstructs the cluster after all members are out of the group. forceQuorumUsingPartitionOf() redefines membership around a surviving partition and carries explicit split-brain risk if excluded members are not fenced.

The similarity—both sound like “make the cluster work again”—is operationally dangerous. A runbook should start by classifying the incident, not by choosing the command the operator remembers. For either case, capture GTID sets, reachable members, member identities, and application fencing state before issuing a destructive or forceful recovery operation.

AdminAPI automation needs stop conditions

AdminAPI is scriptable, but automation should become more conservative around ambiguous state. A good automation can run status checks, verify the expected topology, confirm that a target is empty before clone, and execute a known rejoin. It should stop and escalate if GTID sets diverge unexpectedly, if a member identity changes, if multiple potential authoritative partitions exist, or if the operation would require force. Automation that retries increasingly forceful operations is not resilience; it is a mechanism for destroying evidence quickly.

Production judgment

AdminAPI reduces the number of inconsistent ways operators can change a topology, but it does not eliminate judgment. Before clone, remove, force quorum, or reboot-after-outage operations, record target identity, GTID/state evidence, backup/recovery position, and the expected postcondition. Automation should stop when the state does not match the runbook rather than adding force:true until a command succeeds.

Next, we connect an application to this metadata-managed topology. MySQL Router can follow primary changes and expose stable ports, but it cannot replay a failed transaction or preserve every session assumption across a broken backend connection.

Authoritative references

• Configuring production instances for InnoDB Cluster
• Creating an InnoDB Cluster
• Setting up InnoDB Cluster and MySQL Router
• Rejoining an instance
• Restoring and rebooting an InnoDB Cluster