Rolling Cluster/Replication Upgrades, Rollback Plans, Canaries, and Performance Comparison

Learning outcomes

A low-downtime database upgrade is not “upgrade each node one at a time.” Mixed-version replication or Galera operation is safe only where the exact path is documented, and rollback becomes harder after promotion or new-version writes. A canary is useful only if it receives representative work and has explicit stop criteria.

1. Async replication: canary the replica first

MariaDB’s general major-upgrade guidance recommends upgrading a replica first, validating it, then the remaining replicas, and only later switching/promoting as part of the primary upgrade. That pattern reduces risk because the old primary remains authoritative while the new binary consumes real replication traffic. It is not permission to use any arbitrary old/new pair; verify the exact source→target path and binlog/GTID compatibility.

2. Promotion must include fencing and data-loss checks

SHOW MASTER STATUS;SHOW REPLICA STATUS\GSELECT @@server_id, @@read_only, @@gtid_current_pos, @@gtid_slave_pos;-- Use exact MariaDB GTID/domain semantics from Chapter 14.-- Do not promote until expected transactions are present and the old writer is fenced.

Application routing follows database safety; it does not create it. Disable/fence the old writer at the infrastructure/database layer appropriate to your design, then repoint clients only after the new target passes role/data checks.

3. Galera rolling upgrades are path-specific

Galera adds provider/application protocol compatibility, quorum, SST/IST behavior and cluster membership to the version problem. Follow the exact MariaDB Galera upgrade page for the source/target versions. Never infer “one node at a time” from a generic cluster article when the documented path requires a different order, configuration, full restart or SST.

SHOW STATUS WHERE Variable_name IN('wsrep_cluster_status','wsrep_cluster_size','wsrep_connected','wsrep_ready', 'wsrep_local_state_comment','wsrep_local_recv_queue_avg','wsrep_flow_control_paused');SHOW VARIABLES LIKE 'wsrep_provider_version';

4. Canary criteria must include correctness and performance

Do not invent percentages universally. The threshold belongs to the application SLO and the baseline variability measured in Chapter 18.

5. Wrong approach: “downgrade the package if anything fails”

After a new-version server has written system tables or application data, downgrade may be unsupported. The safer rollback is often failback to an untouched old-version node before divergent writes, or restore/rebuild from a compatible backup. The runbook must state which rollback remains valid at each stage.

6. Free local canary mechanics lab

1. Create disposable source primary + replica using the exact source version you need to test.2. Seed ServiceHub data and capture GTID/file-position, counts, workload baseline and backup.3. Upgrade ONLY the replica using the official source→target guide.4. Run mariadb-upgrade if required; verify replication I/O/apply and data invariants.5. Run the same query/benchmark suite against primary and canary replica under comparable conditions.6. Inject one safe failure: stop canary, restart, verify catch-up; or reject the target if an incompatibility appears.7. Do NOT promote unless your lab goal includes a full promotion/fencing drill from Chapter 14.8. Destroy the topology after recording evidence.

7. Chapter runbook: go/no-go gates

Before production, require signed evidence for: target release/support status, backups/restores, config diff, upgrade-tool result, data invariants, app suite, replica/Galera health, canary latency/error comparison, promotion/fencing procedure, rollback boundary, and post-change observation window. A no-go is a successful outcome of testing—it prevents an uncontrolled production experiment.

Production judgment and bridge to Chapter 21

The safest upgrade strategy minimizes simultaneous unknowns: change one layer at a time, preserve an authoritative old path until evidence closes the gate, and compare behavior under reproducible load. With upgrade/change discipline established, Chapter 21 turns to advanced MariaDB-native features—system-versioned tables, temporal modeling, Oracle compatibility modes and federation—where feature adoption requires the same version/plugin/semantics discipline.

Rolling topology upgrades: prove coexistence, canary behavior, and rollback separately

A rolling upgrade is not “restart nodes one by one.” It depends on whether the exact old/new server, replication protocol, binlog format, GTID behavior, storage format, and Galera provider combination is supported during the coexistence window. Async replication and Galera have different safety models, so use the procedure for the actual topology and target versions rather than a generic cluster checklist.

For asynchronous replication, a common canary pattern is to upgrade a replica first, let it process production-like replicated traffic, and compare correctness/performance while the write primary remains unchanged. Observe receiver/apply health, GTID positions, lag, error logs, query plans, read-serving behavior if used, and backup tooling. Promotion is a separate operation: fence the old writer, prove the candidate is caught up to the declared recovery point, change client routing, and verify writes before considering the old primary a rollback target.

A Galera rolling upgrade requires version-specific provider/server guidance because nodes exchange write sets and state-transfer data while different versions coexist. Check whether rolling upgrade is supported for that exact transition, whether intermediate series are required, and whether IST/SST behavior constrains the procedure. Quorum, primary-component status, donor/joiner state, certification conflicts, and flow control remain application-visible risks even when the cluster stays “up.”

Canary acceptance must include identical workload evidence. Compare error rate, p50/p95/p99 latency, throughput, CPU/I/O, memory, top statement digests/plans, replication/wsrep indicators, and backup/restore compatibility. One node surviving a restart is not a performance canary. Likewise, “no SQL errors” does not prove sorted/collated results, optimizer choices, or application semantics remained acceptable.

Rollback has levels: routing rollback, role/promotion rollback, binary/package rollback, and restore-based rollback. After an upgraded node has accepted new writes or system metadata has changed, some earlier rollback options may no longer be valid. State the boundary for every step and keep an untouched, recoverable copy of the pre-change state until the post-upgrade validation window closes.

Performance regression triage after a rolling upgrade

When a canary is slower, resist changing several variables at once. First classify the regression: plan/cardinality change, cache/warmup effect, CPU instruction/runtime change, storage latency, connection/thread scheduling, redo/flushing behavior, replication/wsrep pressure, connector behavior, or workload mismatch. Compare the same statement digest and parameter class on old and new nodes, then inspect plan and observed counters for that statement.

Some differences are expected during warmup. A newly restarted node has a colder buffer pool and filesystem cache, different prepared-statement/session history, and possibly different statistics state. Define a warmup protocol before comparing. Conversely, do not dismiss persistent p95/p99 regressions as “cache” when the new node has already processed an equivalent workload window.

Keep the upgraded canary reversible until the comparison is complete. Avoid introducing target-only schema or application behavior while it is still serving as the rollback experiment. If it fails acceptance, remove traffic, preserve diagnostic evidence, and either correct/retest or return to the supported prior state. The objective of a canary is to make a bad target cheap to reject.

Closure criteria after the last node is upgraded

Do not close the change when the final package installs. Require a stable observation window with topology health, lag/wsrep state, backups, error logs, application SLOs, query-regression checks, and capacity signals all inside their acceptance bands. Remove temporary bypasses or compatibility settings, update the configuration/version inventory, and schedule cleanup of rollback artifacts only after the restore and failback assumptions have been revalidated.

Authoritative references

• MariaDB Documentation — Upgrading Between Major MariaDB Versions and exact source→target guides.
• MariaDB Documentation — standard replication upgrade/failover semantics and GTIDs.
• MariaDB Galera Cluster documentation — version-specific rolling/full upgrade procedures.
• MariaDB Release Notes and mariadb-upgrade documentation.