Chapter 22 · Production Capstone: Design, Cluster, Secure, Tune, and Recover MariaDB

Define Workload, SLOs, Data Model, Engine Choices, Capacity, and Architecture Record

Turn a concrete ServiceHub workload into measurable SLOs, data-model and engine choices, HA/recovery boundaries, capacity headroom, rejected alternatives, and a defendable MariaDB architecture decision record.

Advanced capstone210–260 minutesarchitecture record and capacity labMariaDB Community 12.3.2 current GA referenceCurriculum anchor: MariaDB 11.8 LTS · verify source/target/tool/topology versionsFree local tooling · Last reviewed: August 2026

Learning outcomes

ServiceHub is moving from “one MariaDB server that seems fast enough” to a database service that must survive growth, operator error and host failure. The first capstone mistake would be to start with a technology answer—“three Galera nodes,” “a 64 GB buffer pool,” or “daily backups”—before defining what the workload and business actually require. This lesson starts with contracts that can be measured.

01

Translate availability, latency, durability, RPO/RTO, concurrency, growth and retention assumptions into database acceptance criteria.

02

Derive a minimal relational model and choose InnoDB or another engine only from correctness and workload needs.

03

Choose a single write-authority + asynchronous GTID replica as the capstone baseline and explain when Galera would be a justified alternative.

04

Build a capacity worksheet that includes data, indexes, redo/binlogs, temporary space, backups and explicit headroom.

05

Write an Architecture Decision Record (ADR) with rejected alternatives, evidence gaps, owners and measurable exit criteria.

Scenario numbers are inputs, not performance promises.

The numeric SLOs and growth rates below are invented requirements for this learning system. They do not claim that any MariaDB version, topology or hardware will meet them. Your lab must replace assumptions with measurements before accepting the design.

1. Freeze the workload contract before choosing topology

Dimension Capstone requirement What must later prove it
Availability 99.95% monthly service objective for database-dependent requests incident history + failover drill; not topology marketing
Latency application DB time p95 ≤ 120 ms; p99 ≤ 250 ms for named critical transactions connector timing + server plan/wait evidence under representative load
Durability committed ticket mutations must survive ordinary process restart; data-loss risk must be explicit for host/site loss InnoDB durability settings + crash/failover drills
RPO ≤ 5 minutes for destructive primary loss in this exercise backup/binlog coverage + replica position; RPO = Recovery Point Objective
RTO ≤ 30 minutes for restore/failover exercise timed runbook; RTO = Recovery Time Objective
Load 200 write transactions/s average, 800/s burst; about 4 reads per write representative benchmark; these are demand assumptions
Growth 60 GB/month net database growth, 24-month online retention measured bytes/row/index + monthly trend
Compliance tenant isolation, least privilege, encrypted network traffic in production, tested retention grants, connector/TLS evidence, retention jobs and audit evidence

A Service-Level Objective (SLO) is an internal target you design and operate toward. It becomes useful only when paired with a measurement definition: which requests count, from where latency is measured, what maintenance is excluded (if anything), and which failure classes are included.

2. Turn user journeys into data and transaction boundaries

The capstone focuses on five user journeys: create a ticket, append a comment, move a ticket through a queue, load one customer’s recent tickets, and publish an integration event. These journeys suggest entities and invariants before they suggest indexes.

Journey Transaction boundary Integrity requirement Likely access path
Create ticket ticket + initial event + outbox entry tenant/customer exist; idempotency key unique tenant + created_at / ticket id
Append comment comment + ticket updated timestamp comment belongs to same tenant/ticket tenant + ticket + created_at
Queue view read open tickets stable order, no cross-tenant rows tenant + status + updated_at + id
Customer history read tickets by customer tenant/customer scope tenant + customer + created_at
Integration publish claim unsent outbox rows retryable ownership; no duplicate business event published_at + id

InnoDB is the default because the capstone requires transactions, foreign keys, crash recovery and row-level concurrency. An alternative engine must earn its place with an explicit invariant it can satisfy and a backup/replication story; “it benchmarked faster once” is not enough.

3. Choose one HA model and name what it does not solve

The capstone baseline is one writable primary plus one asynchronous GTID replica. That is intentionally conservative: the application has one write authority, the replica provides a second copy and read/offload option, and promotion remains an explicit operational action. It does not make commits synchronous across hosts, does not automatically fence the old primary, and does not replace backups.

Option Why it could fit Why it is not the baseline here
Single server only lowest operational complexity host loss has no hot copy; harder RTO
Async GTID replica simple write authority; observable lag; free Community feature non-zero loss window possible; promotion/fencing are external operations
Galera Cluster quorum and multi-primary capabilities; near-synchronous write-set replication more topology/latency/conflict/flow-control constraints than this workload requires
External proxy/orchestrator routing and automation can reduce application burden extra component and operational contract; not required for free mandatory lab
Do not combine guarantees by vocabulary.

“Replica,” “cluster,” “backup,” and “PITR” solve different failure classes. A replica can copy a bad DELETE. A backup can be too old for the RPO. Galera quorum can protect against split-brain but not against a logically valid destructive transaction replicated to every node.

4. Make the architecture observable before building it

sql · inventory the server you actually have
SELECT VERSION() AS server_version, @@version_comment AS build_comment;SHOW ENGINES;SHOW VARIABLES WHERE Variable_name IN ( 'datadir','innodb_buffer_pool_size','innodb_log_file_size','log_bin', 'binlog_format','gtid_strict_mode','read_only','require_secure_transport', 'max_connections','thread_handling');SHOW GLOBAL STATUS WHERE Variable_name IN ( 'Uptime','Threads_connected','Threads_running','Questions','Bytes_received','Bytes_sent');

This is not a sizing exercise yet. It is an effective-state inventory. Configuration files, container environment variables and documentation can all disagree with the running server; production decisions use effective values plus their persistence source.

5. Capacity planning: budget the whole recovery envelope

Raw table bytes are only one consumer. Large operations need temporary free space; rebuilds may need a second copy; physical backups need their own capacity; binlog retention must cover recovery/replica outages; and growth uncertainty needs headroom.

sql · capture current logical footprint
SELECT table_schema,       ROUND(SUM(data_length)/1024/1024,1) AS data_mb,       ROUND(SUM(index_length)/1024/1024,1) AS index_mb,       ROUND(SUM(data_free)/1024/1024,1) AS data_free_mbFROM information_schema.TABLESWHERE table_schema NOT IN ('mysql','information_schema','performance_schema','sys')GROUP BY table_schemaORDER BY data_mb + index_mb DESC;SHOW BINARY LOGS;SHOW GLOBAL STATUS LIKE 'Innodb_os_log_written';
text · capacity worksheet; replace assumptions with measurements
Demand inputs:  net_db_growth_per_month = 60 GiB   # scenario assumption  online_retention_months = 24  uncertainty_headroom = 0.40Budget categories:  current data + indexes  24-month growth  rebuild/online-DDL workspace  InnoDB redo + temporary files  binlog retention for PITR + replica outage  one local restore workspace  backup repository + retention generationsRule:  Do not accept the host if any credible rebuild/restore path consumes the safety margin.  Recalculate from observed monthly growth and measured operation sizes.

Headroom is a risk policy, not a magic percentage. The 40% worksheet value is a capstone assumption to be challenged. A database with unpredictable temp spills, large online DDL and long backup retention may need more; a carefully bounded appliance may need less.

6. Wrong approach: architecture by defaults and product names

Suppose the team immediately creates a three-node Galera cluster, assigns huge global buffers, enables every log and advertises “zero data loss.” The design has not stated write-conflict rate, network latency, quorum placement, log retention, restore time, or memory concurrency. It can fail every business objective while looking sophisticated.

The repair is to make each technology traceable to one requirement and one test. If no test exists, record the decision as provisional rather than “production ready.”

7. Reproducible architecture-record lab

sql · store the capstone contract as data
DROP DATABASE IF EXISTS capstone22_meta;CREATE DATABASE capstone22_meta CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;USE capstone22_meta;CREATE TABLE architecture_requirement (  requirement_key VARCHAR(64) PRIMARY KEY,  requirement_value VARCHAR(255) NOT NULL,  evidence_method VARCHAR(500) NOT NULL,  status ENUM('ASSUMPTION','MEASURED','PASSED','FAILED') NOT NULL DEFAULT 'ASSUMPTION') ENGINE=InnoDB;INSERT INTO architecture_requirement VALUES('latency_p95','<=120ms DB time for critical transactions','connector percentile under representative load','ASSUMPTION'),('latency_p99','<=250ms DB time for critical transactions','connector percentile under representative load','ASSUMPTION'),('rpo','<=5m for destructive primary loss exercise','replica/binlog/restore drill','ASSUMPTION'),('rto','<=30m for restore/failover exercise','timed game-day','ASSUMPTION'),('retention','24 months online','partition/lifecycle + capacity evidence','ASSUMPTION');SELECT * FROM architecture_requirement ORDER BY requirement_key;
text · ADR-022-001
Title: ServiceHub MariaDB production baselineStatus: ProposedContext: workload/SLO/growth/compliance assumptions from capstoneDecision:  InnoDB transactional schema  one writable primary + one asynchronous GTID replica  physical backup + retained binary logs for PITR  application pools connect to one write endpoint; failover requires pool refresh  production network connections require verified TLSRejected for baseline:  single server only — RTO risk  multi-primary Galera — additional conflict/quorum/flow-control complexity not yet justifiedAcceptance criteria:  named p95/p99 transactions pass representative benchmark  restore and failover drills satisfy measured RPO/RTO  grant/TLS/config/index checks pass  capacity forecast retains approved headroom during rebuild/restoreExit criteria / revisit triggers:  write availability or geography requires quorum-based cluster  replica loss window exceeds business RPO  measured growth or restore time exceeds current capacity modelOwner / review cadence: database-platform team / quarterly and after material workload or version change

Check your reasoning

  1. Why is an SLO not a MariaDB tuning variable?
  2. Why choose one write authority for this baseline?
  3. Does a replica satisfy backup requirements?
  4. Why budget rebuild and restore workspace separately from table size?
  5. What makes an ADR falsifiable?
Review the answers
  1. It is a service requirement and measurement contract. MariaDB settings are possible implementation choices that must be tested against it.

  2. It simplifies correctness, retry and fencing semantics while still allowing an observable async replica; it matches the stated workload rather than maximizing topology features.

  3. No. It usually reproduces logical mistakes and may share failure domains. Backups and PITR are independent recovery controls.

  4. Operational procedures can temporarily require another copy, temp files, logs and staging space; a host can run out of disk even when live data fits.

  5. Explicit assumptions, evidence methods, acceptance criteria and exit/revisit triggers rather than a technology preference.

Production judgment and bridge to Lesson 2

A production architecture is a set of measurable contracts, not a diagram. The topology is acceptable only while the workload, recovery tests and operating team can defend it. Lesson 2 converts this proposed architecture into schema, privileges, connector contracts and migrations that can be checked automatically.

Authoritative references

Keep knowledge open

Help the academy stay free and grow.

If these tutorials save you time, a small donation supports new lessons, technical review, diagrams, examples, and long-term maintenance.

ETHEthereum / ERC-20 only
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0

Send only assets compatible with the Ethereum/ERC-20 network. Do not send TRC-20/TRON assets.