Chapter 22 · Production Capstone: Design, Cluster, Secure, Tune, and Recover MariaDB

Tune SQL, InnoDB, Memory, I/O, Threading, and Galera/Replication from Evidence

Generate a representative workload, capture plans and server evidence, isolate the highest-impact bottleneck, change one variable at a time, and preserve reproducible before/after evidence across SQL, InnoDB, memory, I/O, threading, and HA state.

Advanced capstone230–290 minutesevidence-driven performance experimentMariaDB Community 12.3.2 current GA referenceCurriculum anchor: MariaDB 11.8 LTS · verify source/target/tool/topology versionsFree local tooling · Last reviewed: August 2026

Learning outcomes

ServiceHub is deployed to the capstone server and a queue endpoint is slower under load. The temptation is to increase buffer pool size, connection count, I/O capacity and thread-pool settings together. That destroys causality. This lesson makes tuning a controlled experiment: freeze workload and environment, capture a baseline, identify one bottleneck, change one thing, then rerun the same test.

01

Capture query plans, percentile latency, error rate and database/OS counters for a fixed workload window.

02

Diagnose a deliberately missing access path and prove the effect of restoring the workload-derived index.

03

Distinguish buffer-pool pressure, per-session memory, I/O pressure and CPU/thread queueing from one another.

04

Collect replication or Galera evidence only when that topology exists and avoid blaming “lag” without transport/apply/flow-control evidence.

05

Store before/after evidence and residual risks without fabricating benchmark results.

Performance rule

Every number in this lesson comes from your own run. The HTML provides commands, result schemas and decision rules, but no invented ops/s or latency values.

1. Freeze the experiment boundary

text · benchmark manifest
Record before each run:  server: SELECT VERSION(), @@version_comment  connector/runtime: node --version; npm ls mariadb  dataset: row counts + database/index bytes  cache state: warm or intentionally cold  topology: single primary / async replica / Galera; binlog on/off; wsrep state  container/VM CPU + memory limits  storage type and free space  workload seed, mix, concurrency, duration  schema migration versions  changed variable: exactly one planned differenceDo not compare runs if any uncontrolled item materially changed.
sql · capture server baseline
SELECT VERSION(), @@version_comment;SELECT COUNT(*) AS tickets FROM servicehub22.ticket;SELECT COUNT(*) AS comments FROM servicehub22.ticket_comment;SELECT ROUND(SUM(data_length+index_length)/1024/1024,1) AS servicehub_mbFROM information_schema.TABLES WHERE table_schema='servicehub22';SHOW GLOBAL STATUS WHERE Variable_name IN ( 'Threads_connected','Threads_running','Created_tmp_tables','Created_tmp_disk_tables', 'Innodb_buffer_pool_reads','Innodb_buffer_pool_read_requests', 'Innodb_buffer_pool_pages_dirty','Innodb_data_reads','Innodb_data_writes', 'Innodb_os_log_written','Bytes_received','Bytes_sent');

2. Scale the disposable dataset deterministically

sql · create helper sequence and add tickets
USE servicehub22;DROP TABLE IF EXISTS bench_seq;CREATE TABLE bench_seq (n INT PRIMARY KEY) ENGINE=InnoDB;INSERT INTO bench_seq VALUES (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);-- 10,000 deterministic rows from a four-digit Cartesian product.INSERT INTO ticket(tenant_id,ticket_id,customer_id,status,priority,subject,created_at,updated_at)SELECT 1,       100000 + (a.n*1000+b.n*100+c.n*10+d.n),       CASE WHEN MOD(a.n+b.n+c.n+d.n,2)=0 THEN 1001 ELSE 1002 END,       CASE MOD(a.n+b.n+c.n+d.n,4)         WHEN 0 THEN 'OPEN' WHEN 1 THEN 'PENDING'         WHEN 2 THEN 'RESOLVED' ELSE 'CLOSED' END,       1 + MOD(a.n+b.n+c.n+d.n,5),       CONCAT('benchmark ticket ',a.n,b.n,c.n,d.n),       TIMESTAMP('2026-01-01 00:00:00') + INTERVAL (a.n*1000+b.n*100+c.n*10+d.n) SECOND,       TIMESTAMP('2026-01-01 00:00:00') + INTERVAL (a.n*1000+b.n*100+c.n*10+d.n) SECONDFROM bench_seq a CROSS JOIN bench_seq b CROSS JOIN bench_seq c CROSS JOIN bench_seq d;SELECT COUNT(*) FROM ticket WHERE tenant_id=1;

If you rerun the lab, reset or use INSERT IGNORE intentionally; silent duplicate skipping can change workload cardinality. Keep the exact dataset count in the manifest.

3. Create a controlled regression and prove it in the plan

sql · remove one critical index in the disposable lab
ALTER TABLE servicehub22.ticket DROP INDEX ix_ticket_queue;EXPLAINSELECT tenant_id,ticket_id,status,priority,subject,updated_atFROM servicehub22.ticketWHERE tenant_id=1 AND status='OPEN'ORDER BY updated_at DESC, ticket_id DESCLIMIT 50;-- On supported/current versions also capture observed execution:ANALYZE FORMAT=JSONSELECT tenant_id,ticket_id,status,priority,subject,updated_atFROM servicehub22.ticketWHERE tenant_id=1 AND status='OPEN'ORDER BY updated_at DESC, ticket_id DESCLIMIT 50;

Look for access type, chosen key, estimated/observed rows and sort behavior. A full scan is not automatically wrong for a tiny table; the experiment matters only after the dataset and selectivity make the queue path representative.

4. Run a repeatable connector workload with real percentiles

terminal · free local application benchmark
mkdir capstone22-bench && cd capstone22-benchnpm init -ynpm install mariadbnode --versionnpm ls mariadb# Set DB_HOST/DB_USER/DB_PASSWORD in your shell or secret manager.# Do not commit credentials.
javascript · bench.mjs
import mariadb from 'mariadb';import { performance } from 'node:perf_hooks';const concurrency = Number(process.env.CONCURRENCY ?? 8);const iterations = Number(process.env.ITERATIONS ?? 100);const pool = mariadb.createPool({  host: process.env.DB_HOST ?? '127.0.0.1',  port: Number(process.env.DB_PORT ?? 3306),  database: 'servicehub22',  user: process.env.DB_USER,  password: process.env.DB_PASSWORD,  connectionLimit: concurrency,  acquireTimeout: 5000,  connectTimeout: 5000,  resetAfterUse: true});const samples=[];let errors=0;async function one(worker, i) {  let conn;  const t0=performance.now();  try {    conn=await pool.getConnection();    if ((i + worker) % 5 === 0) {      await conn.query(        `UPDATE ticket SET priority=CASE WHEN priority=5 THEN 1 ELSE priority+1 END         WHERE tenant_id=? AND ticket_id=?`,        [1, 100000 + ((i*37 + worker*101) % 10000)]);    } else {      await conn.query(        `SELECT ticket_id,status,priority,subject,updated_at         FROM ticket WHERE tenant_id=? AND status='OPEN'         ORDER BY updated_at DESC,ticket_id DESC LIMIT 50`, [1]);    }  } catch (e) { errors++; }  finally {    samples.push(performance.now()-t0);    if (conn) conn.release();  }}await Promise.all(Array.from({length: concurrency}, (_,w) =>  (async()=>{ for(let i=0;i<iterations;i++) await one(w,i); })()));await pool.end();samples.sort((a,b)=>a-b);const pct=p=>samples[Math.min(samples.length-1, Math.floor((samples.length-1)*p))];console.log(JSON.stringify({  concurrency, samples:samples.length, errors,  p50_ms:pct(.50), p95_ms:pct(.95), p99_ms:pct(.99), max_ms:samples.at(-1)}, null, 2));

Run the script several times and preserve all raw JSON. Warm up before the recorded window. A single p99 is noisy; compare distributions and error counts under identical conditions.

5. Repair one bottleneck and rerun the identical experiment

sql · restore the workload-derived queue index
ALTER TABLE servicehub22.ticket  ADD KEY ix_ticket_queue (tenant_id,status,updated_at,ticket_id);EXPLAINSELECT tenant_id,ticket_id,status,priority,subject,updated_atFROM servicehub22.ticketWHERE tenant_id=1 AND status='OPEN'ORDER BY updated_at DESC,ticket_id DESCLIMIT 50;
text · evidence ledger
experiment_id: 22-L3-index-queuebaseline:  schema_version: record the exact applied migration version  plan: save the complete EXPLAIN/ANALYZE output  p50/p95/p99/errors: save the raw benchmark JSON  server counters before/after: save both snapshotschange:  add ix_ticket_queue(tenant_id,status,updated_at,ticket_id)after:  same dataset/cache/concurrency/mix/duration  plan: save the complete post-change plan  p50/p95/p99/errors: save the post-change raw JSONresult:  accepted / rejected based on predeclared criteriaresidual_risks:  write/index maintenance cost, storage growth, different tenant cardinalities

6. Diagnose resource pressure by symptom family

Evidence family Useful signals What it can support What it cannot prove alone
Buffer pool Innodb_buffer_pool_reads / read_requests; pages dirty physical-read pressure, dirty-page trend the “correct” buffer size from hit ratio alone
Per-session/temp Created_tmp_disk_tables; sort/join plan; concurrency spill/memory multiplication risk that raising every session buffer is safe
I/O/redo Innodb_data_writes; Innodb_os_log_written; OS latency write pressure/checkpoint correlation that higher innodb_io_capacity fixes slow storage
Threads/CPU Threads_running, pool/queue metrics, OS run queue CPU/queue saturation that max_connections should equal peak clients
Async replica SHOW REPLICA STATUS: IO/SQL state, GTID, lag fields receive/apply health zero data loss or correctness of application routing
Galera wsrep_ready, cluster_status, local_state, flow-control metrics quorum/state/backpressure that every write workload scales linearly
sql · topology-aware evidence probes
-- Async replication: run only on a configured replica.SHOW REPLICA STATUS\G-- Galera: run only when wsrep/Galera is actually enabled.SHOW GLOBAL STATUS LIKE 'wsrep_%';-- Always record effective thread/memory configuration.SHOW VARIABLES WHERE Variable_name IN ('max_connections','thread_handling','innodb_buffer_pool_size',  'tmp_table_size','max_heap_table_size','sort_buffer_size','join_buffer_size');

7. Wrong approach: tune five knobs until one run looks faster

Changing buffer pool, thread handling, connection limits, redo capacity and I/O capacity in one restart can produce a better benchmark while leaving you unable to explain why. Worse, the combination can create new memory or durability risks. The repair is a queue of hypotheses: make the highest-evidence, lowest-risk change first, rerun the identical workload, and revert changes that do not meet the predeclared acceptance criterion.

8. Lab checks and cleanup boundary

sql · leave schema correct after the experiment
-- Confirm the critical index was restored.SELECT index_name,GROUP_CONCAT(column_name ORDER BY seq_in_index) columns_in_orderFROM information_schema.STATISTICSWHERE table_schema='servicehub22' AND table_name='ticket'  AND index_name='ix_ticket_queue'GROUP BY index_name;-- Keep benchmark rows for Lesson 4 unless you need a small reset.DROP TABLE IF EXISTS servicehub22.bench_seq;

Check your reasoning

  1. Why is a warm run not comparable to a cold run?
  2. Why can a missing index be a stronger first hypothesis than buffer-pool size?
  3. Why collect error rate with percentiles?
  4. What does Threads_connected fail to measure?
  5. When should wsrep metrics be used?
Review the answers
  1. Cache state changes physical I/O and latency; if cache state is uncontrolled, the changed variable is not the only difference.

  2. EXPLAIN/ANALYZE can directly show an inefficient access/sort path for the affected query, while buffer sizing without pressure evidence is speculation.

  3. A “fast” benchmark that drops or times out requests is not meeting the service contract.

  4. Active execution concurrency. Many pooled/idle sessions can exist while Threads_running remains small.

  5. Only when Galera/wsrep is actually enabled; otherwise they are irrelevant to the async-replication baseline.

Production judgment and bridge to Lesson 4

Performance engineering is now part of the operating record: workload, environment, plan, counters, changed variable and residual risk. Lesson 4 adds the controls that matter when performance is irrelevant because the primary is gone, a backup is unusable, or an operator needs to fail over safely.

Authoritative references

Keep knowledge open

Help the academy stay free and grow.

If these tutorials save you time, a small donation supports new lessons, technical review, diagrams, examples, and long-term maintenance.

ETHEthereum / ERC-20 only
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0

Send only assets compatible with the Ethereum/ERC-20 network. Do not send TRC-20/TRON assets.