Run Restore, Failover, Split-Brain, Performance, and Corruption Drills—and Defend the Design

drill_id,start_utc,detected_utc,recovered_utc,target_rpo_s,target_rto_s,actual_data_loss_s,actual_recovery_s,correctness_pass,slo_pass,evidence_uri,notesRules:  start/detected/recovered use one synchronized clock source  “correctness_pass” requires named invariants, not “site loads”  preserve logs/status/GTID/checksum evidence before cleanup  failure that exceeds target is useful evidence, not a reason to hide the run

primary: writable authority, server_id=221replica: read_only=ON, server_id=222, GTID replication healthyapplication: write endpoint points only to primaryprecondition: SHOW REPLICA STATUS confirms candidate has applied the test transactionbackup/PITR: independent recovery path still exists

# Record evidence before failure.docker compose exec replica mariadb -uroot -pdisposable-root   -e "SHOW REPLICA STATUS\G"# Kill only the disposable primary.docker compose stop primary# Do NOT route writes yet. First record replica state and fence/confirm old primary is stopped.docker compose exec replica mariadb -uroot -pdisposable-root   -e "SELECT @@server_id,@@read_only,@@global.gtid_slave_pos,@@global.gtid_binlog_pos;"# Promote the chosen replica only after the old writer is fenced.docker compose exec replica mariadb -uroot -pdisposable-root   -e "STOP REPLICA; SET GLOBAL read_only=OFF;"# Refresh/recreate application pools so no stale connections point at the dead writer.

# Recreate/reset the original primary+replica topology before this drill.docker network disconnect servicehub22_net servicehub22-replica# On the isolated replica ONLY, deliberately remove the guardrail for the exercise:docker exec servicehub22-replica mariadb -uroot -pdisposable-root   -e "SET GLOBAL read_only=OFF; INSERT INTO servicehub22.outbox_event(tenant_id,aggregate_type,aggregate_id,event_type,payload) VALUES(1,'ticket',5001,'WRONG_SIDE_WRITE','{}');"# Independently write a different event on the real primary.docker compose exec primary mariadb -uroot -pdisposable-root   -e "INSERT INTO servicehub22.outbox_event(tenant_id,aggregate_type,aggregate_id,event_type,payload) VALUES(1,'ticket',5001,'PRIMARY_WRITE','{}');"# Reconnect network only after collecting GTID/data evidence.docker network connect servicehub22_net servicehub22-replica

USE servicehub22;START TRANSACTION;UPDATE ticket SET subject='held by drill A'WHERE tenant_id=1 AND ticket_id=5001;-- Keep the transaction open temporarily; do not COMMIT yet.

SET SESSION innodb_lock_wait_timeout=10;UPDATE servicehub22.ticket SET priority=5WHERE tenant_id=1 AND ticket_id=5001;

SHOW PROCESSLIST;SELECT * FROM information_schema.INNODB_TRX\GSHOW ENGINE INNODB STATUS\G

SELECT NOW(6) AS before_bad_delete;SELECT COUNT(*) AS before_count FROM servicehub22.ticket WHERE tenant_id=1;-- Disposable incident injection.DELETE FROM servicehub22.ticketWHERE tenant_id=1 AND status='CLOSED';SELECT NOW(6) AS after_bad_delete;SELECT COUNT(*) AS after_count FROM servicehub22.ticket WHERE tenant_id=1;

cp -a /tmp/capstone22-full /tmp/capstone22-brokenfind /tmp/capstone22-full -type f -print0 | sort -z | xargs -0 sha256sum > /tmp/capstone22-full.sha256# Damage one disposable copy. Choose an ordinary data file inside the copied backup.TARGET=$(find /tmp/capstone22-broken -type f -name '*.ibd' | head -n 1)printf 'broken' > "$TARGET"# A manifest built for the GOOD tree can be adapted to compare paths, or generate paired manifests.find /tmp/capstone22-broken -type f -print0 | sort -z | xargs -0 sha256sum > /tmp/capstone22-broken.sha256diff -u /tmp/capstone22-full.sha256 /tmp/capstone22-broken.sha256 || true# Never use the broken copy for recovery. Preserve it only as drill evidence, then delete it.

# No production config file is edited.docker run --rm --name mariadb22-bad-config   -e MARIADB_ROOT_PASSWORD=disposable-root   mariadb:12.3.2   --definitely-not-a-real-option=1# Expected: server startup fails and logs identify an unknown/invalid option.# Capture the exit code and logs as evidence.

# Baseline first; then increase only one demand variable.CONCURRENCY=8  ITERATIONS=100 node bench.mjs  > baseline.jsonCONCURRENCY=32 ITERATIONS=100 node bench.mjs  > spike.json# Capture server/OS evidence over the same windows.# Do not claim 32 is “high” universally; it is just this drill's demand multiplier.

-- No cross-tenant foreign-key violations can exist if constraints are intact.SELECT COUNT(*) AS tickets_missing_customerFROM servicehub22.ticket tLEFT JOIN servicehub22.customer c  ON c.tenant_id=t.tenant_id AND c.customer_id=t.customer_idWHERE c.customer_id IS NULL;SELECT COUNT(*) AS comments_missing_ticketFROM servicehub22.ticket_comment cLEFT JOIN servicehub22.ticket t  ON t.tenant_id=c.tenant_id AND t.ticket_id=c.ticket_idWHERE t.ticket_id IS NULL;-- Record selected business counts and checksums appropriate to your dataset.SELECT tenant_id,status,COUNT(*)FROM servicehub22.ticketGROUP BY tenant_id,statusORDER BY tenant_id,status;CHECKSUM TABLE servicehub22.tenant,               servicehub22.customer,               servicehub22.ticket,               servicehub22.ticket_comment;

1. Workload and SLOs   exact user journeys, percentiles, availability, RPO/RTO, growth, retention2. Data correctness   schema invariants, tenant isolation, transactions, idempotency3. Capacity/performance   representative workload, bottleneck evidence, headroom and triggers4. Security   identities/roles, TLS verification, secrets, admin separation5. HA vs DR   write authority, replica/Galera choice, fencing/routing, backups/PITR6. Observability   user-impact signals, DB/OS evidence, alert thresholds and ownership7. Change management   migrations, config inventory, upgrade gate, canary, rollback boundary8. Game-day results   each drill's detection, RPO, RTO, correctness and failed assumptions9. Residual risks   what is consciously NOT solved10. Next 90-day priorities   highest risk-reduction work with measurable exit criteria