Chapter 19 · Application Integration, Connectors, Pools, ORMs, and Reliability Patterns
Health Checks, Graceful Degradation, Read Replicas, Migrations, and Deployment Coordination
Design cheap liveness/readiness checks, define replica-staleness policy, coordinate expand/contract migrations across mixed application versions, and degrade predictably during database failure instead of creating retry storms.
Learning outcomes
ServiceHub is now deployed as multiple application instances. A database restart makes every readiness probe retry aggressively; a deployment expects a new column before the migration is complete; and a read replica serves a just-written work order before it has applied the transaction. Reliability depends on coordinating application behavior with database topology and schema evolution.
Separate process liveness, database readiness, and deeper dependency diagnostics so health checks remain cheap and meaningful.
Define a read-replica consistency contract including read-after-write and acceptable staleness rather than routing reads blindly.
Coordinate expand/contract schema migrations with mixed old/new application versions and a separate migration identity.
Simulate unavailability/lag and apply bounded backoff/circuit-breaking behavior instead of retry storms.
Build deterministic integration acceptance tests that combine driver errors, server session evidence, schema compatibility, and business invariants.
Mandatory labs need only one local Community Server. Replica-specific commands are an optional extension if you still have the disposable Chapter 14 topology. The lesson teaches the consistency contract even when a learner cannot run a second instance.
Liveness, readiness, and deep diagnostics answer different questions
Liveness asks whether the application process should be restarted; it should not fail merely because MySQL is briefly unavailable if the process itself is healthy. Readiness asks whether this instance should receive traffic that requires its dependencies. A deep diagnostic can check privileges, replica state, schema version, or write/read behavior, but should not run on every high-frequency probe.
| Probe | Example | Failure action |
|---|---|---|
| liveness | in-process event loop/thread heartbeat | restart process only for process failure |
| readiness | short pooled checkout + SELECT 1 + required schema gate | remove instance from serving DB-dependent traffic |
| deep diagnostic | role/replica/schema/transaction test | operator/automation investigation, not high-frequency traffic gate |
import timedef database_ready(pool, expected_schema_version=2): started = time.monotonic() try: cnx = pool.get_connection() cur = cnx.cursor() cur.execute("SELECT 1") if cur.fetchone()[0] != 1: return False cur.execute(""" SELECT COALESCE(MAX(version_no),0) FROM servicehub_schema_version """) version = cur.fetchone()[0] return version >= expected_schema_version except Exception: return False finally: try: cur.close(); cnx.close() except Exception: pass elapsed = time.monotonic() - started # Record elapsed; do not sleep/retry inside a high-frequency probe.Repeated retries inside every readiness probe can turn a database outage into connection storms. Let the orchestrator/proxy control probe cadence while the application uses its own bounded retry/circuit policy for real requests.
Create an explicit schema-version gate for deployment coordination
USE servicehub_app_lab;CREATE TABLE IF NOT EXISTS servicehub_schema_version ( version_no INT PRIMARY KEY, applied_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), description VARCHAR(200) NOT NULL) ENGINE=InnoDB;INSERT INTO servicehub_schema_version(version_no,description)VALUES (1,'Chapter 19 base schema')ON DUPLICATE KEY UPDATE description='Chapter 19 base schema';SELECT * FROM servicehub_schema_version ORDER BY version_no;A migration ledger is not a substitute for validating the actual schema, but it gives deployment automation a deterministic compatibility gate. Use your framework’s migration system in real applications and a migration account with DDL privileges separate from the runtime account.
Compatibility failure: new code expects a column before old code can tolerate it
The unsafe deployment adds a required column and immediately switches all code to it. A safer expand/contract sequence first expands the schema in a backward-compatible way, deploys code that can write/read both forms, backfills if necessary, changes reads, then contracts only after old versions are gone.
-- Run as a migration identity, not servicehub_app.ALTER TABLE servicehub_app_lab.work_orders ADD COLUMN dispatch_zone VARCHAR(24) NULL, ALGORITHM=INSTANT;INSERT INTO servicehub_app_lab.servicehub_schema_version(version_no,description)VALUES (2,'Expand: nullable dispatch_zone')ON DUPLICATE KEY UPDATE description='Expand: nullable dispatch_zone';SHOW CREATE TABLE servicehub_app_lab.work_orders\GBefore relying on ALGORITHM=INSTANT, preflight the exact operation on the target MySQL version and representative schema. A compatibility error should stop the canary; do not silently fall back to a copy operation during a tight deployment window.
def create_order_v2(cur, customer_id, key, summary, dispatch_zone=None): cur.execute(""" INSERT INTO work_orders (customer_id,idempotency_key,summary,dispatch_zone,status,priority) VALUES (%s,%s,%s,%s,'open',3) """, (customer_id, key, summary, dispatch_zone))# Old v1 code that omits dispatch_zone still works because the column is nullable.# Contract (for example NOT NULL/removing old columns) happens only after v1 is gone.Read replicas: “readable” is not “read-after-write correct”
Asynchronous replicas can be healthy but behind. If a request creates a work order on the source and the next request immediately reads from a lagging replica, the row may be absent. Decide which endpoints tolerate staleness. Common policies include pinning a user/session to the writer for a short window, routing consistency-critical reads to the writer, or waiting for a known GTID to be applied before serving the read. The correct policy depends on product semantics.
-- On the replica, when that optional topology exists:SHOW REPLICA STATUS\GSELECT CHANNEL_NAME, SERVICE_STATE, LAST_ERROR_NUMBER, LAST_ERROR_MESSAGEFROM performance_schema.replication_applier_status_by_worker;SELECT @@global.gtid_executed;Seconds_Behind_Source and connection state are useful but incomplete. Correctness also needs GTID/application invariants. A delayed replica is intentionally stale and should never sit behind a generic “read pool” without policy.
Graceful degradation: bound the blast radius of an unavailable database
When MySQL is unavailable, allow requests that do not require it if the product can do so safely; reject or queue database-dependent work predictably. Keep retries bounded with exponential backoff and jitter, cap concurrent in-flight DB work, and open a circuit after sustained failure. Do not pretend a cached response is fresh if freshness matters.
import random, timeclass DatabaseUnavailable(RuntimeError): passdef call_db_with_budget(operation, attempts=3): for attempt in range(attempts): try: return operation() except DatabaseUnavailable: if attempt + 1 == attempts: raise time.sleep(min(0.8, 0.1 * (2 ** attempt)) + random.uniform(0, 0.05))# Production additionally needs a shared concurrency limit/circuit state so# thousands of requests do not each run an independent retry loop.Health checks must not use this retry loop. A probe should report the current dependency state quickly; request handling can apply the bounded policy.
Deterministic deployment acceptance tests
SELECT @@version AS server_version, @@read_only AS read_only, @@super_read_only AS super_read_only, @@transaction_isolation AS isolation_level, @@sql_mode AS sql_mode, @@time_zone AS session_time_zone;SELECT version_no, description, applied_atFROM servicehub_app_lab.servicehub_schema_versionORDER BY version_no;SELECT COUNT(*) AS duplicate_idempotency_keysFROM ( SELECT idempotency_key FROM servicehub_app_lab.work_orders GROUP BY idempotency_key HAVING COUNT(*) > 1) AS d;A deployment gate should also run an application integration test through the exact connector/ORM configuration: connect with TLS policy, verify account, execute one bound read, perform a disposable transaction with rollback, confirm schema compatibility, and verify pool/session reset. On a replica, add staleness/GTID checks appropriate to the endpoint’s consistency contract.
Wrong approach: one /health endpoint that runs writes and retries forever
A health endpoint that opens a new connection, performs DDL/DML, waits on replica catch-up, and retries until success creates workload precisely when the database is unhealthy. Split probes by purpose and make each cheap enough that monitoring cannot become the incident.
Chapter lab cleanup and bridge to Chapter 20
Keep servicehub_app_lab if you want to use it for the next chapter’s schema-migration and upgrade exercises. Otherwise drop only the disposable schema and user created by this chapter.
DROP DATABASE IF EXISTS servicehub_app_lab;DROP USER IF EXISTS 'servicehub_app'@'127.0.0.1';Chapter 20 turns the migration/deployment concerns from this lesson into the primary subject: schema migration, version upgrades, compatibility preflight, and zero/low-downtime change. Application reliability and database change management meet at that boundary.
Knowledge check
- Why should liveness usually not depend on MySQL being reachable?
- What does readiness need that liveness does not?
- Why can a healthy read replica violate read-after-write behavior?
- What is the purpose of expand/contract migration?
- Why should a readiness probe avoid internal retries?
Reveal answers
- A healthy application process can survive a dependency outage; tying liveness to MySQL can create restart storms without fixing the database.
- It needs enough dependency/schema evidence to decide whether the instance can safely receive the traffic it claims to serve.
- Asynchronous apply can lag, so the replica may not yet contain a transaction that the client just committed on the writer.
- Keep old and new application versions compatible during staged deployment, then remove legacy schema only after old code is gone.
- High-frequency retrying probes amplify outages; they should report state quickly while request traffic has a separate bounded retry/circuit policy.