Chapter 17 · SQL Dialects, Tools, and Application Access
Migrations, Seed Data, and Database Changes in Git
A database schema is shared, stateful infrastructure. Changing model code is not enough: teams need an ordered history that transforms every supported database safely, records what ran, survives concurrent deployments, and can be reviewed like production code.
Learning outcomes
Learning outcomes
Treat migrations as an ordered, immutable transformation history rather than generated deployment side effects.
Design expand-and-contract changes compatible with rolling application deployments.
Separate reference, development, demonstration, and test seed data.
Track applied revisions, checksums, ownership, and execution evidence.
Review and test database changes in Git and CI, including fresh install, upgrade, rollback or forward-fix, and drift detection.
Desired model versus transformation history
Model state
ORM metadata or schema files describe what developers want now.
Migration history
Ordered revisions describe how existing databases reach that state.
Applied state
A version table records which revisions actually ran in an environment.
Drift
Manual or failed changes create differences not explained by versioned history.
Never assume that rebuilding an empty test database proves an upgrade is safe for real data.
Migration contract
| Property | Required behavior |
|---|---|
| Unique identity | Every revision has a stable ID and parent/dependency relation |
| Ordering | The tool computes a deterministic path from current to target revision |
| Immutability | An applied revision is not silently edited; a new corrective revision is added |
| Atomicity | Use a transaction where the engine and operation support it |
| Idempotent deployment | The orchestrator applies each revision once and handles concurrent deployers |
| Observability | Record revision, checksum, actor/build, timestamps, duration, and outcome |
| Compatibility | Schema and application versions overlap safely during rollout |
| Recovery | Define rollback when safe; otherwise define tested forward repair |
A minimal migration ledger
CREATE TABLE IF NOT EXISTS schema_migration ( revision_id TEXT PRIMARY KEY, parent_revision TEXT, checksum_sha256 TEXT NOT NULL, description TEXT NOT NULL, applied_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, applied_by TEXT NOT NULL, duration_ms INTEGER NOT NULL CHECK(duration_ms >= 0)) STRICT;SELECT revision_id, parent_revision, checksum_sha256, applied_atFROM schema_migrationORDER BY applied_at, revision_id;Expand, migrate, contract
| Phase | Database change | Application behavior |
|---|---|---|
| Expand | Add nullable column/table/index or compatible object | Old and new application versions continue to work |
| Dual compatibility | Optionally write/read both representations | Telemetry verifies the new path |
| Backfill | Move existing data in bounded restartable batches | Normal traffic remains available |
| Enforce | Add validated constraints and switch reads | New version becomes authoritative |
| Contract | Remove obsolete columns or compatibility code | Only after old application versions are gone |
ALTER TABLE customer ADD COLUMN normalized_email TEXT;CREATE INDEX idx_customer_normalized_email ON customer(normalized_email);-- Backfill in bounded batches in a separate operational job.UPDATE customerSET normalized_email = lower(trim(email))WHERE normalized_email IS NULL;-- Contract only after every deployed version uses normalized_email.-- ALTER TABLE customer ALTER COLUMN normalized_email SET NOT NULL;-- ALTER TABLE customer DROP COLUMN legacy_email;Alembic revision example
from alembic import opimport sqlalchemy as sarevision = "20260805_01"down_revision = "20260730_03"branch_labels = Nonedepends_on = Nonedef upgrade() -> None: op.add_column( "customer", sa.Column("normalized_email", sa.String(320), nullable=True), ) op.create_index( "idx_customer_normalized_email", "customer", ["normalized_email"], unique=False, )def downgrade() -> None: op.drop_index("idx_customer_normalized_email", table_name="customer") op.drop_column("customer", "normalized_email")Autogeneration is a draft. Review names, data movement, locking, defaults, constraint validation, server-version behavior, and downgrade safety.
Seed data taxonomy
| Seed category | Examples | Policy |
|---|---|---|
| Reference data | Country codes, workflow states, permission definitions | Versioned, deterministic, stable keys, safe upsert |
| Bootstrap identity | Initial service role or tenant | Secret supplied externally; rotate immediately |
| Development fixtures | Synthetic customers and orders | Never deployed to production |
| Demonstration data | Curated product tour dataset | Clearly labeled and removable |
| Test fixtures/factories | Minimal edge-case records | Owned by tests; isolated per run |
| Backfill data | Derived values for a schema transition | Migration or dedicated resumable job, not permanent seed |
INSERT INTO order_status(status_code, display_name, terminal)VALUES ('draft','Draft',0), ('submitted','Submitted',0), ('paid','Paid',1), ('cancelled','Cancelled',1)ON CONFLICT(status_code) DO UPDATE SET display_name = excluded.display_name, terminal = excluded.terminal;Database changes in Git
db/ migrations/ 20260805_01_add_normalized_email.py 20260807_01_backfill_normalized_email.py 20260812_01_enforce_normalized_email.py seeds/ reference_order_status.sql checks/ verify_normalized_email.sql README.md OWNERS| Pull-request question | Evidence |
|---|---|
| What is the data and lock impact? | Plan, table size, estimated duration, batch design |
| Can old and new app versions coexist? | Compatibility matrix and rollout order |
| How is failure detected? | Metrics, logs, revision status, verification query |
| How is it recovered? | Rollback or forward-fix procedure and backup dependency |
| Was generated SQL reviewed? | Offline SQL artifact or migration logs |
| Does it work from every supported state? | Fresh install, previous release upgrade, representative data tests |
| Who owns approval? | Database/platform owner plus application owner |
CI migration matrix
set -euo pipefail# 1. Build an empty database from the complete revision history.alembic upgrade headalembic current --check-headspytest tests/database# 2. Upgrade a copy representing the previous production release.restore_fixture previous_release.dbalembic upgrade headpython db/checks/verify_schema.pypytest tests/integration# 3. Render SQL for review where supported.alembic upgrade base:head --sql > migration-plan.sqlgit diff --exit-code -- migration-plan.sql || trueMigration review
- Why must applied migrations be immutable?
- Why is a nullable additive column often safer than an immediate required column?
- Why are large data backfills frequently separated from schema revisions?
- What does fresh-install testing fail to prove?
Review the answers
Editing history destroys reproducibility and checksum trust. Additive nullable changes allow old and new versions to coexist while data is populated. Backfills need batching, restartability, throttling, and separate monitoring. Fresh installs do not expose upgrade-time data conversion, locking, volume, or compatibility problems.
SQLite migration exercise
import hashlibimport sqlite3from pathlib import PathMIGRATIONS = [ ("001", "create product", "CREATE TABLE product(id INTEGER PRIMARY KEY, sku TEXT UNIQUE)"), ("002", "add active", "ALTER TABLE product ADD COLUMN active INTEGER NOT NULL DEFAULT 1"),]with sqlite3.connect("academy.db") as db: db.execute("""CREATE TABLE IF NOT EXISTS schema_migration( revision_id TEXT PRIMARY KEY, checksum_sha256 TEXT NOT NULL, description TEXT NOT NULL, applied_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP )""") applied = {row[0]: row[1] for row in db.execute( "SELECT revision_id, checksum_sha256 FROM schema_migration" )} for revision, description, sql in MIGRATIONS: checksum = hashlib.sha256(sql.encode()).hexdigest() if revision in applied: if applied[revision] != checksum: raise RuntimeError(f"edited applied migration: {revision}") continue db.execute(sql) db.execute( "INSERT INTO schema_migration(revision_id,checksum_sha256,description) VALUES(?,?,?)", (revision, checksum, description), )Chapter summary
- Version both the desired schema and the ordered path that changes existing databases.
- Use expand-and-contract for rolling compatibility and isolate large data movement.
- Keep reference seeds deterministic and production-safe; keep test/demo data separate.
- Make migration evidence, ownership, review, and verification part of the Git delivery workflow.
- Chapter 18 applies the complete course in a design-and-query capstone.