Chapter 20 · Production Capstone: Build and Operate a Complete SQLite Application Database
Implement the Schema, Constraints, Indexes, Views, and Migrations
Implement and verify the FieldNotes schema as versioned SQLite migrations with STRICT tables, intentional keys, controlled JSON, workload-backed indexes, one stable view, and one documented trigger.
From approved model to release-controlled schema
The logical plan is now stable enough to implement. The capstone uses two migrations on purpose: release 001 creates the normalized core and release 002 demonstrates controlled evolution. The application never edits sqlite_schema directly and never treats a production database as an ad-hoc scratchpad.
Implement the FieldNotes schema with intentional ROWID/WITHOUT ROWID, STRICT typing, foreign keys, checks, generated columns, defaults, and controlled JSON.
Create only access-pattern-backed indexes and prove them with EXPLAIN QUERY PLAN.
Use one view as a stable read interface and one trigger for a clearly documented invariant.
Create migration 001 and migration 002 using PRAGMA user_version as the release state.
Validate row counts, constraints, indexes, triggers, views, foreign keys, and integrity after migration.
Produce schema documentation and a reproducible schema dump/manifest.
Why ordinary ROWID for events, WITHOUT ROWID for small composite/text keys
site, device, inspection, and maintenance_note are ordinary rowid tables with INTEGER PRIMARY KEY. They need compact generated identifiers that are convenient foreign-key targets. device_tag has a natural composite key, so WITHOUT ROWID avoids a separate hidden rowid structure. sync_outbox uses a text event identifier as its primary key and also benefits from a single primary-key b-tree rather than a separate rowid plus unique index.
Use it when the primary-key organization matches the data model. The capstone does not convert every table simply because WITHOUT ROWID exists.
Migration 001: create the production core
The migration assumes connection initialization has already enabled foreign-key enforcement. JSON text is validated because the design explicitly relies on JSON for variable device metadata and inspection measurements. The generated protocol column promotes one frequently queried JSON property into an indexable relational interface.
-- migrations/001_initial.sql-- Run only after connection initialization has enabled foreign_keys.BEGIN IMMEDIATE;CREATE TABLE site ( site_id INTEGER PRIMARY KEY, site_code TEXT NOT NULL UNIQUE, site_name TEXT NOT NULL, active INTEGER NOT NULL DEFAULT 1 CHECK (active IN (0,1)), created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP) STRICT;CREATE TABLE device ( device_id INTEGER PRIMARY KEY, site_id INTEGER NOT NULL REFERENCES site(site_id), device_code TEXT NOT NULL UNIQUE, device_name TEXT NOT NULL, status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active','inspection_due','retired')), metadata_json TEXT NOT NULL DEFAULT '{}' CHECK (json_valid(metadata_json)), protocol TEXT GENERATED ALWAYS AS (CASE WHEN json_valid(metadata_json) THEN json_extract(metadata_json,'$.protocol') END) STORED, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP) STRICT;CREATE TABLE inspection ( inspection_id INTEGER PRIMARY KEY, device_id INTEGER NOT NULL REFERENCES device(device_id), request_id TEXT NOT NULL UNIQUE, technician_name TEXT NOT NULL, started_at TEXT NOT NULL, finished_at TEXT, outcome TEXT NOT NULL CHECK (outcome IN ('pass','follow_up','failed')), summary TEXT NOT NULL, measurements_json TEXT NOT NULL DEFAULT '{}' CHECK (json_valid(measurements_json)), created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, CHECK (finished_at IS NULL OR finished_at >= started_at)) STRICT;CREATE TABLE maintenance_note ( note_id INTEGER PRIMARY KEY, device_id INTEGER NOT NULL REFERENCES device(device_id), inspection_id INTEGER REFERENCES inspection(inspection_id), occurred_at TEXT NOT NULL, severity TEXT NOT NULL DEFAULT 'info' CHECK (severity IN ('info','warning','critical')), note_text TEXT NOT NULL) STRICT;CREATE TABLE device_tag ( device_id INTEGER NOT NULL REFERENCES device(device_id), tag TEXT NOT NULL, PRIMARY KEY (device_id, tag)) STRICT, WITHOUT ROWID;CREATE TABLE sync_outbox ( event_id TEXT PRIMARY KEY, aggregate_type TEXT NOT NULL CHECK (aggregate_type IN ('inspection','device')), aggregate_id INTEGER NOT NULL, payload_json TEXT NOT NULL CHECK (json_valid(payload_json)), created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, delivered_at TEXT) STRICT, WITHOUT ROWID;CREATE INDEX idx_device_site_status ON device(site_id, status);CREATE INDEX idx_device_protocol ON device(protocol) WHERE protocol IS NOT NULL;CREATE INDEX idx_inspection_device_started ON inspection(device_id, started_at DESC);CREATE INDEX idx_note_device_occurred ON maintenance_note(device_id, occurred_at DESC);CREATE INDEX idx_outbox_pending ON sync_outbox(created_at) WHERE delivered_at IS NULL;CREATE VIEW v_active_device ASSELECT s.site_code, d.device_id, d.device_code, d.device_name, d.status, d.protocolFROM site AS sJOIN device AS d ON d.site_id = s.site_idWHERE s.active = 1 AND d.status <> 'retired';CREATE TRIGGER trg_critical_note_marks_dueAFTER INSERT ON maintenance_noteWHEN NEW.severity = 'critical'BEGIN UPDATE device SET status = 'inspection_due' WHERE device_id = NEW.device_id AND status = 'active';END;PRAGMA user_version = 1;COMMIT;Why these constraints belong in the database
| Rule | Schema mechanism | Why database-side |
|---|---|---|
| Known device/site status values | CHECK | Every writer—not only one UI path—gets the invariant. |
| Valid JSON metadata/measurements | CHECK(json_valid(...)) | Malformed JSON is rejected before later extraction/index logic depends on it. |
| Device belongs to real site | FOREIGN KEY | Prevents orphan operational records. |
| Inspection retry identity | UNIQUE request_id | Makes the idempotency contract durable across process restarts. |
| finish not before start | CHECK | Rejects an obvious domain-invalid chronology in the chosen normalized text format. |
| Protocol extraction | STORED generated column | Centralizes a stable JSON path and makes it indexable without duplicating application parsing. |
| Critical note marks due | AFTER trigger | One small automatic invariant is acceptable because it is documented, testable, and local to the database. |
Why these indexes exist—and why others do not
Each persistent index has a named access path. UNIQUE constraints already create supporting indexes, so the capstone does not duplicate site_code, device_code, or request_id with redundant manual indexes.
| Index | Supported access pattern |
|---|---|
| idx_device_site_status | List non-retired or due devices at one site. |
| idx_device_protocol | Find devices by the promoted protocol property without indexing rows where it is NULL. |
| idx_inspection_device_started | Recent history for one device, already ordered by time. |
| idx_note_device_occurred | Recent notes for one device. |
| idx_outbox_pending | Oldest undelivered sync work without indexing already delivered rows. |
| idx_inspection_open_priority (migration 002) | Prioritize unfinished work after the requirement is introduced. |
Use EXPLAIN QUERY PLAN as an acceptance test
Populate enough rows for the planner to have a choice, run ANALYZE or current PRAGMA optimize as appropriate, and inspect the plan. Exact output text is not an application API, but SCAN/SEARCH evidence tells you whether the access path matches your hypothesis.
EXPLAIN QUERY PLANSELECT device_id, device_code, device_name, protocolFROM deviceWHERE site_id = 1 AND status = 'inspection_due'ORDER BY device_code;The filter can use idx_device_site_status; the ORDER BY may still require a temporary b-tree because device_code is not the next index column. That is not automatically a bug. Add a wider index only after measuring this real query often enough to justify extra write/storage cost.
EXPLAIN QUERY PLANSELECT inspection_id, started_at, outcome, summaryFROM inspectionWHERE device_id = ?ORDER BY started_at DESCLIMIT 20;View and trigger: small, visible, testable
v_active_device is a read interface for UI/reporting code. It stores no materialized rows. The trigger has one side effect: a newly inserted critical note moves an active device to inspection_due. We do not hide synchronization, HTTP, audit chains, or multi-table workflows behind triggers.
SELECT type, name, tbl_name, sqlFROM sqlite_schemaWHERE type IN ('view','trigger','index')ORDER BY type, name;The application repository and schema documentation must mention trg_critical_note_marks_due. A future developer should not have to reverse-engineer why device.status changed.
Migration 002: evolve with a release requirement
Release 2 adds priority to open inspections. The column receives a safe default so existing rows are valid, then a partial index supports only the unfinished-work queue. The migration is not written as “IF NOT EXISTS everywhere” because the policy is apply exactly once when user_version=1. Unexpected schema state is an error to investigate, not something to silently skip.
-- migrations/002_inspection_priority.sql-- Policy: migration 002 is applied once when user_version=1.BEGIN IMMEDIATE;ALTER TABLE inspectionADD COLUMN priority TEXT NOT NULL DEFAULT 'normal'CHECK (priority IN ('low','normal','high'));CREATE INDEX idx_inspection_open_priority ON inspection(priority, started_at) WHERE finished_at IS NULL;PRAGMA user_version = 2;COMMIT;A tiny migration runner
The host application serializes migration at startup/release time. It checks the current version, applies only the expected next migration, and leaves an unexpected newer version unopened rather than guessing compatibility.
from pathlib import PathMIGRATIONS = { 1: Path("migrations/001_initial.sql"), 2: Path("migrations/002_inspection_priority.sql"),}TARGET_VERSION = max(MIGRATIONS)def migrate(con): current = con.execute("PRAGMA user_version").fetchone()[0] if current > TARGET_VERSION: raise RuntimeError(f"database schema {current} is newer than app {TARGET_VERSION}") for version in range(current + 1, TARGET_VERSION + 1): sql = MIGRATIONS[version].read_text(encoding="utf-8") # Each migration file owns its explicit BEGIN IMMEDIATE / COMMIT boundary. try: con.executescript(sql) except sqlite3.Error: if con.in_transaction: con.execute("ROLLBACK") raise actual = con.execute("PRAGMA user_version").fetchone()[0] if actual != version: raise RuntimeError(f"migration {version} did not set user_version")Post-migration verification
Do not declare a migration successful because COMMIT returned. Verify structural and domain expectations on the migrated database.
SELECT sqlite_version();PRAGMA user_version;PRAGMA foreign_keys;PRAGMA quick_check;PRAGMA foreign_key_check;SELECT type, name, tbl_nameFROM sqlite_schemaWHERE name NOT LIKE 'sqlite_%'ORDER BY type, name;SELECT name, "unique", origin, partialFROM pragma_index_list('inspection');-- Validate the generated JSON projection.SELECT device_code, protocol, json_valid(metadata_json)FROM deviceORDER BY device_id;Expected: user_version=2, foreign keys ON for the current connection, quick_check returns ok, foreign_key_check returns no rows, and the inspection priority column/index exist.
Schema manifest and current dump
The CLI .schema command is convenient when available. Applications can also produce an auditable manifest directly from sqlite_schema. For a logical full-data export, Chapter 16's .dump or other intentional tooling is separate from the physical backup strategy.
SELECT type, name, tbl_name, sqlFROM sqlite_schemaWHERE sql IS NOT NULL AND name NOT LIKE 'sqlite_%'ORDER BY CASE type WHEN 'table' THEN 1 WHEN 'index' THEN 2 WHEN 'view' THEN 3 WHEN 'trigger' THEN 4 END, name;Migration acceptance checkpoint
Prove the schema rather than admire it
Answer from the release engineer’s point of view.
- Why does migration 002 intentionally fail if the priority column already exists but user_version still says 1?
- Which indexes are already implied by UNIQUE constraints?
- Why is protocol a generated column instead of asking every query to repeat json_extract?
- What is the operational risk of adding a trigger without documenting its side effect?
- Why must foreign_key_check be run after rebuild-style migrations in addition to integrity_check?
- What does user_version=2 mean—and what does it not prove by itself?
Review the answers
Unexpected duplicate schema with an old user_version signals drift and should not be hidden. UNIQUE site_code, device_code and request_id already receive supporting indexes. The generated protocol expression centralizes a stable JSON path and can be indexed. Undocumented triggers create non-local behavior that surprises application developers. Structural integrity checking does not report foreign-key violations, so foreign_key_check is separate. user_version is application metadata saying which migration state the release claims; post-migration checks still have to prove the schema/data are healthy.
Bridge to application ownership
The database now enforces its invariants, but SQL does not call itself. Lesson 3 builds the primary Python access layer and deliberately maps each driver operation back to the transaction/concurrency rules learned in Chapters 8, 9, and 15.