Chapter 20 · Production Capstone: Build and Operate a Complete SQLite Application Database

Implement the Schema, Constraints, Indexes, Views, and Migrations

Implement and verify the FieldNotes schema as versioned SQLite migrations with STRICT tables, intentional keys, controlled JSON, workload-backed indexes, one stable view, and one documented trigger.

Beginner165–210 minutesMigration 001/002 + planner/schema verificationSQLite 3.53.4 baselineRequires JSON + STRICT/RETURNING-compatible runtimeLast reviewed: August 2026

From approved model to release-controlled schema

The logical plan is now stable enough to implement. The capstone uses two migrations on purpose: release 001 creates the normalized core and release 002 demonstrates controlled evolution. The application never edits sqlite_schema directly and never treats a production database as an ad-hoc scratchpad.

01

Implement the FieldNotes schema with intentional ROWID/WITHOUT ROWID, STRICT typing, foreign keys, checks, generated columns, defaults, and controlled JSON.

02

Create only access-pattern-backed indexes and prove them with EXPLAIN QUERY PLAN.

03

Use one view as a stable read interface and one trigger for a clearly documented invariant.

04

Create migration 001 and migration 002 using PRAGMA user_version as the release state.

05

Validate row counts, constraints, indexes, triggers, views, foreign keys, and integrity after migration.

06

Produce schema documentation and a reproducible schema dump/manifest.

Why ordinary ROWID for events, WITHOUT ROWID for small composite/text keys

site, device, inspection, and maintenance_note are ordinary rowid tables with INTEGER PRIMARY KEY. They need compact generated identifiers that are convenient foreign-key targets. device_tag has a natural composite key, so WITHOUT ROWID avoids a separate hidden rowid structure. sync_outbox uses a text event identifier as its primary key and also benefits from a single primary-key b-tree rather than a separate rowid plus unique index.

WITHOUT ROWID is not a performance badge

Use it when the primary-key organization matches the data model. The capstone does not convert every table simply because WITHOUT ROWID exists.

Migration 001: create the production core

The migration assumes connection initialization has already enabled foreign-key enforcement. JSON text is validated because the design explicitly relies on JSON for variable device metadata and inspection measurements. The generated protocol column promotes one frequently queried JSON property into an indexable relational interface.

sql · migration 001
-- migrations/001_initial.sql-- Run only after connection initialization has enabled foreign_keys.BEGIN IMMEDIATE;CREATE TABLE site (    site_id     INTEGER PRIMARY KEY,    site_code   TEXT NOT NULL UNIQUE,    site_name   TEXT NOT NULL,    active      INTEGER NOT NULL DEFAULT 1 CHECK (active IN (0,1)),    created_at  TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP) STRICT;CREATE TABLE device (    device_id       INTEGER PRIMARY KEY,    site_id         INTEGER NOT NULL REFERENCES site(site_id),    device_code     TEXT NOT NULL UNIQUE,    device_name     TEXT NOT NULL,    status          TEXT NOT NULL DEFAULT 'active'                    CHECK (status IN ('active','inspection_due','retired')),    metadata_json   TEXT NOT NULL DEFAULT '{}'                    CHECK (json_valid(metadata_json)),    protocol        TEXT GENERATED ALWAYS AS                    (CASE WHEN json_valid(metadata_json)                          THEN json_extract(metadata_json,'$.protocol') END) STORED,    created_at      TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP) STRICT;CREATE TABLE inspection (    inspection_id      INTEGER PRIMARY KEY,    device_id          INTEGER NOT NULL REFERENCES device(device_id),    request_id         TEXT NOT NULL UNIQUE,    technician_name    TEXT NOT NULL,    started_at         TEXT NOT NULL,    finished_at        TEXT,    outcome            TEXT NOT NULL                       CHECK (outcome IN ('pass','follow_up','failed')),    summary            TEXT NOT NULL,    measurements_json  TEXT NOT NULL DEFAULT '{}'                       CHECK (json_valid(measurements_json)),    created_at         TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,    CHECK (finished_at IS NULL OR finished_at >= started_at)) STRICT;CREATE TABLE maintenance_note (    note_id        INTEGER PRIMARY KEY,    device_id      INTEGER NOT NULL REFERENCES device(device_id),    inspection_id  INTEGER REFERENCES inspection(inspection_id),    occurred_at    TEXT NOT NULL,    severity       TEXT NOT NULL DEFAULT 'info'                   CHECK (severity IN ('info','warning','critical')),    note_text      TEXT NOT NULL) STRICT;CREATE TABLE device_tag (    device_id  INTEGER NOT NULL REFERENCES device(device_id),    tag        TEXT NOT NULL,    PRIMARY KEY (device_id, tag)) STRICT, WITHOUT ROWID;CREATE TABLE sync_outbox (    event_id        TEXT PRIMARY KEY,    aggregate_type  TEXT NOT NULL CHECK (aggregate_type IN ('inspection','device')),    aggregate_id    INTEGER NOT NULL,    payload_json    TEXT NOT NULL CHECK (json_valid(payload_json)),    created_at      TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,    delivered_at    TEXT) STRICT, WITHOUT ROWID;CREATE INDEX idx_device_site_status    ON device(site_id, status);CREATE INDEX idx_device_protocol    ON device(protocol) WHERE protocol IS NOT NULL;CREATE INDEX idx_inspection_device_started    ON inspection(device_id, started_at DESC);CREATE INDEX idx_note_device_occurred    ON maintenance_note(device_id, occurred_at DESC);CREATE INDEX idx_outbox_pending    ON sync_outbox(created_at) WHERE delivered_at IS NULL;CREATE VIEW v_active_device ASSELECT s.site_code,       d.device_id,       d.device_code,       d.device_name,       d.status,       d.protocolFROM site AS sJOIN device AS d ON d.site_id = s.site_idWHERE s.active = 1  AND d.status <> 'retired';CREATE TRIGGER trg_critical_note_marks_dueAFTER INSERT ON maintenance_noteWHEN NEW.severity = 'critical'BEGIN    UPDATE device       SET status = 'inspection_due'     WHERE device_id = NEW.device_id       AND status = 'active';END;PRAGMA user_version = 1;COMMIT;

Why these constraints belong in the database

RuleSchema mechanismWhy database-side
Known device/site status valuesCHECKEvery writer—not only one UI path—gets the invariant.
Valid JSON metadata/measurementsCHECK(json_valid(...))Malformed JSON is rejected before later extraction/index logic depends on it.
Device belongs to real siteFOREIGN KEYPrevents orphan operational records.
Inspection retry identityUNIQUE request_idMakes the idempotency contract durable across process restarts.
finish not before startCHECKRejects an obvious domain-invalid chronology in the chosen normalized text format.
Protocol extractionSTORED generated columnCentralizes a stable JSON path and makes it indexable without duplicating application parsing.
Critical note marks dueAFTER triggerOne small automatic invariant is acceptable because it is documented, testable, and local to the database.

Why these indexes exist—and why others do not

Each persistent index has a named access path. UNIQUE constraints already create supporting indexes, so the capstone does not duplicate site_code, device_code, or request_id with redundant manual indexes.

IndexSupported access pattern
idx_device_site_statusList non-retired or due devices at one site.
idx_device_protocolFind devices by the promoted protocol property without indexing rows where it is NULL.
idx_inspection_device_startedRecent history for one device, already ordered by time.
idx_note_device_occurredRecent notes for one device.
idx_outbox_pendingOldest undelivered sync work without indexing already delivered rows.
idx_inspection_open_priority (migration 002)Prioritize unfinished work after the requirement is introduced.

Use EXPLAIN QUERY PLAN as an acceptance test

Populate enough rows for the planner to have a choice, run ANALYZE or current PRAGMA optimize as appropriate, and inspect the plan. Exact output text is not an application API, but SCAN/SEARCH evidence tells you whether the access path matches your hypothesis.

sql · plan the hot device query
EXPLAIN QUERY PLANSELECT device_id, device_code, device_name, protocolFROM deviceWHERE site_id = 1  AND status = 'inspection_due'ORDER BY device_code;

The filter can use idx_device_site_status; the ORDER BY may still require a temporary b-tree because device_code is not the next index column. That is not automatically a bug. Add a wider index only after measuring this real query often enough to justify extra write/storage cost.

sql · plan recent history
EXPLAIN QUERY PLANSELECT inspection_id, started_at, outcome, summaryFROM inspectionWHERE device_id = ?ORDER BY started_at DESCLIMIT 20;

View and trigger: small, visible, testable

v_active_device is a read interface for UI/reporting code. It stores no materialized rows. The trigger has one side effect: a newly inserted critical note moves an active device to inspection_due. We do not hide synchronization, HTTP, audit chains, or multi-table workflows behind triggers.

sql · inspect dependencies
SELECT type, name, tbl_name, sqlFROM sqlite_schemaWHERE type IN ('view','trigger','index')ORDER BY type, name;
Document hidden behavior

The application repository and schema documentation must mention trg_critical_note_marks_due. A future developer should not have to reverse-engineer why device.status changed.

Migration 002: evolve with a release requirement

Release 2 adds priority to open inspections. The column receives a safe default so existing rows are valid, then a partial index supports only the unfinished-work queue. The migration is not written as “IF NOT EXISTS everywhere” because the policy is apply exactly once when user_version=1. Unexpected schema state is an error to investigate, not something to silently skip.

sql · migration 002
-- migrations/002_inspection_priority.sql-- Policy: migration 002 is applied once when user_version=1.BEGIN IMMEDIATE;ALTER TABLE inspectionADD COLUMN priority TEXT NOT NULL DEFAULT 'normal'CHECK (priority IN ('low','normal','high'));CREATE INDEX idx_inspection_open_priority    ON inspection(priority, started_at)    WHERE finished_at IS NULL;PRAGMA user_version = 2;COMMIT;

A tiny migration runner

The host application serializes migration at startup/release time. It checks the current version, applies only the expected next migration, and leaves an unexpected newer version unopened rather than guessing compatibility.

python · Python migration runner
from pathlib import PathMIGRATIONS = {    1: Path("migrations/001_initial.sql"),    2: Path("migrations/002_inspection_priority.sql"),}TARGET_VERSION = max(MIGRATIONS)def migrate(con):    current = con.execute("PRAGMA user_version").fetchone()[0]    if current > TARGET_VERSION:        raise RuntimeError(f"database schema {current} is newer than app {TARGET_VERSION}")    for version in range(current + 1, TARGET_VERSION + 1):        sql = MIGRATIONS[version].read_text(encoding="utf-8")        # Each migration file owns its explicit BEGIN IMMEDIATE / COMMIT boundary.        try:            con.executescript(sql)        except sqlite3.Error:            if con.in_transaction:                con.execute("ROLLBACK")            raise        actual = con.execute("PRAGMA user_version").fetchone()[0]        if actual != version:            raise RuntimeError(f"migration {version} did not set user_version")

Post-migration verification

Do not declare a migration successful because COMMIT returned. Verify structural and domain expectations on the migrated database.

sql · release verification
SELECT sqlite_version();PRAGMA user_version;PRAGMA foreign_keys;PRAGMA quick_check;PRAGMA foreign_key_check;SELECT type, name, tbl_nameFROM sqlite_schemaWHERE name NOT LIKE 'sqlite_%'ORDER BY type, name;SELECT name, "unique", origin, partialFROM pragma_index_list('inspection');-- Validate the generated JSON projection.SELECT device_code, protocol, json_valid(metadata_json)FROM deviceORDER BY device_id;

Expected: user_version=2, foreign keys ON for the current connection, quick_check returns ok, foreign_key_check returns no rows, and the inspection priority column/index exist.

Schema manifest and current dump

The CLI .schema command is convenient when available. Applications can also produce an auditable manifest directly from sqlite_schema. For a logical full-data export, Chapter 16's .dump or other intentional tooling is separate from the physical backup strategy.

sql · schema manifest
SELECT type, name, tbl_name, sqlFROM sqlite_schemaWHERE sql IS NOT NULL  AND name NOT LIKE 'sqlite_%'ORDER BY CASE type           WHEN 'table' THEN 1           WHEN 'index' THEN 2           WHEN 'view' THEN 3           WHEN 'trigger' THEN 4         END,         name;

Migration acceptance checkpoint

Prove the schema rather than admire it

Answer from the release engineer’s point of view.

  1. Why does migration 002 intentionally fail if the priority column already exists but user_version still says 1?
  2. Which indexes are already implied by UNIQUE constraints?
  3. Why is protocol a generated column instead of asking every query to repeat json_extract?
  4. What is the operational risk of adding a trigger without documenting its side effect?
  5. Why must foreign_key_check be run after rebuild-style migrations in addition to integrity_check?
  6. What does user_version=2 mean—and what does it not prove by itself?
Review the answers

Unexpected duplicate schema with an old user_version signals drift and should not be hidden. UNIQUE site_code, device_code and request_id already receive supporting indexes. The generated protocol expression centralizes a stable JSON path and can be indexed. Undocumented triggers create non-local behavior that surprises application developers. Structural integrity checking does not report foreign-key violations, so foreign_key_check is separate. user_version is application metadata saying which migration state the release claims; post-migration checks still have to prove the schema/data are healthy.

Bridge to application ownership

The database now enforces its invariants, but SQL does not call itself. Lesson 3 builds the primary Python access layer and deliberately maps each driver operation back to the transaction/concurrency rules learned in Chapters 8, 9, and 15.

Authoritative references

Keep knowledge open

Help the academy stay free and grow.

If these tutorials save you time, a small donation supports new lessons, technical review, diagrams, examples, and long-term maintenance.

ETHEthereum / ERC-20 only
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0

Send only assets compatible with the Ethereum/ERC-20 network. Do not send TRC-20/TRON assets.