Chapter 21 · TTL, Retention, Expiration, Backfills, and Lifecycle Automation
Soft Delete vs TTL vs Scheduled / Batch Cleanup vs Archive: Compliance and Product Requirements
Choose deliberately among soft delete, TTL, scheduled cleanup, archive, and legal hold by matching product semantics, deletion latency, compliance, recoverability, and operational cost.
1. AtlasMart problem: “delete after 90 days” hides four different requirements
A product manager says events should be deleted after 90 days. Security says users need an immediate “delete my account” experience. Legal says some audit records must survive seven years and can be placed on hold. Operations wants a recovery window for accidental deletion. Those are not one TTL policy. They are distinct lifecycle contracts requiring different mechanisms.
- Choose among soft delete, TTL, scheduled/batch cleanup, and archive based on semantics rather than convenience.
- Separate user-visible removal from physical deletion and long-term retention.
- Design legal-hold exclusions so automated cleanup cannot erase protected evidence.
- Account for query correctness, storage cost, recovery, trigger side effects, and subcollections.
- Document retention classes as policy data that can be tested and audited.
Use the Emulator Suite, a Firebase demo project, or an isolated test project for destructive, security-sensitive, billing-sensitive, migration, backup/restore, or write-heavy exercises unless the lesson explicitly marks managed verification as required. Treat shown output as expected evidence unless it is explicitly identified as captured output, and re-check current Firebase/Google Cloud edition, mode, quota, pricing, and security documentation before production execution.
AtlasMart keeps the course-wide project identity
demo-atlasmart-firestore. Mandatory work is local
and no-cost: Node.js 22+, Firebase CLI 15.30.0,
Firestore emulator 127.0.0.1:8080, Auth emulator
127.0.0.1:9099, Emulator UI
127.0.0.1:4000, and the same Standard-edition
Native-mode mental model used by earlier chapters. The default
database is (default). TTL sweeping itself is
not treated as an emulator guarantee: the lab
models eligibility and lifecycle rules deterministically,
while an optional real-project check verifies actual managed
TTL deletion only in an isolated billed project. TTL deletes
are excluded from Firestore's free usage and require billing.
Expiration is a business fact; TTL deletion is an
asynchronous storage-cleanup mechanism.
A document whose expireAt is in the past may
still exist and be returned by queries until the managed
sweeper deletes it. Therefore product behavior must filter or
reject expired state explicitly when timeliness matters. TTL
is not a scheduler, not a transaction boundary, not a
legal-hold engine, and not a substitute for backups/PITR.
2. Mechanism comparison
| Mechanism | Best fit | Timing | Recoverability | Main risk |
|---|---|---|---|---|
| Soft delete | Immediate product invisibility with reversible state | Application-controlled | High until purge | Every query must exclude deleted state; stale copies remain |
| TTL | Low-urgency physical cleanup | Asynchronous; typically within ~24h | Depends on backup/PITR, not TTL itself | Mistaking eligibility for exact deletion |
| Scheduled/batch cleanup | Bounded time window, explicit ordering/side effects | Scheduler/worker-controlled | Can stage and audit | Worker retries/idempotency/rate control required |
| Archive | Long retention, lower-cost/analytical or compliance store | Pipeline-controlled | Designed for retained copy | Access/security/export governance |
| Legal hold | Override that blocks deletion | Until authorized release | Intentional retention | Automation must never bypass hold |
3. Retention classes as explicit policy
{ "session-30d": {"visibility":"expires-at-business-time", "physical":"ttl", "days":30, "legalHoldAllowed":false}, "event-90d": {"visibility":"normal", "physical":"ttl", "days":90, "legalHoldAllowed":true}, "user-delete": {"visibility":"immediate-soft-delete", "physical":"workflow", "days":0, "legalHoldAllowed":true}, "audit-7y": {"visibility":"restricted", "physical":"archive-or-explicit-workflow", "days":2557, "legalHoldAllowed":true}}Store the policy identifier with each document, but do not let an untrusted client choose a retention class that shortens or extends regulated retention. Server-side validation or Security Rules should constrain allowed transitions. For audit records on hold, leave TTL absent/null or otherwise keep them outside the TTL deletion path until hold release is authorized.
4. Soft delete is a query contract, not a boolean decoration
If AtlasMart adds deletedAt but forgets one
collection-group query, listener, export, search index, or
materialized view, the supposedly deleted record can resurface.
A soft-delete design therefore needs a complete query inventory
and regression tests.
import assert from "node:assert/strict";const visible = docs => docs.filter(d => d.deletedAt == null && !(d.expireAtMs <= NOW_MS));assert.deepEqual(visible([ {id:"a",deletedAt:null,expireAtMs:NOW_MS+1000}, {id:"b",deletedAt:NOW_MS-10,expireAtMs:NOW_MS+1000}, {id:"c",deletedAt:null,expireAtMs:NOW_MS-10}]).map(x=>x.id), ["a"]);5. Scheduled cleanup when timing or sequencing matters
Use an explicit worker when the business requirement is stronger than TTL: release reservations promptly, export before delete, delete nested children, write audit evidence, or coordinate external systems. The worker must be idempotent, resumable, scoped to the correct project/database, and rate-limited. After the workflow completes, TTL can still serve as a last-mile stale-record cleanup mechanism.
6. Archive before delete: prove the handoff
An archive workflow is not complete when the write call returns. Record source ID/version, archive object/key, canonical hash or row count, export timestamp, encryption/access policy, and a durable “archive verified” checkpoint. Only then should destructive cleanup become eligible. If the archive is subject to legal retention, deleting the Firestore source does not delete the archive—and the reverse is also true.
7. Controlled failure: put audit logs under the same TTL as sessions
Seed an audit document with legalHold=true and a
past expireAt. The lifecycle contract must fail
before any destructive action. Repair by making legal hold an
explicit override, removing/neutralizing the TTL field while
held, and recording who authorized any later release. A UI
checkbox is not a legal-hold control plane.
8. Compliance and recovery are separate axes
TTL tells Firestore when data becomes eligible for deletion; it does not create a recovery copy. Multi-region availability does not protect against logical deletion. Chapter 22 will add backup/PITR recovery design. Here, every destructive mechanism must declare whether recovery is required and which system provides it.
Production judgment
Choose the simplest mechanism that meets the strongest requirement. If visibility must change instantly but physical deletion may lag, use soft delete plus later purge. If deletion can be eventual and no pre-delete workflow is required, TTL fits. If actions must happen in sequence or on a bounded schedule, use an explicit job. If retention outlives operational data, archive. If a legal hold exists, automated deletion must be disabled until authorized release.
Verification checklist and cleanup
- Every retention class states visibility, physical-delete mechanism, recovery source, and hold behavior.
- All queries/listeners/search projections respect soft-delete/expiry rules.
- Archive verification precedes destructive cleanup.
- Legal hold cannot be bypassed by client data.
- Nested subcollections are included explicitly.
Bridge to Lesson 4
Once the mechanism is chosen, the hardest operational step is often enabling it on existing data. Lesson 4 treats TTL backfill as a production migration with checkpoints, monitoring, failure injection, and cost controls.
Knowledge check
- When is soft delete preferable to TTL alone?
- Why is a legal hold incompatible with blind TTL?
- When should scheduled cleanup replace TTL?
- Does archiving automatically solve deletion compliance?
- What does TTL provide for recovery?
Review the answers
1. When product visibility must change immediately or records need a reversible state before later purge.
2. A hold intentionally overrides deletion; the TTL field/path must be disabled or withheld for protected data.
3. When timing, ordering, nested deletion, export-before-delete, or external side effects must be controlled explicitly.
4. No. Archive access, retention, encryption, deletion, and verification need their own policies.
5. Nothing by itself. Recovery requires backups/PITR/export/another retained system.
Summary and next step
This lesson established the working contract for Soft Delete vs TTL vs Scheduled/Batch Cleanup vs Archive: Compliance and Product Requirements. Keep its edition/mode assumptions, trust boundary, verification evidence, and operational constraints explicit when reusing the pattern.
Next, continue to Backfilling TTL Fields Safely, Monitoring Deletion, Event Side Effects, and Cost.
Authoritative references
- Firebase · Manage data retention with TTL policies
- Firebase · Firestore index overview and index exemptions
- Firebase · Cloud Firestore pricing and TTL billing
- Firebase · Firestore quotas and limits
- Firebase · Cloud Firestore triggers
- Firebase · Enterprise TTL indexes
- Google Cloud · MongoDB compatibility TTL indexes
- Google Cloud · MongoDB compatibility release notes
- Google Cloud · Firestore Monitoring metrics