Expiration/Retention Policies and the Difference Between Event Time and Ingest Time

Learning objectives

1. Retention is a data policy expressed through the timeField

AtlasMart wants raw freezer telemetry for 30 days, while long-term rollups live elsewhere. On a time-series collection, expireAfterSeconds defines expiration relative to the collection's timeField. That means the choice between event time and ingest time is not cosmetic: it determines which clock the retention policy follows.

MongoDB does not delete each measurement the instant it reaches the threshold. Time-series TTL removes expired buckets. A bucket is eligible only after all measurements in it are expired, then a background process removes it on a later pass. The TTL monitor runs periodically (documented as every 60 seconds), and actual deletion can take longer under workload.

2. Create event-time retention with intentionally short bucket spans

docker rm -f atlasmart-ch19-l4 2>/dev/null || truedocker volume rm atlasmart-ch19-l4-data 2>/dev/null || truedocker run -d --name atlasmart-ch19-l4 \  -p 127.0.0.1:27163:27017 \  -v atlasmart-ch19-l4-data:/data/db \  mongodb/mongodb-community-server:8.3.8-ubuntu2204-slim \  --bind_ip_alldocker exec atlasmart-ch19-l4 mongosh --quiet --eval 'printjson(db.adminCommand({buildInfo:1}).version);printjson(db.adminCommand({getParameter:1,featureCompatibilityVersion:1}).featureCompatibilityVersion);' 

const d=db.getSiblingDB("atlasmart");d.telemetry_ch19_l4.drop();d.createCollection("telemetry_ch19_l4",{  timeseries:{    timeField:"eventAt",    metaField:"meta",    bucketMaxSpanSeconds:60,    bucketRoundingSeconds:60  },  expireAfterSeconds:120});const now=new Date();d.telemetry_ch19_l4.insertMany([  {eventAt:new Date(now.getTime()-10*60*1000),ingestedAt:now,meta:{tenantId:"tenant-a",sensorId:"late-old"},v:10},  {eventAt:new Date(now.getTime()-9*60*1000),ingestedAt:now,meta:{tenantId:"tenant-a",sensorId:"late-old"},v:11},  {eventAt:new Date(now.getTime()-20*1000),ingestedAt:now,meta:{tenantId:"tenant-a",sensorId:"fresh"},v:20}]);printjson(d.runCommand({listCollections:1,filter:{name:"telemetry_ch19_l4"}}).cursor.firstBatch[0]);printjson(d.getCollection("system.buckets.telemetry_ch19_l4").find({}, {meta:1,"control.min.eventAt":1,"control.max.eventAt":1}).toArray());print("public count immediately after insert",d.telemetry_ch19_l4.countDocuments({}));

The two old event-time measurements are already beyond the 120-second threshold even though ingestedAt is current. They can still be visible immediately because deletion is asynchronous and bucket-based. The fresh measurement is in a different metadata series/bucket and should not be considered expired.

3. Poll for evidence; never assert exact deletion time

const d=db.getSiblingDB("atlasmart");for (let i=0;i<15;i++) {  const c=d.telemetry_ch19_l4.countDocuments({});  print(new Date().toISOString(),"count",c);  if (c===1) break;  sleep(10000);}printjson(d.telemetry_ch19_l4.find({}, {_id:0,eventAt:1,ingestedAt:1,"meta.sensorId":1,v:1}).toArray());

The expected invariant is eventual removal of fully expired buckets, not “delete at exactly T+120 seconds.” If the bounded poll still shows expired data, record that as real retention lag rather than declaring the lab failed. Workload and bucket timing affect when removal becomes visible.

4. Event-time versus ingest-time retention changes the answer for late data

const d=db.getSiblingDB("atlasmart");d.telemetry_ingest_ch19_l4.drop();d.createCollection("telemetry_ingest_ch19_l4",{  timeseries:{timeField:"ingestedAt",metaField:"meta",bucketMaxSpanSeconds:60,bucketRoundingSeconds:60},  expireAfterSeconds:120});const now=new Date();d.telemetry_ingest_ch19_l4.insertOne({eventAt:new Date(now.getTime()-24*60*60*1000),ingestedAt:now,meta:{tenantId:"tenant-a",sensorId:"gateway-replay"},v:42});printjson(d.telemetry_ingest_ch19_l4.findOne({"meta.sensorId":"gateway-replay"},{_id:0,eventAt:1,ingestedAt:1,v:1}));

The same 24-hour-old event is “new” under ingest-time retention because its timeField is the current arrival time. That may be the right policy for a replay/audit queue, but it is different from “keep measurements for 30 days after they occurred.” Choose the clock from the business rule, not from whichever timestamp is easiest to populate.

5. Change retention safely and understand bulk eligibility

collMod can enable, change, or disable expireAfterSeconds on a time-series collection. Lowering retention can suddenly make a large historical population eligible for deletion, which can create storage/I/O pressure. In production, estimate eligible volume and observe deletion throughput before tightening retention aggressively.

const d=db.getSiblingDB("atlasmart");printjson(d.runCommand({listCollections:1,filter:{name:"telemetry_ch19_l4"}}).cursor.firstBatch[0].options);printjson(d.runCommand({collMod:"telemetry_ch19_l4",expireAfterSeconds:300}));printjson(d.runCommand({collMod:"telemetry_ch19_l4",expireAfterSeconds:"off"}));printjson(d.runCommand({listCollections:1,filter:{name:"telemetry_ch19_l4"}}).cursor.firstBatch[0].options);

6. Wrong assumptions: TTL is a scheduler, backup, or legal hold

TTL is inappropriate when a workflow must fire at an exact second; use an application scheduler/queue. TTL is also not backup: expiration is a delete that becomes part of database state, and independent recovery copies are still required. For legal/regulated retention, document whether “event time” or “ingest time” is the governing clock, how delayed deletion is handled, and how holds override normal expiration. A separate archive or policy-controlled collection is often clearer than dynamically defeating TTL on individual measurements.

7. Production judgment

Use time-series TTL when bounded raw-history retention aligns with bucket-level asynchronous removal. Monitor oldest retained event time, bucket ranges, TTL deletion activity, storage trends, and delayed-arrival distributions. Test retention changes against representative volumes. If stale measurements must disappear before authorization or compliance decisions, do not rely on eventual TTL deletion as the enforcement boundary—filter by time in the read path and enforce policy explicitly.

Bridge. Lesson 5 turns bucket density, index size, retention horizon, and ingestion rate into a capacity worksheet and a repeatable local load-measurement harness.

Authoritative references

• Time Series Collections — time-series model, writable non-materialized view, automatic index, and sharding boundaries.
• Create and Query a Time Series Collection — timeField/metaField, granularity, custom bucketing, and collection options.
• About Time Series Data and Bucketing — system.buckets organization, bucket catalog, bucket creation/closure, and out-of-order timestamps.
• Time Series Collection Considerations — metadata cardinality, bucket density, granularity, compression, and zone-sharding boundary.
• Set Granularity for Time Series Data — granularity and custom bucket span/rounding behavior.
• Add Secondary Indexes to Time Series Collections — automatic and additional indexes plus sort/query support.
• Time Series Collection Limitations — bucket limits and index/query/update restrictions.
• Automatic Removal for Time Series Collections — expireAfterSeconds, bucket-level TTL timing, and collMod.
• Time Series Compression — zstd and column-compression mechanisms and tradeoffs.
• $collStats — time-series storageStats and bucket diagnostic fields.
• serverStatus — bucket catalog counters and server-level time-series diagnostics.
• $setWindowFields — time-range windows, ordering requirements, and analytics behavior.
• Explain Results — queryPlanner/executionStats interpretation and plan-format caveats.
• MongoDB 8.3 Release Notes — current stable 8.3 series and time-series compatibility changes.
• mongosh Release Notes — mongosh 2.10.0 baseline.
• PyMongo Release Notes — PyMongo 4.17.x driver baseline where client measurement is used.