Defaults, Missing Fields, Derived Data, and Repairing Historical Documents

Learning outcomes

AtlasMart's new order model expects a shipping method and caches totalCents for fast reads. Historical documents, however, contain three different states: the field is missing, the field is explicitly null, or the derived total is stale. Treating all three as “bad data” and overwriting them with one default can erase meaning. A repair migration must distinguish absence from explicit unknown state and must recompute derived values only from trustworthy source fields.

1. A validator constrains values; it does not execute defaults

MongoDB's $jsonSchema implementation supports a documented subset of JSON Schema and explicitly omits the default keyword. Even in JSON Schema ecosystems where default is treated as annotation, validation itself is not a database defaulting engine. If AtlasMart needs every new order to store shippingMethod:"standard", the writer should set it, an update/migration should materialize it, or a server-side update expression should do so. Do not rely on a validator to create data.

2. Missing is not null, and {field:null} is intentionally broad

MongoDB equality to null matches documents where the field is explicitly null or does not exist. That is useful for some queries but dangerous in a repair script. In AtlasMart, an explicit null shipping method means “business process marked the method unknown,” while a missing field means “legacy writer predated the field.” Those are different histories and should not be collapsed accidentally.

// WRONG: equality to null matches explicit null and missing.db.orders_ch07_l4.updateMany(  { shippingMethod:null },  { $set:{shippingMethod:"standard"} });// This destroys the business distinction "explicitly unknown" vs "not written yet".// Safer when the business rule is “default only absent historical values”:db.orders_ch07_l4.updateMany(  { shippingMethod:{ $exists:false } },  { $set:{shippingMethod:"standard"} });

If the business wants explicit null to mean the same thing as missing, document that invariant first and then merge them deliberately. The database syntax should follow semantics, not create them.

3. Repair only what can be derived from trustworthy stored components

docker rm -f atlasmart-mongo-ch07-l4 2>/dev/null || truedocker volume rm atlasmart-mongo-ch07-l4-data 2>/dev/null || truedocker run -d --name atlasmart-mongo-ch07-l4 \  -p 127.0.0.1:27050:27017 \  -v atlasmart-mongo-ch07-l4-data:/data/db \  mongodb/mongodb-community-server:8.3.8-ubuntu2204-slimmongosh "mongodb://127.0.0.1:27050/atlasmart?directConnection=true" --quiet --eval \'printjson({server:db.version(), hello:db.hello().isWritablePrimary}); printjson(db.getSiblingDB("admin").runCommand({getParameter:1,featureCompatibilityVersion:1}))' 

const orders = db.getCollection("orders_ch07_l4");orders.drop();orders.insertMany([  { _id:"o-7401", schemaVersion:NumberInt(2), subtotalCents:NumberInt(10000), shippingCents:NumberInt(500), taxCents:NumberInt(800), discountCents:NumberInt(0), totalCents:NumberInt(11300), shippingMethod:"standard" },  { _id:"o-7402", schemaVersion:NumberInt(2), subtotalCents:NumberInt(5000), shippingCents:NumberInt(500), taxCents:NumberInt(400), discountCents:NumberInt(300), totalCents:NumberInt(999999) },  { _id:"o-7403", schemaVersion:NumberInt(2), subtotalCents:NumberInt(2000), shippingCents:NumberInt(0), taxCents:NumberInt(160), discountCents:NumberInt(0), totalCents:NumberInt(2160), shippingMethod:null },  { _id:"o-7404", schemaVersion:NumberInt(2), subtotalCents:NumberInt(3000), shippingCents:NumberInt(500), discountCents:NumberInt(0), totalCents:NumberInt(3500) }]);print("{shippingMethod:null} matches null AND missing:");printjson(orders.find({shippingMethod:null},{_id:1,shippingMethod:1}).sort({_id:1}).toArray());print("missing only:", orders.countDocuments({shippingMethod:{$exists:false}}));print("explicit null only:", orders.countDocuments({ $and:[ {shippingMethod:null}, {shippingMethod:{$exists:true}} ] }));// Safe default repair: only missing, not explicit null.printjson(orders.updateMany(  { shippingMethod:{$exists:false} },  { $set:{ shippingMethod:"standard" } }));const expectedTotal = {  $subtract:[    { $add:["$subtotalCents","$shippingCents","$taxCents"] },    "$discountCents"  ]};print("repairable total mismatches:");printjson(orders.find({  subtotalCents:{$type:"int"}, shippingCents:{$type:"int"}, taxCents:{$type:"int"}, discountCents:{$type:"int"},  $expr:{ $ne:["$totalCents", expectedTotal] }},{_id:1,totalCents:1}).toArray());print("unresolved: missing source fields:");printjson(orders.find({ $or:[{taxCents:{$exists:false}},{subtotalCents:{$exists:false}},{shippingCents:{$exists:false}},{discountCents:{$exists:false}}] },{_id:1}).toArray());const repair = orders.updateMany(  {    "migration.ch07L4.applied":{$ne:true},    subtotalCents:{$type:"int"}, shippingCents:{$type:"int"}, taxCents:{$type:"int"}, discountCents:{$type:"int"},    $expr:{ $ne:["$totalCents", expectedTotal] }  },  [    { $set:{      "migration.ch07L4.previousTotalCents":"$totalCents",      "migration.ch07L4.applied":true,      totalCents:expectedTotal    } }  ]);printjson(repair);print("remaining repairable mismatch:", orders.countDocuments({  subtotalCents:{$type:"int"}, shippingCents:{$type:"int"}, taxCents:{$type:"int"}, discountCents:{$type:"int"},  $expr:{ $ne:["$totalCents", expectedTotal] }}));

Order o-7402 has all four components needed to recompute the total, so the repair can be deterministic. Order o-7404 lacks taxCents; inventing tax from today's policy would rewrite history incorrectly. It remains in an unresolved set for business-specific reconciliation. This is the difference between data cleanup and data fabrication.

The migration marker records the original totalCents only on the first repair and prevents a rerun from overwriting that rollback value. In production, a migration marker may be temporary or kept in an audit trail; either way, define ownership and cleanup before adding arbitrary metadata to every document.

4. Derived data needs a freshness contract

A cached total, review count, inventory summary, or normalized search field is derived data: it can be recomputed from an authoritative source. Duplication can be valuable for read locality, but a stale derived field is a correctness defect if consumers mistake it for the source of truth. Specify when it is updated, how retries are idempotent, how drift is detected, and which query/report can reconcile it.

For high-value financial history, “recompute” may itself be unsafe if formulas, currencies, tax policy, or rounding rules changed. Store the inputs/version needed to reproduce the original calculation or treat the historical amount as authoritative. Schema repair should never silently apply current business policy to past events.

5. Verification, cleanup, and production judgment

Verification checklist

• The broad {shippingMethod:null} query returns both explicit-null and missing documents.
• The safe default backfill changes only documents where the field is missing.
• The derived-data query detects o-7402 as a repairable mismatch.
• The script separately reports records that lack source components instead of fabricating a value.
• The repair stores the previous total and marks the document so a retry is a no-op.
• A post-repair mismatch query reaches zero for the repairable subset while unresolved records remain visible.

Defaults and derived fields are modeling choices with read/write/index/storage consequences. Materializing a default makes it indexable and explicit but costs writes and storage. Leaving it missing avoids migration work but forces every consumer to preserve the same interpretation. Derived copies reduce read computation but increase update amplification and reconciliation burden. None of these choices improves durability or consistency automatically; those guarantees depend on the write path, concerns, topology, and invariant scope.

Backfills can be hot-aggregate or shard-skew amplifiers when many affected documents share a partition pattern. Observe latency tails, replication lag, disk/cache pressure, write tickets, and index growth. Use least-privilege maintenance identities and tenant-scoped filters. Keep rollback data or restore capability for destructive repairs.

The next lesson combines the chapter into a deployment gate: production-like samples, validator tests, migration idempotency, failure injection, cutover, and rollback rehearsal before enforcement reaches live data.

docker rm -f atlasmart-mongo-ch07-l4docker volume rm atlasmart-mongo-ch07-l4-data

Authoritative references

• MongoDB 8.3 release notes — Current server-series and patch-release status; re-check before reproducing the lab.
• MongoDB versioning — Explains major/minor release series and why the latest stable patch should be used.
• mongosh release notes — Current mongosh release and version-sensitive shell behavior.
• PyMongo release notes — Current Python driver line used when application-side migration behavior is demonstrated.
• $jsonSchema operator omissions — Documents unsupported keywords including default and the MongoDB bsonType extension.
• Query for null or missing fields — Official null-versus-missing query semantics.
• Updates with aggregation pipeline — Field-to-field deterministic repair expressions.
• Computed values pattern — Official guidance for storing calculated values when read savings justify update/reconciliation cost.
• Data consistency — Context for separating stored duplication from actual read consistency guarantees.
• Schema validation overview — Server-side write validation scope and purpose.