Test Validators and Migration Scripts Against Production-Like Samples Before Enforcement

Learning outcomes

AtlasMart is ready to enforce version 2 of the customer schema, but a migration that works on three hand-picked documents is not evidence that it is safe for millions. A production-like sample preserves the important shape/cardinality/error strata of production without copying secrets or personal data. A migration acceptance harness runs candidate validation, repair, idempotency, compatibility, and rollback tests before enforcement reaches the live collection.

1. Production-like does not mean production data copied to a laptop

The sample must preserve behavioral dimensions: schema versions, optional-field absence, explicit null, long/empty strings, unusual but valid values, repairable legacy aliases, malformed types, and enough documents to exercise batches. It should not contain real customer identifiers, credentials, tokens, payment data, or confidential text. Generate deterministic synthetic values or sanitize data under a governed process.

2. Seed a deterministic 500-document AtlasMart sample

docker rm -f atlasmart-mongo-ch07-l5 2>/dev/null || truedocker volume rm atlasmart-mongo-ch07-l5-data 2>/dev/null || truedocker run -d --name atlasmart-mongo-ch07-l5 \  -p 127.0.0.1:27051:27017 \  -v atlasmart-mongo-ch07-l5-data:/data/db \  mongodb/mongodb-community-server:8.3.8-ubuntu2204-slimmongosh "mongodb://127.0.0.1:27051/atlasmart?directConnection=true" --quiet --eval \'printjson({server:db.version(), hello:db.hello().isWritablePrimary}); printjson(db.getSiblingDB("admin").runCommand({getParameter:1,featureCompatibilityVersion:1}))' 

from pymongo import MongoClientURI = "mongodb://127.0.0.1:27051/atlasmart?directConnection=true"client = MongoClient(URI, serverSelectionTimeoutMS=3000)coll = client.atlasmart.customers_ch07_l5coll.drop()docs = []# 350 ordinary v1 documents.for i in range(350):    docs.append({"_id": f"c-{i:04d}", "name": f"Customer {i}", "email": f"c{i}@example.test", "phone": None if i % 11 == 0 else f"+994-{100000+i}"})# 80 v1 documents with no optional phone field.for i in range(350, 430):    docs.append({"_id": f"c-{i:04d}", "name": f"Customer {i}", "email": f"c{i}@example.test"})# 50 already-v2 documents.for i in range(430, 480):    docs.append({"_id": f"c-{i:04d}", "schemaVersion": 2, "name": f"Customer {i}", "contact": {"email": f"c{i}@example.test", "phone": None}})# 15 repairable legacy variants: email stored under legacyEmail.for i in range(480, 495):    docs.append({"_id": f"c-{i:04d}", "name": f"Customer {i}", "legacyEmail": f"c{i}@example.test"})# 5 deliberately poisoned v2 samples: contact.email has wrong type but a trusted old email remains.for i in range(495, 500):    docs.append({"_id": f"c-{i:04d}", "schemaVersion": 2, "name": f"Customer {i}", "email": f"c{i}@example.test", "contact": {"email": i, "phone": None}})coll.insert_many(docs, ordered=True)print("seeded", coll.count_documents({}))client.close()

The fixture deliberately contains five documents whose schemaVersion already says 2 while contact.email has the wrong BSON type. A migration that filters only “version < 2” would miss them. This catches a common governance mistake: trusting a version marker more than the actual candidate schema.

3. Run the acceptance harness before strict enforcement

import statistics, timefrom pymongo import MongoClient, UpdateOnefrom pymongo.errors import WriteErrorURI = "mongodb://127.0.0.1:27051/atlasmart?directConnection=true"client = MongoClient(URI, serverSelectionTimeoutMS=3000)db = client.atlasmartcoll = db.customers_ch07_l5schema_v2 = {    "bsonType": "object",    "required": ["_id", "schemaVersion", "name", "contact"],    "properties": {        "_id": {"bsonType": "string"},        "schemaVersion": {"bsonType": "int", "enum": [2]},        "name": {"bsonType": "string", "minLength": 1},        "contact": {            "bsonType": "object",            "required": ["email"],            "properties": {                "email": {"bsonType": "string", "pattern": "@example\\.test$"},                "phone": {"bsonType": ["string", "null"]},            },        },    },}invalid = {"$nor": [{"$jsonSchema": schema_v2}]}def invalid_count(): return coll.count_documents(invalid)def distribution():    return list(coll.aggregate([        {"$group": {"_id": {"$ifNull": ["$schemaVersion", 1]}, "count": {"$sum": 1}}},        {"$sort": {"_id": 1}},    ]))print("initial distribution", distribution())print("candidate invalid", invalid_count())# Observation mode first; inserts that violate are allowed but logged.db.command({"collMod": "customers_ch07_l5", "validator": {"$jsonSchema": schema_v2}, "validationLevel": "moderate", "validationAction": "warn"})batch_times_ms = []BATCH = 50  # lab visibility only; not a production tuning value.while True:    source = list(coll.find({"$or": [{"schemaVersion": {"$exists": False}}, {"schemaVersion": 1}]}).sort("_id", 1).limit(BATCH))    if not source: break    ops = []    for d in source:        email = d.get("email") or d.get("legacyEmail")        if not email:            continue        ops.append(UpdateOne(            {"_id": d["_id"], "$or": [{"schemaVersion": {"$exists": False}}, {"schemaVersion": 1}]},            {"$set": {"schemaVersion": 2, "contact": {"email": email, "phone": d.get("phone")}}},        ))    t0 = time.perf_counter()    if ops: coll.bulk_write(ops, ordered=True)    batch_times_ms.append((time.perf_counter() - t0) * 1000)# Repair the deliberately poisoned synthetic v2 records from their retained trusted source.repair = coll.update_many(    {"schemaVersion": 2, "contact.email": {"$not": {"$type": "string"}}, "email": {"$type": "string"}},    [{"$set": {"contact.email": "$email"}}],)print("poison repair modified", repair.modified_count)print("after migration distribution", distribution())print("candidate invalid after repair", invalid_count())# Idempotency gate: source-version migration must now do zero work.rerun = coll.update_many(    {"$or": [{"schemaVersion": {"$exists": False}}, {"schemaVersion": 1}]},    [{"$set": {"schemaVersion": 2}}],)print("idempotency rerun modified", rerun.modified_count)if batch_times_ms:    ordered = sorted(batch_times_ms)    p95 = ordered[max(0, int(0.95 * len(ordered)) - 1)]    print("lab batch ms median/p95", round(statistics.median(ordered), 2), round(p95, 2))assert invalid_count() == 0assert rerun.modified_count == 0# Cutover only after the gates pass.db.command({"collMod": "customers_ch07_l5", "validator": {"$jsonSchema": schema_v2}, "validationLevel": "strict", "validationAction": "error"})try:    coll.insert_one({"_id": "old-client", "name": "Old", "email": "old@example.test"})    raise AssertionError("old-client write unexpectedly passed")except WriteError:    print("old-client write rejected as expected")coll.insert_one({"_id": "new-client", "schemaVersion": 2, "name": "New", "contact": {"email": "new@example.test", "phone": None}})print("new-client accepted", coll.count_documents({"_id": "new-client"}))client.close()

The exact median/p95 batch times depend on CPU, storage, cache warmth, Python/runtime, container virtualization, and host load; they are not supplied as expected constants. What the harness can assert deterministically is the state contract: after repair the candidate-invalid count is zero, the source-version rerun modifies zero documents, the old writer is rejected after cutover, and the new writer is accepted.

4. Deliberately wrong: bypass validation to make the migration green

MongoDB exposes bypassDocumentValidation for privileged maintenance workflows. That is a capability boundary, not a correctness shortcut. If a migration bypasses validation and the acceptance test only checks that writes were acknowledged, corrupted documents can enter the collection and strict enforcement can later trap them there. With access control enabled, use of the option requires the corresponding privilege action; application roles should not receive it casually.

# Anti-pattern: make the migration "look successful" by bypassing validation# and then enable strict enforcement without checking stored data.collection.insert_many(import_rows, bypass_document_validation=True)# A successful write result proves only the bypassed command ran; it does not prove data quality.# Safer gate:# 1) candidate-invalid count by $jsonSchema# 2) migration rerun modifies 0# 3) distribution matches supported versions# 4) old/new writer compatibility tests pass# 5) rollback/restore rehearsal succeeds# 6) only then switch to strict/error

A useful test suite also asserts that the ordinary application identity cannot bypass the rule. This chapter does not build production authentication—that is covered later—but the deployment checklist must record who can execute collMod and who can bypass validation.

5. Enforcement gate, rollback rehearsal, and production judgment

Verification checklist

• The synthetic dataset has 500 documents and contains all declared schema strata.
• The preflight candidate-invalid count is nonzero, proving the test actually contains migration work and poisoned target-state data.
• The migration repairs source-version documents and separately repairs the five deliberately poisoned v2 documents from retained trustworthy values.
• The final candidate-invalid count is zero.
• An immediate rerun of the source migration modifies zero documents.
• Strict/error cutover rejects the old-client shape and accepts the new-client shape.
• The reset path can recreate the exact original synthetic sample; production rollout has a separately tested backup/restore or reversible-data plan.

Validation rollout is a production change with latency, storage, index, replication, sharding, security, and organizational consequences. Test with realistic document sizes and index sets, not only counts. On replica sets, monitor oplog growth and lag; on sharded clusters, understand whether the migration predicate targets shards or broadcasts. If a backfill touches every document, it can compete with foreground traffic and backup windows. Use measured pacing and pause criteria.

Validators guarantee only that qualifying writes satisfy the declared predicate at write time. They do not guarantee semantic correctness, authorization, uniqueness unless separately indexed, cross-document invariants, backup recoverability, or future compatibility. A migration is complete when data, applications, operational runbooks, rollback, observability, and enforcement all agree—not when one script exits zero.

This chapter closes the flexible-schema foundation. Chapter 08 moves into the aggregation pipeline, where document streams pass through ordered stages and the same schema/type discipline determines whether transformations are correct and efficient.

docker rm -f atlasmart-mongo-ch07-l5docker volume rm atlasmart-mongo-ch07-l5-data

Authoritative references

• MongoDB 8.3 release notes — Current server-series and patch-release status; re-check before reproducing the lab.
• MongoDB versioning — Explains major/minor release series and why the latest stable patch should be used.
• mongosh release notes — Current mongosh release and version-sensitive shell behavior.
• PyMongo release notes — Current Python driver line used when application-side migration behavior is demonstrated.
• Query valid or invalid documents — Candidate schema as a preflight/postflight quality query.
• Handle invalid documents — Warn versus error enforcement during acceptance testing.
• Modify schema validation — Cutover with collMod after tests pass.
• Privilege actions — Defines bypassDocumentValidation and its authorization significance.
• PyMongo bulk write — Driver batching used by migration harnesses.
• Schema Versioning pattern — Mixed-version design tested by the harness.
• Production notes for schema validation — Official validation behavior and deployment context.