Full Document/Pre-Image Options, Update Descriptions, Deletes, and Schema Evolution

Learning objectives

1. Four payload ideas that are easy to conflate

For an update, updateDescription describes the changed/removed fields. fullDocument:"updateLookup" performs a later lookup of the current majority-committed document; intervening writes can make that lookup newer than the event's delta. MongoDB 6.0+ pre/post images instead preserve document images associated with the change when the feature was enabled and the image remains retained. A delete has no current post-image because the document no longer exists.

2. Enable images and capture update + delete evidence

const app=db.getSiblingDB("atlasmart");app.products_ch18_l3.drop();app.createCollection("products_ch18_l3", {changeStreamPreAndPostImages:{enabled:true}});app.products_ch18_l3.insertOne({_id:"p-3",schemaVersion:1,name:"Router",price:99,tags:["network"]});printjson(app.getCollectionInfos({name:"products_ch18_l3"})[0].options);printjson(db.getSiblingDB("config").system.preimages.stats());

from pprint import pprintfrom pymongo import MongoClientclient = MongoClient("mongodb://127.0.0.1:27157/?replicaSet=atlasmart-rs18-l3")c = client.atlasmart.products_ch18_l3with c.watch(    full_document="whenAvailable",    full_document_before_change="whenAvailable",    max_await_time_ms=1000,) as stream:    c.update_one({"_id":"p-3"},{"$set":{"schemaVersion":2,"price":89,"attributes":{"wifi":"6E"}}})    update_event = stream.next()    pprint({        "op": update_event["operationType"],        "delta": update_event.get("updateDescription"),        "before": update_event.get("fullDocumentBeforeChange"),        "after": update_event.get("fullDocument"),    })    c.delete_one({"_id":"p-3"})    delete_event = stream.next()    pprint({        "op": delete_event["operationType"],        "documentKey": delete_event["documentKey"],        "before": delete_event.get("fullDocumentBeforeChange"),        "after": delete_event.get("fullDocument"),    })client.close()

Expected invariant: the update has a delta plus available pre/post image; the delete retains documentKey and can have a pre-image, but there is no post-delete document. Exact resume tokens and timestamps are runtime values.

3. required versus whenAvailable is a correctness decision

whenAvailable lets the stream continue with a null/missing image if the image is unavailable. required asks the server to fail when the requested image cannot be returned. Use required only when missing an image is itself a correctness failure and the retention/storage design supports that contract. Pre-images are removed asynchronously and also disappear when their corresponding change event falls out of the oplog, regardless of a longer configured image retention period.

4. Deliberately wrong consumer: “all changes are update events with fullDocument”

MongoDB may represent a replacement as replace rather than update, and delete events have no current document. A consumer that assumes operationType=="update" and blindly dereferences fullDocument will silently miss replacements or crash on deletes. Normalize operations into an application-owned event envelope and keep schema versioning explicit.

def normalize(change):    op = change["operationType"]    if op in {"insert", "replace", "update"}:        doc = change.get("fullDocument")        return {            "kind": "product-upsert",            "sourceOp": op,            "id": change["documentKey"]["_id"],            "schemaVersion": (doc or {}).get("schemaVersion"),            "document": doc,            "delta": change.get("updateDescription"),        }    if op == "delete":        return {            "kind": "product-delete",            "sourceOp": op,            "id": change["documentKey"]["_id"],            "before": change.get("fullDocumentBeforeChange"),        }    return {"kind":"control-or-unknown", "sourceOp":op}

5. Production judgment

Use deltas when downstream state can apply patches safely; use post-images when consumers need a full snapshot; use pre-images for audit/compensation only when the additional storage, retention, privacy, and event-size costs are justified. Treat image retention as part of the consumer recovery objective, and monitor config.system.preimages size. Do not persist sensitive pre-images indefinitely just because CDC makes it convenient.

Bridge. Lesson 4 turns the resume token into a durable recovery protocol and shows the special invalidation/history-loss boundaries that ordinary reconnect logic cannot hide.

Cleanup/reset

Everything in this lesson is disposable. Remove only the chapter-specific container and volume:

docker rm -f atlasmart-ch18-l3 2>/dev/null || truedocker volume rm atlasmart-ch18-l3-data 2>/dev/null || true

Authoritative references

• MongoDB Change Streams — deployment requirements, majority-committed notification, scopes, sharded behavior, resume tokens, and pre/post images.
• Change Stream Events — event fields, operation types, resume token, update/replace behavior, and expanded events.
• db.collection.watch() — pipeline stages, options, resumability, and mongosh versus driver behavior.
• db.watch() — database-scoped streams.
• Mongo.watch() — deployment-scoped streams.
• update Event — updateDescription, documentKey, full document and pre-image behavior.
• delete Event — delete event and pre-image behavior.
• invalidate Event — stream invalidation and startAfter boundary.
• Change Streams Production Recommendations — sharded total ordering and latency considerations.
• Privilege Actions — changeStream/find authorization requirements.
• Replica Set Oplog — retained history and oplog window.
• PyMongo Driver — official Python driver baseline and change-stream cursor APIs.
• PyMongo 4.17 Release Notes — current driver line used by the course.
• MongoDB 8.3 Release Notes — current stable minor and patch status.
• MongoDB 8.3 Compatibility Changes — current expanded-event field behavior inherited from 8.2.x.
• mongosh Release Notes — mongosh 2.10.0 baseline.