Retryable Reads/Writes, Transactions, Idempotency, Command Monitoring, and Error Classification

Learning objectives

1. Retry layers solve different failures

Official drivers enable retryable reads/writes for supported operations by default. A generic while True: retry() loop is different: it can amplify overload, repeat non-retryable failures, and exceed user deadlines.

2. Idempotent AtlasMart order acceptance

docker ps --filter name=atlasmart-rs27-lab --format "{{.Names}} {{.Status}}"# If absent, run Lesson 1 setup first.

from pymongo import MongoClient, ASCENDINGfrom pymongo.errors import DuplicateKeyErrorfrom datetime import datetime, timezoneclient=MongoClient("mongodb://localhost:27209,localhost:27210,localhost:27211/?replicaSet=atlasmart-rs27&retryWrites=true&w=majority")db=client.atlasmartorders=db.orders_ch27_l2orders.drop()orders.create_index([("tenantId",ASCENDING),("idempotencyKey",ASCENDING)],unique=True,name="uq_tenant_idempotency")def place_order(tenant,key,total):    existing=orders.find_one({"tenantId":tenant,"idempotencyKey":key})    if existing: return existing["orderId"],True    doc={"tenantId":tenant,"idempotencyKey":key,"orderId":f"ord-{key}",         "totalCents":total,"createdAt":datetime.now(timezone.utc)}    try:        orders.insert_one(doc)        return doc["orderId"],False    except DuplicateKeyError:        # Concurrent replay won the race: return the durable first result.        return orders.find_one({"tenantId":tenant,"idempotencyKey":key})["orderId"],Trueprint(place_order("tenant-a","req-1001",4200))print(place_order("tenant-a","req-1001",4200))print(orders.count_documents({"tenantId":"tenant-a","idempotencyKey":"req-1001"}))client.close()

Verification is the invariant: both calls return the same order identifier and exactly one order exists for the tenant/idempotency key.

3. Transactions: database atomicity is not side-effect atomicity

ClientSession.with_transaction() may retry the commit or the entire callback. Therefore a callback that sends email, charges a card, or publishes a non-idempotent message can repeat those effects even when the database transaction ultimately commits once. Keep external effects after commit, or use an outbox/idempotent external API.

from pymongo import MongoClientfrom pymongo.read_concern import ReadConcernfrom pymongo.write_concern import WriteConcernfrom pymongo.read_preferences import ReadPreferenceclient=MongoClient("mongodb://localhost:27209,localhost:27210,localhost:27211/?replicaSet=atlasmart-rs27")db=client.atlasmartdef transfer(session):    db.wallets_ch27_l2.update_one({"_id":"buyer","cents":{"$gte":500}}, {"$inc":{"cents":-500}}, session=session)    db.wallets_ch27_l2.update_one({"_id":"seller"},{"$inc":{"cents":500}},session=session)    db.outbox_ch27_l2.update_one({"eventId":"pay-1001"},{"$setOnInsert":{"kind":"payment.captured","sent":False}},upsert=True,session=session)with client.start_session() as s:    s.with_transaction(transfer,read_concern=ReadConcern("snapshot"),                       write_concern=WriteConcern("majority"),read_preference=ReadPreference.PRIMARY)client.close()

4. Classify first; then decide retry, reconcile, or fail

from pymongo.errors import (DuplicateKeyError, NetworkTimeout, ServerSelectionTimeoutError,                            ExecutionTimeout, OperationFailure, PyMongoError)def classify(exc):    if isinstance(exc, DuplicateKeyError): return "conflict-not-retry"    if isinstance(exc, (NetworkTimeout, ServerSelectionTimeoutError, ExecutionTimeout)): return "deadline-or-transient"    if isinstance(exc, OperationFailure):        if exc.has_error_label("NoWritesPerformed"): return "safe-no-write-performed"        if exc.has_error_label("TransientTransactionError"): return "retry-whole-transaction-within-budget"        if exc.has_error_label("UnknownTransactionCommitResult"): return "retry-commit-or-reconcile"        return f"server-error-{exc.code}"    if isinstance(exc, PyMongoError): return "driver-error-requires-classification"    return "application-error"

Error labels are evidence, not permission for infinite retry. Every retry policy still needs an attempt/deadline budget, jitter/backoff where appropriate, and a reconciliation path for ambiguous outcomes.

5. Observe attempts without leaking commands

from pymongo import monitoringclass Outcomes(monitoring.CommandListener):    def started(self,e): print("start",e.command_name,e.request_id)    def succeeded(self,e): print("ok",e.command_name,e.request_id,round(e.duration_micros/1000,2))    def failed(self,e): print("fail",e.command_name,e.request_id,type(e.failure).__name__)

Production logs should correlate request/trace IDs with command name, duration, server address, retry/error classification, and result state. Do not dump authentication commands, full customer documents, encryption material, or payment data into logs.

6. Production judgment

Prefer driver retryability for supported transient database failures, database uniqueness/idempotency for business commands, and explicit reconciliation for ambiguous outcomes. Transactions protect MongoDB state, not arbitrary external effects. Lesson 3 turns the same discipline toward data-flow volume: pagination, batches, schema evolution, change streams, and backpressure.

Authoritative references

Driver defaults and deployment behavior evolve. Re-check the exact server patch, PyMongo release, topology, and managed-service tier before freezing production assumptions.

• PyMongo driver documentation
• Connect to MongoDB with PyMongo
• PyMongo connection pools
• PyMongo client-side operation timeout
• PyMongo monitoring
• PyMongo CRUD configuration / retries
• PyMongo transactions
• PyMongo bulk writes
• PyMongo release notes
• MongoDB connection strings
• Connection string options
• Retryable writes
• Retryable reads
• Transactions
• Change streams
• cursor.skip() and range pagination
• Explain results
• Read concern
• Write concern
• Read preference
• Replica sets
• Sharding
• Choose a shard key
• Security checklist
• Backup methods
• serverStatus
• MongoDB 8.3 release notes
• mongosh changelog