Diagnose Disk, Cache, Checkpoint, and Write Amplification Problems with Server Metrics

Learning objectives

1. A storage incident needs a timeline, not a favorite metric

Suppose AtlasMart checkout p99 climbs from its normal baseline while CPU, disk, cache, and write volume are all changing. The useful question is not “is WiredTiger slow?” but which mechanism became the bottleneck during the same interval. Capture application latency, operation mix, cache page traffic, dirty bytes, checkpoint duration, journal sync work, execution queueing, collection/index size, and host storage latency/throughput. Then alter one cause and repeat the exact workload.

2. Start the server and create lean versus heavily indexed order collections

docker rm -f atlasmart-ch24-l5 2>/dev/null || truedocker volume rm atlasmart-ch24-l5-db 2>/dev/null || truedocker run -d --name atlasmart-ch24-l5 \  -p 127.0.0.1:27190:27017 \  -v atlasmart-ch24-l5-db:/data/db \  mongodb/mongodb-community-server:8.3.8-ubuntu2204-slim \  --bind_ip_all --port 27017until mongosh "mongodb://127.0.0.1:27190/admin" --quiet --eval 'db.runCommand({ping:1}).ok' 2>/dev/null | grep -q 1; do sleep 1; donemongosh "mongodb://127.0.0.1:27190/admin" --quiet --eval '  printjson(db.adminCommand({buildInfo:1}));  printjson(db.adminCommand({getParameter:1,featureCompatibilityVersion:1}));  printjson(db.serverStatus().storageEngine);'

const d=db.getSiblingDB("atlasmart");for (const n of ["orders_lean_ch24_l5","orders_heavy_ch24_l5"]) d[n].drop();d.createCollection("orders_lean_ch24_l5"); d.createCollection("orders_heavy_ch24_l5");d.orders_lean_ch24_l5.createIndex({tenantId:1,status:1,createdAt:-1},{name:"idx_workload"});const h=d.orders_heavy_ch24_l5;h.createIndex({tenantId:1,status:1,createdAt:-1},{name:"idx_workload"});h.createIndex({customerId:1},{name:"idx_customer"});h.createIndex({paymentId:1},{name:"idx_payment"});h.createIndex({warehouseId:1,status:1},{name:"idx_warehouse_status"});h.createIndex({totalCents:1},{name:"idx_total"});h.createIndex({updatedAt:-1},{name:"idx_updated"});printjson({lean:d.orders_lean_ch24_l5.getIndexes().map(x=>x.name),heavy:h.getIndexes().map(x=>x.name)});

3. Use one snapshot function for every diagnostic phase

const admin=db.getSiblingDB("admin"); const d=db.getSiblingDB("atlasmart");function snap(label) {  const s=admin.serverStatus(), c=s.wiredTiger.cache, l=s.wiredTiger.log, t=s.wiredTiger.transaction;  function cs(name) { const x=d.getCollection(name).stats(); return {size:x.size,storageSize:x.storageSize,totalIndexSize:x.totalIndexSize,indexSizes:x.indexSizes}; }  printjson({label,    cache:{used:c["bytes currently in the cache"],dirty:c["tracked dirty bytes in the cache"],pagesRead:c["pages read into cache"],pagesWritten:c["pages written from cache"],evicted:c["unmodified pages evicted"],appEvictionMicros:c["application thread time evicting (usecs)"]},    journal:{bytes:l["log bytes written"],syncOps:l["log sync operations"],syncMicros:l["log sync time duration (usecs)"]},    checkpointMs:t["transaction checkpoint most recent time (msecs)"],    executionQueue:s.queues?.execution,    lean:cs("orders_lean_ch24_l5"), heavy:cs("orders_heavy_ch24_l5")  });}snap("before");

4. Same logical writes, different index amplification

from statistics import medianfrom time import perf_counterfrom pymongo import MongoClient, WriteConcernclient=MongoClient("mongodb://127.0.0.1:27190")db=client.atlasmartdef run(name):    c=db.get_collection(name,write_concern=WriteConcern(w=1,j=True))    insert_ms=[]; update_ms=[]    for batch in range(40):        docs=[]        for j in range(250):            i=batch*250+j            docs.append({"_id":i,"tenantId":f"tenant-{i%8}","status":"paid","createdAt":i,                         "updatedAt":i,"customerId":f"C-{i%3000}","paymentId":f"P-{i}",                         "warehouseId":f"W-{i%20}","totalCents":1000+i%20000,"payload":"x"*300})        t=perf_counter(); c.insert_many(docs); insert_ms.append((perf_counter()-t)*1000)    for r in range(30):        t=perf_counter(); c.update_many({"tenantId":f"tenant-{r%8}"},{"$inc":{"totalCents":1},"$set":{"updatedAt":100000+r}}); update_ms.append((perf_counter()-t)*1000)    def q(v,p):        s=sorted(v); return s[int(p*(len(s)-1))]    return {"insert_p50_ms":median(insert_ms),"insert_p95_ms":q(insert_ms,.95),"update_p50_ms":median(update_ms),"update_p95_ms":q(update_ms,.95)}for name in ["orders_lean_ch24_l5","orders_heavy_ch24_l5"]:    print(name,run(name))

The heavily indexed collection must maintain more B-tree structures per write, but the magnitude of latency or bytes written is environment-dependent. Use the post-snapshot deltas rather than inserting a canned “6× slower” claim.

const admin=db.getSiblingDB("admin"), d=db.getSiblingDB("atlasmart"), s=admin.serverStatus();const c=s.wiredTiger.cache,l=s.wiredTiger.log,t=s.wiredTiger.transaction;for (const name of ["orders_lean_ch24_l5","orders_heavy_ch24_l5"]) {  const st=d[name].stats(); printjson({name,count:st.count,storageSize:st.storageSize,totalIndexSize:st.totalIndexSize,indexSizes:st.indexSizes});}printjson({cache:{dirty:c["tracked dirty bytes in the cache"],pagesWritten:c["pages written from cache"],pagesRead:c["pages read into cache"]},journal:{bytes:l["log bytes written"],syncOps:l["log sync operations"],syncMicros:l["log sync time duration (usecs)"]},checkpointMs:t["transaction checkpoint most recent time (msecs)"],executionQueue:s.queues?.execution});

5. What each symptom suggests—and what it cannot prove alone

6. Add host/device evidence without changing the host

docker stats --no-stream atlasmart-ch24-l5# If sysstat is already installed on the host, collect device latency/queue evidence:command -v iostat >/dev/null && iostat -xz 1 5 || echo "iostat not installed; use your platform/cloud disk telemetry instead"# Do not install tools or alter I/O schedulers merely to complete this lesson.

MongoDB’s serverStatus can reveal time spent synchronizing the journal and checkpoint duration, but it cannot by itself prove the physical device is the root cause. Cloud disk throttling, filesystem behavior, RAID, virtualization, noisy neighbors, and host memory pressure require platform evidence.

7. The change-one-factor loop

1. Freeze the workload definition: data shape, indexes, read/write concern, concurrency, client pool, cache warmup, and query mix.
2. Capture a baseline interval with application p50/p95/p99 and MongoDB/host metric deltas.
3. State one causal hypothesis, such as “two unused indexes are increasing write amplification.”
4. Apply one reversible change in the disposable/staging environment.
5. Rerun the identical workload and compare both latency and mechanism metrics.
6. Keep or roll back the change based on the invariant and evidence—not because one metric moved in the desired direction.

8. Production judgment and bridge to backup

WiredTiger tuning should be the end of a diagnosis, not the beginning. Prefer correct query/index design, capacity, healthy storage, bounded concurrency, and appropriate durability settings. Treat internal statistics as version-sensitive and build monitoring around documented MongoDB interfaces. Crucially, storage-engine durability is still not backup: a healthy checkpoint and journal can faithfully persist an application mistake. Chapter 25 therefore moves from local durability mechanics to independent logical/physical backups, consistent snapshots, point-in-time recovery, and restore drills.

Authoritative references

WiredTiger metrics and internal field names are implementation- and version-sensitive. The lesson uses documented MongoDB interfaces for evidence and requires re-checking the current server manual before relying on exact metric names or defaults in a later release.

• WiredTiger Storage Engine
• Storage FAQ
• serverStatus
• Self-Managed Diagnostics FAQ
• Journaling
• Configure Journaling
• Write Concern
• Write Operation Performance
• Configuration File Options
• Server Parameters
• mongod Options
• db.collection.stats()
• $collStats
• dbStats
• db.createCollection() Storage Engine Options
• Create Indexes and Storage Engine Options
• Performance Tuning
• Production Notes
• Log Messages
• MongoDB 8.3 Release Notes
• mongosh Release Notes
• PyMongo Release Notes