Compression for Collections and Indexes: Space, CPU, Cache Density, and Workload Tradeoffs

Learning objectives

1. Compression changes both capacity and the amount of work around I/O

WiredTiger uses Snappy block compression by default for most ordinary collections and prefix compression for indexes. zstd and zlib can often compress more, while none avoids block-compression CPU. Time-series collections have a different default (zstd), but this lesson uses ordinary AtlasMart event documents. The critical distinction is that compressed files can improve disk capacity and filesystem-cache density while collection data in the WiredTiger internal cache is represented uncompressed for manipulation.

2. Create one collection per compressor and verify the creation strings

docker rm -f atlasmart-ch24-l3 2>/dev/null || truedocker volume rm atlasmart-ch24-l3-db 2>/dev/null || truedocker run -d --name atlasmart-ch24-l3 \  -p 127.0.0.1:27188:27017 \  -v atlasmart-ch24-l3-db:/data/db \  mongodb/mongodb-community-server:8.3.8-ubuntu2204-slim \  --bind_ip_all --port 27017until mongosh "mongodb://127.0.0.1:27188/admin" --quiet --eval 'db.runCommand({ping:1}).ok' 2>/dev/null | grep -q 1; do sleep 1; donemongosh "mongodb://127.0.0.1:27188/admin" --quiet --eval '  printjson(db.adminCommand({buildInfo:1}));  printjson(db.adminCommand({getParameter:1,featureCompatibilityVersion:1}));  printjson(db.serverStatus().storageEngine);'

const d=db.getSiblingDB("atlasmart");for (const n of ["cmp_none","cmp_snappy","cmp_zstd","cmp_zlib"]) d[n].drop();d.createCollection("cmp_none",  {storageEngine:{wiredTiger:{configString:"block_compressor=none"}}});d.createCollection("cmp_snappy",{storageEngine:{wiredTiger:{configString:"block_compressor=snappy"}}});d.createCollection("cmp_zstd",  {storageEngine:{wiredTiger:{configString:"block_compressor=zstd"}}});d.createCollection("cmp_zlib",  {storageEngine:{wiredTiger:{configString:"block_compressor=zlib"}}});for (const n of ["cmp_none","cmp_snappy","cmp_zstd","cmp_zlib"]) {  const s=d[n].stats();  print(n, (s.wiredTiger?.creationString||"").split(",").filter(x=>x.startsWith("block_compressor="))[0]);}

3. Load identical compressible AtlasMart documents and record storage plus write cost

python -m venv /tmp/atlasmart-ch24-compress-venv. /tmp/atlasmart-ch24-compress-venv/bin/activatepython -m pip install --disable-pip-version-check "pymongo==4.17.0"python - <<'PY'from statistics import medianfrom time import perf_counterfrom pymongo import MongoClientclient=MongoClient("mongodb://127.0.0.1:27188")db=client.atlasmartnames=["cmp_none","cmp_snappy","cmp_zstd","cmp_zlib"]for name in names:    c=db[name]; samples=[]    for batch in range(50):        docs=[]        for j in range(500):            i=batch*500+j            docs.append({                "_id":i,                "tenantId":f"tenant-{i%8}",                "kind":"inventory-snapshot",                "sku":f"SKU-{i%2000:05d}",                "payload":"sensor="+("A"*700)+";warehouse="+("W"*(i%5+20))+";status=healthy;",                "counter":i%100            })        t=perf_counter(); c.insert_many(docs,ordered=True); samples.append((perf_counter()-t)*1000)    q=sorted(samples)    st=db.command("collStats",name)    print(name,{        "logical_size":st["size"],        "storage_size":st["storageSize"],        "p50_batch_ms":median(q),        "p95_batch_ms":q[int(.95*(len(q)-1))]    })PY

Do not rank compressors from one run. The generated payload is intentionally compressible so the storage effect is visible; real payloads may already be compressed/encrypted or have very different entropy. Re-run with representative BSON shapes and report CPU and storage-device context.

4. Add the same index and inspect prefix-compression evidence separately

const d=db.getSiblingDB("atlasmart");for (const n of ["cmp_none","cmp_snappy","cmp_zstd","cmp_zlib"]) {  d[n].createIndex({tenantId:1,sku:1,counter:1},{name:"idx_tenant_sku_counter"});  const s=d[n].stats({indexDetails:true,indexDetailsName:"idx_tenant_sku_counter"});  const creation=s.indexDetails?.idx_tenant_sku_counter?.creationString || "";  printjson({name:n,indexBytes:s.indexSizes?.idx_tenant_sku_counter,    prefixSetting:creation.split(",").filter(x=>x.startsWith("prefix_compression="))[0]});}

Index prefix compression is distinct from the collection block compressor. Also note that index bytes in the WiredTiger internal cache use an internal representation that can still benefit from prefix compression, while collection values are uncompressed there.

5. Read-path experiment: storage savings do not automatically mean lower query latency

const d=db.getSiblingDB("atlasmart");for (const n of ["cmp_none","cmp_snappy","cmp_zstd","cmp_zlib"]) {  const t=Date.now();  const docs=d[n].find({tenantId:"tenant-3",counter:{$gte:40,$lt:60}}).hint("idx_tenant_sku_counter").limit(1000).toArray();  const ms=Date.now()-t;  printjson({name:n,returned:docs.length,wallClockMs:ms,stats:d[n].stats().storageSize});}

6. Production judgment

Start with MongoDB defaults unless representative evidence justifies a change. Compare logical bytes, storage bytes, index bytes, CPU, read/write distributions, cache/page I/O, and host disk metrics. If the application encrypts or pre-compresses payloads, database compression may produce little space savings. If the workload is I/O-bound, stronger compression can improve total latency despite added CPU; if CPU-bound, the opposite can occur. Existing collections keep the compressor chosen at creation, so changing the server default affects new collections rather than rewriting old data. The next lesson deliberately constrains cache to make eviction and admission signals observable.

Authoritative references

WiredTiger metrics and internal field names are implementation- and version-sensitive. The lesson uses documented MongoDB interfaces for evidence and requires re-checking the current server manual before relying on exact metric names or defaults in a later release.

• WiredTiger Storage Engine
• Storage FAQ
• serverStatus
• Self-Managed Diagnostics FAQ
• Journaling
• Configure Journaling
• Write Concern
• Write Operation Performance
• Configuration File Options
• Server Parameters
• mongod Options
• db.collection.stats()
• $collStats
• dbStats
• db.createCollection() Storage Engine Options
• Create Indexes and Storage Engine Options
• Performance Tuning
• Production Notes
• Log Messages
• MongoDB 8.3 Release Notes
• mongosh Release Notes
• PyMongo Release Notes