Build a Memory Budget Including Data, Indexes, Replication Buffers, Fork Headroom, and Failover

Learning outcomes

AtlasMart must choose a memory limit for a production-like Redis node before enabling replicas, Search indexes, and persistence. “Dataset is 20 GiB, so allocate 20 GiB” omits exactly the transients that appear during backup, synchronization, rewrite, failover, and workload spikes.

1. Start from the process envelope, not from maxmemory

A practical budget begins with the smallest hard ceiling: container memory limit, cgroup, VM RAM available to Redis, or managed-service database limit. From that ceiling reserve OS/platform requirements and transient Redis process headroom. Only then choose a maxmemory that leaves room for what eviction does not control.

2. Build a measured budget fixture

docker volume create atlasmart-redis-ch18-budget_datadocker run --rm -v atlasmart-redis-ch18-budget_data:/data redis:8.10.1 sh -lc 'printf "%s\n" "user default off" "user academy-admin on >AtlasMart-Admin-Lab-Only-2026 ~* &* +@all" > /data/users.acl'docker run -d --name atlasmart-redis-ch18-budget --memory 192m -p 127.0.0.1:6395:6379 -v atlasmart-redis-ch18-budget_data:/data redis:8.10.1 redis-server --dir /data --aclfile /data/users.acl --appendonly yes --appendfsync everysec --save '' --maxmemory 48mb --maxmemory-policy noevictiondocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch18-budget redis-cli --user academy-admin PINGdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch18-budget redis-cli --user academy-admin ACL WHOAMI

import os, redisHOST = "127.0.0.1"PORT = 6395r = redis.Redis(host=HOST, port=PORT, username="academy-admin", password="AtlasMart-Admin-Lab-Only-2026", decode_responses=False)assert r.ping()payload = b"catalog:" + b"x" * 4096pipe = r.pipeline(transaction=False)for i in range(2500): pipe.set(f"atlasmart:ch18:l5:catalog:{i}", payload)pipe.execute()for i in range(200): r.hset(f"atlasmart:ch18:l5:inventory:{i}", mapping={b"available": b"100", b"reserved": b"0"})print("keys", r.dbsize())

docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch18-budget redis-cli --user academy-admin INFO memorydocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch18-budget redis-cli --user academy-admin MEMORY STATSdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch18-budget redis-cli --user academy-admin INFO persistencedocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch18-budget redis-cli --user academy-admin INFO replicationdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch18-budget redis-cli --user academy-admin CONFIG GET maxmemory maxmemory-policy appendonly appendfsyncdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch18-budget redis-cli --user academy-admin DBSIZE

3. Index memory belongs in the budget even when source documents look small

Redis Search, vector indexes, JSON structures, time-series metadata, probabilistic structures, and compact-hash templates can add memory beyond the source payload. Do not estimate those structures from source-file size. Measure the feature actually used, including index build/rebuild peaks.

docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch18-budget redis-cli --user academy-admin INFO memory# If AtlasMart uses Redis Search, capture FT.INFO for each index and compare INFO MEMORY before/after representative indexing.# If vectors/time series/probabilistic structures are used, load production-like cardinality and measure their actual process footprint.

4. Measure fork/COW rather than reserving a folklore percentage

Chapter 17 showed that BGSAVE and AOF rewrite fork a child on the Linux-based container. Copy-on-write grows with pages modified while the child is alive. The correct headroom is workload- and duration-dependent.

docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch18-budget redis-cli --user academy-admin BGSAVEdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch18-budget redis-cli --user academy-admin INFO persistence# Wait until rdb_bgsave_in_progress returns 0, then record rdb_last_cow_size.# Repeat under representative write concurrency on a disposable environment; do not invent a percentage.

5. Replication/failover can change who must carry the memory budget

Redis replication is asynchronous and replicas have their own buffers and full-sync transients. Current Redis documentation also notes that replicas ignore maxmemory by default for normal replication behavior; after promotion, the new primary must enforce the configured policy while also handling new client/write pressure. That is why a primary-only steady-state budget is insufficient for Sentinel or Cluster failover planning.

6. A reusable worksheet

Hard process/container/node envelope = ______ bytes- platform/OS/agent reserve             = ______- measured RSS/allocator slack peak      = ______- mem_not_counted_for_evict peak         = ______- client/output buffer peak              = ______- fork/COW peak (RDB/AOF/full sync)       = ______- failover/reconnect/sync contingency     = ______= maximum safe Redis operating envelope   = ______Choose maxmemory below that envelope, then verify dataset + index growth fits the chosen eviction semantics.

7. Export facts; calculate decisions outside Redis

The server exposes measurements, but the capacity decision belongs in an auditable worksheet, infrastructure-as-code variable set, or monitoring calculation. Keep the raw evidence with the chosen limit so future operators know why it exists.

import os, redisHOST = "127.0.0.1"PORT = 6395r = redis.Redis(host=HOST, port=PORT, username="academy-admin", password="AtlasMart-Admin-Lab-Only-2026", decode_responses=False)assert r.ping()m = r.info("memory")persistence = r.info("persistence")replication = r.info("replication")fields = {    "used_memory": m.get("used_memory"),    "used_memory_dataset": m.get("used_memory_dataset"),    "used_memory_rss": m.get("used_memory_rss"),    "used_memory_overhead": m.get("used_memory_overhead"),    "mem_not_counted_for_evict": m.get("mem_not_counted_for_evict"),    "mem_clients_normal": m.get("mem_clients_normal"),    "mem_replication_backlog": m.get("mem_replication_backlog"),    "mem_aof_buffer": m.get("mem_aof_buffer"),    "allocator_frag_bytes": m.get("allocator_frag_bytes"),    "rdb_last_cow_size": persistence.get("rdb_last_cow_size"),    "aof_last_cow_size": persistence.get("aof_last_cow_size"),    "role": replication.get("role"),}for k,v in fields.items(): print(k, v)

8. Failure modes the budget must survive

9. Deliberately wrong budget: dataset bytes + 10%

A fixed percentage can accidentally work in one environment and fail in another. If the write rate during a fork doubles, a replica reconnects and needs a full synchronization, or a Search index rebuilds, the previously “safe” margin may disappear. The repair is to budget each transient from observed production-representative evidence and test the same node role during failover.

10. Acceptance criteria for an AtlasMart capacity decision

• The hard process/container/node ceiling and every subtraction are documented.
• maxmemory is lower than the hard ceiling by measured non-evictable/transient headroom.
• Dataset and all index/data-type structures fit the intended eviction semantics at expected peak cardinality.
• RDB/AOF/full-sync/failover COW and buffers were exercised on a production-representative disposable environment.
• p50/p95/p99 latency, hit/miss rate, evictions, rejected writes, RSS, and allocator metrics remain within service objectives during pressure.
• Every replica/failover candidate can assume the primary role without relying on the old primary's memory envelope.
• Capacity alarms trigger before the kernel/container OOM killer becomes the control plane.

Production judgment and next bridge

Chapter 18 ends with a memory budget that survives more than steady-state dataset growth. Chapter 19 now makes replication itself observable: initial sync, partial resynchronization, backlog sizing, replica reads, lag, and failure recovery will turn several “future buffer” lines in this worksheet into measured topology-specific values.

Summary and next step

Build a Memory Budget Including Data, Indexes, Replication Buffers, Fork Headroom, and Failover is now connected to observable Redis behavior, bounded failure cases, and production tradeoffs. Keep the evidence and cleanup state from this lesson; next, continue with Primary/Replica Replication, Asynchronous Semantics, Initial Sync, and Command Propagation.

Authoritative references

• Redis key eviction
• Redis memory optimization
• INFO
• MEMORY STATS
• MEMORY USAGE
• MEMORY DOCTOR
• MEMORY PURGE
• MEMORY MALLOC-STATS
• OBJECT FREQ
• OBJECT IDLETIME
• UNLINK
• Redis CLI keyspace analysis
• Redis latency monitoring
• Redis persistence
• Redis replication
• Redis Sentinel
• Redis Cluster specification
• Redis Search and query
• Redis 8.10 release notes
• Redis 8.10 whats new
• Redis ACLs
• Redis security
• Redis licenses
• Docker Official Redis image