Chapter 17 · Persistence: RDB Snapshots, AOF, fsync, Rewrites, and Crash Recovery

AOF Rewrite/Compaction, Base/Incremental Files, Fork Costs, and Disk Headroom

Inspect Redis 8.10 multi-part AOF rewrite mechanics, base/incremental files, copy-on-write memory, rewrite state, and the disk headroom required to compact safely.

Advanced180–250 minutesMP-AOF rewrite, COW, disk headroom, manifestsRedis Open Source 8.10.1Docker local labFree/local-firstLast reviewed: September 6, 2026

Learning outcomes

AtlasMart has a healthy AOF but the incremental log grows continuously. Redis compacts AOF with a background rewrite. The dangerous operational mistake is to plan only for the final compacted size and ignore temporary base/increment files, fork cost, copy-on-write memory, and failure/retry behavior.

Explain Redis 7+ base/incremental/manifest rewrite mechanics.

Measure pre/post AOF size instead of inventing a compaction ratio.

Observe aof_rewrite_in_progress, aof_last_cow_size, and rewrite status.

Calculate disk headroom from measured artifacts plus safety margin rather than a universal percentage.

Avoid copying an MP-AOF directory mid-rewrite without a consistency plan.

Exact Chapter 17 lab boundary

The normal course baseline remains Redis Open Source 8.10.1 from pinned image redis:8.10.1. Because this chapter intentionally restarts/crashes persistence processes, destructive exercises use dedicated disposable Chapter 17 containers and named volumes rather than the shared atlasmart-redis-ch01 container. All ports bind only to 127.0.0.1; the temporary lab password is public classroom data, not a production secret; TLS is omitted only on loopback. Mandatory topology is standalone, logical DB 0, no explicit maxmemory/eviction policy. This lesson uses a dedicated AOF node on host port 6383.

1. Why rewrite exists

If a counter is incremented ten thousand times, its final state may require only one compact representation while the AOF history contains many commands. BGREWRITEAOF constructs a new base representing current state while the parent continues accepting writes into a new incremental file.

Shell · create one isolated persistence node

docker volume create atlasmart-redis-ch17-rewrite_datadocker run --rm -v atlasmart-redis-ch17-rewrite_data:/data redis:8.10.1 sh -lc 'printf "%s\n" "user default off" "user academy-admin on >AtlasMart-Admin-Lab-Only-2026 ~* &* +@all" > /data/users.acl'docker run -d --name atlasmart-redis-ch17-rewrite -p 127.0.0.1:6383:6379 -v atlasmart-redis-ch17-rewrite_data:/data redis:8.10.1 redis-server --dir /data --aclfile /data/users.acl --appendonly yes --appendfsync everysec --save  --auto-aof-rewrite-percentage 0docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin PINGdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin ACL WHOAMI# users.acl lives in the named volume, so academy-admin remains available after the deliberate restart/crash.

2. Redis 7+ MP-AOF rewrite sequence

Stage	Parent	Child / artifact
Start	opens a new incremental AOF	forked child begins new base
During rewrite	continues writes to increment	child serializes current dataset
Commit	builds/persists temporary manifest	new base + increment become active set
Finish	atomically swaps manifest and cleans old files	old unused components can be removed

This sequence protects correctness if rewrite fails, but it temporarily requires multiple persistence components and a forked child.

3. Create bounded redundant history and measure before

redis-cli · deterministic rewrite fixture

docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin SET atlasmart:ch17:l3:counter 0docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INCRBY atlasmart:ch17:l3:counter 100docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INFO persistence

Shell · measure files and filesystem headroom

docker exec atlasmart-redis-ch17-rewrite sh -lc 'find /data -maxdepth 2 -type f -printf "%p %s bytes\n" | sort; df -h /data'# Record aof_current_size, aof_base_size, total file bytes, and free filesystem bytes before rewrite.

4. Trigger rewrite and observe fork/COW state

redis-cli · start and poll

docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin BGREWRITEAOFdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INFO persistencedocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin SET atlasmart:ch17:l3:during-rewrite mutationdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INFO persistence

During an active child, inspect current_cow_size, current_cow_peak, and progress fields. After completion, record aof_last_cow_size, aof_last_bgrewrite_status, rewrite duration, and current/base size. Small classroom datasets may finish too quickly to catch the in-progress state; that is a measurement limitation, not permission to fabricate values.

5. Disk headroom is a transient-peak problem

A safe rewrite budget must account for the active base/increment set, the newly generated base, continued writes during rewrite, temporary/manifest files, filesystem metadata/reserve, and growth if rewrite is slow or retried. There is no universal “double the dataset” rule that is correct for every write rate and encoding.

Input to capacity model	Measure from
Current AOF/base/increment bytes	INFO persistence + filesystem
Write bytes/sec during rewrite	application metrics / AOF growth
Observed rewrite duration	INFO persistence
COW peak	current_cow_peak / aof_last_cow_size
Disk free and filesystem reserve	df/storage metrics
Failure/retry overlap	rewrite logs + policy

6. Wrong approach: copy the live MP-AOF directory during rewrite

The manifest and component files change as rewrite commits. A naive recursive copy can combine a manifest from one generation with files from another. Redis documentation recommends preventing rewrites while copying the AOF set, or in Redis 8.10 using the node-side BACKUP workflow designed to produce a self-contained restorable set.

Consistency beats convenience

A directory that copied without errors is not necessarily a valid backup. The acceptance test is restore in an isolated node plus data/application checks.

7. Measure after rewrite

redis-cli/shell · completion evidence

docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin INFO persistencedocker exec atlasmart-redis-ch17-rewrite sh -lc 'find /data -maxdepth 2 -type f -printf "%p %s bytes\n" | sort; df -h /data'docker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin GET atlasmart:ch17:l3:counterdocker exec -e REDISCLI_AUTH=AtlasMart-Admin-Lab-Only-2026 atlasmart-redis-ch17-rewrite redis-cli --user academy-admin GET atlasmart:ch17:l3:during-rewrite

Compare measured before/after sizes; do not hard-code an expected compaction percentage. Workload history, data types, compression, base format, and write concurrency all matter.

8. Cleanup

Shell · bounded cleanup

docker rm -f atlasmart-redis-ch17-rewritedocker volume rm atlasmart-redis-ch17-rewrite_data# Removes only the named Chapter 17 container/volume. Never use docker system prune for this lab.

9. Production judgment

Schedule rewrite based on observed growth, disk bandwidth, COW headroom, latency, and recovery needs. Alert on failed/scheduled rewrites, growing base/current-size ratios, disk exhaustion risk, COW spikes, and slow child completion. In container platforms, persistent-volume capacity and host memory limits matter even when Redis maxmemory is unset.

Check your understanding

Why can rewrite need more disk than the final AOF?
What proves rewrite succeeded?
Why can writes during rewrite increase memory?
Is a fixed 2x disk rule universally safe?
What is safer than copying MP-AOF files mid-rewrite?

Review the answers

The old active set remains while a new base is created and new writes continue into an increment.

aof_last_bgrewrite_status=ok plus expected current data and a coherent active manifest/file set.

Forked child shares pages; parent mutations trigger copy-on-write copies.

No. Model measured active bytes, growth rate, rewrite duration, retries, and filesystem reserve.

Freeze rewrite activity per documented procedure or use the Redis 8.10 BACKUP workflow, then restore-test the result.

10. Summary and bridge

Rewrite is safe background compaction, not free compaction: it consumes temporary disk, fork time, and COW memory. Lesson 4 combines RDB and AOF and proves which persistence source Redis uses at normal startup.

Authoritative references

Keep knowledge open

Help the academy stay free and grow.

If these tutorials save you time, a small donation supports new lessons, technical review, diagrams, examples, and long-term maintenance.

ETHEthereum / ERC-20 only

0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0

Send only Ethereum or ERC-20 compatible assets to this address.