Storage Drivers, overlay2, Copy-on-Write Behavior, Disk Usage, Pruning, and Storage Troubleshooting: Concepts, Architecture, and Mental Model

1. The practical problem: “Docker uses too much disk” is not a diagnosis

Disk-pressure incidents often begin with a host-volume alert or a Docker Desktop disk image growing unexpectedly. The tempting response is deletion. The correct response is classification. Docker can consume storage through OCI image blobs, unpacked snapshots or classic graph-driver layers, each container’s writable layer, named and anonymous volumes, BuildKit cache, local log storage, metadata, and—on Desktop—the Linux VM disk image that contains these resources.

Those categories have different ownership and recovery rules. An unused image might be rebuildable. A volume might contain the only copy of application data. Build cache may be disposable but expensive to regenerate. A container log can grow without changing docker ps --size. Therefore this chapter treats deletion as the last step after the object and its dependencies are proven.

2. Causal storage model

flowchart TD
  A[Registry/image content blobs] --> F[Docker storage domains]
  B[Snapshot or classic image layers] --> F
  C[Container writable layer] --> F
  D[Volumes] --> F
  E[BuildKit cache and logs] --> F
  F --> G[Backing filesystem or Desktop disk image]
  G --> H[Bytes + inodes + I/O pressure]
  H --> I[Read-only inventory]
  I --> J[Exact retention decision]
  J --> K[Targeted deletion or no action]
            

The top nodes are intentionally separate. Registry/image content gives Docker immutable bytes. A snapshotter or storage driver materializes filesystem state. Containers add copy-on-write changes. Volumes live outside the writable layer. Builders retain intermediate cache. Logging drivers can retain stdout/stderr independently. All of that eventually competes for capacity, inodes, and I/O on one or more filesystems.

The causal order for operations is therefore: capacity signal → identify storage domain → inventory exact objects → establish references/retention → predict reclaimed state → remove only owned targets → remeasure. Skipping the inventory step turns cleanup into guesswork.

3. Engine 29 has two storage architectures you may encounter

4. Copy-on-write: why starting many containers is cheaper than copying images

An image filesystem is read-only from the container’s perspective. Containers share those image layers and receive a writable top layer or writable snapshot. Reads can come from lower image content; the first modification of a file can require materializing changed data into the writable layer. This is the essence of copy-on-write (CoW).

CoW explains two common surprises. First, virtual size is not additive across containers because image bytes are shared. Second, write-heavy application data in a container writable layer can create extra copy-up work and makes lifecycle coupling worse. Durable or high-write data normally belongs in volumes or external storage, not the ephemeral writable layer.

5. What docker ps --size proves—and what it omits

docker ps --size reports a container’s writable-layer size plus a “virtual” size that combines the image’s read-only data with that writable state. Docker’s documentation warns that shared image layers make virtual-size totals misleading.

It also does not account for everything attributable to a workload: volume bytes, bind-mounted host data, logging-driver files, swap, and some metadata are outside that writable-layer number. Use it to answer “how much did this container write into its own copy-on-write layer?”, not “how much disk does this application consume in total?”

6. docker system df is the Docker-wide starting inventory

docker system df
docker system df -v

The summary separates images, containers, local volumes, and build cache and reports active/total counts, size, and reclaimable estimates. The verbose form adds object-level detail, including shared and unique image bytes. “Reclaimable” is a candidate estimate based on references known to Docker; it is not permission to delete. Business retention, disaster recovery, future rollback, and preexisting lab ownership remain separate decisions.

7. Builder cache is its own lifecycle

docker buildx ls
docker buildx du
docker buildx du --format=pretty

BuildKit cache can be large even when few containers exist. docker buildx du reports the currently selected builder’s records, whether they are reclaimable, shared, mutable/private, and their sizes. Shared cache may refer to storage also needed by images; deleting cache metadata is not the same as deleting a uniquely owned blob.

BuildKit also has periodic garbage collection. Current defaults use ordered policies based on cache type, age, sharing, and space thresholds. Manual pruning should complement an understood cache budget, not replace one.

8. Volumes are persistent data, not “leftover container files”

Chapter 20 established the lifecycle separation: deleting a container does not imply deleting its named volume. Disk-pressure work must preserve that boundary. A volume can be unused right now and still be intentionally retained for restore, rollback, or a stopped service.

Record each candidate’s name, driver, labels, consuming containers, backup status, and retention owner before removal. Never infer safety solely from “not mounted by a running container.”

9. Logs can exhaust disk without growing the writable layer

Docker’s default json-file logging driver does not rotate by default. High-volume stdout/stderr can therefore consume substantial storage while docker ps --size remains small. Docker recommends the local driver for many non-Kubernetes use cases because it rotates automatically and stores logs efficiently.

Log files are daemon-managed internals. Inspect logging configuration through Docker; do not truncate or edit Docker’s internal log files behind the daemon. A safe prevention policy sets bounded rotation for newly created containers and treats existing containers explicitly because daemon logging defaults are not retroactive.

10. Capacity includes inodes, not only bytes

A filesystem can report free gigabytes and still reject file creation if its inode pool is exhausted. Workloads that create huge numbers of tiny files—package caches, temporary trees, extracted dependency stores, or application churn—can hit this failure mode.

On an authorized local Linux daemon, pair byte evidence (df -h) with inode evidence (df -i) for the filesystems that actually contain Docker and managed containerd state. With remote contexts or Docker Desktop, host-shell filesystem commands may observe the wrong machine, so label that evidence carefully.

11. Engine 29 data-root nuance

Docker’s daemon documentation now makes an important split explicit. General daemon data such as volumes/configs remains under Docker’s data root (by default /var/lib/docker on Linux). When the Engine 29 containerd image store is active, image contents and container snapshots are stored under managed containerd state, documented under /var/lib/containerd by default.

Therefore setting Docker’s data-root alone does not relocate all image/snapshot bytes in the containerd-image-store architecture. Capacity planning and migration plans must identify every active storage domain first. This chapter does not mutate those roots.

12. Safety contract for cleanup

The mandatory labs use exact container/image/volume/builder names, preflight ownership guards, and labels. Prune commands are studied as policy mechanisms, but broad destructive execution is not the learning path.

13. Read-only first-pass checklist

docker context show
docker version
docker info
docker system df
docker system df -v
docker ps -a --size
docker volume ls
docker buildx ls
docker buildx du
docker info --format 'root={{.DockerRootDir}} driver={{.Driver}} logging={{.LoggingDriver}}'
docker info --format '{{json .DriverStatus}}'

Capture this before touching anything. It establishes endpoint identity, Engine version, storage architecture, logging baseline, object inventory, and builder-cache state.

Knowledge check

Official references and version notes

• Docker Docs — containerd image store with Docker Engine — Engine 29 fresh-install default, snapshotter reporting, switching, and migration boundaries.
• Docker Docs — Storage drivers — image layers, container writable layers, copy-on-write, and container-size accounting caveats.
• Docker Docs — OverlayFS storage driver — classic overlay2, backing-filesystem requirements, lower/upper/merged/workdir semantics, and migration cautions.
• Docker Docs — Select a storage driver — current containerd snapshotter versus classic-driver guidance and platform distinctions.
• Docker CLI — docker system df — daemon disk-usage summary and verbose object-level accounting.
• Docker CLI — docker system prune — deletion scope and supported filters; referenced here to understand risk, not as the lab cleanup path.
• Docker Buildx — buildx du — builder-cache usage, shared/private/reclaimable records, and detailed evidence.
• Docker Buildx — buildx prune — cache filters and space-budget controls.
• Docker Build — Build garbage collection — periodic BuildKit GC, default policy shape, and builder-specific configuration.
• Docker Docs — Docker daemon configuration overview — Docker data-root and the separate managed containerd root used by the Engine 29 containerd image store.
• Docker Docs — Configure logging drivers — json-file growth risk, rotation guidance, and the rotating local driver.
• Docker Engine 29 release notes — current Engine 29 behavior and storage/image-store fixes.