Container Filesystems, Writable Layers, docker cp, exec Workflows, Processes, Signals, and PID 1: Concepts, Architecture, and Mental Model

1. The problem: a container can look correct while its declared image is wrong

A common early Docker habit is to enter a running container, edit a configuration file, install a package, or copy a missing file into place until the application works. The immediate process may recover, but the next container created from the same image starts from the old declared state. The runtime object and the image have diverged.

Chapter 04 established immutable image identity. Chapter 05 separated the Docker container object from its primary process. Chapter 06 adds the two runtime layers that explain most “it worked in this container” surprises: the container-specific writable filesystem layer and the process tree rooted at PID 1.

2. Mental model: immutable layers → writable layer + mounts → process tree

At runtime Docker presents a filesystem assembled from read-only image content plus a thin writable layer unique to the container. Explicit volumes, bind mounts, tmpfs mounts, and other mount types can then be attached at particular paths. A mount at a path can obscure what the image or writable layer contains underneath that path.

Inside that filesystem, Docker starts the configured primary process. In a Linux container that process becomes PID 1 in the container PID namespace. Additional processes may be children of PID 1 or may be launched later through docker exec. Docker lifecycle signals target the primary process, so the startup form and the application's signal behavior matter operationally.

flowchart TD
  IMG[Immutable image layers] --> ROOT[Container root filesystem view]
  RW[Per-container writable layer] --> ROOT
  MNT[Explicit mounts] --> ROOT
  ROOT --> P1[Primary process / PID 1]
  P1 --> CH[Child processes]
  EXEC[docker exec] --> EP[Additional exec process]
  SIG[Docker stop signal] --> P1
  RW --> DIFF[docker diff evidence]
  ROOT --> CP[docker cp / inspect evidence]
  P1 --> LOG[logs / exit / signal evidence]
            

The diagram separates three questions that must not be collapsed: where did a file come from?, which process changed or consumes it?, and will the state survive creation of a new container from the same image?

3. The writable layer is container-local runtime state

Docker's current storage documentation describes the default container filesystem as immutable image layers plus a unique writable container layer. New files, modifications, and deletion markers that are not redirected into a mount live in this writable layer. Deleting the container deletes that writable state; the underlying image is unchanged.

This makes the writable layer appropriate for ephemeral runtime output such as temporary files, caches with no durability requirement, PID files, and short-lived diagnostic artifacts. It is a poor place for database data, configuration that must survive replacement, or release content that should be identical across environments.

4. docker diff shows filesystem changes, not intent

docker diff CONTAINER reports paths Docker sees as Added, Changed, or Deleted compared with the container's initial filesystem. It is useful evidence for runtime drift, but it does not tell you why a path changed, whether the change was legitimate, whether the data is sensitive, or whether a mount owns the path.

docker inspect CONTAINER --format 'Image={{.Image}} Status={{.State.Status}}'
docker inspect CONTAINER --format '{{json .Mounts}}'
docker diff CONTAINER

Expect applications and base images to create their own runtime paths. Diagnose only after correlating the path with mounts, process behavior, application logs, and the change you intentionally made.

5. docker cp moves files, but it does not create a release

Current Docker documentation allows docker cp between the local filesystem and either a running or stopped container. It recursively copies directories, preserves permissions when possible, applies destination ownership rules, supports archive mode with -a, and can follow source symlinks with -L. Parent destination directories are not automatically created in every case.

These mechanics make docker cp useful for bounded diagnostics: extract a generated report from a failed container, place a harmless fixture into a disposable container, or preserve a non-secret file before removal. They do not make it a configuration-management or secret-delivery system.

6. PID 1 is operationally special

In a Linux container, the configured primary process normally becomes PID 1 in the container PID namespace. Linux treats PID 1 specially: among other responsibilities, it must reap orphaned child processes, and default signal behavior differs from ordinary processes. An application that was never designed to be PID 1 can therefore exhibit surprising shutdown or zombie-process behavior.

Docker does not magically make every process a correct init system. If the application handles signals and child processes correctly, it can be PID 1 directly. If it spawns children and does not reap them, Docker's --init option can insert a small Tini-backed init as PID 1 to forward signals and reap children.

7. Exec-form and shell-form startup change the process tree

Dockerfile CMD and ENTRYPOINT can be written in exec/JSON form or shell form. Current Docker guidance recommends JSON arguments for predictable signals. With shell form, Docker launches a shell such as /bin/sh -c; the intended application becomes a child and may not receive the stop signal because the shell does not necessarily forward it.

With exec form, the executable is launched directly as the primary process. If a wrapper script is needed, that script should normally use the shell's exec builtin for the final application so the application replaces the wrapper and becomes PID 1.

# Less predictable for application signals
CMD my-server --foreground

# Direct process; no implicit shell parent
CMD ["my-server", "--foreground"]

Chapter 07 teaches Dockerfile syntax in depth. Here, the point is the runtime consequence: startup syntax changes which process owns PID 1 and therefore which process Docker signals.

8. docker exec is a runtime interaction, not image engineering

docker exec asks the daemon to start an additional command inside a running container. That command exists only while the container's primary process is running and is not automatically restarted with the container. It can inspect state or run a bounded administrative task, but it does not rewrite the image.

An interactive edit made through exec can also make incident diagnosis harder: the current filesystem no longer represents the original failure. Capture docker diff, logs, process state, and copied evidence before mutating a failing container.

9. Mounts can hide image content without deleting it

A bind mount, volume, or tmpfs placed over a non-empty path obscures the underlying image/writable-layer content at that path. The underlying files have not necessarily been deleted; they are simply hidden by the mounted filesystem in that container. Docker documentation recommends recreating the container without the mount to reveal the original path again.

This distinction prevents a common false diagnosis: “the image lost its configuration file” when the real cause is “a mount covers the directory.” Always inspect .Mounts before rebuilding or copying files into an apparently empty path.

10. DevOps connection: declare state, observe runtime drift

A production Docker operating model should let you prove the chain from source and Dockerfile to image digest, container configuration, mounts, runtime filesystem delta, process tree, signal/exit behavior, and external health. When a runtime edit is necessary for investigation, record it as evidence and then convert the intended change into source-controlled image or configuration state.

The goal is not “never touch a running container.” The goal is to keep diagnosis, emergency action, and deployable configuration as distinct states with an auditable path between them.

11. Small challenge: classify four changes

For each change below, state whether it belongs in the image, a mount/configuration mechanism, the writable layer, or a one-time diagnostic action—and explain what survives container replacement:

1. An application binary required by every deployment.
2. A database data directory that must survive replacement.
3. A temporary stack trace copied out after a crash.
4. A missing configuration file manually copied into a failing container to test a hypothesis.

The fourth item can be a valid experiment, but the successful experiment is not the final deployment. The durable fix must be expressed in declared source/image/configuration state.

Knowledge check

Official references and version notes

• Docker storage overview — explains the ephemeral per-container writable layer and why persistent data belongs in explicit mounts rather than the container layer.
• Storage drivers and writable layers — copy-on-write concepts, writable-layer behavior, and the Engine 29 distinction between the containerd image store and classic storage-driver examples.
• docker container diff — reports added, changed, and deleted paths in the container filesystem relative to its initial image-backed state.
• docker container cp — current copy semantics, stopped-container support, destination ownership, archive mode, symlink handling, and path rules.
• docker container exec — starts a new command only while the container's primary PID 1 is running and does not make image changes durable.
• docker container run — current runtime options including --init, stop signals/timeouts, mounts, TTY behavior, and security controls.
• docker container stop — graceful signal delivery and timeout-to-SIGKILL escalation.
• docker container attach — documents special PID 1 signal behavior and terminal signal-proxy implications.
• JSONArgsRecommended build check — why exec-form CMD/ENTRYPOINT avoids an unintended shell parent and improves signal handling.
• Docker build best practices — entrypoint scripts should normally exec the final application so it becomes PID 1 and receives signals directly.
• Bind mounts — mounting over an existing container path obscures image/writable-layer content until the container is recreated without that mount.
• Docker Engine 29 release notes — Engine 29.8.1 is the current patch baseline at verification time; recent 29.x releases also include multiple docker cp security and compatibility fixes.