Service Containers, Job Containers, Docker Builds, and Integration-Test Environments: Diagnostics, Failure Modes, and Production Practices

1. Evidence-first diagnostic sequence

Preserve the first failed run and attempt. Record the source SHA and workflow revision. Confirm the event/trust level and token permissions. Inspect job graph/queue state. Confirm the job runs on Linux and whether steps are on the host or in a job container. Inspect service health and the exact address the application used. Only then inspect Docker build output, images, artifacts or external registries.

Rerun the smallest equivalent scope only after a causal hypothesis exists. A rerun can pull a different mutable image tag or execute on a newer runner image, so “the rerun passed” is not itself diagnosis.

2. Intentionally broken example: localhost inside a job container

The following topology is valid YAML and Redis can become healthy, but the client address is wrong. The Python process runs inside the job container, where 127.0.0.1 points back to the Python container itself.

jobs:
  broken:
    runs-on: ubuntu-24.04
    container:
      image: python:3.13-slim
    services:
      redis:
        image: redis:7.4-alpine
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 3s
          --health-retries 10
    steps:
      - name: Broken network assumption
        shell: sh
        run: |
          python - <<'PY'
          import socket
          socket.create_connection(('127.0.0.1', 6379), timeout=3)
          PY

Expected evidence is a connection failure from the application step while the service health setup succeeded. The repair is one field: connect to redis:6379. Do not add host networking, privileged mode or random port mappings.

3. Health-check failure is a different layer

If the service health command itself is wrong—for example, it probes a nonexistent command/port—the job may fail during service initialization before application steps run. Preserve the service-startup logs and inspect the configured health command. Repair the health check or image contract, not the application hostname.

Health checks should be bounded and representative. Extremely long retries hide startup regressions; overly aggressive timeouts create flaky infrastructure failures.

4. Unnecessary published ports increase attack surface and confusion

A job container does not need host port publication to reach sibling services. Publishing ports anyway creates an extra host exposure and can mislead future maintainers into using localhost. Expose only the ports required by processes outside the Docker network.

On persistent self-hosted runners, unintended published ports can also collide with local services or create network reachability beyond the job. Treat port publication as a security/configuration change.

5. Registry credentials belong outside layers and logs

Do not write registry tokens into Dockerfiles, ENV instructions, build arguments, command history or copied configuration files. Those values can survive in layers or metadata. Prefer scoped registry login/federation and BuildKit secret mechanisms for build-time secrets.

Pull requests from forks or other untrusted sources must not be allowed to use privileged registry write credentials to publish tags. Build/test locally and publish only from a trusted release path.

6. Privileged containers and Docker socket are host-adjacent capabilities

--privileged grants a broad Linux capability/device surface. Mounting /var/run/docker.sock into a container often gives effective control over the host Docker daemon. These are not normal fixes for “Docker command failed.”

If a workflow truly requires Docker-in-Docker or host daemon access, isolate the runner, document the threat model and keep untrusted PR code away from that runner. Chapter 9 and Chapter 10's runner isolation rules still apply.

7. Container filesystems do not survive job boundaries

A file written inside a job container exists only in that job's workspace/container mount state. Another job normally starts on a fresh runner with a fresh filesystem. Use job outputs for small control values, artifacts for explicit cross-job files, and caches only for best-effort reusable acceleration state.

Do not “fix” a missing file in a later job by assuming Docker volumes from the earlier hosted runner remain attached.

8. Mutable tags can turn reruns into different experiments

If redis:7.4-alpine or python:3.13-slim moves between attempts, the same workflow text can execute different image content. In the beginner lab we accept that convenience but record the observed identities. Production workflows that require reproducibility should pin image digests and review updates deliberately.

The same rule applies to executable actions: pin full SHAs rather than moving action tags.

9. Do not push an unverified latest image from a PR

A pull request tests proposed code; it is not a release authorization. Publishing latest from PR context can overwrite a trusted consumer reference, leak registry credentials or create a supply-chain confusion event. Keep PR builds local or publish into isolated, run-scoped, non-production namespaces only when policy explicitly supports it.

10. Failure classification matrix

Knowledge check

Official references and version notes

• GitHub Docs — workflow syntax: job containers and services — Linux requirement, shells, volumes, options and service networking.
• GitHub Docs — Redis service containers — host versus job-container topology and health checks.
• GitHub Docs — run jobs in a container — container credentials, ports, volumes and default shell behavior.
• docker/setup-buildx-action v4.1.0 — current optional Buildx setup action, production reference d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5.
• docker/build-push-action v7.3.0 — current optional Docker build action, production reference 53b7df96c91f9c12dcc8a07bcb9ccacbed38856a.
• docker/login-action v4.6.0 — registry authentication action, production reference dbcb813823bdd20940b903addbd779551569679f.