Git, Build Tools, Agents, and CI Runner Hosts

1. A CI runner executes repository-controlled code

A pipeline step can run shell commands, compile code, start containers, read accessible files, use credentials, and contact networks. Pull requests, dependencies, build scripts, test fixtures, and third-party actions can all influence execution. The runner’s trust boundary must therefore match the least-trusted code it may receive.

flowchart TD
  R["Repository event and workflow definition"] --> Q["Scheduler and runner selection"]
  Q --> H["Runner host or ephemeral instance"]
  H --> C["Checkout and dependency resolution"]
  C --> B["Build and test processes"]
  B --> A["Artifacts, logs, reports, caches"]
  S["Secrets and identity tokens"] --> B
  N["Network and internal services"] --> B
  P["Isolation, policy, cleanup, monitoring"] --> H

Do not place public-fork jobs, trusted release signing, production deployment, and internal-network builds on the same persistent runner pool. Separate by repository access, event type, sensitivity, network zone, credentials, and required hardware.

2. Ephemeral runners minimize cross-job contamination

A persistent runner can retain workspaces, processes, containers, caches, credentials, modified tools, and kernel state between jobs. Cleanup scripts reduce but cannot prove complete restoration after arbitrary code execution. An ephemeral runner is provisioned for a bounded job or small trusted batch and destroyed afterward.

Ephemeral does not mean unmonitored. Forward runner, provisioning, system, and job logs before destruction. Preserve the image version, runner version, job identity, and teardown result.

3. Separate runner registration, job identity, and deployment identity

Runner registration credentials enroll an executor. Job tokens grant bounded workflow access. Deployment identities authorize changes to environments. Do not reuse one long-lived host credential for all three roles. Prefer short-lived, audience-bound credentials such as workload identity or OIDC federation where the target platform supports it.

# Inventory service identity and accessible groups.
systemctl show actions.runner.service   -p User -p Group -p SupplementaryGroups -p Environment   -p FragmentPath 2>/dev/null || true
id RUNNER_USER 2>/dev/null || true
sudo -l -U RUNNER_USER 2>/dev/null || true

# Search configuration locations for risky broad permissions without printing secrets.
find /etc/systemd/system /opt /srv -maxdepth 4 -type f   -iname '*runner*' -printf '%m %u:%g %p
' 2>/dev/null | head -n 100

# Runtime-control groups are highly privileged on rootful hosts.
getent group docker 2>/dev/null || true
getent group libvirt 2>/dev/null || true

4. Checkout must define the exact commit and repository trust

Build from an immutable commit SHA, not a moving branch after approval. Clean the workspace, control submodules and Git LFS, avoid unsafe credential persistence, and record repository URL, commit, tree, submodule commits, and dirty state. For forked pull requests, distinguish the untrusted head repository from the trusted base repository.

set -Eeuo pipefail
repo=${1:?Usage: build-checkout REPOSITORY_URL COMMIT_SHA}
commit=${2:?Usage: build-checkout REPOSITORY_URL COMMIT_SHA}
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT

git -c protocol.version=2 clone --no-checkout --filter=blob:none "$repo" "$work/repo"
git -C "$work/repo" fetch --depth=1 origin "$commit"
git -C "$work/repo" checkout --detach FETCH_HEAD

git -C "$work/repo" rev-parse HEAD
git -C "$work/repo" rev-parse HEAD^{tree}
git -C "$work/repo" status --porcelain=v1

git -C "$work/repo" submodule status --recursive 2>/dev/null || true
git -C "$work/repo" remote -v

Never interpolate untrusted branch names, issue titles, matrix values, or commit messages directly into shell code. Pass them through environment variables and quote them, or use structured APIs.

5. Reproducibility requires pinned tools and dependency policy

The same commit can produce different output when compilers, base images, package repositories, locales, clocks, CPU features, or dependency resolution change. Pin toolchain versions, lock dependencies, capture environment metadata, and decide whether network access is allowed during the build.

evidence="$PWD/build-evidence"
mkdir -p "$evidence"
{
  date -u --iso-8601=seconds
  uname -a
  locale
  umask
  git rev-parse HEAD 2>/dev/null || true
  env | sed -E 's/(TOKEN|SECRET|PASSWORD|KEY)=.*/\1=[REDACTED]/I' | sort
} > "$evidence/environment.txt"

for cmd in gcc clang cmake make ninja python node npm java mvn gradle go rustc cargo; do
  command -v "$cmd" >/dev/null || continue
  { "$cmd" --version || "$cmd" -version; } >> "$evidence/toolchains.txt" 2>&1 || true
done

find . -maxdepth 3 -type f   \( -name '*lock*' -o -name 'go.sum' -o -name 'pom.xml'      -o -name 'requirements*.txt' \) -print > "$evidence/dependency-files.txt"
sha256sum "$evidence"/* > "$evidence/SHA256SUMS"

6. Caches are performance inputs with trust and invalidation rules

A cache key should include every input that affects compatibility: operating system, architecture, toolchain, lockfile, build mode, and relevant feature flags. Untrusted jobs should not be able to poison caches later consumed by trusted release jobs. Separate namespaces or permit trusted jobs to read but not blindly execute untrusted cached content.

Example cache-key dimensions:
  trust-domain / operating-system / architecture / toolchain-version /
  dependency-lock-hash / build-mode / feature-set

Unsafe:
  linux-build-cache

Safer:
  trusted-main/linux/x86_64/gcc-15/sha256(lockfile)/release/v3

Do not cache credentials, signing material, deployment configuration, or entire home directories. Record cache hit/miss rates, size, eviction, and restore source so performance tuning does not become an opaque supply-chain path.

7. Containers on runners do not automatically isolate the runner host

Containerized jobs share the runner kernel. Mounting the Docker socket gives the job control of the rootful daemon. Docker-in-Docker often requires privileged mode or broad capabilities. Host networking, host PID namespace, device passthrough, and writable host mounts can collapse isolation.

# Host checks for risky runner access.
runner_user=${1:-runner}
id "$runner_user" 2>/dev/null || true
getent group docker 2>/dev/null || true

# List rootful runtime sockets and permissions.
find /run /var/run -maxdepth 3 -type s   \( -name '*docker*' -o -name '*containerd*' -o -name '*podman*' -o -name '*crio*' \)   -exec stat -Lc '%A %U:%G %n' {} + 2>/dev/null

# Look for persistent containers and mounts after jobs.
docker ps -a --no-trunc 2>/dev/null || true
podman ps -a --no-trunc 2>/dev/null || true
findmnt -R /var/lib/docker 2>/dev/null | head -n 80 || true

For hostile or cross-tenant builds, prefer a fresh VM or similarly strong isolation boundary. Use separate runner pools for jobs that require privileged containers, kernel modules, hardware devices, or internal network access.

8. Harden the runner service but preserve required functionality

Run the agent as a dedicated service account, with no interactive login, minimal groups, bounded filesystem access, explicit environment, resource limits, and controlled network routes. systemd can add protections, but some build tools need namespaces, writable paths, or executable temporary storage; validate each directive.

# Example drop-in: /etc/systemd/system/runner.service.d/hardening.conf
[Service]
User=ci-runner
Group=ci-runner
UMask=0077
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/var/lib/ci-runner /var/cache/ci-runner
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryMax=8G
TasksMax=4096
LimitNOFILE=65536

# Add SystemCallFilter, RestrictAddressFamilies, and network isolation only
# after testing the exact runner, toolchains, containers, and service hooks.

sudo systemd-analyze verify /etc/systemd/system/runner.service 2>/dev/null || true
sudo systemctl daemon-reload
systemd-analyze security --no-pager runner.service
systemctl show runner.service -p User -p ControlGroup -p MemoryMax -p TasksMax

9. Capacity planning combines concurrency, duration, and headroom

For an arrival rate of λ jobs per minute and mean execution time W minutes, Little’s Law gives average jobs in the system:

Track queue duration, pickup latency, job duration distributions, failure rate, cancellation, cache performance, disk pressure, CPU and memory pressure, and cleanup failures. Autoscaling without image prewarming or external quota awareness can move the bottleneck rather than remove it.

10. Cleanup and evidence close the execution boundary

At job end, terminate descendants, unmount transient filesystems, stop and remove containers, revoke short-lived credentials, archive authorized artifacts and logs, and destroy or reset the executor. A persistent runner should fail closed when cleanup cannot prove completion.

# Illustrative post-job evidence; adapt to the runner product.
out="/var/lib/ci-runner/evidence/${JOB_ID:-manual}-$(date -u +%Y%m%dT%H%M%SZ)"
sudo install -d -m 0700 -o root -g root "$out"

{
  date -u --iso-8601=seconds
  hostname
  systemctl --failed --no-pager
  ps -eo pid,ppid,user,etimes,cmd --sort=ppid
  findmnt
  df -h
  df -i
} | sudo tee "$out/host-after.txt" >/dev/null

sudo docker ps -a --no-trunc > "$out/docker-after.txt" 2>&1 || true
sudo journalctl --since -30min -u runner.service --no-pager   > "$out/runner-journal.txt" 2>&1 || true
sudo sha256sum "$out"/* | sudo tee "$out/SHA256SUMS" >/dev/null

Evidence may contain commands, repository paths, usernames, IP addresses, and secrets. Restrict access and redact before broad sharing.

11. Hands-on lab: create a bounded local build worker

This lab uses a transient user service to execute a small build-like task with limits. It does not register a real CI runner or access external repositories.

set -Eeuo pipefail
lab="$HOME/devops-academy/linux/chapter19/lesson04/lab"
rm -rf "$lab"
mkdir -p "$lab/src"
printf '#include <stdio.h>
int main(void){puts("runner lab");}
' > "$lab/src/main.c"

if command -v gcc >/dev/null; then
  systemd-run --user --wait --collect     -p WorkingDirectory="$lab/src"     -p MemoryMax=256M -p CPUQuota=50% -p TasksMax=64     /bin/sh -c 'set -eu; gcc -Wall -Wextra -O2 main.c -o app; ./app; sha256sum app'     > "$lab/build-output.txt" 2>&1
else
  printf 'gcc is unavailable; no package was installed.
' > "$lab/build-output.txt"
fi

{
  date -u --iso-8601=seconds
  uname -a
  command -v gcc >/dev/null && gcc --version | head -n 1 || true
  sha256sum "$lab/src/main.c"
} > "$lab/build-metadata.txt"
sha256sum "$lab"/*.txt > "$lab/SHA256SUMS"
printf 'Review %s
' "$lab"

12. Common runner-host mistakes

13. Knowledge check

14. Summary

• CI runners execute repository-influenced code and must be scoped to explicit trust domains.
• Ephemeral executors reduce state contamination and simplify reset assurance.
• Checkout, toolchains, dependencies, caches, artifacts, secrets, and network access are supply-chain inputs.
• Runtime sockets, privileged builds, and broad host groups can collapse isolation.
• Runner operations require hardening, capacity metrics, external logs, cleanup evidence, and separate deployment identity.

15. Further reading

• GitHub Actions — Self-hosted runners reference
• GitHub Actions security hardening
• GitLab Runner security
• Reproducible Builds documentation
• systemd execution environment