Runner Fleets, Autoscaling, Docker Machine Migration, Kubernetes Runners, Ephemeral Workers, and Capacity Planning: Concepts, Architecture, and Mental Model

1. The practical problem: green jobs can hide an unhealthy fleet

Chapter 29 treated infrastructure mutation as a reviewed state transition. Chapter 30 turns to the infrastructure that executes the pipeline itself. A fleet can produce green jobs while developers wait ten minutes in queue, while idle VMs burn money, while an old Runner version silently misses fixes, or while reused workers retain state from previous untrusted jobs. Conversely, a low-utilization fleet is not automatically wasteful if warm capacity is what keeps critical feedback inside an agreed latency objective.

The operating question is therefore not “how many runners do we have?” It is: which jobs arrive, which runners are eligible, how quickly can managers acquire work, how many workers can be provisioned, how long do they live, what isolation boundary do they provide, and what observable evidence proves cost and queue behavior?

2. Mental model: demand becomes a bounded worker lifecycle

A GitLab pipeline creates jobs with requirements such as tags, protection/trust context, platform and image. Eligible runner managers poll for work. The manager is not necessarily the worker: with Docker Autoscaler or Instance executors, it asks a Fleeting plugin to manage instances; with Kubernetes executor, it asks the Kubernetes API for a new job pod. The worker executes the job, sends logs/artifacts back, and is then reused or destroyed according to policy. Queue duration, provisioning latency, running jobs, errors, worker state and cost feed the next sizing decision.

The loop is the important part. Autoscaling is control feedback, not a one-time configuration choice.

3. State domains to keep separate

4. Read-only inspection before any fleet change

Start with evidence that does not register, delete, scale or restart anything. On a disposable/self-managed runner manager, typical read-only checks are:

gitlab-runner --version
gitlab-runner list
# If metrics listening is already enabled, read it without changing configuration:
curl -fsS http://127.0.0.1:9252/metrics   | grep -E 'gitlab_runner_(version_info|concurrent|limit|request_concurrency|jobs|job_queue_duration_seconds|errors_total)'   | head -80

In GitLab, record the runner ID/scope/tags/protection and the waiting job’s CI_PIPELINE_SOURCE, CI_COMMIT_SHA, pipeline/job IDs and requested tags. The metrics endpoint is optional and must already be enabled; do not expose it publicly merely for a lab.

5. Current executor map

6. Docker Autoscaler and Instance: Fleeting is the provider boundary

Current GitLab Runner uses Taskscaler for autoscaling logic and Fleeting as a plugin abstraction over AWS Auto Scaling Groups, Google instance groups, Azure VM Scale Sets, and supported community providers. Docker Autoscaler wraps Docker executor behavior; Instance executor gives jobs host-level access. Both use [runners.autoscaler].

concurrent = 12

[[runners]]
  name = "example-ephemeral-linux"
  executor = "docker-autoscaler"
  limit = 12
  request_concurrency = 4

  [runners.docker]
    image = "alpine:3.22"

  [runners.autoscaler]
    plugin = "aws"
    capacity_per_instance = 1
    max_use_count = 1
    max_instances = 12

    [[runners.autoscaler.policy]]
      idle_count = 2
      idle_time = "10m0s"

This is architecture syntax, not a runnable lab: it deliberately omits provider credentials and provider-specific resource identifiers. The hard cost/isolation controls are visible: at most 12 instances, one job per instance, one use per instance. concurrent is global to the runner process; limit is per runner configuration.

7. One autoscaling resource has one controller

GitLab explicitly requires every Docker Autoscaler configuration to own a dedicated cloud autoscaling resource. Do not attach two runner managers—or two [[runners]] entries—to the same provider group. Each manager maintains instance bookkeeping; competing controllers can issue conflicting scale commands, causing job failures and cost surprises.

8. Kubernetes executor: one pod per job is not one node per job

The Kubernetes executor creates a pod for each GitLab CI job. That pod is a worker execution unit, while node capacity belongs to the cluster. If pods queue in Pending, the bottleneck may be CPU/memory requests, quotas, taints, topology, image pulls, or node-autoscaler latency—not Runner job polling. Current Runner documentation also includes pause-pod prewarming for Kubernetes executor; treat it as a latency/cost tradeoff and verify the exact Runner version before using it.

9. Four different capacity ceilings

Current Runner guidance notes that high-volume runners may need request_concurrency above its default of 1. The adaptive request-concurrency feature flag can tune effective concurrency up to the configured ceiling; it cannot exceed a ceiling of 1 unless you raise it.

10. Capacity planning starts with queue duration, not CPU alone

Runner exposes Prometheus metrics including current jobs, job duration, queue duration, manager errors, configured concurrency/limit/request concurrency, autoscaling machine state and creation duration. A healthy fleet tracks at least queue latency, provisioning latency, job duration, utilization, errors/retries and infrastructure cost per successful job.

11. Ephemeral is a lifecycle property, not a security slogan

For Fleeting-backed VM workers, capacity_per_instance = 1 and max_use_count = 1 provide a strong single-job instance lifecycle when combined with a secure base image and teardown verification. Reuse can reduce cold-start cost but increases cross-job residue risk. Kubernetes creates a fresh job pod, yet shared nodes, caches, volumes, service accounts and privileged settings can still cross trust boundaries. Chapter 31 threat-models those boundaries in depth.

12. Runner upgrades are fleet changes

Pin and record Runner versions. Current stable patch on this chapter’s verification date is 19.3.2. Use a canary pool, pause/drain it before manager upgrade where practical, verify representative jobs and metrics, then expand. For the Kubernetes Helm chart, GitLab recommends pausing the runner and waiting for jobs to complete before upgrading. Keep the prior package/image/chart and configuration as an explicit rollback target.

13. Docker Machine migration is now an operational deadline

Docker Machine executor is deprecated and scheduled for removal in GitLab 20.0 (May 2027). A migration should inventory MachineOptions, instance image assumptions, caches, network/security rules, idle policy, job concurrency, special devices and provider quotas; then map those requirements to Docker Autoscaler or Instance plus a Fleeting plugin. Do not perform an in-place “syntax translation” without a capacity and rollback test.

14. Misconceptions to remove now

Knowledge check

Official references and version notes

Documentation verification date: 2026-09-12. Current stable GitLab Runner patch verified for this chapter is 19.3.2 (tagged 2026-09-10). The Docker Machine executor was deprecated in GitLab 17.5 and is scheduled for removal as a supported feature in GitLab 20.0 (May 2027); GitLab directs users toward the Instance or Docker Autoscaler executors. Docker Autoscaler is GA and, together with the Instance executor, uses Taskscaler/Fleeting. A Docker Autoscaler configuration must have its own dedicated provider autoscaling resource and must not share that resource with another manager or another [[runners]] entry. The mandatory exercises in this chapter are local simulations and require no cloud account, runner registration token, privileged executor, or managed Kubernetes cluster. Runner 19.3 introduced gitlab-runner config lint; use it during real fleet changes to validate manager configuration syntax before rollout, while still testing behavior on a canary.

• Executors — official reference.
• GitLab Runner autoscaling — official reference.
• Docker Autoscaler executor — official reference.
• Instance executor — official reference.
• Fleeting — official reference.
• Plan and operate a runner fleet — official reference.
• Advanced Runner configuration — official reference.
• Monitor GitLab Runner usage — official reference.
• Kubernetes executor — official reference.
• GitLab Runner Helm chart — official reference.
• Docker Machine executor — official reference.
• GitLab deprecations and removals — official reference.
• Runner troubleshooting — official reference.
• GitLab Runner changelog — official reference.
• GitLab Runner tags — official reference.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.