Runner Scale Sets, Actions Runner Controller, and Kubernetes Autoscaling: Guided Hands-On Workflow

1. Two paths, one learning objective

The mandatory path never asks for a token. The live path is an extension, not a prerequisite for course completion.

2. Mandatory simulation: make the lifecycle observable

Create a temporary directory and save the following as arc_sim.py. It models one queued job with min=0 and max=2, writes JSONL evidence, creates one synthetic runner directory, executes a harmless synthetic step, then deletes the runner directory.

from pathlib import Path
import json, shutil, time

root = Path("arc-sim-state")
evidence = Path("arc-evidence.jsonl")
if root.exists(): shutil.rmtree(root)
if evidence.exists(): evidence.unlink()
root.mkdir()

def emit(state, **data):
    row = {"state": state, **data}
    evidence.open("a", encoding="utf-8").write(json.dumps(row) + "\n")
    print(json.dumps(row))

min_runners, max_runners, queued_jobs = 0, 2, 1
emit("preflight", minRunners=min_runners, maxRunners=max_runners, pods=0)
emit("job_available", queuedJobs=queued_jobs)
desired = min(max_runners, min_runners + queued_jobs)
emit("listener_ack", desiredRunners=desired)
runner = root / "runner-001"
runner.mkdir()
emit("runner_pod_created", pod="runner-001", serviceAccount="arc-lab-no-permission")
emit("jit_registered", runner="runner-001", job="job-001")
(runner / "result.txt").write_text("synthetic job ok\n", encoding="utf-8")
emit("job_completed", job="job-001", conclusion="success")
shutil.rmtree(runner)
emit("runner_pod_deleted", pod="runner-001", remainingPods=0)

mkdir arc-ch10-lab && cd arc-ch10-lab
python arc_sim.py
cat arc-evidence.jsonl
find arc-sim-state -maxdepth 2 -type f -print

Expected evidence has an ordered chain from job_available through runner_pod_deleted; the final filesystem contains no runner directory. This is a faithful control-plane simulation, not proof that ARC itself is installed.

3. Reconcile the simulated states

Before you change anything, write predictions: one queued job should request one runner, maxRunners=2 should never be exceeded, one job should complete, and runner state should disappear. Then compare each prediction with the JSONL evidence. If the file skips a transition, the experiment is incomplete even if the final line says success.

4. Optional local Kubernetes preflight

If Docker and kind are available, a local cluster gives you real Kubernetes objects without requiring a GitHub account mutation. Guard the exact context before destructive commands.

kind create cluster --name arc-course-lab
kubectl config current-context
# Expected exactly: kind-arc-course-lab
kubectl get nodes -o wide
kubectl get namespaces
helm version

If the context is not exactly kind-arc-course-lab, stop. Do not run cleanup commands against another cluster.

5. Optional live ARC preflight

Proceed only with a throwaway repository that you own and can delete. Record the repository URL, default branch, current source SHA, Kubernetes context, namespaces and intended scale-set name. Prefer a GitHub App for repository/organization registration. Keep private-key material in a local file with restrictive permissions and inject it into a Kubernetes Secret by file reference; never paste the private key into Helm command history or the lesson log.

kubectl config current-context
helm list -A
kubectl get ns arc-systems arc-runners 2>/dev/null || true
# Record metadata only; do not dump secret values.
kubectl get secrets -A

6. Pin controller and scale-set charts

For a live disposable lab, create the authentication Secret first according to your GitHub App policy, then use a values file that references only the Secret name.

# values.arc-lab.yaml
githubConfigUrl: "https://github.com/OWNER/DISPOSABLE_REPO"
githubConfigSecret: arc-github-auth
runnerScaleSetName: arc-ch10-lab
minRunners: 0
maxRunners: 2
template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]

helm upgrade --install arc-controller   oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller   --version 0.14.2 --namespace arc-systems --create-namespace

helm upgrade --install arc-ch10-lab   oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set   --version 0.14.2 --namespace arc-runners --create-namespace   -f values.arc-lab.yaml

helm list -A
kubectl -n arc-systems get pods -o wide
kubectl -n arc-runners get pods -o wide

7. Run one synthetic job

Add only this workflow to the disposable repository. It needs no checkout, secrets, packages or external target.

name: arc-ch10-probe
on:
  workflow_dispatch:
permissions: {}
jobs:
  probe:
    runs-on: arc-ch10-lab
    steps:
      - name: Record bounded runtime evidence
        shell: bash
        run: |
          printf 'run=%s attempt=%s sha=%s\n'             "$GITHUB_RUN_ID" "$GITHUB_RUN_ATTEMPT" "$GITHUB_SHA"
          printf 'runner=%s os=%s arch=%s\n'             "$RUNNER_NAME" "$RUNNER_OS" "$RUNNER_ARCH"
          uname -a

Predict before dispatch: zero runner pods at idle; one queued job; listener requests one runner; one ephemeral pod appears; the job starts; after completion the pod disappears. Record run ID, attempt and SHA from the job log.

8. Observe scale-up and scale-down without racing the evidence

Use separate terminals or a timestamped log capture. Ephemeral pods can appear and disappear quickly.

kubectl -n arc-runners get pods -w
kubectl -n arc-runners get events --sort-by=.lastTimestamp
kubectl -n arc-runners logs -l app.kubernetes.io/component=runner-scale-set-listener --tail=200
kubectl -n arc-systems logs deploy/arc-controller-gha-rs-controller --tail=200

Exact deployment/resource names can vary with installation name and chart version, so discover names with kubectl get before hard-coding a log command.

9. What to capture

• ARC chart version and resolved controller image.
• GitHub configuration scope, scale-set name, runner group/labels if used.
• minRunners/maxRunners and namespace names.
• run ID, attempt, source SHA and job conclusion.
• listener/controller timestamps showing demand and reconciliation.
• runner pod UID, service account, node and image digest.
• proof that idle state returns to zero pods.
• limitations: local cluster, synthetic job, no production network or cloud credentials.

10. Cleanup is part of the lab

For the simulator, remove only the lab directory. For the kind/live path, first verify the context, then uninstall the exact release names and delete the disposable cluster.

# Simulator
cd .. && rm -rf arc-ch10-lab

# Optional live path — guard exact context first
kubectl config current-context
helm uninstall arc-ch10-lab -n arc-runners
helm uninstall arc-controller -n arc-systems
kind delete cluster --name arc-course-lab

Also revoke/delete the lab GitHub App installation or PAT according to the credential owner’s process, and delete the disposable repository if it was created solely for this lab. Pod deletion is not the credential-revocation step.

11. Small challenge: choose the failing layer

A job is queued for arc-ch10-lab. The listener log shows demand=1 and desired=1, but no runner pod appears. Which layer should you inspect first?

Answer before revealing the knowledge check: this is already past trigger/workflow selection and past listener demand recognition. Inspect Kubernetes reconciliation, events, quota, admission policy and scheduling—not workflow YAML or GitHub token scopes first.

Knowledge check

Official references and version notes

• GitHub Docs — Actions Runner Controller — control-plane architecture, listener/JIT registration and ephemeral runner lifecycle.
• GitHub Docs — Deploy runner scale sets — Helm deployment, runner groups, min/max runners, pod templates, container modes and security guidance.
• GitHub Docs — Use ARC in a workflow — scale-set names and scale-set labels in runs-on.
• GitHub Docs — Troubleshoot ARC — controller/listener/runner diagnostic workflow.
• ARC release 0.14.2 — pinned release used for concrete examples in this chapter.
• GitHub Actions runner container image — minimal runner image published with runner releases.