Kubernetes Plugin, Pod Templates, Dynamic Kubernetes Agents, Volumes, Service Accounts, and Cluster Scaling: Concepts, Architecture, and Mental Model

1. The problem: “ephemeral agent” is not one state

Chapter 19 separated a container from the Docker daemon that launched it. Kubernetes adds another orchestration layer. A Jenkins build may be queued, a pod may be created, the pod may be Pending, containers may be Running, and yet the Jenkins agent can still be offline because Remoting never connected. Conversely, a Jenkins stage can succeed while evidence remains only in an emptyDir that disappears seconds later.

The operating question is therefore not “did Kubernetes run my build?” It is: which Jenkins build requested which pod template, which Kubernetes pod/UID and images executed it, under which identity and resource boundary, which evidence escaped the pod, and what was deleted afterward?

2. Mental model: queue demand to deleted pod

Read this chain left to right. Jenkins first has a queue item with an agent requirement. The Kubernetes cloud resolves that requirement to a pod template. The plugin asks the Kubernetes API to create a pod. Kubernetes admission, RBAC and scheduling determine whether that pod can exist and where it runs. The agent container then establishes Remoting back to Jenkins. Only after that connection does Jenkins have an executor/workspace in which Pipeline steps run. Evidence that matters must leave pod-scoped storage before the pod is deleted.

flowchart TD
  A[Queued Jenkins task] --> B[Kubernetes cloud + pod template]
  B --> C[Kubernetes API creates Pod]
  C --> D[Admission + scheduler + node capacity]
  D --> E[ServiceAccount + containers + volumes]
  E --> F[Inbound Remoting connection]
  F --> G[Jenkins node/executor + workspace]
  G --> H[Build/test steps]
  H --> I[Archive or external evidence]
  I --> J[Pod termination and capacity release]

Every arrow changes a different state. The Kubernetes plugin orchestrates the path, but it does not turn these layers into one transaction.

3. State inventory before changing anything

4. What the Kubernetes plugin actually provisions

The Kubernetes plugin creates a pod for an agent and removes it after the build according to retention policy. A Pipeline-scoped podTemplate { ... } creates an ephemeral template that exists only while that block executes; new Pipelines should normally use this rather than creating a large collection of globally shared static templates.

If the template omits an explicit label, the plugin generates one and exposes it as POD_LABEL. That generated label ties the node(POD_LABEL) request to the ephemeral template without inventing a global label namespace.

5. Two identities: provisioner versus build pod

The Jenkins Kubernetes cloud needs authority to create/watch/delete agent pods. The agent pod itself also has a Kubernetes ServiceAccount, but ordinary build code usually does not need Kubernetes API access. Treat these as separate principals.

For a build pod with no API requirement, use a dedicated ServiceAccount and set automountServiceAccountToken: false. If a later stage really needs API access, grant a narrowly-scoped Role to a purpose-specific ServiceAccount rather than reusing the Jenkins provisioner identity.

6. Workspace lifetime is a design decision

The plugin's default workspace volume is an ephemeral empty directory. Kubernetes emptyDir exists for the lifetime of a pod: it survives a container restart, but when the pod is removed from the node, that data is deleted. This is an excellent default for disposable CI workspaces when outputs are archived or published before teardown.

7. Requests, limits and queue demand are different capacity signals

Jenkins decides that an agent is needed; Kubernetes decides whether its pod can be scheduled. CPU/memory requests participate in scheduler accounting. If no node can satisfy them, the pod stays Pending. Limits constrain runtime behavior but do not guarantee scheduling by themselves.

In production, a node autoscaler may react to unschedulable pods and provision nodes. That is a separate control loop. Jenkins should still have cloud/template caps, the namespace should still have quotas, and pod requests should still be realistic. “Autoscaling” is not permission for infinite agents.

8. Read-only inspection first

set -eu
kubectl config current-context
kubectl cluster-info
kubectl get namespace ch20-lab 2>/dev/null || true
kubectl -n ch20-lab get pods -o wide 2>/dev/null || true
kubectl -n ch20-lab get serviceaccounts,roles,rolebindings 2>/dev/null || true
kubectl -n ch20-lab get resourcequota,limitrange 2>/dev/null || true

Inside Jenkins, also record the Kubernetes plugin version, cloud name, build number/URL, queue cause, source SHA, and node/label. Inspection before mutation prevents “fixing” the wrong cluster, namespace, cloud, or job.

9. Pod evidence: name is not enough

kubectl -n ch20-lab get pod "$POD" \
  -o jsonpath='name={.metadata.name}{"\\n"}uid={.metadata.uid}{"\\n"}node={.spec.nodeName}{"\\n"}sa={.spec.serviceAccountName}{"\\n"}'
kubectl -n ch20-lab get pod "$POD" -o jsonpath='{range .status.containerStatuses[*]}{.name}{" imageID="}{.imageID}{"\\n"}{end}'
kubectl -n ch20-lab describe pod "$POD"

The UID and container ImageIDs are especially useful after a retry creates a replacement pod. Jenkins build identity can remain constant while Kubernetes execution identity changes.

10. Trust boundary: short-lived does not mean low-privilege

An ephemeral pod can still be highly privileged during its short lifetime. Dangerous examples include a broad ServiceAccount, privileged containers, host networking, or host filesystem mounts. Ephemerality improves cleanup; it does not neutralize authority.

Use namespace separation, dedicated identities, Pod Security controls, network policy where available, non-root containers, bounded resources, reviewed images, and folder/cloud restrictions appropriate to the trust of the source being built.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

• Jenkins LTS changelog — baseline Jenkins 2.568.3 LTS, released 2026-09-02 and tested with Java 21 and 25; labs use Java 21 for Jenkins components.
• Jenkins Java Support Policy — current Jenkins system components, including agents, require a supported JVM; this chapter uses Java 21.
• Jenkins Kubernetes plugin — reviewed version 4547.v52f3080db_8cd, requires Jenkins 2.516.3, and has no current security advisory shown by the plugin health page at the chapter timestamp.
• Kubernetes plugin Pipeline steps — podTemplate, container, pod retention, workspace volumes, and related fields.
• kind v0.33.0 — local lab pin; its Kubernetes v1.37.0 node image is kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5.
• Kubernetes RBAC good practices and ServiceAccounts for Pods — least privilege, namespaced roles, dedicated service accounts, and token-automount guidance.
• Kubernetes volumes — emptyDir is pod-lifetime storage; persistent volume types have a different lifecycle.
• Resource management and Node autoscaling — scheduling uses requests; node autoscaling is a separate control loop from Jenkins agent provisioning.
• Jenkins inbound-agent image — reviewed agent tag 3391.va_37fa_a_305d6d-2-jdk21; record the actual architecture-specific digest/ImageID used by the pod.