Kubernetes Plugin, Pod Templates, Dynamic Kubernetes Agents, Volumes, Service Accounts, and Cluster Scaling: Guided Hands-On Workflow and Core Operations

1. Disposable lab topology

Use your disposable Jenkins controller from earlier chapters, Jenkins 2.568.3 LTS on Java 21, and Kubernetes plugin 4547.v52f3080db_8cd. Keep the built-in Jenkins node at zero executors. The cluster is local only and contains no production workloads.

The reproducible cluster pin is kind v0.33.0 with kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5. If your host architecture cannot use that image, choose an upstream kind-supported digest for your platform and record the change.

2. Create the local cluster and namespace

set -eu
kind version
kubectl version --client

cat > /tmp/ch20-kind.yaml <<'YAML'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ch20
nodes:
- role: control-plane
YAML

kind create cluster \
  --config /tmp/ch20-kind.yaml \
  --image kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
kubectl create namespace ch20-lab
kubectl get nodes -o wide

Do not point these commands at a shared or production context. Immediately confirm kubectl config current-context contains the intended kind cluster.

3. Create separate provisioner and build-pod identities

The first ServiceAccount represents the Jenkins Kubernetes cloud in this disposable namespace. The second is assigned to build pods and has no Kubernetes API permissions or token automount.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-provisioner
  namespace: ch20-lab
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: jenkins-provisioner
  namespace: ch20-lab
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","get"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["get","list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-provisioner
  namespace: ch20-lab
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins-provisioner
subjects:
- kind: ServiceAccount
  name: jenkins-provisioner
  namespace: ch20-lab
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ch20-agent
  namespace: ch20-lab
automountServiceAccountToken: false

Apply the manifest, then verify with kubectl auth can-i. The plugin's upstream sample permissions can evolve; if the current plugin reports a specific namespaced denial, add only the exact required verb/resource after preserving the denial. Do not jump to cluster-admin.

4. Build a short-lived local cloud credential

For this disposable lab, create a one-hour token for jenkins-provisioner and configure the Jenkins Kubernetes cloud with the local API server, CA, namespace ch20-lab, and a Jenkins credential that contains only this lab token. Record the credential ID, not the token value.

set -eu
kubectl -n ch20-lab create token jenkins-provisioner --duration=1h > /tmp/ch20-token
chmod 600 /tmp/ch20-token
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'; echo
kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | wc -c

If your Jenkins controller is outside the kind network, use the plugin's WebSocket agent mode when that simplifies reverse-proxy/firewall traversal. Keep TLS verification enabled.

5. Configure the bounded Jenkins Kubernetes cloud

Create cloud ch20-kind with namespace ch20-lab. Set a small container/pod cap suitable for a one-node local cluster (for example 3), restrict cloud use to the lab folder where supported, and test the Kubernetes API connection before running a job.

The cloud/provisioner credential controls pod lifecycle. The ch20-agent ServiceAccount below controls what code inside the pod can do against the Kubernetes API. Keep those roles separate.

6. Run an ephemeral pod with bounded resources

podTemplate(
  cloud: 'ch20-kind',
  namespace: 'ch20-lab',
  serviceAccount: 'ch20-agent',
  podRetention: never(),
  yaml: '''
apiVersion: v1
kind: Pod
spec:
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
  containers:
  - name: jnlp
    image: jenkins/inbound-agent:3391.va_37fa_a_305d6d-2-jdk21
    resources:
      requests: { cpu: "100m", memory: "192Mi" }
      limits:   { cpu: "500m", memory: "512Mi" }
  volumes:
  - name: evidence
    emptyDir: {}
''') {
  node(POD_LABEL) {
    stage('Identity') {
      sh '''
        set -eu
        printf 'job=%s build=%s node=%s workspace=%s\\n' \
          "$JOB_NAME" "$BUILD_NUMBER" "$NODE_NAME" "$WORKSPACE"
        printf 'pod=%s\\n' "$(hostname)" | tee pod-name.txt
        java -version 2>&1 | tee java.txt
      '''
      archiveArtifacts artifacts: 'pod-name.txt,java.txt', fingerprint: true
    }
  }
}

The version tag pins the agent software line, while Kubernetes reports the actual imageID pulled for your architecture. Capture that ImageID from the pod before teardown for immutable evidence.

7. Inspect the live pod from the cluster side

kubectl -n ch20-lab get pods -o wide
POD=$(kubectl -n ch20-lab get pods --sort-by=.metadata.creationTimestamp \
  -o jsonpath='{.items[-1:].metadata.name}')
kubectl -n ch20-lab get pod "$POD" -o jsonpath='uid={.metadata.uid}{"\\n"}sa={.spec.serviceAccountName}{"\\n"}node={.spec.nodeName}{"\\n"}'
kubectl -n ch20-lab get pod "$POD" -o jsonpath='{range .status.containerStatuses[*]}{.name}{" imageID="}{.imageID}{"\\n"}{end}'
kubectl -n ch20-lab top pod "$POD" 2>/dev/null || true

Also run kubectl -n ch20-lab auth can-i get pods --as=system:serviceaccount:ch20-lab:ch20-agent. The expected result is no for this build identity.

8. Simulate pod loss with a retry-safe workload

Use only synthetic, repeatable work inside the retry. The plugin documents a Kubernetes-agent retry condition for pod-loss infrastructure failures:

podTemplate(cloud: 'ch20-kind', namespace: 'ch20-lab', serviceAccount: 'ch20-agent', yaml: readTrusted('ci/ch20-pod.yaml')) {
  retry(count: 2, conditions: [kubernetesAgent(), nonresumable()]) {
    node(POD_LABEL) {
      stage('Attempt') {
        sh '''
          set -eu
          POD=$(hostname)
          printf 'pod=%s build=%s source=%s\\n' "$POD" "$BUILD_NUMBER" "${GIT_COMMIT:-synthetic}" \
            | tee "attempt-${POD}.txt"
        '''
        archiveArtifacts artifacts: 'attempt-*.txt', fingerprint: true
        echo 'From a separate terminal, delete this disposable agent pod during the following wait.'
        sleep 90
        sh 'echo survived-or-reprovisioned'
      }
    }
  }
}

From another terminal, identify the current pod, preserve kubectl describe and events, then delete only that pod:

kubectl -n ch20-lab get pods -o wide
kubectl -n ch20-lab describe pod <exact-agent-pod> > /tmp/ch20-before-delete.txt
kubectl -n ch20-lab delete pod <exact-agent-pod> --wait=false

A qualifying infrastructure failure can cause the node block to run on a fresh pod. Do not wrap deployments, publication, or other irreversible side effects in this retry without idempotency/guard design.

9. Compare first and replacement execution identities

After recovery, compare the archived attempt-*.txt files and Jenkins console log. The Jenkins build number may be unchanged while pod names/UIDs differ. Verify the first pod is gone and the new attempt has its own Kubernetes identity.

kubectl -n ch20-lab get pods -o wide
kubectl -n ch20-lab get events --sort-by=.lastTimestamp | tail -40

This is the central recovery lesson: Jenkins durable build state and Kubernetes ephemeral execution state can diverge and reconnect. Preserve both identities.

10. Layer-selection challenge

Jenkins shows “waiting for next available executor,” the Kubernetes cloud has already created a pod, and kubectl describe pod says 0/1 nodes are available: Insufficient memory. Should you change the Jenkins label, restart Remoting, or inspect Kubernetes resource/capacity state?

Correct layer: Kubernetes scheduling/capacity. The pod exists but cannot be placed. Inspect requests, node allocatable resources, namespace quota, taints/affinity, and—if production uses one—node-autoscaler constraints.

11. Cleanup and token revocation

set -eu
kubectl -n ch20-lab get pods,pvc,serviceaccounts,roles,rolebindings
# Remove the Jenkins ch20-kind cloud/credential from the disposable controller first.
kubectl delete namespace ch20-lab
kind delete cluster --name ch20
rm -f /tmp/ch20-token /tmp/ch20-kind.yaml /tmp/ch20-before-delete.txt

Verify kind get clusters no longer lists ch20. Cleanup should target the named lab cluster/namespace only; never use broad deletion against an unknown context.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

• Jenkins LTS changelog — baseline Jenkins 2.568.3 LTS, released 2026-09-02 and tested with Java 21 and 25; labs use Java 21 for Jenkins components.
• Jenkins Java Support Policy — current Jenkins system components, including agents, require a supported JVM; this chapter uses Java 21.
• Jenkins Kubernetes plugin — reviewed version 4547.v52f3080db_8cd, requires Jenkins 2.516.3, and has no current security advisory shown by the plugin health page at the chapter timestamp.
• Kubernetes plugin Pipeline steps — podTemplate, container, pod retention, workspace volumes, and related fields.
• kind v0.33.0 — local lab pin; its Kubernetes v1.37.0 node image is kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5.
• Kubernetes RBAC good practices and ServiceAccounts for Pods — least privilege, namespaced roles, dedicated service accounts, and token-automount guidance.
• Kubernetes volumes — emptyDir is pod-lifetime storage; persistent volume types have a different lifecycle.
• Resource management and Node autoscaling — scheduling uses requests; node autoscaling is a separate control loop from Jenkins agent provisioning.
• Jenkins inbound-agent image — reviewed agent tag 3391.va_37fa_a_305d6d-2-jdk21; record the actual architecture-specific digest/ImageID used by the pod.