Kubernetes Plugin, Pod Templates, Dynamic Kubernetes Agents, Volumes, Service Accounts, and Cluster Scaling: Configuration, Design Choices, and Tradeoffs

1. Design from state ownership, not from “Kubernetes is scalable”

A Kubernetes agent is attractive because it is disposable, schedulable and declarative. Those properties do not automatically make it cheaper, faster or more secure. Start by asking which state must survive, who may run code, which tools/images are trusted, how fast feedback must start, and what cluster capacity is actually available.

2. Static agents versus dynamic pods

3. emptyDir, dynamic PVC, existing PVC, or external artifact storage

emptyDir fits the default ephemeral-agent mental model: pod deleted, workspace deleted. That is a feature when all durable evidence is already archived/published. A PVC can be useful for large caches or explicit persistence, but it creates a new lifecycle that must be owned.

The Kubernetes plugin exposes both pod volumes and workspaceVolume. Its dynamicPVC() is managed with the pod and is deleted with it, so do not equate “PVC” with “release storage.” If a release candidate matters after the build, publish it to Jenkins artifact storage or an external repository with exact build/source identity.

4. Single-container versus multi-container pods

A single agent/tool container gives the simplest filesystem, UID and resource model. A multi-container pod can put Maven, Node, scanners or databases in separate images while sharing pod networking and selected volumes. That flexibility is useful when each image has a clear owner/version.

Current plugin guidance notes that containers in the same pod should normally use compatible UIDs; different UIDs can cause workspace permission problems. Resource requests/limits are also container-specific, so “the pod has 1 GiB” is an incomplete description.

spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
  containers:
  - name: jnlp
    image: jenkins/inbound-agent:3391.va_37fa_a_305d6d-2-jdk21
    resources:
      requests: { cpu: "100m", memory: "192Mi" }
  - name: tools
    image: alpine:3.24.1
    command: ["cat"]
    tty: true
    resources:
      requests: { cpu: "50m", memory: "64Mi" }

5. Default versus dedicated ServiceAccount

Kubernetes assigns the namespace default ServiceAccount if no ServiceAccount is named. Even when that account currently has almost no RBAC, a dedicated identity is more reviewable and prevents unrelated policy changes from silently changing the authority of every build pod.

For ordinary build pods, prefer automountServiceAccountToken: false. If a stage needs Kubernetes API access, create a separate ServiceAccount and namespaced Role for that exact workflow. Do not reuse the Jenkins provisioner credential inside arbitrary build code.

6. Pod-template inheritance: reuse can also hide privilege

The plugin supports inheritFrom and YAML merge/override behavior. Inheritance is useful for a reviewed base template—non-root security context, standard agent image, resource defaults—but reviewers must be able to determine the final merged pod.

ServiceAccount and node selector overrides replace inherited values rather than partially merging them. Treat a child template that changes identity, host access, image source, or volumes as a security-sensitive change even if the Pipeline diff looks small.

7. Controller inside versus outside the cluster

If the controller and agents share cluster networking, the normal inbound-agent connection can use internal service discovery. If the controller is outside the cluster or behind an ingress/reverse proxy, the plugin supports WebSocket agent connections over HTTP(S), often avoiding a separate Jenkins agent TCP port.

WebSocket changes transport, not trust. Keep TLS validation and controller identity verification. Do not “fix” a certificate problem by globally disabling HTTPS verification.

8. Jenkins provisioning versus Kubernetes node autoscaling

A backlog can therefore multiply across layers. Bound all of them; otherwise a transient SCM outage or test flake can create many pods and potentially many nodes.

9. Worked scenario: PR tests and trusted release builds

Suppose pull requests need CPU-only tests with no deployment credentials, while signed release builds need a private registry and signing service. A sound design uses separate namespaces/clouds or strongly separated node pools/templates, restricts each Jenkins cloud to authorized folders, gives PR pods no Kubernetes API token, and keeps release identities out of untrusted source paths.

Observable justification: PR builds show the unprivileged ServiceAccount, restricted image set, modest requests and ephemeral workspace; release builds show a different template/identity and explicit artifact digest/signing evidence. The distinction is state, not a comment in the Jenkinsfile.

10. Rollback means restoring a reviewed execution contract

If a new pod template causes permission failures or Pending pods, rollback the template/configuration version and verify new pods use the prior ServiceAccount, images, resources and volume model. Existing build records remain historical evidence; do not delete them to make the rollback “clean.”

Knowledge check

Answer before revealing the explanation.

Official references and version notes

• Jenkins LTS changelog — baseline Jenkins 2.568.3 LTS, released 2026-09-02 and tested with Java 21 and 25; labs use Java 21 for Jenkins components.
• Jenkins Java Support Policy — current Jenkins system components, including agents, require a supported JVM; this chapter uses Java 21.
• Jenkins Kubernetes plugin — reviewed version 4547.v52f3080db_8cd, requires Jenkins 2.516.3, and has no current security advisory shown by the plugin health page at the chapter timestamp.
• Kubernetes plugin Pipeline steps — podTemplate, container, pod retention, workspace volumes, and related fields.
• kind v0.33.0 — local lab pin; its Kubernetes v1.37.0 node image is kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5.
• Kubernetes RBAC good practices and ServiceAccounts for Pods — least privilege, namespaced roles, dedicated service accounts, and token-automount guidance.
• Kubernetes volumes — emptyDir is pod-lifetime storage; persistent volume types have a different lifecycle.
• Resource management and Node autoscaling — scheduling uses requests; node autoscaling is a separate control loop from Jenkins agent provisioning.
• Jenkins inbound-agent image — reviewed agent tag 3391.va_37fa_a_305d6d-2-jdk21; record the actual architecture-specific digest/ImageID used by the pod.