Production Capstone: Design, Automate, Secure, Scale, Upgrade, and Recover an Enterprise Jenkins Platform: Requirements, Constraints, and Target Architecture

1. The production problem: Jenkins is a platform, not one job

A production Jenkins installation coordinates privileged configuration, untrusted or semi-trusted repository content, build workers, credentials, external APIs, artifacts and recovery state. A Pipeline can be syntactically correct while the platform remains unsafe: the controller might execute builds, a trusted library might float to an unreviewed branch, an agent image might drift, a release might be rebuilt during promotion, or the only backup might be untested.

The capstone therefore treats Jenkins as an operating system for delivery automation. Every important behavior needs an owner, a version or immutable identity, observable evidence, an authorization boundary, and a recovery path.

2. Target mental model

flowchart TD
  A["Requirements + threat model + SLOs"] --> B["Versioned controller / Java / plugins / JCasC"]
  B --> C["SCM onboarding + Jenkinsfiles + reviewed shared libraries"]
  C --> D["Trusted isolated agent pools"]
  D --> E["Scoped credentials / external secret provider"]
  E --> F["Test + security + SBOM + digest + provenance evidence"]
  F --> G["Immutable artifact / promotion / delivery"]
  G --> H["Metrics + logs + audit + queue telemetry"]
  H --> I["Backup + restore + upgrade + incident governance"]
  I --> J["Auditable production operating model"]
  

The arrows are causal. Requirements determine the configuration and trust baseline. The baseline governs how repositories become jobs and Pipelines. Pipelines allocate trusted execution contexts, bind narrow credentials, create independently verifiable evidence, publish one immutable artifact, and expose operational signals. Observability, backup, upgrades and incident playbooks then keep that path supportable over time.

3. Requirements become acceptance evidence

A requirement without evidence is only an intention. For example, “least privilege” becomes testable only when a named lab identity is denied an administrative action while still being allowed to perform its intended build action.

4. State contract before change

5. Read-only discovery first

Before applying JCasC, installing plugins or creating jobs, record what exists. The mandatory lab starts from a disposable controller, but the same discipline prevents configuration drift and accidental ownership conflicts in real environments.

# Read-only inventory examples; authenticate only with a lab account when required.
JENKINS_URL=http://127.0.0.1:8080
curl -fsS "$JENKINS_URL/api/json?tree=mode,numExecutors,quietingDown,nodeDescription"
curl -fsS "$JENKINS_URL/computer/api/json?tree=computer[displayName,offline,numExecutors,busyExecutors,offlineCauseReason]"
curl -fsS "$JENKINS_URL/queue/api/json?tree=items[id,why,blocked,stuck,inQueueSince,task[name,url]]"
java -version
# On the disposable controller host:
ls -1 "$JENKINS_HOME/plugins" | sed -n '1,40p'
find "$JENKINS_HOME/jobs" -maxdepth 3 -name config.xml -print | sed -n '1,30p'

Store the outputs with UTC timestamps. Do not export credentials or copy controller secrets into the general evidence packet.

6. Threat model and trust boundaries

The controller JVM and trusted Shared Libraries are privileged. Plugin code runs inside the controller. Agents execute repository-controlled build commands and should therefore be treated as disposable or at least isolated workers, not as extensions of controller trust. Pull requests from forks or other untrusted contributors must not gain secret-bearing credentials or access to privileged agent pools simply because a Jenkinsfile asks for them.

7. Create the platform contract

This YAML is a learning artifact, not a Jenkins-native schema. Its purpose is to make assumptions reviewable before implementation.

platform:
  core: "2.568.3"
  java: "21"
  built_in_executors: 0
  config_owner: "git-reviewed-jcasc"
trust:
  untrusted_prs_use_separate_agents: true
  shared_libraries_require_reviewed_refs: true
  credentials_are_folder_or_job_scoped: true
evidence:
  source_sha: required
  artifact_sha256: required
  sbom: required_for_release_candidate
  build_url: required
operations:
  queue_wait_slo_seconds: 120
  restore_test_required: true
  upgrade_requires_canary: true
  exceptions_require_owner_and_expiry: true

Commit the contract beside the JCasC, plugin manifest, Job DSL, library and runbooks. A later exception must identify which contract line it relaxes, who owns the risk, why it exists and when it expires.

8. Reference target architecture

1. Controller: Java 21, LTS baseline, zero built-in executors, reviewed plugins, JCasC-owned global configuration.
2. SCM onboarding: folders/multibranch jobs generated or configured from reviewed code; source SHA captured for every build.
3. Libraries: shared APIs versioned and reviewed; privileged libraries have a smaller maintainer set than ordinary application repositories.
4. Agents: trusted pools by workload class; dynamic/ephemeral is preferred where operationally practical, but isolation is still verified rather than assumed.
5. Evidence: unit/quality/security reports, artifact digest, SBOM/provenance metadata and build/source identity are retained separately from secrets.
6. Delivery: external artifact repository or faithful local simulation stores one immutable build output that is promoted without rebuilding.
7. Operations: queue/JVM/agent/build/external-service signals feed actionable SLOs; backup, upgrade and incident runbooks are tested.

9. Disposable architecture lab

Create capstone/platform-contract.yaml from the example, then add a one-page THREAT_MODEL.md and SLO.md. Do not change Jenkins yet. Review every requirement and name the evidence that Lesson 5 must contain.

Prediction 1: setting built-in executors to 0 will move routine execution to capstone-lab agents.
Prediction 2: an artifact digest will remain identical when promoted between lab repository paths.
Prediction 3: a restore drill will recover job/config metadata but still require separately protected controller key material and external dependencies.

10. Common capstone mistakes

• Starting with plugin installation instead of requirements and trust boundaries.
• Calling an agent “isolated” merely because it runs in a container.
• Treating JCasC as a complete backup of runtime/build/secret state.
• Calling an archived artifact a release repository.
• Using a green build as proof that authorization, traceability and recovery are correct.
• Tracking dashboards without defining an operator action or SLO.

Knowledge check

11. Summary

You now have a target architecture and evidence contract. Lesson 2 turns that contract into a disposable, code-managed Jenkins platform without hiding which layer each automation step changes.

Official references and version notes

• Jenkins Java support policy
• Jenkins security advisories
• Jenkins upgrade guides
• Configuration as Code plugin
• Job DSL plugin
• Matrix Authorization Strategy plugin
• Pipeline Shared Libraries
• Backing up Jenkins