Production Capstone: Design, Automate, Secure, Scale, Upgrade, and Recover an Enterprise Jenkins Platform: Implementation and Automation Build-Out

1. Disposable scenario and preflight

Use one disposable Jenkins controller and at least one disposable/local agent labeled capstone-lab. Keep the controller's built-in executors at zero. The application is synthetic and the “artifact repository” is a guarded local directory so the mandatory exercise needs no cloud account, commercial product or external SCM organization.

2. Build a platform repository

Separating platform, application, library, evidence and runbook concerns makes configuration ownership explicit. It also lets code review show whether a change touches controller configuration, job generation, Pipeline interfaces or operations.

capstone-platform/
├── platform-contract.yaml
├── plugins.txt
├── casc/jenkins.yaml
├── jobs/seed.groovy
├── library/
│   ├── vars/ciBuild.groovy
│   └── resources/
├── app/
│   ├── Jenkinsfile
│   ├── src/
│   └── tests/
├── repo-sim/releases/
├── evidence/
├── runbooks/
│   ├── backup-restore.md
│   └── upgrade.md
└── exceptions.yaml

3. Pin the reviewed plugin baseline

The manifest is not a guarantee of compatibility; it is the input to compatibility testing. Resolve dependencies in the disposable environment and archive the resulting effective plugin inventory. The September 16 advisory makes the Script Security, Groovy Libraries and Multibranch minimums especially important in this dated lab.

configuration-as-code:2121.v86fe99d4b_b_a_b_
job-dsl:3732.v9a_c49a_61a_313
credentials:1511.v2e3cb_0008ef0
matrix-auth:3.3
workflow-aggregator:608.v67378e9d3db_1
workflow-cps:4383.v04fa_a_3d67b_d9
workflow-multibranch:842.v3a_b_59b_57b_e6e
script-security:1422.v06869826dd9b_
pipeline-groovy-lib:806.v408277b_33d1d

Do not blindly copy this manifest into a future controller. Re-check minimum-core requirements, advisories and transitive dependency changes first.

4. Apply controller configuration from code

This excerpt owns non-secret global state and disables routine controller execution. Authentication, matrix permissions and credentials should live in a reviewed environment-specific overlay because a realistic realm differs across organizations.

jenkins:
  systemMessage: "Disposable Jenkins production-capstone lab"
  numExecutors: 0
  mode: NORMAL
  labelString: "controller-only"
  markupFormatter:
    plainText
unclassified:
  location:
    url: "http://127.0.0.1:8080/"
# Authentication/authorization and credential provider settings are deliberately
# environment-specific. In this lab, define them in reviewed local JCasC overlays;
# never commit real password/token values to this repository.

Apply JCasC only to the disposable controller, capture startup/reload logs, and compare exported/observed state with the source commit. If another mechanism also edits the same setting manually, you have configuration ownership conflict.

5. Least-privilege access as a test, not a slogan

Use Matrix Authorization Strategy or another reviewed authorization mechanism in the lab. Define at least three synthetic identities: platform admin, developer, and read-only auditor. Verify the developer can read/build the intended folder but cannot administer Jenkins or read unrelated credentials. Verify the auditor can inspect permitted job/evidence state but cannot trigger or configure builds.

6. Generate the job definition

Job DSL demonstrates that job topology can be reproduced and reviewed. In a production platform, a seed job is itself privileged: changes to generated jobs can alter SCM, credentials, triggers and execution. Treat the seed repository and approvals accordingly.

folder('capstone') {
  description('Disposable production-capstone jobs')
}
pipelineJob('capstone/app') {
  definition {
    cps {
      script(readFileFromWorkspace('app/Jenkinsfile'))
      sandbox(true)
    }
  }
  logRotator { numToKeep(20) }
}

Run the seed only in the disposable namespace. Record the seed build URL and the exact source revision that created capstone/app.

7. Give teams a narrow shared-library interface

A shared library should reduce duplication without becoming an invisible superuser API. Keep the interface small, validate parameters, and version releases. Trusted libraries receive controller-effective privileges, so their maintainer and review boundary must be stricter than an ordinary application repository.

def call(Map cfg = [:]) {
    if (!cfg.artifactName) {
        error('artifactName is required')
    }
    sh(label: 'build synthetic artifact', script: "mkdir -p dist && printf '%s\n' '${env.GIT_COMMIT ?: 'synthetic-sha'}' > dist/${cfg.artifactName}")
    sh(label: 'digest artifact', script: "sha256sum dist/${cfg.artifactName} | tee evidence/artifact.sha256")
}

Record both application source SHA and library revision in build evidence. Never use a floating library branch for a release path unless policy explicitly accepts that mutability.

8. Build, test and publish one immutable artifact

The following mandatory Pipeline needs only a lab agent and local filesystem. It separates test-report ingestion from artifact identity and makes the local repository coordinate the SHA-256 digest itself. Promotion in later exercises copies the same bytes; it never reruns the build stage.

pipeline {
  agent { label 'capstone-lab' }
  options { timestamps(); disableConcurrentBuilds() }
  stages {
    stage('Identity') {
      steps {
        sh 'mkdir -p evidence dist repo-sim/releases'
        sh 'printf "build=%s\nurl=%s\n" "$BUILD_NUMBER" "$BUILD_URL" > evidence/build.txt'
        sh 'git rev-parse HEAD > evidence/source.sha || printf "synthetic-source\n" > evidence/source.sha'
      }
    }
    stage('Test evidence') {
      steps {
        sh 'printf "<testsuite tests=\"1\" failures=\"0\"><testcase classname=\"capstone\" name=\"smoke\"/></testsuite>\n" > evidence/junit.xml'
        junit 'evidence/junit.xml'
      }
    }
    stage('Build once') {
      steps {
        sh 'printf "capstone artifact from build %s\n" "$BUILD_NUMBER" > dist/app.txt'
        sh 'sha256sum dist/app.txt | tee evidence/artifact.sha256'
      }
    }
    stage('Publish immutable simulation') {
      steps {
        sh 'set -eu; d=$(cut -d" " -f1 evidence/artifact.sha256); p="repo-sim/releases/$d"; test ! -e "$p" || cmp -s dist/app.txt "$p"; test -e "$p" || cp dist/app.txt "$p"; cmp -s dist/app.txt "$p"'
      }
    }
  }
  post {
    always {
      archiveArtifacts artifacts: 'evidence/**,dist/**', fingerprint: true
    }
  }
}

9. Add an observability pack

At minimum capture controller uptime, queue length/wait, executor/agent state, build result/duration and external repository simulation latency. Metrics and Prometheus plugins are useful optional paths, but the lab can remain free/local by storing Jenkins JSON API snapshots and timestamped logs.

mkdir -p evidence/observability
curl -fsS "$JENKINS_URL/queue/api/json" > evidence/observability/queue.json
curl -fsS "$JENKINS_URL/computer/api/json" > evidence/observability/computers.json
printf '%s\n' "$(date -u +%FT%TZ)" > evidence/observability/captured-at.txt

10. Rehearse backup and recovery

Quiesce or stop the disposable controller according to the Chapter 36 runbook, capture the required JENKINS_HOME state and plugin/core/runtime inventory, protect the backup off-controller, and protect controller key material separately. Restore to an alternate path/port, not over the only working copy. Validate job metadata, configuration ownership, plugin baseline and a canary build.

11. Layer-selection challenge

A developer can run capstone/app, but a release Pipeline cannot access a folder-scoped credential after the job is moved to another folder. Which layer changed? The correct starting layer is credential scope/item hierarchy, not agent networking or controller heap. State the negative and positive authorization evidence you would preserve.

12. Verification and guarded cleanup

1. Confirm built-in executors remain zero and all work ran on capstone-lab.
2. Verify archived source/build/artifact identity and compare repository-sim bytes to the build artifact.
3. Record the effective plugin list and JCasC/Job DSL/library commits.
4. Run one restore canary before calling the backup usable.
5. Delete only the exact disposable controller, lab agent, synthetic repository and repo-sim namespace after evidence review.

Knowledge check

13. Summary

You now have a disposable platform assembled from versioned components and observable evidence. Lesson 3 evaluates whether its security, governance and reliability claims actually hold.

Official references and version notes

• Jenkins Java support policy
• Jenkins security advisories
• Jenkins upgrade guides
• Configuration as Code plugin
• Job DSL plugin
• Matrix Authorization Strategy plugin
• Pipeline Shared Libraries
• Backing up Jenkins