Checkpoint Lab — Secret Masking Limits, Credentials Scope, External Secret Providers, Vault Integration, and Rotation Patterns

1. Scenario and success criteria

You operate vault-lab/rotation-checkpoint. The application credential lives only in local disposable Vault KV v2. Jenkins stores two folder-scoped AppRole bootstrap identifiers, uses them on a trusted vault-lab agent to mint a five-minute token, reads the current value, performs a local non-logging validation, revokes the token, and archives only safe metadata.

Success requires two builds: build A records KV version 1; an authorized setup operator rotates the provider to version 2; build B records KV version 2 without changing the application secret in Jenkins. Both builds preserve immutable build/source identities and provider audit evidence.

2. Pinned assumptions

3. Preflight and exact disposable resource guard

• Jenkins folder must be exactly vault-lab; stop if it already contains non-training work.
• Agent label must be vault-lab and point to the same disposable host as loopback Vault.
• Vault mount must be jenkins-lab/; auth role jenkins-lab; policy jenkins-lab-read.
• Jenkins credential IDs must be vault-lab-role-id and vault-lab-secret-id; no application api_key credential exists in Jenkins.
• Vault audit file must be exactly /tmp/jenkins-vault-audit.json.
• Only fake values LAB_PROVIDER_VALUE_V1_ONLY and LAB_PROVIDER_VALUE_V2_ONLY may be used.

4. Make predictions before execution

5. Provider setup

Use the Lesson 2 setup exactly: Vault 2.1.0 dev mode bound to 127.0.0.1:8200, KV-v2 mount jenkins-lab, AppRole enabled, policy limited to jenkins-lab/data/app, jenkins-lab/metadata/app, token self-lookup and self-revoke, role token TTL 5 minutes/max 10 minutes, SecretID TTL 30 minutes, and file audit enabled.

vault kv put jenkins-lab/app api_key='LAB_PROVIDER_VALUE_V1_ONLY'
vault kv metadata get -format=json jenkins-lab/app | jq '.data.current_version'

6. Checkpoint Pipeline

pipeline {
  agent { label 'vault-lab' }
  options { skipDefaultCheckout(true); timestamps(); disableConcurrentBuilds() }
  stages {
    stage('Preflight evidence') {
      steps {
        sh '''
          set -eu
          rm -rf evidence && mkdir evidence
          printf 'job=%s\nbuild=%s\nurl=%s\nnode=%s\nworkspace=%s\n' \
            "$JOB_NAME" "$BUILD_NUMBER" "$BUILD_URL" "$NODE_NAME" "$WORKSPACE" \
            > evidence/jenkins.txt
          vault version > evidence/vault-cli.txt
        '''
      }
    }
    stage('Provider fetch') {
      steps {
        withCredentials([
          string(credentialsId: 'vault-lab-role-id', variable: 'VAULT_ROLE_ID'),
          string(credentialsId: 'vault-lab-secret-id', variable: 'VAULT_SECRET_ID')
        ]) {
          sh '''
            set -eu; set +x; umask 077
            export VAULT_ADDR='http://127.0.0.1:8200'
            TOKEN_FILE=$(mktemp)
            trap 'rm -f "$TOKEN_FILE"' EXIT
            vault write -format=json auth/approle/login \
              role_id="$VAULT_ROLE_ID" secret_id="$VAULT_SECRET_ID" \
              | jq -r '.auth.client_token' > "$TOKEN_FILE"
            chmod 600 "$TOKEN_FILE"
            export VAULT_TOKEN="$(cat "$TOKEN_FILE")"

            VERSION=$(vault kv metadata get -format=json jenkins-lab/app | jq -r '.data.current_version')
            TTL=$(vault token lookup -format=json | jq -r '.data.ttl')
            APP_SECRET=$(vault kv get -field=api_key jenkins-lab/app)
            test -n "$APP_SECRET"
            printf 'provider=vault\nauth=approle\nmount=jenkins-lab\npath=app\nkv-version=%s\ntoken-ttl-seconds=%s\nlocal-consumer-ok=true\n' \
              "$VERSION" "$TTL" > evidence/provider.txt
            unset APP_SECRET
            vault token revoke -self >/dev/null
            unset VAULT_TOKEN
          '''
        }
      }
    }
    stage('Evidence safety gate') {
      steps {
        sh '''
          set -eu
          ! grep -R -n -E 'LAB_PROVIDER_VALUE_V[12]_ONLY|hvs\.' evidence
        '''
        archiveArtifacts artifacts: 'evidence/*.txt', fingerprint: true
      }
    }
  }
  post {
    always {
      sh 'rm -f .vault-token 2>/dev/null || true'
    }
  }
}

7. Run build A and freeze its evidence

• Record build number/URL/cause and Jenkinsfile source SHA if SCM-backed.
• Verify kv-version=1.
• Verify token TTL is bounded and no token/value appears in evidence.
• Extract a reviewed provider-audit summary containing only timestamp, request operation/path, and error/success context.
• Do not edit build A after rotation; it is historical evidence.

8. Rotate from version 1 to version 2

In the authorized setup terminal:

vault kv put jenkins-lab/app api_key='LAB_PROVIDER_VALUE_V2_ONLY'
vault kv metadata get -format=json jenkins-lab/app \
  | jq '{current_version:.data.current_version, updated_time:.data.updated_time}'

Do not update the Jenkins application secret—there is none. Do not expose either value to prove rotation. Provider version metadata is sufficient.

9. Run build B and compare

• Build B must have a new Jenkins build number and new provider authentication event.
• Its evidence must report kv-version=2.
• Build A must still report version 1.
• The two Jenkins credential IDs remain unchanged; provider application secret was never copied into Jenkins.
• Both builds revoke their own short-lived Vault token after successful use.

10. Create a value-free provider audit summary

Review the audit file locally. Extract only fields needed for this checkpoint; do not archive raw provider logs by default. A typical summary should show chronological request paths such as auth/approle/login, auth/token/lookup-self, jenkins-lab/metadata/app, jenkins-lab/data/app, and auth/token/revoke-self. If your Vault version structures JSON differently, adapt the read-only extraction rather than dumping the entire record.

11. Required evidence packet

• baseline.md: Jenkins/Java/plugin/Vault versions and assumption timestamp.
• source.md: job full name, Jenkinsfile/source SHA, build cause.
• build-a-jenkins.txt and build-b-jenkins.txt: build/node/workspace identity.
• build-a-provider.txt and build-b-provider.txt: auth method, path, KV version, TTL, consumer success.
• rotation.md: provider metadata showing transition 1 → 2 and operator/time.
• audit-summary.md: reviewed provider request paths/times/results with no token/secret value.
• security-review.md: confirmation that no secret value/token/private bootstrap secret is in logs/artifacts/workspace evidence.
• assumptions-limitations.md: dev-mode loopback simulation, KV is versioned static data rather than a dynamic leased secret, and production workload identity/TLS/HA are outside the mandatory lab.

12. Cleanup and rollback

1. Archive/download only the reviewed evidence packet.
2. Delete the disposable Jenkins vault-lab folder only after confirming it contains no unrelated work.
3. Terminate the Vault dev server; its in-memory provider state disappears.
4. Delete /tmp/jenkins-vault-audit.json and /tmp/jenkins-lab-read.hcl after evidence review.
5. Remove the disposable agent/controller resources used only for the lab.
6. In a real incident or production rotation, never substitute “stop the server” for provider-supported revocation, rotation, backup, audit, and recovery procedures.

13. What Chapter 16 adds to the production operating model

You can now treat secret handling as a lifecycle across independent systems: Jenkins authorization and source trust, provider authentication and authorization, token/lease lifetime, secret version/rotation, trusted agent use, masking behavior, external side effects, revocation, and audit evidence. A successful build is not proof of safe secret handling; a masked console is not proof of containment.

Chapter 17 moves from identity/secret trust into execution capacity: nodes, agents, labels, executors, workspaces, offline causes, and agent-capacity design.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

• Jenkins LTS changelog and Java support policy — lab baseline Jenkins 2.568.3 LTS, Java 21; this LTS line is tested with Java 21 and 25.
• Jenkinsfile — Handling credentials and Credentials Binding step reference — bounded credential binding and masking caveats.
• Credentials Binding plugin — baseline 728.v902a_273b_8947; masking is intended to reduce accidental disclosure, not prevent a build from exfiltrating a value it can read.
• Credentials plugin — baseline 1511.v2e3cb_0008ef0.
• HashiCorp Vault Jenkins plugin — optional integration baseline 384.vda_86ec66c537; review current security advisories before installation or upgrade.
• Vault dev server mode — explicitly disposable/insecure development mode; never a production configuration.
• Vault AppRole auth, tokens and TTL, and leases, renewal, and revocation.
• Vault KV v2 — versioned static secret data. KV values are versioned but are not dynamic leased credentials.
• Vault audit devices — provider-side request evidence without intentionally logging cleartext secrets.
• Vault release notes — lab tool baseline Vault 2.1.0, released 2026-09-01.