Secret Masking Limits, Credentials Scope, External Secret Providers, Vault Integration, and Rotation Patterns: Guided Hands-On Workflow and Core Operations

1. Lab topology and preflight

Use a disposable Jenkins controller at 2.568.3 LTS, Java 21, one trusted disposable agent labeled vault-lab, Vault CLI/server 2.1.0, and jq. The Vault server and Jenkins agent run on the same disposable machine so Vault can remain bound to 127.0.0.1:8200.

vault version
jq --version
java -version
id
ss -lnt | grep ':8200' || true

2. Start Vault dev mode without giving Jenkins the root token

In a separate setup terminal on the trusted lab host, start Vault:

vault server -dev -dev-listen-address='127.0.0.1:8200'

Vault prints a disposable dev root token. In the setup terminal only, set VAULT_ADDR=http://127.0.0.1:8200 and VAULT_TOKEN to that generated dev root token. Do not save the root token in Jenkins, shell history, SCM, lesson evidence, or chat.

export VAULT_ADDR='http://127.0.0.1:8200'
# Set VAULT_TOKEN interactively from the dev-server output in this disposable setup shell.
vault status

3. Create a bounded provider policy and AppRole

Create a dedicated KV-v2 mount, enable AppRole, and allow read-only access to one lab secret plus its metadata. The setup identity performs these administrative mutations; Jenkins will not.

vault secrets enable -path=jenkins-lab -version=2 kv
vault auth enable approle

cat >/tmp/jenkins-lab-read.hcl <<'EOF'
path "jenkins-lab/data/app" {
  capabilities = ["read"]
}
path "jenkins-lab/metadata/app" {
  capabilities = ["read"]
}
path "auth/token/lookup-self" {
  capabilities = ["read"]
}
path "auth/token/revoke-self" {
  capabilities = ["update"]
}
EOF
vault policy write jenkins-lab-read /tmp/jenkins-lab-read.hcl

vault write auth/approle/role/jenkins-lab \
  token_policies='jenkins-lab-read' \
  token_ttl='5m' token_max_ttl='10m' \
  secret_id_ttl='30m' secret_id_num_uses=20

vault kv put jenkins-lab/app api_key='LAB_PROVIDER_VALUE_V1_ONLY'

These values are synthetic. The 30-minute SecretID is a bootstrap credential for the exercise; the actual per-build Vault token is five minutes and is explicitly revoked after use.

4. Enable disposable provider audit evidence

rm -f /tmp/jenkins-vault-audit.json
vault audit enable file file_path=/tmp/jenkins-vault-audit.json
vault audit list

Vault audit devices protect many sensitive fields rather than storing cleartext values. The evidence packet will extract only request paths, operations, times, and success/failure context. Never archive the whole audit file without reviewing its sensitivity and retention policy.

5. Create Jenkins bootstrap credentials in a disposable folder

In Jenkins create folder vault-lab. Read the AppRole identifiers in the setup terminal:

vault read -field=role_id auth/approle/role/jenkins-lab/role-id
vault write -field=secret_id -f auth/approle/role/jenkins-lab/secret-id

Add the returned values as folder credentials named vault-lab-role-id and vault-lab-secret-id. Treat the SecretID as sensitive. The role ID is commonly an identifier rather than a password, but keeping both inside the disposable folder makes the lab boundary unambiguous. Do not create a Jenkins credential containing the application api_key.

6. Fetch on demand with a short-lived build token

Create vault-lab/fetch-demo as an SCM-backed Pipeline or a disposable Pipeline job. The shell receives only the AppRole bootstrap values from Jenkins, exchanges them for a five-minute Vault token, reads the current KV value, performs a local non-logging check, records metadata, and revokes the token.

pipeline {
  agent { label 'vault-lab' }
  options { skipDefaultCheckout(true); timestamps(); disableConcurrentBuilds() }
  stages {
    stage('Fetch and use') {
      steps {
        withCredentials([
          string(credentialsId: 'vault-lab-role-id', variable: 'VAULT_ROLE_ID'),
          string(credentialsId: 'vault-lab-secret-id', variable: 'VAULT_SECRET_ID')
        ]) {
          sh '''
            set -eu; set +x; umask 077
            export VAULT_ADDR='http://127.0.0.1:8200'
            rm -rf evidence && mkdir evidence
            TOKEN_FILE=$(mktemp)
            trap 'rm -f "$TOKEN_FILE"' EXIT

            vault write -format=json auth/approle/login \
              role_id="$VAULT_ROLE_ID" secret_id="$VAULT_SECRET_ID" \
              | jq -r '.auth.client_token' > "$TOKEN_FILE"
            chmod 600 "$TOKEN_FILE"
            export VAULT_TOKEN="$(cat "$TOKEN_FILE")"

            VERSION=$(vault kv metadata get -format=json jenkins-lab/app \
              | jq -r '.data.current_version')
            TTL=$(vault token lookup -format=json | jq -r '.data.ttl')
            API_KEY=$(vault kv get -field=api_key jenkins-lab/app)
            test -n "$API_KEY"

            printf 'job=%s\nbuild=%s\nnode=%s\nworkspace=%s\n' \
              "$JOB_NAME" "$BUILD_NUMBER" "$NODE_NAME" "$WORKSPACE" \
              > evidence/jenkins.txt
            printf 'provider=vault\nmount=jenkins-lab\npath=app\nkv-version=%s\ntoken-ttl-seconds=%s\nconsumer-ok=true\n' \
              "$VERSION" "$TTL" > evidence/provider.txt

            unset API_KEY
            vault token revoke -self >/dev/null
            unset VAULT_TOKEN
          '''
        }
      }
    }
    stage('Retain safe evidence') {
      steps {
        archiveArtifacts artifacts: 'evidence/*.txt', fingerprint: true
      }
    }
  }
}

7. Verify build 1 before rotating anything

• Build cause, number, URL, source revision, agent name/label/workspace are recorded.
• provider.txt reports kv-version=1 and a TTL no greater than the configured role maximum.
• No Jenkins credential contains LAB_PROVIDER_VALUE_V1_ONLY.
• The Vault audit file contains requests for AppRole login, token self-lookup, KV metadata/data, and token self-revocation.
• No build artifact, log, or workspace evidence contains the application value or Vault client token.

8. Rotate provider state, not Jenkins application-secret state

In the setup terminal—not Jenkins—write a new KV version:

vault kv put jenkins-lab/app api_key='LAB_PROVIDER_VALUE_V2_ONLY'
vault kv metadata get -format=json jenkins-lab/app | jq '.data.current_version'

Predict before the next build: provider current_version becomes 2; Jenkins job configuration and credential IDs do not change; build 2 authenticates again, receives a new short-lived token, and records kv-version=2.

9. Run build 2 and compare evidence

Trigger the same job again. Compare only safe evidence:

# Example after downloading the two evidence files locally
printf 'build1: '; grep '^kv-version=' build1-provider.txt
printf 'build2: '; grep '^kv-version=' build2-provider.txt

Build 1 remains historical evidence of version 1. Build 2 proves fetch-on-demand observed version 2. Rotation did not require copying the application value into Jenkins.

10. Challenge: choose the correct layer

Build 3 fails with permission denied reading jenkins-lab/data/app, while AppRole login succeeds. Should you broaden Jenkins folder permissions, increase agent executors, or inspect Vault policy? The evidence places authentication success before authorization failure, so inspect the provider policy/path/capability first. Do not grant a root token just to make the build green.

11. Cleanup

1. Preserve only reviewed value-free evidence.
2. Delete the disposable Jenkins folder credentials/jobs.
3. In the setup shell, disable the lab audit device or simply terminate the dev server after capturing the required audit summary.
4. Stop the Vault dev server and delete /tmp/jenkins-vault-audit.json and /tmp/jenkins-lab-read.hcl.
5. Because dev mode is in-memory, stopping it destroys provider state; do not mistake this for a production deletion procedure.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

• Jenkins LTS changelog and Java support policy — lab baseline Jenkins 2.568.3 LTS, Java 21; this LTS line is tested with Java 21 and 25.
• Jenkinsfile — Handling credentials and Credentials Binding step reference — bounded credential binding and masking caveats.
• Credentials Binding plugin — baseline 728.v902a_273b_8947; masking is intended to reduce accidental disclosure, not prevent a build from exfiltrating a value it can read.
• Credentials plugin — baseline 1511.v2e3cb_0008ef0.
• HashiCorp Vault Jenkins plugin — optional integration baseline 384.vda_86ec66c537; review current security advisories before installation or upgrade.
• Vault dev server mode — explicitly disposable/insecure development mode; never a production configuration.
• Vault AppRole auth, tokens and TTL, and leases, renewal, and revocation.
• Vault KV v2 — versioned static secret data. KV values are versioned but are not dynamic leased credentials.
• Vault audit devices — provider-side request evidence without intentionally logging cleartext secrets.
• Vault release notes — lab tool baseline Vault 2.1.0, released 2026-09-01.