ID Tokens, OIDC Workload Identity, Cloud Federation, Vault Authentication, and Secretless Deployments: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first diagnostic sequence

1. Preserve pipeline ID, job ID, first failure status and provider request/correlation ID.
2. Confirm CI_PIPELINE_SOURCE, ref, CI_COMMIT_SHA and compiled configuration.
3. Confirm whether the job exists because of the intended rules.
4. Confirm id_tokens requested the intended audience—without printing the token.
5. Inspect safe selected claims or provider-side evaluated claims.
6. Inspect provider/Vault issuer, audience and bound-condition policy.
7. If exchange succeeds, inspect the returned role/scope/expiry metadata.
8. Only then inspect the external API/deployment action and read-back state.
9. Apply the smallest correction; rerun only the smallest safe scope.

2. Failure: wildcard trust allows any branch

Symptom: a feature branch can assume the production role. The pipeline may be “working,” but authorization is wrong.

{
  "effect": "Allow",
  "issuer": "https://gitlab.example.test",
  "subject_pattern": "project_path:academy/app:ref_type:branch:ref:*"
}

The cause is provider governance, not runner capacity or token generation. Repair by binding exact project identity plus production ref/protection/environment conditions supported by that provider. Preserve one pre-fix successful attacker-like exchange and one post-fix denial so the correction is provable.

3. Failure: wrong audience

Symptom: GitLab creates the job and token, but provider authentication returns 401/unauthorized or “no matching federated identity.”

Compare the audience requested in id_tokens with the provider's configured/expected audience. Do not change the issuer, disable validation, or reuse another service's audience just to make the exchange pass.

deploy:
  id_tokens:
    CLOUD_ID_TOKEN:
      aud: https://vault.example.com   # BROKEN for cloud STS
  script:
    - ./cloud-exchange "$CLOUD_ID_TOKEN"

Repair only the audience to the provider's exact expected value and re-run the exchange. Keep the original denial response.

4. Failure: token printed to log or artifact

Symptom: a troubleshooting step writes the raw JWT to console or stores it in evidence/token.txt.

Stop using the leaked assertion immediately. Because an ID token is short-lived, the exposure window is bounded, but do not assume zero impact. Cancel/stop unnecessary jobs, preserve metadata about the incident without copying the token again, and inspect provider audit logs for exchanges/actions during the token lifetime. Remove the logging code and prove future runs log only selected safe metadata.

5. Failure: provider role is administrator

Symptom: the OIDC trust is narrow, but the assumed role can mutate unrelated accounts/namespaces/resources. Authentication is correct; authorization scope is not.

Record the role/policy identifier and hash, then reduce permissions to the exact API/resource set required. Prove one intended action succeeds and one neighboring out-of-scope action is denied. Short credential lifetime does not compensate for excessive privileges.

6. Failure: deprecated implicit JWT assumption

Symptom: old configuration expects CI_JOB_JWT or CI_JOB_JWT_V2. Current GitLab removed these in 17.0.

The repair is explicit id_tokens with an audience that matches the relying party, followed by corresponding provider/Vault configuration. Do not emulate the old implicit token with a long-lived secret.

7. Failure: source project versus job project confusion

Merge-request/fork scenarios can make “project” ambiguous. Current tokens include source-project-oriented project_id/project_path and job-project-oriented job_project_id/job_project_path. If a trust policy authorizes the wrong one, a cross-project execution path can be denied unexpectedly—or worse, accepted too broadly.

Preserve the pipeline source and MR/project relationship, identify which project actually runs the job, then bind the provider to the correct claim for your trust invariant.

8. Failure: long timeout mistaken for authentication fix

An exchange may fail because the audience/subject/issuer is wrong. Increasing job timeout changes token lifetime; it does not fix those mismatches. Diagnose the provider rejection first. Keep timeouts aligned with real work duration, then configure provider credential duration separately.

9. Intentionally broken lab: broad trust, then least-destructive repair

Start from the Chapter 27 local simulator and temporarily set ref policy to *. Modify the verifier to interpret that wildcard. Run the attacker-like claim set and preserve the unexpected allow result as evidence/broken-wildcard-allow.json. Do not delete it after fixing.

Then restore exact main matching and rerun only the exchange. Expected: attacker-like claims are denied; the allowed claim set still succeeds. The external marker does not need to be recreated until authorization is proven.

10. Map symptoms to layers

11. Security-sensitive actions

• Changing cloud/Vault trust policy is authorization-policy mutation; review and record it.
• Changing provider role permissions can widen production blast radius.
• Changing OIDC issuer/JWKS settings affects every federated workload that relies on them.
• Changing protected environment/ref rules alters eligibility and identity claims.
• Revoking or deleting provider identities can break deployments; target exact disposable resources in labs.

12. Performance and availability

OIDC adds network round trips to discovery/JWKS/provider STS and sometimes secret retrieval. Cache provider metadata only according to provider guidance; never cache bearer tokens as generic CI cache artifacts. If the identity provider or Vault is unavailable, fail closed for privileged mutations rather than falling back automatically to a broad static secret.

Knowledge check

Official references and version notes

Documentation verification date: 2026-09-12. GitLab documents explicit id_tokens as available on Free, Premium, and Ultimate across GitLab.com, Self-Managed, and Dedicated. CI_JOB_JWT/CI_JOB_JWT_V2 were removed in GitLab 17.0; use explicit ID tokens instead. ID tokens are RS256-signed. Their expiry is the job timeout when one is specified, otherwise five minutes. Current custom claims include project/namespace identity, job identity, ref/ref protection, pipeline source, runner identity, source SHA, CI configuration identity, and—when the job declares an environment—environment name/protection/tier/action. GitLab recommends stable IDs such as project/namespace IDs in trust policy conditions where the target provider supports them. Native secrets:vault integration is Premium/Ultimate, while ID-token federation itself is Free. GitLab troubleshooting documentation mentions decoding tokens for diagnosis. This course deliberately prefers provider-side evaluation and synthetic/local claim fixtures so learners do not normalize printing bearer assertions.

• OIDC authentication using ID tokens — official reference.
• Connect to cloud services — official reference.
• CI/CD YAML — id_tokens — official reference.
• AWS OIDC tutorial — official reference.
• Azure OIDC tutorial — official reference.
• Google Cloud workload identity federation — official reference.
• Use HashiCorp Vault secrets — official reference.
• Vault authentication tutorial — official reference.
• External secrets in CI/CD — official reference.
• GitLab deprecations and removals — official reference.
• OpenID Connect Core 1.0 — official reference.
• RFC 7519 — JSON Web Token — official reference.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.