Compliance Pipelines, Pipeline Execution Policies, Governance, Audit Evidence, and Enterprise Controls: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first diagnostic sequence

1. Preserve pipeline/job IDs and first-failure evidence.
2. Confirm pipeline source, ref, SHA, and project configuration.
3. Confirm applicable policy project/version/scope and effective configuration/job graph.
4. Confirm workflow/job rule decisions and non-secret inputs.
5. Inspect job graph, queue, runner, executor, image, and tool versions.
6. Inspect failing script/tool/network evidence.
7. Inspect reports, artifacts, caches, and registry state.
8. Inspect deployment/environment/external target state if relevant.
9. Inspect policy/audit/exception state.
10. Apply the smallest correction and rerun only the smallest safe scope.

2. Failure: project tries to override a mandatory job

A project can define a job with the same visible name. Do not assume the project won or lost based on the label. Inspect pipeline compilation and the policy conflict strategy.

# Intentionally suspicious project-owned job
policy:governance-proof:
  stage: test
  script:
    - echo "project-owned lookalike"

3. Failure: policy creates duplicate scans

Preserve both job IDs and traces. Compare policy/project configuration origins. A common cause is that the project already includes a scanner and the policy injects another copy. The least destructive fix is usually to remove one ownership path or narrow rollout—not to disable scanning globally.

curl --fail --silent --show-error \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "$GITLAB_URL/api/v4/projects/$PROJECT_ID/pipelines/$PIPELINE_ID/jobs?per_page=100" \
  | jq '[.[] | {id,name,stage,status,runner:.runner.description}]'

4. Failure: exception has no owner or expiry

The pipeline may be technically valid while governance is invalid. A raw project exclusion is not a managed waiver.

{"project_id":456,"reason":"legacy service"}

Repair the governance object: immutable exception ID, affected control, owner, approver, reason/ticket, creation time, expiry, policy version, and closure. Then reconcile that record with the technical scope exclusion.

5. Failure: policy variable leaks a secret

Committing a credential to policy YAML or echoing it in a trace is a secret incident, not a normal policy bug.

# Broken on purpose — never copy this pattern.
variables:
  DEPLOY_TOKEN: "fake-example-do-not-copy"
policy:check:
  stage: .pipeline-policy-pre
  script:
    - echo "$DEPLOY_TOKEN"

Remove the value through your incident process, revoke/rotate any real credential, stop logging it, and use a protected/external secret path. Do not troubleshoot by printing variables or broadening token scope.

6. Failure: audit record cannot map to source version

An audit event can prove that governance state changed while still failing to identify which exact policy CI content governed pipeline 8421. Do not rewrite historical evidence to invent a missing relation. Record the limitation and fix future correlation.

{
  "evidence_schema": 1,
  "target_project": "acme-labs/ch36-app",
  "source_sha": "<target-commit-sha>",
  "pipeline_id": 8421,
  "policy_project": "acme-labs/security-policy-management",
  "policy_ref": "<policy-commit-or-protected-tag>",
  "policy_ci_ref": "ch36-policy-v1.0.0",
  "policy_job_id": 9912,
  "audit_event_id": "<event-id-if-available>",
  "limitations": []
}

7. Intentionally broken local example: preserve, classify, repair

Use Lesson 2’s simulator. Add the mandatory job name to a copy of the project model, capture the failure and exit status, then restore only that file.

cp project/project-model.json evidence/project-model.before-conflict.json
python - <<'PY'
import json
p='project/project-model.json'; d=json.load(open(p))
d['jobs'].append({'name':'policy:governance-proof','stage':'test'})
open(p,'w').write(json.dumps(d,indent=2)+'\n')
PY
set +e
python tools/compile_effective_pipeline.py \
  --project project/project-model.json --policy policy/policy-model.json \
  --exceptions governance/exceptions.json --out evidence/first-failure.json \
  > evidence/first-failure.trace 2>&1
RC=$?
set -e
printf '%s\n' "$RC" > evidence/first-failure.exit
cat evidence/first-failure.json

cp evidence/project-model.before-conflict.json project/project-model.json
python tools/compile_effective_pipeline.py \
  --project project/project-model.json --policy policy/policy-model.json \
  --exceptions governance/exceptions.json --out evidence/repaired.json

The broken evidence remains intact. The repair changes no token, runner, policy scope, or unrelated job.

8. Causal failure map

9. Security and performance consequences of policy mistakes

Central jobs consume runner capacity and critical-path time. Duplicate scans multiply both. A failing pre-policy job can block later stages under current execution semantics. Measure queue time, policy-job duration, failure rate, runner usage, and in-scope project count before broad rollout.

Off-limits shortcuts: broad PATs, printing tokens/variables, TLS disablement, mutable unreviewed dependencies, untrusted code on privileged runners, blind retries, delete/recreate fixes that erase evidence, unbounded autoscaling, policy bypass, or rebuilding a release artifact merely to make governance evidence look clean.

10. Diagnostic summary

Classify first, mutate second. The same visible red pipeline can originate in config compilation, policy scope, runner routing, shell/tool/network behavior, identity/variables, evidence ingestion, deployment state, API limits, or governance data. The smallest causal fix is safer and easier to audit.

Knowledge check

Official references and version notes

Further reading — current official GitLab sources

Behavior and tier assumptions in this chapter were checked against the current documentation on 2026-09-13. Re-check these pages before applying the optional enterprise path because policy schemas and product tiers evolve.

• GitLab Docs — Pipeline execution policies
• GitLab Docs — Security policy projects
• GitLab Docs — Policy enforcement and separation of duties
• GitLab Docs — Compliance pipelines (deprecated)
• GitLab Docs — CI/CD variables and precedence
• GitLab Docs — CI Lint API
• GitLab Docs — Audit events
• GitLab Docs — Audit events API
• GitLab Docs — Compliance frameworks
• GitLab Docs — Deprecations and removals

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.