Chapter 25Lesson 04~200 minutes

SAST, Dependency Scanning, Secret Detection, Container Scanning, DAST, IaC Scanning, and Security Gates: Diagnostics, Failure Modes, Security, and Performance

Diagnose false-green scans, stale/deprecated configuration, leaked synthetic-secret evidence, unsafe DAST targets, wrong-SHA reports, noisy gates, and scanner performance from preserved first-failure evidence.

DiagnosticsFalse greenDAST safetyReport provenanceRecovery

Learning objectives

  • Use an evidence-first sequence to distinguish scanner execution, report ingestion, policy, runner, identity, and target failures.
  • Diagnose a green scanner job whose findings are ignored without destroying the original evidence.
  • Repair deprecated/stale scanner configuration by validating the compiled configuration and current primary docs.
  • Respond safely to secret-detection and DAST mistakes without printing secrets or rerunning against production.
  • Reduce security-scan latency/noise without blind retries, disabled TLS, privileged shortcuts, or invisible policy bypass.

1. Evidence-first diagnostic sequence

Start by recording CI_PIPELINE_SOURCE and CI_COMMIT_SHA; these distinguish a correct scanner result on the wrong pipeline/source from a real scanner failure.

  1. Preserve pipeline/job IDs, source/ref/SHA, scanner report/checksum, job trace, and first observed policy result.
  2. Confirm merged/compiled CI configuration and effective workflow:rules/job rules.
  3. Confirm scanner/analyzer image and rules/config version and the exact input target.
  4. Inspect job graph, queue, runner/executor/image, and whether untrusted code received protected resources.
  5. Inspect scanner command, exit semantics, network/database availability, and report generation.
  6. Validate the report against its real schema/format and bind it to the source/image/URL it scanned.
  7. Inspect the gate/policy and any exception record separately from the scanner result.
  8. For DAST/container/cloud inputs, inspect external target state and authorization.
  9. Apply the smallest correction and rerun only the necessary disposable scope.

2. Failure mode: scan is green but findings are ignored

Many scanners use exit zero to mean “the scan completed,” not “there are zero findings.” This is useful for evidence production, but dangerous if the team assumes job green equals security green.

security_scan_broken:
  stage: test
  image: semgrep/semgrep:1.176.0
  script:
    - semgrep scan --config security/semgrep-rules.yml --json --output security-reports/semgrep.json src/
  artifacts:
    when: always
    paths: [security-reports/]
  allow_failure: true

The evidence may be correct; the missing control is the gate. Preserve the report, then add a deterministic parser/gate or an approved GitLab security policy workflow. Do not replace evidence with grep ERROR job.log.

3. Failure mode: template is stale, deprecated, or points at the wrong migration path

Symptoms include pipeline compilation errors, missing old job names, or a scan continuing on a legacy analyzer after the organization thinks it migrated. Inspect the merged CI configuration and current feature docs. Dependency Scanning is a concrete example: workflows customized around gemnasium-* job names must be revisited when moving to v2, whose producing job is dependency-scanning.

Do not “fix” this by copying old GitLab template contents into the repository. Update the supported include/component and then update any downstream needs or overrides that referenced legacy job names.

4. Failure mode: secret detector exposes the test secret

If a real credential appears in a commit, log, artifact, or scanner debug output, stop treating it as a training finding. Preserve necessary non-secret metadata, revoke/rotate the credential, restrict access to the leaked evidence, and then remove it from future history according to your incident process. Do not paste the value into tickets or print variables to prove the detector worked.

The course avoids this failure by using a synthetic marker and recording only its SHA-256 in the custom report.

5. Failure mode: DAST points at production

Do not run the scan “once to see what happens.” Preserve the pipeline/job/config evidence, cancel before active traffic if possible, verify whether any requests were already sent, notify the target owner if required, and change the target allowlist/config to a disposable environment. Active scanners can generate side effects or load; authorization is part of correctness.

Never repair a DAST connectivity problem by disabling TLS verification globally or widening firewall access to production. Fix the test target, certificate trust, and network path deliberately.

6. Failure mode: gate blocks with no exception workflow

A hard block with no reviewed exception path eventually encourages shadow bypasses. Preserve the finding and gate reason. Then implement a narrow exception contract: specific finding ID/scope, technical rationale, owner/approver, expiry, and audit trail. Emergency override is governance state and must not silently mutate the underlying report.

7. Failure mode: report belongs to the wrong source or image

A valid security report can still be stale evidence. For source scans, compare source SHA and producer pipeline/job. For container scans, compare the immutable image digest, not merely latest. For DAST, bind the environment URL to the deployed artifact/image identity. For IaC, bind the report to the configuration revision. If the binding is missing, treat provenance as unknown.

8. Performance: optimize the expensive layer, not the evidence away

Symptom Likely cause Safer correction
Scans queue for a long time Too many scanners for runner capacity Stage fast scans; parallelize within capacity; use interruptible only for read-only safe scans.
SAST consumes excessive memory Large codebase/analyzer requirements Use supported exclusions/rules and adequate runner resources; record exclusions.
Container scan repeatedly downloads databases Cold cache/offline setup Use documented mirrors/cache/offline workflow; record DB version.
DAST takes too long Broad crawl/active profile Narrow authorized test scope; separate fast passive feedback from deeper scheduled/release scans.
Gate noise overwhelms developers Poor rules/threshold/triage workflow Measure false positives; tune narrowly; add reviewable exceptions rather than disabling scanner.

9. Interpret one intentionally broken case

Observed evidence Interpretation Least-destructive repair
Semgrep job passed; report has one ERROR result Scanner completed and produced a finding. Keep report; add/repair gate policy.
Gate is absent No delivery decision exists. Add deterministic report parser or approved policy.
Source SHA/checksum recorded Finding provenance is usable. Do not regenerate the report just to make it disappear.
No production secrets/targets involved Safe disposable scope. Remediate source or add reviewed synthetic exception, then rerun smallest scope.

Knowledge check

What should you preserve before changing a security job that failed or produced surprising findings?

A scanner job is green and the report contains Critical findings. What is the likely missing layer?

What is the first safe action when DAST is configured for production accidentally?

Why is copying a deprecated GitLab scanner template into the repository a poor repair?

Why does a finding exception need an expiry?

Next lesson

Checkpoint lab

Run two scanner classes, preserve evidence, triage one synthetic issue with an explicit exception, remediate another finding, and prove the gate outcome.

Version and compatibility note

GitLab and GitLab Runner evolve continuously. Treat version-sensitive YAML, runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations as assumptions to verify against the current official GitLab documentation before production use. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for any reproducible lab or incident record.

Official references and version notes

Documentation verification date: 2026-09-12. Basic GitLab SAST, pipeline Secret Detection, Container Scanning, and IaC scanning are documented for Free/Premium/Ultimate. Current GitLab Dependency Scanning and DAST are Ultimate. Security policy enforcement is Ultimate. GitLab's current Dependency Scanning direction is SBOM-based; the legacy Gemnasium-based implementation was deprecated in GitLab 17.9 and is proposed for removal in GitLab 20.0. Stable security templates are recommended for production workflows; Latest templates change more frequently. AST_ENABLE_MR_PIPELINES="true" is the current recommended switch for supported application-security scans in merge-request pipelines. Current GitLab guidance emphasizes supported templates/components, valid security report schemas, and test-only DAST targets. The course deliberately avoids production scanning, credential printing, and broad token workarounds.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.