SAST, Dependency Scanning, Secret Detection, Container Scanning, DAST, IaC Scanning, and Security Gates: Diagnostics, Failure Modes, Security, and Performance
Diagnose false-green scans, stale/deprecated configuration, leaked synthetic-secret evidence, unsafe DAST targets, wrong-SHA reports, noisy gates, and scanner performance from preserved first-failure evidence.
Learning objectives
- Use an evidence-first sequence to distinguish scanner execution, report ingestion, policy, runner, identity, and target failures.
- Diagnose a green scanner job whose findings are ignored without destroying the original evidence.
- Repair deprecated/stale scanner configuration by validating the compiled configuration and current primary docs.
- Respond safely to secret-detection and DAST mistakes without printing secrets or rerunning against production.
- Reduce security-scan latency/noise without blind retries, disabled TLS, privileged shortcuts, or invisible policy bypass.
1. Evidence-first diagnostic sequence
Start by recording CI_PIPELINE_SOURCE and
CI_COMMIT_SHA; these distinguish a correct scanner
result on the wrong pipeline/source from a real scanner failure.
- Preserve pipeline/job IDs, source/ref/SHA, scanner report/checksum, job trace, and first observed policy result.
-
Confirm merged/compiled CI configuration and effective
workflow:rules/job rules. - Confirm scanner/analyzer image and rules/config version and the exact input target.
- Inspect job graph, queue, runner/executor/image, and whether untrusted code received protected resources.
- Inspect scanner command, exit semantics, network/database availability, and report generation.
- Validate the report against its real schema/format and bind it to the source/image/URL it scanned.
- Inspect the gate/policy and any exception record separately from the scanner result.
- For DAST/container/cloud inputs, inspect external target state and authorization.
- Apply the smallest correction and rerun only the necessary disposable scope.
2. Failure mode: scan is green but findings are ignored
Many scanners use exit zero to mean “the scan completed,” not “there are zero findings.” This is useful for evidence production, but dangerous if the team assumes job green equals security green.
security_scan_broken:
stage: test
image: semgrep/semgrep:1.176.0
script:
- semgrep scan --config security/semgrep-rules.yml --json --output security-reports/semgrep.json src/
artifacts:
when: always
paths: [security-reports/]
allow_failure: true
The evidence may be correct; the missing control is the gate.
Preserve the report, then add a deterministic parser/gate or an
approved GitLab security policy workflow. Do not replace evidence
with grep ERROR job.log.
3. Failure mode: template is stale, deprecated, or points at the wrong migration path
Symptoms include pipeline compilation errors, missing old job names,
or a scan continuing on a legacy analyzer after the organization
thinks it migrated. Inspect the merged CI configuration and current
feature docs. Dependency Scanning is a concrete example: workflows
customized around gemnasium-* job names must be
revisited when moving to v2, whose producing job is
dependency-scanning.
Do not “fix” this by copying old GitLab template contents into the
repository. Update the supported include/component and then update
any downstream needs or overrides that referenced
legacy job names.
4. Failure mode: secret detector exposes the test secret
If a real credential appears in a commit, log, artifact, or scanner debug output, stop treating it as a training finding. Preserve necessary non-secret metadata, revoke/rotate the credential, restrict access to the leaked evidence, and then remove it from future history according to your incident process. Do not paste the value into tickets or print variables to prove the detector worked.
The course avoids this failure by using a synthetic marker and recording only its SHA-256 in the custom report.
5. Failure mode: DAST points at production
Do not run the scan “once to see what happens.” Preserve the pipeline/job/config evidence, cancel before active traffic if possible, verify whether any requests were already sent, notify the target owner if required, and change the target allowlist/config to a disposable environment. Active scanners can generate side effects or load; authorization is part of correctness.
6. Failure mode: gate blocks with no exception workflow
A hard block with no reviewed exception path eventually encourages shadow bypasses. Preserve the finding and gate reason. Then implement a narrow exception contract: specific finding ID/scope, technical rationale, owner/approver, expiry, and audit trail. Emergency override is governance state and must not silently mutate the underlying report.
7. Failure mode: report belongs to the wrong source or image
A valid security report can still be stale evidence. For source
scans, compare source SHA and producer pipeline/job. For container
scans, compare the immutable image digest, not merely
latest. For DAST, bind the environment URL to the
deployed artifact/image identity. For IaC, bind the report to the
configuration revision. If the binding is missing, treat provenance
as unknown.
8. Performance: optimize the expensive layer, not the evidence away
| Symptom | Likely cause | Safer correction |
|---|---|---|
| Scans queue for a long time | Too many scanners for runner capacity | Stage fast scans; parallelize within capacity; use interruptible only for read-only safe scans. |
| SAST consumes excessive memory | Large codebase/analyzer requirements | Use supported exclusions/rules and adequate runner resources; record exclusions. |
| Container scan repeatedly downloads databases | Cold cache/offline setup | Use documented mirrors/cache/offline workflow; record DB version. |
| DAST takes too long | Broad crawl/active profile | Narrow authorized test scope; separate fast passive feedback from deeper scheduled/release scans. |
| Gate noise overwhelms developers | Poor rules/threshold/triage workflow | Measure false positives; tune narrowly; add reviewable exceptions rather than disabling scanner. |
9. Interpret one intentionally broken case
| Observed evidence | Interpretation | Least-destructive repair |
|---|---|---|
| Semgrep job passed; report has one ERROR result | Scanner completed and produced a finding. | Keep report; add/repair gate policy. |
| Gate is absent | No delivery decision exists. | Add deterministic report parser or approved policy. |
| Source SHA/checksum recorded | Finding provenance is usable. | Do not regenerate the report just to make it disappear. |
| No production secrets/targets involved | Safe disposable scope. | Remediate source or add reviewed synthetic exception, then rerun smallest scope. |
Knowledge check
What should you preserve before changing a security job that failed or produced surprising findings?
Pipeline/job IDs, source/ref/SHA, merged config, scanner/version/rules, raw report/checksum, job trace, input target identity, and gate/exception result.
A scanner job is green and the report contains Critical findings. What is the likely missing layer?
The gate/policy layer, unless the documented scanner contract itself was supposed to return non-zero on findings.
What is the first safe action when DAST is configured for production accidentally?
Stop/cancel before further active traffic if possible, preserve configuration/evidence, and verify whether requests already reached the target. Do not rerun.
Why is copying a deprecated GitLab scanner template into the repository a poor repair?
It freezes implementation details, breaks upgrade/migration paths, and makes supported upstream fixes harder to adopt.
Why does a finding exception need an expiry?
Without review/expiry, temporary risk decisions become permanent invisible bypasses as code and scanner behavior change.
Version and compatibility note
GitLab and GitLab Runner evolve continuously. Treat version-sensitive YAML, runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations as assumptions to verify against the current official GitLab documentation before production use. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for any reproducible lab or incident record.
Official references and version notes
Documentation verification date: 2026-09-12. Basic
GitLab SAST, pipeline Secret Detection, Container Scanning, and IaC
scanning are documented for Free/Premium/Ultimate. Current GitLab
Dependency Scanning and DAST are Ultimate. Security policy
enforcement is Ultimate. GitLab's current Dependency Scanning
direction is SBOM-based; the legacy Gemnasium-based implementation
was deprecated in GitLab 17.9 and is proposed for removal in GitLab
20.0. Stable security templates are recommended for production
workflows; Latest templates change more frequently.
AST_ENABLE_MR_PIPELINES="true" is the current
recommended switch for supported application-security scans in
merge-request pipelines. Current GitLab guidance emphasizes
supported templates/components, valid security report schemas, and
test-only DAST targets. The course deliberately avoids production
scanning, credential printing, and broad token workarounds.
- SAST — official reference.
- SAST analyzers — official reference.
- Secret detection — official reference.
- Secret detection configuration — official reference.
- Container scanning — official reference.
- Dependency scanning — official reference.
- Dependency scanning migration to SBOM — official reference.
- DAST — official reference.
- DAST browser analyzer — official reference.
- IaC scanning — official reference.
- Security configuration — official reference.
- Security policies — official reference.
- Security scanner integration and report schemas — official reference.
- CI/CD YAML syntax — official reference.
Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.
Keep the academy open
Support free, practical DevOps education.
Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0
Send only Ethereum/ERC-20 compatible assets to this
address.