SAST, Secret Detection, Dependency Scanning, Container Scanning, DAST, and Coverage-Guided Security: Diagnostics, Failure Modes, Security, and Performance

1. Diagnostic sequence: preserve → scope → inspect → minimally correct → verify

Do not start by rerunning scans repeatedly or deleting the evidence. Preserve the pipeline/job URL, commit SHA, pipeline source, runner identity, expanded CI configuration, analyzer image/version, report artifact, report checksum, and target identity. Then classify the problem: execution failure, target mismatch, report-generation failure, schema/ingestion failure, authorization/tier mismatch, or true security finding.

2. Failure A — scanner job passes, but the report is malformed

An intentionally broken job can exit zero and upload invalid SAST JSON:

broken_sast_fixture:
  stage: test
  image: alpine:3.22
  script:
    - printf '{"vulnerabilities": []}\n' > broken-sast.json
  artifacts:
    reports:
      sast: broken-sast.json

The shell job can succeed, but the report lacks required Secure report fields such as a supported version and scan metadata. GitLab validates security reports before ingestion and surfaces a validation error. Repair: do not suppress the error or rename the file. Fix the analyzer/integration so it emits a current supported schema, validate locally where practical, rerun, and compare the new report checksum/schema.

3. Failure B — report exists, but scanner execution failed

If the scanner job itself fails, do not assume the uploaded report is normal ingested security evidence. GitLab’s scanner integration contract says failed jobs are not ingested as ordinary security results, even if artifacts remain downloadable. Preserve stderr/exit code and report, then fix the scanner execution first.

4. Failure C — source scanner analyzed the wrong ref or pipeline source

Compare the report/job to the intended release candidate:

glab api "projects/$PROJECT_ID/pipelines/$PIPELINE_ID" | jq '{id,sha,ref,source,status}'
glab api "projects/$PROJECT_ID/pipelines/$PIPELINE_ID/jobs" | jq 'map({id,name,status,ref,commit:(.commit.id // null)})'

A branch pipeline, merge-request pipeline, schedule, child pipeline, or downstream pipeline can have different inputs/variables/trust. If the report SHA is not the candidate SHA, do not “relabel” the finding. Run the correct pipeline or bind the release gate to the correct scan evidence.

5. Failure D — container scan analyzed the wrong image

A tag such as latest may have moved between build and scan. Preserve the report’s image identity and registry metadata; resolve the candidate digest independently. If digests differ, the scan is not evidence for the candidate image.

candidate=registry.example.invalid/team/app@sha256:<candidate-digest>
scanned=registry.example.invalid/team/app@sha256:<report-digest>
# Evidence is valid for promotion only when the immutable identities match.

Repair the pipeline so the build exports the immutable digest and the scan consumes that identity, then the release/deployment consumes the same digest.

6. Failure E — DAST points at the wrong URL/environment

DAST evidence is meaningful only for the running deployment it exercised. A test URL can be stale, point to another branch, or front a different image. Preserve environment/deployment record, URL, source SHA, image digest, DAST profile/config, and scan timestamp. Never compensate by scanning production “to be sure.” Deploy the intended disposable candidate to a test environment and rescan there.

7. Failure F — a secret finding is “fixed” only by history deletion

This is an incident-response error. If a real credential was exposed, anyone who saw it may already have copied it. Rewriting Git history cannot invalidate the credential.

History rewrite is destructive and can disrupt collaborators. It belongs after credential invalidation and with an explicit recovery/coordination plan.

8. Failure G — template/analyzer update changes behavior unexpectedly

GitLab security templates and analyzer images evolve. A new version can add rules, change language detection, upgrade a scanner, alter advisory content, or fix parsing. Preserve the old/new expanded configuration and analyzer image/version; compare report schema, scanner metadata, finding set, and release notes. Test scanner configuration changes in a merge request before default-branch rollout.

If a temporary pin is necessary, document why, exact version, owner, and removal date. Do not treat a pin as permanent security configuration.

9. Failure H — untrusted MR needs a secret or privileged runner to scan

Do not solve this by exposing the secret or loosening runner protections. Identify why the scanner needs the privileged resource. Options include a reduced untrusted-MR scan, public/mirrored test dependencies, a post-review trusted pipeline, or short-lived narrowly scoped credentials with no production reach. If no safe path exists, fail closed and document the missing evidence.

10. Failure I — “no findings” is treated as proof of security

An empty report means only that the configured scanner found nothing in its scope with its current rules/intelligence. It says nothing about unscanned languages, business logic, architecture, authorization design, runtime configuration, supply-chain compromise, or unknown vulnerabilities. Compare scanner coverage to the threat model and release asset inventory.

11. Performance and cost failures that weaken security

Security jobs can become the critical path. Diagnose before disabling controls:

12. Broken-report repair exercise

Starting from the malformed report fixture above, the correct diagnostic is:

1. Confirm job exited zero.
2. Open the pipeline Security/report validation message.
3. Download the exact broken artifact and checksum it.
4. Inspect declared report type and report version.
5. Replace the hand-crafted malformed report with output from a current analyzer or a fixture validated against the current schema.
6. Rerun and verify the validation error disappears.

Do not change artifacts:reports:sast to a plain artifact merely to hide validation.

13. Minimum diagnostic evidence packet

project: group/project
pipeline_id: 1234
pipeline_source: merge_request_event
source_sha: <40-hex SHA>
job_id/name/status: ...
runner: hosted-or-isolated-description
analyzer/template: exact configured source
report_type: sast|secret_detection|container_scanning|...
report_sha256: ...
target_identity: source SHA | image digest | test URL+deployment
validation_status: valid | invalid + error
triage_decision: ...

Knowledge check

Summary

Scanner troubleshooting is evidence forensics. Preserve the failing state, prove scope and target identity, distinguish job execution from report validation/ingestion, repair the smallest broken layer, and rerun without hiding the original cause.

Official references

Primary sources used for the current GitLab 19.3 behavior taught in this lesson:

• GitLab Docs — Application security testing
• GitLab Docs — SAST
• GitLab Docs — SAST analyzers
• GitLab Docs — Pipeline secret detection
• GitLab Docs — Customize pipeline secret detection
• GitLab Docs — Dependency scanning
• GitLab Docs — Container scanning
• GitLab Docs — DAST
• GitLab Docs — Security scanning results
• GitLab Docs — Security report validation
• GitLab Docs — CI/CD artifacts report types
• GitLab Docs — Security scanner integration
• GitLab Docs — Coverage-guided fuzz testing (deprecated)
• GitLab Docs — Deprecations and removals