Chapter 24Lesson 04~180 minutes

Secret Scanning, Push Protection, Custom Patterns, Bypass Controls, and Incident Response: Diagnostics, Failure Modes, Security, and Performance

Secret incidents often go wrong because teams optimize the repository instead of the credential: they rewrite history before revoking, normalize bypasses, over-broaden custom patterns, forget non-Git surfaces, or rotate a key without migrating consumers. This lesson diagnoses those failures from evidence and repairs the smallest causal boundary.

DiagnosticsNon-Git surfacesRotationNoise controlContainment

Learning objectives

  • Use a repeatable diagnostic sequence for secret alerts, push blocks, bypass noise, custom-pattern failures, and incomplete rotations.
  • Recognize the critical sequencing failure of history cleanup before provider revocation.
  • Diagnose bypass fatigue and over-broad custom patterns without disabling protection wholesale.
  • Scope incidents across Git history and non-Git GitHub surfaces, then verify dependent systems after rotation.
  • Interpret API/permission/plan failures without exposing literal secrets or weakening repository policy.
Diagnostic order: preserve evidence → identify repository/org/account/ref/surface/credential scope → inspect settings, pattern, permission, alert locations, provider status and logs → choose the least destructive correction → verify credential invalidation and consumer recovery. Do not start by rewriting history, resolving the alert, or disabling push protection.

1. Failure: the team cleans Git history but forgets revocation

This is the highest-impact failure in the chapter. A developer notices a token, runs a history-rewrite procedure, force-updates the repository, and posts “fixed” in chat. The token is still active at the provider. Anyone who copied it before the rewrite can continue using it, and old copies may remain in forks, clones, caches, PR references, or logs.

Repair the sequence, not the repository cosmetics: identify provider/owner → revoke or rotate → confirm old authentication fails → inspect provider audit/use logs → migrate legitimate consumers → only then decide whether historical content removal is warranted. Preserve the original detection timestamp and revocation timestamp for incident evidence.

2. Failure: routine bypass turns prevention into alert noise

A team may develop a habit of choosing “used in tests” or “false positive” whenever a deadline is near. Repository push protection can generate alerts for bypassed values, but a high volume of expected bypass alerts conditions responders to ignore the channel. The root cause may be bad fixtures, a noisy custom pattern, a migration workflow, or weak exception authority—not “too much security.”

Evidence Question
Bypass frequency by repository/actor/pattern Is one team, automation, or pattern responsible for most bypasses?
Bypass reason quality Are reasons specific and technically reviewable or boilerplate?
Alert outcomes Do bypassed alerts stay open, get resolved immediately, or recur?
Pattern precision Does the same legitimate string repeatedly match?
Architecture Can test fixtures or automation use clearly non-secret formats instead?

Repair with a narrower pattern, safe fixture convention, delegated review, or redesigned automation. Disabling push protection globally treats the symptom by removing evidence and increases risk.

3. Intentionally broken custom pattern: it matches ordinary code

This training pattern is intentionally poor: token. It would match variable names, comments, documentation, tests, and security guidance across normal source code. Publishing it with push protection would create widespread friction and bypass pressure.

import re
bad = re.compile(r"token", re.I)
examples = [
    "const tokenCount = 3;",
    "// parse token from header",
    "ACME_TEST_ABC123DEF456",
]
for value in examples:
    print(bool(bad.search(value)), value)

The repair is not “tell developers to bypass.” Define the secret grammar and context. A safer training contract might require an exact prefix plus a constrained suffix, then dry-run against representative repositories before publication. Keep positive and negative fixtures in version-controlled security-policy tests outside credential stores.

good = re.compile(r"\bACME_TEST_[A-Z0-9]{12}\b")
positive = ["ACME_TEST_ABC123DEF456"]
negative = ["tokenCount", "ACME_TEST_MODE", "ACME_TEST_ABC123"]
assert all(good.search(x) for x in positive)
assert all(not good.search(x) for x in negative)
print("fixture contract passes")

4. Failure: response only searches Git commits

Current secret scanning can find secrets in issue titles/bodies/comments, pull-request text/reviews, Discussions, wikis, and secret gists in addition to Git history. An incident responder who runs only git log -S may clean source while leaving the same credential plainly visible in a support issue or review comment.

# Read alert metadata without literal secret values.
ALERT=123
gh api -H "X-GitHub-Api-Version: 2026-03-10" \
  "repos/$REPO/secret-scanning/alerts/$ALERT" \
  --jq '{number,state,secret_type,validity,resolution,created_at}'

# Locations can identify Git and non-Git surfaces.
gh api --paginate -H "X-GitHub-Api-Version: 2026-03-10" \
  "repos/$REPO/secret-scanning/alerts/$ALERT/locations?per_page=100" \
  --jq '.[] | {type,details}'

Do not blindly dump location objects into public logs if they contain sensitive context. Use them to build a remediation checklist: current file, historical commit, PR review, issue comment, discussion, wiki, gist, fork/clone owner, and any external log or artifact that may have copied the value.

5. Failure: rotation succeeds, consumers still use the old credential

Rotating a secret creates a new value and invalidates the old one, but production is not recovered until legitimate consumers migrate. A deployment job, scheduled task, developer workstation, webhook client, or external service may continue presenting the dead credential and fail. Teams sometimes respond by re-enabling the old credential, reopening the incident.

Before rotation, identify consumers from secret-manager references, configuration inventories, workflow/environment secrets, application logs, and provider metadata. After rotation, verify old authentication fails and every required consumer succeeds with the replacement. Use rollback at the application/configuration layer—not by resurrecting a compromised secret.

6. Failure: policy assumes a feature the repository cannot use

A personal public repository can demonstrate secret scanning and push protection, but custom patterns and delegated bypass currently have organization/plan requirements. A workflow document that says “request delegated bypass” is incomplete if the repository is user-owned or the organization lacks Secret Protection.

Differentiate three states: feature unavailable (wrong account/repository/product), insufficient permission (feature exists but your role/token cannot manage it), and misconfiguration (you have the feature and role but the pattern/policy is wrong). The correction differs for each; buying a plan is never a mandatory lab requirement.

7. A production diagnostic worksheet

Stage Evidence to preserve Least-destructive correction
Detection Alert number/type/locations/validity, blocked push message, commit/ref if persisted Do not resolve or delete evidence before triage.
Credential scope Issuer, owner, permissions, expiration, environments, consumers Revoke/rotate or disable at issuer.
Use assessment Provider authentication/audit logs and exposure time window Contain accounts/sessions/resources actually affected.
GitHub scope Git + issue/PR/discussion/wiki/gist locations, forks Remove/replace exposed content after revocation.
Policy cause Pattern, bypass actor/reason, repository security state Narrow pattern, change reviewer/exemption model, enable protection.
Recovery Old credential failure + new credential success + healthy consumers Close incident only after independent verification.
History Sensitivity/compliance requirement and rewrite side effects Rewrite only if additional benefit justifies coordination.

8. History rewrite: why the “fix” can create a second incident

SECURITY-SENSITIVE / DESTRUCTIVE — DO NOT RUN IN THIS CHAPTER LAB. GitHub’s official procedure may involve git-filter-repo --sensitive-data-removal and a force/mirror update. This rewrites commit IDs and can break signatures, PR diffs, references and collaborator clones. It also creates recontamination risk if an old clone pushes rewritten-away history back.

If policy requires a rewrite, freeze affected writes, inventory branches/tags/PRs/forks, coordinate every collaborator, make a clean dedicated clone, record the first changed commits, rewrite only the required data, verify all refs, update protections in a controlled window, force-update only after approval, require re-clone/rebase instructions, and contact GitHub Support when cached/internal PR references need server-side removal. Never use history rewrite as the credential-revocation mechanism.

Knowledge check

A team force-pushed a cleaned history but the provider token still works. What is the incident state?

Push protection has dozens of “used in tests” bypasses every day. What should you investigate first?

Why can git log -S be insufficient for secret incident scope?

After rotating a credential, what two opposite checks prove recovery?

A delegated-bypass option is missing in a personal public repo. Is that necessarily a bug?

Why can history rewrite increase risk?

Summary

The most important diagnostic is sequencing: preserve evidence, invalidate access, assess use, migrate consumers, remove exposure, then decide whether disruptive history cleanup adds value. Noise and missing features are policy/configuration problems, not reasons to weaken the incident boundary.

Next the checkpoint turns that sequence into a repeatable runbook with predictions, evidence, a fake leak, a bypass-review policy, and a history-rewrite decision record.

Next lesson

Checkpoint Lab — Secret Scanning, Push Protection, Custom Patterns, Bypass Controls, and Incident Response

Official references

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.