Chapter 21Lesson 04~145 minutes

Signing, Trust, Secret Removal, fsck, Safe Directories, and Repository Integrity: Diagnostics, Failure Modes, Security, and Performance

Diagnose dangerous security shortcuts: unrotated leaked credentials, valid-but-unauthorized signatures, wildcard safe.directory trust, premature pruning, overinterpreted fsck output, and incomplete history cleanup.

DiagnosticsIncident responseTrust failuresEvidence preservation

Learning objectives

  • Apply a contain-preserve-classify-correct-verify incident sequence.
  • Explain why credential rotation must precede history rewriting.
  • Validate signer identity and authorization separately from signature mathematics.
  • Preserve unreachable/reflog evidence before any cleanup that reduces recovery options.
  • Coordinate rewritten history across remote refs, clones, caches, forks, mirrors, and artifacts.

1. Incident diagnostic sequence: preserve before you purge

  1. Contain: if a real credential is involved, revoke/rotate it immediately.
  2. Preserve evidence: refs, object IDs, reflogs, signature output, fsck output, remote state, configuration origins, timestamps, and relevant logs.
  3. Classify the layer: credential secrecy, signature identity/trust, object integrity, ownership trust, authorization, or history reachability.
  4. Choose the least destructive correction.
  5. Verify independently: history scans, signature policy, fsck, remote cleanup, and prevention controls.

2. Capture evidence without exposing secret material

git status --short --branch
git show-ref --head
git reflog --all --date=iso
git log --graph --decorate --oneline --all --max-count=100
git fsck --no-progress
git config --list --show-origin --show-scope
git config --show-origin --get-all safe.directory
git config --show-origin --get gpg.format

Record hashes and metadata; do not paste real secret-bearing blobs, private keys, tokens, or credential-helper contents into tickets or chat systems.

3. Intentionally broken response — rewrite first, rotate later

09:00 token discovered in Git history
09:05 history rewrite started
09:40 rewrite completed
10:10 token finally revoked

The repository may look clean at 09:40, but the token remained usable for seventy minutes after discovery and could still exist in clones/caches. The repair is procedural: revoke/rotate at 09:00, preserve evidence, then coordinate cleanup. Git cannot revoke a credential at its issuing service.

4. Failure mode — trusting any cryptographically valid signature

An attacker or unauthorized contractor can sign with their own valid key. verify-commit can confirm the signature matches that key, but organizational trust requires mapping the key to an approved principal and checking that principal's authorization for the action.

git verify-commit HEAD
git log -1 --show-signature --format='%H %G? %GF %GS %s'

Use the reported signer/fingerprint as evidence for your identity policy; do not stop at “Good signature.”

5. Failure mode — silencing dubious ownership with a wildcard

# Dangerous broad workaround — do not use as the course fix:
git config --global --add safe.directory '*'
This disables the ownership safety check for every repository covered by that protected config. Instead inspect the exact path, owner, mount/container identity, and intended service account. Correct ownership or approve one exact path.

6. Diagnose a dubious-ownership warning by layer

Ask: Which OS user is running Git? Who owns the working tree and Git directory? Is the path a bind mount/network share/container volume? Is this repository expected to be shared? Only after answering those questions should an administrator choose a narrow safe-directory exception.

7. Failure mode — aggressive cleanup destroys recovery/forensic opportunities

Do not expire reflogs, prune immediately, or run aggressive garbage collection while investigating. Those actions can remove unreachable objects that might contain lost work or evidence.
git fsck --unreachable --no-reflogs --no-progress
git reflog --all
git cat-file -t <candidate-object>
git cat-file -p <candidate-object>

Inspect first. Chapter 22 will cover maintenance and retention; this chapter deliberately avoids “prune now” incident advice.

8. Failure mode — examples themselves leak private material

Security documentation sometimes causes a second incident by pasting a real private key, token, or credential URL into a demo. Use placeholders or generated disposable keys. If real material appears in a transcript, treat that transcript location as another exposure surface and rotate accordingly.

9. Failure mode — “fsck is clean, therefore the repository is safe”

A clean fsck result means Git found no selected object/reference integrity problems. A malicious script can be stored in a perfectly valid blob. A secret can be stored in a perfectly valid blob. An unauthorized signed commit can be perfectly well-formed. Integrity checks are necessary evidence, not a security verdict.

10. --connectivity-only intentionally checks less

git fsck --connectivity-only --no-progress

This mode can be faster because it focuses on reachable connectivity and avoids full blob checking. Use it when that is the question you intend to answer; do not report it as equivalent to a full fsck integrity pass.

11. Failure mode — treating dangling output as corruption

Dangling objects commonly arise from ordinary Git workflows. Before deleting anything, identify the object type and whether it corresponds to abandoned but valuable work:

git fsck --no-progress
git show --stat <dangling-commit>
git branch --contains <dangling-commit>

“No branch contains it” means the commit is not currently reachable from those branch refs; it does not mean the data is malicious or worthless.

12. Failure mode — cleaning only the visible branch

Sensitive data can remain in tags, remote-tracking refs, hidden server refs, forks, caches, pull-request refs, mirrors, and colleague clones. Current filter-repo sensitive-data guidance explicitly treats cleanup of other copies as part of the incident. A successful local rewrite is only one stage.

13. Failure mode — assuming old signatures survive rewritten history

Commit IDs and signed payloads change during filtering. Current filter-repo limitations document that commit signatures are stripped and signed tags become ordinary annotated tags. Therefore post-cleanup verification must distinguish “old signature evidence archived in incident records” from “new signed attestations on sanitized history.”

14. Force-update coordination is a publication incident, not a routine push

Publishing rewritten history can overwrite remote refs and disconnect collaborators' old histories. Freeze writes, capture authoritative remote OIDs, communicate the rewrite window, use the narrowest refspecs possible, and prefer expected-old protections such as --force-with-lease where they fit the cleanup plan. Hosted protected refs and hidden refs may require provider/admin-specific procedures.

Do not run a broad mirror force-push from an ordinary training repository. The checkpoint documents coordination rather than teaching an indiscriminate remote overwrite.

15. Caches, forks, artifacts, and mirrors can preserve the old data

A source-host rewrite does not recall existing clones, artifact caches, CI workspaces, release archives, backups, or forks. Incident closure should list each copy class, its owner, whether it can be deleted/garbage-collected, and how to prevent an old clone from reintroducing the contaminated history.

16. Repository trust extends beyond Git object validity

Untrusted repositories may include build scripts, attributes, submodules, package scripts, generated tooling, and instructions that persuade a user to install hooks or drivers. Git's ownership protections reduce one class of risk but do not sandbox arbitrary project code. Review before executing.

17. Security/performance tradeoff — integrity checks cost resources

Full fsck and transfer fsck can be expensive in very large repositories. That is a reason to choose when and where checks run—not a reason to misreport a lighter check as equivalent. Server ingress, periodic integrity jobs, and developer diagnostics can use different schedules while preserving clear semantics.

18. Symptom → layer → least-destructive next action

Symptom Layer Next action
Real token committed Credential secrecy Revoke/rotate first; preserve evidence; plan rewrite
Good signature from unknown key Authenticity/trust mapping Validate principal/key and policy authorization
Dubious ownership warning Repository ownership trust Inspect owner/path; exact exception only if justified
Dangling objects Reachability/recovery Inspect object/reflog; do not prune during incident
Clean fsck but suspicious script Content behavior Code/security review; fsck is not malware analysis

19. Knowledge check

Question 1. What is wrong with rewriting before revoking a real exposed token?

Question 2. Why is a Good signature not enough for authorization?

Question 3. Why is safe.directory=* dangerous as a generic fix?

Question 4. Why avoid pruning while investigating fsck/reflog evidence?

Question 5. Can a malicious script pass git fsck?

20. Summary

Security incidents become manageable when the team classifies the layer correctly. Rotate credentials, verify signer identity and policy separately, preserve fsck/reflog evidence, keep ownership exceptions narrow, and coordinate rewrites across every relevant copy.

Next

Checkpoint a complete source-control incident

Lesson 5 combines fake-secret detection, ephemeral signing, containment, filter-repo planning, integrity verification, post-rewrite re-signing, and a reusable incident checklist.

Authoritative references

 git-fsck
 git-config safe.directory and fsck settings
 git-filter-repo

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.