Production Capstone: Design, Secure, Scale, and Recover a Governed Git Workflow: Failure Injection, Troubleshooting, and Recovery Drill

1. Failure drill rule: preserve → inspect → classify → correct → verify

Every drill is disposable. Capture refs/OIDs/config/logs before mutation. Do not use cleanup, pruning, broad reset, or blind force merely to make an error disappear.

2. Failure 1 — stale explicit force-with-lease

git clone "$SERVER_URL" rewriter
git clone "$SERVER_URL" concurrent
git -C rewriter config user.name "Rewrite Drill"
git -C rewriter config user.email "rewrite@example.invalid"
git -C concurrent config user.name "Concurrent Writer"
git -C concurrent config user.email "concurrent@example.invalid"
EXPECTED=$(git -C rewriter rev-parse refs/remotes/origin/trunk)
printf "concurrent=true\n" > concurrent/concurrent.conf
git -C concurrent add concurrent.conf
git -C concurrent commit -m "concurrent server advancement"
git -C concurrent push origin trunk
TREE=$(git -C rewriter rev-parse HEAD^{tree})
REPLACEMENT=$(printf "replacement history drill\n" | git -C rewriter commit-tree "$TREE")
git -C rewriter push --force-with-lease=refs/heads/trunk:"$EXPECTED" origin "$REPLACEMENT":refs/heads/trunk
echo "expected stale lease exit=$?"

The push must fail because the server no longer equals EXPECTED. The lease protected concurrent work. Fetch and redesign/recompute; do not refresh the lease only to defeat its purpose.

3. Failure 2 — merge conflict

git clone "$SERVER_URL" conflict-lab
git -C conflict-lab config user.name "Conflict Drill"
git -C conflict-lab config user.email "conflict@example.invalid"
git -C conflict-lab switch -c left
printf "mode=left\n" > conflict-lab/mode.conf
git -C conflict-lab add mode.conf
git -C conflict-lab commit -m "left mode"
git -C conflict-lab switch -c right HEAD~1
printf "mode=right\n" > conflict-lab/mode.conf
git -C conflict-lab add mode.conf
git -C conflict-lab commit -m "right mode"
git -C conflict-lab switch left
git -C conflict-lab merge right
git -C conflict-lab status --short
git -C conflict-lab ls-files -u
git -C conflict-lab merge --abort
git -C conflict-lab status --short

Unmerged index stages are evidence. Because this drill intentionally chose the wrong integration, abort returns to the pre-merge state.

4. Failure 3 — unpublished rebase conflict and abort

git -C conflict-lab switch -c rebase-base left
printf "setting=base-new\n" > conflict-lab/rebase.conf
git -C conflict-lab add rebase.conf
git -C conflict-lab commit -m "base changes setting"
git -C conflict-lab switch -c private-topic HEAD~1
printf "setting=topic-new\n" > conflict-lab/rebase.conf
git -C conflict-lab add rebase.conf
git -C conflict-lab commit -m "topic changes setting"
PRIVATE_BEFORE=$(git -C conflict-lab rev-parse HEAD)
git -C conflict-lab rebase rebase-base
git -C conflict-lab status --short
git -C conflict-lab rebase --show-current-patch
git -C conflict-lab rebase --abort
test "$(git -C conflict-lab rev-parse HEAD)" = "$PRIVATE_BEFORE"

The topic is intentionally unpublished. Resolving plus rebase --continue would create replacement commit IDs; abort restores the original topic tip.

5. Failure 4 — lost branch recovery

git -C conflict-lab switch -c recover-me left
printf "recoverable\n" > conflict-lab/recover.txt
git -C conflict-lab add recover.txt
git -C conflict-lab commit -m "recoverable work"
LOST_OID=$(git -C conflict-lab rev-parse HEAD)
git -C conflict-lab switch left
git -C conflict-lab branch -D recover-me
git -C conflict-lab reflog --all --date=iso
git -C conflict-lab cat-file -t "$LOST_OID"
git -C conflict-lab branch recovered "$LOST_OID"
test "$(git -C conflict-lab rev-parse recovered)" = "$LOST_OID"

6. Failure 5 — bad release tag

git -C integrator fetch origin
BAD=$(git -C central.git rev-parse refs/heads/trunk~1)
GOOD=$(git -C central.git rev-parse refs/heads/trunk)
git -C integrator tag -a v1.1-bad -m "bad release drill" "$BAD"
printf "expected-good=%s\nactual=%s\n" "$GOOD" "$(git -C integrator rev-parse v1.1-bad^{commit})"

If the wrong name is already published, preserve evidence and create a corrected new release/version rather than silently moving the original tag. An unpublished local tag can be deleted/recreated after verification.

7. Failure 6 — shallow CI provenance gap

git clone --depth=1 --no-tags --branch trunk "$SERVER_URL" shallow-failure
git -C shallow-failure rev-parse --is-shallow-repository
git -C shallow-failure tag --list
git -C shallow-failure describe --tags HEAD
echo "describe exit=$?"
git -C shallow-failure fetch --unshallow origin
git -C shallow-failure fetch --tags origin
git -C shallow-failure describe --tags --always HEAD

The exact commit can be valid while release-name context is incomplete.

8. Failure 7 — fake secret leak

git clone "$SERVER_URL" secret-drill
git -C secret-drill config user.name "Secret Drill"
git -C secret-drill config user.email "secret-drill@example.invalid"
printf "FAKE_TOKEN=TRAINING_ONLY_NOT_A_SECRET\n" > secret-drill/leaked.env
git -C secret-drill add leaked.env
git -C secret-drill commit -m "inject fake secret marker"
git -C secret-drill log -S'TRAINING_ONLY_NOT_A_SECRET' --all --oneline -- leaked.env

For a real credential: revoke/rotate immediately; preserve evidence; inventory refs/clones/forks/caches; then rewrite affected history in a fresh disposable remediation clone with a current tool such as git filter-repo. Rewriting changes commit IDs and requires coordination.

# Optional external tool; read current git-filter-repo documentation first.
git filter-repo --sensitive-data-removal --invert-paths --path leaked.env

Do not use git filter-branch as the default remediation path.

9. Failure 8 — missing loose object in an isolated repository

git init -b trunk object-drill
git -C object-drill config user.name "Object Drill"
git -C object-drill config user.email "object-drill@example.invalid"
printf "important\n" > object-drill/important.txt
git -C object-drill add important.txt
git -C object-drill commit -m "important object"
BLOB=$(git -C object-drill hash-object important.txt)
OBJ="object-drill/.git/objects/${BLOB%${BLOB#??}}/${BLOB#??}"
cp "$OBJ" "$OBJ.backup"
rm "$OBJ"
git -C object-drill fsck --full
echo "expected fsck failure=$?"
mv "$OBJ.backup" "$OBJ"
git -C object-drill fsck --full

10. Failure 9 — performance diagnosis targets the wrong layer

git status --porcelain >/dev/null
git count-objects -vH
git rev-list --count --all
git config --show-origin --get core.fsmonitor
git config --show-origin --get core.untrackedCache

A slow status in a huge worktree is not automatically a packfile problem. Select optimization only after identifying the layer.

11. Failure 10 — automation points to the wrong remote/ref

git -C release-bot remote -v
git -C release-bot remote get-url --push origin
git -C release-bot rev-parse --verify HEAD
git -C release-bot push --dry-run --porcelain origin HEAD:refs/heads/release-state

If the URL or ref differs from the runbook, stop before mutation.

12. Failure 11 — Git restore succeeds but external dependencies are absent

A full bundle can reconstruct Git refs/objects while LFS object bytes, submodule repositories, server hooks, access rules, CI variables/secrets, hosted issues/PRs/releases/packages, or deployment state remain missing. Restore acceptance must inventory these separately.

13. Red-zone incident actions

14. Knowledge check

15. Summary

The failure drills preserve original causes while exercising concurrency protection, conflict state, ref recovery, release discipline, shallow provenance repair, secret incident order, object recovery, performance-layer diagnosis, target verification, and dependency-aware restore.

Authoritative references

• git-push
• git-reflog
• history-filtering warning
• git-fsck
• git-fetch