Production Capstone: Design, Secure, Scale, and Recover a Governed Git Workflow: Final Operational Review and Handoff

1. Final checkpoint — operate, break, recover, and hand off Atlas Git

This lesson starts from a fresh directory so the final examination is reproducible. You will establish the target topology, capture acceptance evidence, inject at least four failures, recover without hiding causes, prove an offline restore, and leave a runbook another operator can follow.

2. Target architecture and threat model

flowchart TD
H[Human contributor] --> T[topic/*]
T --> I[Integrator validation]
I --> C[(central.git)]
C --> CI[Detached CI checkout]
CI --> A[Artifact + provenance]
C --> RB[Release-state bot]
C --> G[GitOps desired state]
G --> AP[Applied-state marker]
C --> BK[Verified full bundle]
ATT[Threats: stale writers, leaked credentials, wrong refs, corrupt copies] --> I
ATT --> RB
ATT --> CI
ATT --> BK

3. Fresh checkpoint setup

mkdir git-governance-handoff
cd git-governance-handoff
git init --bare -b trunk central.git
git clone central.git source
git -C source config user.name "Learner Example"
git -C source config user.email "learner@example.invalid"
mkdir -p source/app source/environments/lab
printf "message=handoff\n" > source/app/app.conf
printf "replicas=2\nversion=1\n" > source/environments/lab/service.env
cat > source/validate.sh <<'EOF2'
#!/bin/sh
replicas=$(sed -n 's/^replicas=//p' environments/lab/service.env)
version=$(sed -n 's/^version=//p' environments/lab/service.env)
test -n "$replicas" && test "$replicas" -ge 1 && test -n "$version"
EOF2
git -C source add .
git -C source commit -m "seed handoff source"
git -C source tag -a v1.0 -m "handoff v1.0"
printf "message=handoff-build\n" > source/app/app.conf
git -C source commit -am "prepare handoff build"
git -C source push -u origin trunk
git -C source push origin v1.0
SERVER_URL="file://$(pwd)/central.git"

4. Establish integrator and release automation

git clone "$SERVER_URL" integrator
git -C integrator config user.name "Atlas Integrator"
git -C integrator config user.email "integrator@example.invalid"
git -C integrator switch -c release-state trunk
SOURCE_OID=$(git -C integrator rev-parse trunk)
printf "source_commit=%s\nstatus=ready\n" "$SOURCE_OID" > integrator/release.env
git -C integrator add release.env
git -C integrator commit -m "initialize handoff release state"
git -C integrator push -u origin release-state
git -C integrator switch trunk
git clone "$SERVER_URL" release-bot
git -C release-bot config user.name "Atlas Release Automation"
git -C release-bot config user.email "release-bot@example.invalid"
git -C release-bot switch release-state

5. Preflight and predictions

git --version
git -C central.git show-ref
git -C central.git fsck --full
git -C integrator status --porcelain=v2 --branch
git -C integrator config --list --show-origin --show-scope
git -C release-bot remote -v

1. A normal stale bot push should be rejected.
2. Deleting a branch name should not immediately erase a still-existing commit.
3. A depth-1 no-tags runner can build exact HEAD while lacking an older tag.
4. A full bundle restore should reproduce included refs/OIDs but not hosted/server metadata.

6. Exact detached CI build and provenance

git clone --depth=1 --no-tags --branch trunk "$SERVER_URL" runner
BUILD_OID=$(git -C central.git rev-parse refs/heads/trunk)
git -C runner switch --detach "$BUILD_OID"
COMMIT=$(git -C runner rev-parse --verify HEAD^{commit})
TREE=$(git -C runner rev-parse --verify HEAD^{tree})
CLEAN=$(test -z "$(git -C runner status --porcelain=v1)" && echo true || echo false)
printf "commit=%s\ntree=%s\nclean=%s\n" "$COMMIT" "$TREE" "$CLEAN" > provenance.txt
sh -c 'cd runner && sh validate.sh'
test "$COMMIT" = "$BUILD_OID"
test "$CLEAN" = true

7. GitOps-style applied-state baseline

mkdir deployed
printf "none\n" > deployed/applied-commit.txt
if sh -c 'cd runner && sh validate.sh'; then
  cp runner/environments/lab/service.env deployed/service.env
  printf "%s\n" "$BUILD_OID" > deployed/applied-commit.txt
fi
test "$(cat deployed/applied-commit.txt)" = "$BUILD_OID"

8. Failure A — stale release bot and safe recovery

git clone "$SERVER_URL" operator
git -C operator config user.name "Handoff Operator"
git -C operator config user.email "handoff@example.invalid"
git -C operator switch release-state
printf "operator=review\n" > operator/operator.env
git -C operator add operator.env
git -C operator commit -m "operator handoff update"
git -C operator push origin release-state
printf "bot=stale\n" > release-bot/stale.env
git -C release-bot add stale.env
git -C release-bot commit -m "stale bot update"
git -C release-bot push origin HEAD:refs/heads/release-state
echo "expected rejection=$?"
git -C release-bot fetch origin release-state
git -C release-bot merge --no-edit refs/remotes/origin/release-state
git -C release-bot push --dry-run --porcelain origin HEAD:refs/heads/release-state
git -C release-bot push origin HEAD:refs/heads/release-state

9. Failure B — merge conflict and abort

git clone "$SERVER_URL" conflict
git -C conflict config user.name "Handoff Conflict"
git -C conflict config user.email "handoff-conflict@example.invalid"
git -C conflict switch -c conflict-a
printf "policy=A\n" > conflict/policy.conf
git -C conflict add policy.conf
git -C conflict commit -m "policy A"
git -C conflict switch -c conflict-b HEAD~1
printf "policy=B\n" > conflict/policy.conf
git -C conflict add policy.conf
git -C conflict commit -m "policy B"
git -C conflict switch conflict-a
git -C conflict merge conflict-b
git -C conflict status --short
git -C conflict ls-files -u
git -C conflict merge --abort
git -C conflict status --short

10. Failure C — lost branch recovery

git -C conflict switch -c temporary-recovery conflict-a
printf "handoff recovery\n" > conflict/recovery.txt
git -C conflict add recovery.txt
git -C conflict commit -m "temporary recoverable commit"
RECOVER_OID=$(git -C conflict rev-parse HEAD)
git -C conflict switch conflict-a
git -C conflict branch -D temporary-recovery
git -C conflict cat-file -t "$RECOVER_OID"
git -C conflict branch recovered-handoff "$RECOVER_OID"
test "$(git -C conflict rev-parse recovered-handoff)" = "$RECOVER_OID"

11. Failure D — shallow CI release-name failure

git -C runner describe --tags HEAD
echo "expected describe failure=$?"
git -C runner fetch --unshallow origin
git -C runner fetch --tags origin
git -C runner describe --tags --always HEAD
test "$(git -C runner rev-parse HEAD)" = "$BUILD_OID"

12. Failure E — invalid desired state does not advance applied state

printf "replicas=0\nversion=2\n" > source/environments/lab/service.env
git -C source add environments/lab/service.env
git -C source commit -m "inject invalid desired state"
git -C source push origin trunk
BAD_OID=$(git -C source rev-parse HEAD)
git -C runner fetch origin trunk
git -C runner switch --detach "$BAD_OID"
APPLIED_BEFORE=$(cat deployed/applied-commit.txt)
if sh -c 'cd runner && sh validate.sh'; then
  printf "%s\n" "$BAD_OID" > deployed/applied-commit.txt
else
  echo "validation failed; applied marker preserved"
fi
test "$(cat deployed/applied-commit.txt)" = "$APPLIED_BEFORE"

13. Failure F — corrupt object recovery in isolated repository

git init -b trunk handoff-object
git -C handoff-object config user.name "Handoff Object"
git -C handoff-object config user.email "handoff-object@example.invalid"
printf "restore me\n" > handoff-object/value.txt
git -C handoff-object add value.txt
git -C handoff-object commit -m "object recovery proof"
OID=$(git -C handoff-object hash-object value.txt)
OBJ="handoff-object/.git/objects/${OID%${OID#??}}/${OID#??}"
cp "$OBJ" "$OBJ.saved"
rm "$OBJ"
git -C handoff-object fsck --full
echo "expected failure=$?"
mv "$OBJ.saved" "$OBJ"
git -C handoff-object fsck --full

14. Secret incident tabletop

Trigger: FAKE_TOKEN=TRAINING_ONLY_NOT_A_SECRET appears in history.
1. Preserve evidence and identify affected refs/copies.
2. Revoke/rotate the real credential (fictional here).
3. Stop propagation.
4. Rewrite in a fresh clone with current git-filter-repo guidance.
5. Verify marker absence from intended rewritten refs.
6. Coordinate replacement refs and downstream clone/fork/cache cleanup.
7. Add prevention/scanning controls.

15. Performance and integrity acceptance

git -C integrator count-objects -vH
git -C integrator commit-graph write --reachable --changed-paths
git -C integrator commit-graph verify
git -C integrator maintenance run --task=commit-graph
git -C integrator fsck --full

Auxiliary maintenance is verified and does not use aggressive pruning.

16. Final backup/restore proof

git -C central.git bundle create final-handoff.bundle --all
git init --bare -b trunk final-bundle-verifier.git
git -C final-bundle-verifier.git bundle verify ../central.git/final-handoff.bundle
git clone --mirror central.git/final-handoff.bundle final-restore.git
git -C central.git for-each-ref --sort=refname --format='%(refname) %(objectname)' > handoff-source-refs.txt
git -C final-restore.git for-each-ref --sort=refname --format='%(refname) %(objectname)' > handoff-restored-refs.txt
git diff --no-index -- handoff-source-refs.txt handoff-restored-refs.txt
git -C final-restore.git fsck --full

17. External restore dependency inventory

Core Git: refs/OIDs, objects/connectivity, signatures where required.
External: Git LFS store, every submodule repo, server hooks, permissions/protected refs,
required checks, CI variables/secrets/webhooks, hosted issues/PRs/releases/packages,
and GitOps/deployment applied-state storage.

18. Final handoff package

19. Minimal day-2 command runbook

Before integration: fetch; inspect exact topic OID/range; validate; record target ref.
Before release: validate approved commit; create immutable tag; verify signature if required.
CI: detach exact commit; assert clean; record commit/tree/toolchain provenance.
Bot: fetch owned ref; generate next commit; dry-run; normal fast-forward push.
Incident: preserve refs/reflogs/config/object evidence; do not prune/force blindly.
Recovery: verify bundle; restore separately; compare refs/OIDs; fsck; restore external dependencies.

20. Verification checklist

• Exact source OIDs recorded for integration/release/build.
• Automation uses dedicated ref and normal push.
• At least four failures injected and recovered with evidence.
• Applied state advances only on successful validation.
• Integrity checks pass after recovery.
• Maintenance is measured and recovery-aware.
• Offline bundle restore reproduces intended refs/OIDs.
• External restore dependencies are documented.
• No real credentials or private keys appear in examples.
• Hosted enforcement requirements are assigned to the chosen platform.

21. Cleanup

cd ..
pwd
rm -rf git-governance-handoff

PowerShell equivalent: Set-Location ..; Remove-Item -Recurse -Force git-governance-handoff.

22. Knowledge check

23. Final operational review

The course now forms one operating model: fundamentals/object state → commits/branches/remotes/collaboration → history editing/recovery → releases/worktrees/dependencies/large repositories → forensics/hooks/plumbing/refs/merge engineering → trust/integrity → maintenance → patch workflows → migration/offline transfer → CI/CD/GitOps → governed production operations.

Next, implement the server-side controls in GitHub/GitLab, translate portable checkout/provenance rules into CI systems, and connect desired-state repositories to IaC/Kubernetes/GitOps controllers. Those products add orchestration and enforcement; the Git safety model remains foundational.

24. Course completion summary

You have moved from command fluency to an auditable change-control system that can be inspected, automated, stressed, recovered, migrated, and handed off. The production discipline is explicit state, immutable provenance, least privilege, guarded ref updates, preserved evidence, measured maintenance, verified recovery, and clear boundaries between Git and the surrounding platform.

Authoritative references

• Git documentation
• Pro Git
• git-push
• git-fsck
• git-bundle