Production Capstone: Build, Test, Secure, Publish, and Optimize a Multi-Module JVM Platform: Failure Injection, Troubleshooting, and Recovery Drill

1. One diagnostic sequence for every failure

Use the same order so investigation does not become guesswork:

1. Preserve concise failure output and the exact command.
2. Confirm Wrapper / Maven-or-Gradle / JDK identity.
3. Inspect settings, build scripts, toolchain and dependency declarations.
4. Inspect dependency graph and task/lifecycle graph.
5. Inspect repository, verification metadata, caches and filesystem outputs.
6. Inspect test/compiler/plugin/publication evidence.
7. Apply the least destructive correction.
8. Re-run the controlled failing path, then the required clean verification.
9. Record root cause, detection signal, corrective action, verification and prevention.

2. Drill A — wrong target/toolchain contract

Prediction: changing Java release to an impossible value should fail compilation before publication. Preserve the valid file first:

cp core/build.gradle.kts evidence/core-build.before-toolchain-drill.kts
python - <<'PY'
from pathlib import Path
p=Path('core/build.gradle.kts')
s=p.read_text()
p.write_text(s.replace('options.release.set(17)','options.release.set(99)'))
PY
set +e
./gradlew :core:clean :core:compileJava 2>&1 | tee evidence/drill-a-toolchain.log
status=${PIPESTATUS[0]}
set -e
test "$status" -ne 0
cp evidence/core-build.before-toolchain-drill.kts core/build.gradle.kts
./gradlew :core:clean :core:compileJava
javap -verbose core/build/classes/java/main/dev/academy/capstone/MessageNormalizer.class \
  | grep 'major version' | tee evidence/drill-a-repaired-major.txt

Root cause: declared compiler target is outside the selected toolchain’s supported release set. Preventive control: version/toolchain policy plus bytecode-major verification. Do not “fix” this by changing the runtime contract without product approval.

3. Drill B — dependency checksum mismatch / supply-chain incident

Prediction: if the reviewed Commons Lang checksum is altered, dependency verification should reject the artifact even though its coordinate/version still match. Preserve the metadata first:

cp gradle/verification-metadata.xml evidence/verification.before-drill.xml
python - <<'PY'
from pathlib import Path
p=Path('gradle/verification-metadata.xml')
s=p.read_text()
needle='org.apache.commons'
pos=s.find(needle)
if pos < 0:
    raise SystemExit('Commons group not found; inspect metadata before drill')
# Alter only the first sha256 value after the known group marker.
sha_pos=s.find('value="', pos)
if sha_pos < 0:
    raise SystemExit('checksum value not found')
start=sha_pos+7
old=s[start:start+64]
s=s[:start]+('0' if old[0] != '0' else '1')+old[1:]+s[start+64:]
p.write_text(s)
PY
set +e
./gradlew --refresh-dependencies :core:compileJava 2>&1 | tee evidence/drill-b-verification.log
status=${PIPESTATUS[0]}
set -e
test "$status" -ne 0

# Investigate; do NOT regenerate metadata to force green.
diff -u evidence/verification.before-drill.xml gradle/verification-metadata.xml || true
cp evidence/verification.before-drill.xml gradle/verification-metadata.xml
./gradlew --refresh-dependencies :core:compileJava

Incident response: preserve requested coordinate, repository configuration, verification failure, cached artifact hash, and upstream change evidence. A checksum mismatch can mean a local drill, a changed artifact, repository corruption, or tampering. The correction is restoring independently reviewed policy—not bypassing verification.

4. Drill C — false-green test lifecycle

Prediction: disconnecting integrationTest from check lets check finish without the intended integration suite. That is a build-model failure even if exit code is zero:

cp app/build.gradle.kts evidence/app-build.before-test-drill.kts
python - <<'PY'
from pathlib import Path
p=Path('app/build.gradle.kts')
s=p.read_text()
old='tasks.check {\n    dependsOn(testing.suites.named("integrationTest"))\n}'
if old not in s:
    raise SystemExit('expected check wiring not found')
p.write_text(s.replace(old,'// DRILL: integrationTest intentionally disconnected from check'))
PY
rm -rf app/build/test-results/integrationTest
./gradlew :app:clean :app:check --info | tee evidence/drill-c-false-green.log
if test -d app/build/test-results/integrationTest; then
  echo 'Unexpected: integration report exists; inspect task graph' >&2
  exit 1
fi
./gradlew :app:check --dry-run | tee evidence/drill-c-dry-run.txt
cp evidence/app-build.before-test-drill.kts app/build.gradle.kts
./gradlew :app:clean :app:check
test -d app/build/test-results/integrationTest

Root cause: lifecycle/task wiring omitted required verification. Preventive control: CI must assert report/suite presence or task-graph evidence, not only check exit status.

5. Drill D — distinguish stale workspace/cache state from source truth

Prediction: warm derived state can change task outcomes (UP-TO-DATE/FROM-CACHE) without changing accepted artifact semantics. Compare warm and isolated no-cache executions; do not clear global caches:

./gradlew :core:jar --build-cache --info | tee evidence/drill-d-warm.log
mkdir -p .drill-gradle-fresh
GRADLE_USER_HOME="$PWD/.drill-gradle-fresh" \
  ./gradlew --no-build-cache :core:clean :core:jar --info \
  | tee evidence/drill-d-fresh.log
sha256sum core/build/libs/core-1.0.0.jar | tee evidence/drill-d-core.sha256

If warm and clean behavior disagree functionally, inspect declared task inputs/outputs and cache provenance. Deleting ~/.gradle would erase evidence and is not a root-cause analysis.

6. Drill E — staged artifact checksum mismatch blocks promotion

Prediction: mutating the staged JAR after acceptance must fail promotion against the preserved accepted checksum:

jar="staging-repo/dev/academy/capstone/core/1.0.0/core-1.0.0.jar"
cp "$jar" evidence/core-1.0.0.accepted.jar
accepted="$(sha256sum evidence/core-1.0.0.accepted.jar | awk '{print $1}')"
printf 'harmless-drill-byte' >> "$jar"
actual="$(sha256sum "$jar" | awk '{print $1}')"
printf 'accepted=%s\nactual=%s\n' "$accepted" "$actual" | tee evidence/drill-e-mismatch.txt
if test "$accepted" = "$actual"; then
  echo 'drill failed: checksum unexpectedly equal' >&2
  exit 1
fi
# Promotion is blocked. Restore the preserved accepted artifact, not a rebuild.
cp evidence/core-1.0.0.accepted.jar "$jar"
test "$(sha256sum "$jar" | awk '{print $1}')" = "$accepted"

This recovery intentionally uses the already-built accepted artifact. Rebuilding from source is a different action: it can prove reproducibility, but it does not preserve the identity of an already-approved candidate unless the resulting checksum is independently matched.

7. Drill F — fake credential containment

Prediction: a fake credential committed into project configuration should be detected by the source/evidence scan and removed. Use only an unmistakably non-secret value:

cat > evidence/fake-credential-drill.properties <<'EOF'
repoUser=drill-user
repoPassword=FAKE_DO_NOT_USE_12345
EOF

grep -RIn 'FAKE_DO_NOT_USE_12345' . 
rm evidence/fake-credential-drill.properties
if grep -RIn 'FAKE_DO_NOT_USE_12345' .; then
  echo 'fake credential still present' >&2
  exit 1
fi

In a real incident, removal from the working tree is insufficient: revoke/rotate the credential, purge it from inappropriate history/log/artifact stores according to policy, determine exposure scope, and review authorization. This drill only demonstrates containment mechanics without creating a real secret.

8. Rebuild, cache restore, and artifact restore are three different recovery actions

9. Incident record template for every drill

incident:
  id: DRILL-X
  prediction: "What should fail and why?"
  exact_command: "..."
  root_cause: "..."
  detection_signal: "log/report/checksum/task graph"
  preserved_evidence:
    - "..."
  corrective_action: "least destructive change"
  verification:
    - "controlled retry"
    - "clean required gate"
  time_to_recovery_estimate: "record observed/estimated duration"
  preventive_control: "policy/test/automation/ownership improvement"
  owner: "role, not an individual secret"

The time-to-recovery field is measured or estimated by the learner; this static lesson does not invent a timing number for an environment it has not executed.

10. Recovery acceptance checklist

Knowledge check

Official references and version notes

• Gradle 9.7.1 release notes — pinned Gradle baseline for the capstone.
• Gradle Wrapper and release checksums — wrapper/distribution identity and integrity.
• Gradle JVM toolchains — build JVM versus compiler/test launchers.
• Dependency verification — checksum/signature verification metadata and review workflow.
• Repository declarations and content filtering — dependency origin controls.
• Java testing and JVM Test Suite — unit/integration verification model.
• Maven Publish — Gradle publication to Maven repository layout and clean Maven consumption.
• Build Cache and Configuration Cache — bounded build-state reuse.
• Gradle performance guidance — measurement-first optimization and profiling.
• Apache Maven release history — Maven 3.9.16 GA consumer baseline; Maven 4.0.0-rc-6 remains pre-GA.
• Apache Maven Wrapper 3.3.4 — stable Maven Wrapper baseline.
• Maven Compiler Plugin 3.15.0 — clean Maven consumer compilation baseline.
• JUnit 6.1.3 — Java 17+ test runtime baseline.
• Apache Commons Lang release notes — Commons Lang 3.20.0 dependency baseline.

Version-sensitive statements were rechecked against primary documentation on 2026-08-24. Mandatory work remains local/free. Hosted CI, commercial build analytics, production repository managers, remote caches, and real signing/credential systems are optional integration boundaries only.