Tool Installations, JDKs, Maven, Gradle, Node.js, Docker CLI, and Reproducible Build Environments: Diagnostics, Failure Modes, Security, and Performance

1. Preserve the first failure before changing the environment

Toolchain incidents are easy to erase accidentally. Restarting on a different agent, updating PATH, reinstalling a tool, pulling a fresh image, or rerunning after cache warm-up can remove the original evidence. Record the job/build number, source SHA, agent name, OS/architecture, PATH/JAVA_HOME, resolved executable paths, tool versions, wrapper files, image identity, and failing command first.

2. Diagnostic sequence

1. Confirm controller core and Java baseline.
2. Confirm job/Jenkinsfile/source revision and build cause.
3. Identify queue item, assigned agent, label, and workspace.
4. Capture agent OS/architecture and Jenkins Remoting runtime Java.
5. Resolve the build executable with command -v/where and print exact versions.
6. Inspect wrapper/configuration/image identity.
7. Check dependency resolution and caches.
8. Compare artifact checksum or failure output.
9. Change one hypothesis at a time and rerun the smallest safe scope.

3. Failure mode: Jenkins runtime Java confused with build JDK

Symptom: an operator points the Jenkins agent process at an old application JDK because the project compiles with it, and Remoting no longer starts under a current Jenkins line.

Evidence: agent startup logs show the JVM used to launch Remoting; the application build never starts. This differs from a compiler error inside a running build.

Correction: run Jenkins controller/agent components on a supported Java runtime, then select the application build JDK separately through a named tool, environment path, wrapper/toolchain configuration, or controlled image.

4. Failure mode: PATH drift across agents

Symptom: build #41 passes on lab-linux-a and build #42 fails on lab-linux-b although source SHA is identical.

printf 'node=%s\n' "$NODE_NAME"
printf 'PATH=%s\n' "$PATH"
command -v java; java -version
command -v mvn; mvn -B -V --version

If paths/versions differ, the root cause is execution-environment drift, not source. Fix the image/tool definition or constrain labels; do not add random retries.

5. Failure mode: mutable installer or image changes behavior

Symptom: a clean agent downloads a newer tool or a mutable container tag resolves to different content and a previously green build changes behavior.

Evidence: compare installer logs, tool version output, image repo digest, and timestamps. A Jenkinsfile that did not change can still resolve a different tool if the referenced external name was mutable.

Correction: pin an explicit reviewed version/digest and document the update workflow. Preserve human-readable release identity alongside the immutable identifier.

6. Failure mode: wrapper absent, executable bit lost, or wrapper tampered

Symptom: ./mvnw or ./gradlew fails, so someone proposes using system Maven/Gradle instead.

Diagnosis: compare the exact source SHA, wrapper files, permissions, wrapper properties, distribution URL/checksum, and recent changes. The wrapper is part of the project’s executable toolchain contract.

7. Failure mode: Docker socket turns a build into a host administrator

Symptom: a supposedly isolated test job can start privileged containers or mount arbitrary host paths because the agent exposes the host daemon socket.

Correction: remove that capability from general-purpose agents, rebuild the boundary around a dedicated isolated builder, and rotate/review credentials that may have been reachable. Treat this as a security-boundary failure, not a minor Pipeline syntax problem.

8. Performance: measure tool download and cache cost

Clean ephemeral agents improve isolation but can repeatedly download JDKs, Maven/Gradle distributions, Node packages, and dependencies. Before adding executors or parallel stages, measure queue time separately from tool/bootstrap/download time and build time.

Safe optimizations include pre-baked reviewed images, repository proxies, bounded read-only caches where appropriate, dependency-lock enforcement, and cache keys tied to relevant lockfiles. Never let a shared writable cache become an unreviewed cross-trust code channel.

9. Intentionally broken example: wrong Maven wins PATH

Assume /opt/maven-old/bin appears before the Jenkins-selected Maven directory:

command -v mvn
mvn -B -V --version
printf '%s\n' "$PATH" | tr ':' '\n'

If the version is wrong, compare the Pipeline/tool configuration and environment construction. Fix the ownership conflict so exactly one intended strategy controls Maven. Do not merely append another directory and hope shell ordering remains stable.

10. Security and evidence checklist

• No credentials in PATH/version logs.
• No untrusted build on the built-in controller node.
• No unrestricted Docker socket on general-purpose agents.
• No mutable-only tool/image identity for release-sensitive builds.
• No wrapper fallback that hides integrity errors.
• No “retry until green” before preserving the first failure.

Knowledge check

Official references and version notes

• Jenkins LTS changelog — current LTS line and tested Java configurations.
• Java Support Policy — Java versions supported for running Jenkins controller, agents, and CLI.
• Upgrade to Java 21 — controller/agent runtime inspection and migration guidance.
• Declarative Pipeline tools directive — preconfigured JDK, Maven, and Gradle tool installations and PATH behavior.
• Pipeline examples — selecting named JDK/Maven installations and recording versions.
• Using Docker with Pipeline — containerized execution environment, image selection, caching, and Docker Pipeline prerequisites.
• Docker Pipeline plugin — plugin release and compatibility information for Pipeline container steps.
• Apache Maven Wrapper and Gradle Wrapper — project-owned build-tool version selection.
• Node.js downloads and project lockfile documentation — runtime/package-manager pinning belongs with the project or build image.