JENKINS_HOME, Filesystem Layout, Controller Configuration, System Settings, Tools, and Global Properties: Concepts, Architecture, and Mental Model

1. The problem: Jenkins is easy to start and easy to misunderstand

A Jenkins controller is a long-running stateful service. The executable can be replaced in minutes, but the controller becomes valuable because it accumulates configuration, item definitions, build records, plugin state, users, cryptographic keys, and operational history. If that durable state is mixed with throwaway workspaces or if administrators edit internal files casually, routine maintenance becomes a recovery incident.

Chapter 02 separated the replaceable container image from the persistent /var/jenkins_home volume. This chapter goes one layer deeper: what is inside that persistent boundary, which parts are authoritative, which parts are reconstructable, and which parts are dangerous to expose or edit directly.

2. Mental model: controller process → home → durable state → recovery boundary

The controller process reads configuration from JENKINS_HOME, loads core and plugins, reconstructs items and historical records, exposes configured global behavior, then coordinates builds. A restart replaces process memory; it should not erase durable state. A container replacement replaces the container filesystem; it should not erase the mounted home volume.

flowchart TD
  A[Jenkins core + Java process] --> B[JENKINS_HOME]
  B --> C[Global configuration XML and plugin state]
  B --> D[Jobs, folders, build records, artifacts]
  B --> E[Plugins and plugin data]
  B --> F[Secrets and cryptographic keys]
  B --> G[Tool/cache/runtime files]
  C --> H[Controller behavior after restart]
  D --> H
  E --> H
  F --> H
  H --> I[Queue and agent execution]
  I --> J[Agent workspace - execution state]
  I --> K[External SCM / registry / cloud / deployment state]
  B --> L[Backup + restore boundary]
  F --> M[Protected recovery material stored separately]

The diagram deliberately separates an agent workspace and an external deployment target from controller home. A successful restore of controller files does not reconstruct a deleted cloud resource, and a clean agent workspace should be reproducible from source and declared dependencies rather than from a controller backup.

3. A practical JENKINS_HOME anatomy

Exact files vary with Jenkins core, plugins, installation mode, and history. Never assume every controller has the same tree. The safe approach is to inspect the actual instance and understand categories rather than memorize one sample listing.

4. Discover home path and ownership before touching files

Inside the official container used in this course, JENKINS_HOME is /var/jenkins_home. Native packages use different defaults. Jenkins also shows the active home directory under Manage Jenkins → System. Inspect the running instance instead of assuming a path copied from a tutorial.

docker exec jenkins-ch02-controller sh -lc '
  printf "JENKINS_HOME=%s\n" "$JENKINS_HOME"
  id
  stat -c "owner=%U:%G mode=%a path=%n" "$JENKINS_HOME"
  find "$JENKINS_HOME" -maxdepth 1 -mindepth 1 -printf "%f\n" | sort | sed -n "1,80p"
'

If your Chapter 02 container no longer exists, use the standalone setup in Lesson 2. Notice what the command does not do: it does not recursively print XML, credentials, tokens, secret files, or build logs. File names, ownership, size, and timestamps are normally enough for a first inventory.

5. Configuration has an owner and a persistence path

Manage Jenkins → System changes global controller settings and plugin-provided system settings. Manage Jenkins → Tools configures tool installations/providers. Plugins can add their own persistent files or sections. Later in the course, Jenkins Configuration as Code (JCasC) will provide a version-controlled source for supported configuration. These are different configuration ownership models.

The critical production question is not merely “where is the value stored?” but “what is the accountable source of truth, who changes it, and how would we reconstruct it after loss?”

6. Durable state, reconstructable state, and ephemeral state

7. Backup scope is a recovery-design decision

Jenkins documentation recommends backups and, importantly, restore validation. A full home snapshot is simplest conceptually, while selective backups can reduce size and time. Consistent filesystem snapshots are preferable when available because they avoid copying different files at different points in time.

Secret-key material deserves a separate trust boundary. The credentials/security guidance warns that anyone who obtains both encrypted Jenkins data and its secret keys may recover the protected information. This course therefore keeps secret-key archives separate from ordinary controller backup archives even though both are required for a full restore.

8. Read-only inspection patterns

# Size by top-level directory without reading secret contents
docker exec jenkins-ch02-controller sh -lc '
  du -sh "$JENKINS_HOME"/* 2>/dev/null | sort -h | tail -20
'

# Inspect only safe metadata for selected files
docker exec jenkins-ch02-controller sh -lc '
  stat -c "%y %s %n" "$JENKINS_HOME/config.xml"
  find "$JENKINS_HOME/jobs" -maxdepth 3 -name config.xml -printf "%TY-%Tm-%Td %TH:%TM %s %p\n" 2>/dev/null | head -20
'

These commands answer “what exists, how large is it, and when did it change?” without dumping configuration into a terminal transcript. When troubleshooting, preserve this metadata before restarts or repair.

9. Common misconceptions

• “JENKINS_HOME is just job XML.” It also contains controller/global/plugin data, histories, cryptographic material, and other operational state.
• “A mounted volume is a backup.” Persistence protects against container replacement; it does not provide independent recovery from corruption or operator mistakes.
• “Workspace files are Jenkins state.” Workspaces are execution state and should usually be reproducible from source and declared dependencies.
• “If XML is human-readable, editing it live is supported.” Jenkins owns serialization; direct edits can be overwritten, ignored, or made inconsistent with in-memory state.
• “JCasC means backups are unnecessary.” Configuration as code does not automatically preserve all job histories, plugin data, encrypted records, or recovery keys.

10. Summary

• JENKINS_HOME is the controller’s durable operational state boundary.
• Configuration, build history, plugins, secret keys, caches/tools, workspaces, and external systems have different recovery requirements.
• Supported UI/configuration interfaces should own changes; raw filesystem edits are an exception handled with the controller stopped and a recovery plan.
• Backups are not credible until a restore is validated on an isolated controller.
• Secret-key recovery material is necessary but must be protected separately from ordinary backup evidence.

Knowledge check

Official references and version notes

• Configuring the System — current Jenkins home-directory and global system-configuration guidance.
• Managing Jenkins — current administrative surfaces for System, Tools, Plugins, status, and troubleshooting.
• System Information — controller system properties, environment variables, plugins, memory information, and diagnostics.
• Managing Tools — built-in tool-provider concepts and global tool configuration.
• Backing-up/Restoring Jenkins — backup scope, controller-key separation, and restore validation.
• Credentials security — why Jenkins secret-key material needs protection and separation from ordinary backups.
• Storing Secrets — technical description of Jenkins encryption keys under $JENKINS_HOME/secrets.
• Controller Isolation — current guidance for keeping routine builds off the built-in controller node.
• Jenkins LTS changelog and Java Support Policy — version/runtime assumptions for this chapter.