Pipeline Fundamentals, Jenkinsfile, Pipeline Engine, Durable Execution, Nodes, Workspaces, and Stages: Concepts, Architecture, and Mental Model

1. The problem: a shell script is not a delivery control plane

A long shell script can compile, test, and deploy software, but by itself it has no Jenkins-aware model of queueing, agents, stage boundaries, approvals, restarts, build history, or retained evidence. Jenkins Pipeline solves a different problem: it lets the controller persist and coordinate a delivery program while individual steps execute where their required resources exist.

That distinction matters during failure. If the controller restarts, Pipeline may resume from persisted program state. If an agent disappears, a node-bound step may need reconnection or reallocation depending on the step and design. If an external deployment API accepted a request before the restart, Jenkins cannot magically reverse that external side effect. Durable orchestration is not transactional rollback.

2. Mental model: source → program → allocation → evidence

Start with the Jenkinsfile stored at an exact SCM revision. Jenkins loads that definition for a particular Pipeline build, interprets the Pipeline program, records stage/step progress, requests nodes when steps require executors/workspaces, and keeps a persisted run record under Jenkins controller state.

flowchart TD
  A[Jenkinsfile at exact SCM SHA] --> B[Pipeline job + build number]
  B --> C[Controller Pipeline program / CPS state]
  C --> D[Stage and step graph]
  D --> E{Step needs an agent?}
  E -->|yes| F[Queue + node/label allocation]
  F --> G[Workspace on selected agent]
  G --> H[Durable external process / tool step]
  E -->|no| I[Controller-managed Pipeline step]
  H --> J[Logs / reports / artifact bytes]
  I --> J
  J --> K[Persisted run result + evidence]
  K --> L[External outcome verification]

The controller owns the run identity and orchestration state. The agent owns the process and filesystem context for node-bound work. Artifacts copied into Jenkins or an external repository are a different durability layer from the workspace that produced them.

3. Define the objects before using them

4. Groovy orchestration is not “running on the agent”

Pipeline Groovy is interpreted by the Pipeline: Groovy/CPS engine on the controller. A node block or Declarative agent allocates an executor/workspace so specific steps can run remotely. External commands launched by sh, bat, or powershell execute on that agent, not inside the controller JVM.

pipeline {
  agent { label 'lab-linux' }
  stages {
    stage('Observe') {
      steps {
        echo "run=${env.JOB_NAME} #${env.BUILD_NUMBER}"
        sh '''
          set -eu
          printf 'node=%s\nworkspace=%s\n' "$NODE_NAME" "$WORKSPACE"
          uname -a
        '''
      }
    }
  }
}

The echo step and Pipeline program are controller-orchestrated; the shell process runs in the allocated workspace. Keep controller-side Groovy small and orchestration-focused. Heavy CPU/file/network processing belongs in external tools on agents.

5. Durability means persisted continuation, not infinite survivability

Jenkins Pipeline was designed to survive planned and many unplanned controller restarts. Pipeline frequently records execution state to controller storage. Current Jenkins also exposes speed/durability tradeoffs: more aggressive performance settings reduce disk writes but can reduce survivability after abrupt shutdown.

A durable step such as a long-running shell command is implemented so Jenkins can reconnect to or monitor the external process across controller interruptions when the agent/process remains viable. Controller-managed steps such as sleep can also be persisted. But an agent disk loss, deleted workspace, killed external process, revoked credential, or changed external API state is a separate failure domain.

6. Workspace is a lease on scratch state

When Jenkins allocates a node, it chooses a workspace path for the job. That directory may survive between builds on a static agent, may disappear immediately on an ephemeral agent, and may be cleaned by policy. Do not use workspace leftovers as a database or as proof that an artifact was retained.

7. agent any, labels, and agent none

A top-level Declarative agent any usually holds one node/workspace across stages. agent none allocates no global agent; each stage that needs a workspace declares its own agent. The second pattern can reduce idle executor consumption around approvals or controller-only coordination, but it also means stages may execute on different machines and therefore cannot assume workspace continuity.

pipeline {
  agent none
  stages {
    stage('Build') {
      agent { label 'linux-builder' }
      steps { sh './build.sh' }
    }
    stage('Approve') {
      steps { input message: 'Promote synthetic build?' }
    }
  }
}

Later chapters go deeper into Declarative syntax. Here the important point is resource ownership: no stage should hold an executor just to wait for a human if it does not need a workspace.

8. Read-only inspection before editing a Pipeline

For an existing build, capture the run URL/number and source revision first. Then inspect stage/step visualization, console log, node/workspace lines, artifacts, and installed Pipeline plugin versions. On an agent, use bounded non-secret commands:

printf 'job=%s\nbuild=%s\nnode=%s\nworkspace=%s\n' \
  "$JOB_NAME" "$BUILD_NUMBER" "$NODE_NAME" "$WORKSPACE"
git rev-parse HEAD
pwd
java -version

Do not dump all environment variables. Pipeline environments often include credentials or provider tokens.

9. Evidence chain for one Pipeline run

10. DevOps connection: durable orchestration is observable state

A production Pipeline should let an operator answer “what definition ran, where did it run, what was persisted, what was merely workspace state, and what side effects happened outside Jenkins?” If those questions cannot be answered after a restart, the automation is not operationally complete.

Knowledge check

Official references and version notes

• Jenkins LTS changelog — current LTS line and tested Java configurations.
• Jenkins Pipeline handbook — Pipeline concepts, Jenkinsfile, development tools, shared libraries, and execution model.
• Getting Started with Pipelines — durability, pausing, and the reasons Pipeline differs from Freestyle automation.
• Pipeline Syntax — Declarative pipeline, agent, stages, steps, options, and restart-related directives.
• Scaling Pipelines — Pipeline durability modes, persistence tradeoffs, and restart implications.
• Pipeline plugin — the aggregator and current Pipeline plugin suite.
• Pipeline: Declarative — Declarative Pipeline implementation and compatibility.
• Pipeline: Groovy — CPS execution model and controller-side Groovy interpretation.
• Pipeline: Job — persisted Pipeline run/job implementation.
• Pipeline: Nodes and Processes — node/workspace allocation and durable external process steps.
• Pipeline: Stage View — optional visualization; not the source of Pipeline execution truth.