Continuous Integration and Delivery Foundations, GitLab Pipeline Architecture, and Delivery Flow: Concepts, Architecture, and Mental Model

1. The problem: a green pipeline can still be misunderstood

Teams often begin CI/CD by copying a working .gitlab-ci.yml file. That can produce a green icon quickly, but it does not automatically produce an operational mental model. When something later fails, the team may not know whether GitLab rejected the configuration, declined to create a pipeline, omitted a job because of rules, queued a job with no eligible runner, assigned a runner whose executor could not prepare the environment, ran a command that exited non-zero, failed to upload an artifact, or successfully called a deployment API while the target application remained unhealthy. Those are different states owned by different layers.

The first skill in this course is therefore evidence tracing, not YAML memorization. A reproducible pipeline should let another engineer answer: what caused this pipeline, which exact commit was evaluated, which configuration was compiled, which jobs existed, which runner and executor handled a job, what the job actually produced, what GitLab stored, and whether anything outside GitLab changed. A pipeline that cannot answer those questions is difficult to debug, audit, secure, or recover.

2. Automation, CI, continuous delivery, and continuous deployment

CI/CD is a family of practices, not one switch. GitLab can automate all of them, but the purpose and evidence differ. Continuous integration asks whether an exact change integrates safely with the shared codebase. Continuous delivery asks whether a verified result is maintained in a deployable state. Continuous deployment goes further and automatically changes a target when policy allows it. Repository automation may do useful work that is neither CI nor deployment.

Mature delivery systems promote evidence and immutable outputs rather than merely chaining shell commands. Later chapters will add artifacts, registries, environments, approvals, OIDC, security reports, and policies. Chapter 01 only needs the conceptual separation so those later controls have a place in the model.

3. The causal GitLab CI/CD execution model

Start with intent. The intent may be a Git push, a merge request, a schedule, a user clicking New pipeline, an API request, a trigger token, or a parent/downstream pipeline. GitLab records the pipeline source and associates the run with a ref and commit. Before any runner executes a job, GitLab resolves the CI configuration, evaluates pipeline-level logic, constructs the job graph, and determines which jobs are eligible to exist. Only then can jobs enter the queue and be matched to runners.

            flowchart TD
              A[Commit or user / API intent] --> B[Pipeline source + ref + immutable SHA]
              B --> C[Resolve .gitlab-ci.yml + includes]
              C --> D[Validate and compile configuration]
              D --> E[workflow / rule decisions]
              E --> F[Pipeline ID + job graph]
              F --> G[Eligible job enters queue]
              G --> H[Matching runner selection]
              H --> I[Executor prepares workspace / container / pod]
              I --> J[Checkout + variables + job script]
              J --> K[Logs + status + reports + artifacts]
              K --> L[Environment / deployment record]
              L --> M[External target health + policy / audit evidence]
          

The arrows are operational boundaries. Valid YAML can still compile to no desired job. A pipeline can exist while a job remains pending because no runner matches its tags or protection state. A runner can start successfully while the job's toolchain is missing. A deployment API can return success while the application later fails a health check. When diagnosing, ask: which state definitely exists, and which next state is missing or wrong?

4. Core nouns: name the objects before debugging them

Precise language makes incident reports useful. “GitLab failed” is vague. “Pipeline 812 exists for source push and SHA …a41f; job 2941 is pending because no protected runner matches tag release” points directly to the scheduling layer.

5. Control plane versus execution plane

GitLab performs control-plane work: it stores project configuration, resolves CI configuration, creates pipelines and jobs, evaluates eligibility, queues work, tracks permissions, receives traces, and stores configured artifacts/reports. GitLab Runner performs execution-plane work: a registered runner requests eligible work, an executor prepares an environment, source is obtained, variables are exposed according to job context, scripts run, and results are sent back.

This distinction prevents expensive misdiagnosis. If CI Lint rejects the configuration, changing runner settings cannot help. If a pipeline exists but a job remains pending, the runner/scheduling layer is a better first suspect than shell syntax. If a job is running and a command exits 1, begin with the job trace and tool state. If the job is green but a deployment is unhealthy, move outward to the target.

6. Identity: a branch name is not enough

Branches and tags move; commit SHAs do not. Record the immutable revision that the pipeline is evaluating. Also record why the pipeline exists. A push pipeline and a manually started pipeline can target the same branch and commit while having different CI_PIPELINE_SOURCE values and therefore different rule outcomes in later chapters.

Inside a running job, print only the non-secret fields that are needed for evidence. The following observation block does not print CI_JOB_TOKEN, environment dumps, or arbitrary variables:

printf 'pipeline_source=%s\n' "$CI_PIPELINE_SOURCE"
printf 'ref=%s\n' "$CI_COMMIT_REF_NAME"
printf 'sha=%s\n' "$CI_COMMIT_SHA"
printf 'pipeline_id=%s\n' "$CI_PIPELINE_ID"
printf 'job_id=%s\n' "$CI_JOB_ID"
printf 'runner_id=%s\n' "${CI_RUNNER_ID:-unknown}"
printf 'runner_version=%s\n' "${CI_RUNNER_VERSION:-unknown}"
test "$CI_COMMIT_SHA" = "$(git rev-parse HEAD)"

7. Compilation happens before runner execution

GitLab does not hand the raw YAML file to a runner and ask the runner to invent a pipeline. GitLab first parses and resolves the CI/CD configuration, including configuration brought in by supported reuse mechanisms, then evaluates pipeline/job creation logic. Later chapters will explore workflow:rules, job rules, include, components, child pipelines, and generated configuration. For now, remember one invariant: the graph must exist before a runner can execute its jobs.

CI Lint is therefore a control-plane diagnostic. Current GitLab documentation states that it can validate syntax and logic and can simulate pipeline creation to expose more complicated configuration problems. A successful simulation still does not prove that a compatible runner exists or that a shell command will succeed; it proves a different, earlier layer.

8. Variable availability has phases

GitLab documents predefined variables by availability phase. Pre-pipeline variables are available before a pipeline exists and can participate in decisions such as configuration inclusion. Pipeline variables become available while GitLab creates the pipeline and can influence job rules. Job-only variables appear only when a runner starts a job. This timing is an architectural fact, not trivia.

For example, CI_PIPELINE_SOURCE and CI_COMMIT_SHA are available early enough to describe pipeline creation context. Runner-specific values such as CI_RUNNER_ID are job-only because no runner has been selected while GitLab is deciding whether the job should exist. Therefore a rule cannot sensibly depend on a value that exists only after the rule decision.

9. Runner and executor state: execution is not durable storage

Jobs execute on runners, but the runner is not the pipeline database. Depending on executor and fleet design, workspaces may be ephemeral, reused, or destroyed. Even on a reused host, relying on accidental files from a prior job creates hidden coupling and security risk. Explicit artifacts, registries, or external stores are the durable mechanisms taught later.

GitLab's current runner security guidance warns that the Shell executor has high security risk for untrusted builds because jobs run with the runner user's permissions and can expose host or neighboring project state. The Shell executor is currently in maintenance mode. You do not need to operate a runner in Chapter 01, but you should already know that “it ran on a runner” is not enough; runner scope, executor, protection, network reachability, version, and privilege are part of the evidence.

10. What green proves—and what it does not

This state separation is the backbone of the course. It scales from a two-line tutorial job to a multi-project enterprise delivery platform because every layer still needs its own identity and proof.

11. Read-only inspection before mutation

Before adding credentials, changing runner settings, or introducing deployments, inspect what the system already knows. In the GitLab UI, Build → Pipelines identifies pipelines and sources; open a pipeline to inspect its jobs and graph, then open a job to inspect its trace and runner metadata. Settings → CI/CD → Runners shows runner availability when your role permits it. Build → Pipeline editor → Validate provides validation and simulation without requiring a successful runner execution.

Ask these questions in order:

1. What source caused or attempted pipeline creation?
2. Which exact ref and SHA are involved?
3. Which configuration was compiled and validated?
4. Does the pipeline exist, and which jobs actually exist in it?
5. Which jobs are queued, running, skipped, failed, or successful?
6. Which runner/executor handled a running job?
7. What durable evidence—artifact, report, registry digest, deployment record—was produced?
8. If an external system changed, what independent target-side evidence confirms the change?

12. Foundation mistakes to eliminate now

• “The YAML parses, so CI works.” Parsing is only one early control-plane state.
• “The branch tells me exactly what ran.” Preserve source, ref, and immutable SHA.
• “A job in the YAML must exist in every pipeline.” Compilation and rules can omit jobs.
• “A pending job is a script bug.” The script has not run; inspect runner eligibility/capacity.
• “The workspace is durable.” Persist required results explicitly; do not depend on incidental runner state.
• “A green deploy job means users are healthy.” Verify the external target independently.
• “Printing all variables is the fastest debugging method.” It can leak credentials and expand incident scope.

13. Micro-lab: predict the states before running anything

Imagine a disposable project with one .gitlab-ci.yml job in an observe stage. A push to a lab branch creates a pipeline. Before Lesson 2 makes this real, write down these predictions:

1. The source should be push for the branch push pipeline.
2. The pipeline and job should identify the same immutable CI_COMMIT_SHA.
3. The job cannot have a runner ID until GitLab assigns an eligible runner.
4. If no runner is eligible, the job may remain pending even though configuration and pipeline creation succeeded.
5. If the job runs, its checked-out HEAD should match the recorded job SHA for this simple branch scenario.
6. A file remains useful across jobs only if it is persisted through an explicit mechanism such as an artifact; incidental workspace state is not a contract.
7. A successful observation job changes no cloud, cluster, registry, or production environment.

Lesson 2 converts these predictions into a safe experiment. The goal is not a green badge; it is a defensible evidence chain.

Knowledge check

Official references and version notes

• Get started with GitLab CI/CD — current first-principles overview of pipeline configuration, jobs, stages, and runners.
• CI/CD pipelines — pipeline sources, basic/stage/DAG pipeline forms, and current pipeline behavior.
• CI/CD YAML syntax reference — authoritative keyword semantics and compatibility notes.
• Predefined CI/CD variables — availability phases and runtime identity variables such as source, SHA, pipeline, job, and runner metadata.
• Runners — current runner scheduling and execution model.
• Validate GitLab CI/CD configuration — current CI Lint behavior, including syntax/logic validation and pipeline simulation.
• Security for self-managed runners — executor trust boundaries and current runner-security guidance.
• Shell executor — current maintenance-mode status and security limitations for untrusted builds.