YAML Reuse with include, extends, Anchors, !reference, Hidden Jobs, and Template Architecture: Concepts, Architecture, and Mental Model

1. The practical problem: duplicated YAML hides dependencies instead of removing them

Copying the same image, setup, rules, cache, and script fragments into many jobs feels simple when a pipeline is small. As the pipeline grows, copies drift. One job receives a security fix while another keeps the old behavior; a branch-specific edit changes only one copy; reviewers must compare long blocks instead of reasoning about one shared contract.

Reuse helps only when you know exactly where the effective configuration comes from. GitLab does not execute the source fragments independently. It first resolves included configuration, applies GitLab/YAML reuse mechanisms, merges definitions, applies caller overrides, evaluates pipeline/job rules, and only then creates final jobs. Therefore the object to audit is the compiled configuration, not merely the root file.

2. Keep five reuse states separate

These states fail independently. A remote include can fail before a pipeline exists. A valid merged job can later be omitted by rules. A correct job can wait forever for a runner. Reuse diagnosis therefore begins in the configuration layer and moves outward only after compilation is proven.

3. Mental model: from root file to final jobs

Read the diagram left to right. The root file names dependencies. GitLab resolves the include graph. File-local YAML features and GitLab reuse features shape candidate definitions. GitLab merges them, applies the root/caller overrides, validates the result, and only then produces final jobs that can enter pipeline creation and execution.

The key causality is that changing an include ref or an inherited parent changes the compiled program even when the root file itself is unchanged.

4. Include is dependency resolution, not textual paste

Current GitLab supports local, project, remote, template, and component includes, plus subkeys such as inputs, rules, and integrity where applicable. Includes are evaluated first; then the root configuration is merged and can override included keys. The resolution step has a 30-second time limit and a default maximum of 150 includes including nested includes.

5. Pipeline configuration is a snapshot

When GitLab creates a pipeline, it resolves and stores the configuration snapshot used for that pipeline. Retrying a job does not refetch changed include files; rerunning/creating a new pipeline resolves the includes again. This explains an important diagnostic pattern: “the template changed after my failure” does not explain why a retry of the same job suddenly should compile differently—it does not.

For audit evidence, preserve the pipeline ID, source SHA, and expanded configuration together. A moving include branch can produce different future pipelines from the same root-file text.

6. Hidden jobs are named configuration objects, not executable jobs

A job whose name begins with a dot is hidden. It is not scheduled by itself, but it can hold reusable configuration for extends or anchors. Hidden jobs create a useful “template vocabulary” inside the compiled configuration, but naming alone does not create an interface contract. Callers still need to know which keys they may override safely.

.python_base:
  image: python:3.12.6-slim-bookworm
  before_script:
    - python --version
  variables:
    PIP_DISABLE_PIP_VERSION_CHECK: "1"

unit_tests:
  extends: .python_base
  script:
    - python -m pytest -q

7. YAML anchors are file-local structural reuse

Anchors and aliases are YAML features, not GitLab cross-file references. They are useful for sharing maps or arrays inside one file, but they cannot cross an include boundary. When duplicate keys occur, later YAML keys override earlier ones. For cross-file reuse, use extends or !reference instead.

.common_scripts: &common_scripts
  - python --version
  - python -m pip --version

lint:
  script:
    - *common_scripts
    - python -m compileall src

8. extends is GitLab-aware hash inheritance

extends can reuse jobs defined in included files. GitLab performs a reverse deep merge by key. Hashes such as variables can merge; scalar values and arrays such as script, tags, and rules are replaced when a child provides that key. That replacement behavior is one of the most important failure modes in this chapter.

GitLab supports multiple inheritance levels (up to eleven), but the documentation recommends avoiding more than about three because the compiled behavior becomes difficult to review.

9. !reference selects configuration by path across files

!reference is a GitLab custom YAML tag that can select a keyword or nested configuration from another job, including a job loaded through an include. This is useful when you need an array or a precise subsection rather than a whole-job hash merge. It is resolved during configuration parsing, not at shell runtime.

# templates.yml
.rules_for_tests:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_PIPELINE_SOURCE == "push"'

# .gitlab-ci.yml
include:
  - local: templates.yml

test:
  script: echo test
  rules: !reference [.rules_for_tests, rules]

Do not confuse !reference with variables or CI/CD inputs. Current GitLab documentation describes parsing constraints between !reference and inputs because references are resolved at a different phase.

10. Read-only inspection before editing reuse

Start by locating the root source SHA and every include. Then open CI Lint or the pipeline editor’s expanded configuration. The expanded view resolves includes, anchors, extends, and !reference, allowing you to inspect the final job rather than mentally simulate inheritance.

printf 'source=%s
ref=%s
sha=%s
pipeline=%s
'   "$CI_PIPELINE_SOURCE" "$CI_COMMIT_REF_NAME" "$CI_COMMIT_SHA" "$CI_PIPELINE_ID"

Do not dump all variables or tokens while “debugging templates.” Include dependencies are configuration state; secret values are unrelated and should stay redacted.

11. Common wrong models

• “include is just copy/paste.” No. It has access, resolution, snapshot, limit, merge, and trust semantics.
• “anchors work across files.” They do not.
• “extends appends arrays.” It does not; array values are generally replaced.
• “main is pinned.” A branch is moving. Prefer a full SHA or protected version tag for cross-project executable configuration.
• “remote HTTPS means trusted.” HTTPS protects transport to a host; it does not prove that the content is the reviewed template. Use immutable identity/integrity where practical.

Knowledge check

Official references and version notes

Documentation verification date: 2026-09-11. CI reuse and include behavior evolves across GitLab releases, especially include subkeys, integrity/caching, inputs/components, and template deprecations. Re-check the deployed GitLab version before relying on newer include features on Self-Managed or Dedicated installations.

• CI/CD YAML syntax reference — current include types/subkeys, merge behavior, extends, hidden jobs, defaults, and pipeline syntax.
• Use CI/CD configuration from other files — local/project/remote/template include behavior, variables, rules, overrides, troubleshooting, and trust guidance.
• Optimize GitLab CI/CD configuration files — anchors, hidden jobs, extends, reverse deep merge, and !reference.
• Pipeline editor — validation and expanded configuration showing includes, anchors, extends, and !reference resolved.
• CI Lint — validate and simulate configuration before creating a real pipeline.
• CI/CD inputs — typed configuration interfaces that become increasingly important in Chapter 13.