Chapter 28Lesson 05~260 minutes

Checkpoint Lab — Kubernetes Agent, Cluster Access, GitOps-Oriented Delivery, Kubernetes Deployments, and Environment Integration

Deploy one digest-pinned workload through a narrowly scoped local identity, prove rollout and environment evidence independently, and remove exactly the disposable resources created by the lab.

CheckpointServiceAccountDigest pinningEvidenceCleanup

Learning objectives

Checkpoint objectives

  • Deploy a tiny digest-pinned workload to a disposable local cluster with a namespace-scoped identity.
  • Predict and verify authorization, manifest, runtime, environment and health state changes independently.
  • Prove one cluster-scoped action is denied.
  • Build an evidence packet tying source, image digest, manifest digest, identity, rollout and cleanup together.
  • Explain how the same evidence maps to a real Agent/Flux production architecture without claiming the local lab is one.

1. Checkpoint scenario

You are the release engineer for the synthetic service glci-ch28-web. Security requires that CI may modify only namespace glci-ch28-cp, production-style manifests must use an image digest, and the release record must not be accepted until rollout and runtime image identity are verified. The lab uses a local kind cluster and a short-lived Kubernetes ServiceAccount token. No cloud, paid GitLab feature or real production Agent is required.

Guardrail: if the current context is not exactly kind-glci-ch28-cp, the mutation steps must stop. The lab intentionally uses a unique namespace and labels so cleanup can target only lab-owned resources.

2. Assumptions and recorded versions

  • GitLab semantics verified 2026-09-12; Agent CI/CD workflow is available on all tiers.
  • The checkpoint’s Kubernetes identity is a local simulation of least privilege, not Agent access_as: ci_job.
  • Tools: Git, Docker-compatible engine, kind, kubectl and Python 3. Record the actual versions before starting.
  • Use only a synthetic/public image. No proprietary registry credential is needed.
  • A human-readable source image tag is resolved once; all deployment evidence uses the resulting @sha256: reference.
git --version
docker --version
kind version
kubectl version --client --output=yaml
python3 --version

3. Predict before changing state

Prediction How you will verify it
The simulated CI identity can create/patch Deployments and Services only inside glci-ch28-cp. Short-lived kubeconfig + kubectl auth can-i allowed/denied matrix.
The rendered Deployment contains the exact resolved image digest. Manifest inspection + SHA-256 of manifest + expected image-ref file.
A successful apply will create GitLab-like deployment intent but is not rollout proof. Separate apply output from rollout status, Pod Ready and runtime imageID.
Cleanup removes only the labeled checkpoint workload/security bootstrap. Before/after resource inventory and final namespace absence.

4. Create the disposable cluster and evidence directory

set -eu
CLUSTER=glci-ch28-cp
NS=glci-ch28-cp
mkdir -p evidence .lab-private lab
chmod 700 .lab-private
if ! kind get clusters | grep -qx "$CLUSTER"; then
  kind create cluster --name "$CLUSTER"
fi
test "$(kubectl config current-context)" = "kind-$CLUSTER"
printf 'repo_sha=%s\n' "$(git rev-parse HEAD)" | tee evidence/source.txt
kubectl get nodes -o wide > evidence/nodes-before.txt
kubectl get namespace "$NS" --ignore-not-found > evidence/namespace-before.txt

If this checkpoint is later run inside GitLab, append CI_PIPELINE_SOURCE, CI_COMMIT_SHA, pipeline/job IDs and merged-config evidence to evidence/source.txt.

5. Bootstrap the namespace and narrow ServiceAccount

apiVersion: v1
kind: Namespace
metadata:
  name: glci-ch28-cp
  labels:
    app.kubernetes.io/part-of: glci-ch28-checkpoint
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-deployer
  namespace: glci-ch28-cp
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-deployer
  namespace: glci-ch28-cp
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["pods", "services"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer
  namespace: glci-ch28-cp
subjects:
  - kind: ServiceAccount
    name: ci-deployer
    namespace: glci-ch28-cp
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ci-deployer

Save as lab/checkpoint-rbac.yaml and apply it only after the context guard:

test "$(kubectl config current-context)" = "kind-glci-ch28-cp"
kubectl apply -f lab/checkpoint-rbac.yaml | tee evidence/rbac-apply.txt
kubectl -n glci-ch28-cp get serviceaccount,role,rolebinding -o yaml   > evidence/rbac-state.yaml

6. Build a short-lived kubeconfig and prove positive/negative authorization

TOKEN_FILE=.lab-private/token
KCFG=.lab-private/kubeconfig
kubectl -n glci-ch28-cp create token ci-deployer --duration=15m > "$TOKEN_FILE"
chmod 600 "$TOKEN_FILE"
python3 - "$TOKEN_FILE" "$KCFG" <<'PY2'
import json, pathlib, subprocess, sys
source=json.loads(subprocess.check_output([
  'kubectl','config','view','--raw','--minify','-o','json'
], text=True))
cluster=source['clusters'][0]['cluster']
token=pathlib.Path(sys.argv[1]).read_text().strip()
config={
 'apiVersion':'v1','kind':'Config',
 'clusters':[{'name':'checkpoint','cluster':cluster}],
 'users':[{'name':'ci-deployer','user':{'token':token}}],
 'contexts':[{'name':'checkpoint-ci','context':{
   'cluster':'checkpoint','user':'ci-deployer','namespace':'glci-ch28-cp'}}],
 'current-context':'checkpoint-ci'}
pathlib.Path(sys.argv[2]).write_text(json.dumps(config), encoding='utf-8')
PY2
chmod 600 "$KCFG"
{
  KUBECONFIG="$KCFG" kubectl config current-context
  KUBECONFIG="$KCFG" kubectl auth can-i create deployments.apps -n glci-ch28-cp
  KUBECONFIG="$KCFG" kubectl auth can-i patch deployments.apps -n glci-ch28-cp
  KUBECONFIG="$KCFG" kubectl auth can-i get secrets -n glci-ch28-cp
  KUBECONFIG="$KCFG" kubectl auth can-i create clusterroles.rbac.authorization.k8s.io
} | tee evidence/authorization.txt

Expected sequence: context name, yes, yes, no, no. A surprising yes for the last two is a failed checkpoint, not a convenience.

7. Resolve and record immutable image identity

SOURCE_IMAGE=nginx:1.27.5-alpine
docker pull "$SOURCE_IMAGE"
IMAGE_REF="$(docker inspect --format='{{index .RepoDigests 0}}' "$SOURCE_IMAGE")"
case "$IMAGE_REF" in *@sha256:*) ;; *) exit 3 ;; esac
printf 'source_tag=%s\nresolved=%s\n' "$SOURCE_IMAGE" "$IMAGE_REF"   | tee evidence/image-identity.txt
sha256sum evidence/image-identity.txt > evidence/image-identity.txt.sha256

This does not claim the upstream tag can never move. It proves which digest this checkpoint resolved and deployed. For a release pipeline, supply a digest that was already approved upstream.

8. Create exact desired state and hash it

KCFG=.lab-private/kubeconfig
IMAGE_REF="$(sed -n 's/^resolved=//p' evidence/image-identity.txt)"
KUBECONFIG="$KCFG" kubectl -n glci-ch28-cp create deployment glci-ch28-web   --image="$IMAGE_REF" --replicas=1 --dry-run=client -o yaml   > evidence/deployment.yaml
KUBECONFIG="$KCFG" kubectl -n glci-ch28-cp create service clusterip glci-ch28-web   --tcp=80:80 --dry-run=client -o yaml   > evidence/service.yaml
sha256sum evidence/deployment.yaml evidence/service.yaml   | tee evidence/manifest-digests.txt
grep -F '@sha256:' evidence/deployment.yaml

The grep is a guard: the checkpoint must stop if the rendered manifest does not contain an immutable digest.

9. Deploy with the narrow identity

KCFG=.lab-private/kubeconfig
KUBECONFIG="$KCFG" kubectl apply -f evidence/deployment.yaml   | tee evidence/apply-deployment.txt
KUBECONFIG="$KCFG" kubectl apply -f evidence/service.yaml   | tee evidence/apply-service.txt
KUBECONFIG="$KCFG" kubectl -n glci-ch28-cp rollout status   deployment/glci-ch28-web --timeout=180s   | tee evidence/rollout.txt

Do not retry immediately if this fails. Save the Deployment, Pod and Event state first, as Lesson 4 demonstrated.

10. Verify manifest, runtime and health independently

KCFG=.lab-private/kubeconfig
KUBECONFIG="$KCFG" kubectl -n glci-ch28-cp get deployment/glci-ch28-web -o json   > evidence/deployment-runtime.json
KUBECONFIG="$KCFG" kubectl -n glci-ch28-cp get pods -l app=glci-ch28-web -o json   > evidence/pods-runtime.json
KUBECONFIG="$KCFG" kubectl -n glci-ch28-cp get service/glci-ch28-web -o json   > evidence/service-runtime.json
python3 - <<'PY2'
import json
expected=open('evidence/image-identity.txt', encoding='utf-8').read().split('resolved=',1)[1].splitlines()[0]
pods=json.load(open('evidence/pods-runtime.json', encoding='utf-8'))
print('expected', expected)
for pod in pods['items']:
  for st in pod.get('status',{}).get('containerStatuses',[]):
    print('pod', pod['metadata']['name'], 'ready', st.get('ready'), 'imageID', st.get('imageID'))
PY2

The runtime imageID may include a runtime-specific prefix before the same digest. Record and compare the sha256: value, not only the human-readable tag.

11. Record the GitLab environment mapping you would use

The checkpoint is local, so it cannot create an authentic GitLab deployment record. Record the faithful GitLab mapping as configuration evidence:

deploy_checkpoint:
  stage: deploy
  variables:
    KUBE_CONTEXT: platform/cluster-agents:checkpoint-agent
  script:
    - kubectl config use-context "$KUBE_CONTEXT"
    - kubectl apply -f k8s/checkpoint.yaml
    - kubectl rollout status deployment/glci-ch28-web -n glci-ch28-cp --timeout=180s
  environment:
    name: checkpoint/ch28
    url: https://checkpoint.example.test
    kubernetes:
      agent: platform/cluster-agents:checkpoint-agent
      dashboard:
        namespace: glci-ch28-cp

In a real disposable Agent project, add actual pipeline/job/deployment IDs and environment URL health evidence. Do not claim the local YAML itself creates a GitLab record.

12. Deliberate failure: prove cluster-wide privilege is denied

KCFG=.lab-private/kubeconfig
if KUBECONFIG="$KCFG" kubectl auth can-i create namespaces | grep -qx yes; then
  echo "FAIL: checkpoint identity unexpectedly has cluster-wide privilege" >&2
  exit 4
else
  echo "PASS: namespace creation is denied" | tee evidence/denial.txt
fi

This negative test is part of the security evidence packet. A deployment that works only because the identity is cluster-admin fails the chapter goal.

13. Evidence packet

Evidence Checkpoint file/state
Source identity evidence/source.txt; add GitLab source/SHA/pipeline/job IDs when run in CI
Tool/cluster identity Recorded tool versions, kind-glci-ch28-cp context and node inventory
Authorization evidence/authorization.txt plus RBAC objects; no token/kubeconfig
Artifact identity evidence/image-identity.txt and its SHA-256
Desired state Deployment/Service YAML plus manifest-digests.txt
Execution Apply outputs and rollout.txt
Runtime Deployment/Pod/Service JSON, Ready state and imageID
Negative control evidence/denial.txt proving cluster-wide create is denied
GitLab mapping Agent/context/environment YAML and assumptions note; authentic IDs only when actually run in GitLab
Limitations Local ServiceAccount identity is not a GitLab Agent; no Flux reconciliation is performed

14. Exact cleanup and proof

set -eu
CLUSTER=glci-ch28-cp
NS=glci-ch28-cp
test "$(kubectl config current-context)" = "kind-$CLUSTER"
test "$NS" = "glci-ch28-cp"
KUBECONFIG=.lab-private/kubeconfig kubectl -n "$NS" delete   -f evidence/service.yaml -f evidence/deployment.yaml --ignore-not-found   | tee evidence/workload-cleanup.txt
rm -f .lab-private/token .lab-private/kubeconfig
rmdir .lab-private 2>/dev/null || true
kubectl delete -f lab/checkpoint-rbac.yaml --ignore-not-found   | tee evidence/security-cleanup.txt
kubectl get namespace "$NS" --ignore-not-found | tee evidence/namespace-after.txt
# Optional after evidence is complete:
# kind delete cluster --name glci-ch28-cp

Because the Namespace is part of the guarded checkpoint bootstrap manifest, deleting that exact manifest may remove it after its contained workload is gone. The final namespace-after.txt should be empty.

15. Verification checklist

  • Source SHA and, when applicable, GitLab pipeline source/SHA/IDs are recorded.
  • Narrow identity can mutate only required namespaced workload resources.
  • Cluster-wide action and Secret read are denied.
  • Deployment manifest contains an @sha256: image reference.
  • Manifest hashes and image digest are recorded before mutation.
  • Rollout completed under the narrow identity.
  • Runtime Pod imageID matches the intended digest.
  • GitLab environment/Agent mapping is clearly marked as mapping unless a real Agent run supplied authentic deployment IDs.
  • Temporary token/kubeconfig are absent from the evidence packet.
  • All disposable Kubernetes resources are removed and cleanup evidence is retained.

16. What the checkpoint proves—and does not prove

It proves the mechanics of namespace-scoped Kubernetes identity, immutable image deployment, desired-state hashing, independent rollout/runtime verification and exact cleanup. It does not prove a real GitLab Agent is configured, a GitLab environment record exists, Flux is reconciling the state, or a production cluster’s network/policy controls are safe. Those require an authorized integration exercise with real Agent/Flux evidence.

Knowledge check

Why is the denied namespace-creation test required?

What must you compare to prove the runtime used the intended image bytes?

If rollout status succeeds but an application health endpoint fails, is the checkpoint complete?

Why is the GitLab environment YAML labeled a mapping in this local checkpoint?

What production direction does this chapter recommend for declarative Kubernetes delivery?

17. What Chapter 28 adds to the production operating model

You can now separate Kubernetes delivery into verifiable control planes: GitLab source/configuration and Agent authorization, Kubernetes RBAC and desired state, immutable artifact identity, GitLab environment/deployment history, controller reconciliation, runtime image identity and external health. This prevents a green CI job from becoming an unexamined claim that “production is deployed.”

Next chapter

Infrastructure-as-Code Pipelines, Terraform/OpenTofu Workflows, Plan Reviews, State Safety, and Drift Controls

Chapter 29 applies the same state discipline to infrastructure as code: exact Terraform/OpenTofu configuration and provider identity, plan evidence, protected state, reviewed apply authorization, drift, locking and recovery.

Version and compatibility note

GitLab and GitLab Runner evolve continuously. Treat version-sensitive YAML, runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations as assumptions to verify against the current official GitLab documentation before production use. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for any reproducible lab or incident record.

Official references and version notes

Documentation verification date: 2026-09-12. Current GitLab documentation lists the Agent CI/CD workflow as Free, Premium, and Ultimate on GitLab.com, Self-Managed, and Dedicated. Authorized CI/CD jobs receive a KUBECONFIG containing contexts for authorized agent connections; the context identifies the agent configuration project and agent name. ci_access can authorize projects/groups and can restrict access to named environment patterns. The current per-agent authorization limit is 500 projects and 500 groups. CI-job impersonation with access_as: ci_job is Premium/Ultimate; the mandatory lab therefore demonstrates true Kubernetes namespace-scoped ServiceAccount RBAC locally instead of pretending that paid impersonation is Free. GitLab recommends Flux for GitOps; the Agent’s legacy built-in pull-based GitOps functionality was removed in GitLab 17.0. Certificate-based cluster integration was deprecated in GitLab 14.5. The Kubernetes dashboard is currently Beta and available on all tiers. For environment:kubernetes, dashboard namespace/Flux resource settings belong under dashboard:; the older direct namespace/flux_resource_path form is deprecated. For a real Agent checkpoint, add the Agent configuration project/ref, exact context, ci_access scope, authentic GitLab deployment ID, and—if using Flux—the Source/Kustomization/HelmRelease revision and reconciliation conditions. Never substitute screenshots for machine-readable evidence when the API/object state is available.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.