Kubernetes Agent, Cluster Access, GitOps-Oriented Delivery, Kubernetes Deployments, and Environment Integration: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first diagnostic sequence

1. Preserve pipeline/job IDs, first failing trace and any GitLab deployment ID.
2. Confirm CI_PIPELINE_SOURCE, ref, CI_COMMIT_SHA and compiled configuration.
3. Confirm workflow/job-rule decisions and non-secret effective inputs.
4. Inspect job graph, queue, runner/executor/image/tool versions.
5. Inspect Agent context name and kubectl auth can-i; never expose kubeconfig credentials.
6. Inspect manifest/image digest and the command/tool/network failure.
7. Inspect GitLab environment/deployment record.
8. Inspect Kubernetes Deployment/ReplicaSet/Pod/Service and runtime imageID.
9. Verify the external endpoint independently.
10. Apply the least destructive correction; rerun only the failed safe scope.

2. Failure-layer map

3. Failure: production cluster credential stored in a variable

Broken pattern: a generic project variable contains a kubeconfig with cluster-admin privileges and every branch can run the deployment job. This combines long-lived credential exposure, broad source trust and broad cluster privilege.

4. Failure: Agent authorizes too many projects

Suppose the Agent config grants a whole group while only one deployer project needs access:

ci_access:
  groups:
    - id: platform

Before changing it, list which projects depend on that connection and preserve the current Agent config commit. Then narrow to explicit projects/environments:

ci_access:
  projects:
    - id: platform/deployer
      environments:
        - staging
        - production

Authorization changes can take time to propagate. Verify context availability and denials before retrying.

5. Failure: namespace collision

Two projects both deploy deployment/web into namespace staging. One pipeline patches the other team’s workload. The fix is not a blind rename in the running cluster. Preserve resource UIDs, owner labels, pipeline/job IDs and manifest revisions; then move to deterministic project-owned namespaces or stronger naming/ownership conventions plus RBAC.

kubectl -n staging get deployment/web -o json > evidence/web-before.json
kubectl -n staging get rolebinding -o yaml > evidence/rolebindings-before.yaml
kubectl -n staging get events --sort-by=.lastTimestamp > evidence/events-before.txt

6. Intentionally broken example: green job before rollout health

deploy_staging:
  stage: deploy
  script:
    - kubectl config use-context "$KUBE_CONTEXT"
    - kubectl apply -f k8s/staging.yaml
  environment:
    name: staging

This job can succeed as soon as the Kubernetes API accepts the manifest. It does not wait for Pods to become Ready. Preserve the successful job/deployment record, then inspect the workload. Repair by adding an explicit bounded rollout/health check:

script:
  - kubectl config use-context "$KUBE_CONTEXT"
  - kubectl apply -f k8s/staging.yaml
  - kubectl rollout status deployment/example -n app-staging --timeout=180s
  - ./verify-staging-health.sh

If rollout fails, do not replace the original job trace. The failed rollout is the causal evidence.

7. Failure: image tag drifts

A manifest uses example/app:stable. Pipeline 410 deploys it successfully. Later the tag moves. A restarted Pod now pulls different bytes even though no manifest commit changed. Diagnose by comparing the original registry digest, manifest reference and current Pod imageID. Correct the manifest to the verified digest and preserve the old runtime evidence.

kubectl -n app-staging get deployment/example   -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n app-staging get pods -l app=example   -o jsonpath='{range .items[*]}{.metadata.name}{"  "}{.status.containerStatuses[0].imageID}{"\n"}{end}'
cat evidence/approved-image-digest.txt

The safe correction is to promote the already-approved digest, not rebuild the image during troubleshooting.

8. Failure: correct permissions, wrong cluster context

A job is authorized to two Agents and relies on whichever context is current. The script succeeds against staging while the GitLab environment says production—or vice versa. Make context selection an explicit reviewed variable and record it before mutation:

kubectl config get-contexts -o name
printf 'selected_context=%s\n' "$KUBE_CONTEXT"
kubectl config use-context "$KUBE_CONTEXT"
kubectl config current-context
kubectl get namespace "$TARGET_NAMESPACE" -o jsonpath='{.metadata.uid}{"\n"}'

Do not infer cluster identity only from a namespace name, because the same namespace name can exist in many clusters.

9. Failure: template copied from deprecated certificate-based integration

Symptoms include advice to configure project/group/instance cluster certificates, a gitlab-deploy context assumption, or legacy built-in Agent gitops: manifest_projects. Preserve the source/template version, then migrate to the Agent CI/CD workflow or Flux. Do not add feature flags to resurrect deprecated integration in a new design.

10. Runtime evidence without destructive debugging

kubectl -n app-staging get deployment,replicaset,pod,service -o wide
kubectl -n app-staging describe deployment/example
kubectl -n app-staging get events --sort-by=.lastTimestamp
kubectl -n app-staging logs deployment/example --tail=200
kubectl -n app-staging get pod -l app=example   -o jsonpath='{range .items[*]}{.metadata.name}{"  "}{.status.phase}{"  "}{.status.containerStatuses[0].imageID}{"\n"}{end}'

These commands preserve observable state. Avoid deleting Pods merely to see whether they come back; that destroys first-failure evidence and can create side effects before you know the cause.

11. Security-sensitive actions

• Changing Agent project/group authorization is an authorization change: review and record it.
• Changing an Agent service-account Role/ClusterRole changes external privilege: use least privilege and preserve before/after RBAC.
• Rotating a leaked kubeconfig or Agent token is incident-response work: do not post token material into tickets or logs.
• Changing production Flux sources/reconciliation targets can deploy broadly: use reviewed exact revisions.
• Never troubleshoot by disabling TLS verification, granting cluster-admin, bypassing protected environments or using an unreviewed mutable image.

12. Performance and reliability: avoid making the control plane the bottleneck

Large pipelines can overload an API server with many parallel imperative jobs. Serialize conflicting deployments (Chapter 18), prefer controller reconciliation for production desired state, bound kubectl rollout status timeouts, and keep manifests small enough to review. A queued runner is not a slow cluster; distinguish job queue time, API request latency, reconciliation time and application readiness.

13. Least-destructive recovery map

Knowledge check

Official references and version notes

Documentation verification date: 2026-09-12. Current GitLab documentation lists the Agent CI/CD workflow as Free, Premium, and Ultimate on GitLab.com, Self-Managed, and Dedicated. Authorized CI/CD jobs receive a KUBECONFIG containing contexts for authorized agent connections; the context identifies the agent configuration project and agent name. ci_access can authorize projects/groups and can restrict access to named environment patterns. The current per-agent authorization limit is 500 projects and 500 groups. CI-job impersonation with access_as: ci_job is Premium/Ultimate; the mandatory lab therefore demonstrates true Kubernetes namespace-scoped ServiceAccount RBAC locally instead of pretending that paid impersonation is Free. GitLab recommends Flux for GitOps; the Agent’s legacy built-in pull-based GitOps functionality was removed in GitLab 17.0. Certificate-based cluster integration was deprecated in GitLab 14.5. The Kubernetes dashboard is currently Beta and available on all tiers. For environment:kubernetes, dashboard namespace/Flux resource settings belong under dashboard:; the older direct namespace/flux_resource_path form is deprecated. For production GitOps diagnosis, add Flux Source/Kustomization/HelmRelease conditions and the exact desired-state revision to the same evidence sequence before changing reconciliation settings.

• Using GitLab CI/CD with a Kubernetes cluster — official reference.
• GitLab Agent for Kubernetes — official reference.
• Get started connecting a Kubernetes cluster — official reference.
• Install the Agent for Kubernetes — official reference.
• Migrate legacy Agent GitOps to Flux — official reference.
• Dashboard for Kubernetes — official reference.
• CI/CD YAML syntax — environment:kubernetes — official reference.
• Deprecated CI/CD keywords — official reference.
• Migrate from certificate-based Kubernetes integration — official reference.
• GitLab-managed Kubernetes resources — official reference.
• Kubernetes RBAC authorization — official reference.
• Kubernetes Deployments — official reference.
• kind quick start — official reference.
• Flux documentation — official reference.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.