Environments, Required Reviewers, Protection Rules, and Deployment Gates: Diagnostics, Failure Modes, and Production Practices

1. Evidence-first diagnostic sequence

1. Preserve run ID, attempt, source ref/SHA, workflow revision, timestamps, and first-failure logs.
2. Confirm the job actually references the intended environment name and whether deployment is true or false.
3. Inspect branch/tag policy, wait timer, required reviewers, self-review setting, custom rules, and bypass configuration.
4. Confirm evaluated inputs, job permissions, and environment secret/variable names without printing values.
5. Inspect job graph, queue/wait state, concurrency group, and any older in-progress deployment.
6. Confirm runner/image/toolchain only after the control-plane checks above.
7. Inspect artifact/build identity and then the GitHub deployment/status record.
8. Inspect the external target independently.
9. Apply the least-destructive correction and rerun the smallest equivalent scope.

This order prevents a governance denial from being “fixed” by broadening permissions or rerunning until timing changes.

2. Causal failure map

3. Failure: assuming environment secrets exist before approval

A common design mistake is to perform secret-dependent work in a job that does not reference the environment, then expect the environment secret to be available. Another is to assume a job waiting for approval can somehow use the secret to prepare a deployment. Both violate the environment boundary.

# Broken: this job does not reference production.
jobs:
  prepare:
    runs-on: ubuntu-24.04
    env:
      PROD_TOKEN: ${{ secrets.PROD_TOKEN }}
    steps:
      - run: test -n "$PROD_TOKEN"

Interpret an empty secret in context: verify secret scope/name, job environment binding, fork/Dependabot restrictions where applicable, and protection state. Repair by moving only the secret-dependent operation into the correctly protected environment job—not by copying the secret to repository scope.

4. Failure: production environment accepts arbitrary refs

A workflow trigger such as workflow_dispatch can be useful for recovery, but without environment ref restrictions it may allow deployment from an unintended branch or tag. Preserve the run's GITHUB_REF and SHA. Then inspect the environment's Deployment branches and tags policy.

Repair the environment policy to the intended branch/tag set, and keep manual dispatch only if it serves an explicit operational use case. Do not rely only on a shell if inside the privileged job when GitHub can enforce the ref policy before the runner and environment credential exposure.

5. Failure: branch name is treated as sole production authorization

if: github.ref == 'refs/heads/main' is source selection, not operational approval. It cannot express reviewer state, wait timers, change windows, custom App decisions, or target-specific credential release. Keep source conditions where appropriate, but use a governed environment for the deployment target.

6. Failure: bypassing protection as a troubleshooting shortcut

Administrators may be able to bypass environment protection depending on configuration and policy. A bypass changes the authorization history and must be treated as a security-sensitive operational action, not a normal debugging step. Preserve why the gate blocked first. If emergency bypass is allowed by your policy, record actor, reason, affected environment, exact run/SHA, and subsequent external verification.

For this academy lab, do not bypass. Fix the synthetic rule or wait for the configured timer/reviewer.

7. Failure: one concurrency group cancels or blocks unrelated targets

# Broken design: unrelated staging and production share one global lock.
concurrency:
  group: deploy-global
  cancel-in-progress: true

This has two distinct hazards. First, staging can unnecessarily block production or vice versa. Second, cancellation can interrupt a state-changing deployment after an external write has occurred. Repair with a group derived from the real target boundary and normally queue state-changing deployments instead of cancelling them:

concurrency:
  group: deploy-${{ github.repository }}-${{ inputs.environment_name }}
  cancel-in-progress: false

Validate the input against an allow-list before using it for privileged environment selection. Do not let arbitrary user text create or target environments.

8. Failure: approval is treated as proof of artifact integrity

A reviewer can approve the correct change request while a deployment job later fetches a mutable tag or rebuilds source into a different artifact. Approval cannot solve that. Preserve the build artifact digest/source SHA before the gate, then make the deploy job consume exactly that evidence. Chapter 23 will add attestations and provenance verification.

9. Intentionally broken example: typo creates the wrong environment

jobs:
  deploy:
    runs-on: ubuntu-24.04
    environment: prodution   # typo: intended production
    steps:
      - run: echo "fake deployment"

If the referenced environment does not exist, GitHub can create a new environment with that name. Outside special implicit Pages behavior, the new environment has no protection rules or secrets. The run evidence therefore shows the wrong environment name and missing expected secret/policy. Preserve that run. Repair the literal name to production; then consider organization lint/policy to prevent dynamic or unapproved environment names.

10. Failure: no deployment record where one was expected

Check whether the workflow used deployment: false. If so, no deployment object is expected even though wait/reviewer rules and environment configuration can still apply. If a real deployment should be audited, restore the default deployment: true behavior rather than fabricating a separate history later.

11. Do not use always() as a privileged recovery hammer

A cleanup step that removes a harmless temporary directory can run broadly. A privileged rollback, package publication, environment mutation, or cloud deletion should not be hidden under unconditional always() simply because a previous step failed. Model state-changing recovery explicitly, require the necessary authorization, and make it idempotent.

12. Rerun only after the cause is understood

Preserve the original run/attempt first. A rerun may observe different reviewer timing, environment state, concurrency occupancy, or external target state. If the problem was a workflow/configuration revision, create a new commit and run that exact revision instead of pretending a rerun changed the workflow definition.

13. Production incident record

Knowledge check

Official references and version notes

Version-sensitive GitHub Actions behavior in this lesson was rechecked on 2026-09-10. Re-verify current plan, repository visibility, API, environment, and protection-rule behavior before production rollout.

• GitHub Docs — Deployments and environments
• GitHub Docs — Managing environments for deployment
• GitHub Docs — Reviewing deployments
• GitHub Docs — Deploying with GitHub Actions
• GitHub Docs — Deploying to a specific environment
• GitHub Docs — Workflow syntax: jobs..environment
• GitHub Docs — REST API for deployment environments
• GitHub Docs — REST API for deployments
• GitHub Docs — REST API for deployment statuses
• GitHub Docs — Secrets reference