Checkpoint Lab — Pipeline Schedules, Trigger Tokens, Pipeline API, Webhooks, ChatOps, and Event-Driven Automation

1. Checkpoint scenario

An external release coordinator receives event release:training:33:001. It must trigger a pipeline for disposable project 4242/ref main, handle one rate-limit response, wait for that exact pipeline to succeed, reconcile a local external marker once, and prove that replaying the event reuses the existing pipeline correlation instead of creating another one.

2. Assumptions and preflight

python3 --version
mkdir -p glci-ch33-checkpoint && cd glci-ch33-checkpoint
mkdir -p evidence .local external_state
printf 'checkpoint_root=%s
' "$PWD"

3. Use the exact mock API and controller from Lesson 2

Save the Lesson 2 Python blocks as mock_gitlab.py and orchestrator.py. This checkpoint intentionally reuses the same code instead of creating a second subtly different implementation. Record their SHA-256 values before execution.

sha256sum mock_gitlab.py orchestrator.py > evidence/controller-code.sha256

4. Write predictions before execution

P1: first handling of release:training:33:001 receives one simulated 429, then creates exactly one pipeline.
P2: the created pipeline has source=trigger, ref=main and a recorded 40-hex source SHA.
P3: polling the returned pipeline ID moves pending/running to terminal success; HTTP creation alone is not marked success.
P4: replaying the same external event reuses the journaled pipeline ID and leaves mock pipeline_count=1.
P5: external reconciliation writes exactly one marker only after terminal success; cleanup removes only this disposable lab.

cat > evidence/predictions.txt <<'EOF'
P1 one bounded rate-limit retry, one pipeline creation
P2 source=trigger ref=main exact SHA preserved
P3 terminal success checked by exact pipeline ID
P4 replay reuses existing correlation; one pipeline total
P5 one external marker after success only
EOF

5. Start the disposable API and capture baseline state

python3 mock_gitlab.py > evidence/mock-server.log 2>&1 &
echo $! > .local/mock-server.pid
sleep 0.3
python3 - <<'PY' > evidence/state-before.json
from urllib.request import urlopen
print(urlopen('http://127.0.0.1:18765/debug/state').read().decode())
PY
cat evidence/state-before.json

Expected baseline: no pipelines. The server PID is evidence for exact process cleanup; it is not a GitLab runner registration.

6. Trigger with one simulated 429 and correlate exact pipeline ID

python3 orchestrator.py 'release:training:33:001' --simulate-429   | tee evidence/first-orchestration.txt
cat evidence/journal.json
cat evidence/final-pipeline.json

The client waits only because the server returned Retry-After, then creates one pipeline. Inspect final-pipeline.json: the source is trigger, ref is main, SHA is explicit and status is terminal success.

7. Reconcile external state only after terminal success

from pathlib import Path
import json
p=json.loads(Path('evidence/final-pipeline.json').read_text())
if p['status'] != 'success': raise SystemExit('DENY: pipeline not successful')
key=p['request_key'].replace(':','_')
out=Path('external_state')/f'{key}.txt'
out.write_text(f'pipeline_id={p["id"]}\nsha={p["sha"]}\nsource={p["source"]}\n',encoding='utf-8')
print('reconciled=',out)

Run the snippet only after verifying final-pipeline.json. This marker models an external create-or-update action keyed by the business request, not by “latest pipeline.”

8. Replay the event and prove no second pipeline is created

python3 orchestrator.py 'release:training:33:001'   | tee evidence/replay-orchestration.txt
python3 - <<'PY' | tee evidence/replay-proof.txt
from urllib.request import urlopen
import json
s=json.loads(urlopen('http://127.0.0.1:18765/debug/state').read())
print('pipeline_count=',len(s['pipelines']))
print('pipeline_ids=',sorted(map(int,s['pipelines'].keys())))
assert len(s['pipelines']) == 1
PY

This proves the checkpoint's local duplicate-suppression invariant. Document the limitation: if the first client dies after server acceptance but before journaling, a stronger distributed idempotency design is required.

9. Inject one policy failure: wrong ref

python3 - <<'PY' | tee evidence/wrong-ref.txt
from urllib.parse import urlencode
from urllib.request import Request, urlopen
from urllib.error import HTTPError
payload=urlencode({'token':'training-trigger-token','ref':'production','inputs[request_key]':'negative-33-001'}).encode()
try:
    urlopen(Request('http://127.0.0.1:18765/api/v4/projects/4242/trigger/pipeline',data=payload,method='POST'))
except HTTPError as e:
    print('expected_http=',e.code)
    print('response=',e.read().decode())
    assert e.code == 400
else:
    raise SystemExit('DENY TEST FAILED: wrong ref unexpectedly accepted')
PY

Keep the denial evidence. The correct repair is the intended ref/configuration, not broadening the server to accept arbitrary targets.

10. Add webhook-verification evidence

Save the Lesson 2 HMAC example as webhook_demo.py, then run:

python3 webhook_demo.py | tee evidence/webhook-proof.txt
chmod 600 .local/webhook-signing-token.txt

The training HMAC models current GitLab signing-token verification. Do not put the signing key in evidence/.

11. Preserve final mock/API and external state

python3 - <<'PY' > evidence/state-after.json
from urllib.request import urlopen
print(urlopen('http://127.0.0.1:18765/debug/state').read().decode())
PY
find external_state -maxdepth 1 -type f -print -exec cat {} \;
sha256sum evidence/*.json evidence/*.txt external_state/*.txt   > evidence/evidence-digests.sha256

12. Map checkpoint evidence to a real GitLab run

13. Evidence packet and limitations

cat > evidence/assumptions-limitations.txt <<'EOF'
PROVES IN THIS CHECKPOINT:
- one external event is correlated to one mock pipeline ID after a bounded 429 retry
- pipeline source/ref/SHA/status are preserved
- replay after journal persistence does not create a second mock pipeline
- external marker is written only after terminal success
- wrong ref is rejected and preserved as first-failure evidence
- a Standard-Webhooks-style HMAC can be verified without printing the key
DOES NOT PROVE:
- exactly-once creation across a crash between server acceptance and journal commit
- real GitLab token permissions, runner execution or webhook delivery
- production webhook endpoint/network security
- that pipeline success alone proves a real deployment is healthy
EOF
printf 'gitlab_evidence_fields=CI_PIPELINE_SOURCE,CI_COMMIT_SHA,CI_PIPELINE_ID,CI_JOB_ID,CI_RUNNER_ID
'   >> evidence/assumptions-limitations.txt
find evidence -maxdepth 1 -type f -printf '%f
' | sort

14. Verification checklist

• Predictions were written before creation/mutation.
• One 429 is preserved and retried only after Retry-After.
• Exact returned pipeline ID is journaled.
• Source/ref/SHA are recorded and source is trigger.
• Controller polls only the exact pipeline ID.
• External marker appears only after terminal success.
• Replay reports idempotent_reuse and pipeline count remains one.
• Wrong ref fails with preserved HTTP evidence.
• Webhook HMAC proof is present while signing key is excluded from evidence.
• Limitations describe the client-crash uncertainty window and external-health boundary.

15. Cleanup and rollback

test "$(basename "$PWD")" = glci-ch33-checkpoint
kill "$(cat .local/mock-server.pid)" 2>/dev/null || true
rm -f .local/mock-server.pid .local/webhook-signing-token.txt
cd ..
rm -rf glci-ch33-checkpoint
printf 'cleanup=verified-exact-checkpoint-directory-removed
'

A real GitLab cleanup must likewise target exact disposable trigger/schedule/webhook IDs. Retain evidence required by policy before deleting lab resources.

Knowledge check

16. What Chapter 33 adds to the production operating model

You can now automate pipeline creation without treating automation as a black box. Source, identity, target, inputs, returned pipeline ID, asynchronous execution, callbacks, rate limits, retries and external side effects are explicit states with separate evidence. This lets operations recover from duplicate events and partial failures without broad credentials or ambiguous “latest” selection.

Official references and version notes

Documentation verification date: 2026-09-12. Current GitLab documentation places pipeline schedules, pipeline trigger tokens, the Pipelines API, project webhooks and ChatOps on Free/Premium/Ultimate unless a narrower feature is explicitly noted. Pipeline inputs for schedules and trigger/API creation are generally available in current GitLab releases. Scheduled pipelines execute with the schedule owner's permissions; a manual run of a schedule uses the permissions of the user who starts that manual run. Trigger-token pipelines report CI_PIPELINE_SOURCE=trigger; pipelines created through the Pipelines API report api; schedules report schedule; ChatOps reports chat; and a job-token call to the trigger endpoint creates a downstream multi-project pipeline with source pipeline. New webhooks can use HMAC-SHA256 signing tokens following Standard Webhooks; GitLab 19.1 documentation recommends signing tokens over the legacy plain X-Gitlab-Token secret. The mandatory labs below are local-only and use Python's standard library; no GitLab token, network account, webhook endpoint or live project is required. The checkpoint deliberately teaches application-level idempotency with a local journal. For production, select a durable store and uniqueness/transaction model appropriate to concurrent orchestrators and the external side effect being protected.

• Scheduled pipelines — official reference.
• Pipeline schedules API — official reference.
• Trigger pipelines with the API — official reference.
• Pipeline trigger tokens API — official reference.
• Pipelines API — official reference.
• REST API pagination and rate limits — official reference.
• CI/CD pipeline creation limits — official reference.
• Predefined CI/CD variables — official reference.
• Job rules and CI_PIPELINE_SOURCE values — official reference.
• CI/CD job token — official reference.
• Fine-grained job-token permissions — official reference.
• Webhooks — official reference.
• Webhook events — official reference.
• ChatOps — official reference.
• Access token scopes — official reference.

Current assumptions used in this chapter: Version-sensitive YAML, Runner/executor behavior, APIs, security features, policy controls, tiers, and deprecations must be verified against the exact GitLab, GitLab Runner, tool, and external-system versions used in production. Preserve the exact project/ref/SHA, compiled configuration, pipeline/job IDs, runner/tool versions, and external target evidence used for reproducible work.