REST API, GraphQL API, Webhooks, System Hooks, Pagination, Rate Limits, and Automation: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first diagnostic sequence

1. Preserve status code, selected response headers, x-request-id, endpoint/operation name, sanitized request shape, resource IDs, and timestamps.
2. Identify instance/offering, namespace/project/group, actor/token type, role, tier, and API/client version.
3. For hooks, capture hook ID, event type, delivery ID, timestamp, verification result, attempt count, and receiver status—never the signing token.
4. For pagination, prove continuation metadata and compare an independent count/sentinel when practical.
5. Choose the least destructive correction, rerun once, and read back state.

2. Failure: first page silently becomes “all resources”

Broken: items=$(glab api projects/:fullpath/issues) and the script assumes the JSON array is complete. General REST pagination defaults to 20 items. The failure is dangerous because it produces valid JSON and exit code 0.

# Repair for shell automation:
glab api projects/:fullpath/issues --paginate --output ndjson   | jq -s 'length'
# Raw clients: follow Link rel="next" until absent.

Do not repair by guessing ?page=2 forever. Endpoint pagination can be keyset-based, and server-provided continuation data is the contract.

3. Failure: blind retry duplicates a mutation

A client POSTs “create label,” times out before receiving the response, then repeats the POST. The first request may have committed. Repair it with the Chapter 29 desired-state sequence: query by natural key, create only if absent, update by stable ID if different, verify afterwards.

4. Failure: GraphQL transport succeeded, requested operation did not

GraphQL can return top-level errors next to data, and a mutation payload can have its own errors. An authorization failure may surface as null for a field/resource. Clients that check only HTTP 200 can write corrupt downstream assumptions.

resp="$(glab api graphql -f query='query { currentUser { username } }')"
if jq -e '.errors and (.errors|length>0)' <<<"$resp" >/dev/null; then
  jq '.errors | map({message,path})' <<<"$resp" >&2
  exit 1
fi
jq '.data' <<<"$resp"

5. Failure: receiver acts before authenticating the exact body

Broken order: parse JSON → create ticket → compare a legacy token later. An attacker or replay can trigger side effects first. Correct order: read raw bytes → verify signing token HMAC → verify timestamp window → deduplicate stable webhook ID → parse only needed fields → enqueue once → acknowledge.

6. Failure: every worker retries 429 at once

A fleet that sleeps a fixed one second after 429 can create a synchronized retry storm. Honor Retry-After where provided, add bounded jitter, lower concurrency, and distinguish reads from uncertain writes. Generic rate headers do not describe every application-specific limit, so code must handle 429 regardless of a previous “remaining” value.

7. Failure: broad personal api token where narrower identity works

A long-lived personal token with full api scope expands impact if logs, environment, or receiver storage leaks. First determine whether a CI job token supports the endpoint, then whether project/group-owned automation identity is appropriate. Rotate/revoke leaked credentials before log/history cleanup.

8. Failure: tier/offering mismatch misdiagnosed as bad JSON

A group webhook setup on Free or an instance System Hooks API call on an unsupported offering can look like missing endpoints/permissions. Before rewriting requests, re-check the feature’s current tier, offering, and required role. System-hook feature documentation and its REST API offering matrix are intentionally treated separately in this chapter.

Knowledge check

9. Summary and next bridge

The failure model is now explicit. The checkpoint combines pagination, GraphQL, webhook replay/tamper handling, idempotent mutation, and cleanup into one evidence-producing automation exercise.

Primary sources and version notes

These lessons were finalized against current official GitLab documentation on 2026-08-22. API fields, rate limits, webhook event schemas, CLI flags, and tier/offering availability can change, so production clients should pin/document assumptions and re-check the API/CLI documentation for the deployed GitLab version.

• GitLab 19.3 release
• REST API
• REST API authentication
• REST API troubleshooting
• GraphQL API
• Run GraphQL queries and mutations
• glab api
• GitLab CLI authentication
• Webhooks
• Webhook events
• System hooks
• System Hooks API
• User and IP rate limits
• Project labels API