Web API, Webhooks, Automation, Provisioning, and Policy as Code: Diagnostics, Failure Modes, and Production Practices

1. Evidence-first diagnostic sequence

1. Preserve sanitized request method/path, HTTP status, response headers/body, timestamp, automation version, and target SonarQube version.
2. Confirm token owner/type/expiration and the exact permission required by the documented endpoint.
3. Confirm endpoint generation/changelog and request content type/parameters.
4. For paginated reads, prove the loop reached the terminal page.
5. For mutations, reread the exact target object and determine whether the request changed state.
6. For analysis automation, preserve revision, scanner log, report-task.txt, ceTaskId, Compute Engine result, gate result, and webhook delivery.
7. For webhooks, preserve delivery metadata and HMAC-verification outcome; do not log the secret.
8. Only then apply the smallest correction and rerun the smallest equivalent request.

2. Failure: building on a deprecated or undocumented endpoint

Automation may continue working for months after an endpoint becomes deprecated, then break on upgrade. The first fix is not to pin an old server forever. Record the endpoint changelog in the target instance’s /web_api, identify the documented replacement when one exists, and write contract tests for the replacement.

3. Failure: printing bearer tokens or secrets

Common leak paths include set -x, CI debug output, exception strings that include headers, URLs containing credentials, and artifact uploads of raw environment dumps. Redaction belongs at the logging abstraction—not as a best-effort search after the job finishes.

SENSITIVE_HEADERS = {"authorization", "x-sonar-passcode"}

def safe_headers(headers):
    return {k: ("[REDACTED]" if k.lower() in SENSITIVE_HEADERS else v)
            for k, v in headers.items()}

Webhook secrets and scanner tokens should never appear in exception messages or policy manifests either.

4. Failure: ignoring HTTP error bodies

A wrapper that reduces every response to raise_for_status() without preserving a sanitized body loses the reason for failure. A 400 might name a missing parameter; a 401 points to authentication; a 403 points to permissions; a create conflict/validation response may prove the object already exists.

Preserve the original response before retrying. The evidence packet should include method, path, status, non-secret response headers, body, target version, and request intent.

5. Intentionally broken example: blind project-create retry

Fault: an old script always calls POST /api/projects/create. The first run succeeds; the second returns an error because the key already exists. A wrapper catches every exception and retries three times.

# Deliberately broken; run only against the disposable Chapter 23 project.
for attempt in 1 2 3; do
  curl --fail-with-body -sS -X POST \
    -H "Authorization: Bearer $SONAR_API_TOKEN" \
    --data-urlencode "project=sq-ch23-policy-lab" \
    --data-urlencode "name=SQ Chapter 23 Policy Lab" \
    "$SONAR_HOST_URL/api/projects/create" \
    >"evidence/broken-create-$attempt.out" 2>&1 || true
done

Interpretation: retrying cannot repair an already-existing project. The causal fix is to read current project state, compare the exact key, skip create when it exists, and verify the existing object matches the desired ownership/visibility. Preserve the broken responses as first-failure evidence.

6. Failure: page one looked clean

If an automation checks only the first 100 rules/issues/projects, its “not found” result is not evidence. Force a small page size in tests and assert that the client reads beyond page one. Test empty results, exact-multiple page counts, one extra row, and maximum supported page sizes.

7. Failure: deleting by an ambiguous display name

A human-readable name is often not the durable API identifier. The webhook API returns an auto-generated webhook key. Project identity is a project key. If two objects have the same or similar display name, stop and require explicit resolution instead of deleting whichever appears first.

8. Failure: trusting webhook JSON as an authenticated command

An Internet-accessible receiver that accepts any POST body can be triggered by anyone who can reach it. HMAC verification mitigates origin/integrity spoofing when the secret is configured, but downstream actions still need their own authorization and idempotency logic.

Reject missing/invalid X-Sonar-Webhook-HMAC-SHA256, cap request size, parse only after signature verification, apply timeouts, and deduplicate based on stable event/task identity before changing downstream state.

9. Failure: webhook delivered, therefore gate passed

Webhook delivery success means the HTTP receiver accepted a payload. The payload can describe a failed Compute Engine task or failed Quality Gate. Preserve and inspect status, task ID, and qualityGate.status. If the downstream action is high impact, perform a read-back API check before proceeding.

10. Causal failure map

11. Production anti-patterns to reject

• Do not use administrator tokens for routine project reconciliation.
• Do not print bearer tokens, webhook secrets, passcodes, or secured settings.
• Do not disable TLS verification to make API/webhook connectivity work.
• Do not retry create/delete/comment mutations blindly.
• Do not delete by ambiguous names or search result position.
• Do not use undocumented internal endpoints as a compatibility contract.
• Do not edit the database/search index to repair application state.
• Do not replace a failing project key with a new key to escape history or policy.

Knowledge check

Official references and version notes

• Community Build — Web API — bearer authentication, token-expiration header, POST form-data guidance, and gradual Web API V2 transition.
• Community Build — managing tokens — User/project/global token semantics, expiration and revocation.
• Community Build — automating project creation/import — local project creation through POST /api/projects/create and examples of current V2 DevOps-platform endpoints.
• SonarQube — webhooks — event payload, project/global webhooks, delivery monitoring and HMAC protection.
• SonarQube public Web API reference — webhooks — create/list/update/delete/delivery endpoint family and permission requirements.
• SonarQube public Web API reference — components — documented component discovery/read operations.
• SonarQube downloads — current Community Build/Server/LTA release identities used in the dated compatibility note.