Checkpoint Lab — Web API, Webhooks, Automation, Provisioning, and Policy as Code

1. Checkpoint scenario

You own a disposable automation user and a desired-state file for sq-ch23-policy-lab. Your job is to:

1. inventory the exact Community Build/API baseline;
2. reconcile the private project and one HMAC-protected project webhook;
3. run reconciliation twice and prove the second run is safe;
4. trigger a local analysis and correlate ceTaskId with the webhook event;
5. preserve one deliberate failure—either blind duplicate create, wrong webhook secret, or one-page-only inventory—and repair only the causal layer;
6. revoke credentials and remove only the lab-owned objects.

2. Exact assumptions and preflight

export SONAR_HOST_URL="http://localhost:9000"
export PROJECT_KEY="sq-ch23-policy-lab"
sonar-scanner --version
curl -fsS "$SONAR_HOST_URL/api/server/version"

# Review this instance's endpoint docs before the checkpoint:
# $SONAR_HOST_URL/web_api

3. Predictions before action

Write at least four predictions before running anything:

• Prediction A: first reconciliation creates missing owned state; second reconciliation creates nothing.
• Prediction B: a valid HMAC receiver accepts SonarQube’s analysis webhook, while the same payload signed with a different secret is rejected.
• Prediction C: scanner exit/report upload does not by itself prove Compute Engine success or Quality Gate pass.
• Prediction D: revoking the API User token causes authenticated API calls to fail without changing project state.

4. Desired-state manifest

{
  "project": {
    "key": "sq-ch23-policy-lab",
    "name": "SQ Chapter 23 Policy Lab",
    "visibility": "private"
  },
  "webhook": {
    "name": "ch23-local-receiver",
    "urlFromEnvironment": "CH23_WEBHOOK_URL",
    "secretFromEnvironment": "CH23_WEBHOOK_SECRET"
  },
  "ownership": {
    "deleteUnmanagedObjects": false,
    "ambiguousMatchPolicy": "fail-closed"
  }
}

Secret values are references, not policy-file content.

5. Start receiver and perform a negative HMAC test

Use the Lesson 2 receiver. Before SonarQube sends anything, prove the receiver rejects a fake request signed with the wrong secret. Keep the real secret only in the receiver/SonarQube webhook configuration.

export CH23_WEBHOOK_SECRET="$(python -c 'import secrets; print(secrets.token_hex(32))')"
export CH23_WEBHOOK_URL="http://127.0.0.1:9123/sonar"   # host-process SonarQube
python receiver.py &
RECEIVER_PID=$!

# Negative test: do not use the real secret.
printf '%s' '{"project":{"key":"sq-ch23-policy-lab"}}' > /tmp/ch23-fake.json
BAD_SIG="$(python - <<'PY'
import hashlib,hmac
body=open('/tmp/ch23-fake.json','rb').read()
print(hmac.new(b'wrong-secret', body, hashlib.sha256).hexdigest())
PY
)"
curl -sS -o evidence/bad-hmac-body.txt -w '%{http_code}\n' \
  -H "X-Sonar-Webhook-HMAC-SHA256: $BAD_SIG" \
  -H 'Content-Type: application/json' \
  --data-binary @/tmp/ch23-fake.json \
  http://127.0.0.1:9123/sonar \
  | tee evidence/bad-hmac-status.txt

Expected status: 401. This verifies receiver behavior without exposing or testing the real secret.

6. Run the policy client twice

python reconcile.py | tee evidence/reconcile-1.txt
python reconcile.py | tee evidence/reconcile-2.txt

# Capture sanitized current state.
curl --fail-with-body -sS \
  -H "Authorization: Bearer $SONAR_API_TOKEN" \
  "$SONAR_HOST_URL/api/webhooks/list?project=$PROJECT_KEY" \
  > evidence/webhooks.json

Verification: exactly one owned project key and one owned webhook; second run reports present/compliant state rather than another create.

7. Deliberate API fault: run the old create-only logic once

Now preserve a real deterministic failure by calling create on the already-created project. Do not delete/recreate the project to “make the command work.”

set +e
curl --fail-with-body -sS -X POST \
  -H "Authorization: Bearer $SONAR_API_TOKEN" \
  --data-urlencode "project=$PROJECT_KEY" \
  --data-urlencode "name=Duplicate Attempt" \
  "$SONAR_HOST_URL/api/projects/create" \
  > evidence/duplicate-create.out 2>&1
CREATE_RC=$?
set -e
echo "$CREATE_RC" | tee evidence/duplicate-create-exit.txt

Repair: rerun reconcile.py. It reads first, recognizes the existing exact key, and makes no destructive change. The broken response remains in the packet.

8. Run analysis and preserve asynchronous evidence

git rev-parse HEAD | tee evidence/revision.txt
sonar-scanner -X 2>&1 | tee evidence/scanner.log
cp .scannerwork/report-task.txt evidence/report-task.txt
CE_TASK_ID="$(sed -n 's/^ceTaskId=//p' .scannerwork/report-task.txt)"
echo "$CE_TASK_ID" | tee evidence/ce-task-id.txt

curl --fail-with-body -sS \
  -H "Authorization: Bearer $SONAR_API_TOKEN" \
  "$SONAR_HOST_URL/api/ce/task?id=$CE_TASK_ID" \
  | tee evidence/ce-task.json

Poll with bounded delay until the task is terminal if necessary. Never infer completion from scanner exit code.

9. Prove event correlation and receiver state

cat evidence/webhook-events.jsonl | tee evidence/webhook-events-copy.txt
python - <<'PY'
import json
ce = open('evidence/ce-task-id.txt').read().strip()
events = [json.loads(x) for x in open('evidence/webhook-events.jsonl') if x.strip()]
matched = [e for e in events if e.get('taskId') == ce]
assert matched, f'no validated webhook event matched ceTaskId={ce}'
print(json.dumps(matched[-1], indent=2, sort_keys=True))
PY

Record task status and Quality Gate status separately. A matching event proves correlation, not that the gate passed.

10. API-version assumptions record

server: Community Build 26.9.0.129388
web_api_policy: use only endpoints documented by this target instance
existing_api_endpoints_used:
  - GET /api/server/version
  - GET /api/components/search
  - POST /api/projects/create
  - GET /api/webhooks/list
  - POST /api/webhooks/create|update|delete
  - GET /api/ce/task
v2_statement: migration is gradual; no mechanical prefix conversion
content_type_for_post: application/x-www-form-urlencoded unless endpoint docs say otherwise
auth: Bearer User token for API; project-analysis token for scanner
expiration_header: SonarQube-Authentication-Token-Expiration
webhook_signature: X-Sonar-Webhook-HMAC-SHA256 over raw body

11. Verification checklist

• ☐ Exact server/scanner versions are recorded.
• ☐ API token owner/type/permissions/expiration are recorded without token value.
• ☐ Endpoint methods/paths/content types are documented from the target instance.
• ☐ Pagination logic is exercised with a small page size.
• ☐ First reconcile creates missing state; second reconcile produces no duplicate objects.
• ☐ Duplicate-create failure is preserved and repaired by read-before-create logic.
• ☐ HMAC-negative test returns 401 without using the real secret.
• ☐ Actual webhook event matches scanner ceTaskId.
• ☐ Compute Engine task status and Quality Gate status are recorded independently.
• ☐ No bearer/scanner/webhook secret appears in source or evidence.

12. Required evidence packet

13. Cleanup and rollback

1. Export webhook/project evidence first.
2. Delete the owned webhook by exact webhook key.
3. Delete the disposable project by exact project key only if this checkpoint created it.
4. Revoke API User token and project-analysis token; verify later use fails.
5. Stop the local receiver and remove temporary payload files.
6. Remove the temporary key-pattern permission template and restore global permission changes.
7. Unset all token/secret environment variables.

14. What this adds to the governed operating model

Chapter 23 adds an observable automation control plane: identities are scoped; endpoints are versioned/documented; reads precede mutations; pagination is complete; reruns converge; asynchronous task identity is retained; webhook authenticity is verified; and destructive cleanup requires exact ownership evidence.

Chapter 24 continues with Monorepos, Multi-Module Builds, Generated Code, and Complex Repository Layouts, applying these control principles to repositories where scope and project boundaries are substantially more complex.

Knowledge check

Official references and version notes

• Community Build — Web API — bearer authentication, token-expiration header, POST form-data guidance, and gradual Web API V2 transition.
• Community Build — managing tokens — User/project/global token semantics, expiration and revocation.
• Community Build — automating project creation/import — local project creation through POST /api/projects/create and examples of current V2 DevOps-platform endpoints.
• SonarQube — webhooks — event payload, project/global webhooks, delivery monitoring and HMAC protection.
• SonarQube public Web API reference — webhooks — create/list/update/delete/delivery endpoint family and permission requirements.
• SonarQube public Web API reference — components — documented component discovery/read operations.
• SonarQube downloads — current Community Build/Server/LTA release identities used in the dated compatibility note.