Web API, Webhooks, Automation, Provisioning, and Policy as Code: Guided Hands-On Workflow

1. Disposable workflow contract

2. One-time local bootstrap: make least privilege possible

A project webhook requires project administration, while project creation requires the global Create Projects permission. Do not give the automation user Administer System merely to make the script easy.

Using the local administrator through the UI, create a temporary permission template named SQ Ch23 Lab Template with project key pattern ^sq-ch23-.*. Grant the project creator the project permissions needed for the lab: Browse, Execute Analysis, and Administer. Grant user ch23-bot the global Create Projects permission. Record these bootstrap mutations for rollback.

3. Generate a bounded User token and inspect expiration metadata

Sign in as ch23-bot and generate a short-lived User token. The token needs API actions matching the user’s permissions. Keep it out of source, command history, screenshots, and artifacts.

export SONAR_HOST_URL="http://localhost:9000"
export PROJECT_KEY="sq-ch23-policy-lab"
export PROJECT_NAME="SQ Chapter 23 Policy Lab"

read -rsp "ch23-bot user token: " SONAR_API_TOKEN; echo
export SONAR_API_TOKEN

mkdir -p evidence/http
curl --fail-with-body -sS \
  -o evidence/http/validate.json \
  -H "Authorization: Bearer $SONAR_API_TOKEN" \
  "$SONAR_HOST_URL/api/authentication/validate"

# The validate endpoint is one of the documented exceptions that does not return
# token-expiration metadata. Capture the header from a normal authenticated API call instead.
curl --fail-with-body -sS \
  -D evidence/http/inventory.headers \
  -o evidence/http/inventory.json \
  -H "Authorization: Bearer $SONAR_API_TOKEN" \
  "$SONAR_HOST_URL/api/components/search?qualifiers=TRK&p=1&ps=1"

grep -i '^SonarQube-Authentication-Token-Expiration:' \
  evidence/http/inventory.headers || true

4. Build the idempotent project/webhook reconciler

The script owns exactly two things: project existence/visibility and a webhook named ch23-local-receiver. It never deletes or rewrites objects merely because their names are similar.

# reconcile.py
import json, os, sys, urllib.parse, urllib.request, urllib.error

BASE = os.environ["SONAR_HOST_URL"].rstrip("/")
TOKEN = os.environ["SONAR_API_TOKEN"]
PROJECT = os.environ.get("PROJECT_KEY", "sq-ch23-policy-lab")
NAME = os.environ.get("PROJECT_NAME", "SQ Chapter 23 Policy Lab")
HOOK_NAME = "ch23-local-receiver"
HOOK_URL = os.environ["CH23_WEBHOOK_URL"]
HOOK_SECRET = os.environ["CH23_WEBHOOK_SECRET"]


def call(method, path, form=None):
    data = None
    headers = {"Authorization": f"Bearer {TOKEN}", "Accept": "application/json"}
    if form is not None:
        data = urllib.parse.urlencode(form).encode()
        headers["Content-Type"] = "application/x-www-form-urlencoded"
    req = urllib.request.Request(BASE + path, data=data, headers=headers, method=method)
    try:
        with urllib.request.urlopen(req, timeout=15) as r:
            body = r.read()
            return r.status, dict(r.headers), json.loads(body or b"{}")
    except urllib.error.HTTPError as e:
        body = e.read().decode("utf-8", "replace")
        raise RuntimeError(f"{method} {path} -> HTTP {e.code}: {body}") from None


def project_exists():
    # Search is paginated; inspect every page and compare the exact key.
    page = 1
    while True:
        q = urllib.parse.urlencode({"qualifiers":"TRK", "q":PROJECT, "p":page, "ps":100})
        _, _, payload = call("GET", "/api/components/search?" + q)
        if any(c.get("key") == PROJECT for c in payload.get("components", [])):
            return True
        pg = payload.get("paging", {})
        size, total = int(pg.get("pageSize", 0)), int(pg.get("total", 0))
        index = int(pg.get("pageIndex", page))
        if size == 0 or index * size >= total:
            return False
        page += 1


def ensure_project():
    if project_exists():
        print("project: already present")
        return
    call("POST", "/api/projects/create", {
        "project": PROJECT, "name": NAME, "visibility": "private"
    })
    if not project_exists():
        raise RuntimeError("project create returned but exact key is not readable")
    print("project: created")


def ensure_webhook():
    q = urllib.parse.urlencode({"project": PROJECT})
    _, _, payload = call("GET", "/api/webhooks/list?" + q)
    matches = [h for h in payload.get("webhooks", []) if h.get("name") == HOOK_NAME]
    if len(matches) > 1:
        raise RuntimeError("ambiguous duplicate webhook names; stop for manual review")
    if not matches:
        call("POST", "/api/webhooks/create", {
            "project": PROJECT, "name": HOOK_NAME,
            "url": HOOK_URL, "secret": HOOK_SECRET
        })
        print("webhook: created")
        return
    hook = matches[0]
    if hook.get("url") == HOOK_URL and str(hook.get("hasSecret", "")).lower() in {"true", "yes"}:
        print("webhook: already compliant")
        return
    call("POST", "/api/webhooks/update", {
        "webhook": hook["key"], "name": HOOK_NAME,
        "url": HOOK_URL, "secret": HOOK_SECRET
    })
    print("webhook: reconciled")


ensure_project()
ensure_webhook()

Run the script twice. The first run should create missing state; the second should report compliant state rather than creating another project or webhook.

5. Build an HMAC-validating local webhook receiver

The receiver stores only validated event metadata. It does not treat the payload as permission to deploy or mutate SonarQube.

# receiver.py
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import hashlib, hmac, json, os, pathlib, time

SECRET = os.environ["CH23_WEBHOOK_SECRET"].encode()
OUT = pathlib.Path("evidence/webhook-events.jsonl")
OUT.parent.mkdir(parents=True, exist_ok=True)

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get("Content-Length", "0"))
        body = self.rfile.read(n)
        got = self.headers.get("X-Sonar-Webhook-HMAC-SHA256", "")
        expected = hmac.new(SECRET, body, hashlib.sha256).hexdigest()
        if not hmac.compare_digest(got, expected):
            self.send_response(401); self.end_headers(); return
        payload = json.loads(body)
        record = {
            "receivedAt": int(time.time()),
            "projectKey": payload.get("project", {}).get("key"),
            "taskId": payload.get("taskId") or payload.get("taskID"),
            "taskStatus": payload.get("status"),
            "qualityGate": payload.get("qualityGate", {}).get("status"),
        }
        with OUT.open("a", encoding="utf-8") as f:
            f.write(json.dumps(record, sort_keys=True) + "\n")
        self.send_response(204); self.end_headers()
    def log_message(self, fmt, *args):
        pass

ThreadingHTTPServer(("127.0.0.1", 9123), Handler).serve_forever()

export CH23_WEBHOOK_SECRET="$(python -c 'import secrets; print(secrets.token_hex(32))')"

# If SonarQube runs directly on this host:
export CH23_WEBHOOK_URL="http://127.0.0.1:9123/sonar"

# Docker Desktop alternative when SonarQube is in a container:
# export CH23_WEBHOOK_URL="http://host.docker.internal:9123/sonar"

python receiver.py

Before configuring the webhook, verify the SonarQube runtime can actually route to the receiver. A host browser reaching 127.0.0.1 does not prove a container can reach that same loopback address.

6. Reconcile twice and prove rerun safety

python reconcile.py | tee evidence/reconcile-run-1.txt
python reconcile.py | tee evidence/reconcile-run-2.txt

curl --fail-with-body -sS \
  -H "Authorization: Bearer $SONAR_API_TOKEN" \
  "$SONAR_HOST_URL/api/webhooks/list?project=$PROJECT_KEY" \
  | tee evidence/webhooks-after.json

The second run should produce no duplicate project and no second webhook with the same ownership name. If the script detects duplicate owned webhook names, it stops instead of deleting one by guesswork.

7. Create a tiny source fixture and run an analysis

mkdir -p src
cat > sonar-project.properties <<'EOF'
sonar.projectKey=sq-ch23-policy-lab
sonar.sources=src
sonar.sourceEncoding=UTF-8
sonar.analysis.chapter=23
EOF

cat > src/app.py <<'PY'
def normalize(value: str) -> str:
    return value.strip().lower()
PY

git init
git config user.email "learner@example.invalid"
git config user.name "SonarQube Learner"
git add sonar-project.properties src/app.py
git commit -m "chapter23 automation fixture"
git rev-parse HEAD | tee evidence/revision.txt

# A project-analysis token is preferred for scanning; keep it separate from the API user token.
read -rsp "Project-analysis token: " SONAR_TOKEN; echo
export SONAR_TOKEN
sonar-scanner -X 2>&1 | tee evidence/scanner.log
cp .scannerwork/report-task.txt evidence/report-task.txt
CE_TASK_ID="$(sed -n 's/^ceTaskId=//p' .scannerwork/report-task.txt)"
echo "$CE_TASK_ID" | tee evidence/ce-task-id.txt

8. Correlate Compute Engine and webhook evidence

curl --fail-with-body -sS \
  -H "Authorization: Bearer $SONAR_API_TOKEN" \
  "$SONAR_HOST_URL/api/ce/task?id=$CE_TASK_ID" \
  | tee evidence/ce-task.json

cat evidence/webhook-events.jsonl

After the webhook arrives, compare its task ID with ceTaskId. Matching identity proves the event belongs to the same background task. Then compare task status and Quality Gate status; do not infer one from the other.

9. Force a pagination-aware read path

Set a deliberately small page size when reading issues or rules so the client proves it can follow pages even on a tiny fixture.

for p in 1 2 3; do
  curl --fail-with-body -sS \
    -H "Authorization: Bearer $SONAR_API_TOKEN" \
    "$SONAR_HOST_URL/api/issues/search?componentKeys=$PROJECT_KEY&p=$p&ps=2" \
    > "evidence/issues-page-$p.json"
done

Stop according to the response paging metadata. Do not hard-code “three pages” in real automation; the loop above is only a visible teaching probe.

10. Challenge: choose the correct layer

The webhook receiver returns 401 even though SonarQube shows the delivery was attempted. Which layer should you debug first?

1. Preserve the delivery metadata and raw request bytes if your receiver safely captures them.
2. Verify the receiver is using the same webhook secret as the SonarQube webhook configuration.
3. Verify it computes HMAC-SHA256 over the raw request body and compares against X-Sonar-Webhook-HMAC-SHA256.
4. Do not change the scanner token, quality gate, source code, or database; none of those repair receiver HMAC verification.

11. Guarded cleanup

• Revoke the ch23-bot User token and verify a later API request fails authentication.
• Revoke the project-analysis token separately.
• Delete the project webhook by its exact webhook key, not by fuzzy name matching.
• Delete sq-ch23-policy-lab only after exporting evidence and only if the project was created by this lab.
• Remove the temporary key-pattern permission template and restore any changed global permission state.
• Unset SONAR_API_TOKEN, SONAR_TOKEN, and CH23_WEBHOOK_SECRET.

Knowledge check

Official references and version notes

• Community Build — Web API — bearer authentication, token-expiration header, POST form-data guidance, and gradual Web API V2 transition.
• Community Build — managing tokens — User/project/global token semantics, expiration and revocation.
• Community Build — automating project creation/import — local project creation through POST /api/projects/create and examples of current V2 DevOps-platform endpoints.
• SonarQube — webhooks — event payload, project/global webhooks, delivery monitoring and HMAC protection.
• SonarQube public Web API reference — webhooks — create/list/update/delete/delivery endpoint family and permission requirements.
• SonarQube public Web API reference — components — documented component discovery/read operations.
• SonarQube downloads — current Community Build/Server/LTA release identities used in the dated compatibility note.