Checkpoint Lab — Server Logs, Monitoring, Health, Elasticsearch, and Operational Diagnostics

1. Checkpoint scenario

You are the on-call operator for a disposable Community Build instance. A learner reports: “The scanner succeeded but SonarQube did not update.” Your task is to prove two cases independently:

1. a normal scanner → CE → search/UI analysis path; and
2. a deliberate project/component key clash that produces (or, on scanner versions that prevalidate it, faithfully models) a failed CE report-processing path.

You must preserve first-failure evidence, identify the owning subsystem, recover with the smallest topology change, and leave the server in a known-good state.

2. Exact assumptions and preflight

date -u +%FT%TZ
java -version
mvn -version
sonar-scanner --version
curl -fsS "$SONAR_HOST_URL/api/server/version"
curl -fsS "$SONAR_HOST_URL/api/system/status"
df -h

3. Write predictions before acting

Record at least these predictions in evidence/predictions.md:

• P1: the normal project scan will upload a report, create a ceTaskId, complete SUCCESS, and update the project analysis timestamp.
• P2: when academy:shared already exists as a standalone project, the parent reactor’s reuse of that component key will be rejected at scanner or CE; if CE receives it, task status will be FAILED while global system health may remain GREEN.
• P3: removing only the disposable standalone conflict project and rerunning the same parent Git SHA will allow the parent task to succeed.
• P4: no Quality Gate or source-code change is necessary to repair the ownership failure.

4. Baseline evidence packet

export LAB_TS="$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "evidence/$LAB_TS"/{baseline,success,failure,recovery,cleanup}

date -u +%FT%TZ > "evidence/$LAB_TS/baseline/timestamp.txt"
curl -fsS "$SONAR_HOST_URL/api/system/status" \
  > "evidence/$LAB_TS/baseline/status.json"
curl -fsS -H "X-Sonar-Passcode: $SONAR_SYSTEM_PASSCODE" \
  "$SONAR_HOST_URL/api/system/health" \
  > "evidence/$LAB_TS/baseline/health.json"
df -h > "evidence/$LAB_TS/baseline/disk.txt"

# Preserve logs rather than deleting/rotating them.
docker cp sonarqube:/opt/sonarqube/logs/. \
  "evidence/$LAB_TS/baseline/server-logs/"

If the baseline is already YELLOW/RED or contains unexplained failed tasks, stop and diagnose that state first.

5. Successful task proof

Run the tiny sq-ch27-success project from Lesson 2. Preserve:

• Git SHA and clean-worktree status;
• scanner version and -X log;
• .scannerwork/report-task.txt;
• ceTaskId and terminal CE JSON;
• matching ce.log lines;
• project analysis timestamp and Quality Gate/measure output;
• system health and disk snapshot after completion.

Do not proceed until this control task is SUCCESS.

6. Failed CE task proof

Create/analyze the standalone academy:shared project, then submit the parent academy:sq-ch27-parent reactor from the same fixture. If a report-task.txt is produced, extract the task:

FAIL_REPORT="$(find . -name report-task.txt -print -quit)"
cp "$FAIL_REPORT" "evidence/$LAB_TS/failure/report-task.txt"
FAIL_TASK="$(sed -n 's/^ceTaskId=//p' "$FAIL_REPORT")"
printf '%s\n' "$FAIL_TASK" > "evidence/$LAB_TS/failure/ceTaskId.txt"

curl -fsS -H "Authorization: Bearer $SONAR_API_TOKEN" \
  "$SONAR_HOST_URL/api/ce/task?id=$FAIL_TASK" \
  | tee "evidence/$LAB_TS/failure/ce-task.json"

# Preserve server logs at failure time.
docker cp sonarqube:/opt/sonarqube/logs/ce.log \
  "evidence/$LAB_TS/failure/ce.log"
docker cp sonarqube:/opt/sonarqube/logs/es.log \
  "evidence/$LAB_TS/failure/es.log"
docker cp sonarqube:/opt/sonarqube/logs/web.log \
  "evidence/$LAB_TS/failure/web.log"

Record whether the task is FAILED and the exact server error. Also record current health. A successful health response does not invalidate the failed task; it helps classify it as project/report-specific rather than global infrastructure failure.

7. Least-destructive recovery

Before deleting anything, export the standalone academy:shared project identity, last analysis timestamp, task history, and the exact reason it is disposable. Then delete only that lab project so the same component key may belong to the parent reactor.

# After guarded UI deletion of exactly academy:shared:
cd sq-ch27-reactor
git status --porcelain | tee "../evidence/$LAB_TS/recovery/worktree.txt"
git rev-parse HEAD | tee "../evidence/$LAB_TS/recovery/revision.txt"

export SONAR_TOKEN="$PARENT_PROJECT_TOKEN"
mvn -B verify org.sonarsource.scanner.maven:sonar-maven-plugin:5.7.0.6970:sonar \
  2>&1 | tee "../evidence/$LAB_TS/recovery/scanner.log"

REC_REPORT="$(find . -name report-task.txt -print -quit)"
cp "$REC_REPORT" "../evidence/$LAB_TS/recovery/report-task.txt"
REC_TASK="$(sed -n 's/^ceTaskId=//p' "$REC_REPORT")"
printf '%s\n' "$REC_TASK" > "../evidence/$LAB_TS/recovery/ceTaskId.txt"

Poll to SUCCESS and capture matching ce.log, project measures/gate, and health. Compare the failure and recovery Git SHAs; they must match.

8. Verification checklist

• ☐ Baseline timestamp, version, health, disk, and server logs preserved.
• ☐ One successful scanner report has a recorded ceTaskId and CE SUCCESS.
• ☐ The failed experiment preserves scanner outcome, task/error details when uploaded, and process logs.
• ☐ The failure is classified as scanner/project topology or CE project-processing—not guessed from UI symptoms.
• ☐ Global health and the failed task are recorded independently.
• ☐ No database/search direct edits, TLS bypass, gate lowering, mass suppression, or project-key evasion occurred.
• ☐ Recovery changes only the exact disposable conflicting project ownership.
• ☐ Recovery scan uses the same parent Git SHA and completes successfully.
• ☐ Quality Gate and project analysis timestamp belong to the recovered completed analysis.
• ☐ Temporary credentials/projects are revoked/deleted only after evidence export.

9. Required evidence packet

evidence/<UTC timestamp>/
  assumptions.md
  predictions.md
  baseline/
    status.json
    health.json
    disk.txt
    server-logs/
  success/
    revision.txt
    scanner.log
    report-task.txt
    ceTaskId.txt
    ce-task.json
    ce.log
    gate-or-measures.json
  failure/
    revision.txt
    scanner.log
    scanner-exit.txt
    report-task.txt          # only if uploaded
    ceTaskId.txt             # only if uploaded
    ce-task.json             # failed-task evidence
    web.log
    ce.log
    es.log
    topology-before.md
  recovery/
    revision.txt
    scanner.log
    report-task.txt
    ceTaskId.txt
    ce-task.json
    health.json
  cleanup/
    revoked-tokens.md
    deleted-lab-objects.md
  limitations.md

10. Assumptions and limitations

The local lab does not model production HA, Data Center nodes, an external production database, real reverse proxies, provider decoration, or paid CE worker scaling. It does prove the diagnostic method: preserve revision/task/process evidence, distinguish scanner/CE/DB/search layers, avoid destructive shortcuts, and recover at the owner.

11. What Chapter 27 adds to the governed operating model

Previous chapters established what SonarQube analyzes and how policy reaches developers. Chapter 27 adds the operational proof that the service processing those analyses is itself observable: task identity, process logs, health, queue state, resource pressure, database/search dependencies, and supported recovery become part of the evidence packet.

Chapter 28 continues naturally into Database Backup, Restore, Disaster Recovery, and Data Protection: once you can diagnose the live persistence/search path, the next responsibility is preserving and recovering authoritative data safely.

Knowledge check

Official references and version notes

• Community Build — Server logs — current process-specific files, log levels, rotation, JSON output, DEBUG/TRACE privacy/performance cautions.
• Community Build — Monitoring the instance — api/system/health, Web/CE/Elasticsearch JVMs, CPU/RAM/disk monitoring.
• Community Build — Monitoring on Kubernetes — /api/monitoring/metrics, OpenMetrics, and CE/Web JMX exporter model.
• Community Build — Web API — bearer authentication, system-passcode use for monitoring, and continuing Web API V2 transition.
• Current Web API reference — api/system/health — GREEN/YELLOW/RED semantics and Administer System/system-passcode requirement.
• Current Web API reference — Compute Engine — CE activity/task APIs and permission boundaries.
• Background tasks — scanner success vs CE completion, task failure diagnosis, and project/module key-clash example.
• Community Build — Elasticsearch-related issues — disk-watermark/read-only behavior and supported recovery.
• Community Build — Database-related issues — HikariCP timeout/exhaustion evidence and supported connection-pool tuning.
• Community Build — Installing database — supported production DB engines/versions and evaluation-H2 boundary.
• SonarScanner CLI 8.1.0.6389 — current CLI baseline.
• SonarScanner for Maven 5.7.0.6970 — current Maven scanner baseline used by the optional key-clash failure fixture.