Server Logs, Monitoring, Health, Elasticsearch, and Operational Diagnostics: Guided Hands-On Workflow

1. Disposable lab contract

2. Establish a timestamped operational baseline

export SONAR_HOST_URL="http://localhost:9000"
export LAB_TS="$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "evidence/$LAB_TS"/{system,logs,success,conflict,recovery}

date -u +%FT%TZ | tee "evidence/$LAB_TS/system/timestamp.txt"
df -h | tee "evidence/$LAB_TS/system/disk.txt"
free -h 2>/dev/null | tee "evidence/$LAB_TS/system/memory.txt" || true

curl -fsS "$SONAR_HOST_URL/api/system/status" \
  | tee "evidence/$LAB_TS/system/status.json"

# Requires system passcode or an Administer System identity.
curl -fsS -H "X-Sonar-Passcode: $SONAR_SYSTEM_PASSCODE" \
  "$SONAR_HOST_URL/api/system/health" \
  | tee "evidence/$LAB_TS/system/health.json"

# Container deployment: capture without deleting/rotating logs.
docker cp sonarqube:/opt/sonarqube/logs/. "evidence/$LAB_TS/logs/"

Record the log time zone and container/host time. The baseline should say whether health is GREEN/YELLOW/RED, how much disk/RAM is free, and whether the server already had pending/failed tasks. Do not induce a failure on top of an unexplained unhealthy baseline.

3. Trace one normal analysis end to end

Create a tiny standalone project sq-ch27-success, create a project-analysis token, and scan one source file. Keep the token in the environment, not the command line.

mkdir -p sq-ch27-success/src
cd sq-ch27-success
cat > src/math.py <<'PY'
def total(values):
    return sum(values)
PY
cat > sonar-project.properties <<'EOF'
sonar.projectKey=sq-ch27-success
sonar.projectName=Chapter 27 Successful Task
sonar.sources=src
sonar.sourceEncoding=UTF-8
EOF

git init
git config user.email "learner@example.invalid"
git config user.name "SQ Learner"
git add . && git commit -m "chapter27 success baseline"
git rev-parse HEAD | tee "../evidence/$LAB_TS/success/revision.txt"

export SONAR_TOKEN="$SUCCESS_PROJECT_TOKEN"
sonar-scanner -X 2>&1 | tee "../evidence/$LAB_TS/success/scanner.log"
cp .scannerwork/report-task.txt "../evidence/$LAB_TS/success/report-task.txt"
CE_TASK_ID="$(sed -n 's/^ceTaskId=//p' .scannerwork/report-task.txt)"
printf '%s\n' "$CE_TASK_ID" | tee "../evidence/$LAB_TS/success/ceTaskId.txt"

curl -fsS -H "Authorization: Bearer $SONAR_API_TOKEN" \
  "$SONAR_HOST_URL/api/ce/task?id=$CE_TASK_ID" \
  | tee "../evidence/$LAB_TS/success/ce-task.json"

Poll deliberately until the task is terminal; do not infer completion from scanner exit 0. After SUCCESS, capture project measures/gate and the matching ce.log lines by task ID. This becomes the known-good control for the failure experiment.

4. Safe failed-task experiment: a project/component ownership clash

Sonar’s current background-task documentation explicitly lists a clash between an existing project/module key and a component in a new report as a cause of failed background processing. We can use that failure class without memory exhaustion, disk filling, database corruption, or Elasticsearch mutation.

Create these two disposable projects in the local SonarQube UI before scanning:

• Parent project key: academy:sq-ch27-parent
• Standalone child project key: academy:shared

Create separate project-analysis tokens for each. The source tree uses Maven’s natural groupId:artifactId keys, so no artificial sonar.projectKey override is necessary.

mkdir -p sq-ch27-reactor/shared/src/main/java/academy
cd sq-ch27-reactor
cat > pom.xml <<'XML'
<project xmlns="http://maven.apache.org/POM/4.0.0">
  <modelVersion>4.0.0</modelVersion>
  <groupId>academy</groupId>
  <artifactId>sq-ch27-parent</artifactId>
  <version>1.0.0</version>
  <packaging>pom</packaging>
  <modules><module>shared</module></modules>
</project>
XML
cat > shared/pom.xml <<'XML'
<project xmlns="http://maven.apache.org/POM/4.0.0">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>academy</groupId>
    <artifactId>sq-ch27-parent</artifactId>
    <version>1.0.0</version>
  </parent>
  <artifactId>shared</artifactId>
</project>
XML
cat > shared/src/main/java/academy/Shared.java <<'JAVA'
package academy;
public final class Shared { public static int twice(int x) { return x * 2; } }
JAVA

git init
git config user.email "learner@example.invalid"
git config user.name "SQ Learner"
git add . && git commit -m "chapter27 conflict fixture"
git rev-parse HEAD | tee "../evidence/$LAB_TS/conflict/revision.txt"

5. First make the child key belong to its standalone project

cd shared
export SONAR_TOKEN="$SHARED_PROJECT_TOKEN"
mvn -B verify org.sonarsource.scanner.maven:sonar-maven-plugin:5.7.0.6970:sonar \
  2>&1 | tee "../../evidence/$LAB_TS/conflict/shared-standalone-scanner.log"
cp target/sonar/report-task.txt \
  "../../evidence/$LAB_TS/conflict/shared-standalone-report-task.txt" 2>/dev/null || \
cp .scannerwork/report-task.txt \
  "../../evidence/$LAB_TS/conflict/shared-standalone-report-task.txt"
cd ..

Wait for this standalone child task to complete successfully. Now the server owns academy:shared as a project. Preserve the project page/API evidence before creating the conflict.

6. Submit the parent reactor and preserve the first failure

export SONAR_TOKEN="$PARENT_PROJECT_TOKEN"
set +e
mvn -B verify org.sonarsource.scanner.maven:sonar-maven-plugin:5.7.0.6970:sonar \
  2>&1 | tee "../evidence/$LAB_TS/conflict/parent-scanner.log"
SCAN_RC=${PIPESTATUS[0]}
set -e
printf 'scanner_exit=%s\n' "$SCAN_RC" \
  | tee "../evidence/$LAB_TS/conflict/scanner-exit.txt"

# If report-task.txt exists, the report was uploaded and CE owns the next step.
find . -name report-task.txt -print \
  | tee "../evidence/$LAB_TS/conflict/report-task-locations.txt"

Expected current behavior: the scanner can upload a parent report that contains the academy:shared component, and Compute Engine rejects the ownership clash. Preserve the task ID, failed task JSON/error details, scanner context, and matching ce.log lines.

7. Repair only the ownership conflict

After exporting all evidence, delete only the disposable standalone project academy:shared. This releases that component key so it can belong to the parent reactor. Do not change the parent project key, deactivate rules, or alter the database/search engine.

# Perform deletion in the UI after verifying exact key academy:shared.
# Then rerun the SAME source revision from the parent root.
git status --porcelain
git rev-parse HEAD | tee "../evidence/$LAB_TS/recovery/revision.txt"
export SONAR_TOKEN="$PARENT_PROJECT_TOKEN"
mvn -B verify org.sonarsource.scanner.maven:sonar-maven-plugin:5.7.0.6970:sonar \
  2>&1 | tee "../evidence/$LAB_TS/recovery/scanner.log"

find . -name report-task.txt -print \
  | tee "../evidence/$LAB_TS/recovery/report-task-locations.txt"

The repaired parent task should complete successfully at the same Git SHA. That is the causal proof: source did not change; component ownership did.

8. Challenge: choose the owner, not the loudest symptom

You see all of the following at once: the UI works, api/system/health is GREEN, scanner exit code is 0, and one CE task is FAILED with “component already belongs to another project.” Which layer owns the correction?

Answer before revealing: the server/project topology represented by project/component keys. A server restart, more heap, lower gate, or Elasticsearch edit cannot resolve an ownership conflict.

Knowledge check

Official references and version notes

• Community Build — Server logs — current process-specific files, log levels, rotation, JSON output, DEBUG/TRACE privacy/performance cautions.
• Community Build — Monitoring the instance — api/system/health, Web/CE/Elasticsearch JVMs, CPU/RAM/disk monitoring.
• Community Build — Monitoring on Kubernetes — /api/monitoring/metrics, OpenMetrics, and CE/Web JMX exporter model.
• Community Build — Web API — bearer authentication, system-passcode use for monitoring, and continuing Web API V2 transition.
• Current Web API reference — api/system/health — GREEN/YELLOW/RED semantics and Administer System/system-passcode requirement.
• Current Web API reference — Compute Engine — CE activity/task APIs and permission boundaries.
• Background tasks — scanner success vs CE completion, task failure diagnosis, and project/module key-clash example.
• Community Build — Elasticsearch-related issues — disk-watermark/read-only behavior and supported recovery.
• Community Build — Database-related issues — HikariCP timeout/exhaustion evidence and supported connection-pool tuning.
• Community Build — Installing database — supported production DB engines/versions and evaluation-H2 boundary.
• SonarScanner CLI 8.1.0.6389 — current CLI baseline.
• SonarScanner for Maven 5.7.0.6970 — current Maven scanner baseline used by the optional key-clash failure fixture.