Server Logs, Monitoring, Health, Elasticsearch, and Operational Diagnostics: Core Concepts and Mental Model

1. The practical problem: “the UI opens” is only one signal

Chapters 6 through 26 repeatedly separated scanner success, report upload, ceTaskId, Compute Engine completion, analysis result, Quality Gate, CI status, and provider decoration. Chapter 27 turns that evidence chain into an operating model for the server itself.

A browser can render the SonarQube home page while the Compute Engine queue is backed up. A scanner can print EXECUTION SUCCESS while its uploaded report later fails in Compute Engine. The database can accept web requests while the search engine is under disk pressure. A Quality Gate can remain green because it reflects the last completed analysis even though the newest task is pending or failed. Operational diagnosis therefore starts by naming the subsystem and timestamp, not by asking whether “SonarQube is up.”

2. Mental model: one analysis crosses multiple owning processes

              flowchart TD
                A["Scanner process"] -->|"analysis report"| B["Web server / API"]
                B --> C["CE queue"]
                C --> D["Compute Engine task"]
                D --> E[("SonarQube database")]
                D --> F["Embedded Elasticsearch"]
                E --> G["Project state"]
                F --> G
                G --> H["UI / Web API"]
                B -. "web.log" .-> L1["Process evidence"]
                D -. "ce.log" .-> L1
                F -. "es.log" .-> L1
                A -. "scanner log + report-task.txt" .-> L1
            

The scanner does local indexing/analyzer work and uploads a report. The web process receives HTTP/API traffic and interacts with the database/search layer for request handling. Compute Engine consumes queued background tasks and turns the uploaded report into persisted analysis state. Elasticsearch serves indexed/search-oriented views. If any transition fails, the later stage cannot be inferred from the earlier one.

3. Current server logs: identify the owner before reading noise

On ZIP installs the files live under <sonarqubeHome>/logs by default. Container deployments can preserve the same files on a mounted logs volume and may also expose stdout/stderr. Capture timestamps and the server’s time zone when correlating scanner, CI, container, database, and host logs.

4. Health, status, queue, and gate are different dimensions

That distinction is essential during incidents: a GREEN system health response can coexist with one project’s failed CE task; a RED or YELLOW system signal can be unrelated to a project’s source code; and a green gate can simply be stale.

5. Metrics complement logs; they do not replace them

Current Community Build exposes OpenMetrics-formatted monitoring data at /api/monitoring/metrics. Monitoring authentication is intentionally separate from ordinary project API calls and can use the system passcode configured through sonar.web.systemPasscode/SONAR_WEB_SYSTEMPASSCODE. Kubernetes deployments can additionally expose CE/Web JMX metrics through the supported Prometheus integration.

Use metrics to answer how much and how often: queue depth/latency, JVM memory, CPU, disk, request rates, database pool pressure. Use logs and task records to answer which operation failed and why. A graph tells you when CE latency rose; ce.log and the failed task’s error detail tell you which report or dependency caused the incident.

6. Resource domains: three Java processes plus shared dependencies

Community Build has three main Java processes: Web, Compute Engine, and Elasticsearch. Each has independent memory behavior. The database is an external dependency in production, and the host/container adds CPU, RAM, file-descriptor, process/thread, and disk constraints.

7. Elasticsearch is embedded, owned state—not an operator playground

SonarQube uses embedded Elasticsearch as part of its application architecture. Treat it as SonarQube-owned state. Diagnose through es.log, supported health/monitoring surfaces, disk/IO evidence, and SonarSource recovery procedures. Do not “repair” a SonarQube incident by issuing arbitrary Elasticsearch cluster/index mutations or deleting search directories.

Current Sonar guidance documents disk watermarks: warnings begin before the critical threshold, and at roughly 95% used disk non-Data-Center installations can put indices into read-only mode. After freeing space, the supported non-DCE recovery includes restarting SonarQube so indices return to read/write. The safe causal sequence is preserve logs → prove disk pressure → free space safely → follow documented recovery → verify health.

8. DEBUG and TRACE are temporary diagnostic states

INFO is the default server log level. DEBUG adds advanced information and can include personal user information. TRACE includes very detailed SQL/Elasticsearch request information, can slow the server, and can grow logs rapidly. Current SonarQube lets an administrator change log level temporarily from Administration → System; that temporary choice resets at restart.

9. Read-only inspection first

# Scanner-side identity
sonar-scanner --version
git rev-parse HEAD
cat .scannerwork/report-task.txt

# Public/basic lifecycle endpoint (availability depends on auth policy)
curl -fsS "$SONAR_HOST_URL/api/system/status"

# Admin/system-passcode health request; do not put the passcode in source.
curl -fsS -H "X-Sonar-Passcode: $SONAR_SYSTEM_PASSCODE" \
  "$SONAR_HOST_URL/api/system/health"

# Correlate one uploaded task with its server state.
curl -fsS -H "Authorization: Bearer $SONAR_API_TOKEN" \
  "$SONAR_HOST_URL/api/ce/task?id=$CE_TASK_ID"

# Container lab: preserve process-specific logs before changing anything.
docker cp sq-ch27:/opt/sonarqube/logs/. evidence/logs-before/

The goal is not to collect everything forever. The goal is to preserve the smallest evidence set that can prove ownership, timing, and causal order.

Knowledge check

Official references and version notes

• Community Build — Server logs — current process-specific files, log levels, rotation, JSON output, DEBUG/TRACE privacy/performance cautions.
• Community Build — Monitoring the instance — api/system/health, Web/CE/Elasticsearch JVMs, CPU/RAM/disk monitoring.
• Community Build — Monitoring on Kubernetes — /api/monitoring/metrics, OpenMetrics, and CE/Web JMX exporter model.
• Community Build — Web API — bearer authentication, system-passcode use for monitoring, and continuing Web API V2 transition.
• Current Web API reference — api/system/health — GREEN/YELLOW/RED semantics and Administer System/system-passcode requirement.
• Current Web API reference — Compute Engine — CE activity/task APIs and permission boundaries.
• Background tasks — scanner success vs CE completion, task failure diagnosis, and project/module key-clash example.
• Community Build — Elasticsearch-related issues — disk-watermark/read-only behavior and supported recovery.
• Community Build — Database-related issues — HikariCP timeout/exhaustion evidence and supported connection-pool tuning.
• Community Build — Installing database — supported production DB engines/versions and evaluation-H2 boundary.
• SonarScanner CLI 8.1.0.6389 — current CLI baseline.
• SonarScanner for Maven 5.7.0.6970 — current Maven scanner baseline used by the optional key-clash failure fixture.