Server Logs, Monitoring, Health, Elasticsearch, and Operational Diagnostics: Configuration, Design Patterns, and Trade-Offs

1. Design principle: monitor states that fail independently

A useful SonarQube dashboard should not collapse the service into one “up/down” light. Monitor at least four independent planes: request serving, background analysis processing, persistence/search dependencies, and host/JVM capacity. Then link every alert to the first evidence an operator should inspect.

2. Application health versus analysis queue health

api/system/health is a coarse global signal: GREEN means operational, YELLOW means usable but needs attention, RED means not operational. It does not tell you that every queued analysis is succeeding or that queue latency is acceptable. Queue state belongs to CE activity and CE metrics.

3. Logs versus metrics

Logs preserve discrete events with identities: task X failed at 12:04:11 because of Y. Metrics compress repeated observations into a time series: CE pending count rose from 2 to 80 over 20 minutes while heap and CPU changed. Both are necessary for reliable incident analysis.

4. DEBUG/TRACE versus normal logging

Use INFO continuously and collect it centrally with retention appropriate to your environment. Increase verbosity only after the first incident window has been preserved and only for the process implicated by evidence. Current properties allow process-specific levels: sonar.log.level.app, .web, .ce, and .es.

5. Process restart versus task remediation

A restart is appropriate only when the owning subsystem’s documented recovery requires it—for example, after resolving the non-DCE Elasticsearch disk watermark/read-only condition. It is not a universal task-retry mechanism.

6. Infrastructure failure versus project-specific failure

Project-specific failures tend to cluster around one ceTaskId, one scanner context, one project key, or one report shape. Infrastructure failures recur across unrelated projects and correlate with host/DB/search/JVM signals. Do not infer either class from a single error message—compare neighboring tasks and time windows.

incident triage questions
1. Did unrelated projects fail in the same minute?
2. Did system health or search health change?
3. Did CE queue age grow globally?
4. Is the error tied to one project key/report context?
5. Are DB pool / disk / GC / host metrics abnormal?
6. What changed immediately before the first failure?

7. Database operational evidence

Production SonarQube uses a supported external database. Current Community Build supports PostgreSQL 14–18, SQL Server, and Oracle under documented requirements. The application uses a JDBC/HikariCP connection pool; timeouts or exhausted connections appear in web/CE logs and must be correlated with DB-side connections/CPU/latency and firewall behavior.

Do not tune sonar.jdbc.maxActive, keepalive, lifetime, or timeout values because “the UI feels slow.” Capture pool errors and DB metrics first. Directly editing SonarQube tables is outside supported incident repair.

8. Search operational evidence

Monitor disk space and IO latency continuously. Search degradation can surface in web and CE because both depend on indexed state. Current SonarQube guidance documents ES warnings at disk pressure and a read-only safety action around the 95% used watermark. Preserve es.log before freeing space so the root cause remains auditable.

9. Worked decision table

10. Preserve analysis evidence across operational decisions

Even in an infrastructure incident, keep the course’s established chain: scanner process result → report upload → report-task.txt/ceTaskId → CE terminal status → analysis result → Quality Gate → CI job → provider decoration. A restart or metric alert never substitutes for this project-level provenance.

Knowledge check

Official references and version notes

• Community Build — Server logs — current process-specific files, log levels, rotation, JSON output, DEBUG/TRACE privacy/performance cautions.
• Community Build — Monitoring the instance — api/system/health, Web/CE/Elasticsearch JVMs, CPU/RAM/disk monitoring.
• Community Build — Monitoring on Kubernetes — /api/monitoring/metrics, OpenMetrics, and CE/Web JMX exporter model.
• Community Build — Web API — bearer authentication, system-passcode use for monitoring, and continuing Web API V2 transition.
• Current Web API reference — api/system/health — GREEN/YELLOW/RED semantics and Administer System/system-passcode requirement.
• Current Web API reference — Compute Engine — CE activity/task APIs and permission boundaries.
• Background tasks — scanner success vs CE completion, task failure diagnosis, and project/module key-clash example.
• Community Build — Elasticsearch-related issues — disk-watermark/read-only behavior and supported recovery.
• Community Build — Database-related issues — HikariCP timeout/exhaustion evidence and supported connection-pool tuning.
• Community Build — Installing database — supported production DB engines/versions and evaluation-H2 boundary.
• SonarScanner CLI 8.1.0.6389 — current CLI baseline.
• SonarScanner for Maven 5.7.0.6970 — current Maven scanner baseline used by the optional key-clash failure fixture.