Server Logs, Monitoring, Health, Elasticsearch, and Operational Diagnostics: Diagnostics, Failure Modes, and Production Practices

1. Evidence-first diagnostic sequence

1. Preserve first failure: scanner/CI log, report-task.txt, task JSON/error, relevant process logs, timestamps, and resource snapshots.
2. Confirm versions: Community Build/Server edition, exact release, Java, database, scanner, plugins/integrations, container image.
3. Confirm source and scanner context: Git SHA, base directory, indexed files, effective parameters.
4. Confirm report/task transition: was a report uploaded? What is the ceTaskId? PENDING, IN_PROGRESS, SUCCESS, FAILED?
5. Confirm project/policy state: gate/new-code/profile/security evidence only after analysis completion.
6. Confirm shared dependencies: auth/network, DB pool/latency, search health/disk, JVM/GC, host/container resources.
7. Apply the least destructive correction at the owning layer.
8. Rerun the smallest equivalent scenario and independently verify recovery.

2. Failure: deleting or rotating logs before diagnosis

Deleting logs can erase the only copy of a startup stack trace, CE failure, disk-watermark transition, or DB timeout. Copy the incident window first. If storage pressure requires cleanup, move/compress already-preserved logs according to retention policy; do not destroy the root-cause evidence and then ask why the service failed.

3. Failure: leaving DEBUG/TRACE enabled indefinitely

DEBUG can log personal user information; TRACE adds SQL/Elasticsearch request detail and can materially increase volume and overhead. A useful runbook contains four fields: process, temporary level, start time, stop condition. “Turn on debug and see what happens” is not a controlled diagnostic action.

4. Failure: assuming UI availability means CE/search are healthy

The web server can serve cached/previous project state while the CE queue is delayed or a new task has failed. Search may be degraded while some pages still load. Always compare the newest analysis timestamp to the task submission/completion time and confirm the exact ceTaskId.

5. Failure: repeated restart as a queue strategy

Restarting can interrupt in-flight work, add a new startup event to the evidence stream, and leave the causal defect unchanged. A component-key clash, malformed scanner context, project permission issue, or report-specific failure will return after restart. Use task evidence first; restart only for a supported recovery reason.

6. Failure: touching Elasticsearch internals directly

Do not use Elasticsearch APIs, delete indices, or modify search settings directly to “unstick” SonarQube. Diagnose search via es.log, system health, host disk/IO, and documented SonarQube recovery. For disk-watermark read-only behavior, free disk and follow Sonar’s recovery procedure instead of overriding ES safety settings.

7. Failure: ignoring DB saturation

Errors such as HikariPool-1 - Connection is not available or Cannot acquire connection from data source point toward the DB connection path/pool. Correlate web/CE logs with database connection count, CPU, latency, firewall timeouts, and the current JDBC pool configuration. Do not edit SonarQube tables.

8. Failure: treating memory pressure as “just slow”

Monitor the Web, CE, and Elasticsearch JVMs independently. Sustained high heap, long GC pauses, or host swapping can manifest as request latency, queue delay, or search instability. The correct response is capacity/evidence-driven tuning—not one global heap change copied from a forum post.

9. Intentionally broken example: scanner succeeds, CE fails

Use the disposable key-clash fixture from Lesson 2. Preserve the contradiction rather than “fixing” it immediately:

scanner process: SUCCESS / report uploaded
report-task.txt: contains ceTaskId=AX...
api/ce/task: status=FAILED
ce.log: task AX... reports project/component ownership clash
api/system/health: GREEN
UI: last successful analysis still visible
Quality Gate: belongs to last completed analysis, not the failed task

This is exactly why operational state must remain multi-dimensional. The server is globally healthy enough to reject an invalid topology correctly. Restarting the server would only replay the same invalid ownership relationship.

10. Failure-layer matrix

11. Security-sensitive actions

12. Production operating practices

• Centralize logs with timestamp normalization and retention; restrict DEBUG/TRACE access.
• Alert on system health plus CE queue age/depth, host disk, JVM memory/GC, DB pool/latency, and search health.
• Correlate every analysis incident with ceTaskId and revision.
• Subscribe responsible project/admin users to background-task failure notifications where appropriate.
• Keep API deprecation logs in upgrade readiness reviews.
• Document restart prerequisites and reasons; count restarts as operational events, not fixes.
• Never modify SonarQube database or embedded Elasticsearch state directly.

Knowledge check

Official references and version notes

• Community Build — Server logs — current process-specific files, log levels, rotation, JSON output, DEBUG/TRACE privacy/performance cautions.
• Community Build — Monitoring the instance — api/system/health, Web/CE/Elasticsearch JVMs, CPU/RAM/disk monitoring.
• Community Build — Monitoring on Kubernetes — /api/monitoring/metrics, OpenMetrics, and CE/Web JMX exporter model.
• Community Build — Web API — bearer authentication, system-passcode use for monitoring, and continuing Web API V2 transition.
• Current Web API reference — api/system/health — GREEN/YELLOW/RED semantics and Administer System/system-passcode requirement.
• Current Web API reference — Compute Engine — CE activity/task APIs and permission boundaries.
• Background tasks — scanner success vs CE completion, task failure diagnosis, and project/module key-clash example.
• Community Build — Elasticsearch-related issues — disk-watermark/read-only behavior and supported recovery.
• Community Build — Database-related issues — HikariCP timeout/exhaustion evidence and supported connection-pool tuning.
• Community Build — Installing database — supported production DB engines/versions and evaluation-H2 boundary.
• SonarScanner CLI 8.1.0.6389 — current CLI baseline.
• SonarScanner for Maven 5.7.0.6970 — current Maven scanner baseline used by the optional key-clash failure fixture.