Troubleshooting Analysis Failures, Index Problems, CI Drift, and Incidents: Core Concepts and Mental Model

1. The practical problem: the first visible error is often downstream of the cause

By Chapter 31 you learned to follow work through scanner, queue, Compute Engine, database/search and host resources. Troubleshooting uses the same ownership model under failure. A CI job that says “SonarQube failed” is not a diagnosis. It might mean the scanner could not resolve the server URL, the token lacked Execute Analysis, TLS trust failed, indexing excluded every intended source file, the report uploaded but its Compute Engine task failed, the project policy evaluated as failed, or the CI wrapper interpreted a later gate result as job failure.

The practical discipline is therefore preserve, classify, correlate, test. Preserve the earliest evidence before rerunning. Classify the layer that produced the evidence. Correlate source revision, effective scanner inputs, report/task identifiers and server state. Then test one hypothesis with the least destructive change. If you make several unrelated changes at once, a successful rerun tells you very little about causality.

2. The incident loop: symptom → evidence → hypothesis → same-input proof

A symptom starts the investigation but does not own the fix. “401,” “PKIX path building failed,” “0 files indexed,” “report uploaded but dashboard unchanged,” “CE task FAILED,” “works locally but not in CI,” and “issues API returns 503 while indexing” each point at different boundaries. Classification narrows what evidence can falsify a hypothesis.

Preservation comes before minimal reproduction because a reproduction may warm caches, rotate logs, generate a different task ID or run a different revision. Once the original packet exists, reproduce the smallest equivalent scenario: same project key, same commit, same scanner family/version, same effective analysis scope, same report inputs and—when diagnosing CI drift—the same working-directory/path semantics. Test one hypothesis. Apply the narrowest reversible correction. Finally, rerun the same input and verify both the immediate symptom and the durable server result.

3. Eight layers to keep separate

The strongest troubleshooting habit is to name the state owner before touching configuration. These layers interact, but they are not interchangeable.

4. State before action: build the incident manifest

An incident manifest is a compact inventory of facts that must remain stable while you test hypotheses. Record secret identifiers or token type/owner—not token values.

5. Read-only inspection first

The following commands are examples for a disposable lab or authorized workstation. They deliberately avoid printing SONAR_TOKEN. On Windows PowerShell, use the equivalent commands shown below. API endpoints evolve; verify them in the target instance's built-in Web API documentation before operational automation.

# Source and scanner identity
printf 'revision='; git rev-parse HEAD
git status --porcelain
sonar-scanner -v
java -version 2>&1 | head -n 2

# Server identity and basic reachability: no token printed
curl -fsS http://localhost:9000/api/server/version ; echo
curl -fsS http://localhost:9000/api/system/status ; echo

# Preserve the scanner-to-CE bridge if analysis reached report upload
if [ -f .scannerwork/report-task.txt ]; then
  cp .scannerwork/report-task.txt evidence-report-task.txt
  sed -n '1,20p' evidence-report-task.txt
fi

# Container/process evidence in the disposable lab
docker ps --format 'table {{.Names}}	{{.Image}}	{{.Status}}'
docker stats --no-stream sq32-sonarqube sq32-db

$Revision = git rev-parse HEAD
"revision=$Revision"
git status --porcelain
sonar-scanner -v
java -version

Invoke-RestMethod http://localhost:9000/api/server/version
Invoke-RestMethod http://localhost:9000/api/system/status

if (Test-Path .scannerwork/report-task.txt) {
  Copy-Item .scannerwork/report-task.txt evidence-report-task.txt
  Get-Content evidence-report-task.txt
}

docker ps --format "table {{.Names}}	{{.Image}}	{{.Status}}"
docker stats --no-stream sq32-sonarqube sq32-db

6. The scanner-to-task evidence chain

A successful scanner process is only one link. When the report uploads, the scanner writes a small task bridge file in .scannerwork. Preserve it. Its task identifier lets you ask the server what happened after upload. A scanner can finish upload successfully while the Compute Engine later marks the task failed.

For a task whose ID is known, the documented api/ce/task service returns task details to appropriately authorized callers. Do not assume anonymous access or administrator permission. In a lab, use a project-scoped identity with the minimum permission that can read the task.

CE_TASK_ID="AX_FAKE_TASK_ID_FROM_REPORT_TASK_TXT"
curl -fsS   -H "Authorization: Bearer ${SONAR_TOKEN}"   "http://localhost:9000/api/ce/task?id=${CE_TASK_ID}"   > evidence-ce-task.json
python -m json.tool evidence-ce-task.json | sed -n '1,120p'

If there is no report-task.txt, do not jump to CE logs. First prove whether scanning reached report generation/upload. That distinction saves time and avoids unrelated server changes.

7. Translate common symptoms into first hypotheses

These mappings are starting hypotheses, not final answers. Evidence can contradict them.

8. What not to do first

Deleting .scannerwork, restarting every service, clearing CI caches, rotating tokens, changing project keys, disabling TLS verification or reinstalling plugins all change evidence. Some can be valid later, but only after a hypothesis points to them and rollback is explicit. Direct edits to the SonarQube database or embedded search indexes are not normal troubleshooting mechanisms and can turn a diagnosable incident into data corruption.

Controlled retry: a retry is useful when it tests the hypothesis “this was transient.” It must preserve the original attempt, use the same input, and be bounded. An unrecorded loop of retries is evidence destruction by noise.

9. Why this matters in DevOps

A SonarQube control is reproducible only when another engineer can independently verify the exact revision, effective analysis inputs, asynchronous task/result, policy context and owning system. Troubleshooting is therefore part of governance. The incident record should show which layer owned the fault, what evidence proved it, what minimal correction changed, what remained unchanged, and what recovery criterion passed.

This prevents two common organizational failures: blaming application developers for platform faults and weakening quality/security policy to hide operational faults. A failed gate is not repaired by changing the scanner. A TLS failure is not repaired by lowering a gate. A CE queue problem is not repaired by changing the project key.

10. Mini lab — classify before you repair

For each observation below, write the first layer you would inspect and one artifact you would preserve. Do not propose a fix yet.

1. Scanner log ends with HTTP 401 before any “report uploaded” line.
2. Scanner reports 148 files indexed and gives a task URL, but the project page shows an analysis warning.
3. Local run succeeds at commit abc123; CI fails and its manifest shows commit abc120.
4. Both scanner and UI requests slow down while database latency spikes.

The exercise trains sequence: identity and evidence first, repair second.

11. Knowledge check

Official references and version notes

• Troubleshooting the analysis — Current scanner-to-server asynchronous troubleshooting model and analysis progress behavior.
• Server logs — Current log split: sonar.log, web.log, ce.log, es.log and access.log, plus API deprecation logging.
• SonarScanner CLI — Scanner invocation, verbose/debug options, runtime requirements and TLS troubleshooting entry points.
• Scanner environment requirements — Current scanner runtime and JRE auto-provisioning requirements.
• TLS certificates on client side — Supported truststore/keystore paths instead of disabling TLS verification.
• Web API — Bearer authentication guidance and Web API V2 migration warning.
• SonarQube downloads — Current Community Build, Server release train and active LTA identities.
• SonarScanner CLI releases — Current scanner release identity and release notes.
• Docker Official Image — Current Community Build image tags for a pinned disposable local fixture.