Troubleshooting Analysis Failures, Index Problems, CI Drift, and Incidents: Diagnostics, Failure Modes, and Production Practices

1. Production troubleshooting starts by protecting evidence

Incidents create pressure to “make it green.” The dangerous shortcuts in this chapter all remove information or widen blast radius: deleting .scannerwork and logs, restarting every service, editing database/search internals, disabling TLS verification, rotating tokens without proving authentication, changing project keys, or fixing only local execution while CI remains divergent.

The production objective is not merely a successful next run. It is a durable recovery whose cause, correction and unchanged inputs are independently reviewable.

2. Evidence-first diagnostic sequence

The sequence is not rigid bureaucracy. It is a way to avoid skipping from a scanner symptom directly to a database restart. If evidence localizes a TLS failure before report upload, you stop there. If the report uploaded and CE failed, you move to CE/server state.

3. Preserve before cleanup

# Disposable/local example evidence capture. Run from project root.
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
E="incident-$STAMP"
mkdir -p "$E"

git rev-parse HEAD > "$E/revision.txt"
git status --porcelain > "$E/worktree-status.txt"
sonar-scanner -v > "$E/scanner-version.txt" 2>&1 || true

# Copy, do not delete, scanner bridge/workspace evidence.
[ -f .scannerwork/report-task.txt ] && cp .scannerwork/report-task.txt "$E/"
[ -d .scannerwork ] && find .scannerwork -maxdepth 2 -type f -printf '%P
' > "$E/scannerwork-files.txt"

# Preserve server/container logs before restart in this disposable lab.
docker logs sq32-sonarqube > "$E/sonarqube-container.log" 2>&1 || true
docker logs sq32-db > "$E/postgres-container.log" 2>&1 || true
docker stats --no-stream sq32-sonarqube sq32-db > "$E/container-stats.txt" || true

On a ZIP/service deployment, copy the relevant files from <sonarqubeHome>/logs. In container/Kubernetes deployments, preserve logs/events from the correct container/pod before replacement. Never collect secrets indiscriminately; redact tokens, passwords, private keys and sensitive source paths before sharing evidence.

4. Intentionally broken example — upload succeeded, Compute Engine failed

The following is synthetic but mirrors the evidence shape you should recognize. The scanner ended successfully and wrote:

projectKey=sq32:incident-lab
serverUrl=http://localhost:9000
serverVersion=26.9.0.129388
dashboardUrl=http://localhost:9000/dashboard?id=sq32%3Aincident-lab
ceTaskId=AX_SQ32_BROKEN_001
ceTaskUrl=http://localhost:9000/api/ce/task?id=AX_SQ32_BROKEN_001

An authorized read of the task returns:

{
  "task": {
    "id": "AX_SQ32_BROKEN_001",
    "type": "REPORT",
    "componentKey": "sq32:incident-lab",
    "status": "FAILED",
    "executionTimeMs": 832,
    "errorMessage": "Failed to execute project analysis"
  }
}

And the preserved ce.log window contains:

2026.09.08 14:22:31 INFO  ce[][o.s.c.t.CeWorkerImpl] Execute task | project=sq32:incident-lab | id=AX_SQ32_BROKEN_001
2026.09.08 14:22:32 ERROR ce[][o.s.c.t.CeWorkerImpl] Failed to execute task AX_SQ32_BROKEN_001
java.lang.NoClassDefFoundError: example/plugin/RemovedApi
    at example.plugin.CustomSensor.execute(CustomSensor.java:42)

The first failed transition is server-side Compute Engine execution after successful report upload. The classloading signature makes plugin/runtime compatibility a stronger hypothesis than token, URL or scanner source scope.

5. Repair the broken example without hiding the cause

Preserve the installed plugin inventory, SonarQube version, Java runtime and startup logs. Check the plugin's maintainer compatibility guidance for the exact target release. In a disposable lab, the reversible options are: remove the incompatible third-party plugin through the supported plugin lifecycle and restart once as required, or move to a plugin version explicitly compatible with the current SonarQube release. Record the artifact/checksum/source and rollback package.

After repair, rerun the same commit with the same project key and analysis parameters. Compare the new task ID to the original failed task, but do not replace the original evidence. Recovery is proven when the new CE task succeeds and the durable project result updates as expected.

6. Seven failure modes and their causal correction

7. Database/search symptoms: observe before acting

If CE tasks fail with database connectivity/timeout errors, correlate ce.log/web.log with database health and network latency. If project/issues APIs return temporary 503 during indexing or es.log shows search health transitions, preserve that evidence. Do not open the SonarQube database to “fix” rows or manipulate embedded search files as normal troubleshooting.

Database restoration, search reindexing, upgrade recovery and storage repair are operational procedures with version/support boundaries covered in earlier chapters. Incident response should invoke those procedures only when the owning layer is proven.

8. Auth/TLS incidents: distinguish identity, authorization and trust

HTTP 401 suggests authentication failure; 403 suggests an authenticated identity lacks required authorization; a certificate/hostname validation exception is a TLS trust failure. They can look similar in a CI job because all prevent analysis, but they require different corrections.

Use a least-privilege fake/local token in labs. For production, verify token owner/type/expiry and required project/global permission without printing the token. Current API guidance recommends bearer authentication, and responses using tokens can expose an expiration header on supported endpoints. That lets operations detect upcoming expiry without logging secrets.

9. CI drift incidents: preserve the runner as part of the system

A CI runner can change underneath a repository: action/plugin version updates, base image updates, Java changes, working directory changes, cache restore behavior, shallow checkout, proxy changes or secret scope. Capture the job definition revision and runner/action versions alongside SonarQube evidence. Pin dependencies where the provider supports it and review updates intentionally.

If only the local run is repaired, the governed delivery path remains broken. Recovery must include the CI-owned outcome, while still distinguishing a technically successful analysis from gate pass and provider decoration.

10. Production incident record

11. Knowledge check

Official references and version notes

• Troubleshooting the analysis — Current scanner-to-server asynchronous troubleshooting model and analysis progress behavior.
• Server logs — Current log split: sonar.log, web.log, ce.log, es.log and access.log, plus API deprecation logging.
• SonarScanner CLI — Scanner invocation, verbose/debug options, runtime requirements and TLS troubleshooting entry points.
• Scanner environment requirements — Current scanner runtime and JRE auto-provisioning requirements.
• TLS certificates on client side — Supported truststore/keystore paths instead of disabling TLS verification.
• Web API — Bearer authentication guidance and Web API V2 migration warning.
• SonarQube downloads — Current Community Build, Server release train and active LTA identities.
• SonarScanner CLI releases — Current scanner release identity and release notes.
• Docker Official Image — Current Community Build image tags for a pinned disposable local fixture.