Troubleshooting Analysis Failures, Index Problems, CI Drift, and Incidents: Guided Hands-On Workflow

1. Workflow goal: one fixture, several faults, one diagnostic sequence

This lesson turns the Chapter 32 mental model into a reproducible local workflow. The fixture stays intentionally small so you can rerun it repeatedly. Every drill uses the same synthetic Git repository, stable project key sq32:incident-lab, pinned Community Build image, recorded scanner identity, fake local token and PostgreSQL container. The purpose is not to create every possible SonarQube failure. It is to practice preserving evidence and isolating the layer.

Commercial features are not required. A server-side plugin/Compute Engine mismatch is represented by a faithful saved fixture rather than installing an incompatible plugin into a live server. That keeps the mandatory path free and safe while preserving the reasoning task.

2. Version, edition and trust assumptions

3. Create the disposable server and database

The exact PostgreSQL patch behind postgres:17 can move. For a classroom run, record the resolved image digest before starting. If you need byte-for-byte reproducibility, pin that digest in your own lab manifest.

set -euo pipefail
LAB="$PWD/sq32-lab"
mkdir -p "$LAB"/{repo,evidence,ci-workspace}
cd "$LAB"

docker pull postgres:17
docker pull sonarqube:26.9.0.129388-community

docker image inspect postgres:17 --format '{{index .RepoDigests 0}}' > evidence/postgres-image.txt
docker image inspect sonarqube:26.9.0.129388-community --format '{{index .RepoDigests 0}}' > evidence/sonarqube-image.txt

docker network create sq32-net 2>/dev/null || true
docker run -d --name sq32-db --network sq32-net   -e POSTGRES_USER=sonar -e POSTGRES_PASSWORD=sonar_lab_only   -e POSTGRES_DB=sonar postgres:17

docker run -d --name sq32-sonarqube --network sq32-net -p 9000:9000   -e SONAR_JDBC_URL='jdbc:postgresql://sq32-db:5432/sonar'   -e SONAR_JDBC_USERNAME=sonar   -e SONAR_JDBC_PASSWORD=sonar_lab_only   sonarqube:26.9.0.129388-community

# Read-only startup evidence. Do not hide startup failures with restart loops.
docker ps --filter name=sq32 --format 'table {{.Names}}	{{.Image}}	{{.Status}}'
docker logs sq32-sonarqube > evidence/server-startup.log 2>&1

Once the disposable instance is ready, sign in locally and create a project with key sq32:incident-lab. Create a non-admin lab user, grant that user Execute Analysis on this project, then open User menu → My Account → Security and generate a local token for that user. Store it in SONAR_TOKEN for the shell session; do not write it into sonar-project.properties or the evidence packet. Record only token owner/type/expiry and the permission outcome.

4. Create one synthetic repository and freeze its baseline revision

cd "$LAB/repo"
git init
cat > app.py <<'EOF'
def greet(name):
    if name is None:
        return "hello"
    return "hello " + name

print(greet("SonarQube"))
EOF
cat > sonar-project.properties <<'EOF'
sonar.projectKey=sq32:incident-lab
sonar.projectName=SQ32 Incident Lab
sonar.sources=.
sonar.sourceEncoding=UTF-8
EOF
git add app.py sonar-project.properties
git -c user.name='SQ32 Lab' -c user.email='sq32@example.invalid' commit -m 'baseline fixture'
git rev-parse HEAD | tee ../evidence/baseline-revision.txt
sonar-scanner -v > ../evidence/scanner-version.txt 2>&1

Do one successful baseline analysis before injecting faults. Preserve its scanner log, report-task file and CE task JSON. This gives you a known-good comparison and verifies that the local token/server/project path is valid before fault drills begin.

5. Baseline: prove the complete evidence chain

cd "$LAB/repo"
export SONAR_HOST_URL='http://localhost:9000'
# SONAR_TOKEN is set interactively or by a local secret mechanism; do not echo it.
sonar-scanner -X > ../evidence/baseline-scanner.log 2>&1
cp .scannerwork/report-task.txt ../evidence/baseline-report-task.txt

CE_TASK_ID=$(awk -F= '$1=="ceTaskId"{print $2}' .scannerwork/report-task.txt)
curl -fsS -H "Authorization: Bearer ${SONAR_TOKEN}"   "http://localhost:9000/api/ce/task?id=${CE_TASK_ID}"   > ../evidence/baseline-ce-task.json

git rev-parse HEAD > ../evidence/baseline-rerun-revision.txt

6. Fault drill A — invalid token: authentication before analysis

Do not overwrite the known-good token. Shadow it for one command so you can prove the repair is limited to credential selection.

cd "$LAB/repo"
set +e
SONAR_TOKEN='sq_fake_invalid_token'   sonar-scanner -X > ../evidence/fault-a-invalid-token.log 2>&1
A_RC=$?
set -e
printf 'exit_code=%s
' "$A_RC" > ../evidence/fault-a-status.txt

grep -Ei '401|unauthor|auth|token|fail|error' ../evidence/fault-a-invalid-token.log   | head -n 30 > ../evidence/fault-a-key-lines.txt || true

Interpretation: if authentication fails before report upload, no CE task exists for this attempt. Rotating server certificates, restarting Compute Engine or inspecting database indexes would be wrong-layer activity. Restore the known-good fake/local token and rerun the same revision. Preserve both logs.

7. Fault drill B — wrong URL: network/addressing state

cd "$LAB/repo"
set +e
SONAR_HOST_URL='http://localhost:9900'   sonar-scanner -X > ../evidence/fault-b-wrong-url.log 2>&1
B_RC=$?
set -e
printf 'exit_code=%s
' "$B_RC" > ../evidence/fault-b-status.txt

# Read-only reachability comparison from the same host namespace
curl -sS -o /dev/null -w 'good_url_http=%{http_code}
' http://localhost:9000/api/system/status   > ../evidence/fault-b-reachability.txt
curl -sS -o /dev/null -w 'bad_url_http=%{http_code}
' http://localhost:9900/api/system/status   >> ../evidence/fault-b-reachability.txt || true

Repair only SONAR_HOST_URL. Do not create a second project key to “see if it works.” Project identity is not the failed layer.

8. Fault drill C — wrong source path: deterministic scanner configuration

Command-line analysis parameters have high precedence. This drill intentionally overrides repository configuration for one run without editing the committed file.

cd "$LAB/repo"
set +e
sonar-scanner -X -Dsonar.sources=src-does-not-exist   > ../evidence/fault-c-bad-scope.log 2>&1
C_RC=$?
set -e
printf 'exit_code=%s
' "$C_RC" > ../evidence/fault-c-status.txt

grep -Ei 'base dir|source|index|file|does-not-exist|error|fail'   ../evidence/fault-c-bad-scope.log | head -n 60   > ../evidence/fault-c-key-lines.txt || true

Because this is deterministic configuration, blind retry should reproduce it. The minimal repair is to remove the bad override and prove the effective source path returns to the committed value.

9. Fault drill D — external report path drift without pretending SonarQube runs tests

SonarQube imports reports produced by other tools. To practice report-path diagnosis, create a harmless synthetic coverage placeholder and point the scanner at a deliberately wrong path only if the selected language/report importer supports the format. Otherwise use the path mismatch as a configuration/evidence exercise rather than claiming a real coverage import.

10. Fault drill E — TLS/network trust simulation

The mandatory lab does not weaken TLS. Instead, use a saved failure snippet such as PKIX path building failed and inspect what the supported repair would change: the scanner truststore or authorized reverse-proxy certificate chain. Current scanner guidance supports PKCS#12 truststores for recent CLI/NPM scanners. The wrong fix is curl -k, disabled certificate verification or accepting an arbitrary certificate.

# Example supported shape for a recent scanner; use only a fake/local certificate.
# Do not run against a production trust boundary without authorization.
sonar-scanner   -Dsonar.scanner.truststorePath="$HOME/.sonar/ssl/truststore.p12"   -Dsonar.scanner.truststorePassword="$SQ32_TRUSTSTORE_PASSWORD"

The password itself belongs in a secret mechanism, not the repository. Preserve the original TLS error before changing trust.

11. Safe Compute Engine / plugin mismatch drill: interpret a fixture, do not corrupt the server

Deliberately installing an incompatible plugin merely to force CE failure is disproportionate for a beginner lab. Use this synthetic task/log pair and practice correlation instead:

{
  "task": {
    "id": "AX_SQ32_FAKE_CE_TASK",
    "type": "REPORT",
    "componentKey": "sq32:incident-lab",
    "status": "FAILED",
    "submittedAt": "2026-09-08T12:00:00+0000",
    "executionTimeMs": 1240
  }
}

# Synthetic ce.log excerpt for reasoning only
2026.09.08 12:00:04 ERROR ce[][o.s.s.c.t.CeWorkerImpl] Failed to execute task AX_SQ32_FAKE_CE_TASK
java.lang.NoClassDefFoundError: example/plugin/RemovedApi
    at example.plugin.Sensor.execute(Sensor.java:42)

The evidence points to server-side executable compatibility after report submission, not scanner auth. The production pattern is to compare plugin inventory against the target SonarQube compatibility matrix, preserve startup/CE evidence, and roll back or upgrade the plugin through a supported change window. Never edit plugin state in the database.

12. CI drift drill — same repository, different workspace semantics

Copy the repository into a second directory that mimics a CI checkout and deliberately run from its parent. Relative paths now resolve differently. Capture pwd, revision and the scanner base directory. The repair should change the CI working directory or repository configuration—not the global SonarQube server.

rm -rf "$LAB/ci-workspace/repo"
git clone -q "$LAB/repo" "$LAB/ci-workspace/repo"
cd "$LAB/ci-workspace"
{
  printf 'cwd='; pwd
  printf 'revision='; git -C repo rev-parse HEAD
  sonar-scanner -v
} > "$LAB/evidence/ci-drift-manifest.txt" 2>&1

# Deliberately wrong cwd: scanner cannot use repo/sonar-project.properties as intended.
set +e
sonar-scanner -X > "$LAB/evidence/ci-drift-failure.log" 2>&1
set -e

Then rerun from $LAB/ci-workspace/repo and prove the revision matches the baseline. If CI still differs, compare runner image, environment variable names, secret scope, proxy/trust state, cache keys and generated report paths one by one.

13. Challenge — choose the layer, not the command

You receive three artifacts: (1) a scanner log that shows report upload and a CE task URL, (2) the CE task JSON says SUCCESS, and (3) the CI job is red because its quality-gate wait step timed out while the SonarQube project later shows a passing gate. Which layer should you investigate first?

Do not change scanner scope, database indexes or quality-gate conditions. Start with CI/wrapper timing/network behavior and the exact wait integration, because analysis and CE processing already succeeded and the durable gate later passed.

14. Knowledge check

Official references and version notes

• Troubleshooting the analysis — Current scanner-to-server asynchronous troubleshooting model and analysis progress behavior.
• Server logs — Current log split: sonar.log, web.log, ce.log, es.log and access.log, plus API deprecation logging.
• SonarScanner CLI — Scanner invocation, verbose/debug options, runtime requirements and TLS troubleshooting entry points.
• Scanner environment requirements — Current scanner runtime and JRE auto-provisioning requirements.
• TLS certificates on client side — Supported truststore/keystore paths instead of disabling TLS verification.
• Web API — Bearer authentication guidance and Web API V2 migration warning.
• SonarQube downloads — Current Community Build, Server release train and active LTA identities.
• SonarScanner CLI releases — Current scanner release identity and release notes.
• Docker Official Image — Current Community Build image tags for a pinned disposable local fixture.