Checkpoint Lab — Troubleshooting Analysis Failures, Index Problems, CI Drift, and Incidents

1. Checkpoint mission

Your task is to diagnose three deliberately different failures—authentication, deterministic scanner/workspace configuration, and asynchronous server/Compute Engine behavior—without mixing corrections. For each failure you must write a hypothesis before changing anything, preserve the first-failure packet, make one minimal repair, and prove the rerun uses the same source revision and intended workload/configuration.

The server-side failure uses a faithful synthetic CE/plugin mismatch fixture so the mandatory path does not require intentionally destabilizing a real SonarQube server. All executable resources are disposable and local.

2. Exact assumptions and resource preflight

3. Predict state changes before you execute

Write at least these predictions, plus your own two:

4. Setup and baseline

Reuse the Chapter 32 disposable stack or create a new one with names prefixed sq32cp-. Create a tiny Git repository and commit it before analysis. Generate a baseline evidence directory.

set -euo pipefail
LAB="$PWD/sq32-checkpoint"
mkdir -p "$LAB"/{repo,evidence,ci-workspace,fixtures}
cd "$LAB/repo"
git init
cat > hello.py <<'EOF'
def normalize(value):
    return value.strip().lower() if value else ""
print(normalize(" SonarQube "))
EOF
cat > sonar-project.properties <<'EOF'
sonar.projectKey=sq32:checkpoint
sonar.projectName=SQ32 Checkpoint
sonar.sources=.
sonar.sourceEncoding=UTF-8
EOF
git add .
git -c user.name='SQ32 Checkpoint' -c user.email='sq32@example.invalid' commit -m 'checkpoint baseline'
REVISION=$(git rev-parse HEAD)
printf '%s
' "$REVISION" | tee ../evidence/revision.txt
sonar-scanner -v > ../evidence/scanner-version.txt 2>&1

Start the pinned Community Build/PostgreSQL containers as shown in Lesson 2. In the disposable UI create project sq32:checkpoint, create/use a non-admin lab user with Execute Analysis only for that project, then generate its local token under User menu → My Account → Security and inject it through SONAR_TOKEN. Run one baseline analysis to prove the stack before injecting faults; record permission/token metadata without recording the token value.

5. Hypothesis/evidence worksheet

Use one row per fault and do not fill “repair” until the preserved evidence supports it.

6. Failure A — authentication

cd "$LAB/repo"
BASE_SHA=$(git rev-parse HEAD)
set +e
SONAR_TOKEN='sq32_fake_invalid_checkpoint_token'   sonar-scanner -X > ../evidence/a-first-failure.log 2>&1
A_RC=$?
set -e
printf 'exit_code=%s
revision=%s
' "$A_RC" "$BASE_SHA" > ../evidence/a-manifest.txt

Preserve the log before rerun. State the hypothesis: invalid credentials prevent analysis before report submission. Repair by restoring the valid fake/local token only. Rerun the same SHA and compare project key/scope. Do not restart the server.

7. Failure B — CI workspace/configuration drift

rm -rf "$LAB/ci-workspace/repo"
git clone -q "$LAB/repo" "$LAB/ci-workspace/repo"
cd "$LAB/ci-workspace"
{
  printf 'cwd='; pwd
  printf 'revision='; git -C repo rev-parse HEAD
  sonar-scanner -v
} > "$LAB/evidence/b-first-manifest.txt" 2>&1

set +e
sonar-scanner -X > "$LAB/evidence/b-first-failure.log" 2>&1
B_RC=$?
set -e
printf 'exit_code=%s
' "$B_RC" >> "$LAB/evidence/b-first-manifest.txt"

Hypothesis: the simulated CI runner starts in the wrong directory, so repository scanner configuration is not applied as intended. Repair by setting the CI working directory to $LAB/ci-workspace/repo—not by changing global SonarQube settings. Prove the commit equals evidence/revision.txt and the intended project key remains sq32:checkpoint.

8. Failure C — asynchronous Compute Engine/plugin mismatch fixture

Create these two evidence files. They are explicitly simulated; do not install an incompatible plugin to reproduce them.

projectKey=sq32:checkpoint
serverVersion=26.9.0.129388
ceTaskId=AX_SQ32_CP_FAILED_003
ceTaskUrl=http://localhost:9000/api/ce/task?id=AX_SQ32_CP_FAILED_003

{
  "task": {
    "id": "AX_SQ32_CP_FAILED_003",
    "componentKey": "sq32:checkpoint",
    "type": "REPORT",
    "status": "FAILED",
    "errorMessage": "Failed to execute project analysis"
  }
}

2026.09.08 16:10:10 ERROR ce[][o.s.c.t.CeWorkerImpl] Failed to execute task AX_SQ32_CP_FAILED_003
java.lang.NoClassDefFoundError: example/plugin/RemovedApi
    at example.plugin.CustomSensor.execute(CustomSensor.java:42)

Hypothesis: report submission reached CE, but a server-side executable compatibility fault broke processing. Your repair proposal must identify the supported plugin lifecycle/compatibility action and rollback evidence; because no plugin is actually installed in the mandatory lab, record the repair as a simulation decision. Do not claim you executed a server mutation.

9. Prove same revision, workload and configuration after each repair

A successful rerun is comparable only if you can show the inputs that should remain constant actually remained constant.

cd "$LAB/repo"
{
  printf 'revision='; git rev-parse HEAD
  printf 'project_key='; awk -F= '$1=="sonar.projectKey"{print $2}' sonar-project.properties
  printf 'sources='; awk -F= '$1=="sonar.sources"{print $2}' sonar-project.properties
  printf 'tracked_tree='; git ls-tree -r --name-only HEAD | sha256sum
  sonar-scanner -v
} > ../evidence/final-input-manifest.txt 2>&1

diff -u ../evidence/revision.txt <(git rev-parse HEAD)

For CI, create the equivalent manifest from inside the corrected workspace. If a generated report is part of the workload, hash it too. A changed token is expected only for the auth repair; a changed source SHA is not.

10. Required evidence packet

Package evidence, not secrets. At minimum include:

• revision/build manifest and dirty-state result;
• scanner version/runtime and non-secret effective analysis parameters;
• baseline indexed-file/report evidence;
• first-failure scanner logs for A and B;
• baseline and repaired report-task.txt/CE task status where executable;
• synthetic CE task/ce.log fixture for C, clearly labeled simulation;
• server/container logs or health snapshot from the disposable lab;
• token owner/type/permission/expiry record without token value;
• plugin inventory proving the executable lab has no third-party plugin;
• CI workspace/cwd/revision manifests;
• hypothesis/evidence worksheet, repairs, rollback criteria and limitations note.

11. Verification checklist

12. Cleanup and rollback

Cleanup is intentionally last. Preserve the evidence packet outside the disposable directory first. Then remove only resources whose names match the lab prefix.

# Guard ownership by exact disposable names.
for c in sq32-sonarqube sq32-db sq32cp-sonarqube sq32cp-db; do
  if docker ps -a --format '{{.Names}}' | grep -Fxq "$c"; then
    docker rm -f "$c"
  fi
done
for n in sq32-net sq32cp-net; do
  docker network inspect "$n" >/dev/null 2>&1 && docker network rm "$n" || true
done
# Keep the evidence packet; delete the synthetic source only when you no longer need the lab.

Do not delete shared Docker volumes/networks by broad filters. If you pinned images only for this lab, image removal is optional and should also be guarded by exact identity.

13. Knowledge check

14. What Chapter 32 adds to the governed SonarQube operating model

Chapter 32 adds an incident evidence contract. Operational teams now have a repeatable method to preserve first failure, identify state ownership, distinguish scanner/report/upload/CE/gate/CI outcomes, test one hypothesis, apply a narrow reversible correction and prove recovery with the original workload identity. That is the difference between “we reran until it passed” and an auditable incident response.

Chapter 33 builds on this discipline at organizational scale: enterprise standards, auditability, developer adoption and quality programs. The evidence packet you created here becomes a model for change records, exceptions, operating standards and reviewable quality governance.

Official references and version notes

• Troubleshooting the analysis — Current scanner-to-server asynchronous troubleshooting model and analysis progress behavior.
• Server logs — Current log split: sonar.log, web.log, ce.log, es.log and access.log, plus API deprecation logging.
• SonarScanner CLI — Scanner invocation, verbose/debug options, runtime requirements and TLS troubleshooting entry points.
• Scanner environment requirements — Current scanner runtime and JRE auto-provisioning requirements.
• TLS certificates on client side — Supported truststore/keystore paths instead of disabling TLS verification.
• Web API — Bearer authentication guidance and Web API V2 migration warning.
• SonarQube downloads — Current Community Build, Server release train and active LTA identities.
• SonarScanner CLI releases — Current scanner release identity and release notes.
• Docker Official Image — Current Community Build image tags for a pinned disposable local fixture.