Capstone: Operate a Governed Production SonarQube Quality Platform: Diagnostics, Failure Modes, and Production Practices

1. Production anti-patterns become incident multipliers

Capstone failures are often caused by plausible shortcuts: install another plugin, grant more permission, rotate credentials, restart everything, lower a gate, copy an index, change the project key, or downgrade. Each shortcut can erase evidence or move the symptom without repairing the owning layer.

2. Evidence-first incident sequence

This sequence is deliberately the same one taught in Chapter 32. Production maturity means the organization follows it even under pressure.

1. Preserve scanner, server, CI/provider and API evidence before cleanup or retry.
2. Confirm versions: exact Community/Server edition, Java, database, scanner, plugins and integrations.
3. Confirm source and effective inputs: revision, workspace state, scope/report paths and parameter precedence.
4. Inspect scanner/index/report: indexed files, analyzer/sensor/import warnings, upload and report-task.txt.
5. Inspect Compute Engine: task ID/status, timestamps and ce.log correlation.
6. Inspect policy/result: profile, gate, New Code, security/review state and durable measures/issues.
7. Inspect trust/external state: token permission, TLS/network, CI/provider/IDE integration.
8. Inspect persistence/resources: database/search/JVM/host/container only if evidence points there.
9. Apply one least-destructive correction at the owning layer.
10. Rerun the smallest equivalent scenario with the same project key/revision/workload and verify durable recovery.

3. Runnable broken example: authentication without privilege escalation

This fault changes exactly one trust input: the bearer token. The expected evidence is an authentication failure response while server health and project identity remain unchanged. Because auth is now proven as the owning layer, rotating unrelated tokens, changing TLS, restarting services or granting administrator privileges would be unjustified.

# Preserve a baseline successful read first.
curl -fsS -D evidence/api/auth-good.headers \
  -H "Authorization: Bearer $SONAR_TOKEN" \
  "http://localhost:9000/api/qualitygates/project_status?projectKey=sq34%3Acapstone" \
  -o evidence/api/auth-good.json

# Intentionally use a fake invalid token. Never overwrite the real lab token value.
BAD_TOKEN='sq34-intentionally-invalid'
curl -sS -D evidence/api/auth-bad.headers \
  -H "Authorization: Bearer $BAD_TOKEN" \
  "http://localhost:9000/api/qualitygates/project_status?projectKey=sq34%3Acapstone" \
  -o evidence/api/auth-bad.body

# Repair: restore the known-good scoped credential; do not promote to admin.
curl -fsS -H "Authorization: Bearer $SONAR_TOKEN" \
  "http://localhost:9000/api/qualitygates/project_status?projectKey=sq34%3Acapstone" \
  > evidence/api/auth-recovered.json

4. Intentionally broken Compute Engine incident: interpret before repair

Creating a real incompatible plugin merely to crash a current server is an unnecessary safety risk. The mandatory free path therefore uses a clearly labeled synthetic CE transcript while preserving the same diagnostic reasoning. Optional plugin lifecycle testing belongs only in an isolated extension lab.

# Synthetic CE evidence for interpretation only — do NOT install an incompatible plugin to create this.
2026.09.08 12:15:31 INFO  ce[][o.s.c.t.CeWorkerImpl] Execute task | project=sq34:capstone | type=REPORT | id=AY-sq34-demo
2026.09.08 12:15:32 ERROR ce[][o.s.c.t.CeWorkerImpl] Failed to execute task AY-sq34-demo
java.lang.IllegalStateException: SQ34 simulated extension incompatible with current server API
... first causal stack frames preserved ...
Task status: FAILED
Scanner evidence: report uploaded successfully; ceTaskId=AY-sq34-demo

5. Turn backup creation into recovery proof

The goal is not to overwrite the working lab. Restore to an isolated destination with a fresh search/data volume. SonarQube backup guidance centers the supported database and uses search reindexing during recovery. This exercise therefore refuses to “restore” a copied embedded-search directory as the authoritative state.

# compose.restore.yaml — isolated restore target, different port and volumes.
services:
  restore-db:
    image: postgres:17
    environment:
      POSTGRES_USER: sonar
      POSTGRES_PASSWORD: sq34-restore-db-only
      POSTGRES_DB: sonarqube
    volumes: [sq34_restore_db:/var/lib/postgresql/data]
  restore-sonar:
    image: sonarqube:26.9.0.129388-community
    depends_on: [restore-db]
    environment:
      SONAR_JDBC_URL: jdbc:postgresql://restore-db:5432/sonarqube
      SONAR_JDBC_USERNAME: sonar
      SONAR_JDBC_PASSWORD: sq34-restore-db-only
    ports: ["9001:9000"]
    volumes:
      - sq34_restore_data:/opt/sonarqube/data
      - sq34_restore_logs:/opt/sonarqube/logs
volumes:
  sq34_restore_db:
  sq34_restore_data:
  sq34_restore_logs:

# Verify checksum before restore.
sha256sum -c evidence/backup/sonarqube.dump.sha256

# Start ONLY the isolated database first.
docker compose -f compose.restore.yaml -p sq34restore up -d restore-db
# Wait for pg_isready, then restore the database backup.
docker compose -f compose.restore.yaml -p sq34restore exec -T restore-db \
  pg_restore -U sonar -d sonarqube --clean --if-exists \
  < evidence/backup/sonarqube.dump

# Start SonarQube with a fresh search/data volume so indexes are built for the restored DB.
docker compose -f compose.restore.yaml -p sq34restore up -d restore-sonar
curl -fsS http://localhost:9001/api/system/status \
  | tee evidence/backup/restore-system-status.json

# Verify project/configuration in UI/API, then run a fresh equivalent analysis against port 9001.
# Record the restored project's prior analysis evidence AND the new validation task.

# Cleanup only the restore target after evidence capture.
docker compose -f compose.restore.yaml -p sq34restore down -v

6. Unsupported upgrade/downgrade shortcuts

An update changes multiple compatibility contracts at once. Production recovery must know whether the database schema, plugin API, Java runtime, scanner or automation endpoint changed. “Install the old binaries again” is not a safe generic rollback.

7. Vanity metrics and commercial dependencies as incident causes

Operational incidents can be organizational. A misleading KPI creates pressure to mutate analysis scope or policy; a paid feature can become an undocumented single point of governance. Treat incentives and ownership as causal layers, not “soft” afterthoughts.

8. Post-incident record: prove durable recovery

Closing an incident means more than a green rerun. Preserve equivalence, causal evidence, correction, verification and residual risks.

incident_id: SQ34-INC-001
first_failure_utc: 2026-09-08T12:15:32Z
project_key: sq34:capstone
revision: <exact SHA from evidence/00-revision.txt>
classification: authentication | scanner-config | compute/plugin | persistence | policy | provider | infrastructure
first_failure_artifacts:
  - evidence/scanner/...
  - evidence/server/...
  - evidence/api/...
hypothesis: <one falsifiable statement>
change_applied: <smallest reversible correction>
rerun_equivalence:
  same_project_key: true
  same_revision: true
  same_build_reports: true
verification:
  scanner: <status>
  compute_engine: <status>
  gate: <status>
  ci_provider: <status or simulated>
residual_risks:
  - <risk, owner, due date>
prevention:
  - <compatibility test / permission check / runbook / training change>

9. Knowledge check

10. Summary and bridge

You can now operate the incident side of the platform without destructive shortcuts. Lesson 5 is the final checkpoint: assemble architecture, analysis, policy, security/RBAC, automation, observability, recovery, upgrade, capacity and governance into one production-readiness evidence packet with explicit residual risk.

Official references and version notes

Further reading

Verify version-sensitive behavior against primary documentation before using these patterns outside the disposable lab.

• SonarQube downloads — current Community Build, commercial release and LTA identities
• SonarQube Server documentation — server, administration, security and operations
• SonarQube Community Build documentation — free/local product behavior
• Web API — authentication and the ongoing Web API V2 migration
• Backup and restore — database backup/restore and reindex guidance
• Official SonarQube Docker image — current image tags and deployment notes
• Performance troubleshooting — host/search/database/CE performance guidance