Data Center Edition, Clustering, High Availability, and Scale: Diagnostics, Failure Modes, and Production Practices

1. Evidence-first DCE incident sequence

1. Preserve scanner/CI evidence, report-task.txt, ceTaskId, load-balancer status, node logs, cluster/search health, and timestamps.
2. Confirm DCE license, exact server release, Java/database versions, plugins, Helm/chart/image versions, and node role/count.
3. Confirm the exact source revision/effective analysis parameters before blaming infrastructure for a project-specific error.
4. Inspect scanner upload and specific CE task state.
5. Inspect CE queue/workers and application-node Web/CE logs.
6. Inspect search-cluster health, disk, heap/GC, and es.log.
7. Inspect database connectivity, latency, pool pressure, storage, and locks.
8. Inspect load-balancer health/backends and private 9000/9001/9002/9003 network reachability.
9. Apply the least-destructive correction to the owning layer.
10. Rerun the smallest equivalent scenario and compare evidence at the same source revision when possible.

2. Failure: “we have five servers, therefore we have HA”

Five independent SonarQube instances do not share cluster membership, application coordination, or one search cluster. Putting a load balancer in front of them can route users to inconsistent state and does not create a supported DCE architecture. The correction is architectural: deploy licensed DCE with explicit node roles and shared database/search membership, or operate a single supported server.

3. Intentionally broken example: mixed versions/plugins

Start from the Lesson 2 manifest and make two changes:

python - <<'PY'
import json
p='dce-topology.json'; d=json.load(open(p))
for n in d['nodes']:
    if n['name']=='app-2':
        n['version']='2026.3.1'
        n['plugins_hash']='different-plugin-set'
json.dump(d, open('dce-broken-mixed.json','w'), indent=2)
PY
python dce_sim.py dce-broken-mixed.json \
  > evidence/mixed-version/validator.stdout 2> evidence/mixed-version/validator.stderr || true

Preserve before repair. The validator should report mixed SonarQube versions and divergent application plugin state. In a real DCE incident, preserve exact binaries/image digests, plugin manifests/checksums, node startup logs, and the deployment event that created drift.

Repair: return app-2 to the exact cluster release and approved plugin artifact set. Do not “fix” the symptom by removing the app node from the load balancer forever or by downgrading the rest of the cluster.

4. Failure: exposing or blocking the wrong cluster port

5. Failure: unsuitable search storage

Search nodes are latency-sensitive and current Sonar guidance says SSDs perform significantly better than HDDs. If search heap is healthy but disk latency or capacity pressure is high, adding application nodes increases incoming work without repairing the bottleneck. Preserve es.log, disk latency/utilization, heap/GC, and search health first. Never edit Elasticsearch indexes directly or move live search data with generic file-copy commands as a repair.

6. Failure: shared database is the actual bottleneck

Every application node depends on the same supported database. Adding application nodes can increase database concurrency. If JDBC pool waits, DB CPU/IO/locks, or network latency are already saturated, horizontal app scaling can worsen the incident. Use database/vendor metrics and Sonar Web/CE evidence together. Current server support includes PostgreSQL 14–18, SQL Server 2017/2019/2022, and supported Oracle versions; verify the exact release matrix before changing engines or versions.

7. Failure: treating sticky sessions as the HA fix

Sticky sessions can hide routing symptoms but do not repair a broken application cluster, inconsistent JWT secret, mixed node version, or unhealthy backend. Current DCE explicitly does not require session affinity. Preserve the load-balancer backend/health evidence, then repair the owning cluster or network layer.

8. Failure: adding app nodes when search/database is saturated

Capacity management is causal. If CE queue/CPU is high and downstream tiers are healthy, app scaling may help. If es.log, search heap, or disk is the limiting layer, fix search. If JDBC/DB latency is limiting, fix the database path. Node count is not a generic “performance” knob.

9. Production shortcuts to reject

• Do not expose 9001/9002/9003 to the public Internet.
• Do not direct-edit Elasticsearch state or the SonarQube database.
• Do not repeatedly restart the whole cluster before preserving first-failure evidence.
• Do not run mixed server/plugin versions as a tolerated steady state.
• Do not bypass TLS verification or place cluster/shared secrets in source.
• Do not add nodes before identifying the bottleneck.
• Do not treat node HA as database/region disaster recovery.
• Do not change project keys, gates, or profiles to hide infrastructure failures.

Knowledge check

Official references and version notes

• DCE topology — current role model and minimum/default 2 application + 3 search topology.
• DCE installation requirements — node identity, search-node placement, hardware guidance, Docker/network/load-balancer requirements.
• DCE-specific system properties — current cluster roles, host lists and default application-cluster port.
• DCE network rules — default 9000/9001/9002/9003 trust paths.
• DCE scaling — application-node scaling, currently up to ten application nodes.
• Installing the database — current supported database engines/versions including PostgreSQL 14–18.
• Official SonarQube DCE Helm chart — current DCE chart/image structure and application/search node configuration.
• Performing an update — coordinated DCE Helm/database migration behavior and post-update reanalysis.