Data Center Edition, Clustering, High Availability, and Scale: Guided Hands-On Workflow

1. Lab contract: simulate DCE faithfully; do not emulate its license

2. Create the topology manifest

The manifest is intentionally declarative: it records what a real production review should know before anyone touches a node.

{
  "edition": "Data Center Edition (simulation)",
  "server_version": "2026.4.1",
  "license_present": false,
  "load_balancer": {"name": "lb-1", "backends": ["app-1", "app-2"]},
  "database": {"engine": "PostgreSQL", "version": "17", "healthy": true},
  "nodes": [
    {"name":"app-1","role":"application","healthy":true,"version":"2026.4.1","plugins_hash":"demo-same"},
    {"name":"app-2","role":"application","healthy":true,"version":"2026.4.1","plugins_hash":"demo-same"},
    {"name":"search-1","role":"search","healthy":true,"version":"2026.4.1"},
    {"name":"search-2","role":"search","healthy":true,"version":"2026.4.1"},
    {"name":"search-3","role":"search","healthy":true,"version":"2026.4.1"}
  ],
  "ports": {"lb_to_app":9000,"app_to_search":9001,"search_to_search":9002,"app_to_app":9003}
}

3. Build the local validator/failure simulator

# dce_sim.py -- educational topology simulator, not SonarQube software.
import json, sys
from pathlib import Path

p = Path(sys.argv[1] if len(sys.argv) > 1 else "dce-topology.json")
d = json.loads(p.read_text())
nodes = d["nodes"]
apps = [n for n in nodes if n["role"] == "application"]
search = [n for n in nodes if n["role"] == "search"]
healthy_apps = [n for n in apps if n["healthy"]]
healthy_search = [n for n in search if n["healthy"]]

errors = []
if len(apps) < 2: errors.append("minimum topology requires >=2 application nodes")
if len(search) < 3: errors.append("minimum topology requires >=3 search nodes")
if len({n["version"] for n in nodes}) != 1: errors.append("mixed SonarQube versions")
if len({n.get("plugins_hash") for n in apps}) != 1: errors.append("application plugin state differs")
if not d["database"]["healthy"]: errors.append("shared database unavailable")
if sorted(d["load_balancer"]["backends"]) != sorted(n["name"] for n in apps):
    errors.append("load balancer backend inventory differs from application inventory")

service_available = bool(healthy_apps) and len(healthy_search) >= 2 and d["database"]["healthy"]
print(json.dumps({
    "healthy_application_nodes": len(healthy_apps),
    "healthy_search_nodes": len(healthy_search),
    "validation_errors": errors,
    "simulated_service_available": service_available,
}, indent=2))
raise SystemExit(2 if errors else 0)

The simulator intentionally models the documented default topology’s one-search-node failure tolerance as “at least two of three search nodes remain.” It is not a substitute for Elasticsearch quorum logic, Sonar Support guidance, or real cluster health APIs.

4. Baseline: validate the untouched five-node design

mkdir -p evidence/baseline evidence/app-failure evidence/search-failure evidence/recovery
python dce_sim.py dce-topology.json | tee evidence/baseline/topology-result.json
sha256sum dce-topology.json dce_sim.py | tee evidence/baseline/files.sha256
python --version | tee evidence/baseline/python-version.txt

Expected result: two healthy application nodes, three healthy search nodes, no validation errors, and simulated service availability true.

5. Predict before failure injection

• Prediction A: marking app-2 unhealthy leaves one healthy application node; the service remains available but loses application-tier redundancy and capacity.
• Prediction B: restoring app-2 and marking search-3 unhealthy leaves two healthy search nodes; the default topology remains serviceable but loses search redundancy/capacity.
• Prediction C: changing one application node’s version or plugin hash should fail the configuration validator even if both nodes are “healthy.”
• Prediction D: neither scenario changes source revision, Quality Gate policy, or scanner project configuration.

6. Model one application-node failure

python - <<'PY'
import json
p='dce-topology.json'; d=json.load(open(p))
for n in d['nodes']:
    if n['name']=='app-2': n['healthy']=False
json.dump(d, open('dce-app-failure.json','w'), indent=2)
PY
python dce_sim.py dce-app-failure.json | tee evidence/app-failure/result.json

Interpretation: the load balancer must stop routing to the unhealthy app node. Existing traffic can continue through app-1, but the cluster no longer has application-node fault tolerance. This is precisely why health checks matter; sticky sessions are not the recovery mechanism.

7. Model one search-node failure

python - <<'PY'
import json
p='dce-topology.json'; d=json.load(open(p))
for n in d['nodes']:
    if n['name']=='search-3': n['healthy']=False
json.dump(d, open('dce-search-failure.json','w'), indent=2)
PY
python dce_sim.py dce-search-failure.json | tee evidence/search-failure/result.json

Expected result: two search nodes remain and the default topology’s single-search-node failure assumption remains satisfied. The correct production evidence would also include actual search-cluster health, disk/heap/GC, and index state—not just node count.

8. Keep real analysis evidence separate from the DCE simulation

Run a tiny Community Build analysis so you retain the same task chain used throughout the course:

export SONAR_HOST_URL="http://localhost:9000"
export SONAR_PROJECT_KEY="sq-ch30-dce-simulation"
# SONAR_TOKEN is a disposable project-analysis token in the environment.
sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" 2>&1 | tee evidence/baseline/scanner.log
cp .scannerwork/report-task.txt evidence/baseline/report-task.txt
CE_TASK_ID="$(sed -n 's/^ceTaskId=//p' .scannerwork/report-task.txt)"
echo "$CE_TASK_ID" | tee evidence/baseline/ceTaskId.txt

This does not make Community Build a cluster. It simply gives you a real scanner/report/ceTaskId chain to map onto the conceptual application/CE role.

9. Optional licensed observation path

If you have an authorized DCE sandbox, remain read-only for the lesson: inventory application/search nodes, exact server/plugin versions, load-balancer backends, database endpoint/version, search-cluster health, CE queue/worker evidence, and configured cluster ports. Do not stop a node merely to satisfy the course. A planned maintenance sandbox may reproduce the one-app-node/one-search-node failures, but production fault injection is outside the mandatory lab.

10. Challenge: choose the owning layer

The CE queue doubles after traffic growth, but search latency, search heap, disk, and database latency remain normal. Which layer should be capacity-tested first?

Answer: application/Compute Engine capacity. Adding search nodes attacks the wrong bottleneck.

Knowledge check

Official references and version notes

• DCE topology — current role model and minimum/default 2 application + 3 search topology.
• DCE installation requirements — node identity, search-node placement, hardware guidance, Docker/network/load-balancer requirements.
• DCE-specific system properties — current cluster roles, host lists and default application-cluster port.
• DCE network rules — default 9000/9001/9002/9003 trust paths.
• DCE scaling — application-node scaling, currently up to ten application nodes.
• Installing the database — current supported database engines/versions including PostgreSQL 14–18.
• Official SonarQube DCE Helm chart — current DCE chart/image structure and application/search node configuration.
• Performing an update — coordinated DCE Helm/database migration behavior and post-update reanalysis.