Checkpoint Lab — Data Center Edition, Clustering, High Availability, and Scale

1. Checkpoint scenario

You are reviewing a proposed SonarQube deployment for a global engineering organization. The proposal claims “high availability” because it contains multiple containers. Your job is to produce an edition-aware architecture and failure walk-through that proves whether the proposal actually matches current Data Center Edition semantics.

Your output must include the minimum/default DCE topology, private network paths, load-balancer contract, database/search assumptions, node-version/plugin consistency, one application-node failure, one search-node failure, a capacity note, a DR note, and a Community Build-safe simulation of every mandatory concept.

2. Exact assumptions

3. Resource and credential preflight

mkdir -p evidence/{baseline,app-loss,search-loss,mixed-version,recovery,analysis}
python --version | tee evidence/baseline/python-version.txt
sonar-scanner --version | tee evidence/analysis/scanner-version.txt
curl -fsS http://localhost:9000/api/system/status | tee evidence/analysis/community-status.json

git rev-parse HEAD | tee evidence/analysis/revision.txt
# SONAR_TOKEN must be a disposable project-analysis token in the environment.
test -n "$SONAR_TOKEN"

Do not proceed if the working directory contains proprietary source or if the token points to a production project. The checkpoint is synthetic and local.

4. Required trust-boundary diagram

flowchart TB
  Internet[Users / CI / scanners] --> TLS[HTTPS reverse proxy / load balancer]
  TLS -->|9000 or configured web port| A1[App 1: Web + CE]
  TLS -->|9000 or configured web port| A2[App 2: Web + CE]
  A1 <-->|private 9003| A2
  A1 -->|private 9001| S1[Search 1]
  A1 -->|private 9001| S2[Search 2]
  A1 -->|private 9001| S3[Search 3]
  A2 -->|private 9001| S1
  A2 -->|private 9001| S2
  A2 -->|private 9001| S3
  S1 <-->|private 9002| S2
  S2 <-->|private 9002| S3
  S1 <-->|private 9002| S3
  A1 --> DB[(External supported database)]
  A2 --> DB

Annotate the actual production design with subnet/AZ and firewall ownership. Search nodes may be spread across availability zones, but current guidance keeps them in the same region. Each node should have an independent machine/failure domain where practical.

5. Make predictions before changing simulated state

• P1: baseline validator returns 2 healthy app / 3 healthy search / available.
• P2: losing one app returns 1 healthy app / 3 healthy search / available, with lost app redundancy.
• P3: losing one search returns 2 healthy app / 2 healthy search / available, with reduced search redundancy/capacity.
• P4: a mixed-version/plugin app node is rejected as a deployment-consistency defect even if both app nodes are marked healthy.
• P5: a real Community analysis produces a ceTaskId and CE result without changing any DCE simulation state.

6. Execute and independently verify the failure walk-through

Use the dce-topology.json and dce_sim.py from Lesson 2. Preserve each manifest rather than overwriting it.

python dce_sim.py dce-topology.json | tee evidence/baseline/result.json
python dce_sim.py dce-app-failure.json | tee evidence/app-loss/result.json
python dce_sim.py dce-search-failure.json | tee evidence/search-loss/result.json
python dce_sim.py dce-broken-mixed.json \
  > evidence/mixed-version/stdout.txt 2> evidence/mixed-version/stderr.txt || true

sha256sum dce-*.json dce_sim.py | tee evidence/baseline/all-artifacts.sha256

Verification is independent when you compare the output against the manifest itself: count healthy role nodes, compare versions/plugin hashes, verify LB backend inventory, and confirm the database flag. Do not mark a prediction “verified” solely because the simulator prints what you expected.

7. Real scanner/task evidence

export SONAR_HOST_URL="http://localhost:9000"
export SONAR_PROJECT_KEY="sq-ch30-dce-checkpoint"
sonar-scanner -Dsonar.projectKey="$SONAR_PROJECT_KEY" \
  2>&1 | tee evidence/analysis/scanner.log
cp .scannerwork/report-task.txt evidence/analysis/report-task.txt
CE_TASK_ID="$(sed -n 's/^ceTaskId=//p' .scannerwork/report-task.txt)"
echo "$CE_TASK_ID" | tee evidence/analysis/ceTaskId.txt
curl -fsS -H "Authorization: Bearer $SONAR_TOKEN" \
  "$SONAR_HOST_URL/api/ce/task?id=$CE_TASK_ID" \
  | tee evidence/analysis/ce-task.json

Record scanner exit, report upload, CE terminal state, analysis ID, and Quality Gate separately. In licensed DCE, the same external task identity is processed by the distributed application tier; the caller should not need to know which app node executed it.

8. Capacity and failure-budget worksheet

application_nodes: 2 minimum, current planned count: __
search_nodes: 3 minimum, current planned count: __
max documented application nodes: 10
app sizing evidence: CPU / heap / CE queue / request latency
search sizing evidence: heap / GC / disk latency / index size / search health
database evidence: engine/version / latency / pool / storage / HA / backup
load_balancer evidence: backend health / TLS / timeouts / max connections
failure budget: tolerate 1 app + 1 search node? yes/no + evidence
AZ/region: search nodes same region; placement: __
DR: backup ID / restore rehearsal ID / RPO / RTO
upgrade: exact supported path / maintenance window / DB backup

Do not pre-fill a “recommended node count” beyond the documented minimum. Chapter 31 will use observed workload and reference architectures for large-instance sizing.

9. Verification checklist

• ☐ DCE is explicitly licensed/optional; Community Build fallback does not claim clustering.
• ☐ Architecture has exactly defined application/search roles and at least 2+3 nodes.
• ☐ Every node’s server version and application plugin artifact set is controlled.
• ☐ Shared database engine/version is supported and protected by backup/HA policy.
• ☐ Load balancer routes only to healthy app nodes; no sticky-session assumption is required.
• ☐ 9001/9002/9003 backplane paths are private and source/destination scoped.
• ☐ Search storage/heap/disk and database capacity are measured independently of app capacity.
• ☐ One app-loss and one search-loss prediction were independently checked.
• ☐ Mixed-version/plugin failure evidence is preserved before repair.
• ☐ Real Community scan retains revision, scanner log, report-task.txt, ceTaskId, CE status, and gate state.
• ☐ HA and DR are documented as different controls.
• ☐ Upgrade/plugin changes include coordinated cluster downtime/maintenance assumptions where current docs require it.

10. Required evidence packet

11. Cleanup and rollback

• Revoke the disposable Community project-analysis token and delete only the synthetic Community project if you created it.
• Keep the simulation manifests/evidence packet; remove only temporary generated files after checksums are recorded.
• If an optional licensed sandbox was used, restore any planned test node to service and verify cluster/search health before ending maintenance.
• Do not delete logs, search data, database state, or cluster configuration merely to “reset” the lab.
• Never carry fake simulation secrets into a real DCE configuration.

12. What Chapter 30 adds to the production operating model

The course now has a distributed-service operating model: a SonarQube analysis is still tied to an exact revision and ceTaskId, but the Web/CE tier can be horizontally resilient, search becomes an explicit cluster tier, load-balancer/backplane networks become first-class trust boundaries, and node/version/plugin consistency becomes a release invariant. HA decisions are tied to measured failure budgets rather than container counts.

Chapter 31, Performance Sizing, Reference Architectures, and Large-Instance Tuning, continues from this architecture by turning workload evidence—LOC/issues, CE throughput, user/API load, heap/GC, search indexes, disk, and database behavior—into defensible sizing and tuning decisions.

Knowledge check

Official references and version notes

• DCE topology — current role model and minimum/default 2 application + 3 search topology.
• DCE installation requirements — node identity, search-node placement, hardware guidance, Docker/network/load-balancer requirements.
• DCE-specific system properties — current cluster roles, host lists and default application-cluster port.
• DCE network rules — default 9000/9001/9002/9003 trust paths.
• DCE scaling — application-node scaling, currently up to ten application nodes.
• Installing the database — current supported database engines/versions including PostgreSQL 14–18.
• Official SonarQube DCE Helm chart — current DCE chart/image structure and application/search node configuration.
• Performing an update — coordinated DCE Helm/database migration behavior and post-update reanalysis.