Performance Sizing, Reference Architectures, and Large-Instance Tuning: Diagnostics, Failure Modes, and Production Practices

1. Diagnostic sequence: preserve first failure, then narrow the owning layer

Performance incidents tempt teams to restart services and erase the evidence that identifies the constrained layer. Use the same sequence whether the symptom is a slow scanner, growing CE queue, failed background task, sluggish UI or unstable search. Each step either rules in or rules out a state owner.

Project policy is included even in a performance incident because an “analysis is slow” report can actually be a quality-gate wait, provider-decoration delay, permission/API failure or scanner-scope drift. Do not assume the symptom names the layer.

2. Intentionally broken example: heap everything, cache nothing

Assume a disposable single-node lab host has 8 GiB available to the SonarQube container. An operator sees search latency and decides to set the search JVM maximum close to the entire container limit while Web and CE remain on the same container. After restart, container memory stays near the limit, the host reclaims aggressively, search queries become erratic and CE tasks slow during indexing. The change looked like “more memory for search,” but it removed memory from the OS cache and competing processes.

# INTENTIONALLY BAD — educational example only. Do not apply to production.
# Container memory: 8 GiB
SONAR_SEARCH_JAVAOPTS=-Xms7g -Xmx7g
# Web + CE + native memory + Lucene page cache now have almost no safe budget.

Interpret the evidence

Repair: restore the previous supported heap values first, restart through the normal deployment procedure, and rerun the smallest equivalent workload. Then build a measured process/OS memory budget. If search still needs capacity, change one supported resource at a time while retaining page-cache headroom.

3. Failure mode: increasing CE workers into a constrained database/search path

A classic secondary failure begins with a real queue. An Enterprise operator increases workers from two to six. Pending count briefly falls, but each task becomes slower, database CPU and active connections rise, disk latency increases and the queue soon returns. The worker change did increase concurrency—exactly as configured—but concurrency amplified contention in the actual bottleneck.

Preserve worker count, PendingCount/PendingTime, task durations, DB pool/activity and disk/network evidence from both states. The least-destructive correction is usually to return to the previous worker count while diagnosing the downstream limit. Do not “compensate” with progressively more workers or unreviewed database changes.

4. Failure mode: performance-sensitive search data on remote network storage

Search can look CPU-idle while queries and indexing stall on storage latency. Current SonarSource host guidance warns against remote-mounted NFS, SMB/CIFS and NAS for search-sensitive storage because latency is higher and more variable. A deployment can pass a trivial smoke test and still fail under concurrent indexing/search.

Evidence should include actual mount/storage class, latency/IOPS observations, search/server logs, disk free space and UI/API/CE timing correlation. The correction is a supported migration to suitable low-latency storage—not copying an embedded search directory while the service runs, editing search metadata or deleting indices. Preserve the database as durable application state and follow supported recovery/migration paths.

5. Failure mode: sizing only by LOC

A 20M-LOC portfolio can be low-frequency and mostly small repositories; a 5M-LOC estate can generate continuous PR traffic from a few huge monorepos. LOC matters for stored/analyzed scale, but it is one dimension. When a reference architecture is selected only from LOC, high-frequency analysis, large-repository tails, API automation and plugin/analyzer cost disappear from the model.

6. Failure mode: ignoring bursty CI

Suppose 120 analyses arrive over an eight-hour day: average 15/hour. That sounds small. If 80 of them start within ten minutes after a scheduled merge train, the relevant arrival rate during the burst is 480/hour. A capacity worksheet that only stores the daily average will systematically underpredict queueing.

Preserve CI scheduling/run-start evidence beside CE pending/time evidence. If the service is healthy outside the burst, test a reversible scheduling/concurrency change before a server expansion. If the burst is legitimate merge-critical traffic, use its peak shape in the scaling model rather than smoothing it away.

7. Failure mode: trusting one warmed benchmark

Caches are real operational mechanisms, but a single warmed run can hide cold-start and change-related cost. Scanner analyzer downloads, filesystem cache, JVM JIT and database buffers can make later repetitions faster. Conversely, one cold run can exaggerate normal steady-state time. A trustworthy test labels cache state and keeps multiple repetitions.

run,revision,cache_state,scanner_s,queue_wait_s,ce_s
warmup,abc123,cold_or_unknown,48.2,0.0,31.4
B1,abc123,warm,31.0,0.0,27.9
B2,abc123,warm,30.7,0.0,28.1
B3,abc123,warm,31.4,0.0,27.6

Do not delete caches before every run unless “cold cache” is the scenario you are explicitly testing. Do not delete them after a failure before preserving the failure. State which scenario the benchmark represents.

8. Failure mode: tuning production without a rollback envelope

Changing heap, workers, container limits, storage, database tier or cluster topology can trade one bottleneck for another. A production tuning plan therefore needs a before snapshot, exact configuration diff, one primary hypothesis, validation window, guardrails and a rollback condition. “We can restart again if it is bad” is not a rollback plan.

Before:
  CE task p95: 70 s
  pending-time p95: 210 s
  DB p95 latency: 8 ms
  search disk p95: 4 ms
Change:
  one supported variable only
Success:
  pending-time p95 <= 60 s AND CE task p95 <= 77 s
Guardrails:
  DB p95 <= 12 ms; no health degradation; UI/API p95 <= baseline + 10%
Rollback:
  restore exact previous config/deployment value and verify smallest equivalent workload

9. Causal separation checklist

10. Evidence preservation bundle for a local incident

$Incident = "evidence/incident-$(Get-Date -Format yyyyMMdd-HHmmss)"
New-Item -ItemType Directory -Force $Incident | Out-Null

# Preserve identities before changing anything.
curl.exe -fsS http://localhost:9000/api/server/version | Set-Content "$Incident/server-version.txt"
sonar-scanner -v 2>&1 | Set-Content "$Incident/scanner-version.txt"
git rev-parse HEAD | Set-Content "$Incident/revision.txt"
docker inspect sq31-sonarqube | Set-Content "$Incident/sonarqube-inspect.json"
docker stats --no-stream sq31-sonarqube sq31-postgres | Set-Content "$Incident/docker-stats.txt"
docker logs --since 30m sq31-sonarqube 2>&1 | Set-Content "$Incident/server.log"

# If an analysis ran, preserve its task identity and raw server response.
Copy-Item .scannerwork/report-task.txt "$Incident/report-task.txt" -ErrorAction SilentlyContinue
# Poll api/ce/task for that exact id using a least-privilege bearer token.

# Only after this packet exists should you test the smallest correction.

Knowledge check

Official references and version notes

• SonarQube downloads — Current Community Build, commercial Server release train, editions and active LTA.
• Server host requirements — Current disk, memory, CPU, local-storage and production database-host guidance.
• Community Build host requirements — Equivalent Community Build host/search guidance for the mandatory free path.
• Monitoring the instance — Web/CE/search JVMs, read-only JMX MBeans and ComputeEngineTasks/Database signals.
• Reference architecture up to 10 M LOC — Planning anchor and its stated normal-usage assumptions.
• Improving performance — Enterprise+ CE-worker guidance and the requirement to measure external bottlenecks.
• Performance issues — Current troubleshooting starting points for storage, scope, workers and database-related performance.
• Server release notes — 2026.1 runtime/database changes, including JDK 21/25 and PostgreSQL 14–18.
• SonarScanner CLI releases — Current scanner release identity; 8.1.0.6389 is the latest listed release at this chapter recheck.
• Web API — Bearer authentication guidance and the ongoing Web API V2 migration.