High Availability, Resiliency, Shared Storage, Blob Store Groups, Node Coordination, and Failure Recovery: Diagnostics, Failure Modes, Security, and Performance

1. Diagnostic sequence for an HA incident

1. Preserve client errors, timestamps, load-balancer target health, node identities/versions, and recent changes.
2. Confirm every node’s Nexus version, Java runtime, edition/license, and nexus.properties.
3. Verify the client URL/TLS/authentication path and which node actually served the request.
4. Check cluster/node status and repository authorization.
5. Check the PostgreSQL writer endpoint, connectivity, pool saturation, failover state, and latency.
6. Check shared blob connectivity, latency, quota/capacity, credentials, and provider errors.
7. Check node-local logs, tasks, JVM heap/direct memory, file handles, and temporary disk.
8. Compare an artifact digest/metadata request across repeated load-balanced calls.
9. Apply the smallest supported correction, then verify through the same client path.
10. If shared state is corrupted, stop treating the incident as “HA failover” and enter the tested recovery runbook.

2. Failure: “two independent Nexus servers are HA”

Symptom: clients intermittently see different repositories/users/components depending on the node.

Cause: each Nexus server has its own database and/or blob store. A load balancer in front does not merge them.

Safe correction: stop write traffic to the invalid topology, preserve both states for forensics, and migrate to one supported clustered state using official migration/recovery procedures. Do not copy database directories or blob files between live servers.

3. Failure: shared filesystem is reachable but too slow

Symptom: package downloads time out, upload latency spikes, cleanup/repair tasks stretch for hours, and JVMs appear underutilized.

Cause: “shared” met the namespace requirement but not the IO/latency requirement.

Evidence: storage latency/IOPS, Nexus request latency, task duration, DB latency, client-cache state. Measure them separately before changing heap or adding nodes.

4. Failure: every node shares one failure domain

Symptom: one rack/host cluster/AZ/network outage removes every Nexus node simultaneously.

Cause: node count increased but failure-domain diversity did not.

Correction: distribute nodes across supported independent hosts/AZs while keeping the cluster in one region and retaining low-latency access to shared DB/blob services.

5. Failure: database requests load-balanced to read replicas

Symptom: write failures, stale reads, transaction anomalies, or unsupported topology errors.

Cause: an operator inserted pgpool/generic SQL load balancing because the application tier is active-active.

Correction: restore the supported PostgreSQL writer/failover endpoint model. Do not “solve” the issue by disabling constraints or forcing writes to replicas.

6. Failure: mixed Nexus versions linger

Normal HA requires the same Nexus version/configuration across nodes. Mixed mode is supported only during an explicitly enabled rolling upgrade. Current guidance says to keep that interval as short as possible; new features remain unavailable until the database schema is finalized, and mixed mode outside an active upgrade is unsupported.

7. Failure: failover hides corrupted content

Symptom: every surviving node can serve the same wrong/missing artifact, so uptime dashboards stay green while clients fail checksum or business validation.

Cause: the corruption lives in shared authoritative state. HA replicated/shared the problem correctly.

Correction: preserve evidence, stop harmful writes if needed, identify the consistency point, and restore through Chapter 25’s tested database/blob recovery process.

8. Failure: health endpoint says 200 but storage is failing

Current Status API documentation explicitly says these endpoints report Nexus state and are not hardware checks; they do not replace dedicated checks for external database reachability or disk/storage access. A load balancer should use an appropriate application probe, while monitoring also observes the DB and blob services directly.

9. Performance: add nodes only after locating the bottleneck

10. Security-sensitive HA operations

• Do not expose node management endpoints publicly.
• Keep database/blob credentials in supported secret mechanisms; never duplicate plaintext secrets in lesson files.
• Review support ZIPs per node before sharing because logs/configuration can contain sensitive metadata.
• Do not bypass TLS verification to “fix” cross-node or storage connectivity.
• Do not edit shared blob files or PostgreSQL rows directly during failover.

11. Intentionally broken topology classifier

bad = {
  "nexus_db": {"nx-a":"db-a", "nx-b":"db-b"},
  "blob": {"nx-a":"/local/a", "nx-b":"/local/b"},
  "same_version": False,
  "rolling_upgrade_active": False,
  "db_mode": "round-robin-writer-and-replica",
}

problems=[]
if len(set(bad["nexus_db"].values())) != 1:
    problems.append("independent databases: not one HA service")
if len(set(bad["blob"].values())) != 1:
    problems.append("private blob stores: content can diverge")
if not bad["same_version"] and not bad["rolling_upgrade_active"]:
    problems.append("unsupported mixed Nexus versions")
if bad["db_mode"] != "stable-writer-endpoint":
    problems.append("unsupported database routing")

for p in problems: print("FAIL:", p)

Expected: four failures. The lesson is not to patch each symptom independently; the topology violates the shared-state contract and must be redesigned before production traffic continues.

Knowledge check

Summary and next step

HA troubleshooting follows the shared dependency chain: front door → live Nexus nodes → coherent PostgreSQL writer → shared blob state → node-local resources. Failover is useful only while the shared state itself remains correct.

Lesson 5 turns the chapter into a production-style architecture and failure-validation checkpoint.

Official references and version notes

• Sonatype: High Availability Deployment — current Pro-only clustered application model.
• Sonatype: Requirements for High Availability — same-version nodes, single-region scope, load balancer, shared blob storage, and external PostgreSQL requirements.
• Sonatype: Manual High Availability Deployment — clustered mode, shared storage, and per-node operational state.
• Sonatype: Validating Your HA Deployment — node status, system information, and per-node support evidence.
• Sonatype: Status API — read/writable status endpoints and their monitoring limitations.
• Sonatype: Nodes API — Pro cluster-node inventory and node identities.
• Sonatype: System Requirements — Java 21, PostgreSQL guidance, supported providers, and the explicit pgpool/load-balancing restriction.
• Sonatype: Blob Stores — shared/object storage behavior, health indicators, and group blob-store semantics.
• Sonatype: Storage Planning — group blob stores, storage layout tradeoffs, and performance implications.
• Sonatype: Nexus Repository Professional Features — HA, cloud blob stores, and group blob-store licensing boundaries.
• Sonatype: Rolling Upgrades in High Availability — mixed-version limits, sticky-session requirement during rolling upgrades, schema-finalization behavior, and restore-based rollback after finalization.
• Sonatype: Prepare a Backup — recovery-set consistency across database/configuration and blob content.
• Sonatype: Cross-Region Disaster Recovery — a separate DR pattern rather than an extension of a single-region HA cluster.