Cluster Seeding, Membership Changes, Maintenance, Capacity Headroom, and Rebalancing Operational Risk

AtlasMart needs to patch one cluster host and add capacity before a campaign. The unsafe plan is “stop server03, add server05, and let Neo4j figure it out.” That ignores quorum headroom, allocation movement, store-copy I/O, system-database availability, server state and capacity. Safe maintenance is a topology transition with explicit preconditions and rollback.

Learning outcomes

Lab baseline and scope

AtlasMart remains the same customer/order/product graph used throughout earlier chapters. This chapter changes deployment topology, not business identity. The optional licensed topology uses a separate database name atlasmart on disposable test infrastructure so cluster administration cannot damage the single-server continuity database.

1. Joining a server does not make it a permanent role

A new server first needs discovery configuration so it can learn the cluster. Its initial constraints can restrict which database modes/names it may host, but those are placement constraints—not a global “primary server” identity. After discovery and enablement, database allocations can be moved or created on it according to topology and constraints.

# On a new Enterprise server, configure discovery to existing members.
server.default_listen_address=0.0.0.0
server.default_advertised_address=server05.example.test
dbms.cluster.discovery.resolver_type=LIST
dbms.cluster.endpoints=server01.example.test:6000,server02.example.test:6000,server03.example.test:6000,server04.example.test:6000
server.cluster.system_database_mode=SECONDARY
initial.server.mode_constraint=NONE

# Start server05. It is discovered, then inspect/enable it from system database.

2. Cordon before evacuation

Cordoning marks a server unsuitable for new allocations; it does not automatically force current allocations away. That separation is useful: you first prevent new placement, then preview/deallocate existing database copies while observing quorum and capacity. Current Cypher 25 removes the old uncordon procedure path; ENABLE SERVER is the current replacement when a server is returned to service.

:use system
SHOW SERVERS YIELD serverId, name, address, state, health, hosting, requestedHosting, modeConstraint RETURN *;

# For a newly discovered server not already enabled, use its actual ID/name:
ENABLE SERVER 'server05';

# Preview rebalancing before moving allocations:
DRYRUN REALLOCATE DATABASES;

# For planned evacuation of a server:
CALL dbms.cluster.cordonServer('server04');
DRYRUN DEALLOCATE DATABASES FROM SERVER 'server04';
DEALLOCATE DATABASES FROM SERVER 'server04';

# After it hosts no databases, DROP SERVER is the removal step if decommissioning.
# DROP SERVER 'server04';

3. Reallocation moves load and consumes resources

REALLOCATE DATABASES can rebalance database placement, but movement may trigger store copy and catch-up. During that interval the cluster uses extra disk bandwidth, network, CPU and page cache while still serving application traffic. A maintenance window can therefore fail by overload even when quorum arithmetic is correct.

4. Seeding is a creation/recreation mechanism

When a database must be created or recreated from known data, current Neo4j supports seed URI workflows. The exact provider can be file/cloud/server dependent. Starting in 2026.04, server:// can reference a backup/dump staged in a configured seeds directory. This is not an invitation to use cluster servers as permanent backup storage: seeding artifacts are operational inputs; durable backup retention remains the Chapter 16 problem.

# Current 2026.07 concepts: seed from a validated backup/dump URI when creating/recreating.
# Exact provider/credentials depend on configured seed provider and deployment.
CREATE DATABASE atlasmart_restore
TOPOLOGY 3 PRIMARIES 1 SECONDARY
OPTIONS {seedURI: 'file:///validated-course-backup.dump'};

# Starting in Neo4j 2026.04, ServerSeedProvider also supports server:// URIs
# for a backup/dump placed in a configured seeds directory on a cluster server.
# Treat the seeds directory as transient staging, not long-term backup storage.

5. Maintenance headroom is a fault-tolerance budget

For a three-primary database, taking one primary down intentionally leaves two. You can still write, but another primary loss removes quorum. For five primaries, maintenance of one leaves four and still tolerates another failure. This is not a universal recommendation for five; it is a way to convert an operational requirement (“patch one host while tolerating another failure”) into topology math.

6. Controlled maintenance runbook

7. Wrong operations pattern and repair

Wrong: schedule simultaneous rolling restarts across three primaries because “the cluster has replicas.” Mechanism: overlapping restarts can remove the majority or force repeated elections/catch-up while capacity is already reduced. Repair: one controlled topology transition at a time, explicit current-primary count, health gate between steps, and a stop-the-line rule on unexpected failure.

8. Verification checklist

• New server discovery and enablement are distinct and observable.
• Server mode constraints are treated as placement constraints, not permanent server roles.
• Every deallocation/reallocation is previewed where supported and checked against per-database quorum.
• Store-copy bandwidth/disk/cache impact is included in maintenance acceptance criteria.
• Seed artifacts are validated and not confused with long-term backup storage.
• Maintenance includes a concurrent-failure stop rule.

Production judgment

Summary and next step

You can now move topology deliberately rather than restarting machines blindly. Lesson 5 combines the chapter into an HA drill that injects failures and verifies driver recovery, transaction outcomes, lag, alerts and operator decisions.

Authoritative references

• Current Neo4j versions — Current Neo4j 2026.07.1, 5.26.30 LTS, Cypher Shell and GDS release snapshot.
• Clustering architecture — Current server/database decoupling, primaries, secondaries, majority writes, and causal consistency.
• Deploy a basic cluster — Current discovery/bootstrap configuration and TOPOLOGY examples.
• Cluster server discovery — LIST/DNS/Kubernetes discovery and the current discovery-service model.
• Leadership, routing, and load balancing — Per-database Raft leadership, routing tables, readers/writers/routers, and routing policy.
• Managing databases in a cluster — Primary/secondary allocation and topology changes.
• Managing servers in a cluster — Server enablement, constraints, deallocation, removal, and operational lifecycle.
• Server management command syntax — SHOW/ENABLE/ALTER/DEALLOCATE/REALLOCATE/DROP server commands.
• Show databases — Role, writer, allocation counts, replication lag, and database-state evidence.
• Monitor databases in a cluster — SHOW DATABASES-based per-allocation monitoring and status interpretation.
• Resilient multi-data-center cluster — Failure-domain placement, latency/fault-tolerance tradeoffs, and recommended/anti-pattern layouts.
• Cluster disaster recovery — Recovery when allocations or whole failure domains are lost.
• Built-in procedures — Current routing, cordon, deallocation and related procedure names/deprecations.
• Python driver transactions/routing — Read/write routing and explicit database selection in the official driver.
• Python driver bookmarks — Bookmark propagation and causal-consistency coordination across sessions.