Why Batches Are Not a Bulk-Load Performance Tool and When Single-Partition Batches Help

Learning outcomes

AtlasMart needs to backfill millions of catalog projections. Someone proposes 5,000-row logged batches because “one request is faster than 5,000.” Cassandra can indeed save some client round trips, but a giant cross-partition batch concentrates coordination, memory, network, batchlog, and replica work on one coordinator. This lesson distinguishes a correctness batch from throughput concurrency.

Core terms for this chapter

Apache Cassandra is a peer-to-peer distributed database. CQL is the Cassandra Query Language. A coordinator is the node handling one client request; it routes mutations to the replicas that own the target partition. A partition groups rows sharing a partition key, and the partitioner's token mapping determines which replica set owns it. A batch is one native-protocol request carrying multiple CQL mutations. A logged batch uses Cassandra's distributed batch log to support the documented atomic batch guarantee across the mutations; an unlogged batch skips that batch log and therefore can be partially applied on failure. Isolation means readers do not observe intermediate changes within the documented same-partition scope. A counter is a special 64-bit distributed value changed only by increment/decrement operations. Idempotent means repeating an operation produces the same final database state as executing it once. A retry re-executes a request after certain failures; because write outcomes can be ambiguous, retry safety depends on idempotency and error classification. A request ID is an application-supplied stable identifier used to recognize duplicate attempts. Reconciliation is application or Cassandra logic that converges divergent/partial state after failures; it is not the same as pretending every write happened exactly once.

1. One request can still fan out into many distributed writes

Client-side request count and server-side work are different quantities. A 1,000-partition logged batch may save client/coordinator network round trips, but the coordinator must decode and retain the request, persist batchlog state, route hundreds or thousands of partition mutations, track acknowledgments at the requested CL, and replay incomplete work after failure. This creates a single-request burst and often worse tail latency than a bounded number of independent asynchronous writes distributed across coordinators by a token-aware driver.

A same-partition batch is different: the mutations share one replica set, the batchlog is optimized away, and the application may actually need same-partition atomic/isolation semantics. Even there, “more statements” is not free—payload size, tombstones, SAI/vector index maintenance, and replica CPU/disk work still matter.

2. Build a same-partition comparison

# Verify the existing course cluster first.docker exec atlasmart-cass-1 nodetool versiondocker exec atlasmart-cass-1 nodetool statusdocker exec atlasmart-cass-1 java -version# If the shared course cluster does not exist, recreate the same local topology.docker network inspect atlasmart-cassandra >/dev/null 2>&1 || docker network create atlasmart-cassandradocker volume create atlasmart-cass-1-datadocker volume create atlasmart-cass-2-datadocker volume create atlasmart-cass-3-datadocker inspect atlasmart-cass-1 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-1 --hostname atlasmart-cass-1 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-cass-1-data:/var/lib/cassandra cassandra:5.0.9# Continue only after node 1 is UN.docker exec atlasmart-cass-1 nodetool statusdocker inspect atlasmart-cass-2 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-2 --hostname atlasmart-cass-2 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-2-data:/var/lib/cassandra cassandra:5.0.9docker inspect atlasmart-cass-3 >/dev/null 2>&1 || docker run -d --name atlasmart-cass-3 --hostname atlasmart-cass-3 --network atlasmart-cassandra -e CASSANDRA_CLUSTER_NAME=atlasmart-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-cass-1 -v atlasmart-cass-3-data:/var/lib/cassandra cassandra:5.0.9docker exec atlasmart-cass-1 nodetool statusdocker exec -it atlasmart-cass-1 cqlsh

CREATE KEYSPACE IF NOT EXISTS atlasmart_writecoordWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_writecoord.orders_by_customer (    customer_id text,    order_month date,    order_id uuid,    status text,    total decimal,    request_id uuid,    updated_at timestamp,    PRIMARY KEY ((customer_id,order_month),order_id)) WITH compaction = {'class':'UnifiedCompactionStrategy'};CREATE TABLE IF NOT EXISTS atlasmart_writecoord.order_events_by_order (    order_id uuid,    event_id uuid,    event_type text,    detail text,    event_time timestamp,    PRIMARY KEY (order_id,event_id)) WITH compaction = {'class':'UnifiedCompactionStrategy'};CONSISTENCY LOCAL_QUORUM;

CONSISTENCY LOCAL_QUORUM;TRACING ON;BEGIN BATCH  INSERT INTO atlasmart_writecoord.orders_by_customer  (customer_id,order_month,order_id,status,total,request_id,updated_at)  VALUES ('cust-bulk','2026-09-01',00000000-0000-0000-0000-000000000171,'CREATED',10.00,10000000-0000-0000-0000-000000000171,toTimestamp(now()));  INSERT INTO atlasmart_writecoord.orders_by_customer  (customer_id,order_month,order_id,status,total,request_id,updated_at)  VALUES ('cust-bulk','2026-09-01',00000000-0000-0000-0000-000000000172,'CREATED',20.00,10000000-0000-0000-0000-000000000172,toTimestamp(now()));  INSERT INTO atlasmart_writecoord.orders_by_customer  (customer_id,order_month,order_id,status,total,request_id,updated_at)  VALUES ('cust-bulk','2026-09-01',00000000-0000-0000-0000-000000000173,'CREATED',30.00,10000000-0000-0000-0000-000000000173,toTimestamp(now()));APPLY BATCH;TRACING OFF;SELECT order_id,status,total FROM atlasmart_writecoord.orders_by_customerWHERE customer_id='cust-bulk' AND order_month='2026-09-01';

Record the trace and compare it with Lesson 1's cross-partition trace. Do not expect identical event text. The useful question is whether all mutations share one partition/replica set and whether the logged-batch optimization removes separate batchlog work.

3. Wrong benchmark: one huge logged batch

Cassandra has server-side batch-size warning/failure thresholds and other guardrails because oversized batches are operationally dangerous, but the thresholds are safeguards—not target sizes. Never tune them upward just to silence warnings. The correct experiment holds data and write count constant while comparing multiple request shapes.

// Pseudocode-level lab: use the same prepared INSERT for each row.PreparedStatement ps = session.prepare(    "INSERT INTO atlasmart_writecoord.orders_by_customer " +    "(customer_id,order_month,order_id,status,total,request_id,updated_at) " +    "VALUES (?,?,?,?,?,?,?)");Semaphore inFlight = new Semaphore(64); // example test input, NOT a universal tuning valueList<CompletionStage<AsyncResultSet>> futures = new ArrayList<>();for (Order row : rows) {  inFlight.acquire();  BoundStatement bs = ps.bind(/* row fields */)      .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM)      .setIdempotent(true);  CompletionStage<AsyncResultSet> f = session.executeAsync(bs)      .whenComplete((ok,err) -> inFlight.release());  futures.add(f);}// Measure p50/p95/p99 and error/retry counts for each tested concurrency level.

$nodes = 'atlasmart-cass-1','atlasmart-cass-2','atlasmart-cass-3'docker stats --no-stream $nodesdocker exec atlasmart-cass-1 nodetool tpstatsdocker exec atlasmart-cass-1 nodetool proxyhistogramsdocker exec atlasmart-cass-1 nodetool tablestats atlasmart_writecoord.orders_by_customer# Also record rows/request, partitions/request, payload bytes, CL, concurrency, warmup, and failures.

4. Verification and benchmark acceptance

Production judgment

Choose batch/counter/retry behavior from the business invariant and partition model, not from a generic “fewer requests is faster” rule. Record RF/CL, partition cardinality and bytes, mutations per request, partitions per batch, coordinator locality, batchlog warnings/timeouts, counter contention, p50/p95/p99 latency, write timeout/failure type, retry/speculation counts, duplicate-attempt rate, request-ID reconciliation backlog, SSTable/compaction/tombstone pressure, disk/network/JVM headroom, and downstream side effects. A logged batch does not turn independent partitions into a relational transaction; a counter does not become an exact ledger; an idempotent database mutation does not automatically make an email/payment/webhook side effect idempotent.

For SAI/vector tables later in the course, remember that every extra mutation also updates index structures; large batches can concentrate write/index pressure. Security and tenancy boundaries still require authentication/authorization/network controls—same partition or LOCAL_QUORUM is not isolation. Managed Cassandra services can cap batch size, hide JMX/internal tables, or expose different metrics, so preserve the semantic tests even when the observability surface changes. Every rollout needs a rollback path: remove unsafe driver idempotence flags, stop duplicate retries, split cross-partition batches into independent async writes, or migrate counters to an event/reconciliation model when exactness requirements change. Lesson 3 introduces counters, whose increment/decrement semantics break the idempotent retry assumptions used by ordinary deterministic upserts.

Summary and next step

This lesson’s concepts, evidence path, failure boundaries, and production judgment should now be explicit enough to verify rather than assume. Re-run the check-your-understanding prompts and preserve any lab evidence you need before changing or cleaning up the environment.

Next, continue to Counter Tables, Distributed Counter Semantics, Restrictions, and Workload Suitability.

Authoritative references

Use these version-sensitive sources as the contract. Re-check them when regenerating the course rather than freezing this lesson's dated snapshot.

• Apache Cassandra downloads / current GA baseline
• CQL data manipulation: BATCH semantics
• CQL counter column usage
• Native protocol BATCH and write-type semantics
• Java Driver 4.x idempotence
• Java Driver 4.x retry policy
• Java Driver statement API