Write Propagation, Consistency Expectations, Operational Limitations, and Failure Considerations

Learning outcomes

AtlasMart's legal view builds successfully. The next risk appears during an incident: a base-table write returns success while one replica is unavailable, and the support team immediately reads the materialized view. “The base write succeeded” and “the derived view is already fresh everywhere” are not the same statement.

Terms and mental model

A base table is the Cassandra table the application writes directly. A materialized view (MV) is an automatically maintained Cassandra table derived from one base table with a different primary-key layout; applications cannot mutate the view directly. In Cassandra 5.0 this feature is explicitly experimental. A derived table is a broader design idea: any table whose rows are produced from authoritative data for another read pattern. An explicit projection or query table is a derived table owned by application/stream/reconciliation logic rather than Cassandra's MV mechanism. Storage-Attached Indexing (SAI) is Cassandra 5.0's storage-integrated secondary indexing framework and is a different way to support non-primary-key predicates. Change Data Capture (CDC) exposes durable portions of commit-log segments for an external consumer; Cassandra does not turn those raw records into a projection for you.

A primary key determines Cassandra partition placement and clustering order. The partition key is hashed to a token; replicas store token ranges according to the keyspace replication strategy. A request-scoped coordinator routes work to replicas and waits for the chosen consistency level (CL). A view replica stores a materialized-view partition, whose partition key can hash to a different token than the corresponding base row. Batchlog is Cassandra's durable coordination mechanism for certain multi-mutation operations and is used internally by MV maintenance; it is not a relational transaction log. Hints, repair, and streaming are separate convergence mechanisms. An immutable SSTable (Sorted String Table) stores flushed table/view data on disk and is managed by compaction.

1. A base write and a view mutation are related—but not identical operations

Each base replica receiving a mutation may need to inspect existing local row state to calculate the corresponding view change. The paired view partition can hash elsewhere, so Cassandra coordinates view mutations internally. Implementation details have evolved, but the operational lesson is stable: the base table and materialized view have separate storage/replica state, and the MV mechanism has additional write/locking/batchlog work. The official feature status is experimental precisely because operators must not infer relational indexed-view guarantees that Cassandra does not promise.

2. Normal propagation: measure instead of assuming

docker network inspect atlasmart-derived-net >/dev/null 2>&1 || docker network create atlasmart-derived-netdocker volume create atlasmart-derived-cass-1-datadocker volume create atlasmart-derived-cass-2-datadocker volume create atlasmart-derived-cass-3-datadocker inspect atlasmart-derived-cass-1 >/dev/null 2>&1 || docker run -d --name atlasmart-derived-cass-1 --hostname atlasmart-derived-cass-1 --network atlasmart-derived-net -e CASSANDRA_CLUSTER_NAME=atlasmart-derived-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-derived-cass-1-data:/var/lib/cassandra atlasmart-cassandra-derived:5.0.9# Wait for node 1 to become UN before starting peers.docker exec atlasmart-derived-cass-1 nodetool statusdocker inspect atlasmart-derived-cass-2 >/dev/null 2>&1 || docker run -d --name atlasmart-derived-cass-2 --hostname atlasmart-derived-cass-2 --network atlasmart-derived-net -e CASSANDRA_CLUSTER_NAME=atlasmart-derived-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-derived-cass-1 -v atlasmart-derived-cass-2-data:/var/lib/cassandra atlasmart-cassandra-derived:5.0.9docker inspect atlasmart-derived-cass-3 >/dev/null 2>&1 || docker run -d --name atlasmart-derived-cass-3 --hostname atlasmart-derived-cass-3 --network atlasmart-derived-net -e CASSANDRA_CLUSTER_NAME=atlasmart-derived-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-derived-cass-1 -v atlasmart-derived-cass-3-data:/var/lib/cassandra atlasmart-cassandra-derived:5.0.9# Continue only when all nodes are UN in dc1.docker exec atlasmart-derived-cass-1 nodetool versiondocker exec atlasmart-derived-cass-1 java -versiondocker exec atlasmart-derived-cass-1 nodetool statusdocker exec atlasmart-derived-cass-1 sh -lc "grep -E '^(materialized_views_enabled|cdc_enabled|materialized_views_on_repair_enabled):' /etc/cassandra/cassandra.yaml || true"

CREATE KEYSPACE IF NOT EXISTS atlasmart_derivedWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_derived.orders_by_day (    tenant_id text,    order_day date,    order_id uuid,    customer_id text,    created_at timestamp,    status text,    total decimal,    shipping_city text,    note text,    updated_at timestamp,    PRIMARY KEY ((tenant_id,order_day),order_id)) WITH compaction = {'class':'UnifiedCompactionStrategy'}  AND cdc = false;CONSISTENCY LOCAL_QUORUM;CREATE MATERIALIZED VIEW IF NOT EXISTS atlasmart_derived.orders_by_customer_mv ASSELECT tenant_id,order_day,order_id,customer_id,created_at,status,total,shipping_cityFROM atlasmart_derived.orders_by_dayWHERE tenant_id IS NOT NULL  AND order_day IS NOT NULL  AND order_id IS NOT NULL  AND customer_id IS NOT NULLPRIMARY KEY ((tenant_id,customer_id,order_day),order_id)WITH compaction = {'class':'UnifiedCompactionStrategy'};

CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_derived.orders_by_day (tenant_id,order_day,order_id,customer_id,created_at,status,total,shipping_city,note,updated_at) VALUES ('tenant-a','2026-09-08',20000000-0000-0000-0000-000000000001,'cust-42','2026-09-08T08:00:00Z','PAID',129.90,'Baku','gift','2026-09-08T08:00:00Z');INSERT INTO atlasmart_derived.orders_by_day (tenant_id,order_day,order_id,customer_id,created_at,status,total,shipping_city,note,updated_at) VALUES ('tenant-a','2026-09-08',20000000-0000-0000-0000-000000000002,'cust-42','2026-09-08T08:10:00Z','PACKING',59.00,'Baku','fragile','2026-09-08T08:10:00Z');INSERT INTO atlasmart_derived.orders_by_day (tenant_id,order_day,order_id,customer_id,created_at,status,total,shipping_city,note,updated_at) VALUES ('tenant-a','2026-09-08',20000000-0000-0000-0000-000000000003,'cust-99','2026-09-08T08:20:00Z','PAID',210.00,'Ganja','priority','2026-09-08T08:20:00Z');INSERT INTO atlasmart_derived.orders_by_day (tenant_id,order_day,order_id,customer_id,created_at,status,total,shipping_city,note,updated_at) VALUES ('tenant-a','2026-09-07',20000000-0000-0000-0000-000000000004,'cust-42','2026-09-07T18:00:00Z','SHIPPED',90.50,'Baku','','2026-09-07T18:00:00Z');SELECT tenant_id,order_day,order_id,customer_id,status,totalFROM atlasmart_derived.orders_by_dayWHERE tenant_id='tenant-a' AND order_day='2026-09-08';

CONSISTENCY LOCAL_QUORUM;TRACING ON;UPDATE atlasmart_derived.orders_by_daySET status='SHIPPED', updated_at='2026-09-08T09:00:00Z'WHERE tenant_id='tenant-a' AND order_day='2026-09-08'  AND order_id=20000000-0000-0000-0000-000000000001;TRACING OFF;SELECT order_id,status,totalFROM atlasmart_derived.orders_by_customer_mvWHERE tenant_id='tenant-a' AND customer_id='cust-42' AND order_day='2026-09-08';

docker exec atlasmart-derived-cass-1 nodetool viewbuildstatus atlasmart_derived orders_by_customer_mvdocker exec atlasmart-derived-cass-1 nodetool tablestats atlasmart_derived.orders_by_daydocker exec atlasmart-derived-cass-1 nodetool tablestats atlasmart_derived.orders_by_customer_mvdocker exec atlasmart-derived-cass-1 nodetool tpstats | grep -i -E 'view|mutation|batch' || true

The trace proves what the coordinator observed for the client mutation; it may not expose every internal MV message. Thread-pool/table metrics and subsequent reads provide complementary evidence. Exact stage names/counters are implementation-dependent and should be treated as diagnostics, not a public API.

3. Failure timeline: pause one replica, write at LOCAL_QUORUM, recover

RF=3 with LOCAL_QUORUM can tolerate one unavailable local replica for the base write. The view also has RF=3, but its replicas are chosen from the view token. On this three-node RF=3 lab all nodes store each replicated partition, which makes the recovery exercise compact without pretending the production topology is equally simple.

docker pause atlasmart-derived-cass-3docker exec atlasmart-derived-cass-1 nodetool status

CONSISTENCY LOCAL_QUORUM;UPDATE atlasmart_derived.orders_by_daySET status='DELIVERED', updated_at='2026-09-08T09:20:00Z'WHERE tenant_id='tenant-a' AND order_day='2026-09-08'  AND order_id=20000000-0000-0000-0000-000000000001;SELECT order_id,statusFROM atlasmart_derived.orders_by_customer_mvWHERE tenant_id='tenant-a' AND customer_id='cust-42' AND order_day='2026-09-08';

docker unpause atlasmart-derived-cass-3# Wait until node 3 returns UN.docker exec atlasmart-derived-cass-1 nodetool status# Confirm the pinned configuration governing MV replay during streaming/repair.docker exec atlasmart-derived-cass-1 sh -lc "grep -E '^materialized_views_on_repair_enabled:' /etc/cassandra/cassandra.yaml || true"# Controlled lab repair; record network/disk cost and duration on your machine.docker exec atlasmart-derived-cass-1 nodetool repair --full atlasmart_deriveddocker exec atlasmart-derived-cass-1 nodetool viewbuildstatus atlasmart_derived orders_by_customer_mv

Current Cassandra documentation says materialized_views_on_repair_enabled defaults true: materialized-view mutations associated with streamed data are replayed through the write path/commit log. Setting it false can make streaming faster but introduces an explicit risk that, in extreme loss scenarios, SSTable data may not be reflected in the view. Treat this as a recovery contract, not a throughput knob.

4. Verify convergence from multiple coordinators

for n in 1 2 3; do  echo "=== coordinator node $n : base ==="  docker exec atlasmart-derived-cass-$n cqlsh -e "CONSISTENCY LOCAL_ONE; SELECT order_id,status,updated_at FROM atlasmart_derived.orders_by_day WHERE tenant_id='tenant-a' AND order_day='2026-09-08' AND order_id=20000000-0000-0000-0000-000000000001;"  echo "=== coordinator node $n : view ==="  docker exec atlasmart-derived-cass-$n cqlsh -e "CONSISTENCY LOCAL_ONE; SELECT order_id,status FROM atlasmart_derived.orders_by_customer_mv WHERE tenant_id='tenant-a' AND customer_id='cust-42' AND order_day='2026-09-08';"done

Because LOCAL_ONE through a coordinator is not a forensic “read this exact SSTable” command, combine the result with tracing/status if you need to prove which replica answered. The acceptance criterion is application-visible convergence after recovery/repair—not a claim that every intermediate state was synchronous.

5. Verification checklist

• The base write succeeds/fails exactly as RF=3/LOCAL_QUORUM availability predicts.
• All nodes are restored to UN.
• View build status is complete before judging query results.
• materialized_views_on_repair_enabled is recorded from the running image.
• Base and view results are compared after explicit repair rather than assuming recovery from one successful read.

Production judgment

Do not choose a derived-data mechanism because it removes code from one layer. Evaluate who owns correctness, replay, backfill, deletion semantics, schema evolution, failure recovery, repair, observability, and rollback. For Cassandra materialized views, start with the documented fact that the feature is experimental and disabled by default. If an organization still evaluates it, test base/view key shape, update/delete mix, view build duration, write amplification, p95/p99 base-write latency, view-query latency/freshness, replica failures, streaming/repair behavior, disk headroom, compaction/tombstones, and upgrade behavior under the exact Cassandra patch. For explicit query tables or CDC/application projections, budget application/stream complexity, idempotency, checkpointing, retries, poison-event handling, backfill, reconciliation lag, and ownership. For SAI, measure selectivity/fanout, index build/disk cost, query limits, and failure behavior.

Replication factor and consistency level protect Cassandra replica operations but do not turn a derived-data pipeline into a global transaction. Security and tenant boundaries must be part of the query/table/index design; a local secondary access path is not authorization. Driver retry and speculative execution require idempotency analysis. Managed Cassandra services may disable, alter, or omit materialized views/CDC and may expose different repair/backup controls, so verify the actual service contract rather than assuming Apache Cassandra behavior. Keep migration reversible: dual-read comparison, backfill/reconciliation, explicit cutover criteria, and the ability to drop or stop the derived mechanism without losing the authoritative base data. Lesson 3 replaces implicit view ownership with an explicit denormalized projection and shows exactly what application-owned backfill and reconciliation buy you.

Summary and next bridge

Materialized-view maintenance is a distributed write/recovery path, not a magical synchronous local index. Base success, view freshness, build, streaming, and repair are separate evidence. Lesson 3 asks whether explicit denormalized tables provide a better operational contract even when they require more application code.

Authoritative references

Materialized-view behavior is version-sensitive and the feature remains experimental in Cassandra 5.0. Re-check these sources before promoting any example beyond a disposable lab.

• Apache Cassandra downloads / current 5.0 patch
• Cassandra 5.0 materialized-view CQL rules and limitations
• cassandra.yaml: materialized views, repair replay, CDC flags
• nodetool command index including viewbuildstatus
• Cassandra repair guidance
• Cassandra Change Data Capture
• SAI index reference
• Cassandra data-modeling introduction