CDC / Streams / Application Dual Writes vs Materialized Views for Derived Read Models

Learning outcomes

AtlasMart now wants one order change to feed a customer dashboard, fraud pipeline, warehouse export, and notification service. A materialized view can only maintain a constrained Cassandra table. The requirement has become an event-distribution problem, so this lesson compares raw Cassandra CDC, an external stream/consumer, and direct application dual writes.

Terms and mental model

A base table is the Cassandra table the application writes directly. A materialized view (MV) is an automatically maintained Cassandra table derived from one base table with a different primary-key layout; applications cannot mutate the view directly. In Cassandra 5.0 this feature is explicitly experimental. A derived table is a broader design idea: any table whose rows are produced from authoritative data for another read pattern. An explicit projection or query table is a derived table owned by application/stream/reconciliation logic rather than Cassandra's MV mechanism. Storage-Attached Indexing (SAI) is Cassandra 5.0's storage-integrated secondary indexing framework and is a different way to support non-primary-key predicates. Change Data Capture (CDC) exposes durable portions of commit-log segments for an external consumer; Cassandra does not turn those raw records into a projection for you.

A primary key determines Cassandra partition placement and clustering order. The partition key is hashed to a token; replicas store token ranges according to the keyspace replication strategy. A request-scoped coordinator routes work to replicas and waits for the chosen consistency level (CL). A view replica stores a materialized-view partition, whose partition key can hash to a different token than the corresponding base row. Batchlog is Cassandra's durable coordination mechanism for certain multi-mutation operations and is used internally by MV maintenance; it is not a relational transaction log. Hints, repair, and streaming are separate convergence mechanisms. An immutable SSTable (Sorted String Table) stores flushed table/view data on disk and is managed by compaction.

1. CDC is a source of durable change records—not a projection engine

Node-wide cdc_enabled is already true only in the isolated custom image, but a table participates only after cdc=true. Cassandra exposes durable portions of commit-log segments in cdc_raw_directory with companion _cdc.idx offsets. A consumer must decode records, checkpoint progress, handle duplicates/restarts, and delete/archive processed segments. If CDC space fills while the default blocking behavior is active, writes to CDC-enabled tables can be rejected. That operational backpressure is part of the feature contract.

docker network inspect atlasmart-derived-net >/dev/null 2>&1 || docker network create atlasmart-derived-netdocker volume create atlasmart-derived-cass-1-datadocker volume create atlasmart-derived-cass-2-datadocker volume create atlasmart-derived-cass-3-datadocker inspect atlasmart-derived-cass-1 >/dev/null 2>&1 || docker run -d --name atlasmart-derived-cass-1 --hostname atlasmart-derived-cass-1 --network atlasmart-derived-net -e CASSANDRA_CLUSTER_NAME=atlasmart-derived-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -v atlasmart-derived-cass-1-data:/var/lib/cassandra atlasmart-cassandra-derived:5.0.9# Wait for node 1 to become UN before starting peers.docker exec atlasmart-derived-cass-1 nodetool statusdocker inspect atlasmart-derived-cass-2 >/dev/null 2>&1 || docker run -d --name atlasmart-derived-cass-2 --hostname atlasmart-derived-cass-2 --network atlasmart-derived-net -e CASSANDRA_CLUSTER_NAME=atlasmart-derived-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack2 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-derived-cass-1 -v atlasmart-derived-cass-2-data:/var/lib/cassandra atlasmart-cassandra-derived:5.0.9docker inspect atlasmart-derived-cass-3 >/dev/null 2>&1 || docker run -d --name atlasmart-derived-cass-3 --hostname atlasmart-derived-cass-3 --network atlasmart-derived-net -e CASSANDRA_CLUSTER_NAME=atlasmart-derived-course -e CASSANDRA_DC=dc1 -e CASSANDRA_RACK=rack3 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_NUM_TOKENS=16 -e CASSANDRA_SEEDS=atlasmart-derived-cass-1 -v atlasmart-derived-cass-3-data:/var/lib/cassandra atlasmart-cassandra-derived:5.0.9# Continue only when all nodes are UN in dc1.docker exec atlasmart-derived-cass-1 nodetool versiondocker exec atlasmart-derived-cass-1 java -versiondocker exec atlasmart-derived-cass-1 nodetool statusdocker exec atlasmart-derived-cass-1 sh -lc "grep -E '^(materialized_views_enabled|cdc_enabled|materialized_views_on_repair_enabled):' /etc/cassandra/cassandra.yaml || true"

CREATE KEYSPACE IF NOT EXISTS atlasmart_derivedWITH replication = {'class':'NetworkTopologyStrategy','dc1':3};CREATE TABLE IF NOT EXISTS atlasmart_derived.orders_by_day (    tenant_id text,    order_day date,    order_id uuid,    customer_id text,    created_at timestamp,    status text,    total decimal,    shipping_city text,    note text,    updated_at timestamp,    PRIMARY KEY ((tenant_id,order_day),order_id)) WITH compaction = {'class':'UnifiedCompactionStrategy'}  AND cdc = false;CONSISTENCY LOCAL_QUORUM;CREATE MATERIALIZED VIEW IF NOT EXISTS atlasmart_derived.orders_by_customer_mv ASSELECT tenant_id,order_day,order_id,customer_id,created_at,status,total,shipping_cityFROM atlasmart_derived.orders_by_dayWHERE tenant_id IS NOT NULL  AND order_day IS NOT NULL  AND order_id IS NOT NULL  AND customer_id IS NOT NULLPRIMARY KEY ((tenant_id,customer_id,order_day),order_id)WITH compaction = {'class':'UnifiedCompactionStrategy'};CREATE TABLE IF NOT EXISTS atlasmart_derived.orders_by_customer_projection (    tenant_id text,    customer_id text,    order_month date,    created_at timestamp,    order_id uuid,    status text,    total decimal,    shipping_city text,    source_updated_at timestamp,    projection_request_id uuid,    PRIMARY KEY ((tenant_id,customer_id,order_month),created_at,order_id)) WITH CLUSTERING ORDER BY (created_at DESC,order_id ASC)  AND compaction = {'class':'UnifiedCompactionStrategy'};

ALTER TABLE atlasmart_derived.orders_by_day WITH cdc = true;DESCRIBE TABLE atlasmart_derived.orders_by_day;CONSISTENCY LOCAL_QUORUM;INSERT INTO atlasmart_derived.orders_by_day(tenant_id,order_day,order_id,customer_id,created_at,status,total,shipping_city,note,updated_at)VALUES ('tenant-a','2026-09-08',20000000-0000-0000-0000-000000000020,'cust-77','2026-09-08T10:30:00Z','PAID',44.00,'Baku','cdc-demo','2026-09-08T10:30:00Z');UPDATE atlasmart_derived.orders_by_day SET status='PACKING'WHERE tenant_id='tenant-a' AND order_day='2026-09-08'  AND order_id=20000000-0000-0000-0000-000000000020;

# Flush can help old commit-log segments become recyclable; exact CDC file timing is version/workload dependent.for n in 1 2 3; do docker exec atlasmart-derived-cass-$n nodetool flush atlasmart_derived orders_by_day; donefor n in 1 2 3; do  echo "=== node $n cdc_raw ==="  docker exec atlasmart-derived-cass-$n sh -lc "ls -lah /var/lib/cassandra/cdc_raw | tail -20"  docker exec atlasmart-derived-cass-$n sh -lc "for f in /var/lib/cassandra/cdc_raw/*_cdc.idx; do [ -f \"$f\" ] && echo ---\ $f && cat \"$f\"; done"done

An index offset/word such as COMPLETED tells a consumer how far the corresponding commit-log segment is safe to parse. It does not tell you the business projection was applied. Parsing binary commit-log records is deliberately not hidden in a one-line shell command; production consumers use Cassandra's commit-log reader APIs or a maintained integration and own schema/version compatibility.

2. Build an explicit event/request contract

CREATE TABLE IF NOT EXISTS atlasmart_derived.projection_events_by_request (    tenant_id text,    request_id uuid,    order_day date,    order_id uuid,    event_type text,    source_updated_at timestamp,    projection_state text,    last_error text,    PRIMARY KEY ((tenant_id),request_id)) WITH compaction = {'class':'UnifiedCompactionStrategy'};INSERT INTO atlasmart_derived.projection_events_by_request(tenant_id,request_id,order_day,order_id,event_type,source_updated_at,projection_state,last_error)VALUES ('tenant-a',40000000-0000-0000-0000-000000000020,'2026-09-08',20000000-0000-0000-0000-000000000020,'ORDER_STATUS_CHANGED','2026-09-08T10:31:00Z','PENDING',null);SELECT * FROM atlasmart_derived.projection_events_by_requestWHERE tenant_id='tenant-a';

This table is a teaching checkpoint, not a claim that Cassandra provides a transactional outbox across unrelated partitions. If the source mutation and event record must be atomic, redesign them into a single partition where appropriate, use a carefully bounded logged batch after measuring its tradeoffs, or choose infrastructure with the transaction semantics the business invariant truly requires. The important point is explicit ownership of “pending/applied/failed” rather than invisible best effort.

3. Dual write failure and reconciliation

UPDATE atlasmart_derived.orders_by_day SET status='SHIPPED'WHERE tenant_id='tenant-a' AND order_day='2026-09-08'  AND order_id=20000000-0000-0000-0000-000000000020;-- Intentionally DO NOT write orders_by_customer_projection yet.UPDATE atlasmart_derived.projection_events_by_requestSET projection_state='PENDING', last_error='simulated worker crash before projection'WHERE tenant_id='tenant-a'  AND request_id=40000000-0000-0000-0000-000000000020;

INSERT INTO atlasmart_derived.orders_by_customer_projection(tenant_id,customer_id,order_month,created_at,order_id,status,total,shipping_city,source_updated_at,projection_request_id)VALUES ('tenant-a','cust-77','2026-09-01','2026-09-08T10:30:00Z',20000000-0000-0000-0000-000000000020,'SHIPPED',44.00,'Baku','2026-09-08T10:31:00Z',40000000-0000-0000-0000-000000000020);UPDATE atlasmart_derived.projection_events_by_requestSET projection_state='COMPLETE', last_error=nullWHERE tenant_id='tenant-a'  AND request_id=40000000-0000-0000-0000-000000000020;SELECT * FROM atlasmart_derived.projection_events_by_requestWHERE tenant_id='tenant-a';

4. Ownership matrix: MV vs CDC/stream vs application dual write

Kafka, Pulsar, managed CDC connectors, and commercial streaming platforms can be useful, but none is required for this lesson. The free/local path is raw Cassandra CDC evidence plus an explicit projection-event simulation. If a managed service does not expose raw CDC or materialized views, its documented connector/change-stream interface becomes a different product contract.

5. Verification and cleanup of CDC state

• cdc_enabled=true is isolated to the Chapter 20 image.
• orders_by_day explicitly shows cdc=true.
• CDC raw/index files are observed without modifying live segments.
• The simulated projection crash leaves a visible PENDING event.
• Reconciliation produces the correct projection and marks the request COMPLETE.

Production judgment

Do not choose a derived-data mechanism because it removes code from one layer. Evaluate who owns correctness, replay, backfill, deletion semantics, schema evolution, failure recovery, repair, observability, and rollback. For Cassandra materialized views, start with the documented fact that the feature is experimental and disabled by default. If an organization still evaluates it, test base/view key shape, update/delete mix, view build duration, write amplification, p95/p99 base-write latency, view-query latency/freshness, replica failures, streaming/repair behavior, disk headroom, compaction/tombstones, and upgrade behavior under the exact Cassandra patch. For explicit query tables or CDC/application projections, budget application/stream complexity, idempotency, checkpointing, retries, poison-event handling, backfill, reconciliation lag, and ownership. For SAI, measure selectivity/fanout, index build/disk cost, query limits, and failure behavior.

Replication factor and consistency level protect Cassandra replica operations but do not turn a derived-data pipeline into a global transaction. Security and tenant boundaries must be part of the query/table/index design; a local secondary access path is not authorization. Driver retry and speculative execution require idempotency analysis. Managed Cassandra services may disable, alter, or omit materialized views/CDC and may expose different repair/backup controls, so verify the actual service contract rather than assuming Apache Cassandra behavior. Keep migration reversible: dual-read comparison, backfill/reconciliation, explicit cutover criteria, and the ability to drop or stop the derived mechanism without losing the authoritative base data. Lesson 5 turns the chapter into a decision framework: base table, SAI, experimental MV, or explicit projection/CDC are selected from workload and ownership requirements rather than convenience.

Summary and next bridge

CDC and stream/application projections expose more moving parts than a materialized view, but they also expose ownership, replay, transformations, and side effects. The final lesson chooses among base tables, SAI, MV, and explicit projections using concrete AtlasMart requirements and operational acceptance criteria.

Authoritative references

Materialized-view behavior is version-sensitive and the feature remains experimental in Cassandra 5.0. Re-check these sources before promoting any example beyond a disposable lab.

• Apache Cassandra downloads / current 5.0 patch
• Cassandra 5.0 materialized-view CQL rules and limitations
• cassandra.yaml: materialized views, repair replay, CDC flags
• nodetool command index including viewbuildstatus
• Cassandra repair guidance
• Cassandra Change Data Capture
• SAI index reference
• Cassandra data-modeling introduction