Change Data Capture from Logs vs Application-Published Events

1. One order change, two very different contracts

AtlasMart updates an order from CREATED to PAID. A database can expose that fact from its transaction log through change data capture (CDC): machinery that converts committed storage/database changes into a consumable change stream. The application can also publish an application event or domain event such as OrderPaid. Both can describe the same underlying moment, but they have different meanings and owners.

A CDC record usually answers questions such as “which table/collection changed, which key changed, what was the before/after image, what transaction/log position produced it?” A domain event answers a business question: “what happened that downstream services are allowed to depend on?” Treating those contracts as interchangeable is a common long-lived integration mistake.

2. Log positions, snapshots, and transaction coupling

A database log is ordered according to the database's commit/log semantics. Consumers need a position—often called an offset, log sequence number, or connector-specific source position—to say how far they have processed. A CDC pipeline commonly combines an initial snapshot of existing rows with a change stream starting at a position that makes the snapshot and subsequent changes consistent.

3. Observable history: before/after images are evidence, not intent

The lab records WAL-like changes at positions 100, 110, and 120. The order table update at 110 has a before image CREATED and after image PAID. Application logic can map that transition to OrderPaid, but the mapping itself is domain knowledge. A generic CDC connector cannot know whether the same column transition represents capture, reconciliation, manual repair, import, or test traffic unless the source contract supplies that meaning.

4. Deliberately wrong approach: “CDC delivers exactly once, so duplicates are impossible”

Suppose a consumer processes source position 110, updates its search projection, but crashes before durably storing checkpoint 110. On restart its persisted checkpoint is still 100. Positions 110 and 120 may be delivered again. PostgreSQL 18's logical-decoding documentation explicitly warns that after a crash a slot can return to an earlier persisted position and resend recent changes; clients must avoid harmful duplicate processing.

The safe design records source position/event identity with the derived effect or makes the operation naturally idempotent. Do not confuse source-order replay semantics with end-to-end exactly-once effects.

5. AtlasMart lab: compare raw CDC to semantic events

The simulator emits WAL-like changes, maps one transition to a domain event, records a snapshot boundary, and intentionally restarts from an older durable checkpoint.

from dataclasses import dataclass
from copy import deepcopy

@dataclass(frozen=True)
class WalChange:
    lsn: int
    txid: int
    table: str
    op: str
    key: str
    before: dict | None
    after: dict | None

wal = [
    WalChange(100, 41, "orders", "INSERT", "o-7", None,
              {"status":"CREATED", "total":120, "version":1}),
    WalChange(110, 42, "orders", "UPDATE", "o-7",
              {"status":"CREATED", "total":120, "version":1},
              {"status":"PAID", "total":120, "version":2}),
    WalChange(120, 42, "payments", "INSERT", "p-7", None,
              {"order_id":"o-7", "captured":True, "version":1}),
]

print("DATABASE-LOG CDC")
for c in wal:
    print(c.lsn, c.txid, c.table, c.op, c.key, "before=", c.before, "after=", c.after)

print("\nDOMAIN MAPPING")
def to_domain_event(change):
    if change.table == "orders" and change.op == "UPDATE":
        if change.before["status"] != "PAID" and change.after["status"] == "PAID":
            return {"type":"OrderPaid", "order_id":change.key,
                    "order_version":change.after["version"], "source_lsn":change.lsn}
    return None
for c in wal:
    evt = to_domain_event(c)
    if evt: print(evt)

print("\nSNAPSHOT + OFFSET")
source = {"o-7":{"status":"PAID","version":2}, "o-8":{"status":"CREATED","version":1}}
snapshot_at_lsn = 120
print("snapshot rows:", deepcopy(source), "resume_after_lsn:", snapshot_at_lsn)

print("\nCRASH / REPLAY")
consumer_checkpoint = 100          # last durable checkpoint
already_seen_in_memory = 110       # processed but not durably checkpointed
print("processed through", already_seen_in_memory, "but durable checkpoint is", consumer_checkpoint)
replayed = [c.lsn for c in wal if c.lsn > consumer_checkpoint]
print("after restart CDC may replay:", replayed)
print("duplicate-tolerant consumer required:", 110 in replayed)

print("\nSEMANTIC WARNING")
print("row UPDATE says what bytes changed; it does not by itself prove why the business changed them")

6. Production judgment

Use log CDC when consumers need a faithful stream of committed source changes, when application modification is difficult, or when many derived systems must follow the same source. Use explicit domain events when the semantic contract matters more than physical schema and the application can own that contract. For CDC, observe source-log retention, connector checkpoint/offset, snapshot phase, lag in time and bytes, schema-history health, replication-slot/WAL retention, and duplicate/replay counts. Apply least privilege to log readers because CDC can expose sensitive columns that ordinary application APIs would filter.

Current optional references: PostgreSQL 18 provides logical decoding from WAL and durable replication slots; Debezium 3.6.1.Final (August 4, 2026) is the latest stable 3.6 maintenance release and supports PostgreSQL 18 among its tested versions. Neither is required for the mandatory lab.

The next lesson solves the application-side atomicity problem: how to commit business state and publication intent without a fragile database-plus-broker dual write.

References

Foundational claims use primary specifications/research or current official documentation where practical. Product references are optional implementation anchors; the mandatory labs are vendor-neutral.

• PostgreSQL 18 logical decoding concepts — Official WAL decoding, replication-slot, checkpoint, and duplicate replay behavior.
• PostgreSQL 18 logical decoding chapter — Official change-stream infrastructure and row-image availability.
• Debezium 3.6 release series — Current stable series details and tested database versions.
• Debezium 3.6.1.Final release notes — August 4, 2026 maintenance release and compatibility notes.