Big Data AcademyQuery · move · process · govern

Chapter 16 · Pub/Sub, Keyspace Notifications, and Messaging Tradeoffs

Pub/Sub vs Streams vs External Brokers: Replay, Consumer State, Ordering, and Durability

Compare Redis Pub/Sub, Redis Streams, and external brokers by replay, consumer state, ordering scope, durability, fan-out, backpressure, and operational responsibility using observable fixtures.

Advanced170–230 minutesPub/Sub vs Streams vs external brokersRedis Open Source 8.10.1redis-py 8.1.0Free/local-firstLast reviewed: September 6, 2026

Learning outcomes

AtlasMart has three messaging needs: live UI fan-out, replayable internal events, and potentially a cross-service integration backbone. Those needs should not be forced into one primitive. This lesson compares Redis Pub/Sub, Redis Streams, and an external durable broker by the semantics that affect correctness and operations.

01

Compare fan-out, replay, consumer state, acknowledgment, retention, and ordering scope without claiming one universal winner.

02

Use one AtlasMart event to prove that Pub/Sub disappears during disconnect while the Stream entry remains addressable.

03

Inspect Stream consumer-group pending and acknowledgment state as a concrete contrast with Pub/Sub.

04

Describe external-broker capabilities as product/configuration dependent instead of importing Kafka/RabbitMQ/NATS guarantees generically.

05

Choose the primitive from failure recovery, scale, operational ownership, latency, retention, and topology requirements.

Exact lab baseline

All Chapter 16 mandatory labs reuse the disposable Chapter 01 environment: Redis Open Source 8.10.1 from pinned Docker image redis:8.10.1, container atlasmart-redis-ch01, standalone topology, endpoint 127.0.0.1:6379, TLS disabled only because traffic is loopback-local, default ACL user disabled, named academy-admin and atlasmart-app users, logical database 0, AOF with appendfsync everysec plus RDB snapshots, persistent /data, and no explicit maxmemory/eviction policy. Python examples target redis-py 8.1.0. The Chapter 01 application ACL intentionally does not assume Pub/Sub command permissions, so Pub/Sub/configuration demonstrations use academy-admin until Lesson 5 creates a temporary least-privilege Chapter 16 user. Fixtures stay under atlasmart:ch16:.

1. Start from the recovery question

The fastest way to choose messaging infrastructure is to ask: what must happen after a consumer is offline or crashes mid-processing? If “nothing; current state is enough,” Pub/Sub may fit. If “replay the event and know whether it was processed,” use a persisted log/queue mechanism such as Redis Streams or an external broker whose durability/consumer model is configured for that requirement.

Capability Redis Pub/Sub Redis Streams External broker (product/config dependent)
Persistence/replay No Yes, while entries retained Usually configurable; verify product
Consumer pending/ack state No Consumer groups + PEL + XACK Often available; semantics vary
Live fan-out Yes Yes via XREAD / multiple groups Usually
Offline catch-up No Yes by ID/group state Often
Retention None for publications Stream trimming/deletion policy Product policy/log retention
Operational surface Small Redis data + group operations Separate service/cluster and tooling

2. Ordering exists, but its scope matters

Redis Pub/Sub delivers messages to a connected subscriber in publish order for the routing path, but there is no replay log or cross-failure cursor. Streams assign ordered IDs inside a Stream and preserve entries for later range/consumer reads. External brokers may order per queue, partition, subject, key, or another scope. “The broker is ordered” is therefore incomplete; document the exact scope your application relies on.

3. One event, two Redis mechanisms

The following fixture writes an identical logical AtlasMart event to a live Pub/Sub channel and a durable Stream. Run it once with an active subscriber and again while the subscriber is disconnected. The Stream grows in both cases; Pub/Sub has no history to inspect later.

redis-cli · publish signal and persist event separately 
PUBLISH atlasmart:ch16:l4:live '{"event_id":"e-1001","type":"shipment-ready"}'XADD atlasmart:ch16:l4:events '*' event_id e-1001 type shipment-readyXLEN atlasmart:ch16:l4:eventsXRANGE atlasmart:ch16:l4:events - +# The Stream entry remains. There is no equivalent Pub/Sub history command.

4. Consumer groups make processing state explicit

A Stream consumer group remembers which entries were delivered but not yet acknowledged in its Pending Entries List (PEL). XREADGROUP establishes ownership and XACK removes pending state after processing. Pub/Sub has no analogous ownership or acknowledgment record.

redis-cli · observe Stream consumer state 
XGROUP CREATE atlasmart:ch16:l4:events notifications 0 MKSTREAMXADD atlasmart:ch16:l4:events '*' event_id e-1002 type shipment-readyXREADGROUP GROUP notifications worker-1 COUNT 1 STREAMS atlasmart:ch16:l4:events '>'XPENDING atlasmart:ch16:l4:events notifications# Copy the returned entry ID, then:XACK atlasmart:ch16:l4:events notifications <entry-id>XPENDING atlasmart:ch16:l4:events notifications# Pending state is observable application-processing metadata that Pub/Sub does not have.

5. Redis persistence still needs the right interpretation

Streams are Redis data, so their retained entries participate in Redis persistence/replication according to the server topology/configuration. That is stronger than Pub/Sub replay, but it is not automatically an off-host backup, zero-data-loss failover, or immutable audit system. AOF everysec, asynchronous replication, memory policy, trimming, and operator mistakes all shape the actual recovery envelope.

Durable does not mean indestructible

If the requirement is regulatory retention, multi-region event durability, long replay windows, or independent failure domains, evaluate those requirements explicitly instead of assuming “Redis Stream” or “external broker” alone satisfies them.

6. External brokers earn their complexity only when requirements need it

Dedicated brokers/log platforms can add long retention, disk-first architectures, partitions, replication policies, consumer offsets, dead-letter routing, cross-region tooling, or ecosystem connectors. But exact semantics vary by product and configuration. Apache Kafka, RabbitMQ, and NATS JetStream are examples of different designs—not interchangeable labels for “more durable than Redis.” Compare documented guarantees, operational skill, cost, client maturity, and failure tests for the product actually selected.

Question Why it may favor an external broker
Long replay window / large retained log Memory-centric Redis capacity may be less economical
Many independent service consumers Broker-native consumer/partition tooling may fit better
Cross-region durability requirements Dedicated replication/failure-domain features may be required
Protocol/ecosystem integration Connectors and broker-specific tooling may reduce application work
Redis already operational, short event window Streams may keep architecture simpler

7. Backpressure differs across the models

Pub/Sub pushes toward live subscribers and can accumulate output-buffer pressure when a consumer cannot keep up. Streams let consumers pull/read in bounded counts and expose lag/pending state, which makes recovery/backpressure visible but still requires application controls. External brokers each expose their own flow-control and lag mechanisms. Choose a model whose overload behavior your team can operate, not just its happy-path latency.

8. Failure/misuse: “replace Pub/Sub with Streams and everything is exactly once”

Streams fix replay and consumer-state gaps, but an at-least-once consumer can still execute an external side effect and crash before XACK, causing redelivery. The application still needs idempotency/reconciliation. Conversely, using a heavy external broker for a disposable typing indicator can add unnecessary operational surface. Match failure semantics to business consequence.

Exactly-once warning

Do not advertise an end-to-end exactly-once guarantee merely because a broker or Redis feature offers deduplication/transaction options. External side effects, retries, clients, and recovery boundaries determine end-to-end behavior.

9. Decision matrix for AtlasMart

AtlasMart use case Recommended starting primitive Why
Live product-price refresh hint Pub/Sub Current state can be re-read; missed hint is recoverable
Order fulfillment work Stream consumer group Replay, pending ownership, acknowledgment needed
Audit/event history across long retention External durable log or purpose-built store Independent retention/failure-domain requirements may dominate
Cache invalidation from Redis mutation Keyspace notification or explicit Pub/Sub hint Signal only; cache re-reads authority
Large cross-service integration backbone Evaluate dedicated broker Scale/retention/connectors/ops may justify separate platform

10. Verification and cleanup

  • You observed a Stream entry after the production moment passed.
  • You inspected consumer-group PEL state and acknowledged a bounded entry.
  • No Pub/Sub replay mechanism was invented.
  • External broker guarantees were kept product/configuration-specific.
  • Persistence/replication were not equated with backup or exactly-once processing.
redis-cli · bounded Chapter 16 cleanup 
UNLINK atlasmart:ch16:l4:events

11. Production judgment

Pub/Sub minimizes state and latency for disposable fan-out. Streams add retained event data and Redis-native consumer state at the cost of memory/retention/operational management. External brokers may be appropriate when event-log scale, retention, partitioning, ecosystem, or independent failure domains outweigh the simplicity of staying in Redis. Re-run failure drills with your actual topology, persistence, consumer concurrency, payload distribution, and recovery objectives.

Check your understanding

  1. What is Pub/Sub missing that a worker queue usually needs?
  2. What does XPENDING prove?
  3. Does a Stream guarantee an external side effect happens exactly once?
  4. Why not assign one guarantee to all external brokers?
  5. What is the simplest fit for a disposable live UI hint?
Review the answers

Replay plus explicit consumer ownership/acknowledgment state.

Which Stream entries are delivered but not yet acknowledged for a consumer group.

No. At-least-once redelivery still requires idempotency/reconciliation.

Products and configurations differ in ordering, retention, replication, acknowledgment, and failure behavior.

Often Pub/Sub, provided current authoritative state can be re-read after a gap.

12. Summary and next step

Message delivery is a design spectrum, not a hierarchy where more machinery is always better. Lesson 5 composes the pieces: durable order state and replayable events remain authoritative, while Pub/Sub accelerates online delivery without becoming a source of truth.

Authoritative references

Keep knowledge open

Help the academy stay free and grow.

If these tutorials save you time, a small donation supports new lessons, technical review, diagrams, examples, and long-term maintenance.

ETHEthereum / ERC-20 only
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0

Send only Ethereum or ERC-20 compatible assets to this address.