Big Data AcademyQuery · move · process · govern

Chapter 07 · Redis Streams and Consumer Groups

Consumer Groups with XGROUP/XREADGROUP: Distribution, Pending Entries, and Acknowledgment

Distribute Stream work with consumer groups while making PEL ownership, acknowledgment order, lag, and at-least-once crash windows observable.

Intermediate → Advanced155–190 minutesConsumer-group delivery and acknowledgment labRedis Open Source 8.10.1Free/local-firstLast reviewed: September 6, 2026

Learning outcomes

AtlasMart fulfillment has several workers. Unlike dashboards, these workers should distribute new events among group members and retain evidence when an event was delivered but not yet acknowledged. A consumer group is Redis-maintained delivery state attached to a Stream. Its Pending Entries List (PEL) records entries delivered to the group but not yet acknowledged.

01

Create groups at the beginning or current end of a Stream and explain the difference between 0 and dollar.

02

Distribute new entries with XREADGROUP and the greater-than marker.

03

Observe group, consumer, PEL, last-delivered, pending, and lag state.

04

Acknowledge only after the application side effect has succeeded.

05

Explain why crash windows create at-least-once processing and why exactly-once claims are unsafe.

Exact lab baseline

All Chapter 07 mandatory labs reuse the disposable Chapter 01 environment: Redis Open Source 8.10.1 from Docker Official Image redis:8.10.1, container atlasmart-redis-ch01, standalone topology, host publication 127.0.0.1:6379, TLS disabled only because traffic stays on loopback, default ACL user disabled, named ACL users atlasmart-app and academy-admin, logical database 0, AOF with appendfsync everysec plus RDB snapshots, persistent /data volume, and no explicit Redis maxmemory limit or eviction policy. The primary interface is the redis-cli shipped in the same pinned image. Mandatory examples use only bounded synthetic keys under atlasmart:ch07:*. No managed service, paid broker, or external API is required.

1. Stream body and consumer-group state are separate

The Stream stores entries. The group stores a delivery cursor plus per-consumer pending ownership. Creating a group does not copy the Stream, and acknowledging an entry does not delete the Stream entry by itself.

redis-cli · create a deterministic group fixture 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch07:group-ordersdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:group-orders 1000-0 order 5001 state createddocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:group-orders 2000-0 order 5002 state createddocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:group-orders 3000-0 order 5003 state createddocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XGROUP CREATE atlasmart:ch07:group-orders fulfillment 0-0docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XINFO GROUPS atlasmart:ch07:group-orders

Starting at 0-0 means the group's next “new entry” delivery can begin with existing history. If you create the group at $, it starts after the Stream's current last entry and intentionally skips existing entries for normal new-message delivery.

2. MKSTREAM creates an empty Stream when needed

XGROUP CREATE ... MKSTREAM can create an empty Stream and its group in one command. Without MKSTREAM, creating a group against a missing key fails.

redis-cli · MKSTREAM boundary 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch07:mkstreamdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XGROUP CREATE atlasmart:ch07:mkstream demo '$' MKSTREAMdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XLEN atlasmart:ch07:mkstreamdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XINFO GROUPS atlasmart:ch07:mkstream

The length is zero, but the group exists. This is operational setup state, not evidence that a producer has emitted anything.

3. The greater-than marker requests never-delivered entries

With XREADGROUP GROUP group consumer ... STREAMS key '>', Redis selects entries that have not yet been delivered to any consumer in that group. Delivery creates a PEL record owned by the named consumer.

redis-cli · deliver work to two consumers 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XREADGROUP GROUP fulfillment worker-a COUNT 2 STREAMS atlasmart:ch07:group-orders '>'docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XREADGROUP GROUP fulfillment worker-b COUNT 2 STREAMS atlasmart:ch07:group-orders '>'docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XPENDING atlasmart:ch07:group-orders fulfillmentdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XINFO GROUPS atlasmart:ch07:group-ordersdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XINFO CONSUMERS atlasmart:ch07:group-orders fulfillment

Worker A receives up to two existing entries, worker B gets the remaining not-yet-delivered entry, and the group's pending count becomes 3 until acknowledgments occur.

4. Pending means delivered, not completed

A PEL record says Redis delivered the entry to a consumer and has not received XACK. It does not know whether the worker started the side effect, completed it, or crashed halfway through. That distinction is why application idempotency and recovery are mandatory.

5. XACK removes pending state, not the Stream body

XACK returns the number of supplied IDs that were actually pending in the group and were acknowledged. It removes their PEL records. The entry remains readable from the Stream until retention/delete removes it.

redis-cli · acknowledge one completed event 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XACK atlasmart:ch07:group-orders fulfillment 1000-0docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XPENDING atlasmart:ch07:group-orders fulfillmentdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XRANGE atlasmart:ch07:group-orders 1000-0 1000-0

Expected: pending decreases by one, but XRANGE still shows 1000-0. Acknowledgment and retention are intentionally separate lifecycle steps.

6. Wrong order: ACK before the business effect can lose work

If worker A acknowledges 2000-0 and then crashes before performing the side effect, Redis no longer tracks that entry as pending. A normal recovery scan of the PEL cannot discover the unfinished work.

redis-cli · controlled bad-order demonstration 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XACK atlasmart:ch07:group-orders fulfillment 2000-0docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XPENDING atlasmart:ch07:group-orders fulfillmentdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XRANGE atlasmart:ch07:group-orders 2000-0 2000-0

The Stream still contains the event, so an explicit reconciliation scan could find it, but the consumer-group acknowledgment contract has been violated. Repair: perform an idempotent business effect first, then XACK.

7. The opposite crash window creates duplicates

Suppose worker B processes 3000-0 successfully but crashes before XACK. The entry remains pending. On restart or claim by another consumer, the event can be processed again. This is the characteristic at-least-once window.

redis-cli · inspect the unacknowledged attempt 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XPENDING atlasmart:ch07:group-orders fulfillment - + 10

The detailed row includes owner, idle milliseconds, and delivery count. Redis can recover ownership state; it cannot make an external payment/email non-duplicating. The side effect must be idempotent or protected by a domain-level deduplication contract.

8. Reading a consumer’s pending history uses an ID, not greater-than

When XREADGROUP is called with an ID other than >, it reads pending messages already owned by that consumer rather than assigning fresh group entries. This is useful for a consumer restarting and replaying its own PEL.

redis-cli · worker-b pending history 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XREADGROUP GROUP fulfillment worker-b COUNT 10 STREAMS atlasmart:ch07:group-orders 0-0

Because 3000-0 is pending for worker-b in this deterministic fixture, it should appear. Once the consumer's pending history is empty, it can return to > for new work.

9. Group lag is useful, but it is not always available

XINFO GROUPS reports pending, last-delivered-id, entries-read, and lag. Lag estimates how many Stream entries remain to be delivered to the group. Redis may report lag as null when group position/history deletions make the logical read counter invalid. Therefore dashboards must handle “unknown” separately from zero.

10. Competing consumers distribute within one group

Consumers in the same group compete for never-delivered entries; groups themselves are independent. Two different groups can each receive the same Stream entry, which is useful for separate services such as fulfillment and analytics. More consumers do not automatically improve throughput if the Stream key, side effect, downstream dependency, or hot partition is the bottleneck.

11. Reproducible acknowledgment lab

Finish the remaining pending event after an idempotent synthetic side effect.

redis-cli · effect then acknowledge 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app HSET atlasmart:ch07:effect:3000-0 order 5003 status fulfilleddocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app HGETALL atlasmart:ch07:effect:3000-0docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XACK atlasmart:ch07:group-orders fulfillment 3000-0docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XPENDING atlasmart:ch07:group-orders fulfillmentdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XINFO GROUPS atlasmart:ch07:group-orders

After the HSET succeeds, XACK removes the pending record. Repeating the same HSET produces the same logical state, which makes this lab side effect idempotent.

12. Production judgment

Consumer groups are appropriate when Redis should track distribution, pending ownership, acknowledgments, and lag. They provide at-least-once recovery building blocks—not an exactly-once business transaction. Bound COUNT, use finite BLOCK timeouts where shutdown matters, monitor pending/lag/consumer idle, retain enough Stream history for recovery, and test crash points before/after the side effect and before/after XACK.

13. Summary and next step

Groups make delivery state explicit. Next we recover abandoned pending entries, distinguish slow from dead consumers, and stop poison events from retrying forever.

Check your understanding

  1. What does a PEL entry prove?
  2. What does XACK remove?
  3. What does the greater-than marker mean in XREADGROUP?
  4. Why must the business effect normally happen before XACK?
  5. Does one group prevent another group from receiving the same Stream entry?
Review the answers

That an entry was delivered in the group and has not yet been acknowledged; it does not prove side-effect completion.

The pending reference for the acknowledged ID in that group, not the Stream body entry.

Request entries not yet delivered to any consumer in that group.

Acknowledging first can lose unfinished work if the consumer crashes afterward.

No. Different consumer groups maintain independent delivery state.

Authoritative references

  • Redis Streams — stream entries, consumer groups, pending entries, acknowledgments, and recovery
  • XADD — entry IDs, trimming, Redis 8.2 reference policies, and Redis 8.6 idempotent production options
  • XRANGE — ordered range inspection and exclusive continuation IDs
  • XREAD — blocking/nonblocking reads, explicit IDs, and the special dollar ID
  • XGROUP CREATE — consumer-group starting position and MKSTREAM
  • XREADGROUP — group delivery, pending history, and new-message marker
  • XPENDING — PEL summary/details, idle time, owner, and delivery count
  • XACK — acknowledgment semantics
  • XCLAIM — explicit ownership transfer and retry-count behavior
  • XAUTOCLAIM — cursor-like stale-pending recovery
  • XINFO GROUPS — pending count, last-delivered ID, entries-read, and lag
  • XTRIM — exact/approximate retention and PEL reference policies
  • Redis 8.10 release notes — pinned server release family

Keep knowledge open

Help the academy stay free and grow.

If these tutorials save you time, a small donation supports new lessons, technical review, diagrams, examples, and long-term maintenance.

ETHEthereum / ERC-20 only
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0

Send only Ethereum or ERC-20 compatible assets to this address.