Chapter 07 · Redis Streams and Consumer Groups
XADD IDs, Stream Ordering, Approximate Trimming, and Event Retention
Append, inspect, and retain Redis Stream entries with exact ID/order semantics, bounded history, and version-aware trimming behavior.
Learning outcomes
AtlasMart wants to keep an append-oriented record of order-state events so operators can inspect history, consumers can later process entries, and retention can be bounded. A Redis Stream is a persisted Redis data structure made of ordered entries. Each entry has a unique stream ID and one or more field/value pairs. The stream entry is stored data; consumer-group ownership and acknowledgments are separate metadata introduced later in this chapter.
Explain stream IDs, automatic versus explicit IDs, and total ordering inside one stream.
Append and inspect entries with XADD, XRANGE, XREVRANGE, XLEN, and XINFO STREAM.
Distinguish exact MAXLEN/MINID trimming from approximate trimming and explain why retention is not “free.”
Explain how trimming interacts with consumer-group references, including Redis 8.2 reference policies.
Design retention from replay, recovery, memory, persistence, replication, and business-audit requirements rather than arbitrary caps.
All Chapter 07 mandatory labs reuse the disposable Chapter 01
environment: Redis Open Source 8.10.1 from Docker
Official Image redis:8.10.1, container
atlasmart-redis-ch01, standalone topology, host
publication 127.0.0.1:6379, TLS disabled only
because traffic stays on loopback, default ACL user disabled,
named ACL users atlasmart-app and
academy-admin, logical database 0, AOF with
appendfsync everysec plus RDB snapshots,
persistent /data volume, and no explicit Redis
maxmemory limit or eviction policy. The primary
interface is the redis-cli shipped in the same
pinned image. Mandatory examples use only bounded synthetic
keys under atlasmart:ch07:*. No managed service,
paid broker, or external API is required.
1. Stream storage: append entries, do not mutate rows in place
Think of a Stream as an ordered sequence of immutable entries identified by IDs. Redis lets you delete or trim entries, but normal application flow appends new facts rather than editing an old entry in place. This is different from a Hash record, where fields are repeatedly overwritten.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch07:ordersdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:orders '*' type order-created order 5001 amount-cents 1299docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:orders '*' type payment-authorized order 5001 provider-ref p-77docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:orders '*' type fulfillment-requested order 5001 warehouse BAK-1docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XLEN atlasmart:ch07:ordersdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XRANGE atlasmart:ch07:orders - +
Expected evidence: XLEN is 3 and
XRANGE returns the entries from the lowest to
highest ID. Generated IDs will differ on every run because their
first component normally reflects server time.
2. What an auto-generated ID means
An auto-generated ID has the form
milliseconds-sequence. The first number is normally
Unix time in milliseconds on the Redis server; the second
distinguishes entries generated in the same millisecond. Redis
guarantees that newly inserted IDs increase relative to existing
entries in that stream. If the server clock moves backward,
Redis preserves monotonic stream ordering rather than generating
an ID lower than the current top ID.
| Property | What Redis guarantees | What you must not infer |
|---|---|---|
| Uniqueness | ID is unique within the stream | a globally unique business ID across every stream/system |
| Ordering | IDs increase inside the stream | business causality across independent producers/services |
| Time component | auto IDs normally use server milliseconds | a trusted audit timestamp or exact wall-clock truth |
| Sequence | separates same-millisecond IDs | priority or semantic sequence chosen by the application |
3. Explicit IDs are possible, but constrained
Explicit IDs can align a Stream with another monotonic ID
source, but Redis requires each inserted ID to be greater than
the stream's current top ID. The minimum explicit ID is
0-1. A lower or repeated ID fails rather than
silently reordering history.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch07:explicitdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:explicit 1000-0 event firstdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:explicit 1000-1 event seconddocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XRANGE atlasmart:ch07:explicit - +docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:explicit 999-0 event too-old
The final command should return an error because
999-0 is not greater than the current top ID. That
failure is useful evidence: a Stream is ordered by its IDs, not
by insertion timestamp stored in a field.
4. XRANGE and XREVRANGE expose retained boundaries
XRANGE reads ascending IDs;
XREVRANGE reads descending IDs. A bounded
COUNT is important because an unbounded history
read can create large replies and client memory pressure.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XRANGE atlasmart:ch07:orders - + COUNT 2docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XREVRANGE atlasmart:ch07:orders + - COUNT 2docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XINFO STREAM atlasmart:ch07:orders
XINFO STREAM exposes stream-level metadata such as
length, first/last entries, the last generated ID, and lifetime
entry counters. Treat implementation-detail fields as
version-sensitive; use them for observability, not as an
undocumented API contract.
5. Exact MAXLEN trimming versus approximate trimming
MAXLEN = N enforces an exact length after trimming.
MAXLEN ~ N allows Redis to trim in larger internal
chunks and may retain somewhat more than N.
Approximate trimming is therefore a performance/retention
tradeoff, not a promise that XLEN will equal the
threshold after every append.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch07:exact-capdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:exact-cap MAXLEN = 3 '*' n 1docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:exact-cap MAXLEN = 3 '*' n 2docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:exact-cap MAXLEN = 3 '*' n 3docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:exact-cap MAXLEN = 3 '*' n 4docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XLEN atlasmart:ch07:exact-capdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XRANGE atlasmart:ch07:exact-cap - +
Expected exact evidence: length 3 and the oldest entry is gone.
With ~, the observed length depends on the stream's
internal packing and version; the only safe statement is that
approximate trimming can temporarily keep more than the
requested threshold.
6. MINID retention is a time/order boundary, not a count boundary
MINID trims entries whose IDs are below a
threshold. That is useful when retention is expressed as “keep
entries from this ID onward,” but remember that an
auto-generated Stream ID's millisecond component is not
automatically your business event time.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch07:miniddocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:minid 1000-0 event adocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:minid 2000-0 event bdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:minid 3000-0 event cdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XTRIM atlasmart:ch07:minid MINID = 2000-0docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XRANGE atlasmart:ch07:minid - +
Only IDs at or above 2000-0 remain. If your
retention policy is based on a domain timestamp, either make the
relationship to IDs explicit or store/query the domain timestamp
separately.
7. Retention and consumer-group references are different state
Once consumer groups exist, trimming an entry and removing its
Pending Entries List (PEL) references are separate concerns.
Redis 8.2+ exposes KEEPREF (default),
DELREF, and ACKED policies on
XADD/XTRIM. This chapter will build groups in Lesson 3; for now,
record the key idea: “not retained in the stream body” and “not
referenced by any group” are not synonymous.
| Redis 8.2+ policy | Retention behavior | Operational consequence |
|---|---|---|
KEEPREF |
trim body according to MAXLEN/MINID, keep group PEL references | a pending reference can outlive the body entry |
DELREF |
trim body and remove group references to trimmed entries | recovery metadata is discarded too |
ACKED |
trim only entries acknowledged by all groups | retention can remain above a nominal MAXLEN when references block deletion |
8. Current Redis 8.6+ producer idempotency is separate from consumer delivery
Redis 8.6 introduced IDMP/IDMPAUTO
options for deduplicating producer retries. That can prevent
duplicate Stream entries from ambiguous producer retries, but it
does not turn consumer processing into
exactly-once execution. Consumer redelivery and side-effect
idempotency remain separate problems taught in Lessons 3–5.
9. Wrong approach: “never trim because history is useful”
An unbounded Stream grows with lifetime traffic. That increases memory, persistence/AOF volume, replication traffic, restart/recovery work, and the cost of broad inspections. “Keep everything” is a business retention decision that needs explicit capacity and archive design—not an accidental default.
10. Wrong approach: “trim aggressively because consumers will catch up”
Retention can remove history that lagging consumers, incident responders, or reconciliation jobs still need. Before choosing MAXLEN/MINID, define the maximum replay horizon, worst outage/catch-up duration, PEL behavior, legal/audit requirements, backup/restore needs, and whether another durable system is the system of record.
11. Reproducible retention lab
Create a bounded stream, verify its retained endpoints, then clean up only Chapter 07 keys.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch07:retention-labdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:retention-lab MAXLEN = 4 '*' seq 1docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:retention-lab MAXLEN = 4 '*' seq 2docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:retention-lab MAXLEN = 4 '*' seq 3docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:retention-lab MAXLEN = 4 '*' seq 4docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch07:retention-lab MAXLEN = 4 '*' seq 5docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XLEN atlasmart:ch07:retention-labdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XRANGE atlasmart:ch07:retention-lab - + COUNT 10docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XINFO STREAM atlasmart:ch07:retention-labVerify length 4, the retained first/last IDs, and that sequence 1 is no longer present. This proves body retention only; it says nothing yet about consumer-group recovery state.
12. Production judgment
Use Streams when ordered append history and later consumption/replay matter. Size retention from real recovery objectives; bound range reads; monitor XLEN and retained first/last IDs; include persistence and replica lag in durability analysis; and avoid one global hot Stream if partitioned throughput is required. A Stream is not an external immutable audit log, replication is asynchronous, and trimming must be tested together with consumer-group behavior and restore drills.
13. Summary and next step
Stream entries are ordered persisted records with explicit retention. Next we read them without consumer groups, which makes the caller—not Redis—responsible for tracking its replay position.
Check your understanding
- What are the two components of an auto-generated Stream ID?
- Does MAXLEN ~ 1000 guarantee XLEN is exactly 1000?
- What happens when an explicit XADD ID is lower than the current top ID?
- Does trimming a body entry always remove every consumer-group reference?
- Does Redis 8.6 producer idempotency eliminate consumer redelivery?
Review the answers
A millisecond component and a sequence component.
No. Approximate trimming may retain somewhat more than the threshold.
XADD fails; Redis preserves increasing Stream IDs.
No. In Redis 8.2+ the reference policy is explicit, and KEEPREF is the default.
No. Producer deduplication and consumer processing guarantees are separate concerns.
Authoritative references
- Redis Streams — stream entries, consumer groups, pending entries, acknowledgments, and recovery
- XADD — entry IDs, trimming, Redis 8.2 reference policies, and Redis 8.6 idempotent production options
- XRANGE — ordered range inspection and exclusive continuation IDs
- XREAD — blocking/nonblocking reads, explicit IDs, and the special dollar ID
- XGROUP CREATE — consumer-group starting position and MKSTREAM
- XREADGROUP — group delivery, pending history, and new-message marker
- XPENDING — PEL summary/details, idle time, owner, and delivery count
- XACK — acknowledgment semantics
- XCLAIM — explicit ownership transfer and retry-count behavior
- XAUTOCLAIM — cursor-like stale-pending recovery
- XINFO GROUPS — pending count, last-delivered ID, entries-read, and lag
- XTRIM — exact/approximate retention and PEL reference policies
- Redis 8.10 release notes — pinned server release family