Use blocking list operations without confusing efficient waiting with acknowledged, replayable delivery.
Blocking List Operations and Reliable Queue Limitations Compared with Streams
Choose hashes, Redis JSON, or many keys from structure, update/query patterns, expiration granularity, memory/cardinality, indexing, ACL boundaries, and migration evidence.
Learning outcomes
AtlasMart now wants workers to wait for fulfillment jobs instead of polling every few milliseconds. Blocking list commands can efficiently wait, but destructive pop semantics create a failure window: a worker can receive a job and disappear before completing it. This lesson distinguishes blocking from reliable delivery.
Explain BLPOP/BRPOP blocking, timeout, multi-key priority, and connection behavior.
Use BLMOVE to transfer work to a processing list and explain what reliability state is still missing.
Reproduce the lost-work window of destructive list popping in a disposable lab.
Contrast lists with Streams consumer groups, pending entries, acknowledgments, and replay/history.
Design client timeouts, retries, idempotency, persistence, and recovery around the actual guarantees.
All Chapter 05 mandatory labs reuse the disposable Chapter 01
environment: Redis Open Source 8.10.1 from Docker
Official Image redis:8.10.1, container
atlasmart-redis-ch01, standalone topology, host
publication 127.0.0.1:6379, TLS disabled only
because traffic stays on loopback, default ACL user disabled,
named ACL users atlasmart-app and
academy-admin, logical database 0, AOF with
appendfsync everysec plus RDB snapshots,
persistent /data volume, and no explicit Redis
maxmemory limit or eviction policy. The
application ACL is restricted to ~atlasmart:* and
normal read/write/connection categories. The primary interface
is the redis-cli shipped in the same 8.10.1
image, so server and CLI versions stay aligned. Mandatory
examples use only bounded synthetic keys under
atlasmart:ch05:*.
1. Blocking means “wait on this connection”
BLPOP and BRPOP wait when all supplied
list keys are empty. A non-zero timeout bounds the wait; zero
means wait indefinitely. If several keys are already non-empty,
Redis checks them left-to-right and returns from the first
non-empty key. A blocked connection is dedicated to that wait
until data, timeout, disconnect, or another terminating
condition releases it.
# Terminal A: wait up to 5 seconds.docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app BLPOP atlasmart:ch05:queue:priority atlasmart:ch05:queue:normal 5# Terminal B, before the 5 seconds expires:docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app RPUSH atlasmart:ch05:queue:normal job:3001# Terminal A should return the key plus job:3001.When multiple clients are blocked on the same key, Redis documents service ordering based on waiting time for that key. Do not turn that narrow rule into a claim of global fairness across keys, workers, retries, or heterogeneous workloads.
2. A destructive pop creates a processing gap
BLPOP removes the item before your application
performs its business side effect. If the worker crashes after
the pop but before durable completion, Redis has no
pending-entry record for that item. The queue can be empty even
though the order was never fulfilled.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch05:queue:unsafedocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app RPUSH atlasmart:ch05:queue:unsafe job:lost-demodocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app BLPOP atlasmart:ch05:queue:unsafe 1docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app LLEN atlasmart:ch05:queue:unsafe
Interpretation: LLEN = 0 proves the queue no longer
contains the job. It does not prove the worker completed
anything. The deliberate failure is simply “pretend the worker
crashed now”; no host process killing is needed.
3. BLMOVE narrows the loss window but does not create acknowledgments
BLMOVE source destination LEFT RIGHT timeout
atomically removes one element from the source and inserts it
into the destination. That supports a classic processing-list
pattern: jobs are no longer invisible after acquisition. Redis
6.2 introduced BLMOVE as the modern replacement for
the now-deprecated BRPOPLPUSH pattern.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch05:queue:safe-ish atlasmart:ch05:processingdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app RPUSH atlasmart:ch05:queue:safe-ish job:4001 job:4002docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app BLMOVE atlasmart:ch05:queue:safe-ish atlasmart:ch05:processing LEFT RIGHT 1docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app LRANGE atlasmart:ch05:queue:safe-ish 0 -1docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app LRANGE atlasmart:ch05:processing 0 -1docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app LREM atlasmart:ch05:processing 1 job:4001
The final LREM is an application-defined
acknowledgment. Redis Lists do not attach delivery timestamp,
consumer identity, retry count, pending ownership, or reclaim
policy. You must build those mechanisms yourself, and your
design must tolerate duplicate processing if recovery races with
a slow original worker.
4. Streams add server-side delivery state
A Redis Stream is an append-oriented data
structure whose entries have IDs. A
consumer group tracks what has been delivered
and maintains a Pending Entries List (PEL) for
delivered-but-unacknowledged entries. Consumers acknowledge
successful processing with XACK; pending entries
can be inspected with XPENDING and reassigned with
XCLAIM/XAUTOCLAIM. That is materially
different from a List.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch05:stream:jobsdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XADD atlasmart:ch05:stream:jobs "*" job job:5001docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XGROUP CREATE atlasmart:ch05:stream:jobs workers 0 MKSTREAMdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XREADGROUP GROUP workers worker-a COUNT 1 STREAMS atlasmart:ch05:stream:jobs ">"docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app XPENDING atlasmart:ch05:stream:jobs workers
Stop before XACK and XPENDING should
show one pending entry. That makes the unfinished delivery
observable. It still does not provide exactly-once business
effects: a worker can complete an external side effect and fail
before acknowledging, so the entry may be delivered again.
Idempotency remains an application responsibility.
5. List queue vs Stream consumer group
| Requirement | List / blocking list | Stream + consumer group |
|---|---|---|
| Wait without polling | Yes: BLPOP/BRPOP/BLMOVE | Yes: XREAD/XREADGROUP BLOCK |
| Entry retained after read | No for pop; manual processing-list pattern possible | Yes until trimmed/deleted |
| Server-side pending ownership | No | Yes, consumer-group PEL |
| Explicit acknowledgment | No built-in list ack | Yes, XACK |
| Replay/history | Only what remains in lists | Stream IDs/ranges plus group history |
| Claim abandoned work | Application-built | XCLAIM/XAUTOCLAIM |
| Exactly once | No | No; design idempotency/reconciliation |
6. Client timeout and disconnect behavior are part of correctness
Blocking forever can complicate shutdown, connection-pool health checks, and failover. A finite blocking timeout lets a worker periodically check cancellation, topology changes, or deployment state. Client socket timeout must exceed the intended Redis blocking timeout or be configured in a client-specific blocking-safe way; otherwise the client may disconnect before Redis returns. Retries must account for ambiguity: after a network break, the client may not know whether a move/pop happened.
Use a dedicated blocking connection or a client feature documented for blocking commands. Do not assume an ordinary pooled request connection can safely carry an indefinitely blocked operation while other requests share it.
7. Persistence and replication do not close the processing gap
AOF/RDB can recover Redis data according to their durability windows, but a worker may have removed/moved a job and changed an external system in a different failure order. Redis replication is asynchronous and failover can expose additional ambiguity. End-to-end reliability therefore needs idempotent job identities, durable business state, reconciliation, and explicit retry/dead-letter policy—not merely “Redis persistence is on.”
8. Hands-on failure/recovery drill
Use two list keys so recovery is visible without killing processes.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch05:q atlasmart:ch05:processingdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app RPUSH atlasmart:ch05:q job:6001 job:6002docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app BLMOVE atlasmart:ch05:q atlasmart:ch05:processing LEFT RIGHT 1docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app LRANGE atlasmart:ch05:processing 0 -1# Simulate worker disappearance by NOT acknowledging/removing the processing entry.docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app LMOVE atlasmart:ch05:processing atlasmart:ch05:q LEFT LEFTdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app LRANGE atlasmart:ch05:q 0 -1The move-back step is intentionally simplistic; real systems need ownership/time metadata and duplicate-safe processing. Its purpose is to expose how much machinery a list queue leaves to the application compared with Streams.
9. Production judgment
Blocking Lists fit small, simple work queues when loss/retry semantics are acceptable or when the application already owns a robust processing-list protocol. Prefer Streams when you need retained entries, consumer groups, pending inspection, acknowledgment, recovery of abandoned work, or multiple independent consumption views. External brokers may be a better fit when cross-region durability, very long retention, ecosystem integrations, partitioned throughput, or broker-specific guarantees dominate.
10. Summary and next step
Blocking eliminates polling; it does not automatically add
reliability. BLMOVE can retain acquired work in
another list, but Streams provide explicit server-side delivery
state. Next, Sets switch the problem entirely—from ordered
delivery to unique membership.
Check your understanding
- What does BLPOP timeout 0 mean?
- Why can BLPOP lose work from the queue even with AOF enabled?
- What reliability improvement does BLMOVE provide?
- What does a Stream consumer group PEL track?
- Why can Streams still deliver a business operation more than once?
Review the answers
Block indefinitely until an element becomes available or the connection/operation is otherwise terminated.
The pop removes the item before application processing; persistence can faithfully persist that removal even if the worker crashes before finishing.
It atomically moves the item into another list, so acquired work can remain visible for application-defined recovery.
Delivered but unacknowledged entries and their consumer ownership/history metadata.
A worker can complete the external effect and fail before XACK, making re-delivery necessary; consumers must be idempotent.
Authoritative references
- Redis Open Source 8.10 release notes — 8.10.1 security baseline and 8.10 list/set additions
- Redis lists — list semantics and common queue patterns
- Redis sets — set uniqueness, membership, and algebra
- Redis Streams — persistent entries, consumer groups, pending entries, and acknowledgments
- Redis 8.10 command reference — current command syntax and complexity
- Redis Cluster specification — hash slots and multi-key locality
- BLPOP — blocking timeout, multi-key priority, and client ordering
- BLMOVE — atomic blocking move and BRPOPLPUSH replacement
- XPENDING — consumer-group pending-entry observability