Design rate-limit, scheduler, expiration, and secondary sorted-set indexes with explicit truth, cleanup, reliability, topology, and rollback boundaries.
Use Sorted Sets for Rate Limits, Schedulers, Expiration Indexes, and Secondary Access Paths
Choose hashes, Redis JSON, or many keys from structure, update/query patterns, expiration granularity, memory/cardinality, indexing, ACL boundaries, and migration evidence.
Learning outcomes
AtlasMart's architecture now contains several sorted-set patterns. The final lesson turns them into an operational design discipline: a rate-limit window, a due-time scheduler index, an explicit expiration index, and a secondary access path. The key lesson is that the sorted set is often an index; correctness still depends on the authoritative state, cleanup, atomic transitions, and failure recovery.
Design sorted-set rate-limit, scheduler, expiration-index, and secondary-access-path models.
Distinguish discovery/order from ownership, acknowledgment, deletion, and business truth.
Use bounded range reads and explicit cleanup to control growth.
Identify race conditions between primary data and secondary indexes and plan reconciliation.
Assess hot-key, Cluster, persistence, retry, memory, latency, and migration implications.
All Chapter 06 mandatory labs reuse the disposable Chapter 01
environment: Redis Open Source 8.10.1 from Docker
Official Image redis:8.10.1, container
atlasmart-redis-ch01, standalone topology, host
publication 127.0.0.1:6379, TLS disabled only
because traffic stays on loopback, default ACL user disabled,
named ACL users atlasmart-app and
academy-admin, logical database 0, AOF with
appendfsync everysec plus RDB snapshots,
persistent /data volume, and no explicit Redis
maxmemory limit or eviction policy. The primary
interface is the redis-cli shipped in the same
pinned image. Mandatory examples use only bounded synthetic
keys under atlasmart:ch06:*.
1. Pattern anatomy: member identity + ordering score + lifecycle
| Pattern | Member | Score | Authoritative truth |
|---|---|---|---|
| Sliding-window rate limit | unique request ID | request timestamp | application/API policy + request outcome |
| Delayed scheduler | job ID | due timestamp | job/workflow record |
| Expiration index | object ID | desired expiry timestamp | object/retention policy |
| Secondary time index | entity ID | created/updated timestamp | entity record |
2. Rate-limit index: prune, add, count, decide
A per-subject sorted set can represent requests within a moving horizon. The design must define whether the current request is added before or after checking, inclusive boundaries, duplicate request IDs, and how concurrent clients make one decision atomically.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch06:ratelimit:user7docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZADD atlasmart:ch06:ratelimit:user7 100 req:a 120 req:b 170 req:cdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZREMRANGEBYSCORE atlasmart:ch06:ratelimit:user7 -inf 110docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZCOUNT atlasmart:ch06:ratelimit:user7 111 170docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZRANGE atlasmart:ch06:ratelimit:user7 0 -1 WITHSCORESThis demonstrates data mechanics only. A production limiter should combine the race-prone multi-command decision atomically and define TTL/cleanup for idle subject keys; Chapter 23 does that design explicitly.
3. Scheduler index: discovery is not delivery
For scheduled work, query members with score ≤ now. After discovery, the system needs a claim state, retry deadline, idempotency key, completion state, and recovery path. A sorted set provides ordering; it does not provide a Pending Entries List like Streams.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch06:schedulerdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZADD atlasmart:ch06:scheduler 1000 job:a 2000 job:b 3000 job:cdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZRANGE atlasmart:ch06:scheduler -inf 2100 BYSCORE LIMIT 0 10 WITHSCORESdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZCARD atlasmart:ch06:schedulerIf you remove a job when it is claimed, maintain durable processing state elsewhere or in another explicit structure. If you leave it in place, prevent multiple workers from acting concurrently. Either way, “ZRANGE found it” is not an acknowledgment protocol.
4. Expiration index: TTL and business retention are different mechanisms
An explicit sorted-set expiration index is useful when an application must discover objects by planned retention time, perform custom archival/deletion, or coordinate cleanup across data stores. Redis key TTL can remove a Redis key automatically; a sorted-set expiration index is an application-managed schedule and should not be presented as an exact timer.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch06:expiry-indexdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZADD atlasmart:ch06:expiry-index 100 customer:temp1 150 customer:temp2 500 customer:temp3docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZRANGE atlasmart:ch06:expiry-index -inf 200 BYSCORE WITHSCORESdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZREMRANGEBYSCORE atlasmart:ch06:expiry-index -inf 200docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZCARD atlasmart:ch06:expiry-indexThe removal above only removes index members. It does not delete the authoritative customer records. A cleanup worker must perform and verify the actual retention action, then reconcile the index.
5. Secondary access paths need reconciliation
Suppose atlasmart:order:9001 is authoritative and
atlasmart:ch06:orders-by-time is a secondary
sorted-set index. A failure can happen after one is updated but
before the other. That makes the index stale even though every
Redis command involved was individually atomic.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch06:primary:9001 atlasmart:ch06:orders-by-timedocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SET atlasmart:ch06:primary:9001 paiddocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZADD atlasmart:ch06:orders-by-time 100 order:9001docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch06:primary:9001docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZRANGE atlasmart:ch06:orders-by-time 0 -1 WITHSCORESThe index still contains order:9001 after the synthetic primary key is deleted. Repair requires transactional/function coupling where feasible or a reconciliation process that can detect and remove stale index entries.
6. Wrong approach: one global hot key for every tenant
A global leaderboard, scheduler, or limiter can become a write/read hot key even when each operation has good asymptotic complexity. In Redis Cluster, one key maps to one slot and one primary. A better partition key may be tenant, region, time bucket, or workload class—provided cross-partition queries and correctness requirements remain manageable.
7. Cleanup strategy is part of capacity planning
For every sorted-set index, write down the maximum expected live
members, insert rate, cleanup cadence, maximum catch-up delete,
largest range response, and behavior after a cleanup outage.
Measure ZCARD, MEMORY USAGE, command
latency, and application response bytes. No universal “safe
sorted-set size” replaces workload measurement.
8. Persistence, replication, and retries
AOF/RDB persistence can recover the Redis index according to configured durability windows; asynchronous replication can fail over with some uncertainty. Neither proves that an external email was sent, an order was charged, or an object was deleted. Retried clients may also face ambiguous outcomes after network loss. Design idempotent operations and reconciliation around stable business IDs.
9. Cluster locality and hash tags
A single sorted set needs no multi-key locality, but workflows that atomically combine index and state keys or use multi-key ZUNION/ZINTER need Cluster-aware design. Hash tags can co-locate related keys, but overusing one tag creates a hot slot. Chapter 21 will revisit slot calculation, MOVED/ASK routing, and resharding; this chapter only requires you to record the locality assumption.
10. Search/index boundary
Sorted Sets are excellent one-dimensional numeric/lexicographic access paths. They are not a general substitute for Redis Search indexes over HASH/JSON documents. If AtlasMart needs filters such as category + stock + text relevance + geography, forcing those predicates into composite sorted-set members creates brittle application-side indexing. Chapter 09 covers Search schemas and their memory/write costs.
11. Operational checklist
| Concern | Question to answer before production |
|---|---|
| Correctness | What is authoritative, and what can be stale? |
| Ordering | What exactly does the score mean? How are ties handled? |
| Atomicity | Which multi-command transitions may interleave? |
| Retention | How are old members removed? What if cleanup stops? |
| Reliability | What survives worker/client failure? How is work retried? |
| Capacity | Expected cardinality, skew, memory, response sizes, p99 latency? |
| Topology | Standalone/replication/Sentinel/Cluster; required slot locality? |
| Security | Which ACL key patterns/commands are allowed? TLS in production? |
| Migration | How is a new index backfilled, verified, cut over, and rolled back? |
12. Reproducible capstone lab
Build three bounded indexes, inspect them, then clean only Chapter 06 fixtures.
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch06:cap:window atlasmart:ch06:cap:due atlasmart:ch06:cap:expirydocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZADD atlasmart:ch06:cap:window 10 r:1 20 r:2 30 r:3docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZREMRANGEBYSCORE atlasmart:ch06:cap:window -inf 10docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZADD atlasmart:ch06:cap:due 100 j:1 200 j:2docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZRANGE atlasmart:ch06:cap:due -inf 150 BYSCORE WITHSCORESdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZADD atlasmart:ch06:cap:expiry 50 x:1 500 x:2docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZRANGE atlasmart:ch06:cap:expiry -inf 100 BYSCORE WITHSCORESdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app ZCARD atlasmart:ch06:cap:windowdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app MEMORY USAGE atlasmart:ch06:cap:window
Verification: the window retains r:2/r:3, j:1 is due at 150, x:1
is selected for expiration work, and no command touches
unrelated keys. Cleanup with
DEL atlasmart:ch06:cap:window atlasmart:ch06:cap:due
atlasmart:ch06:cap:expiry.
13. Migration and rollback judgment
When introducing a sorted-set secondary index, backfill from authoritative data, compare counts and sampled records, dual-write only with an explicit failure/reconciliation plan, cut reads over gradually, and keep a rollback path that does not depend on deleting the old source immediately. For a ranking-policy change, version the destination key or policy so old and new scores can be compared before replacement.
14. Production judgment
Sorted Sets are a strong primitive for score-ordered unique membership and bounded range queries. They are only one component of rate limiters, schedulers, retention workflows, and secondary indexing systems. Production readiness requires atomic transitions, idempotency, reconciliation, capacity/headroom, hot-key strategy, Cluster locality, security, monitoring, failure injection, and rollback—not just correct redis-cli syntax.
15. Summary and bridge to Streams
Chapter 06 established score/rank semantics, safe range queries, time-based indexes, weighted aggregation, cleanup, and operational boundaries. Chapter 07 introduces Redis Streams, where append IDs, retained entries, consumer groups, pending ownership, acknowledgment, and replay solve a different class of event/work-delivery problem.
Check your understanding
- What does a sorted-set scheduler natively provide?
- Does removing an item from an expiration index delete the underlying object?
- Why can a secondary index become stale?
- What is the main risk of one global scheduler key in Cluster?
- When should Redis Search replace a sorted-set access path?
Review the answers
Numeric ordering and bounded discovery by due score; ownership/ack/retry are separate.
No. It only removes the index member.
Primary and index updates can fail/interleave independently unless coupled and reconciled.
One key maps to one slot/primary and can become a hot key.
When the query requires richer multidimensional/text/structured predicates rather than one score/lex ordering.
Authoritative references
- Redis sorted sets — data model, score/rank behavior, and common patterns
- ZADD — conditional updates, score precision, and return behavior
- ZRANGE — rank, BYSCORE, BYLEX, REV, LIMIT, and WITHSCORES
- ZINCRBY — relative score updates
- ZUNION — weighted unions and aggregation
- ZINTER — weighted intersections and complexity
- ZDIFF — sorted-set difference semantics
- Redis Cluster specification — slot locality for multi-key commands
- Redis Open Source 8.10 release notes — pinned course release family
- MEMORY USAGE — per-key memory observation
- Redis key expiration — TTL is distinct from custom expiration indexing