Big Data AcademyQuery · move · process · govern

Model unique membership and deduplication with Sets while keeping randomness, lifecycle, and exactly-once claims within their real boundaries.

Sets: SADD/SREM/SISMEMBER, Cardinality, Random Members, and Deduplication

Choose hashes, Redis JSON, or many keys from structure, update/query patterns, expiration granularity, memory/cardinality, indexing, ACL boundaries, and migration evidence.

Beginner → Intermediate125–160 minutesSet membership + dedupe labRedis Open Source 8.10.1Free/local-firstLast reviewed: September 6, 2026

Learning outcomes

AtlasMart needs unique coupon recipients, product tags, and deduplication markers. A Redis Set is an unordered collection of unique binary-safe string members. Uniqueness and constant-time-style membership checks are the central abstraction; ordering and fairness are not.

01

Use SADD/SREM/SISMEMBER/SCARD and interpret their integer results precisely.

02

Design deduplication keys with explicit lifecycle rather than unbounded growth.

03

Use SRANDMEMBER/SPOP without promising order, fairness, or guaranteed uniform sampling.

04

Avoid SMEMBERS on huge sets by using bounded/incremental access where appropriate.

05

Connect membership semantics to memory, persistence, Cluster hot keys, and authorization freshness.

Exact lab baseline

All Chapter 05 mandatory labs reuse the disposable Chapter 01 environment: Redis Open Source 8.10.1 from Docker Official Image redis:8.10.1, container atlasmart-redis-ch01, standalone topology, host publication 127.0.0.1:6379, TLS disabled only because traffic stays on loopback, default ACL user disabled, named ACL users atlasmart-app and academy-admin, logical database 0, AOF with appendfsync everysec plus RDB snapshots, persistent /data volume, and no explicit Redis maxmemory limit or eviction policy. The application ACL is restricted to ~atlasmart:* and normal read/write/connection categories. The primary interface is the redis-cli shipped in the same 8.10.1 image, so server and CLI versions stay aligned. Mandatory examples use only bounded synthetic keys under atlasmart:ch05:*.

1. SADD proves whether a member was new

SADD adds only members not already present and returns the count of newly added members. The member itself is a binary-safe string; Redis does not interpret JSON, numbers, or case-insensitive identity unless your application normalizes them first.

redis-cli · uniqueness evidence 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch05:coupon:fall:recipientsdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SADD atlasmart:ch05:coupon:fall:recipients customer:101 customer:102 customer:102docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SCARD atlasmart:ch05:coupon:fall:recipientsdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SISMEMBER atlasmart:ch05:coupon:fall:recipients customer:102docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SISMEMBER atlasmart:ch05:coupon:fall:recipients Customer:102

The first SADD returns 2, not 3. The capitalized member is distinct. If identity should be case-insensitive, normalize before constructing the Redis member.

2. SREM and key lifecycle

SREM removes named members and returns how many were actually removed. When the last member is removed, the set key disappears. SCARD reports current set cardinality in O(1), which is useful for growth monitoring without transferring members.

redis-cli · removal and cardinality 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SREM atlasmart:ch05:coupon:fall:recipients customer:999 customer:101docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SCARD atlasmart:ch05:coupon:fall:recipientsdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app TYPE atlasmart:ch05:coupon:fall:recipients

A deduplication set is not self-cleaning unless you design expiration/removal. If every request ID is retained forever, cardinality becomes an operational incident rather than a correctness feature. Chapter 02's TTL rules apply to the set key as a whole.

3. Deduplication is a membership decision, not exactly-once execution

A common pattern is “if SADD returns 1, process; if 0, duplicate.” The membership mutation is atomic, but the surrounding business effect may not be. If the process crashes after adding the marker and before completing the order, a retry may be suppressed incorrectly. If it performs the effect before adding the marker, a crash can allow duplicate effects.

redis-cli · dedupe marker state 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch05:dedupe:paymentsdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SADD atlasmart:ch05:dedupe:payments payment:req-7001docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SADD atlasmart:ch05:dedupe:payments payment:req-7001docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SCARD atlasmart:ch05:dedupe:payments

Repair: define an idempotency state machine tied to durable business state, with retry/reconciliation rules. A Set can be a fast membership component; it is not a distributed transaction with your payment processor.

4. Random member commands are not scheduling fairness

SRANDMEMBER returns random members without removing them. SPOP returns and removes random members. Redis explicitly warns not to use SPOP when you require a guaranteed uniform distribution. “Random” therefore cannot be promoted to “fair,” “round robin,” or “statistically guaranteed” without measurement and an appropriate algorithm.

redis-cli · observe nondeterministic selection 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch05:stores:sampledocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SADD atlasmart:ch05:stores:sample store:1 store:2 store:3 store:4 store:5docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SRANDMEMBER atlasmart:ch05:stores:sample 3docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SCARD atlasmart:ch05:stores:sampledocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SPOP atlasmart:ch05:stores:sample 2docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SCARD atlasmart:ch05:stores:sample

Repeated runs may differ. The output order of Sets is not a stable ordering contract. If weighted or ordered scheduling matters, a Sorted Set is usually closer to the requirement.

5. Large-set discovery: cardinality first, then bounded iteration

SMEMBERS returns the entire set and is O(N). For a large operational set, first inspect SCARD; if you genuinely need enumeration, use SSCAN incrementally and design the consumer for cursor semantics similar to Chapter 02's SCAN discussion. Do not use Set output order as pagination state.

6. Deliberately wrong: a Set as an ordered fair queue

Suppose an engineer uses SPOP to choose the “next” store and assumes every store will be selected fairly over time. The data structure guarantees uniqueness of stored members, not a fair service discipline. A destructive random pop also permanently removes the member unless the application reinserts it.

redis-cli · show why order is absent 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch05:worker-pooldocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SADD atlasmart:ch05:worker-pool worker:a worker:b worker:cdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SMEMBERS atlasmart:ch05:worker-pooldocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SRANDMEMBER atlasmart:ch05:worker-pooldocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SRANDMEMBER atlasmart:ch05:worker-pool

Repair: use a queue for ordered service, a Sorted Set for priority/score scheduling, or an application scheduler with explicit fairness policy. Keep the Set for “is this worker eligible?” membership.

7. Memory, security, and topology boundaries

One huge Set is one hot key and one Cluster slot. Membership commands are efficient, but cardinality and member size still consume memory. ACL key patterns can restrict which set keys a user may access, but Set namespaces are not tenant isolation by themselves. For authorization, consider staleness: a cached permission Set can be fast yet wrong after a policy change unless invalidation/versioning is designed and tested.

8. Hands-on lab: coupon audience and bounded dedupe

Create an audience and a dedupe marker set, then record cardinality and memory.

redis-cli · set evidence card 
docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app DEL atlasmart:ch05:audience:vip atlasmart:ch05:dedupe:emaildocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SADD atlasmart:ch05:audience:vip customer:1 customer:2 customer:3 customer:3docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SISMEMBER atlasmart:ch05:audience:vip customer:2docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SCARD atlasmart:ch05:audience:vipdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app MEMORY USAGE atlasmart:ch05:audience:vipdocker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app SADD atlasmart:ch05:dedupe:email send:9001docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app EXPIRE atlasmart:ch05:dedupe:email 3600docker exec -e REDISCLI_AUTH=AtlasMart-App-Lab-Only-2026 atlasmart-redis-ch01 redis-cli --user atlasmart-app TTL atlasmart:ch05:dedupe:email

The TTL is on the entire dedupe Set key, not individual members. If per-member lifecycle matters, model it explicitly with another structure/key strategy.

9. Production judgment

Use Sets when unique membership is the primary query: tags, audiences, capability flags, dedupe membership, adjacency candidates, or permission membership with explicit freshness. Monitor cardinality and memory, bound lifecycle, avoid full enumeration for huge sets, and never infer ordering/fairness. For expensive relationships among multiple Sets, measure algebra cost before putting it on a hot request path.

10. Summary and next step

Sets give exact uniqueness and membership, but no ordering. Next, Redis can combine Sets with union, intersection, and difference—powerful operations whose cost and Cluster locality must be part of the design.

Check your understanding

  1. What does SADD return when every supplied member already exists?
  2. Does SCARD transfer the members?
  3. Why is a Set dedupe marker not enough for exactly-once payment processing?
  4. Can SRANDMEMBER be treated as fair scheduling?
  5. What lifecycle problem appears in an unbounded dedupe Set?
Review the answers

Zero, because no new members were added.

No. It returns cardinality only.

The atomic marker and the external business effect are separate failure domains and can be committed in different orders.

No. Random selection is not a fairness guarantee, and Redis warns about distribution guarantees for destructive SPOP.

Memory/cardinality grows without bound and old markers may suppress valid future work unless retention semantics are defined.

Authoritative references

Keep knowledge open

Help the academy stay free and grow.

If these tutorials save you time, a small donation supports new lessons, technical review, diagrams, examples, and long-term maintenance.

ETHEthereum / ERC-20 only
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0

Send only Ethereum or ERC-20 compatible assets to this address.