Values as Bytes, Documents, or Data Structures: Modeling Beyond Simple Strings

Learning outcomes

AtlasMart initially stores every value as a JSON blob. That is simple until two checkout workers decrement stock concurrently: both deserialize stock=10, both write stock=9, and one decrement disappears. A richer server-side operation can move the critical read-modify-write boundary into the store. This lesson separates value representation from operation semantics.

1. Opaque bytes maximize freedom—and move responsibility to the application

An opaque value may be raw bytes, UTF-8 text, MessagePack, Protocol Buffers, Avro, JSON, or a custom encoding. The store can often fetch/replace it without understanding its fields. This makes the storage contract simple but pushes schema evolution, validation, partial updates, compatibility, and race prevention into application code.

An explicit envelope such as {"schema":2,...} gives readers a deterministic upgrade path. Without a version or self-describing format, a deploy can write data that an older rollback binary cannot parse. “Flexible schema” therefore does not mean “no schema”; the schema simply lives in application/serialization rules instead of a rigid table definition.

2. Structured values let operations move closer to the state

A map/hash can update one field without transferring/replacing an entire document. A set can enforce unique membership. A sorted set associates unique members with scores for rank/range operations. A list represents an ordered sequence. A counter offers atomic numeric change. A stream or append-oriented sequence records ordered entries for later consumption. These structures are useful when the server can perform the exact state transition atomically at the required boundary.

3. Failure case: client-side read-modify-write loses an update

Two clients fetch the same JSON inventory blob. Each decrements from 10 to 9 and replaces the entire value. The second replacement is valid in isolation, but the combined history should have produced 8. This is the classic lost-update mechanism: the client assumed that read + local computation + set was one atomic operation when it was actually three separate steps.

Possible repairs include a native atomic decrement, a server-side function/transaction, compare-and-set (CAS) on a version, or redesigning the invariant. The right repair depends on whether negative inventory is forbidden, whether multiple warehouses are independent, and whether stronger coordination is acceptable.

4. Schema evolution is an operational compatibility problem

When a byte/document value evolves, record what old readers do with new fields, what new readers do with missing fields, whether field types can change, how backfills happen, and whether rollback is possible. A version envelope can support lazy migration on read or explicit migration jobs. The migration must remain idempotent because retries and partial progress are normal distributed-system conditions.

5. AtlasMart lab: bytes, structures, and server-side atomicity

The lab implements a toy in-process data-structure store. It deliberately loses an inventory decrement with JSON bytes, then performs two atomic field decrements correctly. It also demonstrates a set, scored structure, list, stream-like append, and a schema-v1-to-v2 conversion.

import json
from copy import deepcopy

class ToyDataStructureStore:
    def __init__(self):
        self.bytes = {}
        self.hashes = {}
        self.sets = {}
        self.sorted_sets = {}
        self.lists = {}
        self.streams = {}

    def set_bytes(self, key, value: bytes): self.bytes[key] = bytes(value)
    def get_bytes(self, key): return self.bytes.get(key)
    def hset(self, key, **fields): self.hashes.setdefault(key, {}).update(fields)
    def hincrby(self, key, field, delta):
        h = self.hashes.setdefault(key, {})
        h[field] = int(h.get(field, 0)) + delta
        return h[field]
    def sadd(self, key, *members): self.sets.setdefault(key, set()).update(members)
    def zadd(self, key, member, score): self.sorted_sets.setdefault(key, {})[member] = score
    def lpush(self, key, value): self.lists.setdefault(key, []).insert(0, value)
    def xadd(self, key, fields):
        stream = self.streams.setdefault(key, [])
        entry_id = len(stream) + 1
        stream.append((entry_id, dict(fields)))
        return entry_id

store = ToyDataStructureStore()

print("OPAQUE BYTES + LOST UPDATE")
store.set_bytes("inventory:sku-7", json.dumps({"schema": 1, "stock": 10}).encode())
a = json.loads(store.get_bytes("inventory:sku-7"))
b = json.loads(store.get_bytes("inventory:sku-7"))
a["stock"] -= 1
b["stock"] -= 1
store.set_bytes("inventory:sku-7", json.dumps(a).encode())
store.set_bytes("inventory:sku-7", json.dumps(b).encode())
print("expected stock 8, actual", json.loads(store.get_bytes("inventory:sku-7"))["stock"])

print("\nSERVER-SIDE FIELD/COUNTER OPERATION")
store.hset("inventory-hash:sku-7", schema=2, stock=10, warehouse="W1")
print("after decrement 1 ->", store.hincrby("inventory-hash:sku-7", "stock", -1))
print("after decrement 2 ->", store.hincrby("inventory-hash:sku-7", "stock", -1))
print("hash state ->", store.hashes["inventory-hash:sku-7"])

print("\nRICH STRUCTURES")
store.sadd("product:sku-7:tags", "sale", "blue", "summer")
store.zadd("product:popularity", "sku-7", 98.5)
store.zadd("product:popularity", "sku-8", 76.0)
store.lpush("user:42:recent", "sku-7")
store.lpush("user:42:recent", "sku-8")
eid = store.xadd("inventory:events", {"sku": "sku-7", "delta": -1})
print("tags", sorted(store.sets["product:sku-7:tags"]))
print("leaderboard", sorted(store.sorted_sets["product:popularity"].items(), key=lambda x: -x[1]))
print("recent", store.lists["user:42:recent"])
print("stream entry", eid, store.streams["inventory:events"][-1])

print("\nSCHEMA EVOLUTION ENVELOPE")
v1 = {"schema": 1, "name": "Ada", "locale": "en"}
blob = json.dumps(v1).encode()
loaded = json.loads(blob)
if loaded["schema"] == 1:
    loaded = {"schema": 2, "display_name": loaded["name"], "locale": loaded["locale"], "marketing_opt_in": False}
print("migrated document", loaded)

OPAQUE BYTES + LOST UPDATE
expected stock 8, actual 9

SERVER-SIDE FIELD/COUNTER OPERATION
after decrement 1 -> 9
after decrement 2 -> 8
hash state -> {'schema': 2, 'stock': 8, 'warehouse': 'W1'}

RICH STRUCTURES
tags ['blue', 'sale', 'summer']
leaderboard [('sku-7', 98.5), ('sku-8', 76.0)]
recent ['sku-8', 'sku-7']
stream entry 1 (1, {'sku': 'sku-7', 'delta': -1})

SCHEMA EVOLUTION ENVELOPE
migrated document {'schema': 2, 'display_name': 'Ada', 'locale': 'en', 'marketing_opt_in': False}

Interpretation

The result does not prove that every hash/counter/stream product has identical atomicity or replication semantics. It proves a narrower mechanism: when the operation executes atomically where the state lives, the application can avoid a client-side race that whole-value replacement creates.

6. Production judgment

Bridge to Lesson 3: Rich operations become especially useful for ephemeral state: TTL, counters, conditional creation, and CAS. But TTL and eviction have different semantics, and memory pressure can remove data before its TTL.

Authoritative references

• Redis data types — current concrete example of bytes and rich server-side data structures
• Redis — Compare data types — operation-oriented selection guidance
• Redis strings — example of byte/string values and atomic counter operations
• Redis transactions / WATCH — current concrete example of optimistic check-and-set behavior