Benchmark Workload Shapes, Not Marketing Numbers: Keys, Values, Zipf Skew, Concurrency, and Durability

1. “100,000 operations per second” is not a workload

Performance numbers are conditional statements. AtlasMart needs to know whether those operations are 1 KiB point reads or 2 MiB writes, whether keys are uniform or Zipf-like, whether reads hit cache, whether writes fsync, whether three replicas acknowledge, whether indexes are maintained, whether the dataset fits RAM, and whether a node is failed or rebuilding. Change any of those and the comparison may answer a different question.

A benchmark contract freezes the material variables: engine/version/edition, client/driver, host/container/cloud shape, node topology and failure domains, dataset cardinality and bytes, key/value sizes, read/write/scan mix, request distribution, concurrency/offered rate, batch size, consistency/ack level, durability/fsync behavior, indexes, replication factor, compression, warmup, runtime duration, and active maintenance/failure state.

2. Uniform traffic and Zipf-like traffic exercise different systems

Uniform selection gives keys roughly equal probability. A Zipf-like distribution assigns much more probability to the most popular ranks, resembling catalogs, social objects, or “celebrity” keys. Skew can improve cache locality for hot reads but overload one shard, increase lock/conditional-write contention, or create one partition whose replicas dominate network and disk work. Therefore a benchmark that hashes uniformly can miss the production bottleneck created by the real access distribution.

3. Deliberately wrong approach: publish the larger ops/sec number and omit request distribution

The deterministic lab holds operation count, record count, value size, nominal concurrency, replication, and ack policy constant. Only key distribution changes. Uniform traffic has a ~1.1% top-key share; the Zipf-like sample gives the hottest key ~23.9% and the top ten ~63.6%. A toy contention model then predicts far lower service capacity. The modeled ops/sec is intentionally not a product benchmark—the lesson is that shape changes the answer.

4. AtlasMart lab: same count, different workload shape

import random
from collections import Counter

random.seed(2303)
KEYS = [f"k{i:03d}" for i in range(1,101)]
N = 20000

def sample_uniform():
    return [random.choice(KEYS) for _ in range(N)]

def sample_zipf(alpha=1.1):
    weights = [1/(rank**alpha) for rank in range(1,len(KEYS)+1)]
    return random.choices(KEYS, weights=weights, k=N)

def summarize(name, keys):
    c = Counter(keys)
    top1 = c.most_common(1)[0]
    top10 = sum(n for _,n in c.most_common(10))
    # Teaching service model: base 0.20ms + contention penalty from repeated access concentration.
    penalty = sum((n/N)**2 for n in c.values()) * 18
    service_ms = 0.20 + penalty
    est_ops = 1000/service_ms
    print(name)
    print(" top1:", top1, "share=", f"{top1[1]/N:.1%}")
    print(" top10_share=", f"{top10/N:.1%}")
    print(" concentration_index=", round(sum((n/N)**2 for n in c.values()),4))
    print(" modeled_service_ms=", round(service_ms,3), "modeled_single_worker_ops_s=", round(est_ops))

print("BENCHMARK CONTRACT")
print({"records":100, "operations":N, "value_bytes":1024, "read_write":"80/20", "concurrency":64,
       "replication_factor":3, "write_ack":"quorum", "durability":"fsync modeled", "indexes":1, "failure_state":"healthy"})
print()
summarize("UNIFORM", sample_uniform())
print()
summarize("ZIPF-LIKE alpha=1.1", sample_zipf())
print("\nSame operation count, record size, and nominal concurrency; request shape alone changes concentration and modeled cost.")

5. Production judgment: compare guarantees and environments, not just speed

Use the same dataset snapshot, schema/indexes, replication, consistency/durability, client version, warmup policy, test duration, offered rate, and failure state for A/B comparisons. Disclose whether compaction, repair, checkpoints, backups, or rebalance ran. Report distributions, useful throughput, errors, and freshness—not only successful ops/sec. If one candidate disables fsync or lowers acknowledgements, label the changed durability window rather than presenting it as a free speedup.

YCSB is useful precisely because workload properties such as read/write proportions and request distribution are explicit. It is not a substitute for your workload. The next lesson examines another benchmark trap: a closed-loop generator can stop sending during a stall and thereby omit latencies that real scheduled arrivals would have experienced.

References

Foundational claims use primary research or current official documentation where practical. Product references are implementation anchors only; the mandatory labs are vendor-neutral.

• Yahoo! Cloud Serving Benchmark (YCSB) — Open benchmark suite; Apache-2.0 repository and workload machinery.
• YCSB Workload Template — Current workload properties include request distribution and histogram/raw measurement options.
• YCSB Core Workloads — Examples of read/write/scan mixes and application-oriented workload definitions.
• Cooper et al. — Benchmarking Cloud Serving Systems with YCSB — Foundational YCSB paper defining a framework for evaluating cloud data-serving systems under specified workloads.