Choose Storage Behavior by Write Rate, Read Pattern, SSD Characteristics, and Retention

Learning outcomes

AtlasMart now has three materially different workloads: transactional orders, high-ingest telemetry, and short-lived sessions. The wrong question is “B-tree or LSM—which is faster?” The defensible question is “which storage behavior best satisfies this workload's invariants, query shapes, retention, device constraints, failure model, and latency SLO after background maintenance reaches steady state?”

1. Start from the workload, not the engine family

Record at least: logical writes/second and bytes/second; point/range read mix; update/delete frequency; TTL distribution; object/key/value sizes; key skew; concurrency; durability level; replication; p50/p95/p99 latency SLOs; acceptable staleness; backup/restore; and growth. Then measure under realistic compaction/checkpoint state.

2. Device characteristics change constants, not fundamental accounting

SSDs remove mechanical seek penalties, but they still have bandwidth, IOPS, latency tails, controller/firmware behavior, garbage collection, and endurance. If AtlasMart accepts 0.8 TB/day of logical writes and the measured storage stack produces write amplification 8, the device absorbs about 6.4 TB/day before replication/backups. A lower-amplification design at 2 writes about 1.6 TB/day. Those numbers affect endurance, replacement rate, compaction headroom, and cost.

Do not import a write-amplification number from a blog. Measure actual device writes versus logical workload with the intended compression, indexes, compaction policy, retention, replication, and data distribution.

3. Benchmark the maintenance state, not only the ingest honeymoon

An LSM benchmark against an empty database may stop before serious compaction; a B-tree benchmark may run entirely in cache; either can report a flattering average while p99 stalls. A useful test has warm-up, steady-state data volume, realistic overwrite/delete/TTL behavior, cache-state declaration, background maintenance enabled, failure/restart phases, and enough duration to observe debt cycles.

Also avoid coordinated omission in load generators: if the client stops issuing work while the system is slow, measured latency can under-report the queueing users actually experience. Chapter 23 will cover benchmarking rigor in depth; here the rule is simply that storage-engine selection cannot be defended by one headline throughput number.

4. Wrong approach: “LSM for writes, B-tree for reads” as a universal rule

The slogan hides too much. A well-cached B-tree can sustain heavy writes; an LSM with Bloom filters and cache can serve excellent point reads; range behavior depends on layout/compaction; deletes and TTLs can favor specific compaction designs; and implementation quality can dominate family-level intuition. Choosing by slogan can create a cost surprise: high write amplification burns SSD endurance, or low-compaction settings trade ingest speed for read-tail incidents.

Repair by recording the decision as hypotheses, then run production-shaped measurements and failure tests. Keep a rollback path—dual-write/CDC validation or restoreable migration—because engine/file-format choices are difficult to reverse under load.

5. AtlasMart lab — an explicit cost model and tail-latency warning

The following cost units are deliberately invented so the decision process is inspectable. They are not benchmark data and must never be copied as tuning weights. Replace them with measurements from the candidate products and target hardware.

from statistics import median

profiles = {
    "orders":    dict(write=0.40, point=0.55, range=0.25, ttl=0.00, delete=0.10, tail=1.00),
    "telemetry": dict(write=0.90, point=0.15, range=0.65, ttl=0.80, delete=0.05, tail=0.55),
    "sessions":  dict(write=0.55, point=0.95, range=0.00, ttl=0.95, delete=0.10, tail=0.90),
}

# Lower is better. These weights are deliberately illustrative cost units,
# not benchmark results or claims about a specific database.
def btree_cost(p):
    return (1.15*p["write"] + 0.30*p["point"] + 0.35*p["range"] +
            0.90*p["ttl"] + 0.55*p["delete"] + 0.40*p["tail"])

def lsm_cost(p):
    return (0.45*p["write"] + 0.55*p["point"] + 0.70*p["range"] +
            0.55*p["ttl"] + 0.50*p["delete"] + 0.65*p["tail"])

for name, p in profiles.items():
    b, l = btree_cost(p), lsm_cost(p)
    choice = "B-tree-like" if b < l else "LSM-like"
    print(f"{name:9s} B={b:.3f} LSM={l:.3f} illustrative_choice={choice}")

logical_tb_day = 0.8
for wa in [2, 8]:
    print(f"logical={logical_tb_day:.1f} TB/day, WA={wa} -> device_writes={logical_tb_day*wa:.1f} TB/day")

quiet = [2.0, 2.1, 2.2, 2.3, 2.5, 2.7, 3.0, 3.2, 3.5, 4.0]
compacting = [2.1, 2.3, 2.5, 2.8, 3.2, 4.0, 5.5, 8.0, 13.0, 22.0]

def percentile(xs, q):
    xs = sorted(xs)
    i = round((len(xs)-1)*q)
    return xs[i]

print("quiet median/p90/p99:", median(quiet), percentile(quiet, .90), percentile(quiet, .99))
print("compacting median/p90/p99:", median(compacting), percentile(compacting, .90), percentile(compacting, .99))
print("decision rule: validate with production-shaped data, warm state, and background work enabled")

orders    B=1.167 LSM=1.357 illustrative_choice=B-tree-like
telemetry B=2.275 LSM=1.765 illustrative_choice=LSM-like
sessions  B=2.188 LSM=1.928 illustrative_choice=LSM-like
logical=0.8 TB/day, WA=2 -> device_writes=1.6 TB/day
logical=0.8 TB/day, WA=8 -> device_writes=6.4 TB/day
quiet median/p90/p99: 2.6 3.5 4.0
compacting median/p90/p99: 3.6 13.0 22.0
decision rule: validate with production-shaped data, warm state, and background work enabled

Interpretation

The toy scoring happens to prefer different engine families for different profiles. That is the point: the result changes when assumptions/weights change. The SSD calculation makes amplification financially/operationally visible, while the synthetic latency series shows why a similar median can coexist with a much worse tail during compaction.

6. Chapter synthesis: API latency is the visible tip of a storage pipeline

Chapter 09 connects five layers of evidence. WAL defines the recovery ordering behind an acknowledgement. B/B+Tree pages turn logical updates into page/WAL activity while preserving ordered locality. LSM paths batch writes into memtables/SSTables and move reorganization into compaction. Tombstones/filters/caches govern delete safety and read cost. Finally, device and workload measurements decide whether those mechanisms satisfy AtlasMart's SLOs.

No storage engine removes distributed-systems tradeoffs from earlier chapters. Replication still decides how many failure domains hold state; quorum semantics still govern reads/writes; partitions and hot keys still route load; failure detection still decides which nodes are trusted. Storage-engine behavior is one layer in that end-to-end contract.

7. Production decision record

Bridge to Chapter 10: With persistence internals established, the course moves up one abstraction level to key-value/data-structure stores: keys as the API, atomic single-key operations, TTL, eviction, counters, conditional writes, caching, and system-of-record boundaries.

Authoritative references

• PostgreSQL 18 — WAL — page-oriented relational implementation example for durable logging
• Apache Cassandra 5.0 — Storage Engine — current LSM implementation example
• RocksDB Overview — current LSM/compaction implementation example
• SQLite release history — current release/version reference for the optional SQLite example
• RocksDB releases — current release/version reference for the optional RocksDB example