Golden Signals for Databases: Latency, Throughput, Errors, Saturation, and Freshness

1. Start with the request the customer actually waits for

AtlasMart's checkout API can look healthy on a node dashboard while customers see pauses or stale inventory. Performance engineering therefore begins at the client-observed boundary. Latency is the elapsed time from request start until the client receives a usable response. Throughput is completed work per unit time. Errors must be classified—timeouts, overload/rejection, authentication failures, validation failures, and server faults mean different things. Saturation means some finite resource or concurrency boundary is near or beyond its sustainable working range; queue depth is one common symptom. Freshness measures how far a read model or replica trails the authoritative change, often by source version, log position, or elapsed lag.

Do not reduce a latency distribution to one mean. If 970 requests complete in roughly 6–12 ms but a thin tail takes hundreds of milliseconds, the mean can remain attractive while checkout p99 violates the service level objective (SLO). Percentiles answer a different question: p99 is the latency at or below which roughly 99% of observations fall; it says nothing about the worst 1%, and percentile estimates need enough samples.

2. Golden signals are correlated evidence, not five independent scorecards

A slow request can be caused by client-pool waiting, coordinator queueing, a remote replica, storage stalls, compaction, garbage collection (GC), cache misses, or a network retransmission. A rising queue with flat throughput suggests offered load has exceeded effective service rate or downstream work is blocked. A replica can answer quickly yet be stale. Conversely, a replica can be current but unavailable. Therefore AtlasMart correlates latency, throughput, errors, saturation, and freshness on the same timeline.

3. Deliberately wrong approach: “average latency is 14.5 ms, therefore the database is fine”

The lab creates 1,000 client observations. Most are fast, but the tail reaches 450 ms; one replica is 18 seconds behind and one node has a queue depth of 96. The average remains below 20 ms. Treating that mean as the health verdict produces a misleading benchmark/alert: the tail and freshness failure disappear from the headline.

4. AtlasMart lab: measure the distribution, errors, queues, and freshness

from statistics import mean

def pct(xs, p):
    s = sorted(xs)
    rank = max(0, min(len(s)-1, int((p/100) * len(s) + 0.999999) - 1))
    return s[rank]

# 1,000 client-observed requests: most fast, a thin slow tail, and explicit errors.
latency_ms = [6 + (i % 7) for i in range(970)] + [80 + (i % 5) * 20 for i in range(20)] + [250,270,290,310,330,350,370,390,410,450]
errors = {"timeout": 6, "overloaded": 4, "validation": 2}
window_s = 10
freshness_lag_s = {"replica-a": 0.4, "replica-b": 0.7, "replica-c": 18.0}
queue_depth = {"node-a": 3, "node-b": 5, "node-c": 96}

print("CLIENT LATENCY DISTRIBUTION")
for p in (50, 95, 99, 99.9):
    print(f"p{p}: {pct(latency_ms,p)} ms")
print(f"mean: {mean(latency_ms):.1f} ms")
print(f"throughput: {len(latency_ms)/window_s:.1f} req/s")
print("errors:", errors, "error_rate=", f"{sum(errors.values())/len(latency_ms):.2%}")
print("freshness_lag_s:", freshness_lag_s)
print("queue_depth:", queue_depth)

print("\nBROKEN AVERAGE-ONLY JUDGMENT")
print("average under 20 ms?", mean(latency_ms) < 20)
print("p99 under 100 ms?", pct(latency_ms,99) < 100)
print("replica-c freshness under 5 s?", freshness_lag_s["replica-c"] < 5)
print("diagnosis: an acceptable mean coexists with a bad tail, saturation, and stale data")

5. Production judgment and instrumentation contract

Define latency at the client boundary, tag operation class and consistency level, separate success from failure latency, and preserve a histogram rather than only averages. Bound label cardinality: raw tenant IDs, user IDs, or full keys can create expensive telemetry and security leakage. Track freshness with a stable source version where possible rather than guessing from wall-clock timestamps. A trace should show routing and waits, but sampling means absence of a trace is not proof an event did not happen.

Alert on service objectives and causal signals together: “checkout p99 above target while queue depth rises” is more actionable than “CPU above 80%.” During incidents, compare offered load with completed throughput; retry storms can make server operations rise even while useful customer throughput falls. The next lesson moves from cluster-level symptoms to the shard and storage-engine internals that explain them.

References

Foundational claims use primary research or current official documentation where practical. Product references are implementation anchors only; the mandatory labs are vendor-neutral.

• Dean & Barroso — The Tail at Scale — Primary systems work explaining why latency variability and the tail matter at distributed scale.
• PostgreSQL 18 — Monitoring Database Activity — Current official example of replication, I/O, WAL, activity, lock, and progress statistics.
• Google SRE Book — Monitoring Distributed Systems — Four-golden-signals framing and monitoring principles; useful operational context, not a database-specific standard.
• Apache Cassandra 5.0 — Monitoring — Current official example of coordinator/table latency, dropped messages, caches, compaction, and repair metrics.