Search SLOs: Availability, p95/p99 Latency, Freshness, Indexing Lag, Error/Rejection Rate, and Relevance Quality

Learning outcomes

1. From “fast search” to an operational contract

“Search should be fast” cannot be alerted on or reviewed. An SLI is a measured indicator, an SLO is a target for that indicator over a defined population/window, and an error budget is the amount of allowed bad service implied by the target. These are service contracts, not Elasticsearch settings.

2. AtlasMart SLI catalog

3. Tail latency: p95 and p99 are not averages

Percentiles answer distribution questions. p99 is the value at or below which 99% of eligible observations fall for the chosen window/population. Do not derive p99 by averaging per-node p99 values, and do not compare percentiles from different request populations as if they are the same SLI. Record both tail latency and throughput because a system can meet latency only by rejecting work.

4. Error-budget math

For an availability target of T over N eligible requests, the allowed bad-event budget is N × (1 − T). For a time-based SLO, multiply the window duration by (1 − T). This arithmetic is deterministic; the actual good/bad counts are measured.

slo_name: atlasmart_interactive_search
window: 30d
population: prod catalog search requests excluding synthetic/admin traffic
availability_target: 99.90%
latency_objective: p99 <= AGREED_MS
freshness_objective: marker_searchable_within <= AGREED_SECONDS
rejection_objective: rejection_rate <= AGREED_RATE
relevance_gate: golden_query_suite >= AGREED_SCORE
budget_math:
  availability_bad_fraction = 1 - 0.999 = 0.001
  allowed_bad_requests = eligible_requests * 0.001
all_runtime_values: MEASURED

5. Elastic and OpenSearch SLO product features

Elastic's Observability SLO feature can manage SLO definitions and error-budget summaries, but current documentation requires an appropriate license and Elasticsearch nodes with transform and ingest roles. Kibana spaces scope the SLO UI but do not replace index authorization. OpenSearch documentation currently labels its SLO feature experimental and ties it to Prometheus-compatible ruler concepts. Therefore the course does not make either UI a prerequisite: the canonical lab artifact is the product-neutral SLI/SLO worksheet plus raw evidence.

6. Relevance belongs in reliability

A search endpoint can return 200 in 40 ms and still fail users because ranking changed. Maintain a versioned golden query set with judgments and a chosen metric (for example recall@k, NDCG@k, or a simpler must-contain constraint appropriate to the product). Run it before analyzer, synonym, vector-model, or ranking changes and after incident recovery. Do not turn one offline metric into a universal business KPI; state the query set and judgment process.

query_id,q,required_top_k_product
q001,"waterproof hiking backpack",P-1005
q002,"usb c charger",P-1002
q003,"noise cancelling headphones",P-1004

# Gate example:
# PASS when every required product appears within the agreed top-k for the pinned fixture.
# Production quality still needs broader judgments and business validation.

7. Freshness probe design

Use a uniquely identified marker document. Record the write acknowledgment, then poll a normal search path at a bounded interval until the marker appears or the objective expires. Clean up the marker. This measures end-to-end search freshness for that path; it is stronger than reading only a refresh counter.

t_write_ack = now()
index(unique_marker)
while now() - t_write_ack < objective:
    if ordinary_search(unique_marker).found:
        record_freshness_lag(now() - t_write_ack)
        break
    sleep(bounded_poll_interval)
else:
    record_bad_event("freshness")
cleanup(unique_marker)

8. Production judgment

Keep SLOs few, stable, and owned. Exclude traffic only with a documented reason, version the definition, and preserve historical changes. Use internal metrics as explanatory evidence, not as substitutes for the SLI. Pair fast-burn alerts with longer-window budget views so operators distinguish acute incidents from slow degradation.

References and version checks

• Elastic cluster stats API — Cluster-wide node, shard, store, JVM, CPU, and plugin context.
• Elastic node stats API — JVM, filesystem, thread pools, breakers, indexing pressure, search/indexing, caches, merge, refresh, and recovery metrics.
• Kibana alerting — Rules, schedules, alerts, and connector actions.
• Elastic SLO access — Current license, transform/ingest-role, space, and index-security requirements.
• Kibana saved objects — Dashboards, visualizations, data views, import/export, and permissions.
• OpenSearch Nodes Stats API — JVM, filesystem, process, thread-pool, and index metric evidence.
• OpenSearch Alerting — Monitor, trigger, alert, and action model.
• OpenSearch Query Insights: top N queries — Latency, CPU, and memory query evidence.
• OpenSearch Query Insights Dashboards — Live/top-N/configuration UI and query details.
• OpenSearch Observability — Dashboards, alerting/detection, traces, and experimental SLO scope.
• Elastic Stack 9.5.3 release and OpenSearch 3.8.0 artifacts — pinned September/August 2026 server baselines.