Async Search, Cancellation, Timeouts, Partial Results, and Long-Running Analytical Requests

Learning outcomes

An AtlasMart analyst launches a cross-month aggregation that takes tens of seconds. The web tier keeps a synchronous connection open, times out at 15 seconds, retries automatically, and accidentally doubles cluster work. The query may be legitimate; the request lifecycle is not. Long-running search needs ownership, polling, cancellation and retention semantics.

1. Timeout, cancellation and async are different controls

The dangerous pattern is a client timeout followed by an automatic retry while the original expensive server-side operation continues. A user sees one failure; the cluster sees two costly searches. Use request IDs/tracing plus Tasks/async IDs to verify server lifecycle.

2. Elasticsearch async search

Elasticsearch exposes POST /{index}/_async_search. The submit call can wait briefly for completion and otherwise returns an ID that can be polled through GET /_async_search/{id}. Results can be retained for a bounded keep_alive; access is tied to the submitting user/API key when security is enabled. Current Elasticsearch also caps the stored async response size by search.max_async_search_response_size.

POST /atlasmart-search-exec-v1/_async_search?wait_for_completion_timeout=100ms&keep_alive=5m
{
  "size":0,
  "query":{"match_all":{}},
  "aggs":{
    "by_category":{
      "terms":{"field":"category","size":10},
      "aggs":{"avg_price":{"avg":{"field":"price"}}}
    }
  }
}

GET /_async_search/ASYNC_ID
DELETE /_async_search/ASYNC_ID

The delete API releases stored state and cancels an ongoing async search as applicable. Do not keep completed responses “just in case” beyond the application retention requirement.

3. OpenSearch asynchronous search is a plugin namespace

OpenSearch uses the Asynchronous Search plugin and the endpoint namespace /_plugins/_asynchronous_search. Its settings include per-coordinating-node concurrency, maximum running time, maximum keep-alive and maximum wait-for-completion timeout. This is not wire-compatible with Elasticsearch async-search endpoints even though the workflow—submit, poll, optionally receive partial results, delete—is conceptually similar.

POST /_plugins/_asynchronous_search?wait_for_completion_timeout=100ms&keep_alive=5m
{
  "index":"atlasmart-search-exec-v1",
  "size":0,
  "query":{"match_all":{}},
  "aggs":{
    "by_category":{
      "terms":{"field":"category","size":10},
      "aggs":{"avg_price":{"avg":{"field":"price"}}}
    }
  }
}

GET /_plugins/_asynchronous_search/ASYNC_ID
DELETE /_plugins/_asynchronous_search/ASYNC_ID

4. Partial results are a product decision, not only a flag

A response can be incomplete because shards failed, the request timed out, an async search is still running, or cross-cluster components are unavailable. The UI must decide whether partial data is acceptable. A product-search page might reject incomplete price/filter results; an exploratory dashboard might display “partial as of …” with explicit shard/completion metadata.

Elasticsearch has allow_partial_search_results for shard failures and async responses that expose completion state. OpenSearch async search can return partial results while running. In both cases, application code must not silently label partial aggregates as final business totals.

GET atlasmart-search-exec-v1/_search
{
  "timeout":"1ms",
  "size":0,
  "query":{"match_all":{}},
  "aggs":{"by_category":{"terms":{"field":"category","size":10}}}
}
# Inspect timed_out, _shards.failed, failures and aggregation/hit state.
# Do not assume this tiny fixture will always time out.

5. Cancellation and Tasks API

List active search tasks during a deliberately long bounded lab, capture the relevant task ID, then use the task cancellation API only against that disposable task. Cancellation is cooperative; verify task disappearance/completion state instead of assuming the HTTP acknowledgement means all CPU stopped at that instant.

GET /_tasks?detailed=true&actions=*search*

# Cancel only the confirmed disposable task ID:
POST /_tasks/NODE_ID:TASK_ID/_cancel

GET /_tasks?detailed=true&actions=*search*

6. Long-running request contract

7. Production judgment

Async search is a request-lifecycle tool, not a performance optimization. The same expensive query still consumes CPU, heap, I/O and shard capacity. Its value is that the application can represent progress, avoid fragile long HTTP waits, control retries, and explicitly cancel/expire work.

For recurring heavy analytics, precomputation, transforms/rollups, materialized summaries or a separate analytics system may be better than repeatedly launching the same distributed search. Measure total cluster cost, not only user-perceived responsiveness.

Summary and next step

You can now operate long-running search intentionally. Lesson 5 combines the chapter into a tuning exercise: form a hypothesis, change query/mapping/routing/shards/aggregation/pagination, and accept the change only after correctness and tail-latency evidence.

Authoritative references

• Elastic search API — Current search request options, search type, pre-filter shard behavior and partial-results controls.
• Elastic search profiling — Profile operator trees, collectors and DFS profiling; also documents what profile does not measure.
• Elastic pagination — from/size result window, search_after, PIT consistency and PIT cleanup guidance.
• Elastic point in time API — Opening PITs, keep_alive, changing PIT IDs and retained-segment resource implications.
• Elastic async search submit — Long-running asynchronous search submission and response-size constraints.
• Elastic async search results — Polling, ownership/security and keep_alive behavior for async search.
• Elastic slow logs — Shard-level query/fetch slow logs and current query-logging guidance.
• OpenSearch pagination — from/size, search_after and PIT-backed pagination tradeoffs.
• OpenSearch PIT — Create/list/delete PIT APIs, security permissions and resource lifetime.
• OpenSearch profile API — Search component timings and explicit omissions such as network/queue/coordinator idle time.
• OpenSearch asynchronous search — Plugin endpoint, partial results and long-running search model.
• OpenSearch async settings — Maximum running time, concurrency, retention and wait timeout settings.
• OpenSearch logs — Request-level and shard-level search slow logs plus task-resource logging.
• OpenSearch tasks API — Task inspection and cancellation mechanisms.