Dashboards / Notebook / Observability Query Workflows and Security / Resource Boundaries

Learning outcomes

Analytical languages rarely stay in curl. AtlasMart analysts put queries into Kibana/Dashboards, notebooks, observability explorers, saved visualizations, alerts, or application code. That convenience can hide which principal executes the query, which indices it can read, how much data it scans, and who owns a runaway dashboard panel.

1. UI is a client, not a security boundary

Kibana and OpenSearch Dashboards send requests on behalf of authenticated users or service principals. A saved query, visualization, space, or tenant is an object-management boundary; it is not automatically a permission to the backing index. The server-side authorization layer still decides whether the data can be read.

2. OpenSearch Query Workbench and notebooks

Query Workbench offers SQL/PPL editors, tabular results, and Explain. OpenSearch Dashboards notebooks combine Markdown, SQL, PPL, and visualizations in a shareable analysis document. These are strong teaching and incident-analysis surfaces because the narrative can sit beside the query, but every paragraph can still consume cluster resources.

%md
AtlasMart checkout errors by 5-minute bucket. Query owner: SRE. Data: prod only.

%ppl
source=atlasmart-telemetry-v23
| where environment='prod'
| timechart timefield=@timestamp span=5 m sum(error_count) by service

%sql
SELECT service, SUM(error_count) AS errors
FROM atlasmart-telemetry-v23
WHERE environment='prod'
GROUP BY service
ORDER BY errors DESC

Pin the notebook's time range and data source before sharing a performance conclusion. A dashboard panel run over 15 minutes and the same panel run over 90 days are different workloads.

3. Kibana ES|QL workflow

ES|QL can power interactive exploration and visualizations in Kibana where supported. Keep the source query beside a small “query contract”: expected columns/types, data-view/index scope, time range, owner, refresh cadence, and acceptable latency. When a visualization changes a query, review the resulting language text rather than assuming the UI preserved semantics.

name=atlasmart-service-error-rate
owner=search-sre
language=ES|QL
source=atlasmart-telemetry-*
time_range=15m
refresh=60s
expected_columns=bucket,service,requests,errors,error_rate
max_rows=500
p95_panel_latency_slo=MEASURED
required_index_privilege=read
security_scope=prod telemetry only
query_version=git:<commit>

4. Resource boundaries: multiply panel cost by refresh and viewers

A query that costs 200 ms once may become expensive when 20 panels refresh every 10 seconds for 50 viewers. Capacity is a concurrency problem. Analytical interfaces can scan broad windows, run joins, calculate high-cardinality groupings, and materialize tabular results. Design time filters and group cardinality before raising server limits.

panels=MEASURED
viewers=MEASURED
refresh_interval_s=MEASURED
requests_per_minute = panels * viewers * 60 / refresh_interval_s

# Then measure per-query CPU, heap, bytes read, rows returned, p95/p99.
# Do not assume cache hit rate will rescue an unbounded design.

5. Observability workflows and query ownership

For incident work, a PPL/ES|QL query can be excellent because it leaves a readable sequence of filtering and aggregation decisions. Preserve the query text/hash in the incident timeline together with time range, cluster state, user identity, result row count, and whether remote/federated data was partial. This turns a dashboard screenshot into reproducible evidence.

incident=ATLASMART-ANALYTICS-001
query_language=PPL|ESQL|SQL
query_hash=MEASURED
principal=MEASURED
indices=MEASURED
remote_clusters=MEASURED
time_range=MEASURED
rows=MEASURED
p95_ms=MEASURED
partial_or_skipped=MEASURED
null_handling=VERIFIED
approximate_aggregations=DOCUMENTED
saved_object_id=MEASURED

6. Version/feature status belongs in the dashboard contract

Both ecosystems evolve their analytical languages quickly. OpenSearch PPL subsearch features can carry experimental status; ES|QL commands and cross-cluster/lookup capabilities have changed across 9.x releases. If a saved query depends on a preview/experimental command, label it in the dashboard/notebook and add an upgrade regression test. Do not discover the dependency during an incident.

7. Practical lab: same analysis in UI and API

PUT atlasmart-telemetry-v23
{
  "settings": {"number_of_shards": 1, "number_of_replicas": 0},
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "@timestamp": {"type": "date"},
      "service": {"type": "keyword"},
      "environment": {"type": "keyword"},
      "tenant_id": {"type": "keyword"},
      "request_count": {"type": "long"},
      "error_count": {"type": "long"},
      "duration_ms": {"type": "double"}
    }
  }
}

POST atlasmart-telemetry-v23/_bulk?refresh=true
{"index":{"_id":"e01"}}
{"@timestamp":"2026-09-11T10:00:00Z","service":"catalog-api","environment":"prod","tenant_id":"tenant-a","request_count":100,"error_count":2,"duration_ms":120}
{"index":{"_id":"e02"}}
{"@timestamp":"2026-09-11T10:01:00Z","service":"catalog-api","environment":"prod","tenant_id":"tenant-a","request_count":120,"error_count":3,"duration_ms":140}
{"index":{"_id":"e03"}}
{"@timestamp":"2026-09-11T10:02:00Z","service":"checkout-api","environment":"prod","tenant_id":"tenant-a","request_count":80,"error_count":4,"duration_ms":220}
{"index":{"_id":"e04"}}
{"@timestamp":"2026-09-11T10:03:00Z","service":"checkout-api","environment":"prod","tenant_id":"tenant-a","request_count":90,"error_count":5,"duration_ms":260}
{"index":{"_id":"e05"}}
{"@timestamp":"2026-09-11T10:05:00Z","service":"catalog-api","environment":"prod","tenant_id":"tenant-a","request_count":130,"error_count":1,"duration_ms":110}
{"index":{"_id":"e06"}}
{"@timestamp":"2026-09-11T10:06:00Z","service":"catalog-api","environment":"prod","tenant_id":"tenant-a","request_count":140,"error_count":2,"duration_ms":130}
{"index":{"_id":"e07"}}
{"@timestamp":"2026-09-11T10:07:00Z","service":"checkout-api","environment":"prod","tenant_id":"tenant-a","request_count":100,"error_count":2,"duration_ms":210}
{"index":{"_id":"e08"}}
{"@timestamp":"2026-09-11T10:08:00Z","service":"checkout-api","environment":"prod","tenant_id":"tenant-a","request_count":110,"error_count":3,"duration_ms":250}
{"index":{"_id":"e09"}}
{"@timestamp":"2026-09-11T10:10:00Z","service":"catalog-api","environment":"prod","tenant_id":"tenant-a","request_count":150,"error_count":1,"duration_ms":100}
{"index":{"_id":"e10"}}
{"@timestamp":"2026-09-11T10:11:00Z","service":"catalog-api","environment":"prod","tenant_id":"tenant-a","request_count":160,"error_count":1}
{"index":{"_id":"e11"}}
{"@timestamp":"2026-09-11T10:12:00Z","service":"checkout-api","environment":"prod","tenant_id":"tenant-a","request_count":120,"error_count":6,"duration_ms":280}
{"index":{"_id":"e12"}}
{"@timestamp":"2026-09-11T10:13:00Z","service":"checkout-api","environment":"prod","tenant_id":"tenant-a","request_count":130,"error_count":7,"duration_ms":300}

Run the canonical analysis through the REST API, then through the corresponding Kibana Console/ES|QL workflow or OpenSearch Query Workbench/notebook. Save a result table but also save the query text. Verify that the UI does not silently change source scope, time range, or output ordering. Then test with a principal that lacks access to the index and require an authorization failure.

[ ] API and UI query text are semantically equivalent
[ ] exact requests/errors match deterministic fixture
[ ] null duration denominator is documented
[ ] p95 labeled approximate
[ ] forbidden principal cannot read atlasmart-telemetry-v23
[ ] saved object/tenant does not grant index access
[ ] panel time range and refresh cadence are explicit
[ ] query owner and rollback/removal path are recorded

8. Production judgment

DELETE atlasmart-telemetry-v23

Next, translate one canonical requirement across native Query DSL, ES|QL, and PPL and compare semantics before cost.

References

• Elastic ES|QL reference — Current language model, commands, functions, and limitations.
• Elastic ES|QL REST API — /_query, formats, privileges, and async-query surface.
• Elastic ES|QL syntax — Source and processing-command pipeline semantics.
• Elastic ES|QL STATS — Grouping, filtered aggregates, and null handling.
• Elastic ES|QL ENRICH — Query-time use of executed enrich policies.
• Elastic ES|QL LOOKUP JOIN — Lookup-index joins and current cross-cluster behavior.
• Elastic ES|QL across clusters — Remote-index syntax, enrich/lookup placement, and security considerations.
• Elastic ES|QL limitations — Result-size, type, and feature boundaries.
• OpenSearch SQL and PPL — Current SQL/PPL mental models and entry points.
• OpenSearch PPL — Pipeline syntax and SQL-plugin requirement.
• OpenSearch SQL — Relational syntax and response formats.
• OpenSearch SQL/PPL API — Query, explain, cursor, and format APIs.
• OpenSearch Query Workbench — Interactive SQL/PPL workflow and read-only boundary.
• OpenSearch Dashboards notebooks — Markdown, SQL, PPL, visualization, and reporting workflow.
• OpenSearch PPL lookup — Dimension-table enrichment semantics.
• OpenSearch PPL join — Join forms, limits, and performance-sensitive modes.
• Elasticsearch release notes — verify the pinned server baseline and current ES|QL status.
• OpenSearch version history — verify the pinned OpenSearch/Dashboards baseline.