Turn AtlasMart observability evidence into environment-safe dashboards and supported saved artifacts whose UI boundaries never replace underlying index authorization.
Dashboards/Kibana Data Views, Saved Objects, Visualizations, and Environment Separation
Create an operational evidence model that links user-visible search symptoms to cluster, node, index, query, ingest, and relevance signals and turns them into actionable SLOs.
Learning outcomes
Distinguish Elasticsearch/Kibana data views and saved objects from underlying Elasticsearch index authorization.
Organize operational visualizations around user symptoms, diagnostic drill-down, and ownership instead of metric inventory.
Use spaces/tenants and environment naming to reduce accidental cross-environment edits while respecting their security limits.
Export/import dashboard artifacts through supported APIs or UI flows rather than editing Kibana/OpenSearch system indices directly.
Design a portable AtlasMart dashboard contract that can be reproduced in Elastic or OpenSearch without pretending saved-object formats are compatible.
Run mutating, destructive, security, lifecycle, snapshot, failure-injection, and load-test commands only in the disposable AtlasMart lab or an equivalently isolated environment. Verify the target cluster, index, tenant, credentials, and rollback path before execution; treat shown output as an expected invariant unless the lesson explicitly labels it as captured evidence.
Examples are reviewed against
Elasticsearch/Kibana 9.5.3 and
OpenSearch/OpenSearch Dashboards 3.8.0, using
the course's established local TLS/auth conventions and bundled
JVMs. Elastic metrics examples use supported cluster, node,
index, task, slow-log, Kibana rule, saved-object/data-view, and
SLO surfaces. OpenSearch examples use node/index stats, Alerting
monitors, Dashboards, and Query Insights where
installed/enabled. OpenSearch's native SLO feature is currently
documented as experimental; Elastic SLO
management has license and node-role prerequisites, so the
mandatory lab also includes a product-neutral SLI/error-budget
worksheet. No live cluster is available in this generation
environment: commands are reproducible, expected invariants are
stated, and latency/throughput values are labeled
MEASURED instead of fabricated.
1. AtlasMart problem: the dashboard exists, but nobody trusts it
Production, staging, and a game-day cluster all send similarly named telemetry. An engineer opens an old dashboard, changes a filter, and assumes the p99 panel represents production. The technical failure is not chart rendering; it is missing environment identity, artifact ownership, and data authorization.
A dropdown is a convenience, not an authorization boundary. Separate data privileges first, then use spaces/tenants/data-source conventions to reduce human error.
2. Three layers: data, view, saved artifact
| Layer | Elastic/Kibana | OpenSearch Dashboards | Security implication |
|---|---|---|---|
| Data | indices/data streams | indices/data streams | controlled by search-cluster privileges |
| Logical view | data view / time field | index pattern/data source concepts depending workflow | does not grant index access |
| Saved artifact | dashboard, visualization, search, data view saved objects | Dashboards saved objects and plugin objects | UI object access is distinct from data access |
Elastic explicitly warns against writing directly to the
.kibana index. Use documented
saved-object/data-view/dashboard APIs or the UI so migrations
and metadata remain valid. Apply the same principle to
OpenSearch Dashboards system objects: use supported
Dashboards/plugin interfaces, not direct hidden-index surgery.
3. Dashboard information architecture
Build the AtlasMart operations view in four rows. The first row is for the incident commander; deeper rows are for diagnosis.
| Row | Panels | Question |
|---|---|---|
| 1 — SLO | availability, p95/p99, freshness, error/rejection rate, relevance probe | Are users affected? |
| 2 — Workload | search/index rate, concurrency, top queries, ingest rate | What demand changed? |
| 3 — Runtime | CPU, heap/GC, thread pools, breakers, caches | Which resource or guardrail is stressed? |
| 4 — Storage/topology | disk, shards, merges, refresh, recovery, allocation | Is storage or topology driving the symptom? |
4. Environment contract before visualization
Every telemetry document used by the reproducible metric report
should have stable dimensions such as environment,
service, cluster_id, and a timestamp.
Avoid free-form high-cardinality labels unless they are required
for diagnosis.
PUT /atlasmart-ops-v24
{
"mappings": {
"properties": {
"@timestamp": {"type":"date"},
"environment": {"type":"keyword"},
"service": {"type":"keyword"},
"cluster_id": {"type":"keyword"},
"latency_ms": {"type":"double"},
"status": {"type":"keyword"},
"freshness_lag_ms": {"type":"long"},
"relevance_pass": {"type":"boolean"}
}
}
}Production dashboards should default to an explicit production filter and display the environment prominently. The underlying principal must also have only the intended environment's index privileges when isolation is required.
5. Kibana spaces and saved objects
Kibana spaces organize saved objects and can provide feature-level access, but Elastic documents that spaces do not automatically restrict access to the underlying Elasticsearch data. Treat a production space as a UI boundary layered on top of Elasticsearch roles. For promoted dashboards, export supported saved objects as opaque artifacts and import them into the target space/version; do not hand-edit migration metadata.
POST kbn:/api/saved_objects/_export
{
"objects": [
{"type": "dashboard", "id": "atlasmart-operations-v24"}
],
"includeReferencesDeep": true
}
# Preserve the exported NDJSON as an opaque deployment artifact.
# Exported saved objects are not backwards-compatible with older Kibana versions.6. OpenSearch Dashboards separation
OpenSearch Dashboards has its own saved-object, tenant, data-source, and plugin boundaries. The operational intent can match Kibana—separate environments, least privilege, version-controlled exports—but object schemas and APIs are not assumed interchangeable. In Security-plugin deployments, tenant visibility is a Dashboards object boundary; index permissions still decide which data the user can query.
Version-control the dashboard contract—panel purpose, query/aggregation definition, filters, units, SLO thresholds, and owner—plus product-specific exported artifacts. Do not make one product's saved-object JSON the canonical cross-product specification.
7. Free/local reproducible fallback: metric report
If the UI is unavailable or a visualization feature is subscription/managed-service dependent, the mandatory chapter lab remains reproducible with a generated HTML/JSON metric report fed by the same API evidence. The report should include the SLO row, query/runtime/storage evidence, timestamps, and links to the runbook. That preserves the learning objective without requiring a paid observability UI.
{
"panel_id": "search-p99",
"owner": "search-platform",
"environment": "prod",
"sli": "client_search_latency_ms",
"aggregation": "p99",
"window": "5m",
"unit": "ms",
"runbook": "runbooks/search-tail-latency.md",
"drilldowns": ["top_queries", "thread_pool_search", "cpu_gc", "disk_merge"]
}8. Production judgment
Dashboards are operational software. Give them owners, version them, test their filters against fixtures, validate units and aggregation semantics, and review them when index names, mappings, spaces/tenants, or SLO definitions change. Keep a small “golden incident” dataset so a dashboard import can be smoke-tested before promotion.
Check your understanding
- Why is a Kibana space not an Elasticsearch data-security boundary?
- Why avoid direct writes to .kibana?
- What should the first dashboard row answer?
- What is the portable artifact across Elastic and OpenSearch?
- What is the free/local fallback?
Review the answers
1. Spaces organize/authorize Kibana objects; underlying index access is still governed by Elasticsearch privileges.
2. They bypass Kibana saved-object migrations and can corrupt future compatibility.
3. Whether users are affected, using availability, tail latency, freshness, errors/rejections, and relevance.
4. The dashboard contract plus product-specific exports, not a shared saved-object JSON schema.
5. A reproducible metric report built from supported APIs and the same SLO/runbook contract.
Summary and next step
Preserve the evidence, assumptions, version boundaries, and safety checks established in this lesson. Carry them into the next lesson—or, at the end of the capstone, into the production runbook—rather than treating this lesson as an isolated recipe.
References and version checks
- Elastic cluster stats API — Cluster-wide node, shard, store, JVM, CPU, and plugin context.
- Elastic node stats API — JVM, filesystem, thread pools, breakers, indexing pressure, search/indexing, caches, merge, refresh, and recovery metrics.
- Kibana alerting — Rules, schedules, alerts, and connector actions.
- Elastic SLO access — Current license, transform/ingest-role, space, and index-security requirements.
- Kibana saved objects — Dashboards, visualizations, data views, import/export, and permissions.
- OpenSearch Nodes Stats API — JVM, filesystem, process, thread-pool, and index metric evidence.
- OpenSearch Alerting — Monitor, trigger, alert, and action model.
- OpenSearch Query Insights: top N queries — Latency, CPU, and memory query evidence.
- OpenSearch Query Insights Dashboards — Live/top-N/configuration UI and query details.
- OpenSearch Observability — Dashboards, alerting/detection, traces, and experimental SLO scope.
- Elastic Stack 9.5.3 release and OpenSearch 3.8.0 artifacts — pinned September/August 2026 server baselines.