Logs and Metrics Modeling: High Cardinality, Labels, Structured Events, and Field Governance

Learning outcomes

Observability pipelines often fail gradually: each team adds “just one more label,” JSON keys become fields, tenant/request identifiers become facets, and dashboards later pay the cost. AtlasMart therefore treats telemetry schema as a governed API with explicit high-cardinality budgets and security boundaries.

1. Model by query requirement, not producer convenience

2. Strict mappings expose drift early

PUT _index_template/atlasmart-telemetry-template-v1
{
  "index_patterns": ["atlasmart-telemetry"],
  "priority": 300,
  "data_stream": {},
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "@timestamp": {"type":"date"},
        "event_id": {"type":"keyword"},
        "event_kind": {"type":"keyword"},
        "service": {"type":"keyword"},
        "environment": {"type":"keyword"},
        "tenant_id": {"type":"keyword"},
        "host_id": {"type":"keyword"},
        "level": {"type":"keyword"},
        "message": {"type":"text"},
        "duration_ms": {"type":"double"},
        "request_count": {"type":"long"},
        "error_count": {"type":"long"},
        "labels": {"type":"object", "dynamic":"strict"}
      }
    }
  }
}

PUT _data_stream/atlasmart-telemetry

POST atlasmart-telemetry/_doc
{
  "@timestamp":"2026-09-11T06:20:00Z",
  "event_id":"evt-label-bad",
  "event_kind":"log",
  "service":"checkout-api",
  "environment":"lab",
  "tenant_id":"tenant-b",
  "host_id":"node-2",
  "level":"WARN",
  "message":"unexpected label payload",
  "duration_ms":21.1,
  "request_count":1,
  "error_count":0,
  "labels":{"request_uuid":"550e8400-e29b-41d4-a716-446655440000"}
}

Because labels is strict and no child is declared, the write should fail rather than silently creating a new searchable field. Capture the root cause as schema-drift evidence. The repair is to update the governed template/version or normalize/drop the label upstream—not to disable safeguards during an incident.

3. High cardinality is a budget, not a forbidden number

GET atlasmart-telemetry/_search
{
  "size":0,
  "aggs":{
    "event_ids":{"cardinality":{"field":"event_id"}},
    "services":{"cardinality":{"field":"service"}},
    "hosts":{"cardinality":{"field":"host_id"}},
    "tenants":{"cardinality":{"field":"tenant_id"}}
  }
}

cardinality itself is approximate at scale, so use it as reconnaissance, not a legal/compliance count. More importantly, cost depends on how a field is used: filtering by a high-cardinality event ID can be reasonable, while returning a huge terms aggregation over that same field to every dashboard is not.

4. Normalize labels upstream

{
  "@timestamp":"2026-09-11T06:22:00Z",
  "event_id":"evt-1022",
  "event_kind":"log",
  "service":"checkout-api",
  "environment":"lab",
  "tenant_id":"tenant-b",
  "host_id":"node-2",
  "level":"ERROR",
  "message":"payment gateway timeout",
  "duration_ms":2503.0,
  "request_count":1,
  "error_count":1,
  "labels":{}
}

Do not put raw customer headers, arbitrary JSON blobs, access tokens, secrets, personal data, or unbounded query strings into labels. Redact/normalize before indexing and use least-privilege ingest credentials. Search authorization must enforce tenant scope server-side; a UI filter is not a security boundary.

5. Structured logs and metrics need different acceptance tests

6. Observe schema and field growth

GET atlasmart-telemetry/_mapping
GET atlasmart-telemetry/_field_caps?fields=*
GET _cluster/stats?filter_path=indices.mappings,indices.fielddata
GET _data_stream/atlasmart-telemetry/_stats

Track field-count growth as a deployment signal. If fields appear without a reviewed schema change, treat that as drift even when indexing still succeeds. For large environments, store this inventory as a CI/CD artifact or periodic control-plane check.

7. Query safely under tenant scope

GET atlasmart-telemetry/_search
{
  "size":0,
  "query":{
    "bool":{
      "filter":[
        {"term":{"tenant_id":"tenant-b"}},
        {"term":{"service":"checkout-api"}},
        {"range":{"@timestamp":{"gte":"now-15m"}}}
      ]
    }
  },
  "aggs":{
    "levels":{"terms":{"field":"level","size":10}},
    "p95_duration":{"percentiles":{"field":"duration_ms","percents":[95]}}
  }
}

Application-side tenant filtering after a broader search is unsafe because unauthorized documents have already crossed the search boundary. Use product security controls and server-side scoped queries appropriate to the deployment.

Production judgment

Create budgets for total mapped fields, approved labels, expected cardinality bands, per-tenant query load, response bucket counts and raw event size. Monitor field-growth rate, circuit-breaker/rejection signals, p95/p99 search latency and storage per day. High-cardinality design is sustainable only when the query surface is intentional.

Summary and next step

Telemetry quality depends on bounded structure, not merely valid JSON. Next we handle the events that violate the happy path: late arrivals, corrections and deletes against older backing indices.

Authoritative references

• Elastic data streams — Logical stream, hidden backing indices, write index, @timestamp and rollover behavior.
• Elastic use a data stream — Indexing, searching, rollover, and document updates/deletes through backing indices.
• Elastic time series data streams — Elastic-specific TSDS dimensions, metrics and generated time-series identity.
• Elastic time series index settings — index.mode=time_series, routing path, look-back/look-ahead windows and related settings.
• Elastic time-bound indices and dimension routing — Timestamp acceptance windows and dimension-based shard routing.
• Elastic data stream lifecycle — Built-in lifecycle, rollover, retention, downsampling and storage transitions.
• Elastic data stream retention — Effective retention semantics and lifecycle APIs.
• OpenSearch data streams — Backing indexes, timestamp field, rollover, search and ISM integration.
• OpenSearch rollover API — Manual rollover semantics for data streams and aliases.
• OpenSearch modify data stream API — OpenSearch 3.8 experimental backing-index add/remove operation.
• OpenSearch Index State Management — Policy-driven rollover, retention and index-state automation.
• OpenSearch update document API — Document correction semantics when targeting a concrete backing index.