Time-Series Dimensions/Metrics Concepts, Index Modes, Routing, and Compression / Storage Considerations

Learning outcomes

AtlasMart's logs and its metrics are both time stamped, but they have different physical economics. A log event carries free-form text and context; a metric sample repeatedly measures a bounded set of named dimensions. This lesson separates generic data streams from an Elastic-specific time series data stream (TSDS) so you can reason about shard routing and compression without inventing a false cross-platform abstraction.

1. Dimensions identify a series; metrics vary over time

Cardinality matters because each unique dimension combination identifies a distinct time series. A field can be syntactically a keyword yet operationally disastrous as a dimension if almost every event has a unique value.

2. Elastic-specific TSDS mechanics

PUT _index_template/atlasmart-metrics-tsds-v1
{
  "index_patterns":["atlasmart-metrics-tsds"],
  "priority":310,
  "data_stream":{},
  "template":{
    "settings":{
      "index.mode":"time_series",
      "index.routing_path":["service","host_id"]
    },
    "mappings":{
      "properties":{
        "@timestamp":{"type":"date"},
        "service":{"type":"keyword","time_series_dimension":true},
        "host_id":{"type":"keyword","time_series_dimension":true},
        "environment":{"type":"keyword","time_series_dimension":true},
        "request_rate":{"type":"double","time_series_metric":"gauge"},
        "request_total":{"type":"long","time_series_metric":"counter"}
      }
    }
  }
}

Elastic uses dimensions plus timestamp to derive time-series identity and routes documents by dimensions. Current documentation also exposes automatically managed dimension-based routing and a fallback index.routing_path. The relevant point is the mechanism: colocating a series can improve ingest/query locality and storage organization, but a pathological dimension distribution can still create skew.

3. OpenSearch: keep the shared concept, not Elastic-only syntax

OpenSearch 3.8 supports data streams, a required timestamp field (customizable in OpenSearch), backing-index rollover and ISM policies. Do not copy index.mode=time_series, time_series_dimension or time_series_metric into a supposedly portable template unless the target OpenSearch version explicitly documents those exact settings. For the dual-platform AtlasMart lab, ordinary data streams remain the portability baseline.

POST atlasmart-telemetry/_doc?refresh=true
{
  "@timestamp":"2026-09-11T06:10:00Z",
  "event_id":"metric-1010",
  "event_kind":"metric",
  "service":"catalog-api",
  "environment":"lab",
  "tenant_id":"tenant-a",
  "host_id":"node-1",
  "level":"INFO",
  "message":"request window",
  "duration_ms":8.7,
  "request_count":150,
  "error_count":3,
  "labels":{}
}

4. Routing can improve locality and still create hot shards

Routing determines which primary shard receives a document. A good routing dimension distributes load while keeping useful query locality. A bad one sends most traffic to one value—for example, routing all events only by environment=prod. Test the actual frequency distribution before making a field part of routing.

GET atlasmart-telemetry/_search
{
  "size":0,
  "aggs":{
    "services":{"terms":{"field":"service","size":20}},
    "hosts":{"cardinality":{"field":"host_id"}},
    "tenants":{"cardinality":{"field":"tenant_id"}}
  }
}

GET _cat/shards/.ds-atlasmart-telemetry*?v&s=index,shard,prirep

On the tiny one-shard course fixture there is no useful routing benchmark. The lab teaches what to measure; it does not manufacture a throughput win.

5. Compression and storage are emergent, not magic

Repeated dimensions, sorted/time-local data, doc values and segment compression can make metrics compact, but storage depends on mapping, values, source retention, segment state and index codec. Measure store.size, segment counts and representative query latency before/after a design change. Do not extrapolate a production compression ratio from ten course documents.

GET _data_stream/atlasmart-telemetry/_stats
GET _cat/indices/.ds-atlasmart-telemetry*?h=index,docs.count,pri.store.size,store.size
GET .ds-atlasmart-telemetry*/_stats/store,segments,indexing,search

6. High-cardinality budget

7. Portable decision pattern

If you need one schema across Elasticsearch and OpenSearch, use an ordinary data stream with explicit mappings and application-level dimension discipline. If you intentionally specialize for Elasticsearch TSDS, pin the target version and test timestamp windows, duplicate identity, routing and query behavior separately. Portability is a design choice, not an automatic property of similar REST APIs.

Production judgment

Measure series cardinality, per-key ingest skew, shard utilization, query fan-out, segment/store growth and late-event rejection before changing routing or TSDS settings. Retention reduces long-term storage but does not cure hot-shard design. Keep enough operational headroom for rollover, merge, recovery and snapshots.

Summary and next step

Dimensions and metrics are workload concepts; Elastic TSDS adds a product-specific storage/routing contract. Next we govern the actual event fields and labels so AtlasMart telemetry remains queryable without mapping or cardinality explosion.

Authoritative references

• Elastic data streams — Logical stream, hidden backing indices, write index, @timestamp and rollover behavior.
• Elastic use a data stream — Indexing, searching, rollover, and document updates/deletes through backing indices.
• Elastic time series data streams — Elastic-specific TSDS dimensions, metrics and generated time-series identity.
• Elastic time series index settings — index.mode=time_series, routing path, look-back/look-ahead windows and related settings.
• Elastic time-bound indices and dimension routing — Timestamp acceptance windows and dimension-based shard routing.
• Elastic data stream lifecycle — Built-in lifecycle, rollover, retention, downsampling and storage transitions.
• Elastic data stream retention — Effective retention semantics and lifecycle APIs.
• OpenSearch data streams — Backing indexes, timestamp field, rollover, search and ISM integration.
• OpenSearch rollover API — Manual rollover semantics for data streams and aliases.
• OpenSearch modify data stream API — OpenSearch 3.8 experimental backing-index add/remove operation.
• OpenSearch Index State Management — Policy-driven rollover, retention and index-state automation.
• OpenSearch update document API — Document correction semantics when targeting a concrete backing index.