Dynamic Field Detection, Coercion, Date / Numeric Detection, and Failure Modes from Uncontrolled Input

Learning outcomes

AtlasMart receives product events from catalog, marketplace and merchant systems. One feed sends "price":"19.99", another sends 19.99; an optional merchant field called launch_date sometimes looks like a date and sometimes like an arbitrary label. If the first payload silently decides the mapping, later documents can be rejected or—worse—accepted under a search contract nobody intentionally designed. This lesson makes those decisions observable.

1. Dynamic mapping is executable schema creation

Mapping is the index-time contract that tells the engine how each field is parsed, indexed, stored for sorting/aggregation, and queried. With dynamic mapping enabled, the first concrete value for a new field can create that contract. A null value or empty array may create no mapping at all, so “we sent the field” does not prove “the field now has a type.”

2. Reproduce first-document ownership, then inspect the evidence

DELETE atlasmart-schema-detect-v1
PUT atlasmart-schema-detect-v1
{
  "settings": {"number_of_shards":1,"number_of_replicas":0},
  "mappings": {
    "date_detection": true,
    "numeric_detection": false
  }
}

PUT atlasmart-schema-detect-v1/_doc/first?refresh=true
{
  "merchant_code": "10001",
  "launch_hint": "2026-09-11",
  "price_guess": 19.99,
  "promo": {"code":"FALL"}
}

GET atlasmart-schema-detect-v1/_mapping
GET atlasmart-schema-detect-v1/_field_caps?fields=*

The mapping/field-capabilities responses are the evidence. Do not infer the final type from the original JSON alone. Verify whether launch_hint became a date, whether merchant_code stayed a string-derived field, and how price_guess was inferred on your exact server version.

3. Coercion can make bad producers look healthy

DELETE atlasmart-schema-coerce-v1
PUT atlasmart-schema-coerce-v1
{
  "mappings": {
    "properties": {
      "stock": {"type":"integer", "coerce":true},
      "product_id": {"type":"keyword"}
    }
  }
}

PUT atlasmart-schema-coerce-v1/_doc/1?refresh=true
{"product_id":"P-9001", "stock":"12"}

GET atlasmart-schema-coerce-v1/_doc/1

If the write succeeds, it proves only that the mapped field accepted the value under current coercion semantics. It does not prove the producer emitted the agreed JSON type. For critical contracts, validate at the producer/API boundary and keep coercion policy explicit. Do not use coercion to compensate indefinitely for upstream schema drift.

4. Controlled failures: conflict and field explosion

PUT atlasmart-schema-detect-v1/_doc/conflict
{
  "promo": "FALL"
}

The existing promo path was created as an object, so indexing a scalar at the same path should be rejected. Capture the HTTP status and root cause. Repair the producer or route the incompatible payload to a versioned destination; do not try to “change the field type in place.”

# Pseudocode: do not point this at a shared cluster.
for i in range(1, 501):
    document[f"merchant_attribute_{i}"] = "x"

# Observe _mapping growth and index.mapping.total_fields.limit.
# The lesson is the growth curve, not how high you can push the limit.

Unbounded merchant-defined keys multiply mapping metadata, per-segment field structures and cluster-state work. Increasing a field-count limit is not a schema strategy. Prefer an allowlisted explicit schema, a governed dynamic-template family, or a flattened-like representation where the product supports it and query semantics fit.

5. Product-specific dynamic controls are governance tools

The names are not cosmetic. They change whether ingestion succeeds, whether a new field is searchable, and whether the mapping changes. Make the selected mode part of the schema artifact and integration tests.

6. AtlasMart decision: explicit core, controlled extension surface

For products, AtlasMart owns product_id, name, category, price, available, and updated_at explicitly. Merchant extensions are accepted only under a documented path with bounded naming/type rules introduced in the next lesson. This prevents arbitrary request bodies from mutating the search contract.

Production judgment

Schema governance starts before the write reaches the cluster. Track rejected-document rate, new-field rate, mapping size, indexing p95/p99, and producer/version identity. A tolerant mapping can improve availability but can also turn upstream defects into expensive permanent index structures. Backups/snapshots protect bytes; they do not make an accidental mapping compatible with a future contract.

Summary and next step

Dynamic mapping is automated schema mutation. You can now distinguish detection, coercion, date/numeric heuristics and product-specific dynamic modes. Next, we replace arbitrary flexibility with ordered dynamic templates that match by type, field name and path.

Authoritative references

• Elastic mapping overview — Dynamic versus explicit mapping and schema-management guidance.
• Elastic dynamic field mapping — Detection rules, dynamic modes, date detection and numeric detection.
• Elastic dynamic templates — match/path/type conditions, template variables, ordering and runtime-field options.
• Elastic templates — Composable index/component templates, precedence, priority and reusable configuration.
• Elastic simulate index template API — Dry-run composition and overlapping-template evidence.
• Elastic simulate index API — Resolve the configuration that a concrete index name would receive.
• Elastic update mapping examples — Why incompatible field-type changes require a new index and reindex.
• Elastic aliases — Index aliases, write-index behavior and no-downtime reindex patterns.
• OpenSearch mappings — Dynamic mapping rules and dynamic-template controls.
• OpenSearch dynamic parameter — OpenSearch-specific dynamic modes including allow-templates variants.
• OpenSearch index templates — Composable templates, priorities and component composition.
• OpenSearch component template API — Reusable settings/mappings/aliases and creation-time behavior.
• OpenSearch simulate index template API — Preview template resolution before index creation.
• OpenSearch reindex API — Source snapshot behavior and destination-index requirements.
• OpenSearch alias API — Alias creation/update APIs and the distinction from manage-alias actions.