Document/Field-Level Security, Audit Logs, API Keys/Scoped Access Where Available, and Secrets Rotation

Learning outcomes

AtlasMart’s marketplace index contains several tenants and a sensitive cost_internal field. A reader role that can search every document and field violates the business boundary even if it cannot write. This lesson adds DLS/FLS, audit evidence and OpenSearch 3.8 API keys, while keeping the crucial rule that DLS/FLS are read restrictions—not write authorization.

1. DLS filters documents during reads

Document-level security (DLS) attaches a query to an index permission so reads expose only documents that satisfy the query. AtlasMart can therefore constrain a marketplace analyst to tenant-a. DLS does not become a write predicate: if the same role also has write/delete permissions, those operations are governed by index actions, not by the DLS query.

PUT _plugins/_security/api/roles/atlasmart_tenant_a_reader
{
  "cluster_permissions": [],
  "index_permissions": [{
    "index_patterns": ["atlasmart-products-secure-v1"],
    "dls": "{\"term\":{\"tenant_id\":\"tenant-a\"}}",
    "fls": ["sku", "name*", "tenant_id", "category", "price", "updated_at"],
    "masked_fields": [],
    "allowed_actions": ["read"]
  }],
  "tenant_permissions": []
}

The FLS include list omits cost_internal. Because name has a multi-field, name* intentionally includes both the text field and its subfields. When DLS references a field, ensure FLS does not hide that field in a way that breaks the DLS evaluation.

2. Prove what is hidden—and what is not guaranteed

# As tenant-A reader: expect tenant-a hits and no cost_internal in _source.
GET atlasmart-products-secure-v1/_search
{
  "query": {"match_all": {}},
  "_source": true,
  "sort": [{"sku":"asc"}]
}

Validate both document set and returned fields. Also test aggregations, field capabilities, stored fields and any application-specific retrieval path your clients use. Fine-grained security can affect query semantics and performance; measure p95/p99 using the real workload rather than assuming the filtered query is free.

3. Audit logs are evidence, not prevention

OpenSearch audit logging is disabled by default in the upstream self-managed configuration. Enabling it can record authentication, authorization and security events according to configured categories and storage. Audit data helps incident response and compliance; it does not prevent an over-privileged action.

# opensearch.yml — lab illustration
plugins.security.audit.type: internal_opensearch

Production audit design should consider separation of duties: if an attacker controls the protected cluster, audit events stored only on that cluster may be altered or lost with it. Select a storage/forwarding design consistent with your threat model, retention and privacy rules.

4. API keys: scoped machine credentials in OpenSearch 3.8

OpenSearch API keys are created by security administrators and authenticate with Authorization: ApiKey <token>. The plaintext token is returned only once; the plugin stores a SHA-256 hash. Keys can expire and be revoked. Requests authenticated with an API key cannot access protected Security/system indexes or call Security APIs.

config:
  dynamic:
    api_tokens:
      enabled: true
      max_duration_seconds: 7776000   # example policy ceiling: 90 days
      max_tokens: 100

Apply this Security configuration through your reviewed deployment/security-admin workflow; do not overwrite unrelated active configuration.

POST /_plugins/_security/api/apitokens
{
  "name": "atlasmart_ingest_20260911",
  "cluster_permissions": ["indices:data/write/bulk"],
  "index_permissions": [{
    "index_pattern": ["atlasmart-products-secure-v1"],
    "allowed_actions": ["write"]
  }],
  "duration_seconds": 3600
}

Capture once: store the returned token in a secret manager/environment outside source control, record the key ID and expiry, then discard the API response from ordinary logs. The explicit bulk cluster permission reflects the current API-key limitation documented for bulk evaluation.

# Use only the token value returned at creation.
curl -k -H "Authorization: ApiKey $ATLASMART_OS_API_KEY"   'https://localhost:9201/_plugins/_security/authinfo?pretty'

# Administrator revokes by key ID.
curl -k -u "admin:$OPENSEARCH_INITIAL_ADMIN_PASSWORD"   -X DELETE "https://localhost:9201/_plugins/_security/api/apitokens/$ATLASMART_OS_API_KEY_ID"

# Repeat the API-key request: it must now fail authentication.

5. Rotation is an overlap protocol

A safe rotation creates credential B with the same or smaller scope, deploys B to consumers, proves B works and forbidden operations still fail, then revokes credential A. If you revoke A before every consumer has switched, rotation becomes an outage. If you never revoke A, “rotation” only increases the number of valid secrets.

6. Production judgment

Fine-grained access can become complex when users receive multiple roles; test the combined effective result, not each role in isolation. Keep audit retention proportional to investigation/compliance needs. Prefer expiring, scoped machine credentials over long-lived shared passwords when supported, and ensure the administrative path that creates keys is more protected than the application path that consumes them.

References

• OpenSearch 3.8 release/version history — Pinned OpenSearch 3.8.0 release baseline.
• OpenSearch Security overview — Security-plugin architecture, TLS, authentication and authorization overview.
• Configure TLS certificates — Transport/HTTP TLS, node DNs and admin certificate concepts.
• Apply Security configuration safely — Security-index initialization, backup and narrow apply guidance.
• Security APIs — Internal users, roles, mappings, tenants, certificate and related APIs.
• Users and roles — Internal users, roles, role mappings and built-in roles.
• Authentication backends — Authentication flow and chained backend concepts.
• Multi-tenancy — OpenSearch Dashboards tenant mechanics and permissions.
• Document-level security — DLS read filtering and limitations.
• Field-level security — FLS include/exclude semantics and interactions.
• Audit logs — Security audit categories, storage and enablement.
• API keys — OpenSearch 3.7+ scoped API-key lifecycle and limitations.
• OpenSearch downloads/license — Current distribution and Apache-2.0 licensing context.