API Keys, Service Accounts/Tokens, Rotation, Expiration, and Machine-to-Machine Authentication

Learning outcomes

AtlasMart has correct roles, but a deployment still fails security review because the search service stores a never-expiring username/password in three replicas and the same credential is used by CI. Machine-to-machine authentication needs a lifecycle: scope, issuance, storage, expiration or rotation, revocation and evidence that old credentials stop working.

1. API keys are identities plus an authorization envelope

An Elasticsearch API key contains an ID and secret. The secret is shown at creation and should be handled like a password. Role descriptors can narrow the key. The effective authorization is constrained by the privileges of the authenticating principal that creates it; a key-issuing principal cannot mint authority it does not possess. This is why “create all keys as superuser” is technically convenient but creates a large issuance blast radius.

POST /_security/api_key
{
  "name": "atlasmart-search-20260911-a",
  "expiration": "20m",
  "role_descriptors": {
    "atlasmart_search_key": {
      "cluster": [],
      "indices": [
        {
          "names": ["atlasmart-products-secure-v1"],
          "privileges": ["read", "view_index_metadata"]
        }
      ]
    }
  },
  "metadata": {
    "service": "atlasmart-search",
    "environment": "lab",
    "rotation_generation": 1
  }
}

Store the returned id for inventory. Deliver the secret through a secret manager or process-specific environment injection. Do not put the encoded value in Git, Docker Compose source, screenshots or issue trackers.

2. Prove the key’s identity and boundaries

export ATLASMART_API_KEY="<value delivered by your local secret store>"

curl --fail --silent --show-error \
  --cacert "$ES_CA" \
  -H "Authorization: ApiKey $ATLASMART_API_KEY" \
  "$ES_URL/_security/_authenticate?pretty"

# Expected 200
curl --cacert "$ES_CA" \
  -H "Authorization: ApiKey $ATLASMART_API_KEY" \
  "$ES_URL/atlasmart-products-secure-v1/_search?size=1"

# Expected 403
curl --cacert "$ES_CA" \
  -H "Authorization: ApiKey $ATLASMART_API_KEY" \
  -X DELETE "$ES_URL/atlasmart-products-secure-v1"

The key should be useful enough for its job and useless for unrelated cluster/security administration.

3. Rotation is overlap, verification, cutover, then revocation

Do not “rotate” by deleting the old key first. Issue generation B, deploy it to one canary instance, verify authentication/authorization and application metrics, roll it out, then invalidate generation A. This keeps rollback possible until the new credential is proven.

GET /_security/api_key?owner=true

POST /_security/api_key/_invalidate
{
  "ids": ["<generation-a-key-id>"]
}

# Old key: expect 401 after invalidation.
curl --cacert "$ES_CA" \
  -H "Authorization: ApiKey $OLD_API_KEY" \
  "$ES_URL/_security/_authenticate"

# New key: expect 200 and the expected API-key identity.
curl --cacert "$ES_CA" \
  -H "Authorization: ApiKey $NEW_API_KEY" \
  "$ES_URL/_security/_authenticate"

4. Service accounts are predefined Elastic service identities

Elastic service accounts are predefined in code for specific Elastic services (for example Kibana or Fleet Server). They have fixed privileges and authenticate using service-account tokens. They are not a general factory for arbitrary AtlasMart business services, and they are not human accounts. Service tokens do not expire automatically; lifecycle requires explicit deletion/rotation. For ordinary AtlasMart search/ingest applications, a scoped expiring API key is usually the clearer learning path.

5. Derived-key edge case

If the credential creating a new API key is itself an API key, Elasticsearch restricts derived-key privilege behavior: it cannot create a child key with an independently privileged role descriptor. This prevents arbitrary privilege delegation chains. Design a dedicated human/automation issuance path with manage_own_api_key or the appropriate administrative privilege rather than allowing every application key to mint peers.

6. Secrets are deployment state, not application configuration text

Use a secret manager, orchestrator secret, protected environment injection or equivalent. Prevent request headers from appearing in debug logs. Scope CI variables to the environment/job that needs them. Track key IDs, owners, purpose, issue time, expiration and rotation generation in inventory; never inventory raw secrets.

credential_id: "api-key-id-only"
name: atlasmart-search-20260911-b
owner: platform-search
purpose: read product search index
environment: lab
issued_at: 2026-09-11T14:30:00Z
expires_at: 2026-09-11T14:50:00Z
rotation_generation: 2
secret_location: secret-manager-reference-only

7. Production judgment

Pick expiration shorter than the organization can reliably rotate, not an arbitrary “secure” number. Very short keys without automation cause outages; never-expiring keys enlarge breach windows. Monitor authentication failures during rollout, keep a tested rollback key only for a bounded overlap, and remove stale credentials after owners/services disappear.

References

• Elastic: Secure your cluster — Current security architecture and deployment-type responsibility.
• Elastic: Automatic security setup — First-start TLS, elastic password, and Kibana enrollment-token behavior.
• Elastic: TLS for cluster communications — HTTP versus transport TLS and trust boundaries.
• Elastic: Users and roles — Authentication and RBAC model.
• Elasticsearch privileges reference — Cluster, index, application and run-as privilege semantics.
• Elastic: API key API — Expiration and role-descriptor scoping.
• Elastic: Service accounts — Predefined service accounts and service tokens.
• Elastic: DLS/FLS — Read-oriented document/field restrictions and limitations.
• Elastic: Run as — Delegated request semantics and supported authenticators.
• Elastic: Audit logging — Audit-event configuration and subscription notice.
• Elastic self-managed subscriptions — Current Basic/paid feature boundaries.
• Elastic downloads — Pinned Elasticsearch 9.5.3 release baseline.