Build the AtlasMart OpenSearch trust boundary from the Security plugin outward: transport/HTTP TLS, node and admin certificate identities, initial security-index bootstrap, safe configuration change, and evidence that the cluster is not relying on demo credentials.
Security Plugin Architecture, Node/HTTP Certificates, Distinguished Names, and Initialization
Build least-privilege OpenSearch security with observable identity, authorization, fine-grained read controls, credential lifecycle, and explicit Elasticsearch comparison boundaries.
Learning outcomes
AtlasMart’s OpenSearch node is still running the convenient demo security setup from Chapter 01. That is acceptable for a disposable learning cluster, but it is a dangerous production model: demo certificates, an administrator identity used for application traffic, and configuration changes performed without a backup make both compromise and accidental lockout more likely. This lesson builds the Security plugin’s trust chain from the transport layer upward.
Distinguish OpenSearch transport TLS from HTTP TLS and explain why transport TLS is mandatory when the Security plugin is enabled.
Explain node-certificate distinguished names, admin certificates, CA trust and the danger of conflating node and administrator identities.
Initialize and inspect Security-plugin configuration without treating YAML files as automatically live cluster state.
Back up active security configuration before controlled securityadmin changes and prefer REST/Dashboards for routine resources.
Prove the AtlasMart cluster identity with TLS and authinfo evidence before creating application roles.
Examples target upstream self-managed
OpenSearch 3.8.0 / OpenSearch Dashboards 3.8.0, released 4 August 2026, with the bundled Security plugin.
The lab reuses the earlier single-node
atlasmart-os container at
https://localhost:9201 and
OPENSEARCH_INITIAL_ADMIN_PASSWORD only for the
disposable demo bootstrap. The upstream demo certificate is
intentionally not a production PKI contract, so earlier local
labs may use -k only against this isolated demo
endpoint. Production must use hostname-valid certificates and
normal CA verification. OpenSearch 3.8 distributions use the
bundled Java runtime unless you intentionally supply a
supported alternative. Elasticsearch 9.5.3 / Kibana 9.5.3
appears only for equivalent-intent comparisons; its
/_security APIs, realms, Spaces and
service-account model are not OpenSearch Security-plugin APIs.
The Security plugin stores active configuration in its
security system index. The YAML files are
initialization/deployment inputs, not automatically live
state. Before using securityadmin.sh on an
established cluster, back up the current configuration; a
broad directory load can overwrite resources created later
through REST/Dashboards. Prefer REST/Dashboards for ongoing
non-reserved users/roles and apply individual configuration
types only when file-based administration is genuinely
required.
The generation environment does not run the AtlasMart Docker
cluster. Requests below are deterministic lab instructions and
response fragments are expected shapes/invariants, not
fabricated captures. Record your own certificate
DN/fingerprint/expiry, authinfo identity,
role-resolution evidence, 200/403 behavior, tenant visibility,
DLS/FLS results, API-key ID/expiry/revocation, audit events,
and p95/p99 latency under authenticated load.
1. Security plugin architecture: four decisions, not one switch
The upstream Security plugin is an OpenSearch plugin that enforces encrypted transport, HTTP authentication, role-based authorization, optional fine-grained read controls, Dashboards tenancy and audit features. Treat these as separate mechanisms. TLS establishes encrypted, authenticated channels. An authentication backend proves a user identity. Role mappings connect that identity or its backend roles to OpenSearch security roles. Those roles finally authorize cluster, index and tenant actions.
| Layer | Mechanism | AtlasMart evidence |
|---|---|---|
| Transport | Node-to-node TLS and node certificate identity | Every node certificate is trusted and recognized as a node; no application certificate becomes a node. |
| HTTP | Client-to-node HTTPS |
Client verifies the production CA and hostname; the demo
-k exception never leaves the disposable
lab.
|
| Authentication | Internal DB, LDAP/AD, JWT/OIDC/SAML/client cert, etc. |
/_plugins/_security/authinfo identifies the
expected principal/backend roles.
|
| Authorization | Roles, role mappings, action groups, DLS/FLS, tenants | Allowed calls return 2xx; deliberately forbidden calls return 403. |
2. Node certificates and distinguished names
A certificate’s
distinguished name (DN) identifies its subject
using attributes such as CN, OU and O. The Security plugin must
distinguish cluster node certificates from ordinary TLS clients.
A common configuration lists allowed node-certificate DNs in
plugins.security.nodes_dn. If the DN does not match
the certificate subject under the plugin’s rules, the node does
not become a trusted transport peer.
openssl x509 -subject -nameopt RFC2253 -noout -in node1.pem
openssl x509 -issuer -dates -fingerprint -sha256 -noout -in node1.pem
plugins.security.nodes_dn:
- 'CN=os-data-01.atlasmart.internal,OU=Search,O=AtlasMart,C=US'
- 'CN=os-data-02.atlasmart.internal,OU=Search,O=AtlasMart,C=US'
# HTTP TLS should also be enabled for production client traffic.
plugins.security.ssl.http.enabled: true
Wildcard or regex DN rules reduce configuration work but increase the set of certificates that can satisfy the node identity check. Use them only when the PKI issuance policy itself constrains who can obtain matching certificates.
The certificate used by securityadmin.sh is a
privileged client certificate. Do not reuse a node certificate
as the admin certificate, and do not make the admin DN one of
the node DNs. Keep its private key outside application
containers and restrict its use to controlled security
administration.
3. Demo TLS is a teaching fixture, not a production PKI
The OpenSearch demo installer creates certificates, users, mappings and an internal authentication setup so a new cluster can start quickly. That is why the AtlasMart local lab can authenticate immediately. The convenience is itself the boundary: demo certificates and passwords are not production credentials. In a production deployment, issue certificates for the real node/client names, validate hostnames, protect CA/admin private keys and document rotation before expiry.
# Local demo exception only; production clients must verify the proper CA/hostname.
curl -k -u "admin:$OPENSEARCH_INITIAL_ADMIN_PASSWORD" https://localhost:9201/_plugins/_security/authinfo?pretty
Expected invariant: the response names the authenticated user and resolved roles/backend roles. It proves who OpenSearch thinks you are. It does not prove that the identity has an appropriately small permission set.
4. Initialize once; manage active state deliberately
The Security plugin stores active users, roles, mappings and
backend configuration in its security system index. Files under
config/opensearch-security are inputs. Editing them
does nothing until a tool such as
securityadmin.sh applies them. Conversely, applying
an old full directory can replace newer REST-created objects.
# Inside an OpenSearch installation/container, paths vary by deployment.
./plugins/opensearch-security/tools/securityadmin.sh -backup /secure-backup/security-config-$(date +%Y%m%d%H%M%S) -icl -nhnv -cacert config/root-ca.pem -cert config/admin.pem -key config/admin-key.pem
./plugins/opensearch-security/tools/securityadmin.sh -f config/opensearch-security/roles.yml -t roles -icl -nhnv -cacert config/root-ca.pem -cert config/admin.pem -key config/admin-key.pem
For ordinary non-reserved AtlasMart users and roles, the current documentation recommends REST API or Dashboards rather than repeatedly reloading YAML. Reserved/hidden bootstrap resources are a different governance problem and should remain deployment-managed.
5. Deliberately wrong approach: disable the plugin to “fix” certificate errors
Disabling security makes the symptom disappear by deleting the
control. The safe diagnosis is to identify whether the failure
is CA trust, hostname/SAN validation, node-DN matching, admin-DN
configuration, expired certificates or authentication. Use
certificate inspection, OpenSearch logs and
authinfo evidence. Repair the smallest failing
layer.
# Example once AtlasMart has a hostname-valid production certificate.
curl --cacert /etc/atlasmart/search-root-ca.pem -u "$ATLASMART_READER_USER:$ATLASMART_READER_PASSWORD" https://search.atlasmart.internal:9200/_plugins/_security/authinfo?pretty
6. Production judgment
PKI ownership matters more than copying certificate paths.
Define who issues node and client certificates, how renewal is
tested, where admin private keys live, how revoked certificates
are handled, and whether a load balancer terminates or passes
through TLS. Managed OpenSearch services can replace parts of
this model with provider-managed certificates, IAM/domain policy
or provider-specific identity integration; never paste upstream
self-managed opensearch.yml assumptions into a
managed service without checking its control plane.
Check your understanding
- Why is transport TLS different from HTTP TLS?
- What does nodes_dn control?
- Why back up before rerunning securityadmin?
- Does a successful authinfo response prove least privilege?
- Why is disabling the Security plugin a poor certificate fix?
Review the answers
1. Transport TLS authenticates/encrypts node-to-node cluster traffic; HTTP TLS protects client REST traffic.
2. Which certificate subjects are accepted as OpenSearch node identities when DN-based node authentication is used.
3. A broad load can overwrite active resources created later through REST or Dashboards.
4. No. It proves identity/role resolution; authorization must be tested with allowed and forbidden operations.
5. It removes the security boundary instead of diagnosing trust, hostname, DN or credential configuration.
Summary and next step
Preserve the evidence, assumptions, version boundaries, and safety checks established in this lesson. Carry them into the next lesson—or, at the end of the capstone, into the production runbook—rather than treating this lesson as an isolated recipe.
References
- OpenSearch 3.8 release/version history — Pinned OpenSearch 3.8.0 release baseline.
- OpenSearch Security overview — Security-plugin architecture, TLS, authentication and authorization overview.
- Configure TLS certificates — Transport/HTTP TLS, node DNs and admin certificate concepts.
- Apply Security configuration safely — Security-index initialization, backup and narrow apply guidance.
- Security APIs — Internal users, roles, mappings, tenants, certificate and related APIs.
- Users and roles — Internal users, roles, role mappings and built-in roles.
- Authentication backends — Authentication flow and chained backend concepts.
- Multi-tenancy — OpenSearch Dashboards tenant mechanics and permissions.
- Document-level security — DLS read filtering and limitations.
- Field-level security — FLS include/exclude semantics and interactions.
- Audit logs — Security audit categories, storage and enablement.
- API keys — OpenSearch 3.7+ scoped API-key lifecycle and limitations.
- OpenSearch downloads/license — Current distribution and Apache-2.0 licensing context.