OpenSearch Cross-Cluster Replication/Search Concepts and Managed-Service Constraints

Learning outcomes

OpenSearch offers cross-cluster search plus a bundled cross-cluster replication plugin. The architecture resembles Elastic's leader/follower model but the APIs, Security-plugin roles, auto-follow semantics, and managed-service workflow differ enough that operators must maintain product-specific runbooks.

1. OpenSearch CCR plugin is pull-based active-passive replication

The follower cluster connects to a leader cluster and creates a new read-only follower index. The standard OpenSearch distribution bundles the cross-cluster-replication plugin; the minimal distribution may not. If node.roles is explicitly set on follower nodes, include remote_cluster_client. Security should be enabled on both sides with node-to-node encryption in production.

# Follower cluster: configure leader connection
PUT /_cluster/settings
{
  "persistent": {
    "cluster.remote.region_a.seeds": ["region-a-node:9300"]
  }
}

# Start follower replication
PUT /_plugins/_replication/atlasmart-products-dr-v1/_start
{
  "leader_alias": "region_a",
  "leader_index": "atlasmart-products-v1",
  "use_roles": {
    "leader_cluster_role": "atlasmart_ccr_leader",
    "follower_cluster_role": "atlasmart_ccr_follower"
  }
}
GET /_plugins/_replication/atlasmart-products-dr-v1/_status

2. Replication security is a two-cluster contract

The Security plugin includes replication permissions and built-in role concepts, but AtlasMart should create scoped roles rather than reusing all_access. The leader role should expose only indices intended for replication; the follower role should manage only required follower resources. CCS authorization and CCR authorization are related but distinct.

POST /_plugins/_replication/atlasmart-products-dr-v1/_pause
POST /_plugins/_replication/atlasmart-products-dr-v1/_resume
POST /_plugins/_replication/atlasmart-products-dr-v1/_stop

GET /_plugins/_replication/leader_stats
GET /_plugins/_replication/follower_stats
GET /_plugins/_replication/autofollow_stats

Stopping replication changes the follower's relationship; validate the exact current API semantics before using stop as a failover step. Treat promotion as a rehearsed procedure, not a generic “make writable” button.

3. Auto-follow semantics differ from Elastic

OpenSearch replication rules use wildcard patterns. Current OpenSearch documentation states that when a rule is created it begins by replicating existing matching indices as well as automatically following future matching indices. Elastic auto-follow patterns, by contrast, are described around newly created matching indices. This is a concrete portability trap: the same mental model can unexpectedly create more followers on OpenSearch.

POST /_plugins/_replication/_autofollow
{
  "leader_alias": "region_a",
  "name": "atlasmart-products",
  "pattern": "atlasmart-products-*",
  "use_roles": {
    "leader_cluster_role": "atlasmart_ccr_leader",
    "follower_cluster_role": "atlasmart_ccr_follower"
  }
}
GET /_plugins/_replication/autofollow_stats

4. Managed Amazon OpenSearch Service is a different control plane

Amazon OpenSearch Service uses domain connection workflows and IAM/fine-grained access control rather than the self-managed cURL-only setup. Current service docs state that CCR is active-passive and incurs normal inter-domain data-transfer costs. A managed cross-cluster connection can have compatibility and account/Region approval constraints, and a connection previously created as search-only may need recreation for replication.

AWS cross-cluster search documentation also carries service-specific limits such as connection-count/version restrictions and disallows connecting managed domains to self-managed clusters for that service feature. Use AWS documentation, not upstream OpenSearch setup steps, for managed domains.

5. Free/local AtlasMart replication lab

Use two disposable OpenSearch 3.8.0 clusters on one Docker network with the Security plugin enabled and a shared trusted node CA. If your machine cannot host both clusters, use the deterministic trace; do not weaken production conclusions by disabling all security.

A. Leader: create atlasmart-products-v1 and index P-1001..P-1005.
B. Follower: configure region_a remote connection.
C. Start atlasmart-products-dr-v1 replication.
D. Wait until follower count == leader count.
E. Insert P-1006 on leader; record visibility delay on follower.
F. Stop only leader container/network path; record follower read behavior.
G. Restore leader; verify replication catches up.

leader_docs=MEASURED
follower_docs=MEASURED
replication_lag_ms_p95=MEASURED
outage_detection_ms=MEASURED
catchup_ms=MEASURED

Expected invariants: direct writes to the follower fail while replication is active; lag is measurable and non-negative; follower reads can continue during a leader outage if the follower cluster itself is healthy; configuration/security outside the replicated index must still be validated independently.

6. Production judgment

OpenSearch CCR is the practical no-cost replication mechanism for this course, but “free” does not mean operationally free: WAN transfer, follower capacity, observability, PKI, runbooks, and failure rehearsals still cost engineering time and infrastructure. Select only indices whose RPO/RTO justify continuous replication.

References

• Elastic remote clusters — Current connection, security-model, managed-environment, and remote-cluster guidance.
• Elastic remote-cluster connection modes — Sniff versus proxy semantics and network reachability.
• Elastic remote-cluster settings — skip_unavailable, roles, seeds, proxy settings, and defaults.
• Elastic cross-cluster search — Search syntax, optional clusters, response metadata, and supported APIs.
• Elastic cross-cluster replication — Active-passive mechanics, limitations, and DR patterns.
• Elastic auto-follow patterns — Rolling-index/data-stream follower automation.
• Elastic subscriptions — Current cross-cluster licensing boundaries; verify the deployed license before labs.
• OpenSearch cross-cluster search — Security flow, remote connections, permissions, and examples.
• OpenSearch cross-cluster replication — Replication plugin model and operational behavior.
• OpenSearch CCR getting started — Sniff/proxy connectivity, roles, follower startup, and pull replication.
• OpenSearch auto-follow — Pattern-based automatic replication.
• Amazon OpenSearch Service cross-cluster search — Managed-service connection and IAM/network restrictions.
• Amazon OpenSearch Service cross-cluster replication — Managed CCR constraints and domain connection workflow.