Build a Migration Runbook with Data Validation, Relevance Regression, Performance Baselines, RPO, and Rollback Triggers

Learning outcomes

1. AtlasMart migration runbook: approval gates, not a checklist you can ignore

A runbook is executable only if each step has an owner, command/test, expected evidence, stop condition, and rollback action. “Validate target” is not executable. “Run golden-query suite v3; require zero authorization escapes, NDCG@10 degradation ≤ agreed threshold, and p99 within the approved SLO at load level L2” is executable.

2. Phase 0 — freeze the baseline

3. Phase 1 — create target contracts before copying data

Create destination mappings, analyzers, templates, lifecycle policy, security roles, and aliases independently. Do not let automatic index creation decide production mapping. Translate non-portable features explicitly in the compatibility ledger.

All Chapter 29 labs preserve the established local endpoints and security assumptions: Elasticsearch 9.5.3 at https://localhost:9200 with ELASTIC_PASSWORD and the copied CA file atlasmart-es-http-ca; OpenSearch 3.8.0 at https://localhost:9201 with OPENSEARCH_INITIAL_ADMIN_PASSWORD. The shared Docker network remains atlasmart-search. OpenSearch's demo certificate trust bypass (-k) is acceptable only for this disposable local lab. The migration fixture uses one primary and zero replicas to fit a single workstation; production redundancy, recovery headroom, and managed-service networking must be designed separately.

PUT atlasmart-migrate-source-v1
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "sku":       {"type":"keyword"},
      "tenant_id": {"type":"keyword"},
      "name":      {"type":"text", "fields":{"raw":{"type":"keyword"}}},
      "category":  {"type":"keyword"},
      "price":     {"type":"scaled_float", "scaling_factor":100},
      "available": {"type":"boolean"},
      "updated_at": {"type":"date"}
    }
  }
}

POST _bulk?refresh=wait_for
{"index":{"_index":"atlasmart-migrate-source-v1","_id":"P-1001"}}
{"sku":"P-1001","tenant_id":"tenant-a","name":"Waterproof Hiking Boot","category":"footwear","price":129.90,"available":true,"updated_at":"2026-09-12T10:00:00Z"}
{"index":{"_index":"atlasmart-migrate-source-v1","_id":"P-1002"}}
{"sku":"P-1002","tenant_id":"tenant-a","name":"Trail Running Shoe","category":"footwear","price":99.50,"available":true,"updated_at":"2026-09-12T10:01:00Z"}
{"index":{"_index":"atlasmart-migrate-source-v1","_id":"P-1003"}}
{"sku":"P-1003","tenant_id":"tenant-a","name":"Insulated Water Bottle","category":"outdoor","price":32.00,"available":true,"updated_at":"2026-09-12T10:02:00Z"}
{"index":{"_index":"atlasmart-migrate-source-v1","_id":"P-1004"}}
{"sku":"P-1004","tenant_id":"tenant-b","name":"Lightweight Hiking Pack","category":"outdoor","price":74.00,"available":false,"updated_at":"2026-09-12T10:03:00Z"}
{"index":{"_index":"atlasmart-migrate-source-v1","_id":"P-1005"}}
{"sku":"P-1005","tenant_id":"tenant-b","name":"Merino Hiking Sock","category":"apparel","price":18.50,"available":true,"updated_at":"2026-09-12T10:04:00Z"}

For the lab, source is atlasmart-migrate-source-v1; target is atlasmart-migrate-target-v1. The exact source/target product direction can be reversed because the neutral ETL harness uses the shared search/Bulk subset.

4. Phase 2 — backfill and reconcile

# Replace auth/TLS flags with the source or target product's established lab pattern.
GET atlasmart-migrate-source-v1/_count
GET atlasmart-migrate-target-v1/_count

POST atlasmart-migrate-source-v1/_search
{"size":0,"aggs":{"max_update":{"max":{"field":"updated_at"}},"tenants":{"terms":{"field":"tenant_id"}}}}

POST atlasmart-migrate-target-v1/_search
{"size":0,"aggs":{"max_update":{"max":{"field":"updated_at"}},"tenants":{"terms":{"field":"tenant_id"}}}}

Counts and aggregate checkpoints must agree, then sample/canonical-hash selected fields by ID. If production supports writes during backfill, continue incremental sync until measured lag satisfies the cutover RPO.

5. Phase 3 — dual-read behavioral validation

{
  "query_id":"catalog-hiking-tenant-a",
  "source_top_ids":["P-1001","P-1002"],
  "target_top_ids":["P-1001","P-1002"],
  "source_metrics":{"ndcg_at_3":"MEASURE","p99_ms":"MEASURE"},
  "target_metrics":{"ndcg_at_3":"MEASURE","p99_ms":"MEASURE"},
  "auth_negative_tests":{"tenant_b_visible":false,"write_allowed":false},
  "decision":"PASS_OR_FAIL_BY_DECLARED_THRESHOLDS"
}

Do not compare raw relevance scores across products as if they were calibrated. Compare judged ranking outcomes and user/business invariants.

6. Phase 4 — performance and saturation gate

Use the same dataset, cache/warmup disclosure, request mix, concurrency, TLS/auth, and hardware class where possible. Capture p50/p95/p99, throughput, errors/rejections, CPU, heap/GC, disk/network, merge/recovery, and freshness. The target need not reproduce the source internal metrics; it must satisfy the approved user-visible and recovery objectives.

7. Phase 5 — security gate

reader identity:
  ALLOW search tenant-a catalog
  DENY index/update/delete
  DENY tenant-b documents
  DENY cluster admin and snapshot operations

ingester identity:
  ALLOW required write/bulk target
  DENY broad read if application does not need it
  DENY security/plugin administration

migration identity:
  ALLOW only source-read + target-write operations required for migration
  EXPIRE/REVOKE after migration

Security parity means equivalent least-privilege intent, not identical role JSON.

8. Phase 6 — cutover gate

9. Rollback triggers must be machine-readable where possible

cutover_id: atlasmart-search-2026-09
rollback_window: "4h"
triggers:
  - "authorization escape or cross-tenant visibility"
  - "sustained application error rate above approved SLO"
  - "p99 above approved SLO for two consecutive windows"
  - "sync/reconciliation gap above RPO"
  - "critical relevance regression in golden suite"
rollback_actions:
  - "stop/redirect target writes according to write-ownership plan"
  - "replay target-only writes to source if required"
  - "restore source endpoint/service discovery"
  - "verify source smoke/security/relevance checks"
  - "preserve target evidence for incident analysis"

10. Do not delete the source at cutover

The rollback window closes only after the target has accumulated enough evidence under real traffic and source recovery is no longer required. Source retirement includes credentials, snapshots, repositories, monitoring, DNS, client configuration, data-retention obligations, and cost controls—not merely deleting the cluster.

11. Chapter 29 lab rehearsal

1. Create the portable source and target mappings on opposite products.
2. Run the five-document neutral ETL backfill.
3. Compare counts, per-tenant buckets, selected fields by ID, and the golden hiking query.
4. Translate one lifecycle intent (ILM↔ISM) on paper or in the disposable local clusters; do not copy JSON blindly.
5. Create least-privilege reader/ingester test identities using each product’s own security model.
6. Point a tiny test client at source, then target. Record response and latency distributions; do not fabricate values.
7. Inject one failure: remove target read permission or alter a query mapping. Confirm the rollback trigger fires and restore source endpoint/config.
8. Only after repair, rerun the complete gate set.

12. Migration evidence package

13. Bridge to Chapter 30: migration numbers become capacity inputs

Chapter 30 begins with Workload Characterization: QPS, Write Rate, Document Size, Fields, Retention, Aggregations, Vector Dimensions, and Concurrency. The migration performance report is not merely a go/no-go artifact; it becomes a measured workload baseline for capacity planning. Rebenchmark when query mix, mapping, shard layout, vectors, retention, hardware, or product version changes.

References and current-version checks

• Elastic Stack 9.5.3 release
• Elasticsearch snapshot and restore compatibility
• Elasticsearch restore snapshot guidance
• Elasticsearch reindex and reindex-from-remote
• Elasticsearch reindex settings
• Elasticsearch Python client compatibility
• Elasticsearch Python client release notes
• OpenSearch 3.8 version history
• OpenSearch upgrade or migrate guidance
• OpenSearch Reindex Documents API
• OpenSearch reindex data guidance
• OpenSearch language clients and compatibility
• OpenSearch Migration Assistant
• Migration Assistant supported migration paths
• Amazon OpenSearch Service snapshot migration