Run a Restore Drill and Measure RPO/RTO, Data Completeness, Security State, Templates, and Application Cutover

Learning outcomes

The only defensible statement that “AtlasMart can recover in four hours with at most one hour of data loss” is a measured restore drill with evidence. This lesson turns the chapter into a runbook: capture a recovery point, create a known post-snapshot change, simulate source loss, restore under an isolated name, validate data/configuration/security assumptions, cut an alias deliberately, measure RPO/RTO, and prove rollback.

1. Define the clock before the incident

RPO measures recoverable data age at the failure boundary. RTO measures how long service recovery takes from declared incident until the agreed service acceptance gate passes. Decide what “service recovered” means: cluster green? index searchable? API smoke tests passing? full p95/p99 SLO restored? Different definitions produce different RTOs.

observed_rpo = T_incident - T_last_recoverable_event
observed_data_restore_time = T_data_ready - T_restore_start
observed_rto = T_app_ready - T_incident

# Do not substitute:
# schedule_interval for observed_rpo
# restore_API_ack for T_app_ready

2. Prepare recovery point and deliberate post-snapshot evidence

Use snapshot 002 from Lesson 3 or create a fresh snapshot. Then write one document after snapshot completion. That post-snapshot document acts as a visible RPO marker: an ordinary restore of the snapshot should not contain it.

PUT atlasmart-dr-products-v1/_doc/P-AFTER-SNAPSHOT?refresh=wait_for
{
  "sku":"AM-DR-AFTER",
  "name":"Post Snapshot Marker",
  "category":"dr-test",
  "price":1.00,
  "updated_at":"2026-09-11T10:05:00Z"
}

GET atlasmart-dr-products-v1/_doc/P-AFTER-SNAPSHOT

Record the exact snapshot completion time and marker event time. In a real system, the newest recoverable business event should come from a canonical event/transaction sequence, not from wall-clock guessing alone.

3. Simulate source loss safely

The safest drill is to leave the source intact and pretend it is unavailable to the recovery operator. If you want deletion semantics, clone the fixture first and delete only the disposable clone. The goal is to practice restore, not to manufacture real data loss.

POST _aliases
{
  "actions": [
    {"add": {"index":"atlasmart-dr-products-v1", "alias":"atlasmart-dr-products-read"}},
    {"add": {"index":"atlasmart-dr-products-v1", "alias":"atlasmart-dr-products-write", "is_write_index":true}}
  ]
}

GET _alias/atlasmart-dr-products-read,atlasmart-dr-products-write

Clients in the drill should use the aliases, not the physical index name. This gives the recovery procedure an explicit cutover boundary and lets rollback remain an alias operation instead of a destructive restore-overwrite.

4. Restore the snapshot under an isolated name and time it

# Record T_restore_start immediately before this request.
POST _snapshot/atlasmart-es-fs/atlasmart-dr-2026.09.11-002/_restore?wait_for_completion=true
{
  "indices":"atlasmart-dr-products-v1",
  "include_global_state":false,
  "include_aliases":false,
  "rename_pattern":"atlasmart-dr-products-v1",
  "rename_replacement":"atlasmart-dr-products-drill-002"
}

# Then inspect recovery, health and index state.
GET _cluster/health/atlasmart-dr-products-drill-002
GET atlasmart-dr-products-drill-002/_recovery?active_only=false
GET atlasmart-dr-products-drill-002/_count

Expected invariant: restore completes without mutating the source index or aliases. The exact response timing is environment-specific. wait_for_completion=true measures API completion for the restore operation, but your RTO continues until application validation and cutover gates pass.

5. Validate completeness and the intentional RPO gap

GET atlasmart-dr-products-drill-002/_doc/P-1001
GET atlasmart-dr-products-drill-002/_doc/P-1006

# The marker was written AFTER snapshot 002 and should be absent.
GET atlasmart-dr-products-drill-002/_doc/P-AFTER-SNAPSHOT

GET atlasmart-dr-products-drill-002/_mapping
GET atlasmart-dr-products-drill-002/_settings

GET atlasmart-dr-products-drill-002/_search
{
  "size": 0,
  "aggs": {
    "by_category": {"terms": {"field":"category","size":10}},
    "price_sum": {"sum": {"field":"price"}}
  }
}

The missing post-snapshot marker is not a restore bug; it is a visible representation of the recovery point. If the documented RPO is smaller than that gap, the design needs more frequent snapshots, external change replay, or another durability mechanism.

6. Validate dependencies outside the restored index

Elasticsearch feature states and OpenSearch Security-plugin configuration require product-specific privileged handling. The mandatory lab intentionally does not restore them. Instead, record whether the DR environment obtains security configuration from infrastructure-as-code, identity provider integration, separate secure backups, or a product-supported security-state recovery process.

7. Atomic application cutover and smoke tests

POST _aliases
{
  "actions": [
    {"remove": {"index":"atlasmart-dr-products-v1", "alias":"atlasmart-dr-products-read"}},
    {"add":    {"index":"atlasmart-dr-products-drill-002", "alias":"atlasmart-dr-products-read"}}
  ]
}

GET _alias/atlasmart-dr-products-read
GET atlasmart-dr-products-read/_search
{
  "query": {"match": {"name":"wireless headphones"}},
  "size": 3
}

If the recovered index is intended to resume writes, do not simply repoint the write alias until you have decided how to reconcile post-snapshot writes, mapping/version constraints and write ownership. In many DR designs the restored cluster becomes authoritative only after an explicit incident command and application-level replay/catch-up step.

8. Rollback evidence

POST _aliases
{
  "actions": [
    {"remove": {"index":"atlasmart-dr-products-drill-002", "alias":"atlasmart-dr-products-read"}},
    {"add":    {"index":"atlasmart-dr-products-v1", "alias":"atlasmart-dr-products-read"}}
  ]
}

Rollback is only possible because the drill preserved the original source. In a real regional disaster the old cluster may be gone; “rollback” could instead mean abandon the failed target and retry restore into another target. Write that distinction into the runbook.

9. Disaster-recovery report

drill_id: atlasmart-ch18-2026-09-11
source_product_version: ...
target_product_version: ...
repository: ...
snapshot: ...
snapshot_start: ...
snapshot_end: ...
last_recoverable_business_event: ...
incident_time: ...
restore_start: ...
data_ready: ...
application_ready: ...
observed_rpo: ...
observed_rto: ...
source_count: ...
restored_count: ...
mapping_diff: PASS|FAIL
query_fixture: PASS|FAIL
template_check: PASS|FAIL
alias_cutover: PASS|FAIL
security_assumption: documented
post_snapshot_marker_absent: EXPECTED|UNEXPECTED
rollback_test: PASS|FAIL
open_actions:
  - ...

10. Chapter completion criteria

Chapter 18 is complete only when AtlasMart can point to a named snapshot, prove its repository is reachable, restore it into an isolated target, explain exactly what global/security state was excluded, demonstrate the expected RPO gap, pass data/schema/application checks, execute cutover and rollback, and report measured RPO/RTO. This hands off naturally to Chapter 19: the recovered service must now be secured with correct TLS, identities, roles, API keys and audit controls.

References

• Elastic snapshot and restore — Snapshot contents, incremental segment reuse, compatibility, and deployment-specific behavior.
• Elastic create/monitor snapshots and SLM — SLM schedules, retention, snapshot status, feature-state backup, and operational monitoring.
• Elastic restore a snapshot — Restore prerequisites, rename-on-restore, capacity, compatibility, and cross-cluster cautions.
• Elastic searchable snapshots — Enterprise-licensed searchable-snapshot behavior and repository dependency.
• OpenSearch snapshot and restore — Incremental snapshots, repository types, restore compatibility, and security constraints.
• OpenSearch Snapshot Management — Scheduled snapshot creation/deletion, failure metadata, plugin and security requirements.
• OpenSearch Snapshot Management API — SM policy schema, explain state, start/stop, schedules, retention and OCC updates.
• OpenSearch remote-backed storage — Remote segment/translog storage and remote-store recovery concepts.
• OpenSearch release artifacts — Pinned OpenSearch release baseline.
• Elasticsearch downloads — Pinned Elasticsearch release baseline.