Checkpoint Lab — Backup, Restore, Configuration Export, Blob Recovery, Disaster Scenarios, and Recovery Validation

1. Scenario and objectives

Your team operates a small internal artifact service. At 10:20, an operator discovers that a destructive maintenance action removed the current release artifact. The approved recovery objective is:

• RPO: 30 minutes — recovery point must be 09:50 or newer.
• RTO: 120 minutes — validated service must be available by 12:20.
• Critical content: learner-example/app/2.0.0/app-2.0.0.bin.
• Required configuration: repository ch25-release-hosted maps to blob store ch25-file; node identity and synthetic role metadata must survive.

The mandatory lab is a local fixture. An optional disposable Nexus H2 implementation may repeat the drill using the current documented backup/restore procedure.

2. Write predictions before action

1. If only the live blob is deleted, metadata validation must detect a missing binary even though the database record still exists.
2. Restoring from a verified 10:00 recovery set should meet the 30-minute RPO for a 10:20 incident and should restore the same SHA-256.
3. Copying only the database directory into the recovery target must fail acceptance because the critical blob will be absent.

Save these to ch25-drill/evidence/01-predictions.md.

3. Create the source instance fixture

from pathlib import Path
import hashlib, json, shutil
root=Path('ch25-drill/source')
if root.parent.exists(): shutil.rmtree(root.parent)
(root/'db').mkdir(parents=True); (root/'blobs/learner-example/app/2.0.0').mkdir(parents=True)
(root/'keystores/node').mkdir(parents=True); (root/'etc').mkdir()
artifact=root/'blobs/learner-example/app/2.0.0/app-2.0.0.bin'
artifact.write_bytes(b'chapter25-release-2.0.0
')
sha=hashlib.sha256(artifact.read_bytes()).hexdigest()
(root/'db/state.json').write_text(json.dumps({
  'repositories':[{'name':'ch25-release-hosted','blobStore':'ch25-file'}],
  'roles':[{'id':'ch25-reader','privileges':['read:ch25-release-hosted']}],
  'assets':[{'path':'learner-example/app/2.0.0/app-2.0.0.bin','sha256':sha}]
}, indent=2)+'
')
(root/'keystores/node/id').write_text('NODE-CH25-DRILL
')
(root/'etc/runtime.properties').write_text('version=3.95.2-01
java=21
')
print('critical sha256:', sha)

4. Create the 10:00 recovery set and immutable-style manifest

from pathlib import Path
import hashlib, json, shutil
src=Path('ch25-drill/source'); b=Path('ch25-drill/backup-1000')
shutil.copytree(src, b/'payload')
files=[]
for p in sorted((b/'payload').rglob('*')):
    if p.is_file(): files.append({'path':p.relative_to(b/'payload').as_posix(),'sha256':hashlib.sha256(p.read_bytes()).hexdigest()})
manifest={'recoveryPoint':'2026-08-27T10:00:00Z','rpoMinutes':30,'rtoMinutes':120,
          'nexusVersion':'3.95.2-01','database':'H2-model','blobStores':['ch25-file'],'files':files}
(b/'manifest.json').write_text(json.dumps(manifest,indent=2)+'
')
print('manifest entries:',len(files))

In production, backup media and manifests should be protected by access controls, encryption, and preferably deletion-resistant retention appropriate to your threat model.

5. Gate: verify the recovery set before declaring it usable

from pathlib import Path
import hashlib, json
b=Path('ch25-drill/backup-1000'); m=json.loads((b/'manifest.json').read_text())
issues=[]
for item in m['files']:
    p=b/'payload'/item['path']
    if not p.exists(): issues.append([item['path'],'missing'])
    elif hashlib.sha256(p.read_bytes()).hexdigest()!=item['sha256']: issues.append([item['path'],'hash'])
assert not issues, issues
print('recovery-set gate: PASS')

6. Inject the controlled incident

from pathlib import Path
victim=Path('ch25-drill/source/blobs/learner-example/app/2.0.0/app-2.0.0.bin')
assert victim.exists(); victim.unlink()
print('INCIDENT: deleted only disposable critical blob')

Now run the source validator from Lesson 2 logic. It should report a missing binary. Capture that output as 02-incident-validation.txt.

7. Demonstrate the wrong recovery: database only

from pathlib import Path
import shutil
bad=Path('ch25-drill/bad-target')
if bad.exists(): shutil.rmtree(bad)
(bad/'db').mkdir(parents=True)
for p in Path('ch25-drill/backup-1000/payload/db').iterdir(): shutil.copy2(p,bad/'db'/p.name)
print('bad target intentionally contains DB only')

Do not accept this target. The repository definition may be represented, but the critical artifact cannot be served. Record why the attempt fails the acceptance criteria, then leave it intact as evidence.

8. Perform the correct isolated restore

from pathlib import Path
import shutil
src=Path('ch25-drill/backup-1000/payload'); target=Path('ch25-drill/restored-target')
if target.exists(): shutil.rmtree(target)
shutil.copytree(src,target)
print('full recovery set restored')

9. Execute the acceptance validator

from pathlib import Path
import hashlib, json, time
start=time.monotonic(); target=Path('ch25-drill/restored-target')
state=json.loads((target/'db/state.json').read_text())
assert state['repositories']==[{'name':'ch25-release-hosted','blobStore':'ch25-file'}]
assert state['roles'][0]['id']=='ch25-reader'
assert (target/'keystores/node/id').read_text().strip()=='NODE-CH25-DRILL'
for a in state['assets']:
    p=target/'blobs'/a['path']; assert p.exists(), a['path']
    assert hashlib.sha256(p.read_bytes()).hexdigest()==a['sha256'], a['path']
print('ACCEPTANCE: PASS')
print('validationSeconds:', round(time.monotonic()-start,3))

For a live Nexus drill, add controlled client requests using a least-privilege reader, repository browse/search evidence, task/log inspection, and the pre-incident artifact checksum.

10. Evaluate RPO explicitly

Incident detection is 10:20; recovery point is 10:00. The potential loss window is 20 minutes, which is within the 30-minute RPO. This does not mean every write between 10:00 and 10:20 is acceptable to lose operationally; it means the declared objective allows it. Record any business-critical publication that occurred after the recovery point as a gap requiring replay/republication from trusted build evidence.

11. Evaluate RTO with measured timestamps

For the fixture, elapsed time is tiny and therefore not representative. In production, record incident declaration, backup selection, infrastructure provisioning, DB restore, blob restore, Nexus startup, reconciliation, validation, DNS/traffic cutover, and service acceptance timestamps. The sum—not the copy command duration—is your demonstrated recovery time.

12. Optional live Nexus drill

On a disposable H2 archive installation only:

1. Publish a harmless raw asset and record its SHA-256.
2. Capture repository/blob configuration, node-ID backup, and the exact 3.95.2-01/Java 21 baseline.
3. Create/run Admin - Backup H2 Database; back up the associated file blob store and required configuration/node ID as one declared recovery set. Prefer a maintenance-window/offline embedded DB backup as the stronger periodic method per Sonatype guidance.
4. Stop Nexus and preserve the original disposable data directory.
5. Restore into a separate disposable target using the documented H2 restore procedure and corresponding blob backup.
6. Start the restored target, confirm repository/security state, download the known asset, and compare SHA-256.
7. If you intentionally created bounded DB/blob skew, generate/review the current repair plan before executing anything.

13. Required evidence bundle

• 01-predictions.md
• 02-source-manifest.json
• 03-backup-verification.txt
• 04-incident-validation.txt
• 05-bad-restore-result.md
• 06-good-restore-log.txt
• 07-acceptance-validation.txt
• 08-rpo-rto.md
• 09-security-and-secrets-review.md
• 10-runbook-gaps.md

14. Record what this lab does not prove

The fixture proves reasoning and validation mechanics, not production-scale throughput, PostgreSQL PITR, object-store recovery, KMS/secret restoration, HA behavior, reverse-proxy/DNS cutover, or multi-terabyte restore times. A mature runbook turns each of those unknowns into a tested dependency or an explicit risk.

15. Cleanup / rollback

Keep the recovery report, then delete only the disposable ch25-drill tree. For the optional live drill, decommission only the isolated target after saving sanitized evidence. Never destroy the only known-good backup to “clean up” the exercise. Production backup retention follows policy, not lab convenience.

16. Knowledge check

17. Chapter summary and bridge to Chapter 26

You now have a production-oriented recovery model: define RPO/RTO, capture database/configuration and blob content as one recovery set, protect node identity and custom configuration, verify backups before loss, restore into isolation, prove critical bytes and authorization/client paths, and record gaps. Chapter 26 builds on that discipline to handle database and instance migration without confusing migration with backup or rollback.

Official references and version notes

• Sonatype: Backup and Restore — embedded H2 backup-task behavior and the role of database snapshots.
• Sonatype: Prepare a Backup — blob-store, node-ID, database, and custom-configuration backup requirements.
• Sonatype: Configure the Backup Task — current H2 task name and the recommendation for periodic offline embedded-database backups.
• Sonatype: Restore an H2 Database — stop/restore/restart sequence and same-point blob-store requirement.
• Sonatype: Nexus Repository Database — H2/PostgreSQL boundaries and PostgreSQL backup responsibilities.
• PostgreSQL: Backup and Restore — database-native backup methods for external PostgreSQL.
• Sonatype: Storage Guide — blob-store layout and the warning not to modify internal blob files manually.
• Sonatype: Data Repair Tasks — current plan-based reconciliation for database/blob inconsistencies and supported formats.
• Sonatype: Repository Export — Pro-only content export and its operational limits.
• Sonatype: Repository Import — Pro-only content import, generated metadata, and what is not preserved.
• Sonatype: Backup/Same-Site Restore — RPO/RTO and test-restore expectations.
• Sonatype: Resiliency — recovery expectations after database failure and the distinction from HA.
• Sonatype: Nexus Repository 3.95.x Release Notes — dated version line used as the chapter reference.