Rotate Keys, Back Up Key Material, Test Recovery, and Prevent Encrypted Data from Becoming Unrecoverable

Learning objectives

1. Two “rotations” change different things

CMK rotation / DEK rewrapping changes the master key that protects existing DEKs. The DEK itself can remain the same, so encrypted application data does not need to be rewritten merely because the wrapping key changed. DEK rotation creates a new data key for future ciphertext; existing ciphertext encrypted under the old DEK must remain decryptable until it is explicitly re-encrypted or retired through a deliberate migration. Conflating these procedures can either cause unnecessary data rewrites or destroy access to historical ciphertext.

2. Create encrypted evidence and export recovery artifacts

docker rm -f atlasmart-ch23-l5 2>/dev/null || truedocker volume rm atlasmart-ch23-l5-db 2>/dev/null || truedocker run -d --name atlasmart-ch23-l5 \  -p 127.0.0.1:27185:27017 \  -v atlasmart-ch23-l5-db:/data/db \  mongodb/mongodb-community-server:8.3.8-ubuntu2204-slim \  --bind_ip_all --port 27017until mongosh "mongodb://127.0.0.1:27185/admin" --quiet --eval 'db.runCommand({ping:1}).ok' 2>/dev/null | grep -q 1; do sleep 1; donemongosh "mongodb://127.0.0.1:27185/admin" --quiet --eval '  printjson(db.adminCommand({buildInfo:1}));  printjson(db.adminCommand({getParameter:1,featureCompatibilityVersion:1}));' python -m venv /tmp/atlasmart-ch23-recovery-venv. /tmp/atlasmart-ch23-recovery-venv/bin/activatepython -m pip install --disable-pip-version-check "pymongo[encryption]==4.17.0"python - <<'PY'from pathlib import Pathimport osfor name in ('cmk-v1.bin','cmk-v2.bin'):    p=Path('/tmp')/name; p.write_bytes(os.urandom(96)); os.chmod(p,0o600)print('generated two independent 96-byte local test CMKs')PY

from pathlib import Pathfrom bson.codec_options import CodecOptionsfrom bson import BSONfrom pymongo import MongoClientfrom pymongo.encryption import ClientEncryptionclient=MongoClient("mongodb://127.0.0.1:27185")kv=client.encryption.__keyVaultkv.create_index("keyAltNames", unique=True, partialFilterExpression={"keyAltNames":{"$exists":True}})kms1={"local":{"key":Path('/tmp/cmk-v1.bin').read_bytes()}}ce1=ClientEncryption(kms1,"encryption.__keyVault",client,CodecOptions())key_id=ce1.create_data_key("local",key_alt_names=["atlasmart-recovery-key"])ct=ce1.encrypt("recover-me-2305","AEAD_AES_256_CBC_HMAC_SHA_512-Random",key_id=key_id)client.atlasmart.recovery_demo.replace_one({"_id":"demo"},{"_id":"demo","secret":ct},upsert=True)Path('/tmp/key-vault-backup.bson').write_bytes(BSON.encode(kv.find_one({"_id":key_id})))print('key_id=',key_id)print('ciphertext_type=',type(ct).__name__)print('decrypt_before=',ce1.decrypt(ct))

3. CMK rotation rewraps the DEK; it does not magically rotate data encryption

Current key-management APIs provide rewrapManyDataKey so existing DEKs can be decrypted under the old CMK and re-encrypted under a new CMK. The DEK identifier remains stable. The exact driver method signature differs by language; use the current driver API and KMS provider options. In a production cloud/KMIP flow, rotating the provider CMK and then rewrapping DEKs requires both old-key access during transition and new-key access afterward.

# Continue from the previous script.from pymongo.encryption import ClientEncryptionkms_both={    "local:old":{"key":Path('/tmp/cmk-v1.bin').read_bytes()},    "local:new":{"key":Path('/tmp/cmk-v2.bin').read_bytes()},}# Named providers / rewrap parameters are provider- and driver-version sensitive.# In production, use the exact current ClientEncryption.rewrap_many_data_key API# for AWS/Azure/GCP/KMIP and verify the returned bulk write result.# The invariant to verify is: key_id is unchanged, master-key wrapping changes,# and ciphertext still decrypts with the newly configured CMK path.

4. Prove the catastrophic case safely: wrong CMK means no plaintext

from pathlib import Pathfrom bson.codec_options import CodecOptionsfrom pymongo import MongoClientfrom pymongo.encryption import ClientEncryptionclient=MongoClient("mongodb://127.0.0.1:27185")ct=client.atlasmart.recovery_demo.find_one({"_id":"demo"})["secret"]wrong={"local":{"key":Path('/tmp/cmk-v2.bin').read_bytes()}}try:    ce_wrong=ClientEncryption(wrong,"encryption.__keyVault",client,CodecOptions())    print(ce_wrong.decrypt(ct))except Exception as exc:    print('expected_wrong_key_failure=',type(exc).__name__)correct={"local":{"key":Path('/tmp/cmk-v1.bin').read_bytes()}}ce_ok=ClientEncryption(correct,"encryption.__keyVault",client,CodecOptions())print('restored_plaintext=',ce_ok.decrypt(ct))

5. Backup design: encrypted data, key vault, and CMK recovery are one restore contract

Backing up only encrypted database files is incomplete if the key vault or CMK cannot be recovered. Backing up the CMK next to the database under the same access policy defeats separation. A recovery plan needs independent protected copies, restore ordering, KMS identity/permissions, integrity checks, and a periodic decrypt test using synthetic ciphertext. For cloud KMS providers, key ARN/resource IDs, regions, tenant/project ownership, IAM policy, deletion-protection windows, and network reachability all become part of recovery.

• Back up the key-vault namespace with the same rigor as application data; retain its unique index requirements.
• Protect CMK recovery material independently from database backups and test access from the disaster-recovery environment.
• Record which DEKs protect which fields/collections and which key alternate names applications resolve.
• Before destroying an old DEK, prove all required ciphertext was migrated or intentionally crypto-shredded under approved retention policy.
• After every CMK rotation, run a decrypt smoke test against old and new representative ciphertext before removing old-key access.

6. Production judgment

Encryption creates a new failure mode: perfectly intact ciphertext can become permanently useless. Treat key loss as data loss. CMK rotation should be rehearsed, DEK rotation should have migration/rollback telemetry, key vault backups should be restored in drills, and intentionally destructive crypto-shredding should be governed like a destructive database operation. The next chapter moves below the logical encryption layer into WiredTiger pages, journaling, checkpoints, compression, and cache—mechanisms whose files may themselves be protected by Enterprise/Atlas storage encryption but whose durability behavior must still be understood independently.

Authoritative references

Version-, edition-, driver-, topology-, and preview-sensitive encryption behavior was checked against current MongoDB documentation at generation time. Re-check these sources before applying the procedure to a later release.

• MongoDB Encryption
• Configure Encryption at Rest
• Atlas Encryption at Rest
• Client-Side Field Level Encryption
• CSFLE Compatibility
• CSFLE Encryption Schemas
• Queryable Encryption
• Queryable Encryption Compatibility
• Queryable Encryption Features
• Queryable Encryption Supported Operations
• Encrypted Fields and Enabled Queries
• Create an Encryption Schema
• Explicit Queryable Encryption
• Queryable Encryption KMS Providers
• Rotate and Rewrap Encryption Keys
• Key Vault createKey
• Key Vault getKeyVault
• MongoClient Queryable Encryption Options
• PyMongo CSFLE Upgrade Requirements
• MongoDB 8.3 Release Notes
• mongosh Release Notes
• PyMongo Release Notes