Secrets Rotation, Audit/Connection Evidence, Backups, and Incident Response for Credential Exposure

Learning outcomes

This lesson turns Secrets Rotation, Audit/Connection Evidence, Backups, and Incident Response for Credential Exposure into an observable AtlasMart workflow with explicit correctness, failure, and production boundaries.

1. Problem: a leaked app password is now in a support ticket

Assume the old AtlasMart application password appeared in a support transcript. The incident is not resolved by editing one config file: new connections might still use the old secret, existing pooled connections can remain alive, replicas/Sentinels/Cluster clients may have their own credentials, logs/backups may contain evidence or copies, and TLS private keys have a separate lifecycle. The safe pattern is staged rotation plus verification and containment.

2. Inventory secrets before rotating them

Classify each credential by consumer, storage, transport, rotation method, and blast radius. Redis ACL password hashes, application clear secrets, masterauth, Sentinel auth-pass, TLS private keys, CA keys, managed-service tokens, and cloud credentials are different artifacts. A Redis RDB/AOF backup is not automatically a backup of ACL files, TLS keys, or infrastructure configuration.

3. Add the new app password hash before removing the old one

Redis users can have multiple active passwords, which supports overlap. Add the new SHA-256 verifier while keeping the old verifier. Use the administrator over TLS. ACL SAVE persists effective ACL rules to the configured ACL file on the primary; confirm the file location/permissions before relying on it.

export REDISCLI_AUTH=AtlasMart-Ch22-Admin-Lab-Only-2026A="./ch22-cli.sh"$A ACL SETUSER atlasmart-app '#88b29b2f0e87d8e977e283b2422c50320becdd5ea679acf007870f9d8e174f3a'$A ACL GETUSER atlasmart-app$A ACL SAVECH22_HOST=atlasmart-redis-ch22-replica ./ch22-cli.sh ACL SETUSER atlasmart-app '#88b29b2f0e87d8e977e283b2422c50320becdd5ea679acf007870f9d8e174f3a'CH22_HOST=atlasmart-redis-ch22-replica ./ch22-cli.sh ACL SAVE

4. Prove old and new credentials during the overlap window

The overlap is intentional: both old and new passwords should authenticate while every application instance is being moved. Never infer rollout completeness from one successful client. Inventory deployments, pools, workers, cron jobs, scripts, and disaster-recovery environments.

REDISCLI_AUTH=AtlasMart-Ch22-App-Lab-Only-2026 CH22_USER=atlasmart-app ./ch22-cli.sh PINGREDISCLI_AUTH=AtlasMart-Ch22-App-New-Lab-Only-2026 CH22_USER=atlasmart-app ./ch22-cli.sh PING

5. Remove the old verifier only after consumers move

Use !<sha256> to remove the old ACL password hash, save the ACL file, and test both secrets again. The new password should work; a new authentication attempt with the old one should fail. Removing a password is not proof that every already-open connection disappeared—connection pools and long-lived clients must be managed deliberately.

export REDISCLI_AUTH=AtlasMart-Ch22-Admin-Lab-Only-2026A="./ch22-cli.sh"$A ACL SETUSER atlasmart-app '!a753450b5a9cc3f06436e022127d46e54a44ba7d356801bc72887a7c4d261554'$A ACL SAVECH22_HOST=atlasmart-redis-ch22-replica ./ch22-cli.sh ACL SETUSER atlasmart-app '!a753450b5a9cc3f06436e022127d46e54a44ba7d356801bc72887a7c4d261554'CH22_HOST=atlasmart-redis-ch22-replica ./ch22-cli.sh ACL SAVEREDISCLI_AUTH=AtlasMart-Ch22-App-New-Lab-Only-2026 CH22_USER=atlasmart-app ./ch22-cli.sh PINGREDISCLI_AUTH=AtlasMart-Ch22-App-Lab-Only-2026 CH22_USER=atlasmart-app ./ch22-cli.sh PING || true

6. Decide what to do with already-authenticated connections

Credential revocation primarily blocks future AUTH with the removed password. During an incident, inventory CLIENT LIST by user/address/name and decide whether to kill/recycle affected application connections so they must authenticate again. CLIENT KILL USER atlasmart-app is powerful: use it only after the new credential is deployed and after modeling the reconnect storm/availability impact.

REDISCLI_AUTH=AtlasMart-Ch22-Admin-Lab-Only-2026 ./ch22-cli.sh CLIENT LIST# Incident-only after staged rollout, if you intentionally need to force re-authentication:# ... CLIENT KILL USER atlasmart-app

7. Rotate replication/Sentinel/Cluster credentials without breaking availability

Replication credentials require the new user/password to be accepted by the primary before replicas switch masteruser/masterauth. Sentinel credentials must exist across all monitored data nodes before changing sentinel auth-user/auth-pass. Cluster application credentials must be compatible across every node a client may reach after MOVED/ASK. Use overlapping credentials where supported, switch one dependency class at a time, and verify topology health after each stage.

8. Certificate exposure is a different incident

If a client or server private key is exposed, ACL password rotation is not enough. Revoke/remove trust as your PKI supports, issue a new certificate/key, stage trust, update all relevant Redis planes, reconnect clients, and remove old trust. If the CA private key is exposed, assume every certificate it can issue is suspect and execute your CA-compromise plan.

9. Inspect configs/backups for secret-bearing material

The Chapter 22 replica config intentionally contains a public disposable masterauth; production configs can therefore be secret-bearing. Inspect ACL files/configuration manifests, backup bundles, CI artifacts, crash bundles, support uploads, shell history, and logs. Do not copy TLS private keys into general data backups by accident. Conversely, do not assume an RDB/AOF restore will recreate ACL/TLS/infrastructure state.

grep -nE 'masterauth|auth-pass|requirepass|tls-key-file|aclfile' primary.conf replica.conf 2>/dev/null || truels -l users.acl ch22-certs 2>/dev/null || truedocker exec atlasmart-redis-ch22-primary ls -l /data/users.acldocker exec atlasmart-redis-ch22-replica ls -l /data/users.acl# ACL GETUSER returns password hashes, not the original clear password.REDISCLI_AUTH=AtlasMart-Ch22-Admin-Lab-Only-2026 ./ch22-cli.sh ACL GETUSER atlasmart-app

10. ACL LOG and connection state are incident clues, not forensic completeness

Capture recent ACL LOG, relevant CLIENT LIST entries, Redis/security logs, configuration/version evidence, and timestamps before resetting diagnostics. ACL LOG records recent ACL violations but is bounded and mutable. A production incident process should export logs/metrics externally with access controls and retention appropriate to your requirements.

11. Credential-exposure response sequence

A useful runbook is: scope the exposed credential/cert and reachable Redis endpoints → contain network/identity access → preserve evidence → introduce new credentials/trust → roll consumers safely → revoke old material → force/recycle sessions if required → verify data/topology/permissions → search downstream copies → document root cause and prevention. Do not “rotate fast” by breaking replication/Sentinel or deleting evidence before capture.

12. Production judgment and chapter close

Redis security is a layered operating discipline, not a password toggle. Review ACLs with each application/schema/module upgrade, rotate secrets/certificates before expiry or after exposure, keep Redis off untrusted networks, test TLS on every relevant plane, preserve break-glass access securely, and restore-test the security/configuration artifacts that are separate from data persistence. Chapter 23 builds application patterns on top of these boundaries; caching, locks, rate limits, and idempotency are unsafe if attackers or overprivileged workloads can bypass the intended key/command model.

docker rm -f atlasmart-redis-ch22-acl 2>/dev/null || truedocker compose -f ch22-compose.yaml down -v --remove-orphans 2>/dev/null || truedocker volume rm atlasmart-redis-ch22-acl-data 2>/dev/null || true# Delete only your local ch22-lab/ directory after exporting any evidence you need.# Never use docker system prune for this lab.

Summary and next step

Secrets Rotation, Audit/Connection Evidence, Backups, and Incident Response for Credential Exposure is now connected to observable Redis behavior, bounded failure cases, and production tradeoffs. Keep the evidence and cleanup state from this lesson; next, continue with Cache-Aside with TTL/Jitter, Negative Caching, Stampede Protection, and Stale-While-Revalidate.

Authoritative references

• Redis security
• Redis ACL tutorial
• ACL SETUSER
• ACL GETUSER
• ACL LIST
• ACL WHOAMI
• ACL CAT
• ACL DRYRUN
• ACL LOG
• ACL SAVE
• ACL LOAD
• AUTH
• Redis TLS
• CLIENT LIST
• CLIENT KILL
• CONFIG GET
• CONFIG SET
• MODULE LOAD
• Redis replication — ACL authentication
• Redis Sentinel — ACL authentication
• Redis Cluster specification
• Redis 8.10 release notes
• Redis 8.10 — what is new
• Redis licenses
• Docker Official Image — Redis