Cascading Replication, Replication Slots, WAL Retention, and Replica Lag Diagnosis

Learning outcomes

ServiceHub now has a direct standby, but operators need a more scalable topology and a reliable lag playbook. The dangerous shortcut is a single dashboard value called “replica lag.” Physical replication is a pipeline: generation → sender → network → receiver write → receiver flush → replay → query visibility. Different failures separate different stages.

1. Cascading replication changes who sends WAL

Instead of every standby connecting to the primary, a downstream standby can stream from an upstream standby. The upstream standby must accept replication connections and have enough max_wal_senders. The downstream's primary_conninfo points to the upstream standby, not the original primary.

primary:55436  └── standby_a:55437  (streams directly from primary)        └── standby_b:55438  (streams from standby_a)

primary_conninfo = 'host=localhost port=55437 user=ch14_repl application_name=servicehub_cascade2'primary_slot_name = 'ch14_cascade_slot'

On the primary, pg_stat_replication shows standby_a only. On standby_a, its own pg_stat_replication shows standby_b. This monitoring boundary matters during incidents: absence from the primary view does not mean a cascade does not exist.

2. Slot placement follows the sender that must retain WAL

If standby_b streams from standby_a, the slot protecting standby_b's required WAL belongs on standby_a. A slot on the original primary cannot force standby_a to retain downstream WAL once it is the sender.

SELECT * FROM pg_create_physical_replication_slot('ch14_cascade_slot');SELECT slot_name, active, restart_lsn, wal_status,       safe_wal_size, inactive_since, invalidation_reasonFROM pg_replication_slotsWHERE slot_name='ch14_cascade_slot';

Physical slots can exist on a standby where supported by the current configuration/feature set, but your topology must preserve them across promotion or recreate the retention relationship deliberately. Do not assume “slot exists somewhere” protects every chain segment.

3. Measure byte distance, not only interval lag

SELECT application_name, state, sync_state,       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn))   AS not_sent,       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), write_lsn))  AS not_written_remote,       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), flush_lsn))  AS not_flushed_remote,       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn)) AS not_replayed_remote,       write_lag, flush_lag, replay_lagFROM pg_stat_replication;

Byte distance tells you how much WAL separates positions. Lag intervals tell you how long recent acknowledgment stages took. PostgreSQL explicitly warns that write_lag/flush_lag/replay_lag are not catch-up-time predictions.

4. Diagnose by pipeline signature

SELECT status, written_lsn, flushed_lsn, latest_end_lsn,       last_msg_send_time, last_msg_receipt_time,       sender_host, sender_port, slot_nameFROM pg_stat_wal_receiver;SELECT pg_last_wal_receive_lsn(),       pg_last_wal_replay_lsn(),       pg_last_xact_replay_timestamp();SELECT datname, confl_snapshot, confl_lock, confl_bufferpin,       confl_deadlock, confl_tablespaceFROM pg_stat_database_conflicts;

5. Replay conflict versus slow I/O

A large receive-to-replay gap plus increasing pg_stat_database_conflicts suggests recovery conflicts. The same gap without conflict growth can instead come from replay CPU, storage, full-page images, index maintenance, or a bursty primary workload. Correlate with standby wait events and I/O metrics; do not “solve” every replay gap by enabling feedback.

SELECT pid, backend_type, state, wait_event_type, wait_eventFROM pg_stat_activityWHERE backend_type IN ('startup','walreceiver')ORDER BY backend_type;

6. Slot danger states in PostgreSQL 18

wal_status can move through reserved, extended, unreserved, and lost depending on configured retention limits and WAL removal. safe_wal_size estimates how many more bytes may be written before the slot risks becoming lost when a finite max_slot_wal_keep_size applies. inactive_since shows when use stopped.

SELECT slot_name, slot_type, active, restart_lsn,       wal_status,       pg_size_pretty(safe_wal_size) AS safe_wal_remaining,       inactive_since, invalidation_reasonFROM pg_replication_slotsWHERE slot_type='physical'ORDER BY active, inactive_since NULLS LAST;

If wal_status='lost', that slot can no longer supply its required WAL. Reinitializing the standby from a fresh base backup is often safer than trying to improvise around missing physical WAL.

7. Cascading failure semantics

If the primary is healthy but standby_a fails, standby_b loses its upstream even though the original primary still runs. A well-designed topology documents whether standby_b can be re-pointed to another upstream, whether its needed WAL is available there, and how recovery timelines will be followed after promotion. recovery_target_timeline='latest' is the normal failover-friendly behavior.

8. Retention runway is a rate problem

A slot's retained byte distance becomes operationally urgent when combined with the current WAL generation rate and free disk. Ten gigabytes of retained WAL can be harmless on one system and minutes from failure on another. Measure generation over representative intervals and derive a runway; do not alert on a universal retained-byte threshold.

SELECT pg_current_wal_insert_lsn() AS start_lsn \gset\echo 'Run the representative workload for a measured interval.'SELECT pg_size_pretty(         pg_wal_lsn_diff(pg_current_wal_insert_lsn(), :'start_lsn'::pg_lsn)       ) AS generated_wal;

Combine this with filesystem free space and slot safe_wal_size. If a consumer is intentionally offline, document how long it can remain offline before rebuild becomes preferable to continued retention.

9. Cascading reduces primary fan-out but adds failure domains

Cascading can reduce the primary's connection count and inter-site bandwidth, but downstream availability now depends on the upstream relay. A deep chain also compounds replay delay: downstream cannot receive WAL before the upstream has received enough to stream it onward. Keep chain depth justified by topology, not aesthetics.

When an upstream standby is promoted, downstream nodes can follow its new timeline when configured to follow the latest timeline. When a different node is promoted, downstream connection strings may need rerouting. Test both planned switchover and unplanned failover paths rather than assuming the cascade will rewire itself.

10. Differentiate lag from data loss risk

Lag is an observed distance in the current stream. Data-loss risk depends on the durability contract and which candidate would be promoted. An asynchronous standby can have zero current lag but still offer no guarantee that every future acknowledged primary commit will reach it before a sudden primary crash. Conversely, a synchronous standby can be temporarily replay-lagged while still having durably flushed acknowledged WAL. Keep “RPO protection” and “query freshness” as separate dashboards.

SELECT application_name,       pg_wal_lsn_diff(flush_lsn, replay_lsn) AS flushed_not_replayed_bytes,       sync_state, flush_lag, replay_lagFROM pg_stat_replicationWHERE flush_lsn IS NOT NULL AND replay_lsn IS NOT NULL;

11. Production lag runbook

When lag alarms fire: confirm the primary is generating WAL; identify directly connected nodes at each hop; compare send/write/flush/replay positions; check receiver connectivity; inspect standby replay conflicts and I/O; inspect slot retention; estimate disk runway from measured WAL generation; and only then choose remediation. Common fixes include restoring network/storage capacity, ending pathological standby reads, tuning workload-specific feedback/delay policy, or rebuilding a lost standby.

Authoritative references

Replication and HA behavior is topology-, version-, privilege-, and operating-system-sensitive. These primary PostgreSQL sources define the mechanisms used in this lesson.

• PostgreSQL 18 — Cascading Replication and Slots
• PostgreSQL 18 — pg_replication_slots
• PostgreSQL 18 — Monitoring Statistics
• PostgreSQL 18 — Replication Configuration