Promotion, Rewind, Failover Orchestration Concepts, Fencing, and Split-Brain Avoidance

Learning outcomes

A standby can be seconds from the primary and still be unsafe to fail over if the old primary can continue accepting writes. PostgreSQL core provides the primitives to stream, promote, follow timelines, and rewind a diverged old primary. A complete High Availability (HA) system must add failure detection, leader selection, fencing, routing, and operator policy around those primitives.

1. Promotion ends recovery and creates a writable primary

Use the disposable pair. First stop application writes or simulate primary failure. Then promote the standby with pg_ctl promote or pg_promote(). The promoted node exits recovery and begins generating WAL on a new timeline.

SELECT pg_is_in_recovery();SELECT pg_promote(wait => true, wait_seconds => 60);SELECT pg_is_in_recovery();

Expected transition: pg_is_in_recovery() changes from true to false. This proves role transition on that node; it does not prove the old primary is fenced or that clients now reach the new primary.

INSERT INTO app.ch14_work_orders VALUES  (14999, 'post_promotion_marker', clock_timestamp());SELECT pg_current_wal_lsn();

2. Timelines preserve divergent history

When a standby is promoted, WAL history branches. PostgreSQL records the branch in timeline history. If the former primary also accepts writes after the split, the two histories diverge; neither side can be merged by merely reconnecting streaming replication.

pg_controldata ./ch14_primary  | grep -E 'TimeLineID|Database cluster state|Latest checkpoint'pg_controldata ./ch14_standby | grep -E 'TimeLineID|Database cluster state|Latest checkpoint'

After failover, standbys configured to follow recovery_target_timeline='latest' can follow the new history when their upstream changes appropriately. Timeline handling is why “the replica just becomes the same primary” is an incomplete mental model.

3. Split brain is an external coordination failure

Split brain means two nodes accept authoritative writes for the same logical service. PostgreSQL streaming replication does not provide distributed consensus to prevent this. Before exposing the promoted node to writes, the failover process must make the old primary unable to serve writes—power off, revoke storage/network access, disable its service, remove it from routing, or use another fencing mechanism that is independently reliable.

1. Detect primary failure with multiple signals.2. Select the best candidate using replay/flush evidence and policy.3. Fence the old primary (prove it cannot accept writes).4. Promote the chosen standby.5. Verify writable role and business data.6. Move writer routing / service discovery.7. Reconfigure remaining standbys to follow new timeline.8. Rejoin or rebuild old primary only after divergence is handled.9. Record RPO/RTO and any durability-policy exceptions.

4. pg_rewind can rejoin a diverged old primary

pg_rewind synchronizes a target data directory with another copy of the same cluster after timelines diverge. It copies changed blocks/files instead of taking an entirely new base backup. For PostgreSQL 18, the target must have full_page_writes=on and at least one of data checksums or wal_log_hints enabled. PostgreSQL 18 enables data checksums by default for newly initialized clusters unless explicitly disabled.

SHOW full_page_writes;SHOW data_checksums;SHOW wal_log_hints;

pg_rewind also needs WAL history back to the divergence point. If target-side required WAL is gone, an archive plus -c may recover it; otherwise take a new base backup. If rewind fails partway, PostgreSQL documentation recommends treating the target data directory as potentially unrecoverable and taking a fresh backup rather than improvising.

5. Controlled rewind pattern

Keep the new primary running. Shut down the old primary cleanly before using it as the rewind target. The source can be the running new primary through a normal SQL connection with sufficient permissions, or a cleanly stopped source data directory.

pg_rewind \  --target-pgdata=./ch14_primary \  --source-server="host=localhost port=55437 dbname=postgres user=rewind_user" \  --dry-run --progress

pg_rewind \  --target-pgdata=./ch14_primary \  --source-server="host=localhost port=55437 dbname=postgres user=rewind_user" \  --write-recovery-conf --progress

--write-recovery-conf creates standby.signal and appends connection settings so the rewound target can follow the new primary, but inspect copied configuration before restart. Rewind copies configuration files from the source; port/path/SSL differences may need correction.

6. Least privilege for an online rewind source

A superuser is not strictly required when the source is online. PostgreSQL documents a role that can execute the file-inspection functions used by pg_rewind. Treat this as a powerful operational account, protect its credentials, and revoke it when the runbook does not require standing access.

CREATE USER rewind_user LOGIN;GRANT EXECUTE ON FUNCTION pg_catalog.pg_ls_dir(text, boolean, boolean) TO rewind_user;GRANT EXECUTE ON FUNCTION pg_catalog.pg_stat_file(text, boolean) TO rewind_user;GRANT EXECUTE ON FUNCTION pg_catalog.pg_read_binary_file(text) TO rewind_user;GRANT EXECUTE ON FUNCTION pg_catalog.pg_read_binary_file(text, bigint, bigint, boolean) TO rewind_user;

7. Core PostgreSQL versus HA orchestrators

Core PostgreSQL provides streaming replication, slots, synchronous commit, promotion, timeline following, pg_rewind, and monitoring views. It does not provide a built-in distributed leader-election service, virtual IP manager, quorum witness, cloud API fencer, or complete automatic failover controller. Tools such as Patroni/repmgr/Pacemaker-based stacks and managed cloud services add policy/control-plane behavior; their safety depends on their own consensus/fencing design and configuration.

This course does not require any third-party HA product. The free local learning path is to manually execute the same state transitions and verify the invariants that an orchestrator must automate.

8. Failover observability and acceptance

-- On primary, when reachable:SELECT application_name, state, sync_state,       flush_lsn, replay_lsn, flush_lag, replay_lagFROM pg_stat_replication;-- On candidate standby:SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn(),       pg_last_xact_replay_timestamp();

The “best” candidate depends on your durability policy. A synchronous standby with acknowledged flush may have stronger RPO evidence than an asynchronous node that happens to respond faster. A node with current receive LSN but delayed replay may still require recovery before it can expose recent data.

9. Final Chapter 14 drill

Build the disposable pair, create a marker, stop or isolate the primary, record candidate LSNs, fence the old primary, promote the standby, verify the marker and writable state, then either rebuild or rewind the old primary into a standby. Record observed RPO/RTO and every manual routing/fencing step. If you cannot prove old-primary fencing, mark the drill failed even if promotion itself succeeded.

pg_ctl -D ./ch14_primary stop -m fast || truepg_ctl -D ./ch14_standby stop -m fast || true# Confirm paths, preserve drill logs, then remove only disposable ch14 directories.

Authoritative references

Replication and HA behavior is topology-, version-, privilege-, and operating-system-sensitive. These primary PostgreSQL sources define the mechanisms used in this lesson.

• PostgreSQL 18 — Failover
• PostgreSQL 18 — pg_rewind
• PostgreSQL 18 — Recovery Control Functions / pg_promote
• PostgreSQL 18 — Log-Shipping Standbys
• PostgreSQL 18 — Data Checksums