Raft Mental Model: Leaders, Terms, Log Replication, Commit Index, and Elections

1. The problem: an old leader acknowledged itself, then disappeared

AtlasMart's control plane wants to store set-shard-owner=s1:A. Leader A in term 4 appends the command locally and gets it to only follower B. Before a majority receives the entry, A becomes isolated. If the application equates “it is in the leader log” with “it is committed,” it may act on a value that a later legitimate leader can overwrite.

Raft organizes time into monotonically increasing terms. A server is follower, candidate, or leader for a term. Elections select at most one leader per term under the protocol's voting rules. Client commands become log entries identified by index and term. A leader's commit index marks the highest index known committed; only committed entries are safe to apply to the replicated state machine according to Raft's rules.

2. Election safety depends on log freshness

A candidate requests votes with its term and last-log information. A voter grants at most one vote per term and refuses a candidate whose log is less up-to-date than its own. This restriction is not cosmetic: it helps ensure that a candidate capable of winning cannot be missing already committed entries.

In the lab's term-4 scenario, the old entry exists only on A and B, so it was never on a majority. Nodes C, D, and E form a majority whose logs do not contain it; C can legitimately become leader in term 5. That is safe precisely because the term-4 entry was never committed.

3. AppendEntries is both replication and consistency repair

The leader sends AppendEntries messages containing a previous-log index/term anchor plus new entries. Followers accept new entries only if the preceding history matches. When a follower has a conflicting uncommitted suffix, the leader backs up to the last matching point and replaces the conflict. This produces the log matching property: if two logs contain an entry with the same index and term, their preceding histories match.

4. AtlasMart lab: uncommitted entry disappears safely

Term 4's entry reaches only A and B. After A's failure/partition, C wins term 5 from C, D, and E, appends a different current-term entry to that majority, commits it, then repairs A and B's conflicting uncommitted suffix.

from copy import deepcopy

nodes = ["A", "B", "C", "D", "E"]
majority = 3
logs = {n: [] for n in nodes}
term = 4
leader = "A"
commit_index = 0

print("TERM 4: A IS LEADER")
entry1 = (4, "set-shard-owner=s1:A")
logs["A"].append(entry1)
logs["B"].append(entry1)  # only one follower receives it
replicated = [n for n in nodes if logs[n] == [entry1]]
print("entry1 replicas:", replicated)
print("present on leader:", entry1 in logs[leader])
print("majority replicated:", len(replicated) >= majority)
print("commit_index:", commit_index, "=> entry1 is NOT committed")

print("\nA FAILS/PARTITIONS; C,D,E CAN ELECT IN TERM 5")
term = 5
leader = "C"
voters = ["C", "D", "E"]
print("votes for C:", voters, "majority:", len(voters) >= majority)

# New leader's empty log is authoritative because the old entry was never committed.
entry2 = (5, "set-shard-owner=s1:C")
for n in ["C", "D", "E"]:
    logs[n] = [entry2]
replicated2 = [n for n in nodes if logs[n] == [entry2]]
if len(replicated2) >= majority:
    commit_index = 1
print("entry2 replicas:", replicated2)
print("commit_index:", commit_index, "=> current-term entry committed")

print("\nFOLLOWER CATCH-UP / LOG REPAIR")
for n in ["A", "B"]:
    # AppendEntries conflict handling replaces the uncommitted conflicting suffix.
    logs[n] = deepcopy(logs[leader])
print("logs:")
for n in nodes:
    print(n, logs[n])
print("all logs agree after catch-up:", len({tuple(v) for v in logs.values()}) == 1)
print("old uncommitted term-4 entry survived?", any(entry1 in v for v in logs.values()))
print("lesson: leader-local presence is not the same as commitment")

5. The current-term commit rule closes a subtle safety hole

Raft leaders do not simply count replicas of arbitrary older-term entries and mark those entries committed. A leader advances the commit index by counting a majority for an entry from its current term; earlier entries become committed as part of the prefix once that current-term entry commits. This rule is part of Raft's safety proof across leader changes.

Production implementations also persist the current term, voted-for state, and log before responding in ways assumed by the protocol. If a crash erases a vote or accepted log entry that the algorithm assumes survives restart, the paper proof no longer describes the implementation.

6. Deliberately wrong approach: serve authority from an uncommitted leader suffix

Suppose A sends “I own shard s1” to workers immediately after appending the term-4 entry locally. A then partitions, C legitimately wins term 5, and commits C as owner. Now A and C may both act. The correction is to expose only committed ownership decisions, attach the committed term/epoch to work, and enforce a fencing token at the downstream resource. Consensus decides the authority; fencing stops a delayed former authority from using stale credentials.

7. Production judgment

Observe term, leader changes, commit index, applied index, follower match/next indexes, proposal queueing, fsync latency, snapshot installation, and quorum round-trip latency. Frequent elections often inflate tail latency even when safety remains intact. Network placement matters: a leader far from a majority increases commit latency. Membership changes require Raft's safe configuration-change mechanism rather than manual simultaneous edits.

The next lesson reaches the same fundamental agreement problem from Paxos's vocabulary: proposers, acceptors, ballots, promises, accepted values, quorum intersection, and the exact rule that prevents a higher-ballot proposer from replacing an already chosen value.

References

Foundational claims use primary research where practical. Product documentation is used only as a current implementation example and is not required for the mandatory labs.

• Ongaro & Ousterhout — Raft extended paper — Primary source for terms, leader election, log replication, safety, commit rules, and cluster membership.
• Raft Consensus Algorithm — official publication hub — Maintained index of the Raft paper, dissertation, TLA+ specification, and related verification work.
• etcd v3.7 release announcement — Current implementation example: etcd v3.7 uses raft v3.7.0 as a core dependency; not required for the lab.
• etcd v3.7 documentation — Current operational documentation for a production Raft-based system.