Curriculum planned

Stage 05 · Data Formats, Storage & Distributed Systems

Distributed Systems Fundamentals

A rigorous distributed-systems course covering network and failure models, time and clocks, replication, quorum, consensus, Raft/Paxos-family concepts, leader election, consistency models, CAP/PACELC, distributed transactions, serializability, partitioning, rebalancing, storage engines, coordination, messaging, stream processing, distributed compute, caching, IDs and ordering, gossip/failure detectors, backpressure, observability, security, chaos engineering, tail latency, architecture patterns, and a production capstone.

Browse planned curriculum Back to all courses

33planned chapters

165reserved lesson paths

Beginner → Advancedlearning level

Plannedcourse state

Coverage baselineVendor-neutral foundations grounded in modern database, streaming, storage, and cloud architectures, with emphasis on safety/liveness reasoning, partial failure, consensus, consistency, partitioning, distributed transactions, observability, and resilient operations

Course brief

Reason about distributed systems from invariants and failure models so replication, consensus, sharding, messaging, and recovery designs remain correct when machines, networks, clocks, and operators fail.

This syllabus deliberately separates foundations, data/model semantics, internals, reliability, security, performance, operations, and production design so advanced material is not compressed into generic catch-all chapters.

By the end

You will be able to

Model asynchronous networks, partial failures, clocks, retries, duplicates, partitions, and the distinction between safety and liveness
Explain and compare replication, quorum, leader-based and leaderless systems, Raft/Paxos-family consensus, consistency models, and CAP/PACELC tradeoffs
Design distributed transactions, partitioning/rebalancing, messaging/streaming, storage, coordination, caching, and ID/order mechanisms from explicit guarantees
Instrument and test systems using metrics/logs/traces, failure injection, load testing, SLOs, backpressure, retries, and disaster scenarios
Evaluate real architectures by identifying invariants, failure domains, bottlenecks, recovery paths, security boundaries, and operational ownership

Complete planned syllabus

33 chapters · 165 lesson paths.

Every lesson path is reserved now but intentionally not linked until its lesson HTML is actually published. The sequence moves from foundations through advanced implementation, architecture, operations, reliability, security, tuning, and a production capstone.

Chapter 1

Distributed Systems Foundations: Nodes, Messages, State, Invariants, Safety, Liveness, and Tradeoffs

5 lessons

What Makes a System Distributed: Independent Failure, Message Passing, No Shared Global Memory, and ConcurrencyPlanned lesson · reserved path Chapter01/Lesson1.html

Planned02

State Machines, Invariants, Safety Properties, Liveness Properties, Progress, and Why 'Usually Works' Is Not a GuaranteePlanned lesson · reserved path Chapter01/Lesson2.html

Planned03

Latency, Throughput, Availability, Durability, Consistency, Scalability, Cost, and Operability as Competing ObjectivesPlanned lesson · reserved path Chapter01/Lesson3.html

Planned04

Client/Server, Peer-to-Peer, Shared-Nothing, Shared-Storage, Control/Data Planes, and Common TopologiesPlanned lesson · reserved path Chapter01/Lesson4.html

Planned05

Write an Explicit Guarantee/Fault Model for a Familiar Database or Service Before Discussing Its ImplementationPlanned lesson · reserved path Chapter01/Lesson5.html

Planned

Chapter 2

Networks: Latency, Loss, Reordering, Duplication, Partitions, Timeouts, and Partial Failure

5 lessons

Packets vs Application Messages, TCP/QUIC-Like Streams, Connection Failure, and the End-to-End Nature of DeliveryPlanned lesson · reserved path Chapter02/Lesson1.html

Planned02

Messages Can Be Delayed, Lost, Duplicated, Reordered, or Delivered After a Timeout; Interpret Ambiguous OutcomesPlanned lesson · reserved path Chapter02/Lesson2.html

Planned03

Network Partitions, One-Way Reachability, DNS/LB Failures, MTU/Firewall Issues, and Slow Links as Failure ModesPlanned lesson · reserved path Chapter02/Lesson3.html

Planned04

Timeout Selection, False Suspicion, Retries, Exponential Backoff/Jitter, Retry Storms, and Circuit BreakersPlanned lesson · reserved path Chapter02/Lesson4.html

Planned05

Simulate Delay/Loss/Partition and Show Why 'No Response' Cannot Distinguish Remote Failure from Network FailurePlanned lesson · reserved path Chapter02/Lesson5.html

Planned

Chapter 3

Processes and Concurrency: Threads, Async I/O, Races, Atomicity, and Local State Machines

5 lessons

Concurrency Within One Node vs Distribution Across Nodes and Why Both Create InterleavingsPlanned lesson · reserved path Chapter03/Lesson1.html

Planned02

Atomic Operations, Locks, Lock-Free Concepts, Queues, Event Loops, Futures, and CancellationPlanned lesson · reserved path Chapter03/Lesson2.html

Planned03

Race Conditions, Lost Updates, Check-Then-Act Bugs, Reentrancy, Deadlocks, and StarvationPlanned lesson · reserved path Chapter03/Lesson3.html

Planned04

Single-Threaded State-Machine Execution as a Simplification and Its Throughput/Latency LimitsPlanned lesson · reserved path Chapter03/Lesson4.html

Planned05

Build a Concurrent Counter/Queue Example and Identify Which Guarantees Must Be Solved Locally Before DistributionPlanned lesson · reserved path Chapter03/Lesson5.html

Planned

Chapter 4

Time: Physical Clocks, Monotonic Clocks, NTP, Drift, Skew, and Timeout Semantics

5 lessons

Wall Clock vs Monotonic Clock, Leap/Adjustment Behavior, Drift, Skew, and Why Time Can Move UnexpectedlyPlanned lesson · reserved path Chapter04/Lesson1.html

Planned02

NTP/PTP-Like Synchronization Concepts, Accuracy Limits, Clock Failure, and Operational MonitoringPlanned lesson · reserved path Chapter04/Lesson2.html

Planned03

Timeouts/Lease Durations Require Monotonic Reasoning While Human Timestamps Need Calendar/UTC SemanticsPlanned lesson · reserved path Chapter04/Lesson3.html

Planned04

Clock-Based IDs/Ordering, Expiration, TTL, LWW Conflict Resolution, and the Cost of Incorrect Time AssumptionsPlanned lesson · reserved path Chapter04/Lesson4.html

Planned05

Design a Lease/TTL Mechanism and Identify the Maximum Safe Behavior Under Clock Skew and Process PausesPlanned lesson · reserved path Chapter04/Lesson5.html

Planned

Chapter 5

Logical Time: Lamport Clocks, Vector Clocks, Causality, and Happens-Before

5 lessons

Happens-Before/Causality vs Real-Time Order and Why Concurrent Events Need Not Have a Natural OrderPlanned lesson · reserved path Chapter05/Lesson1.html

Planned02

Lamport Logical Clocks: Monotonic Event Ordering, Limitations, and Tie-BreakingPlanned lesson · reserved path Chapter05/Lesson2.html

Planned03

Vector Clocks/Version Vectors: Capturing Causal Histories, Concurrency Detection, and Metadata GrowthPlanned lesson · reserved path Chapter05/Lesson3.html

Planned04

Hybrid Logical Clock Concepts and Combining Physical/Logical Time for Practical SystemsPlanned lesson · reserved path Chapter05/Lesson4.html

Planned05

Given a Message History, Derive Causal Relationships and Identify Which Conflicts Are Truly ConcurrentPlanned lesson · reserved path Chapter05/Lesson5.html

Planned

Chapter 6

Replication Fundamentals: Copies, Failure Tolerance, Read Scaling, and State Convergence

5 lessons

Why Replicate: Availability, Durability, Read Locality/Throughput, Disaster Recovery, and Their Different RequirementsPlanned lesson · reserved path Chapter06/Lesson1.html

Planned02

Synchronous vs Asynchronous Replication, Acknowledgement Points, Commit Durability, Lag, and Data-Loss WindowsPlanned lesson · reserved path Chapter06/Lesson2.html

Planned03

Primary-Backup/Leader-Follower, Multi-Leader, Leaderless, and State-Machine Replication ModelsPlanned lesson · reserved path Chapter06/Lesson3.html

Planned04

Full Snapshot, Log Shipping, Incremental Catch-Up, Anti-Entropy, Merkle-Tree-Like Repair, and Replica BootstrapPlanned lesson · reserved path Chapter06/Lesson4.html

Planned05

Create a Failure Matrix Showing What Happens When Leader, Follower, Network, Disk, and Entire Site FailPlanned lesson · reserved path Chapter06/Lesson5.html

Planned

Chapter 7

Quorums and Leaderless Replication: R/W/N, Sloppy Quorums, Hinted Handoff, and Repair

5 lessons

Read/Write Quorum Arithmetic, Overlap, Consistency Conditions, and Why R+W>N Is Not the Whole StoryPlanned lesson · reserved path Chapter07/Lesson1.html

Planned02

Leaderless Writes, Concurrent Versions, Read Repair, Background Anti-Entropy, and Conflict ResolutionPlanned lesson · reserved path Chapter07/Lesson2.html

Planned03

Sloppy Quorums and Hinted Handoff for Availability During Node Failure with Later ConvergencePlanned lesson · reserved path Chapter07/Lesson3.html

Planned04

Last-Write-Wins, Version Vectors, Application Merge, CRDT-Like Strategies, and Lost-Update RiskPlanned lesson · reserved path Chapter07/Lesson4.html

Planned05

Simulate Conflicting Writes and Repair Under Partitions; State Exactly Which Consistency Guarantee Is AchievedPlanned lesson · reserved path Chapter07/Lesson5.html

Planned

Chapter 8

Consensus Problem: Agreement, Validity, Termination, Fault Assumptions, and Impossibility Boundaries

5 lessons

What Consensus Solves: A Single Ordered Decision/Log Despite Node Failures and Message DelayPlanned lesson · reserved path Chapter08/Lesson1.html

Planned02

Safety vs Liveness in Consensus and Why a Safe System May Stop Progress During UncertaintyPlanned lesson · reserved path Chapter08/Lesson2.html

Planned03

Crash Faults vs Byzantine Faults and Why Algorithms/Quorum Sizes DifferPlanned lesson · reserved path Chapter08/Lesson3.html

Planned04

FLP-Style Asynchronous Impossibility Intuition and the Role of Timeouts/Failure Detectors in Practical SystemsPlanned lesson · reserved path Chapter08/Lesson4.html

Planned05

Separate Consensus, Replication, Leader Election, Membership, and Failure Detection Instead of Using Them as SynonymsPlanned lesson · reserved path Chapter08/Lesson5.html

Planned

Chapter 9

Raft: Terms, Leader Election, Log Replication, Commit Rules, Membership, and Snapshots

5 lessons

Raft Roles, Terms, RequestVote, Heartbeats, Election Timeouts, Majority Quorums, and Split VotesPlanned lesson · reserved path Chapter09/Lesson1.html

Planned02

AppendEntries, Log Matching, Conflict Repair, commitIndex, Leader Completeness, and Application to State MachinesPlanned lesson · reserved path Chapter09/Lesson2.html

Planned03

Why Entries from Prior Terms Have Special Commit Rules and How Safety Is Preserved Across Leader ChangesPlanned lesson · reserved path Chapter09/Lesson3.html

Planned04

Snapshots/Log Compaction and Joint-Consensus-Like Membership Changes for Safe ReconfigurationPlanned lesson · reserved path Chapter09/Lesson4.html

Planned05

Walk Through Node Failure, Partition, New Leader, Conflicting Suffix, and Recovery Without Violating Committed HistoryPlanned lesson · reserved path Chapter09/Lesson5.html

Planned

Chapter 10

Paxos, Multi-Paxos, Zab, Viewstamped Replication, and Consensus-Family Comparison

5 lessons

Single-Decree Paxos Roles/Phases, Proposal Numbers, Quorum Intersection, and Safety IntuitionPlanned lesson · reserved path Chapter10/Lesson1.html

Planned02

Multi-Paxos: Stable Leader Optimization and Repeated Log DecisionsPlanned lesson · reserved path Chapter10/Lesson2.html

Planned03

Zab/Viewstamped-Replication-Like Primary/epoch Concepts and Their Relationship to Ordered ReplicationPlanned lesson · reserved path Chapter10/Lesson3.html

Planned04

Compare Raft/Paxos Families by Model, Membership, Read Path, Implementation Clarity, and Operational EcosystemPlanned lesson · reserved path Chapter10/Lesson4.html

Planned05

Read a High-Level Consensus Protocol Description and Identify the Quorum, Epoch/Term, Log-Matching, and Recovery InvariantsPlanned lesson · reserved path Chapter10/Lesson5.html

Planned

Chapter 11

Leader Election, Leases, Fencing Tokens, Split Brain, and Single-Writer Safety

5 lessons

Leader Election Is Not Enough: Old Leaders May Continue Acting After Losing AuthorityPlanned lesson · reserved path Chapter11/Lesson1.html

Planned02

Leases, Clock/GC Pause Risks, Session Expiration, and Distinguishing Membership from AuthorityPlanned lesson · reserved path Chapter11/Lesson2.html

Planned03

Monotonic Fencing Tokens/Epoch Numbers for Protecting External Storage and Preventing Stale WritersPlanned lesson · reserved path Chapter11/Lesson3.html

Planned04

Split-Brain Scenarios Under Network Partition, Quorum Design, and Why Dual Primaries Are DangerousPlanned lesson · reserved path Chapter11/Lesson4.html

Planned05

Design a Safe Distributed Lock/Leader Lease and Show How a Fencing Token Prevents a Delayed Former Leader from Corrupting StatePlanned lesson · reserved path Chapter11/Lesson5.html

Planned

Chapter 12

Consistency Models: Linearizability, Sequential, Causal, Read-Your-Writes, and Eventual Consistency

5 lessons

Linearizability as Real-Time Single-Copy Semantics and Why It Is Stronger Than SerializabilityPlanned lesson · reserved path Chapter12/Lesson1.html

Planned02

Sequential Consistency, PRAM/Session Guarantees, Read-Your-Writes, Monotonic Reads/Writes, and Consistent PrefixPlanned lesson · reserved path Chapter12/Lesson2.html

Planned03

Causal Consistency: Visibility of Dependencies While Allowing Concurrent Operations to Diverge TemporarilyPlanned lesson · reserved path Chapter12/Lesson3.html

Planned04

Eventual Consistency as Convergence Without a Freshness Bound and the Need to Define Conflict SemanticsPlanned lesson · reserved path Chapter12/Lesson4.html

Planned05

Classify Example Histories Under Multiple Consistency Models Instead of Labeling Systems with One Vague WordPlanned lesson · reserved path Chapter12/Lesson5.html

Planned

Chapter 13

CAP, PACELC, Availability Definitions, and Consistency-Latency Tradeoffs

5 lessons

CAP's Partition Scenario: Consistency vs Availability When Communication Between Required Replicas Is LostPlanned lesson · reserved path Chapter13/Lesson1.html

Planned02

Availability in CAP vs SLA/Uptime Availability and Why Terminology Often Causes Misleading ClaimsPlanned lesson · reserved path Chapter13/Lesson2.html

Planned03

PACELC: During Partitions Choose A/C; Else Trade Latency Against ConsistencyPlanned lesson · reserved path Chapter13/Lesson3.html

Planned04

Per-Operation/Per-Data-Class Tradeoffs, Tunable Consistency, and Geo-Replication LatencyPlanned lesson · reserved path Chapter13/Lesson4.html

Planned05

Evaluate a Database Marketing Claim by Writing the Exact Failure Scenario, Required Response, and Consistency PropertyPlanned lesson · reserved path Chapter13/Lesson5.html

Planned

Chapter 14

Transactions in Distributed Systems: ACID, Atomic Commit, Isolation, and Failure Ambiguity

5 lessons

Local ACID vs Distributed ACID and the Added Problem of Coordinating Multiple Failure-Prone ParticipantsPlanned lesson · reserved path Chapter14/Lesson1.html

Planned02

Atomicity, Consistency, Isolation, Durability, and Why 'Consistency' Here Differs from Replica ConsistencyPlanned lesson · reserved path Chapter14/Lesson2.html

Planned03

Client Timeout After Commit, Unknown Transaction Outcome, Idempotency, Transaction IDs, and Recovery QueriesPlanned lesson · reserved path Chapter14/Lesson3.html

Planned04

Distributed Locking/Validation, Deadlocks, Retries, and Transaction Coordinator StatePlanned lesson · reserved path Chapter14/Lesson4.html

Planned05

Trace a Multi-Service/Shard Transaction Through Crash Points and Define What Recovery Must GuaranteePlanned lesson · reserved path Chapter14/Lesson5.html

Planned

Chapter 15

Two-Phase Commit, Three-Phase Ideas, Sagas, Outbox, and Atomicity Alternatives

5 lessons

2PC Prepare/Commit Phases, Coordinator/Participant Logs, Prepared Locks, Blocking, and RecoveryPlanned lesson · reserved path Chapter15/Lesson1.html

Planned02

Coordinator Failure, Participant Failure, Network Partition, Heuristic Decisions, and In-Doubt TransactionsPlanned lesson · reserved path Chapter15/Lesson2.html

Planned03

Why Consensus and Atomic Commit Solve Different Problems and How Modern Systems Combine ThemPlanned lesson · reserved path Chapter15/Lesson3.html

Planned04

Sagas/Compensations, Transactional Outbox/Inbox, Idempotent Consumers, and Business-Level AtomicityPlanned lesson · reserved path Chapter15/Lesson4.html

Planned05

Choose 2PC, Saga, Outbox, or Redesign for a Workflow and Defend the Choice using Failure and Consistency RequirementsPlanned lesson · reserved path Chapter15/Lesson5.html

Planned

Chapter 16

Isolation and Serializability: MVCC, Snapshot Isolation, SSI, Locks, and Distributed Anomalies

5 lessons

Read Committed, Repeatable Read, Snapshot Isolation, Serializable, and Anomaly-Based ReasoningPlanned lesson · reserved path Chapter16/Lesson1.html

Planned02

MVCC Versions/Snapshots, Write-Write Conflicts, Read Skew, Write Skew, and Garbage CollectionPlanned lesson · reserved path Chapter16/Lesson2.html

Planned03

2PL/Predicate Locks vs Serializable Snapshot Isolation-Like Detection and Abort TradeoffsPlanned lesson · reserved path Chapter16/Lesson3.html

Planned04

Global/Per-Shard Timestamp Ordering, Clock/Consensus Dependencies, and Cross-Partition Serializable TransactionsPlanned lesson · reserved path Chapter16/Lesson4.html

Planned05

Construct Transaction Histories that Demonstrate Lost Update, Read Skew, Write Skew, and Serialization FailurePlanned lesson · reserved path Chapter16/Lesson5.html

Planned

Chapter 17

Partitioning and Sharding: Range, Hash, Directory, Consistent Hashing, and Data Locality

5 lessons

Why Partition: Capacity, Throughput, Parallelism, Locality, Tenant Isolation, and Independent Failure DomainsPlanned lesson · reserved path Chapter17/Lesson1.html

Planned02

Range Partitioning for Ordered Scans vs Hash Partitioning for Distribution; Hotspots and SkewPlanned lesson · reserved path Chapter17/Lesson2.html

Planned03

Consistent Hashing/Rendezvous-Hash-Like Concepts, Virtual Nodes, and Minimizing Movement on Membership ChangePlanned lesson · reserved path Chapter17/Lesson3.html

Planned04

Secondary Indexes, Global Uniqueness, Cross-Shard Joins/Transactions, and Routing MetadataPlanned lesson · reserved path Chapter17/Lesson4.html

Planned05

Design Sharding for a Multi-Tenant Workload and Quantify Cross-Shard Traffic, Hot Keys, and Rebalance CostPlanned lesson · reserved path Chapter17/Lesson5.html

Planned

Chapter 18

Rebalancing, Elasticity, Membership Change, and Online Data Movement

5 lessons

Adding/Removing Nodes Changes Capacity and Requires Controlled Ownership/Data MovementPlanned lesson · reserved path Chapter18/Lesson1.html

Planned02

Copy-Then-Cutover, Dual Read/Write, Ownership Epochs, Range Splitting/Merging, and Consistent-Hash MovementPlanned lesson · reserved path Chapter18/Lesson2.html

Planned03

Throttling Rebalance to Protect Foreground Traffic and Avoiding Cascading Capacity FailuresPlanned lesson · reserved path Chapter18/Lesson3.html

Planned04

Failure During Rebalance, Duplicate/Missing Ownership, Resume/Checkpoint, and Consistency VerificationPlanned lesson · reserved path Chapter18/Lesson4.html

Planned05

Plan an Online Scale-Out from N to N+K Nodes with Traffic Gates, Validation, Rollback, and Completion CriteriaPlanned lesson · reserved path Chapter18/Lesson5.html

Planned

Chapter 19

Distributed Storage Engines: WALs, LSM Trees, B-Trees, Compaction, and Replicated Logs

5 lessons

Write-Ahead Logs, Memtables, Immutable SSTables, B-Trees, Page Stores, and Crash RecoveryPlanned lesson · reserved path Chapter19/Lesson1.html

Planned02

LSM Compaction Strategies, Read/Write Amplification, Tombstones, Bloom Filters, and Space AmplificationPlanned lesson · reserved path Chapter19/Lesson2.html

Planned03

Replicating Logical Commands vs Physical Logs/Pages and the Consequences for Determinism and UpgradesPlanned lesson · reserved path Chapter19/Lesson3.html

Planned04

Snapshots, Checkpoints, Log Truncation, Garbage Collection, and Replica BootstrapPlanned lesson · reserved path Chapter19/Lesson4.html

Planned05

Relate Storage-Engine Behavior to Distributed Symptoms such as Compaction Debt, Tail Latency, and Recovery TimePlanned lesson · reserved path Chapter19/Lesson5.html

Planned

Chapter 20

Coordination and Metadata Services: Namespaces, Watches, Locks, Sessions, and Control Planes

5 lessons

Why Small Strongly Consistent Metadata Systems Often Coordinate Much Larger Data SystemsPlanned lesson · reserved path Chapter20/Lesson1.html

Planned02

Hierarchical Keys/Nodes, Conditional Updates, Versions, Watches, Sessions/Leases, and Ephemeral OwnershipPlanned lesson · reserved path Chapter20/Lesson2.html

Planned03

Distributed Locks/Leader Election Recipes, Herd Effects, Lost Notifications, and Fencing RequirementsPlanned lesson · reserved path Chapter20/Lesson3.html

Planned04

Control Plane vs Data Plane: Keeping Metadata Quorums Small, Stable, and Isolated from Bulk TrafficPlanned lesson · reserved path Chapter20/Lesson4.html

Planned05

Design Coordination for Cluster Membership/Configuration Without Storing Large Application Data in the Consensus ServicePlanned lesson · reserved path Chapter20/Lesson5.html

Planned

Chapter 21

Messaging and Durable Logs: Queues, Pub/Sub, Partitions, Offsets, and Delivery Semantics

5 lessons

Queue vs Pub/Sub vs Append-Only Log Models and How Consumer State Changes SemanticsPlanned lesson · reserved path Chapter21/Lesson1.html

Planned02

Partitions for Parallelism and Ordering Scope; Keys, Rebalancing, Consumer Groups, and Hot PartitionsPlanned lesson · reserved path Chapter21/Lesson2.html

Planned03

At-Most-Once, At-Least-Once, Effectively/Exactly-Once Claims, Idempotency, and Transactional BoundariesPlanned lesson · reserved path Chapter21/Lesson3.html

Planned04

Acknowledgements, Retries, Dead-Letter Queues, Poison Messages, Backlog, Retention, and ReplayPlanned lesson · reserved path Chapter21/Lesson4.html

Planned05

Design a Recoverable Event Pipeline and Test Duplicate, Out-of-Order, Consumer Crash, and Replay ScenariosPlanned lesson · reserved path Chapter21/Lesson5.html

Planned

Chapter 22

Stream Processing: Event Time, Watermarks, Windows, Stateful Operators, and Checkpoints

5 lessons

Event Time vs Processing Time, Out-of-Order Events, Watermarks, Allowed Lateness, and CompletenessPlanned lesson · reserved path Chapter22/Lesson1.html

Planned02

Tumbling/Sliding/Session Windows, Triggers, Late Updates, Retractions, and Result SemanticsPlanned lesson · reserved path Chapter22/Lesson2.html

Planned03

Partitioned State, Keyed Operators, State Backends, Checkpoints/Snapshots, and RestorePlanned lesson · reserved path Chapter22/Lesson3.html

Planned04

Exactly-Once State vs External Side Effects, Transactional Sinks, Idempotency, and Two-Phase Commit-Like SinksPlanned lesson · reserved path Chapter22/Lesson4.html

Planned05

Trace a Stateful Aggregation Through Failure/Restore and Verify Output Correctness with Late/Duplicate EventsPlanned lesson · reserved path Chapter22/Lesson5.html

Planned

Chapter 23

Distributed Compute: MapReduce, DAG Schedulers, Tasks, Shuffles, Stragglers, and Data Locality

5 lessons

Map/Shuffle/Reduce as a Distributed Dataflow Pattern and Why Shuffles Dominate Many WorkloadsPlanned lesson · reserved path Chapter23/Lesson1.html

Planned02

DAG Scheduling, Stage/Task Boundaries, Dependencies, Pipelining, and Intermediate MaterializationPlanned lesson · reserved path Chapter23/Lesson2.html

Planned03

Data Locality, Rack/Zone Awareness, Remote Reads, Shuffle Networks, Spill, and Disk PressurePlanned lesson · reserved path Chapter23/Lesson3.html

Planned04

Task Failure/Retry, Speculative Execution, Stragglers, Determinism, and Duplicate Side EffectsPlanned lesson · reserved path Chapter23/Lesson4.html

Planned05

Analyze a Distributed Job Plan and Identify Parallelism, Shuffle Volume, Failure Scope, and Tail-Latency RisksPlanned lesson · reserved path Chapter23/Lesson5.html

Planned

Chapter 24

Distributed Caching, CDNs, Invalidation, Replication, and Cache Consistency

5 lessons

Cache-Aside, Read-Through, Write-Through/Behind, Near Cache, Shared Cache, and CDN Edge ModelsPlanned lesson · reserved path Chapter24/Lesson1.html

Planned02

TTL, Explicit Invalidation, Versioned Keys, Generational Caches, and the Two-Hard-Things ProblemPlanned lesson · reserved path Chapter24/Lesson2.html

Planned03

Cache Stampede/Thundering Herd, Request Coalescing, Probabilistic Early Refresh, and Negative CachingPlanned lesson · reserved path Chapter24/Lesson3.html

Planned04

Stale Reads, Write Invalidation Races, Multi-Region Cache Coherence, and Consistency RequirementsPlanned lesson · reserved path Chapter24/Lesson4.html

Planned05

Design a Cache Strategy with Explicit Freshness SLO, Failure Behavior, Stampede Protection, and Bypass ModePlanned lesson · reserved path Chapter24/Lesson5.html

Planned

Chapter 25

Distributed IDs, Ordering, Sequences, Snowflake-Like IDs, UUIDs, and Global Uniqueness

5 lessons

Why Auto-Increment Global Sequences Become Coordination Bottlenecks in Sharded SystemsPlanned lesson · reserved path Chapter25/Lesson1.html

Planned02

UUID Versions, Random vs Time-Ordered IDs, Collision Probability, Index Locality, and Privacy LeakagePlanned lesson · reserved path Chapter25/Lesson2.html

Planned03

Snowflake-Like IDs: Timestamp, Worker ID, Sequence Bits, Clock Rollback, and Worker AssignmentPlanned lesson · reserved path Chapter25/Lesson3.html

Planned04

Database/Consensus-Backed Sequences, Hi/Lo Allocation, Range Leasing, and GapsPlanned lesson · reserved path Chapter25/Lesson4.html

Planned05

Choose an ID Scheme for a Multi-Region System and Analyze Uniqueness, Sortability, Availability, and Clock DependenciesPlanned lesson · reserved path Chapter25/Lesson5.html

Planned

Chapter 26

Membership, Gossip, Failure Detectors, Heartbeats, and Cluster Discovery

5 lessons

Static Membership vs Discovery Services vs Gossip/Epidemic DisseminationPlanned lesson · reserved path Chapter26/Lesson1.html

Planned02

Heartbeats, Accrual/Threshold Failure Detectors, False Positives, Suspicion, and Timeout TuningPlanned lesson · reserved path Chapter26/Lesson2.html

Planned03

Gossip Convergence, Anti-Entropy, Membership Versioning, Tombstones, and Network PartitionsPlanned lesson · reserved path Chapter26/Lesson3.html

Planned04

Failure Detection Is Not Failure Proof; Connect Suspicion to Quorum/Consensus Authority SafelyPlanned lesson · reserved path Chapter26/Lesson4.html

Planned05

Simulate Slow/Partitioned Nodes and Compare Detector Sensitivity, Convergence Time, and False-Failure CostPlanned lesson · reserved path Chapter26/Lesson5.html

Planned

Chapter 27

Backpressure, Flow Control, Load Shedding, Rate Limits, and Overload Stability

5 lessons

Queues Hide Overload Temporarily but Increase Latency; Little's Law and Queue Growth IntuitionPlanned lesson · reserved path Chapter27/Lesson1.html

Planned02

Producer/Consumer Backpressure, Bounded Queues, Credits/Windows, Pull vs Push, and Stream DemandPlanned lesson · reserved path Chapter27/Lesson2.html

Planned03

Rate Limiting: Token/Leaky Buckets, Per-Tenant Fairness, Admission Control, and PriorityPlanned lesson · reserved path Chapter27/Lesson3.html

Planned04

Load Shedding, Degradation, Circuit Breakers, Retry Budgets, and Avoiding Positive Feedback LoopsPlanned lesson · reserved path Chapter27/Lesson4.html

Planned05

Run an Overload Test and Identify Saturation Point, Queue Growth, p99 Collapse, and Stable Rejection PolicyPlanned lesson · reserved path Chapter27/Lesson5.html

Planned

Chapter 28

Observability: Metrics, Logs, Distributed Traces, Correlation, and SLOs

5 lessons

Golden Signals, RED/USE Methods, Queue Depth, Replication Lag, Consensus Health, and Resource SaturationPlanned lesson · reserved path Chapter28/Lesson1.html

Planned02

Structured Logs with Request/Trace/Transaction/Shard IDs and Clock/Ordering CaveatsPlanned lesson · reserved path Chapter28/Lesson2.html

Planned03

Distributed Tracing, Context Propagation, Spans, Critical Path, Sampling, and Cross-Service CausalityPlanned lesson · reserved path Chapter28/Lesson3.html

Planned04

SLIs/SLOs/Error Budgets for Latency, Availability, Freshness, Durability, and Correctness ProxiesPlanned lesson · reserved path Chapter28/Lesson4.html

Planned05

Build a Debugging Timeline for a Cross-Service Incident from Traces, Logs, Metrics, and State/Message EvidencePlanned lesson · reserved path Chapter28/Lesson5.html

Planned

Chapter 29

Security in Distributed Systems: Identity, Trust Boundaries, TLS, Secrets, and Multi-Tenancy

5 lessons

Authentication vs Authorization, Human vs Workload Identity, Service Accounts, and Short-Lived CredentialsPlanned lesson · reserved path Chapter29/Lesson1.html

Planned02

TLS/mTLS, PKI, Certificate Rotation, Name Verification, East-West Traffic, and Service-Mesh ConceptsPlanned lesson · reserved path Chapter29/Lesson2.html

Planned03

Secrets Distribution, KMS/HSM, Envelope Encryption, Key Rotation, and Dependency AvailabilityPlanned lesson · reserved path Chapter29/Lesson3.html

Planned04

Tenant Isolation, Quotas, Network Segmentation, Supply-Chain Risk, and Control-Plane PrivilegePlanned lesson · reserved path Chapter29/Lesson4.html

Planned05

Threat-Model Node Compromise, Credential Theft, MITM, Rogue Service, Data Exfiltration, and Malicious Tenant WorkloadsPlanned lesson · reserved path Chapter29/Lesson5.html

Planned

Chapter 30

Resilience Engineering: Redundancy, Bulkheads, Retries, Chaos Testing, and Recovery

5 lessons

Fault Containment, Bulkheads, Cell-Based Architecture, N+1 Capacity, and Failure-Domain IndependencePlanned lesson · reserved path Chapter30/Lesson1.html

Planned02

Timeouts, Retries, Hedged Requests, Circuit Breakers, Fallbacks, and Their Load-Amplification RisksPlanned lesson · reserved path Chapter30/Lesson2.html

Planned03

Chaos/Fault Injection for Process Kill, Disk Full, Latency, Packet Loss, Clock Skew, Dependency Failure, and Region LossPlanned lesson · reserved path Chapter30/Lesson3.html

Planned04

RPO/RTO, Backup vs Replication, Restore Drills, Failover/Failback, and Data ReconciliationPlanned lesson · reserved path Chapter30/Lesson4.html

Planned05

Create a Resilience Test Plan that Maps Each Assumed Fault to Detection, Automated Response, Manual Runbook, and VerificationPlanned lesson · reserved path Chapter30/Lesson5.html

Planned

Chapter 31

Performance and Tail Latency: Queueing, Fan-Out, Stragglers, Batching, and Capacity Planning

5 lessons

Averages Hide Tail Latency: p95/p99/p99.9, Fan-Out Amplification, and User-Visible Critical PathsPlanned lesson · reserved path Chapter31/Lesson1.html

Planned02

Queueing Theory Intuition, Utilization vs Latency, Little's Law, Service Time, and HeadroomPlanned lesson · reserved path Chapter31/Lesson2.html

Planned03

Batching Improves Throughput but Adds Queue Delay; Choose Batch Size/Timeout from SLA and WorkloadPlanned lesson · reserved path Chapter31/Lesson3.html

Planned04

Stragglers, Hedged/Backup Requests, Speculative Work, Cache Misses, GC Pauses, and Noisy NeighborsPlanned lesson · reserved path Chapter31/Lesson4.html

Planned05

Build a Capacity Model from Arrival Rate, Service Time, Concurrency, Failure Headroom, Growth, and p99 ObjectivesPlanned lesson · reserved path Chapter31/Lesson5.html

Planned

Chapter 32

Architecture Patterns: Microservices, Event-Driven Systems, CQRS, Event Sourcing, Cells, and Data Ownership

5 lessons

Service Decomposition, Data Ownership, Database-per-Service, Shared Data Anti-Patterns, and Distributed Transaction PressurePlanned lesson · reserved path Chapter32/Lesson1.html

Planned02

Synchronous RPC vs Asynchronous Events, Coupling, Backpressure, Failure Propagation, and User-Visible ConsistencyPlanned lesson · reserved path Chapter32/Lesson2.html

Planned03

CQRS and Event Sourcing: Separate Models, Event Logs, Projections, Replay, Versioning, and Operational ComplexityPlanned lesson · reserved path Chapter32/Lesson3.html

Planned04

Cell/Shard/Region Architectures, Blast Radius, Tenant Placement, Routing, and Independent RecoveryPlanned lesson · reserved path Chapter32/Lesson4.html

Planned05

Evaluate an Architecture for Invariants, Cross-Boundary Calls, State Ownership, Failure Modes, and Operability Before Choosing PatternsPlanned lesson · reserved path Chapter32/Lesson5.html

Planned

Chapter 33

Production Capstone: Design and Validate a Fault-Tolerant Distributed Data Service

5 lessons

Define Invariants, Consistency, Latency, Availability, Durability, RPO/RTO, Scale, Regions, and Threat ModelPlanned lesson · reserved path Chapter33/Lesson1.html

Planned02

Choose Replication, Consensus/Coordination, Partitioning, Transactions, Messaging, Caching, and ID/ordering mechanismsPlanned lesson · reserved path Chapter33/Lesson2.html

Planned03

Specify Protocols for Timeouts, Retries, Idempotency, Membership, Rebalancing, Backpressure, and Schema/version evolutionPlanned lesson · reserved path Chapter33/Lesson3.html

Planned04

Load-Test and Inject Node/Network/Disk/Clock/Dependency/Region Failures while Measuring SLOs and Data CorrectnessPlanned lesson · reserved path Chapter33/Lesson4.html

Planned05

Present the Design with State Machines, Failure Matrix, Capacity Model, Observability, Security, Recovery Runbooks, and Known TradeoffsPlanned lesson · reserved path Chapter33/Lesson5.html

Planned