Curriculum planned

Stage 06 · Hadoop Ecosystem

Apache Hadoop

A complete Apache Hadoop course covering Hadoop 3.5.x architecture and Java 17-era setup, Hadoop Common, HDFS internals and administration, permissions/ACLs/xattrs/snapshots, HDFS HA, federation and Router-based Federation, erasure coding and storage policies, balancing and maintenance, YARN architecture/scheduling/security/federation, MapReduce programming and internals, Hadoop I/O/compression, DistCp/archives, Kerberos and tokens, KMS/encryption, observability, upgrades, disaster recovery, cloud object-store connectors, ecosystem integration, troubleshooting, and production operations.

Browse planned curriculum Back to all courses

35planned chapters

175reserved lesson paths

Beginner → Advancedlearning level

Plannedcourse state

Coverage baselineCurrent Apache Hadoop 3.5.0-era platform, including Java 17 server requirements, HDFS high availability/federation/erasure coding, modern YARN scheduling/federation, MapReduce, cloud filesystems such as S3A/ABFS/GCS, security, and production administration

Course brief

Understand Hadoop as a distributed storage and resource-management platform—not merely MapReduce—by mastering HDFS, YARN, security, operations, failure recovery, and modern object-store integration.

This syllabus deliberately separates foundations, data/model semantics, internals, reliability, security, performance, operations, and production design so advanced material is not compressed into generic catch-all chapters.

By the end

You will be able to

Install and operate a Hadoop 3.5.x lab with Java 17, Hadoop Common configuration, HDFS, YARN, MapReduce, and repeatable cluster administration
Reason deeply about HDFS blocks, NameNode metadata, DataNodes, HA, federation, Router-based federation, erasure coding, balancing, snapshots, quotas, and storage policies
Configure and troubleshoot YARN resource management, schedulers, containers, application masters, resource types, queues, federation, and security
Develop and tune MapReduce workloads while understanding input splits, shuffle/sort, combiners, partitioners, speculative execution, compression, and output commit
Secure, monitor, upgrade, back up, recover, and integrate Hadoop with cloud/object storage and downstream Hive/HBase/Spark ecosystems

Complete planned syllabus

35 chapters · 175 lesson paths.

Every lesson path is reserved now but intentionally not linked until its lesson HTML is actually published. The sequence moves from foundations through advanced implementation, architecture, operations, reliability, security, tuning, and a production capstone.

Chapter 1

Apache Hadoop Foundations, 3.5.x Architecture, Java 17, Components, and Lab Setup

5 lessons

Hadoop's Historical Role vs Modern Data Lakes/Spark/Cloud Warehouses and Where HDFS/YARN Still MatterPlanned lesson · reserved path Chapter01/Lesson1.html

Planned02

Hadoop Common, HDFS, YARN, and MapReduce: Responsibility Boundaries and Process TopologyPlanned lesson · reserved path Chapter01/Lesson2.html

Planned03

Hadoop 3.5.0-Era Requirements, Java 17 on Servers, Client Compatibility, Native Libraries, and Release AwarenessPlanned lesson · reserved path Chapter01/Lesson3.html

Planned04

Install Pseudo-Distributed/Small Multi-Node Hadoop, Configure SSH/Java, Format HDFS, Start Daemons, and Run Smoke TestsPlanned lesson · reserved path Chapter01/Lesson4.html

Planned05

Build a Reproducible Lab with Configuration-as-Code, Sample Data, Metrics/Logs, Safe Reset, and Failure-Injection ScriptsPlanned lesson · reserved path Chapter01/Lesson5.html

Planned

Chapter 2

Hadoop Common: Configuration, Filesystem Abstraction, RPC, Serialization, and Native Utilities

5 lessons

core-site.xml and Configuration Resolution, Defaults, Overrides, Environment Variables, and Credential/Secret HygienePlanned lesson · reserved path Chapter02/Lesson1.html

Planned02

FileSystem API, URI Schemes, Local/HDFS/Object-Store Implementations, Path Semantics, and Client CachingPlanned lesson · reserved path Chapter02/Lesson2.html

Planned03

Hadoop RPC/IPC Concepts, Writable Serialization, SequenceFile/MapFile-Like Utilities, and CompatibilityPlanned lesson · reserved path Chapter02/Lesson3.html

Planned04

Native Libraries, Checksums, Compression Codecs, Shell Commands, Classpath, and Troubleshooting Environment IssuesPlanned lesson · reserved path Chapter02/Lesson4.html

Planned05

Trace a Client File Operation through Configuration, FileSystem Resolution, RPC, and Underlying StoragePlanned lesson · reserved path Chapter02/Lesson5.html

Planned

Chapter 3

HDFS Architecture: NameNode, DataNodes, Blocks, Metadata, Replication, and Client Data Flow

5 lessons

Namespace/Inode Metadata vs Block Data, NameNode Responsibilities, DataNode Storage, and Scalability BoundariesPlanned lesson · reserved path Chapter03/Lesson1.html

Planned02

Blocks, Replication Factor, Rack Awareness, Placement Policy, Checksums, Block Reports, and HeartbeatsPlanned lesson · reserved path Chapter03/Lesson2.html

Planned03

Client Read Path: Block Locations, Locality, Checksums, Replica Failover, and Short-Circuit Read ConceptsPlanned lesson · reserved path Chapter03/Lesson3.html

Planned04

Client Write Pipeline: Packetization, DataNode Pipeline, Acknowledgements, Pipeline Recovery, and hflush/hsync ConceptsPlanned lesson · reserved path Chapter03/Lesson4.html

Planned05

Trace Create/Write/Close/Read of a Multi-Block File and Identify Metadata, Network, Disk, and Failure StepsPlanned lesson · reserved path Chapter03/Lesson5.html

Planned

Chapter 4

HDFS Namespace and File Lifecycle: Create, Append, Rename, Delete, Trash, and Lease Recovery

5 lessons

Directories, Files, Blocks, Inodes, Quotas, Ownership, Timestamps, Replication, and Namespace OperationsPlanned lesson · reserved path Chapter04/Lesson1.html

Planned02

Create/Append/Truncate-Like Semantics, Open-File Leases, Lease Recovery, and Client FailurePlanned lesson · reserved path Chapter04/Lesson2.html

Planned03

Atomic Rename within HDFS Namespace, Delete/Trash, Delayed Block Deletion, and Snapshot InteractionPlanned lesson · reserved path Chapter04/Lesson3.html

Planned04

Safe Publication Patterns for Jobs: Temporary Outputs, Atomic Rename, Output Commit, and Reader VisibilityPlanned lesson · reserved path Chapter04/Lesson4.html

Planned05

Recover an Abandoned Write/Lease and Verify File Length, Block State, Checksums, and Application SemanticsPlanned lesson · reserved path Chapter04/Lesson5.html

Planned

Chapter 5

HDFS CLI, FileSystem API, WebHDFS/HTTP, and Application Access

5 lessons

hdfs dfs/fs Shell Commands for put/get/cp/mv/rm/du/count/stat/checksum/setrep and Safe AutomationPlanned lesson · reserved path Chapter05/Lesson1.html

Planned02

Java FileSystem API Concepts: Open/Create/List/Status, Streams, Seek, Positioned Reads, and Resource CleanupPlanned lesson · reserved path Chapter05/Lesson2.html

Planned03

WebHDFS/HTTPFS-Like REST Access Concepts, Authentication, Redirects, and Remote IntegrationPlanned lesson · reserved path Chapter05/Lesson3.html

Planned04

Large Directory Listings, Glob Patterns, Batch Operations, Permissions, and Avoiding Namespace HotspotsPlanned lesson · reserved path Chapter05/Lesson4.html

Planned05

Build a Scripted Data Ingest/Verify/Export Workflow with Checksums, Exit Codes, Idempotency, and Failure RecoveryPlanned lesson · reserved path Chapter05/Lesson5.html

Planned

Chapter 6

HDFS Permissions, ACLs, Extended Attributes, Quotas, and Multi-Tenant Namespace Governance

5 lessons

POSIX-Like User/Group/Mode Permissions, superuser Semantics, umask, and HDFS Differences from Local FilesystemsPlanned lesson · reserved path Chapter06/Lesson1.html

Planned02

Access and Default ACLs, Mask Entries, Inheritance, Effective Permissions, and Group MappingPlanned lesson · reserved path Chapter06/Lesson2.html

Planned03

Extended Attributes (xattrs), Trusted/User/Security Namespaces, Encryption Metadata, and Application UsePlanned lesson · reserved path Chapter06/Lesson3.html

Planned04

Namespace/Space/Storage-Type Quotas, Quota Violations, Capacity Governance, and Tenant IsolationPlanned lesson · reserved path Chapter06/Lesson4.html

Planned05

Design a Multi-Team HDFS Namespace with Ownership, ACLs, Quotas, Auditability, and Least PrivilegePlanned lesson · reserved path Chapter06/Lesson5.html

Planned

Chapter 7

HDFS Snapshots: Point-in-Time Namespace Views, Retention, Diff, and Recovery

5 lessons

Snapshottable Directories, Snapshot Creation/Deletion/Rename, Copy-on-Write Metadata Semantics, and LimitsPlanned lesson · reserved path Chapter07/Lesson1.html

Planned02

Snapshot Paths, Read-Only Historical Views, Restoring Accidentally Changed/Deleted Files, and Application RecoveryPlanned lesson · reserved path Chapter07/Lesson2.html

Planned03

Snapshot Diff Reports for Change Discovery, Replication/Backup Workflows, and Audit EvidencePlanned lesson · reserved path Chapter07/Lesson3.html

Planned04

Snapshot Retention, Namespace/Block Storage Impact, Quotas, and Interaction with Encryption ZonesPlanned lesson · reserved path Chapter07/Lesson4.html

Planned05

Run an Accidental-Deletion Recovery Drill with Snapshots and Measure RTO, Permissions, and Verification StepsPlanned lesson · reserved path Chapter07/Lesson5.html

Planned

Chapter 8

HDFS High Availability with Quorum Journal Manager and Automatic Failover

5 lessons

Active/Standby NameNodes, Shared Edit Log, JournalNodes, Edit Quorum, and Namespace State SynchronizationPlanned lesson · reserved path Chapter08/Lesson1.html

Planned02

Standby Tail/Edit Application, Checkpointing, DataNode Block Reports to Both NameNodes, and Client FailoverPlanned lesson · reserved path Chapter08/Lesson2.html

Planned03

ZooKeeper/ZKFC-Like Automatic Failover, Health Monitoring, Leader Election, and Fencing Stale Active NodesPlanned lesson · reserved path Chapter08/Lesson3.html

Planned04

Failure Scenarios: Active Crash, Network Partition, JournalNode Loss, ZK Loss, and Split-Brain PreventionPlanned lesson · reserved path Chapter08/Lesson4.html

Planned05

Execute Planned/Unplanned NameNode Failover and Verify Client Availability, Namespace Correctness, and AlertingPlanned lesson · reserved path Chapter08/Lesson5.html

Planned

Chapter 9

HDFS Federation and Router-Based Federation: Multiple Namespaces and Horizontal Metadata Scale

5 lessons

Federation Concepts: Multiple Independent Namespaces/Block Pools Sharing DataNode StoragePlanned lesson · reserved path Chapter09/Lesson1.html

Planned02

Mount Tables/ViewFs-Like Namespace Composition, Client Routing, Administration, and Application TransparencyPlanned lesson · reserved path Chapter09/Lesson2.html

Planned03

Router-Based Federation Architecture, Routers, State Store, Resolver/Mount Table, and Async RPC ImprovementsPlanned lesson · reserved path Chapter09/Lesson3.html

Planned04

HA for Routers/State Store, Namespace Isolation, Cross-Namespace Operations, and Operational ComplexityPlanned lesson · reserved path Chapter09/Lesson4.html

Planned05

Design a Federated HDFS for Multiple Business Domains and Explain Metadata Scale, Failure Isolation, and RoutingPlanned lesson · reserved path Chapter09/Lesson5.html

Planned

Chapter 10

HDFS Erasure Coding, Storage Policies, Archival Tiers, and Capacity Efficiency

5 lessons

Replication vs Reed-Solomon Erasure Coding: Data/Parity Units, Durability, CPU/Network Cost, and Small-File FitPlanned lesson · reserved path Chapter10/Lesson1.html

Planned02

Erasure Coding Policies, Striped Blocks/Block Groups, Placement, Reconstruction, and Supported OperationsPlanned lesson · reserved path Chapter10/Lesson2.html

Planned03

Storage Types/Policies such as DISK/SSD/ARCHIVE/RAM_DISK Concepts and Moving Data by Access PatternPlanned lesson · reserved path Chapter10/Lesson3.html

Planned04

Archive/Cold Storage, Heterogeneous DataNodes, Storage Policy Satisfaction, and Capacity TradeoffsPlanned lesson · reserved path Chapter10/Lesson4.html

Planned05

Model Replication-3 vs Erasure-Coded Capacity and Run a Reconstruction/Performance ExperimentPlanned lesson · reserved path Chapter10/Lesson5.html

Planned

Chapter 11

HDFS Balancer, Disk Balancer, Decommission, Maintenance, and Data-Node Lifecycle

5 lessons

Cluster Balancer for Redistributing Blocks Across DataNodes, Thresholds, Bandwidth, and Operational TimingPlanned lesson · reserved path Chapter11/Lesson1.html

Planned02

Disk Balancer for Uneven Volumes Within a DataNode and Why Node Balance Does Not Guarantee Disk BalancePlanned lesson · reserved path Chapter11/Lesson2.html

Planned03

Decommission/Recommission, Maintenance State, Replica Safety, Draining, and Rack/Failure-Domain AwarenessPlanned lesson · reserved path Chapter11/Lesson3.html

Planned04

Adding/Replacing Drives/Nodes, Volume Failures, Block Replication Queues, and Capacity HeadroomPlanned lesson · reserved path Chapter11/Lesson4.html

Planned05

Plan a Node-Retirement/Maintenance Window with Data Safety Gates, Throughput Limits, and Completion VerificationPlanned lesson · reserved path Chapter11/Lesson5.html

Planned

Chapter 12

HDFS Performance: Block Size, Replication, Locality, Caching, Short-Circuit Reads, and Small Files

5 lessons

Choose Block Size from File Size, Parallelism, Sequential Scan, Metadata Count, and Recovery CostPlanned lesson · reserved path Chapter12/Lesson1.html

Planned02

Read Locality, Short-Circuit Local Reads, Centralized Cache Management, Page Cache, and Disk ThroughputPlanned lesson · reserved path Chapter12/Lesson2.html

Planned03

Write Pipeline Length, Replication, Network/Rack Traffic, Checksums, and Client BufferingPlanned lesson · reserved path Chapter12/Lesson3.html

Planned04

Small Files and NameNode Metadata Pressure; HAR/Sequence/Container/Lakehouse Compaction AlternativesPlanned lesson · reserved path Chapter12/Lesson4.html

Planned05

Benchmark Sequential/Parallel HDFS Reads/Writes and Correlate Throughput with Network, Disks, CPU, and NameNode LoadPlanned lesson · reserved path Chapter12/Lesson5.html

Planned

Chapter 13

NameNode Metadata, FsImage, Edit Logs, Checkpoints, Safemode, and Namespace Recovery

5 lessons

FsImage Snapshot of Namespace, Edit Logs, TxIDs, Startup Replay, and Metadata DurabilityPlanned lesson · reserved path Chapter13/Lesson1.html

Planned02

Checkpoint/Secondary/Standby NameNode Roles and Why 'Secondary NameNode' Is Not a Hot BackupPlanned lesson · reserved path Chapter13/Lesson2.html

Planned03

Safemode Entry/Exit, Block-Safety Thresholds, Missing/Under-Replicated Blocks, and Startup DiagnosticsPlanned lesson · reserved path Chapter13/Lesson3.html

Planned04

NameNode Heap/GC, Inode/Block Counts, Namespace Growth, Metadata Directories, and Capacity PlanningPlanned lesson · reserved path Chapter13/Lesson4.html

Planned05

Inspect FsImage/Edit Logs in a Lab and Practice Metadata Recovery Without Guessing or Editing Live FilesPlanned lesson · reserved path Chapter13/Lesson5.html

Planned

Chapter 14

YARN Foundations: ResourceManager, NodeManager, ApplicationMaster, Containers, and Application Lifecycle

5 lessons

YARN Separates Cluster Resource Management from Per-Application Scheduling/CoordinationPlanned lesson · reserved path Chapter14/Lesson1.html

Planned02

ResourceManager Scheduler vs ApplicationsManager Responsibilities and HA ConceptsPlanned lesson · reserved path Chapter14/Lesson2.html

Planned03

NodeManager Containers, Resource Monitoring, Localized Resources, Logs, Heartbeats, and HealthPlanned lesson · reserved path Chapter14/Lesson3.html

Planned04

ApplicationMaster Registration, Container Requests, Allocation, Task Coordination, Retry, and CompletionPlanned lesson · reserved path Chapter14/Lesson4.html

Planned05

Trace One MapReduce/Spark-Like Application from Submission Through AM Launch, Containers, Execution, and CleanupPlanned lesson · reserved path Chapter14/Lesson5.html

Planned

Chapter 15

YARN Scheduling: Capacity Scheduler, Queues, Fairness, Reservations, Priorities, and Elasticity

5 lessons

Capacity Scheduler Queue Hierarchy, Capacities, Maximums, User Limits, ACLs, and Multi-Tenant GovernancePlanned lesson · reserved path Chapter15/Lesson1.html

Planned02

Dynamic/Auto-Created Queues, Queue Placement, Application Priorities, and Avoiding Fragmented CapacityPlanned lesson · reserved path Chapter15/Lesson2.html

Planned03

Preemption, Dominant Resource/Multiple Resource Dimensions, Node Labels, and Workload IsolationPlanned lesson · reserved path Chapter15/Lesson3.html

Planned04

ReservationSystem Concepts for Time-Bounded Capacity Guarantees and Admission ControlPlanned lesson · reserved path Chapter15/Lesson4.html

Planned05

Simulate Competing Tenants and Tune Queues for Utilization, Fairness, p95 Wait Time, and Guaranteed CapacityPlanned lesson · reserved path Chapter15/Lesson5.html

Planned

Chapter 16

YARN Containers, Resource Types, cgroups/Isolation, Local Resources, and Node Management

5 lessons

Memory/vcores and Custom Resource Types such as GPUs/FPGAs-Like Resources, Units, and SchedulingPlanned lesson · reserved path Chapter16/Lesson1.html

Planned02

Container Launch, Localization, Environment/Classpaths, Tokens, Logs, Exit Status, and RetryPlanned lesson · reserved path Chapter16/Lesson2.html

Planned03

LinuxContainerExecutor/cgroups-Like Isolation, Process Trees, Memory Enforcement, CPU, Disk, and SecurityPlanned lesson · reserved path Chapter16/Lesson3.html

Planned04

Node Labels/Attributes, Placement Constraints, Opportunistic Containers Concepts, and Specialized HardwarePlanned lesson · reserved path Chapter16/Lesson4.html

Planned05

Design Resource Profiles/Placement for ETL, Memory-Heavy, GPU, and Latency-Sensitive WorkloadsPlanned lesson · reserved path Chapter16/Lesson5.html

Planned

Chapter 17

YARN High Availability, Federation, State Stores, and Large-Cluster Scalability

5 lessons

ResourceManager HA, Active/Standby, State Store, Client/Application Failover, and Recovery of Running AppsPlanned lesson · reserved path Chapter17/Lesson1.html

Planned02

NodeManager Reconnect, ApplicationMaster Restart, Work-Preserving Recovery, and Failure SemanticsPlanned lesson · reserved path Chapter17/Lesson2.html

Planned03

YARN Federation: Multiple Subclusters Presented through a Larger Logical Resource FabricPlanned lesson · reserved path Chapter17/Lesson3.html

Planned04

Router/Policy/State-Store Concepts, Application Placement, Cross-Cluster Capacity, and Failure DomainsPlanned lesson · reserved path Chapter17/Lesson4.html

Planned05

Design Multi-Cluster YARN for Organizational/Regional Scale and Test Subcluster/Router Failure ScenariosPlanned lesson · reserved path Chapter17/Lesson5.html

Planned

Chapter 18

MapReduce Programming Model: Mapper, Reducer, Driver, Keys/Values, and Job Configuration

5 lessons

Map/Reduce Dataflow, Input Key/Value Pairs, Intermediate Pairs, Grouping, and Final OutputPlanned lesson · reserved path Chapter18/Lesson1.html

Planned02

Mapper/Reducer APIs, Context, Counters, Configuration, Setup/Cleanup, and Deterministic Side EffectsPlanned lesson · reserved path Chapter18/Lesson2.html

Planned03

Driver/Job Configuration: Input/Output Paths, Classes, Types, Reducer Count, and SubmissionPlanned lesson · reserved path Chapter18/Lesson3.html

Planned04

Map-Only Jobs, Identity Reducers, Multiple Inputs/Outputs, and Choosing MapReduce vs Newer EnginesPlanned lesson · reserved path Chapter18/Lesson4.html

Planned05

Implement and Test a WordCount-Like and Aggregate Job with Local/Mini-Cluster Tests and CountersPlanned lesson · reserved path Chapter18/Lesson5.html

Planned

Chapter 19

InputFormat, InputSplit, RecordReader, OutputFormat, and Data Locality

5 lessons

InputFormat Chooses Splits; RecordReader Converts Bytes into Mapper Key/Value RecordsPlanned lesson · reserved path Chapter19/Lesson1.html

Planned02

FileInputFormat Splitting, HDFS Blocks vs Input Splits, Non-Splittable Compression, and LocalityPlanned lesson · reserved path Chapter19/Lesson2.html

Planned03

TextInputFormat, KeyValue, Sequence/File-Based and Custom Input Formats for Structured SourcesPlanned lesson · reserved path Chapter19/Lesson3.html

Planned04

OutputFormat/RecordWriter, FileOutputCommitter, Temporary Output, Atomic Commit, and Speculative AttemptsPlanned lesson · reserved path Chapter19/Lesson4.html

Planned05

Create a Custom RecordReader/InputFormat and Prove Correct Split Boundaries and Failure-Safe OutputPlanned lesson · reserved path Chapter19/Lesson5.html

Planned

Chapter 20

MapReduce Shuffle and Sort: Partitioning, Spill, Merge, Grouping, and Network Cost

5 lessons

Mapper Buffer, Spill Thresholds, Sort, Optional Combiner, Partitioning, and Map Output FilesPlanned lesson · reserved path Chapter20/Lesson1.html

Planned02

Shuffle Fetch by Reducers, Parallel Copies, Merge, Sort, Grouping Comparator, and Reduce InputPlanned lesson · reserved path Chapter20/Lesson2.html

Planned03

Default HashPartitioner vs Custom Partitioning, Skew, Hot Reducers, and Secondary Sort PatternsPlanned lesson · reserved path Chapter20/Lesson3.html

Planned04

Map Output Compression, Spill/Merge Tuning, Network Saturation, Disk I/O, and MemoryPlanned lesson · reserved path Chapter20/Lesson4.html

Planned05

Profile a Shuffle-Heavy Job and Reduce Runtime by Addressing Skew, Map-Side Aggregation, Compression, or PartitioningPlanned lesson · reserved path Chapter20/Lesson5.html

Planned

Chapter 21

Combiners, Counters, Side Data, Distributed Cache, and Advanced MapReduce Patterns

5 lessons

Combiner Semantics Require Associative/Commutative-Safe Partial Aggregation and Are Not Guaranteed to RunPlanned lesson · reserved path Chapter21/Lesson1.html

Planned02

Counters for Data Quality/Operational Signals Without Turning Them into High-Cardinality MetricsPlanned lesson · reserved path Chapter21/Lesson2.html

Planned03

Distributed Cache/Localized Files, Side-Data Joins, Map-Side Joins, Bloom Filters, and Memory BoundariesPlanned lesson · reserved path Chapter21/Lesson3.html

Planned04

Chain/Multi-Stage Jobs, Total Order Partitioning, Secondary Sort, Inverted Index, and Top-N PatternsPlanned lesson · reserved path Chapter21/Lesson4.html

Planned05

Select an Advanced Pattern for a Real Data Problem and Prove Correctness Under Retry/SpeculationPlanned lesson · reserved path Chapter21/Lesson5.html

Planned

Chapter 22

MapReduce Fault Tolerance, Speculative Execution, Retries, Committers, and Exactly-Once Illusions

5 lessons

Task Attempts, Retry Limits, Node Failure, ApplicationMaster Failure, and Recomputing Deterministic WorkPlanned lesson · reserved path Chapter22/Lesson1.html

Planned02

Speculative Execution for Stragglers, Duplicate Attempts, and Why External Side Effects Can Become UnsafePlanned lesson · reserved path Chapter22/Lesson2.html

Planned03

Output Commit Protocols, Task/Job Commit, Temporary Paths, Object-Store Committer Challenges, and Partial OutputsPlanned lesson · reserved path Chapter22/Lesson3.html

Planned04

Bad Records, Timeouts, JVM Reuse/Container Lifecycle, and Diagnosing Repeated Task FailurePlanned lesson · reserved path Chapter22/Lesson4.html

Planned05

Inject Mapper/Reducer/Node Failures and Verify Output Uniqueness, Counter Behavior, and Recovery TimePlanned lesson · reserved path Chapter22/Lesson5.html

Planned

Chapter 23

MapReduce Performance Tuning: Parallelism, Memory, Compression, Skew, and Cluster Resources

5 lessons

Mapper Count from Input Splits, Reducer Count from Shuffle/Output Size, and Avoiding Tiny TasksPlanned lesson · reserved path Chapter23/Lesson1.html

Planned02

Container Memory/vcores, JVM Heap, Sort Buffers, Spill Thresholds, Shuffle Parallelism, and GCPlanned lesson · reserved path Chapter23/Lesson2.html

Planned03

Input/Map-Output/Final-Output Compression and Codec Selection for CPU, I/O, and NetworkPlanned lesson · reserved path Chapter23/Lesson3.html

Planned04

Data Skew, Salting/Custom Partitioning, Combiners, Pre-Aggregation, and Straggler MitigationPlanned lesson · reserved path Chapter23/Lesson4.html

Planned05

Create a Benchmark Matrix and Tune a Job using Runtime, CPU, GC, Shuffle Bytes, Spills, Locality, and p95 Task DurationPlanned lesson · reserved path Chapter23/Lesson5.html

Planned

Chapter 24

Hadoop I/O Formats, Serialization, Compression, SequenceFile, and Data Exchange

5 lessons

Writable/WritableComparable, Serialization Interfaces, Type Contracts, and Java Ecosystem CompatibilityPlanned lesson · reserved path Chapter24/Lesson1.html

Planned02

SequenceFile/MapFile-Like Binary Containers, Sync Markers, Compression Modes, and SplittabilityPlanned lesson · reserved path Chapter24/Lesson2.html

Planned03

Text/CSV/JSON/Avro/Parquet/ORC Integration through Ecosystem Libraries and Schema TradeoffsPlanned lesson · reserved path Chapter24/Lesson3.html

Planned04

Gzip, Bzip2, Snappy, LZ4, ZSTD-Like Codecs, Splittability, Native Support, and Codec ConfigurationPlanned lesson · reserved path Chapter24/Lesson4.html

Planned05

Choose an HDFS Storage Format/Codec for Raw Ingest, MapReduce Intermediate, Hive Analytics, and ArchivalPlanned lesson · reserved path Chapter24/Lesson5.html

Planned

Chapter 25

DistCp, Hadoop Archives, Data Movement, Checksums, and Large-Scale Migration

5 lessons

DistCp Architecture, Parallel Copy Listings/Tasks, Update/Overwrite/Delete Modes, and Metadata PreservationPlanned lesson · reserved path Chapter25/Lesson1.html

Planned02

Source/Destination Checksums, Snapshot-Diff-Based Copy Concepts, Bandwidth Throttling, and Failure ResumePlanned lesson · reserved path Chapter25/Lesson2.html

Planned03

Copy Across HDFS Clusters/Object Stores, Credentials, Network Paths, and Cross-Region CostPlanned lesson · reserved path Chapter25/Lesson3.html

Planned04

Hadoop Archives (HAR) and Historical Small-File Packaging Tradeoffs vs Modern Compaction/Table FormatsPlanned lesson · reserved path Chapter25/Lesson4.html

Planned05

Migrate a Large Dataset with DistCp, Validate Counts/Checksums/Permissions, and Design Rollback/Resume ProceduresPlanned lesson · reserved path Chapter25/Lesson5.html

Planned

Chapter 26

Hadoop Security: Kerberos, Delegation Tokens, Service Principals, and Authentication Flow

5 lessons

Simple Mode vs Secure Mode and Why Production Hadoop Traditionally Uses KerberosPlanned lesson · reserved path Chapter26/Lesson1.html

Planned02

Kerberos Principals, Keytabs, KDC, Tickets, SPNEGO, Hostnames/DNS, Clock Synchronization, and Common FailuresPlanned lesson · reserved path Chapter26/Lesson2.html

Planned03

HDFS/YARN Service Principals, Daemon Identities, Proxy Users, and User-to-Group MappingPlanned lesson · reserved path Chapter26/Lesson3.html

Planned04

Delegation Tokens for Long-Running Applications Without Distributing User Keytabs and Token Renewal/CancellationPlanned lesson · reserved path Chapter26/Lesson4.html

Planned05

Trace an Authenticated HDFS/YARN Request and Troubleshoot Expired Ticket, Clock, DNS, Keytab, and Authorization ErrorsPlanned lesson · reserved path Chapter26/Lesson5.html

Planned

Chapter 27

HDFS Encryption, KMS, Encryption Zones, Data Transfer Protection, and Key Operations

5 lessons

Hadoop KMS, Key Providers, Encryption Keys/EDEKs, Envelope Encryption, and Separation of Storage from KeysPlanned lesson · reserved path Chapter27/Lesson1.html

Planned02

HDFS Transparent Data Encryption and Encryption Zones: File Creation, Rename Constraints, and Snapshot InteractionPlanned lesson · reserved path Chapter27/Lesson2.html

Planned03

Key Rotation/Re-Encryption Concepts, KMS HA, Authorization, Audit, Backup, and Disaster RecoveryPlanned lesson · reserved path Chapter27/Lesson3.html

Planned04

Data-in-Transit Protection for RPC, HTTP, and DataNode Transfer; TLS/SASL/QOP-Like ControlsPlanned lesson · reserved path Chapter27/Lesson4.html

Planned05

Create an Encryption Zone and Run a KMS-Outage/Key-Rotation Drill with Data-Recovery DocumentationPlanned lesson · reserved path Chapter27/Lesson5.html

Planned

Chapter 28

YARN Security, Queue ACLs, Container Isolation, Credentials, and Multi-Tenant Controls

5 lessons

YARN Authentication, Delegation Tokens, RM/NM/ApplicationMaster Trust Relationships, and Secure SubmissionPlanned lesson · reserved path Chapter28/Lesson1.html

Planned02

Queue/Application ACLs, Admin/Submit Permissions, Node Labels, and Tenant Resource BoundariesPlanned lesson · reserved path Chapter28/Lesson2.html

Planned03

Container Tokens, Localized Credentials, Secret Exposure, Linux Container Executor, and Privileged OperationsPlanned lesson · reserved path Chapter28/Lesson3.html

Planned04

Docker/Containerized Execution Concepts, Image Trust, Host Mounts, User Mapping, and Runtime SecurityPlanned lesson · reserved path Chapter28/Lesson4.html

Planned05

Threat-Model a Shared YARN Cluster Against Malicious Jobs, Credential Theft, Resource Abuse, and Data ExfiltrationPlanned lesson · reserved path Chapter28/Lesson5.html

Planned

Chapter 29

Hadoop Observability: Web UIs, Metrics2, JMX, Logs, Audit Logs, and Operational Dashboards

5 lessons

NameNode/DataNode/ResourceManager/NodeManager/JobHistory UIs and What Health Signals They ExposePlanned lesson · reserved path Chapter29/Lesson1.html

Planned02

Hadoop Metrics2/JMX, Prometheus/JMX Exporter-Like Integration, OS Metrics, and Time-Series DashboardsPlanned lesson · reserved path Chapter29/Lesson2.html

Planned03

Daemon Logs, GC Logs, Application/Container Logs, Aggregation, Audit Logs, and Correlation IDs/Application IDsPlanned lesson · reserved path Chapter29/Lesson3.html

Planned04

SLIs for HDFS Capacity/Under-Replication, RPC Latency, YARN Queue Wait, Container Failures, and Job RuntimePlanned lesson · reserved path Chapter29/Lesson4.html

Planned05

Build Runbooks for NameNode Load, Missing Blocks, DataNode Failure, Queue Saturation, and MapReduce Job RegressionPlanned lesson · reserved path Chapter29/Lesson5.html

Planned

Chapter 30

Operations, Configuration Management, Rolling Restart, Upgrade, and Compatibility

5 lessons

Configuration Files Across Nodes, Drift Detection, Secrets, Host Lists, Include/Exclude, and Environment PromotionPlanned lesson · reserved path Chapter30/Lesson1.html

Planned02

Daemon Start/Stop/Refresh, Rolling Restart, Maintenance State, Client Compatibility, and Workload DrainPlanned lesson · reserved path Chapter30/Lesson2.html

Planned03

Upgrade Planning, Rolling Upgrade Concepts, HDFS Layout/Metadata Compatibility, Finalize/Rollback BoundariesPlanned lesson · reserved path Chapter30/Lesson3.html

Planned04

Java/JDK, Native Libraries, Ecosystem Component Compatibility, and Staging/Canary ValidationPlanned lesson · reserved path Chapter30/Lesson4.html

Planned05

Build an Upgrade Runbook with Prechecks, HA/Backup Validation, Health Gates, Functional Tests, and Rollback CriteriaPlanned lesson · reserved path Chapter30/Lesson5.html

Planned

Chapter 31

Backup, Disaster Recovery, Metadata Protection, and Cluster Rebuild

5 lessons

HDFS Replication/HA vs Backup: Protecting Against Hardware Failure vs Operator Error/Corruption/Site LossPlanned lesson · reserved path Chapter31/Lesson1.html

Planned02

Protect NameNode Metadata, Journal/Checkpoint State, Configuration, Kerberos/KMS Material, and External DependenciesPlanned lesson · reserved path Chapter31/Lesson2.html

Planned03

Snapshots + DistCp to Secondary Cluster/Object Store for Recoverable Copies and RetentionPlanned lesson · reserved path Chapter31/Lesson3.html

Planned04

RPO/RTO for NameNode Loss, Multi-DataNode Loss, Site Loss, Accidental Delete, KMS Failure, and CorruptionPlanned lesson · reserved path Chapter31/Lesson4.html

Planned05

Run a Recovery Drill that Reconstructs Namespace/Services and Verifies Data, Permissions, Encryption, Applications, and Audit EvidencePlanned lesson · reserved path Chapter31/Lesson5.html

Planned

Chapter 32

Cloud and Object-Store Connectors: S3A, ABFS, GCS, Credentials, Committers, and Semantics

5 lessons

Hadoop FileSystem Abstraction over S3A, Azure ABFS, and Google Cloud Storage; HDFS vs Object-Store SemanticsPlanned lesson · reserved path Chapter32/Lesson1.html

Planned02

Credential Providers, IAM/STS/Managed Identity, Endpoint/Region, TLS, Proxies, and Secret HygienePlanned lesson · reserved path Chapter32/Lesson2.html

Planned03

Multipart Upload, Range Reads, Directory Markers/Listings, Rename Cost, Consistency, and Request EconomicsPlanned lesson · reserved path Chapter32/Lesson3.html

Planned04

S3A/Object-Store Committer Concepts for Safe/Scalable Job Output Without Expensive Rename-Copy PatternsPlanned lesson · reserved path Chapter32/Lesson4.html

Planned05

Benchmark HDFS vs Object Storage and Document Latency, Throughput, Request Count, Failure, and Cost TradeoffsPlanned lesson · reserved path Chapter32/Lesson5.html

Planned

Chapter 33

Hadoop Ecosystem Integration: Hive, HBase, Spark, Trino, and Shared Storage/Resource Layers

5 lessons

Hive Uses HDFS/Object Storage plus Metastore/Table Formats; Separate SQL Metadata from Physical StoragePlanned lesson · reserved path Chapter33/Lesson1.html

Planned02

HBase Uses HDFS for Distributed Storage with Its Own Region/Write Path and Operational RequirementsPlanned lesson · reserved path Chapter33/Lesson2.html

Planned03

Spark on YARN: Driver/Executor/ApplicationMaster Roles, Dynamic Allocation, Locality, and Shuffle InteractionPlanned lesson · reserved path Chapter33/Lesson3.html

Planned04

Trino/Other SQL Engines Reading HDFS/Object Storage via Catalogs and Open File/Table FormatsPlanned lesson · reserved path Chapter33/Lesson4.html

Planned05

Design a Shared Platform Boundary Showing Which Guarantees Belong to HDFS, YARN, Hive/HBase/Spark, and Table FormatsPlanned lesson · reserved path Chapter33/Lesson5.html

Planned

Chapter 34

Troubleshooting Hadoop: Missing Blocks, RPC Saturation, Slow DataNodes, YARN Failures, and Job Pathologies

5 lessons

HDFS fsck, Block Reports, Under/Mis-Replicated/Missing/Corrupt Blocks, Safemode, and DataNode DiagnosticsPlanned lesson · reserved path Chapter34/Lesson1.html

Planned02

NameNode RPC Queues/Handlers, GC, Heap, Large Directories, Block Storms, and Client Retry CascadesPlanned lesson · reserved path Chapter34/Lesson2.html

Planned03

Slow/Failed DataNodes: Disk, Network, Checksum, Volume Failure, Pipeline Recovery, and Decommission DecisionsPlanned lesson · reserved path Chapter34/Lesson3.html

Planned04

YARN Pending Apps, Container Kills, Memory Limits, Lost Nodes, Queue Policies, AM Failures, and Log AnalysisPlanned lesson · reserved path Chapter34/Lesson4.html

Planned05

Use a Symptom→Evidence→Hypothesis→Controlled Change Workflow on a Multi-Layer Hadoop IncidentPlanned lesson · reserved path Chapter34/Lesson5.html

Planned

Chapter 35

Production Capstone: Design, Secure, Operate, and Recover a Modern Hadoop Platform

5 lessons

Define Dataset Scale, File/Object Counts, Ingest, Compute, Multi-Tenancy, Availability, RPO/RTO, Security, and Cloud-Integration RequirementsPlanned lesson · reserved path Chapter35/Lesson1.html

Planned02

Design HDFS HA/Federation/Erasure Coding/Storage Policies, YARN Queues/HA/Federation, Kerberos/KMS, and Network/Failure DomainsPlanned lesson · reserved path Chapter35/Lesson2.html

Planned03

Implement Data Movement, MapReduce/Spark-on-YARN Workloads, Object-Store Connectors, Metrics, Logs, Quotas, and GovernancePlanned lesson · reserved path Chapter35/Lesson3.html

Planned04

Load-Test, Decommission Nodes, Fail NameNode/DataNodes/RM, Restore from Backup, Rotate Keys, Upgrade, and Execute Incident RunbooksPlanned lesson · reserved path Chapter35/Lesson4.html

Planned05

Present the Architecture with Capacity Math, Failure Matrix, Security Model, Compatibility Plan, Measured Performance, and Migration/Future StrategyPlanned lesson · reserved path Chapter35/Lesson5.html

Planned