Disk I/O, Filesystem Latency, and Capacity

1. Storage latency can originate at several layers

An application issues reads, writes, metadata operations, or synchronization calls. The kernel may satisfy reads from page cache, buffer writes in memory, submit block requests, merge or schedule them, and wait for a local disk, virtual device, RAID layer, network volume, or cloud storage service. Filesystem locks, journal commits, device throttling, thin-provisioned pools, and remote storage can all add latency.

flowchart TD
  A["Application read/write/fsync"] --> V["VFS and filesystem"]
  V --> C{"Page cache hit?"}
  C -- yes --> R["Return cached data"]
  C -- no --> B["Block layer queue"]
  V --> W["Dirty pages and writeback"]
  W --> B
  B --> D["Device, RAID, virtual or remote storage"]
  D --> Q["Completion and observed latency"]

Start by identifying the exact mount, filesystem, backing device, and workload. A path under a container may map to overlay storage; a mount may be network-backed; a logical volume may span several devices. Do not assume a filename maps directly to one physical disk.

findmnt -T /var/lib
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS,MODEL
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS

# Resolve the device that backs a path.
df -hT /var/lib
df -i /var/lib

2. Capacity incidents include bytes, inodes, reserved space, and hidden usage

df reports filesystem allocation; du walks visible directory entries. They can disagree because a process still holds a deleted file open, a mount hides files beneath a mount point, snapshots retain blocks, sparse files report different logical and allocated sizes, or privileges prevent traversal. A filesystem can also run out of inodes while free bytes remain.

# Filesystem-level byte and inode capacity.
df -hT
df -i

# Directory-level usage without crossing filesystem boundaries.
du -xhd1 /var 2>/dev/null | sort -h

# Large regular files on one filesystem.
find /var -xdev -type f -size +500M -printf '%s %p\n' 2>/dev/null \
  | sort -nr | head -n 20

# Deleted files that remain allocated because a process holds them open.
lsof +L1 2>/dev/null | head -n 30

3. Interpret device metrics as rates and queues

iostat -xz reports extended block-device statistics. Names vary slightly by sysstat version, but the important dimensions are operations per second, transferred bytes per second, average request size, queue depth, completion latency, and busy time. Compare interval samples, not only the first report since boot.

# Extended device reports every second, five intervals.
iostat -xz 1 5

# Per-process I/O rates and delays.
pidstat -d 1 5

# Kernel-wide block and paging context.
vmstat 1 5

# Optional: inspect one device's scheduler and queue settings.
device=sda
for file in scheduler nr_requests rotational; do
  path="/sys/block/$device/queue/$file"
  [[ -r $path ]] && printf '%s=%s\n' "$file" "$(<"$path")"
done

High await means completed requests spent longer in queue and service. High queue depth with rising latency suggests saturation, but acceptable values depend on device class and workload. A high %util can be meaningful for a single rotating disk yet misleading for parallel arrays, NVMe, virtual devices, and devices that process many requests concurrently. Always correlate with application latency and throughput.

4. Page cache changes what “disk activity” means

Buffered reads may complete without device I/O. Buffered writes may return after copying data into memory, while writeback occurs later. fsync, database durability settings, journal commits, direct I/O, and memory pressure alter this behavior. A workload can therefore show low write latency initially and then encounter throttling when dirty pages must be flushed.

awk '/Cached|Dirty|Writeback|MemAvailable/ {print}' /proc/meminfo

# Observe writeback and block activity at intervals.
vmstat -w 1 10

# Read selected kernel counters before and after a controlled lab.
grep -E '^(pgpgin|pgpgout|pswpin|pswpout|nr_dirty|nr_writeback) ' /proc/vmstat

Dropping caches on production systems is not a neutral diagnostic action. It changes workload behavior and can create a latency storm. Likewise, running sync or a large benchmark can affect every tenant of a shared device. Prefer existing telemetry and reproduce tests on disposable storage.

5. Filesystem and mount semantics shape latency

Metadata-heavy workloads stress directory lookup, inode allocation, journaling, and locking rather than sequential throughput. Small synchronous writes behave differently from large asynchronous streams. Network filesystems add server latency and retransmission behavior. Overlay filesystems can add copy-up work. Mount options, quotas, encryption, compression, and snapshots also influence cost.

# Identify filesystem type and mount options for a specific path.
findmnt -T /srv/app -o TARGET,SOURCE,FSTYPE,OPTIONS

# Compare apparent size with allocated blocks for sparse files.
stat --format='size=%s bytes blocks=%b block_size=%B path=%n' /path/to/file

du -h /path/to/file
du --apparent-size -h /path/to/file

# Look for filesystem and block-device warnings in the current boot.
journalctl -k -b --grep='I/O error|EXT4-fs|XFS|BTRFS|blk_update_request|nvme' --no-pager

Device metrics cannot explain every filesystem wait. If block latency is low while application file operations are slow, investigate locks, metadata contention, path traversal, remote dependencies, or application serialization.

Cloud and virtual disks add another boundary: the guest may see a virtual block queue while the provider enforces burst credits, bandwidth tiers, or multi-tenant limits underneath it. Preserve instance type, volume class, provisioned IOPS or throughput, attachment topology, and provider-side telemetry when available. A guest-only conclusion should explicitly state when the physical service layer could not be observed.

6. Hands-on lab: profile a disposable file safely

This lab operates only inside your home directory. It measures cached and synchronized writes as separate operations and records system evidence. Results are educational, not a storage benchmark.

lab="$HOME/devops-academy/linux/chapter16/lesson02"
rm -rf "$lab"
mkdir -p "$lab"
cd "$lab"

{
  date -u --iso-8601=seconds
  findmnt -T . -o TARGET,SOURCE,FSTYPE,OPTIONS
  df -hT .
  df -i .
} > environment.txt

# Create a 64 MiB file in a disposable directory.
/usr/bin/time -f 'elapsed=%e user=%U system=%S' \
  dd if=/dev/zero of=sample.bin bs=1M count=64 status=none 2> buffered-write.time

# Request durability separately and time that operation.
/usr/bin/time -f 'elapsed=%e user=%U system=%S' \
  sync sample.bin 2> sync.time

stat --format='size=%s blocks=%b block_size=%B' sample.bin > allocation.txt
du -h sample.bin >> allocation.txt
du --apparent-size -h sample.bin >> allocation.txt

printf '%s\n' '--- environment ---'
cat environment.txt
printf '%s\n' '--- timings ---'
cat buffered-write.time sync.time
printf '%s\n' '--- allocation ---'
cat allocation.txt

rm -f sample.bin

7. Common storage mistakes

8. Knowledge check

9. Summary

• Storage latency can arise in applications, filesystems, caches, block queues, devices, or remote layers.
• Capacity analysis includes bytes, inodes, deleted-open files, snapshots, and mount topology.
• Interpret IOPS, throughput, latency, queue depth, and utilization together.
• Page cache and writeback separate apparent application latency from durable completion.
• Production evidence collection should be read-only and workload-aware.

10. Further reading

• iostat(1)
• pidstat(1)
• lsof(8)
• proc_vmstat(5)
• sysstat FAQ