RAID Concepts, Swap, and Storage Troubleshooting

1. RAID changes failure and performance behavior; it does not eliminate failure

Redundant Array of Independent Disks combines devices into one logical block device. Depending on the level, data may be striped for throughput, mirrored for redundancy, or protected by distributed parity. RAID improves availability against selected device failures, but it does not protect against accidental deletion, corruption replicated across members, ransomware, controller defects, site loss, or operator error.

flowchart TB
  A["Application path"] --> B["Filesystem"]
  B --> C["LVM or encryption"]
  C --> D["RAID device"]
  D --> E1["Member disk 1"]
  D --> E2["Member disk 2"]
  D --> E3["Member disk 3"]
  F["Monitoring and backups"] -. protect different failure modes .-> A

2. Choose a RAID level from failure tolerance and workload

Capacity formulas are simplified: metadata, alignment, spares, filesystem overhead, and vendor implementation reduce usable space. Failure tolerance also depends on which members fail. Rebuilds read large portions of surviving devices, increasing latency and exposing latent errors.

3. Linux md RAID exposes health and rebuild state

# Kernel summary for software RAID arrays
cat /proc/mdstat

# Detailed metadata for one confirmed array
sudo mdadm --detail /dev/md0

# Examine member metadata without assembling or modifying it
sudo mdadm --examine /dev/sdX1

# Correlate the array with higher layers
lsblk -o NAME,PATH,TYPE,SIZE,FSTYPE,MOUNTPOINTS
findmnt --source /dev/md0 2>/dev/null || true

Key states include clean, active, degraded, recovering, resyncing, reshaping, or failed. Do not automatically re-add or replace a member until you have confirmed device identity, array metadata, event counters, failure reason, and whether the remaining array is internally consistent.

4. Swap supports memory management but is not replacement RAM

Swap provides disk-backed pages that the kernel can use under memory pressure and for selected features such as hibernation. It can increase resilience against short-lived pressure and preserve more page cache, but sustained swap I/O can make a system extremely slow. A system can also encounter the out-of-memory killer even with swap when limits, cgroups, commitments, or workload behavior make allocation impossible.

# Active swap devices/files and priority
swapon --show --bytes
cat /proc/swaps

# Memory and swap totals
free -h

# Observe paging activity over time
vmstat 1 10

# Relevant policy
sysctl vm.swappiness vm.overcommit_memory vm.overcommit_ratio 2>/dev/null

Swap can be a partition, logical volume, or file. A swap file should be created with safe permissions, initialized by mkswap, activated with swapon, and declared carefully in fstab. Copy-on-write filesystems may require special procedures. Never deactivate swap during pressure without confirming enough available memory.

5. Troubleshoot storage from the symptom through every layer

path=${1:-/var}
findmnt --target "$path" -o TARGET,SOURCE,FSTYPE,OPTIONS
df -hT "$path"
df -i "$path"
lsblk -o NAME,PATH,TYPE,SIZE,RO,FSTYPE,MOUNTPOINTS
cat /proc/mdstat
sudo pvs 2>/dev/null || true
sudo vgs 2>/dev/null || true
sudo lvs -a 2>/dev/null || true
journalctl -k --since '-2 hours' 2>/dev/null \
  | grep -Ei 'I/O error|reset|timeout|nvme|ata|scsi|md|filesystem|read-only' \
  | tail -n 200

6. Repair tools require offline-state and backup decisions

Filesystem repair utilities interpret and modify metadata. ext-family tools use e2fsck; XFS uses xfs_repair; Btrfs has its own check and recovery tools. The correct tool, mount state, options, and recovery expectations are filesystem-specific. Running a generic repair command on a mounted filesystem can worsen damage.

Hardware evidence tools such as smartctl or nvme smart-log help distinguish media, interface, temperature, wear, and controller problems, but virtual or managed cloud devices may expose different telemetry. A clean health summary does not prove that every block is readable.

# Identify type first
lsblk -f
findmnt --target /affected/path

# Read-only health evidence where supported
sudo smartctl -a /dev/CONFIRMED_DISK 2>/dev/null || true
sudo nvme smart-log /dev/CONFIRMED_NVME_DEVICE 2>/dev/null || true

# Do not run filesystem repair until the filesystem-specific
# offline procedure, backups, and recovery path are confirmed.

7. Hands-on lab: capture a storage incident evidence bundle

This lab is read-only and suitable for a lab VM. Review collected data before sharing it because device serials, mount paths, and host details may be sensitive.

lab="$HOME/devops-academy/linux/chapter10/lesson05"
mkdir -p "$lab"
cd "$lab"

target=${1:-/}
{
  printf 'timestamp=%s\n' "$(date --iso-8601=seconds)"
  printf 'host=%s\n' "$(hostname)"
  printf 'target=%s\n' "$target"
} > context.txt

findmnt --target "$target" -o TARGET,SOURCE,FSTYPE,OPTIONS > target-mount.txt
df -hT "$target" > target-capacity.txt
df -i "$target" > target-inodes.txt
lsblk -e 7 -o NAME,PATH,TYPE,SIZE,RO,FSTYPE,UUID,MOUNTPOINTS > block-topology.txt
cat /proc/mdstat > mdstat.txt
swapon --show --bytes > swap.txt
free -h > memory.txt
vmstat 1 5 > vmstat.txt

sudo pvs > pvs.txt 2>&1 || true
sudo vgs > vgs.txt 2>&1 || true
sudo lvs -a > lvs.txt 2>&1 || true
journalctl -k --since '-1 hour' > kernel-last-hour.txt 2>&1 || true

sha256sum ./*.txt > evidence.sha256
printf 'Bundle created at %s\n' "$PWD"
ls -lh

8. Common RAID, swap, and troubleshooting mistakes

9. Knowledge check

10. Summary

RAID changes capacity, performance, and tolerated member failures but does not replace backups. Swap is a memory-management tool whose activity must be interpreted over time. Storage troubleshooting maps the affected path through filesystems, LVM, RAID, and devices, preserves evidence, stabilizes the workload, and uses only layer-specific, tested repair procedures.

11. Further reading

• mdadm(8), Linux MD documentation, and distribution RAID-recovery guidance.
• swapon(8), free(1), vmstat(8), and kernel virtual-memory documentation.
• Filesystem-specific recovery manuals and hardware telemetry documentation for deployed devices.