CPU, Memory, Load Average, and Pressure

1. Performance analysis starts with a resource model

A slow system is not automatically a CPU-bound system. A request can wait for runnable CPU time, memory reclaim, storage completion, a network response, a lock, a dependency, or an application queue. Begin with four categories: utilization describes how busy a resource is; saturation describes queued work that cannot run immediately; errors reveal failed work; and latency expresses the time users or downstream services experience.

flowchart TD
  S["User reports slowness"] --> T["Confirm time window and affected workload"]
  T --> C["Check CPU utilization and runnable work"]
  T --> M["Check memory availability, reclaim, and swap"]
  T --> P["Check PSI for CPU, memory, and I/O stalls"]
  C --> H["Form a bottleneck hypothesis"]
  M --> H
  P --> H
  H --> V["Validate with per-process and interval evidence"]
  V --> A["Change one factor or escalate with evidence"]

Every observation needs a time window and scope. A five-minute host average cannot explain a 200-millisecond request spike by itself. Likewise, a container may be throttled while the host appears mostly idle. Record the host, cgroup or container, workload, and exact interval before interpreting numbers.

2. CPU percentages describe where scheduler time went

Tools such as top, mpstat, vmstat, and sar derive CPU categories from kernel counters. User time represents execution outside the kernel; system time represents kernel execution; idle means no runnable work was scheduled on that logical CPU; I/O-wait is a CPU accounting category associated with waiting for I/O and is not the same as device utilization; steal time indicates time a virtual CPU wanted to run but the hypervisor scheduled another guest.

# Host-wide and per-CPU snapshots.
uptime
nproc
mpstat -P ALL 1 5

# Find tasks consuming CPU and tasks switching frequently.
pidstat -u -w 1 5

# Inspect the busiest processes without an interactive UI.
ps -eo pid,ppid,psr,ni,stat,%cpu,%mem,comm --sort=-%cpu | head -n 20

A high aggregate CPU percentage can hide one saturated core, and a low aggregate value can hide cgroup throttling or a single-threaded bottleneck. Compare per-CPU distribution, process state, context switches, run-queue length, and application throughput. High system time may indicate syscall, networking, storage, or interrupt work, but it does not identify which subsystem without additional evidence.

3. Load average counts runnable and uninterruptible work

Linux load average summarizes tasks that are runnable or waiting in uninterruptible sleep, commonly including storage-related waits. The three values represent exponentially smoothed one-, five-, and fifteen-minute views. They are not percentages and do not mean “CPU usage.” A rough capacity-oriented comparison divides load by the number of logical CPUs:

# Read the same load figures shown by uptime.
cat /proc/loadavg
uptime

# Count logical CPUs visible to this execution context.
nproc

# Inspect runnable and uninterruptible tasks.
ps -eo state,pid,ppid,wchan:28,comm | awk '$1 ~ /^[RD]/ {print}' | head -n 30

A load of 8 may be comfortable on a 32-CPU host and severe on a 2-CPU host, but workload latency and scheduler behavior still decide whether it is acceptable. A rising one-minute value above the five- and fifteen-minute values indicates a recent increase; the reverse suggests recovery. Investigate whether tasks are runnable (R) or in uninterruptible sleep (D) before assuming CPU shortage.

4. “Free memory” is not the same as available memory

Linux deliberately uses unused RAM for caches. MemFree alone therefore overstates pressure. MemAvailable estimates memory that can be allocated without swapping under normal conditions. Separate anonymous memory, page cache, slab, dirty pages, swap usage, and reclaim activity. A system can have little free memory and still be healthy; a system with available memory can still experience cgroup limits or short reclaim stalls.

free -h
awk '/MemTotal|MemAvailable|MemFree|Buffers|Cached|SwapTotal|SwapFree|Dirty|Writeback|Slab/ {print}' /proc/meminfo

# Interval evidence: paging, swapping, run queue, and CPU categories.
vmstat 1 10

# Per-process resident memory and major/minor faults.
pidstat -r 1 5

# Check for kernel OOM decisions in the current boot.
journalctl -k -b --grep='Out of memory|oom-kill|Killed process' --no-pager

Minor faults usually resolve without storage access, while major faults require data to be brought in from backing storage. Swap use is not automatically a fault: inactive pages may remain swapped while the system is healthy. Sustained swap-in and swap-out, increasing major faults, memory PSI, reclaim activity, and latency together are stronger evidence of memory pressure.

5. PSI measures time lost to resource contention

Pressure Stall Information is exported through /proc/pressure/cpu, /proc/pressure/memory, and /proc/pressure/io. The some line reports periods when at least some tasks were stalled. The full line reports periods when all non-idle work was stalled for that resource class. Values such as avg10, avg60, and avg300 are percentages over recent windows; total is cumulative stall time in microseconds.

for resource in cpu memory io; do
  printf '\n=== %s pressure ===\n' "$resource"
  cat "/proc/pressure/$resource"
done

# Container or service cgroups may expose their own pressure files.
service_cgroup=$(systemctl show -p ControlGroup --value ssh.service 2>/dev/null || true)
if [[ -n $service_cgroup && -r /sys/fs/cgroup${service_cgroup}/memory.pressure ]]; then
  cat "/sys/fs/cgroup${service_cgroup}/memory.pressure"
fi

PSI complements utilization. High CPU utilization with little CPU pressure may mean the host is busy but keeping up. Noticeable CPU pressure means runnable work is waiting. Memory full pressure is especially serious because productive work can stop while the system thrashes. Compare pressure with application latency, queue depth, and per-process evidence.

6. Use a disciplined CPU and memory workflow

1. Confirm impact: identify the affected service, hosts, containers, and time window.
2. Capture a host overview: uptime, CPU count, load, memory, swap, and PSI.
3. Sample intervals: use vmstat, mpstat, and pidstat for repeated measurements.
4. Identify scope: determine whether the pressure is host-wide, per CPU, per process, or limited to a cgroup.
5. Correlate: align metrics with request latency, throughput, deploys, batch work, and kernel events.
6. Test one hypothesis: reproduce in a safe environment or make one controlled change.

7. Hands-on lab: capture a read-only performance snapshot

This lab creates a timestamped evidence directory and gathers read-only host data. Commands unavailable on your distribution are skipped rather than installed automatically.

lab="$HOME/devops-academy/linux/chapter16/lesson01"
run_dir="$lab/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$run_dir"

{
  printf 'captured_utc=%s\n' "$(date -u --iso-8601=seconds)"
  printf 'hostname=%s\n' "$(hostname)"
  printf 'kernel=%s\n' "$(uname -r)"
  printf 'logical_cpus=%s\n' "$(nproc)"
  uptime
} > "$run_dir/overview.txt"

free -h > "$run_dir/free.txt"
cat /proc/loadavg > "$run_dir/loadavg.txt"
cat /proc/meminfo > "$run_dir/meminfo.txt"

for resource in cpu memory io; do
  cat "/proc/pressure/$resource" > "$run_dir/pressure-$resource.txt"
done

if command -v vmstat >/dev/null; then vmstat 1 5 > "$run_dir/vmstat.txt"; fi
if command -v mpstat >/dev/null; then mpstat -P ALL 1 5 > "$run_dir/mpstat.txt"; fi
if command -v pidstat >/dev/null; then pidstat -u -r -w 1 5 > "$run_dir/pidstat.txt"; fi

printf 'Evidence written to %s\n' "$run_dir"
find "$run_dir" -maxdepth 1 -type f -printf '%f\n' | sort

8. Common interpretation mistakes

9. Knowledge check

10. Summary

• Performance analysis combines utilization, saturation, errors, throughput, and latency.
• CPU categories are accounting signals, not standalone root causes.
• Load average is a queue-oriented measure and should be interpreted with CPU count and task state.
• Healthy memory analysis emphasizes availability, reclaim, faults, swap movement, and cgroup scope.
• PSI quantifies workload stall time caused by resource contention.

11. Further reading

• Linux kernel documentation — Pressure Stall Information
• proc_loadavg(5)
• proc_meminfo(5)
• vmstat(8)
• pidstat(1)