Performance Tools: vmstat, iostat, sar, lsof, and strace

1. Start with a question, not a favorite command

Tools overlap, but each has a useful scope. vmstat gives a compact host-wide view of runnable tasks, memory, paging, block activity, interrupts, context switches, and CPU categories. iostat focuses on CPUs and block devices. sar provides interval and historical reports. pidstat attributes CPU, memory, I/O, and scheduling activity to tasks. lsof maps processes to open resources. strace shows system calls and signals at the user/kernel boundary.

flowchart LR
  Q["What is the diagnostic question?"] --> H{"Host saturation now?"}
  H -- yes --> V["vmstat, mpstat, iostat"]
  Q --> R{"Which process?"}
  R -- yes --> P["pidstat, ps, top"]
  Q --> F{"Which file, mount, or socket?"}
  F -- yes --> L["lsof and ss"]
  Q --> Y{"What happened earlier?"}
  Y -- yes --> S["sar and sadf archives"]
  Q --> C{"Which syscall is blocking or failing?"}
  C -- yes --> T["narrow strace capture"]

2. vmstat is a compact interval dashboard

With an interval and count, vmstat prints repeated snapshots. The first report generally summarizes since-boot activity; subsequent reports cover the requested intervals. Key fields include runnable tasks (r), uninterruptible tasks (b), swap in/out, block input/output, interrupts, context switches, and CPU categories.

# Ignore the first since-boot line when studying current behavior.
vmstat -w 1 6

# Timestamp output when supported by procps-ng.
vmstat -t 1 6

# Active and inactive memory summary.
vmstat -a 1 5

# Fork counts and selected summary values.
vmstat -s | head -n 30

A sustained run queue larger than available CPU capacity can indicate CPU saturation, but validate with per-CPU and process data. Swap-in/out rates matter more than swap occupancy alone. High context-switch rates may be expected for network servers; compare with a baseline and application throughput.

3. iostat connects block-device rates, queues, and latency

Use extended reports, human-readable device names, and interval sampling. As with vmstat, the first report often covers time since boot while later reports cover the interval. Device names may represent logical mappings rather than physical hardware, so pair the output with lsblk and findmnt.

# Extended statistics, omit zero-activity devices, five intervals.
iostat -xz 1 5

# Display persistent device names when supported.
iostat -xz -j ID 1 5

# Relate devices to filesystems and mappings.
lsblk -o NAME,KNAME,TYPE,MAJ:MIN,SIZE,FSTYPE,MOUNTPOINTS
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS

Do not compare await thresholds across unrelated device classes. A latency acceptable for a bulk backup may violate a database requirement. Also confirm whether device-level delay actually overlaps the application incident.

4. sar turns current sampling into historical evidence

The sysstat collector stores binary activity files, commonly under /var/log/sa or /var/log/sysstat. sar can report CPU, run queue, memory, paging, block devices, network, and other categories for a time range. Collection frequency and retention are distribution-specific, and missing data may reflect a disabled collector rather than a healthy host.

# Current interval samples.
sar -u ALL 1 5
sar -q 1 5
sar -r ALL 1 5
sar -n DEV,TCP,ETCP 1 5

# Historical example for today's default activity file.
sar -u -s 02:00:00 -e 03:00:00
sar -q -s 02:00:00 -e 03:00:00

# Export archive data in JSON when sadf supports it.
sadf -j -- -u -q -r > sysstat.json

Check timezone, host identity, reboot boundaries, collection gaps, and units before correlating historical data. Preserve the source activity file when it is incident evidence; generated summaries can always be recreated later.

5. lsof maps processes to open kernel objects

Linux represents regular files, directories, sockets, devices, pipes, and many other resources through file descriptors. lsof can answer who owns a port, who prevents an unmount, which process holds a deleted file, or what files a service opened. Results may be incomplete without privilege, and a full system scan can be expensive on a large host.

# Files opened by one process.
lsof -p 1234

# Process using a TCP port.
lsof -nP -iTCP:8080 -sTCP:LISTEN

# Processes using a mount or directory tree; scope carefully.
lsof +D /srv/app 2>/dev/null

# Deleted files that are still open.
lsof +L1 2>/dev/null

# Machine-friendly field output for PID, command, FD, type, device, and name.
lsof -F pcftDn -p 1234

+D recursively descends a directory and can be costly. Prefer a specific path, PID, port, or filesystem. Treat names and command lines as potentially sensitive incident data.

6. strace reveals syscalls, failures, and waits

strace intercepts system calls and signals. It is invaluable when a process cannot open a file, waits on a socket, loops on time calls, or receives unexpected signals. Tracing adds overhead, may alter timing, requires ptrace permission, and can capture secrets from arguments or buffers. Begin with a reproducer or a short-lived command whenever possible.

# Trace only file-related calls for a short command.
strace -f -e trace=%file -o trace-file.log -- ls /path/that/may/not/exist

# Summarize syscall counts and time without printing every event.
strace -c -- curl --silent --output /dev/null http://127.0.0.1:8000/

# Trace network calls with timestamps and duration.
strace -f -tt -T -e trace=%network -o trace-network.log -- command args

# Attach briefly to an existing PID; obtain authorization first.
timeout 10s strace -f -tt -T -p 1234 -o trace-attach.log

Filter by syscall class, path, signal, or file descriptor. Use -o to keep trace output separate. -f follows descendants and increases volume. Strings may be truncated unless changed, but increasing string length can expose more sensitive data. A trace explains kernel interactions, not application-level intent.

7. Combine tools in a staged investigation

Suppose requests slow while CPU graphs look moderate. vmstat shows several blocked tasks and I/O pressure. iostat shows rising latency on the volume backing the application. pidstat -d attributes writes to a log processor. lsof +L1 finds a large deleted log held open. The evidence points to a retention/reopen failure rather than “slow Linux.” strace is unnecessary because the existing layers already explain the behavior.

Escalate tool depth only when the current evidence leaves a specific unanswered question.

8. Hands-on lab: create a bounded evidence bundle

The script gathers short interval reports and metadata. It skips unavailable tools and never attaches to a production process.

lab="$HOME/devops-academy/linux/chapter16/lesson04"
run_dir="$lab/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$run_dir"

{
  date -u --iso-8601=seconds
  hostname
  uname -a
  uptime
} > "$run_dir/identity.txt"

command -v vmstat >/dev/null && vmstat -w 1 6 > "$run_dir/vmstat.txt"
command -v iostat >/dev/null && iostat -xz 1 5 > "$run_dir/iostat.txt"
command -v pidstat >/dev/null && pidstat -u -r -d -w 1 5 > "$run_dir/pidstat.txt"
command -v sar >/dev/null && sar -q 1 5 > "$run_dir/sar-queue.txt"
command -v lsof >/dev/null && lsof +L1 > "$run_dir/deleted-open-files.txt" 2>&1 || true

find "$run_dir" -type f -maxdepth 1 -printf '%f %s bytes\n' | sort
sha256sum "$run_dir"/* > "$run_dir/SHA256SUMS"
printf 'Bundle: %s\n' "$run_dir"

9. Common tool-selection mistakes

10. Knowledge check

11. Summary

• Select a tool from the diagnostic question and required scope.
• Understand since-boot versus interval semantics.
• Use sysstat archives as time-bounded evidence with timezone and collection context.
• lsof connects processes to files, sockets, mounts, and deleted resources.
• strace is a narrow, potentially invasive syscall-level instrument.

12. Further reading

• vmstat(8)
• iostat(1)
• sar(1)
• lsof(8)
• strace(1)