Baselines, Bottleneck Analysis, and Incident Evidence

1. A baseline is a comparable known-good distribution

A baseline is not one average captured on an idle host. It describes expected behavior for a specific service, host class, software version, workload shape, time window, and environment. Daily traffic cycles, batch jobs, backups, deploys, cache warmth, and cloud-instance variability all create seasonality. Store percentiles and ranges, not only means.

flowchart TD
  I["Incident impact and time window"] --> B["Choose comparable known-good baseline"]
  B --> D["Measure utilization, saturation, errors, latency"]
  D --> H["State one falsifiable bottleneck hypothesis"]
  H --> E["Collect minimum evidence to test it"]
  E --> R{"Evidence supports hypothesis?"}
  R -- no --> N["Reject or refine hypothesis"]
  N --> H
  R -- yes --> C["Controlled mitigation or reproduction"]
  C --> P["Preserve evidence and document confidence"]

Compare like with like: same service tier, instance type, cgroup limits, storage class, request mix, and software release. A “before” period immediately preceding the incident may already contain degradation. Use several known-good periods when possible.

2. Use resource and service methods together

The USE method asks, for each resource: utilization, saturation, and errors. CPU utilization can be paired with run queues and throttling; memory use with reclaim, faults, swap movement, and PSI; storage throughput with queue depth, latency, and device errors; network throughput with queues, drops, retransmissions, and link errors. For request-driven services, the RED method adds request rate, error rate, and duration.

3. A bottleneck hypothesis must be falsifiable

“The server is overloaded” is too vague. A stronger hypothesis is: “Between 14:02 and 14:08 UTC, the application’s p99 latency increased because its cgroup exhausted CPU quota; host idle remained available, but cpu.stat throttled time and cgroup CPU pressure rose.” This statement names scope, interval, mechanism, and expected evidence.

1. Observation: directly measured fact, including source and timestamp.
2. Inference: interpretation connecting observations.
3. Test: evidence that would support or reject the inference.
4. Conclusion: confidence-weighted explanation after the test.

Observation:
  p99 latency rose from 180 ms to 2.4 s at 14:02 UTC.
  Host CPU remained 42% idle.
  service.slice/cpu.stat nr_throttled increased rapidly.
  service.slice/cpu.pressure avg10 reached 18.6.

Hypothesis:
  The service was CPU-quota saturated inside its cgroup.

Disconfirming evidence would include:
  No throttling delta, no cgroup CPU pressure, or latency remaining high
  after a controlled quota increase with the same request mix.

Conclusion after test:
  Supported / rejected / inconclusive, with confidence and caveats.

Do not skip disconfirming evidence. Several metrics can move together because they share a workload increase. Correlation narrows investigation; a controlled change, reproduction, or mechanism-specific evidence strengthens causation.

4. Build one timeline across telemetry and changes

Convert logs and metrics to a common timezone, preferably UTC. Include deploys, configuration changes, autoscaling, failovers, batch jobs, alerts, user reports, and mitigations. Record collection delay and clock skew. A graph without a version or change timeline can produce false conclusions.

# Record time sources and recent boot/change context.
date -u --iso-8601=seconds
timedatectl status
uptime -s

# Service and kernel events in an exact UTC interval.
journalctl --utc --since '2026-08-05 14:00:00' \
  --until '2026-08-05 14:15:00' -u example.service --no-pager
journalctl --utc -k --since '2026-08-05 14:00:00' \
  --until '2026-08-05 14:15:00' --no-pager

# Package history examples; availability is distribution-specific.
grep -hE '2026-08-05|upgrade|install' /var/log/apt/history.log* 2>/dev/null || true
dnf history list 2>/dev/null | head -n 20 || true

Preserve raw source timestamps and document conversions. If hosts have significant clock drift, ordering across systems may be uncertain even when each log looks precise.

5. Incident evidence must be reproducible and governed

An evidence bundle should identify who collected it, when, from which host and namespace, with what commands and versions. Preserve raw outputs before transforming them. Generate checksums. Restrict permissions. Review for secrets, tokens, IP addresses, usernames, command lines, packet payloads, and customer data before sharing.

#!/usr/bin/env bash
set -Eeuo pipefail

case_id=${1:?Usage: collect-evidence CASE_ID}
umask 077
root="$HOME/devops-academy/linux/chapter16/lesson05"
bundle="$root/${case_id}-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$bundle/raw"

run() {
  local name=$1
  shift
  {
    printf 'command='
    printf '%q ' "$@"
    printf '\nstarted_utc=%s\n' "$(date -u --iso-8601=seconds)"
    "$@"
  } > "$bundle/raw/$name.txt" 2>&1 || true
}

run identity uname -a
run uptime uptime
run load cat /proc/loadavg
run memory free -h
run pressure-cpu cat /proc/pressure/cpu
run pressure-memory cat /proc/pressure/memory
run pressure-io cat /proc/pressure/io
run sockets ss -s
run routes ip route show

command -v vmstat >/dev/null && run vmstat vmstat -w 1 6
command -v iostat >/dev/null && run iostat iostat -xz 1 5
command -v pidstat >/dev/null && run pidstat pidstat -u -r -d -w 1 5

find "$bundle/raw" -type f -print0 | sort -z | xargs -0 sha256sum > "$bundle/SHA256SUMS"
printf 'case_id=%s\ncreated_utc=%s\nhost=%s\n' \
  "$case_id" "$(date -u --iso-8601=seconds)" "$(hostname)" > "$bundle/MANIFEST"

printf 'Review and redact before sharing: %s\n' "$bundle"

6. Write an incident finding that survives review

A defensible report separates facts from interpretations and includes uncertainty. It should state impact, time window, affected scope, baseline, observations, hypothesis, tests, mitigation, recovery evidence, remaining risk, and follow-up actions. Avoid phrases such as “CPU was high” without values, units, scope, and time.

Finding: storage queue saturation on database volume

Impact:
  Checkout p99 latency exceeded 3 s for 11 minutes; 2.1% requests failed.

Scope and time:
  db-prod-03, /var/lib/postgresql, 14:02-14:13 UTC.

Baseline:
  Comparable weekday periods: await 1.8-3.2 ms, queue 0.2-0.6,
  p99 160-230 ms.

Incident observations:
  await 42-88 ms, queue 18-31, IO PSI full avg10 12-19%,
  backup process wrote 220 MiB/s, no device errors.

Test and mitigation:
  Pausing the unplanned backup reduced queue and latency within 90 s.

Conclusion:
  High confidence that concurrent backup I/O saturated the shared volume.

Caveats:
  Cloud-provider device telemetry was unavailable for the first 3 minutes.

Follow-up:
  Move backups to a separate performance class; add IO PSI and queue alerts.

Evidence of recovery is as important as evidence of failure. Show that user-visible latency, queueing, error rates, and resource pressure returned toward baseline after mitigation.

7. Hands-on lab: compare two bounded snapshots

Capture a quiet snapshot, run a small CPU workload in your own shell, capture an active snapshot, and compare. The workload is intentionally short and uses one process.

lab="$HOME/devops-academy/linux/chapter16/lesson05-lab"
rm -rf "$lab"
mkdir -p "$lab"

capture() {
  local label=$1
  {
    printf 'label=%s\n' "$label"
    printf 'utc=%s\n' "$(date -u --iso-8601=seconds)"
    uptime
    cat /proc/loadavg
    cat /proc/pressure/cpu
    free -h
  } > "$lab/$label.txt"
}

capture baseline

# Bounded one-process CPU workload.
timeout 5s bash -c 'while :; do :; done' &
load_pid=$!
sleep 2
capture active
wait "$load_pid" 2>/dev/null || true
sleep 2
capture recovery

diff -u "$lab/baseline.txt" "$lab/active.txt" || true
printf 'Review all snapshots in %s\n' "$lab"

8. Common analytical mistakes

9. Knowledge check

10. Summary

• Baselines must be workload-aware, comparable, and distribution-oriented.
• USE and RED organize evidence but do not replace system knowledge.
• Strong hypotheses name scope, interval, mechanism, and disconfirming evidence.
• Incident bundles require identity, timestamps, raw data, checksums, permissions, and privacy review.
• Reports should distinguish observations, inferences, tests, confidence, and caveats.

11. Further reading

• Linux kernel documentation — Pressure Stall Information
• sysstat feature overview
• sar(1)
• pidstat(1)
• strace(1)