A Systematic Troubleshooting Method

1. Troubleshooting is controlled uncertainty reduction

A production symptom is not yet a root cause. “The API is down,” “the server is slow,” and “DNS is broken” describe observations from a particular vantage point. A reliable operator converts those observations into a bounded incident statement, protects users and evidence, and then reduces uncertainty with reversible tests.

flowchart TD
  S["Detect and state the symptom"] --> T["Triage severity, scope, and safety"]
  T --> E["Preserve timeline and evidence"]
  E --> H["Form ranked hypotheses"]
  H --> X["Run one discriminating test"]
  X --> R{"Result supports hypothesis?"}
  R -- Yes --> M["Mitigate or repair with rollback"]
  R -- No --> H
  M --> V["Verify service and side effects"]
  V --> P["Record cause, prevention, and ownership"]

The method is deliberately slower than guessing for the first few minutes and much faster than guessing over the whole incident. Every command should answer a question, and every change should have an expected result and a rollback path.

2. Write an operational incident statement

Before exploring details, state the affected service, first known failure time, user impact, geography or host scope, recent changes, current mitigation, and evidence confidence. Separate confirmed facts from reports and assumptions.

Incident: checkout-api elevated 5xx responses
Started: 2026-08-05 02:14 UTC; first alert 02:16 UTC
Scope: 3 of 12 application nodes in eu-central; database healthy
Impact: about 18% of requests fail; no confirmed data loss
Recent change: package rollout completed 01:58 UTC
Current mitigation: affected nodes removed from load balancer
Confirmed evidence: service restart loop and missing shared library
Unknowns: why only three nodes diverged; whether rollback artifact is safe
Incident commander: NAME; operations lead: NAME; recorder: NAME

3. Stabilize first when continued operation increases harm

Stabilization limits blast radius while preserving the ability to diagnose. Examples include draining a bad node, stopping an unsafe deployment, switching to a known-good replica, rate limiting abusive traffic, remounting a damaged filesystem read-only, or isolating a suspected host. A mitigation is not proof of root cause.

Do not reboot automatically because it “usually works.” Rebooting may clear the symptom while destroying process state, transient logs, socket ownership, memory pressure, or a reproducible failure.

4. Establish time, change history, and a comparison baseline

Many Linux incidents are change-correlated: a package update, configuration rollout, certificate rotation, disk expansion, key change, firewall edit, kernel reboot, or dependency failure. Correlation does not prove causation, but it prioritizes tests.

date --iso-8601=seconds
uptime -s
who -b
uname -a
cat /etc/os-release

# Current and previous boot evidence.
journalctl --list-boots
journalctl -b -p warning --no-pager
journalctl -b -1 -p warning --no-pager 2>/dev/null || true

# Package and service changes vary by distribution.
grep -hE ' install | upgrade | remove ' /var/log/dpkg.log* 2>/dev/null | tail -n 50
rpm -qa --last 2>/dev/null | head -n 30
systemctl list-units --state=failed --no-pager

# Git-managed configuration should expose the exact revision.
git -C /etc status --short 2>/dev/null || true
git -C /etc log -n 5 --oneline 2>/dev/null || true

Compare one failing instance with one healthy peer using the same commands. Differences in artifact digest, kernel, environment, mount options, resolver state, time, certificates, unit overrides, or resource limits are often more informative than a long standalone dump.

5. Traverse layers without skipping dependencies

A service request crosses multiple layers: client behavior, name resolution, route and transport, listener, process, unit manager, filesystem, identity, dependency, resource budget, and application logic. Start near the observed failure, but verify the next lower layer before concluding that a higher layer is broken.

6. Rank hypotheses and choose discriminating tests

Write hypotheses in falsifiable form: “The service cannot read the key because the deployed file owner differs from the unit UID,” not “permissions issue.” Rank by plausibility, impact, cost of testing, and danger of acting.

A useful test predicts different results under competing hypotheses. For example, testing by IP and by hostname separates transport from DNS; running as the unit user separates application behavior from interactive-shell identity; comparing a healthy host separates general design from drift.

Hypothesis H1: the unit receives an outdated environment file.
Prediction: systemctl show reports an old value while the file on disk is new.
Test: systemctl show api.service -p Environment -p EnvironmentFiles
Result: OLD_ENDPOINT is present in manager state.
Conclusion: H1 supported; daemon-reload alone will not restart the process.
Change: restart one canary instance; verify requests and rollback readiness.

7. Collect a bounded Linux evidence bundle

Collection should be read-only by default, time-bounded, and explicit about privileged data. Avoid recursively copying secrets, full home directories, private keys, tokens, or unbounded journals. Redact before sharing outside the authorized incident group.

#!/usr/bin/env bash
set -Eeuo pipefail
out=${1:-"$PWD/linux-evidence-$(hostname)-$(date -u +%Y%m%dT%H%M%SZ)"}
mkdir -m 0700 -p "$out"

run() { local name=$1; shift; { printf '$'; printf ' %q' "$@"; printf '
'; "$@"; } >"$out/$name.txt" 2>&1 || true; }
run time date --iso-8601=seconds
run identity id
run release sh -c 'uname -a; cat /etc/os-release'
run uptime uptime
run failed-units systemctl list-units --state=failed --no-pager
run services systemctl --failed --no-pager
run processes ps -eo pid,ppid,user,stat,lstart,etime,%cpu,%mem,cmd --sort=-%cpu
run sockets ss -lntup
run addresses ip -brief address
run routes ip route show table all
run mounts findmnt -R /
run filesystems df -hT
run inodes df -i
run memory free -h
run vmstat vmstat 1 5
run pressure sh -c 'for f in /proc/pressure/*; do echo "== $f"; cat "$f"; done'
run journal journalctl -b --since '-30 min' -p info --no-pager
run kernel dmesg --ctime --level=emerg,alert,crit,err,warn

find "$out" -type f -print0 | sort -z | xargs -0 sha256sum > "$out/SHA256SUMS"
tar --numeric-owner -czf "$out.tar.gz" -C "$(dirname "$out")" "$(basename "$out")"
printf 'Evidence bundle: %s.tar.gz
' "$out"

8. Query logs by boot, unit, time, priority, and field

The journal is a structured event store. Narrow it deliberately rather than scrolling from the end. Preserve the command and time range that produced each finding.

# Current boot, one unit, and a bounded interval.
journalctl -b -u api.service --since '2026-08-05 02:10:00 UTC'   --until '2026-08-05 02:30:00 UTC' --no-pager

# Errors across the current boot, with precise timestamps.
journalctl -b -p err -o short-iso-precise --no-pager

# Structured fields and a process-specific query.
journalctl -o verbose -n 1
journalctl _SYSTEMD_UNIT=api.service _PID=1234 --no-pager

# Kernel messages and previous boot.
journalctl -k -b --no-pager
journalctl -b -1 -p warning --no-pager 2>/dev/null || true

# Verify journal files before relying on archived evidence.
journalctl --verify

No log entry is also evidence: the process may fail before logging, write elsewhere, lack permission, use a different namespace, or have its messages rate-limited. Confirm logging path and unit stdout/stderr configuration.

9. Make one bounded change with an explicit rollback

A production change should identify the hypothesis, exact target, expected observable effect, timeout, abort threshold, rollback, and owner. Prefer a canary or one failed instance. Avoid combining package updates, configuration edits, restarts, and firewall changes into one experiment.

Change ID: INC-2041-07
Hypothesis: api.service is loading an obsolete EnvironmentFile value.
Target: api-03 only
Action: systemctl restart api.service
Expected within 30 s: active state, listener on 127.0.0.1:8080, health 200
Abort: restart loop, error rate increase, or dependency timeout
Rollback: restore previous environment file and restart; keep node drained
Evidence: systemctl show/status, journal slice, curl result, artifact hash
Owner: operations lead; approved by incident commander

Configuration validation belongs before activation: systemd-analyze verify for unit files, sshd -t for SSH, nginx -t for NGINX, nft -c -f for nftables, and application-specific dry runs where available.

10. Verify the service, not merely the command

A command returning zero proves only that the command accepted the request. Recovery verification should cover process state, listener, dependency access, synthetic user transaction, error rate, latency, saturation, logs, and side effects. Continue observing long enough to cross the failure interval or workload pattern.

After recovery, confirm that temporary bypasses, elevated permissions, permissive security modes, debug logging, packet captures, drained nodes, and emergency accounts are removed or assigned an expiration owner.

11. Hands-on lab: diagnose a failing user service

This lab creates a safe per-user systemd service with an intentional executable-path failure. It does not require root and removes its files at the end.

set -Eeuo pipefail
unit_dir="$HOME/.config/systemd/user"
lab="$HOME/devops-academy/linux/chapter20/lesson01"
mkdir -p "$unit_dir" "$lab"

cat > "$unit_dir/academy-broken.service" <<'EOF'
[Unit]
Description=DevOps Academy troubleshooting lab

[Service]
Type=oneshot
ExecStart=%h/devops-academy/linux/chapter20/lesson01/missing-command
StandardOutput=journal
StandardError=journal
EOF

systemctl --user daemon-reload
systemctl --user start academy-broken.service || true
systemctl --user status academy-broken.service --no-pager || true
journalctl --user -u academy-broken.service -n 20 --no-pager
systemctl --user show academy-broken.service   -p Result -p ExecMainCode -p ExecMainStatus -p FragmentPath

cat > "$lab/missing-command" <<'EOF'
#!/usr/bin/env bash
printf 'academy service recovered at %s
' "$(date --iso-8601=seconds)"
EOF
chmod 0755 "$lab/missing-command"
systemctl --user start academy-broken.service
systemctl --user status academy-broken.service --no-pager
journalctl --user -u academy-broken.service -n 20 --no-pager

systemctl --user disable --now academy-broken.service 2>/dev/null || true
rm -f "$unit_dir/academy-broken.service"
systemctl --user daemon-reload
systemctl --user reset-failed

12. Common troubleshooting failure modes

13. Knowledge check

14. Summary

• State the incident precisely and stabilize unsafe conditions before deep diagnosis.
• Build a timestamped timeline and compare failing systems with healthy peers.
• Rank falsifiable hypotheses and choose safe tests with high information gain.
• Collect bounded evidence across units, processes, network, storage, identity, policy, and resources.
• Change one variable with rollback, then verify service behavior and prevention ownership.

15. Further reading

• systemd journalctl manual
• systemd-analyze manual
• systemd debugging guidance
• Linux kernel issue-reporting guidance
• Google SRE — Effective Troubleshooting