Security and Performance Incident Scenarios

1. Security incidents change the objective and authority model

Ordinary troubleshooting optimizes rapid restoration. A suspected compromise also requires containment, evidence integrity, legal and privacy handling, credential protection, and coordinated eradication. Do not investigate beyond authorization or contact an attacker-controlled system.

flowchart TD
  A["Alert or user symptom"] --> B{"Possible malicious activity?"}
  B -- Yes --> C["Activate security incident process"]
  C --> D["Contain while preserving evidence"]
  D --> E["Acquire and hash approved evidence"]
  E --> F["Eradicate, recover, rotate, and monitor"]
  B -- No --> G["Performance triage"]
  G --> H["Measure utilization, saturation, errors, demand"]
  H --> I["Test one bottleneck hypothesis"]
  I --> J["Mitigate and verify user metrics"]

If the host may be compromised, its commands, logs, binaries, and timestamps may be untrustworthy. Escalate to the security team and use approved acquisition tooling and trusted analysis systems.

2. Contain proportionally and preserve volatile state

Containment choices include removing a node from service, blocking a narrow indicator, disabling a compromised credential, isolating a network segment, or capturing an image before shutdown. Abrupt power-off may lose memory evidence; leaving a host online may permit continued harm.

3. Build a minimal security triage snapshot

Collect only what the incident process authorizes. Preserve timestamps, command versions, hashes, operator identity, and the destination of evidence. Avoid executing unknown binaries or opening suspicious files on an analyst workstation.

#!/usr/bin/env bash
set -Eeuo pipefail
umask 077
out=${1:-"$PWD/security-triage-$(hostname)-$(date -u +%Y%m%dT%H%M%SZ)"}
mkdir -p "$out"

record() { local n=$1; shift; { printf '$'; printf ' %q' "$@"; printf '
'; "$@"; } >"$out/$n.txt" 2>&1 || true; }
record clock date --iso-8601=ns
record uptime uptime
record users w
record logins last -Faiwx
record processes ps -eo user,pid,ppid,lstart,etime,stat,%cpu,%mem,cmd --forest
record sockets ss -anp
record units systemctl list-units --all --no-pager
record timers systemctl list-timers --all --no-pager
record failed systemctl --failed --no-pager
record journal journalctl -b --since '-2 hours' -o short-iso-precise --no-pager
record auth sh -c 'journalctl -b _COMM=sshd --since "-2 hours" --no-pager; true'
record kernel dmesg --ctime
record mounts findmnt --real
record capabilities sh -c 'getcap -r /usr /opt 2>/dev/null | head -n 500'
record suid sh -c 'find / -xdev -type f \( -perm -4000 -o -perm -2000 \) -printf "%m %u:%g %TY-%Tm-%TdT%TH:%TM:%TS %p
" 2>/dev/null'

find "$out" -type f -print0 | sort -z | xargs -0 sha256sum > "$out/SHA256SUMS"
printf 'Triage directory: %s
' "$out"

Process command lines and journal entries can contain secrets. Redact a working copy, preserve the original under restricted access, and document every transformation.

4. Inspect identity, privilege, and persistence mechanisms

Compromise commonly abuses existing administration paths: SSH keys, sudoers, systemd units and timers, cron, shell startup files, package hooks, containers, CI runners, cloud-init, or application plugins. Compare against a known baseline rather than assuming every unfamiliar item is malicious.

# Identity and privileged configuration.
getent passwd
getent group sudo 2>/dev/null || getent group wheel 2>/dev/null || true
sudo -l -U "$USER" 2>/dev/null || true
find /etc/sudoers.d -maxdepth 1 -type f -ls 2>/dev/null

# SSH authorization metadata without printing private keys.
find /root /home -xdev -path '*/.ssh/authorized_keys'   -printf '%m %u:%g %TY-%Tm-%TdT%TH:%TM:%TS %p
' 2>/dev/null

# systemd and scheduled persistence.
systemctl list-unit-files --state=enabled --no-pager
systemctl list-timers --all --no-pager
find /etc/systemd/system /usr/local/lib/systemd/system -type f -ls 2>/dev/null
crontab -l 2>/dev/null || true
find /etc/cron.d /etc/cron.daily /var/spool/cron -type f -ls 2>/dev/null

# Package verification is distribution-specific.
dpkg -V 2>/dev/null | head -n 100 || true
rpm -Va 2>/dev/null | head -n 100 || true

Package verification reports legitimate configuration changes too. Correlate with ownership, change records, hashes from trusted artifacts, and deployment history.

5. Audit and mandatory-policy logs explain denied or privileged actions

The audit subsystem can record authentication, policy decisions, system calls, and administrator-defined watches. SELinux AVC and AppArmor denial messages are security evidence, not obstacles to suppress automatically.

# Audit status and recent records where auditd is used.
auditctl -s 2>/dev/null || true
ausearch -ts recent -m USER_LOGIN,USER_AUTH,USER_ACCT,CRED_ACQ,CRED_DISP 2>/dev/null | tail -n 100 || true
ausearch -ts recent -m AVC,USER_AVC 2>/dev/null | tail -n 100 || true

# SELinux and AppArmor state.
sestatus 2>/dev/null || true
getenforce 2>/dev/null || true
aa-status 2>/dev/null || true
journalctl -k --since '-2 hours' --grep='DENIED|AVC|apparmor' --no-pager

# Kernel hardening and taint state.
cat /proc/sys/kernel/tainted
sysctl kernel.kptr_restrict kernel.dmesg_restrict kernel.yama.ptrace_scope 2>/dev/null || true

Do not create audit rules or enable verbose tracing during an overloaded incident without capacity assessment. Logging can consume CPU, storage, and I/O and can itself worsen the event.

6. Performance troubleshooting begins with user work and demand

Define the affected transaction, request rate, latency percentile, error rate, concurrency, and change in workload. A host with 90% CPU may be healthy at high throughput; a host with 20% CPU may be stalled on storage, locks, DNS, or a throttled cgroup.

Use RED for services—rate, errors, duration—and USE for resources—utilization, saturation, errors. Compare before/after and healthy/failing cohorts.

7. CPU incidents require utilization, run queue, throttling, and work attribution

High user time suggests computation; high system time may reflect kernel work, networking, storage, or syscall overhead; high steal time indicates hypervisor contention; a long run queue indicates runnable work waiting for CPU. A cgroup may throttle while the host remains mostly idle.

uptime
top -b -n 1 | head -n 30
mpstat -P ALL 1 5 2>/dev/null || true
pidstat -u -w 1 5 2>/dev/null || true
vmstat 1 5
ps -eo pid,ppid,psr,stat,ni,pri,%cpu,%mem,comm,args --sort=-%cpu | head -n 30

# Managed workload/cgroup evidence.
systemd-cgtop --iterations=3 --delay=1
systemctl show example.service -p ControlGroup -p CPUQuotaPerSecUSec
cg=$(systemctl show example.service -p ControlGroup --value)
[[ -n $cg && -r /sys/fs/cgroup$cg/cpu.stat ]] && cat "/sys/fs/cgroup$cg/cpu.stat"

# Sampling/profiling requires authorization and can expose sensitive symbols.
# sudo perf top
# sudo perf record -F 99 -p PID -- sleep 30

Do not equate load average with CPU percentage. Linux load includes runnable and certain uninterruptible tasks; storage stalls can raise load with low CPU utilization.

8. Memory incidents combine working set, reclaim, swap, OOM, and pressure

“Free memory is low” is not sufficient evidence because Linux uses memory for cache. Investigate available memory, reclaim behavior, swap activity, major faults, working sets, cgroup events, and OOM records.

free -h
cat /proc/meminfo
vmstat 1 10
pidstat -r 1 5 2>/dev/null || true
ps -eo pid,user,rss,vsz,%mem,stat,comm,args --sort=-rss | head -n 30
cat /proc/pressure/memory
journalctl -k -b --grep='oom|Out of memory|Killed process' --no-pager

# Per-service cgroup limits and events.
systemctl show example.service -p MemoryCurrent -p MemoryHigh -p MemoryMax -p MemorySwapMax
cg=$(systemctl show example.service -p ControlGroup --value)
for f in memory.current memory.high memory.max memory.events memory.pressure; do
  [[ -r /sys/fs/cgroup$cg/$f ]] && { echo "== $f"; cat "/sys/fs/cgroup$cg/$f"; }
done

An OOM kill is the terminal event, not necessarily the beginning. Correlate allocation growth, workload change, cache behavior, limits, swap, and reclaim pressure before increasing memory.

9. Storage performance needs latency, queueing, utilization, and errors

Filesystem space, device health, and I/O performance are different dimensions. High device utilization is not automatically bad for parallel storage; latency and workload class matter. Network-backed storage adds path and server dependencies.

iostat -xz 1 5 2>/dev/null || true
pidstat -d 1 5 2>/dev/null || true
vmstat 1 5
cat /proc/pressure/io
findmnt --real
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
journalctl -k -b --grep='I/O error|timeout|reset|nvme|ata|EXT4-fs|XFS|BTRFS' --no-pager

# Identify deleted-open files and high-I/O processes when tools exist.
lsof +L1 2>/dev/null | head -n 50 || true
iotop -b -n 3 2>/dev/null | head -n 60 || true

Do not benchmark production storage with destructive writes or unbounded cache-bypassing workloads. Reproduce on an isolated target with realistic block size, queue depth, sync semantics, and data safety.

10. Pressure Stall Information measures lost execution opportunity

PSI reports how much time tasks were stalled because CPU, memory, or I/O resources were unavailable. some means at least one task stalled; full means all non-idle tasks in the scope stalled simultaneously for memory or I/O.

for resource in cpu memory io; do
  echo "== $resource pressure =="
  cat "/proc/pressure/$resource"
done

# cgroup v2 exposes workload-scoped pressure when supported.
cg=$(systemctl show example.service -p ControlGroup --value)
for resource in cpu memory io; do
  f="/sys/fs/cgroup$cg/$resource.pressure"
  [[ -r $f ]] && { echo "== $f =="; cat "$f"; }
done

11. Security and performance controls interact

Cryptography, audit, packet inspection, endpoint agents, integrity scanning, sandboxing, and verbose logging consume resources. Disabling them during an incident can hide compromise or create exposure. Measure overhead and tune scope, buffers, rate limits, and scheduling with the security owner.

12. Hands-on lab: measure and contain a controlled CPU workload

The lab runs a short CPU loop inside a user systemd scope with a 25% CPU quota, records host and cgroup evidence, and guarantees cleanup.

set -Eeuo pipefail
lab="$HOME/devops-academy/linux/chapter20/lesson04/lab"
rm -rf "$lab"
mkdir -p "$lab"

systemd-run --user --unit=academy-cpu-lab --collect   -p CPUQuota=25% -p RuntimeMaxSec=20s   bash -c 'end=$((SECONDS+15)); while (( SECONDS < end )); do :; done'   >"$lab/run.txt"

sleep 2
systemctl --user status academy-cpu-lab.service --no-pager >"$lab/status.txt" 2>&1 || true
systemctl --user show academy-cpu-lab.service   -p ControlGroup -p CPUQuotaPerSecUSec -p CPUUsageNSec -p Result   >"$lab/show.txt"

cg=$(systemctl --user show academy-cpu-lab.service -p ControlGroup --value)
if [[ -n $cg && -r /sys/fs/cgroup$cg/cpu.stat ]]; then
  cp "/sys/fs/cgroup$cg/cpu.stat" "$lab/cpu.stat"
fi
cat /proc/pressure/cpu > "$lab/cpu-pressure.txt"
wait_count=0
while systemctl --user is-active --quiet academy-cpu-lab.service; do
  sleep 1; ((wait_count+=1)); ((wait_count < 30)) || break
done
journalctl --user -u academy-cpu-lab.service --no-pager >"$lab/journal.txt"
systemctl --user reset-failed academy-cpu-lab.service 2>/dev/null || true
find "$lab" -type f -maxdepth 1 -print -exec sed -n '1,40p' {} \;

13. Security and performance incident runbook

1. Decide whether malicious activity is plausible and activate the security incident process when required.
2. Preserve volatile and external evidence before rebooting, killing processes, rotating logs, or modifying the host.
3. Contain proportionally: drain, isolate, revoke, or block the narrowest confirmed scope.
4. For performance, define user work, demand, errors, latency, concurrency, and healthy baseline.
5. Measure utilization, saturation, errors, queueing, and pressure at host and cgroup scopes.
6. Test one bottleneck hypothesis with bounded profiling or load.
7. Recover from trusted artifacts, rotate exposed credentials, restore controls, and monitor recurrence indicators.
8. Document evidence custody, false positives, contributing factors, and prevention owners.

14. Knowledge check

15. Summary

• Suspected compromise requires containment, evidence handling, and trusted recovery—not only service restoration.
• Identity, persistence, sockets, packages, audit, and mandatory-policy evidence must be compared with a known baseline.
• RED describes service behavior; USE describes resource utilization, saturation, and errors.
• CPU, memory, storage, and cgroup pressure require interval measurements and workload context.
• Bounded experiments, least privilege, automatic cleanup, and user-level verification control diagnostic risk.

16. Further reading

• NIST SP 800-61 Revision 3 — Incident Response
• Linux kernel Pressure Stall Information
• Linux perf events and tool security
• Linux Magic SysRq documentation
• systemd-cgtop manual