Vulnerability Patching, Integrity, and Secure Baselines

1. Vulnerability management is a continuous decision system

A scanner finding is an input, not a final risk decision. Package versions may include vendor backports without matching an upstream version string. A vulnerable component may be absent from the running path, protected by configuration, isolated from attackers, or actively exposed. Conversely, a low-scored issue can be urgent on an internet-facing identity system. Use distribution advisories and package metadata to determine whether the installed build contains the vendor fix.

flowchart TD
  A["Asset, package, service, and exposure inventory"] --> F["Vendor advisory or scanner finding"]
  F --> V["Validate installed build and applicability"]
  V --> R["Prioritize by exploitability, exposure, impact, controls"]
  R --> T["Test update, reboot, migration, and rollback"]
  T --> D["Deploy through controlled rings"]
  D --> C["Validate service, security, and observability"]
  C --> E["Preserve evidence and update baseline"]
  E --> M["Monitor drift and new intelligence"]

Ownership matters at every stage: platform teams may patch the OS, application teams own compatibility, security validates risk, and service owners accept downtime or exceptions. Emergency remediation should still record authorization, exact artifacts, validation, and follow-up debt.

2. Inventory must connect packages to running services and provenance

A host package list alone cannot show whether software is running, reachable, loaded into a process, installed outside the package manager, embedded in a container, or supplied by a language ecosystem. Begin with distribution packages, enabled repositories, kernel, booted kernel, services, listeners, containers, and local installations. Record architecture and repository origin.

inventory="$HOME/devops-academy/linux/chapter17/lesson05/inventory"
mkdir -p "$inventory"

uname -a > "$inventory/kernel-running.txt"
cat /etc/os-release > "$inventory/os-release.txt"
systemctl list-units --type=service --state=running --no-pager \
  > "$inventory/running-services.txt"
ss -lntup > "$inventory/listeners.txt"

if command -v dpkg-query >/dev/null; then
  dpkg-query -W -f='${binary:Package}\t${Version}\t${Architecture}\n' \
    > "$inventory/packages.tsv"
  apt-cache policy > "$inventory/apt-policy.txt"
elif command -v rpm >/dev/null; then
  rpm -qa --qf '%{NAME}\t%{EPOCHNUM}:%{VERSION}-%{RELEASE}\t%{ARCH}\n' \
    | sort > "$inventory/packages.tsv"
  dnf repolist --enabled > "$inventory/dnf-repositories.txt" 2>&1 || true
fi

find /usr/local /opt -xdev -maxdepth 3 -type f -perm /111 \
  -printf '%m\t%u\t%g\t%TY-%Tm-%TdT%TH:%TM:%TS\t%p\n' 2>/dev/null \
  > "$inventory/local-executables.tsv"
sha256sum "$inventory"/* > "$inventory/SHA256SUMS"

Software bills of materials can improve component visibility but still require association with an exact artifact, image digest, release, and runtime. For Linux operations, preserve package-manager state and repository trust. Avoid downloading arbitrary replacement binaries from search results when a supported vendor package or verified source exists.

3. Prioritize with context, not severity alone

Risk models can help make assumptions explicit. One simple conceptual model is:

Consider known exploitation, public exploit maturity, authentication requirements, network reachability, privileges gained, data sensitivity, blast radius, service criticality, recovery time, and whether the vulnerable feature is enabled. Keep the vendor advisory, scanner evidence, package query, service configuration, and decision together.

# Debian-family update visibility without installing changes.
sudo apt-get update
apt list --upgradable 2>/dev/null | tee /tmp/upgradable-packages.txt
apt-cache policy openssl libc6 linux-image-generic 2>/dev/null || true

# Red Hat-family advisory visibility without installing changes.
sudo dnf check-update || rc=$?
if [[ ${rc:-0} -ne 0 && ${rc:-0} -ne 100 ]]; then
  printf 'dnf check-update failed with %s\n' "$rc" >&2
fi
sudo dnf updateinfo list --available 2>/dev/null | tee /tmp/updateinfo.txt || true

# Identify the installed package that owns a relevant executable.
dpkg-query -S /usr/bin/ssh 2>/dev/null || rpm -qf /usr/bin/ssh 2>/dev/null || true

4. A patch deployment is a change with preconditions and rollback

Before updating, confirm backups and restore capability, repository trust, free space, package-manager health, maintenance window, service dependencies, load-balancer behavior, database compatibility, and console access. Test on a representative clone or canary. Separate metadata refresh from package installation so the exact candidate set is reviewed. Capture before and after package states.

change="$HOME/devops-academy/linux/chapter17/lesson05/change-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -m 0700 -p "$change"

{
  date -u --iso-8601=seconds
  hostname
  uname -a
  uptime
  df -h / /boot 2>/dev/null || true
} > "$change/preflight.txt"

if command -v apt-get >/dev/null; then
  dpkg-query -W -f='${binary:Package}\t${Version}\n' | sort \
    > "$change/packages-before.tsv"
  sudo apt-get update
  sudo apt-get --simulate upgrade > "$change/apt-simulation.txt"
  # Execute only after review and approval:
  # sudo apt-get upgrade
elif command -v dnf >/dev/null; then
  rpm -qa --qf '%{NAME}\t%{EPOCHNUM}:%{VERSION}-%{RELEASE}\n' | sort \
    > "$change/packages-before.tsv"
  sudo dnf --assumeno upgrade > "$change/dnf-simulation.txt" 2>&1 || true
  # Execute only after review and approval:
  # sudo dnf upgrade
fi

sha256sum "$change"/* > "$change/SHA256SUMS.before"

Rollback may mean package downgrade, filesystem or VM snapshot restoration, image redeployment, database restore, or failover to a previous node pool. Package downgrade alone may not reverse data migrations or configuration changes. A snapshot is not a backup unless its failure domain and restoration procedure are understood.

5. Determine what must restart

Updating files on disk does not replace code already mapped into running processes. Libraries may remain loaded until the service restarts. A new kernel is not active until the host boots it, unless a supported live-patching mechanism covers that specific change—and live patching does not eliminate every reboot requirement. Use distribution-supported tools and service-owner validation.

# Compare running and installed kernel packages.
uname -r
if command -v dpkg-query >/dev/null; then
  dpkg-query -W 'linux-image-*' 2>/dev/null | sort
  command -v needrestart >/dev/null && sudo needrestart -r l || true
  test -f /var/run/reboot-required && cat /var/run/reboot-required || true
elif command -v rpm >/dev/null; then
  rpm -q kernel 2>/dev/null | sort
  command -v needs-restarting >/dev/null && sudo needs-restarting -r || true
  command -v needs-restarting >/dev/null && sudo needs-restarting -s || true
fi

# Deleted mapped libraries often indicate processes needing restart.
sudo lsof +L1 2>/dev/null | grep -E '\.(so|so\.)' | head -n 40 || true

A reboot workflow should drain traffic, stop or fail over stateful work safely, reboot through an approved mechanism, verify boot loader and kernel, validate mounts and network, check service health, restore traffic gradually, and watch errors and latency. Test rollback from a failed boot—not only from a healthy package transaction.

6. Integrity monitoring detects change; it does not decide intent

File-integrity tools such as AIDE can baseline hashes, metadata, ownership, permissions, ACLs, extended attributes, and selected file properties. Package managers can verify packaged files against stored metadata. Linux Integrity Measurement Architecture can measure files into a kernel-maintained log and, with appropriate trust architecture, support attestation. Every approach requires protected baseline generation and a response process.

# Package-level verification examples.
sudo dpkg --verify 2>/dev/null | head -n 80 || true
sudo rpm -Va 2>/dev/null | head -n 80 || true

# AIDE lifecycle varies by distribution; inspect configuration first.
command -v aide >/dev/null && aide --version | head -n 2
sudo grep -Ev '^\s*(#|$)' /etc/aide/aide.conf 2>/dev/null | head -n 80 || true

# Typical controlled lifecycle after reviewing paths and rules:
# sudo aide --init
# sudo install -m 0600 /var/lib/aide/aide.db.new /var/lib/aide/aide.db
# sudo aide --check

# Kernel measurement log, when IMA is configured.
test -r /sys/kernel/security/ima/ascii_runtime_measurements \
  && head -n 20 /sys/kernel/security/ima/ascii_runtime_measurements || true

Create a baseline only on a known-good system using trusted installation media, repositories, images, and configuration. Store the baseline and signing keys outside the monitored host or otherwise protect them from the attacker being detected. Tune dynamic paths such as logs, caches, temporary files, databases, and container layers so expected churn does not bury high-value changes.

7. Secure baselines need drift and exception governance

A baseline can include packages, repositories, services, ports, users, sudo policy, firewall, MAC state, sysctl values, systemd hardening, audit rules, scheduled tasks, SSH configuration, boot integrity, and file-integrity rules. Record the applicable baseline version and host role. Detect drift continuously, classify it as authorized, unauthorized, emergency, or unknown, and link authorized change to a ticket or deployment.

baseline="$HOME/devops-academy/linux/chapter17/lesson05/secure-baseline"
mkdir -m 0700 -p "$baseline"

{
  date -u --iso-8601=seconds
  hostnamectl 2>/dev/null || hostname
  uname -a
} > "$baseline/identity.txt"

systemctl list-unit-files --state=enabled --no-pager > "$baseline/enabled-units.txt"
ss -lntup > "$baseline/listeners.txt"
sysctl -a 2>/dev/null | sort > "$baseline/sysctl.txt"
sudo nft list ruleset > "$baseline/nft-ruleset.txt" 2>&1 || true
sudo auditctl -l > "$baseline/audit-rules.txt" 2>&1 || true
getenforce > "$baseline/selinux-mode.txt" 2>&1 || true
sudo aa-status > "$baseline/apparmor-status.txt" 2>&1 || true

find "$baseline" -maxdepth 1 -type f -exec chmod 0600 {} +
sha256sum "$baseline"/* > "$baseline/SHA256SUMS"

Checksums within the same writable directory are not enough against a privileged attacker. Forward evidence to protected storage, sign manifests, use append-only or immutable retention where appropriate, and separate collection from approval. Baselines contain sensitive security architecture and require access control.

8. Hands-on lab: plan and verify a non-destructive patch cycle

This lab inventories the host, simulates available updates, checks reboot indicators, and writes a validation plan. It does not install packages. Complete it before designing an approved canary update.

lab="$HOME/devops-academy/linux/chapter17/lesson05/lab-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -m 0700 -p "$lab"

cp /etc/os-release "$lab/os-release.txt"
uname -a > "$lab/kernel.txt"
systemctl --failed --no-pager > "$lab/failed-units-before.txt"
ss -lntup > "$lab/listeners-before.txt"

if command -v apt-get >/dev/null; then
  sudo apt-get update
  sudo apt-get --simulate upgrade > "$lab/update-simulation.txt"
elif command -v dnf >/dev/null; then
  sudo dnf --assumeno upgrade > "$lab/update-simulation.txt" 2>&1 || true
fi

cat > "$lab/validation-plan.txt" <<'EOF'
1. Verify package transaction completed without unresolved errors.
2. Restart or reboot components identified by distribution-supported tools.
3. Confirm expected kernel, mounts, interfaces, routes, and time synchronization.
4. Confirm every required service is active and no unexpected service is enabled.
5. Run application health, authentication, data-integrity, and dependency checks.
6. Compare listeners, errors, latency, and resource pressure with baseline.
7. Preserve exact package versions, logs, approvals, and rollback result.
EOF

sha256sum "$lab"/* > "$lab/SHA256SUMS"
printf 'Review the simulated change in %s\n' "$lab"

9. Common vulnerability-management mistakes

10. Knowledge check

11. Summary

• Patch decisions require exact asset, package, vendor, runtime, exposure, and impact context.
• Controlled updates include preflight, simulation, canary, deployment, restart/reboot, validation, evidence, and rollback.
• Distribution-supported version and advisory semantics take precedence over naive banner matching.
• Integrity monitoring detects unexplained change only when the baseline and keys are trusted.
• Secure baselines need continuous drift review, exceptions, ownership, and protected evidence.

12. Further reading

• Debian Security Information
• Ubuntu Security Notices
• Red Hat Product Security
• AIDE documentation
• Linux kernel — IMA template management