Boot, Service, Storage, and Permission Incidents

1. Map the incident to the boot-to-service dependency chain

“The server does not start” can mean no firmware handoff, bootloader failure, kernel panic, missing root filesystem, failed initramfs, emergency target, dependency timeout, or one application unit failing after an otherwise successful boot. Identify the last confirmed stage before changing anything.

flowchart TD
  F["Firmware / virtual platform"] --> B["Bootloader and kernel command line"]
  B --> K["Kernel and hardware discovery"]
  K --> I["initramfs and root filesystem"]
  I --> S["systemd system manager"]
  S --> M["Local and remote mounts"]
  M --> N["Network and identity dependencies"]
  N --> A["Application unit and readiness"]

Use console or out-of-band access for boot incidents. Confirm snapshot and backup state before filesystem repair, bootloader installation, initramfs regeneration, or partition changes.

2. Read the current and previous boot as separate datasets

Persistent journal storage makes the previous failed boot available after recovery. If persistence is disabled or the root filesystem never mounted, use console output, kernel ring buffer, bootloader logs, hypervisor console, serial console, or distribution rescue media.

journalctl --list-boots
journalctl -b -0 -p warning --no-pager
journalctl -b -1 -p warning --no-pager 2>/dev/null || true
journalctl -k -b -1 --no-pager 2>/dev/null || true

systemd-analyze time
systemd-analyze critical-chain
systemd-analyze blame | head -n 30
systemctl --failed --no-pager
systemctl get-default
cat /proc/cmdline

# Verify unit syntax without starting it.
systemd-analyze verify /etc/systemd/system/*.service 2>&1 | head -n 100

systemd-analyze blame measures activation time, not necessarily causation. A unit can appear slow because it waits on a dependency or device. Use critical-chain, unit jobs, and timestamped journal events together.

3. Rescue and emergency targets are controlled recovery environments

Rescue target normally provides basic local filesystems and a root shell with more services than emergency target. Emergency target is more minimal and may leave the root filesystem read-only. Distribution behavior and authentication requirements differ.

# From a running system, schedule a controlled transition only with approval.
# systemctl rescue
# systemctl emergency

# From a bootloader, temporary kernel command-line choices commonly include:
# systemd.unit=rescue.target
# systemd.unit=emergency.target

# In recovery, inspect before remounting or repairing.
findmnt /
mount | grep ' on / '
lsblk -o NAME,TYPE,SIZE,FSTYPE,UUID,FSAVAIL,FSUSE%,MOUNTPOINTS
cat /etc/fstab
systemctl --failed --no-pager
journalctl -b -p warning --no-pager

4. Diagnose the effective systemd unit, not one file

A unit may be composed from vendor configuration, administrator overrides, generated units, environment files, dependencies, credentials, and drop-ins. systemctl cat and systemctl show expose the effective state better than opening one path.

unit=example.service
systemctl status "$unit" --no-pager -l
systemctl cat "$unit"
systemctl show "$unit"   -p LoadState -p ActiveState -p SubState -p Result   -p FragmentPath -p DropInPaths -p UnitFileState   -p User -p Group -p WorkingDirectory   -p ExecStart -p Environment -p EnvironmentFiles   -p Requires -p Wants -p After -p ConditionResult   -p MainPID -p ExecMainCode -p ExecMainStatus
systemctl list-dependencies "$unit" --all
journalctl -b -u "$unit" -o short-iso-precise --no-pager

# After editing unit configuration.
systemd-analyze verify "$(systemctl show -p FragmentPath --value "$unit")"
systemctl daemon-reload
systemctl try-restart "$unit"

A daemon reload updates manager configuration but does not restart an existing process. Conversely, restarting a unit does not validate downstream user transactions. Treat load, activation, readiness, and service behavior as separate states.

5. Service incidents: classify the failure mechanism

Use the unit result, exit status, timing, and logs to distinguish configuration rejection, executable failure, dependency failure, timeout, watchdog, signal termination, resource exhaustion, or restart-rate limiting.

# Inspect the main process and its actual environment and limits.
pid=$(systemctl show -p MainPID --value example.service)
if [[ $pid =~ ^[1-9][0-9]*$ ]]; then
  ps -fp "$pid"
  tr '\0' '
' < "/proc/$pid/environ" | sed -E 's/(TOKEN|SECRET|PASSWORD)=.*/\1=REDACTED/'
  cat "/proc/$pid/limits"
  cat "/proc/$pid/cgroup"
  ls -l "/proc/$pid/fd" | head
fi

6. Trace storage from device to application path

Do not assume a pathname identifies a physical disk. It may traverse LVM, device mapper, RAID, encryption, network storage, bind mounts, overlays, or container namespaces. Build the mapping before repair.

lsblk -e7 -o NAME,PATH,TYPE,SIZE,FSTYPE,FSVER,LABEL,UUID,PARTUUID,FSAVAIL,FSUSE%,MOUNTPOINTS
findmnt --real --output TARGET,SOURCE,FSTYPE,OPTIONS,PROPAGATION
findmnt --verify --verbose
blkid
cat /etc/fstab

# Capacity and inode exhaustion are distinct.
df -hT
df -i
du -xhd1 /var 2>/dev/null | sort -h

# Kernel/storage errors and read-only transitions.
journalctl -k -b --grep='I/O error|read-only|EXT4-fs|XFS|BTRFS|nvme|ata' --no-pager
dmesg --ctime --level=err,warn | tail -n 100

Deleted-but-open files can consume space while remaining invisible to ordinary directory traversal. Use lsof +L1 when available, then determine whether the owning process can be safely restarted. Never truncate arbitrary open files without understanding the application.

7. Mount failures require source, target, type, options, and dependency checks

A failed mount may come from a wrong UUID, missing device, invalid option, unavailable network, absent credential, damaged filesystem, dependency cycle, or a timeout. The generated systemd mount unit name derives from the path.

# Validate fstab syntax and resolvability without rebooting.
findmnt --verify --verbose
systemd-analyze verify /etc/fstab 2>&1 || true

# Show the generated mount unit for /srv/data.
systemd-escape --path --suffix=mount /srv/data
systemctl status srv-data.mount --no-pager 2>/dev/null || true
journalctl -b -u srv-data.mount --no-pager 2>/dev/null || true

# Resolve the configured source and inspect permissions on the mountpoint.
findmnt /srv/data
namei -l /srv/data
stat -c '%A %a %U:%G %n' /srv /srv/data

For optional or slow devices, systemd automounting and bounded device timeouts can improve boot resilience, but they must match application semantics. Do not hide a required durable volume behind nofail without a readiness guard.

8. A read-only filesystem is a symptom that protects data

The kernel may remount a filesystem read-only after detecting errors. Treat this as a data-integrity event. Capture kernel messages and storage health, stop writers, verify backup state, and follow the filesystem-specific recovery process.

findmnt -no TARGET,SOURCE,FSTYPE,OPTIONS /
findmnt -no TARGET,SOURCE,FSTYPE,OPTIONS /var 2>/dev/null || true
journalctl -k -b -p warning --no-pager

# Device health commands are hardware and transport specific.
# smartctl -a /dev/sdX
# nvme smart-log /dev/nvme0

# Examples only: run offline, against the verified device, with backups.
# fsck -f /dev/mapper/vg-lv        # ext-family wrapper; inspect prompts/options
# xfs_repair -n /dev/mapper/vg-lv # no-modify assessment for XFS
# btrfs check --readonly DEVICE    # do not use repair casually

9. Resolve access in the order Linux evaluates it

Start with process identity and pathname traversal, then discretionary permissions, ACLs, mount flags, capabilities, and mandatory policy. Root in a container or user namespace may not be host root.

target=/srv/app/config.yaml
service=example.service

systemctl show "$service" -p User -p Group -p SupplementaryGroups -p DynamicUser
namei -l "$target"
stat -c '%A %a %U:%G %n' "$target"
getfacl -p "$target" 2>/dev/null || true
findmnt -T "$target" -o TARGET,SOURCE,FSTYPE,OPTIONS
getcap "$target" 2>/dev/null || true

# Test using the service identity instead of an administrator shell.
uid=$(systemctl show -p User --value "$service")
[[ -n $uid ]] && sudo -u "$uid" -- test -r "$target" && echo readable

# SELinux and AppArmor state where present.
getenforce 2>/dev/null || true
ls -Z "$target" 2>/dev/null || true
ausearch -m AVC,USER_AVC -ts recent 2>/dev/null | tail -n 50 || true
aa-status 2>/dev/null || true
journalctl -k --grep='apparmor="DENIED"' --no-pager 2>/dev/null | tail -n 50

Do not solve access failures with chmod 777, blanket ownership changes, disabling SELinux/AppArmor, or granting CAP_SYS_ADMIN. Those actions erase boundaries and may still not address the actual identity, label, mount, or namespace.

10. Treat mandatory-access denials as policy evidence

For SELinux, first verify ordinary ownership and mode, then inspect file context, expected policy type, booleans, and AVC records. A mislabeled file commonly results from copying data into a nonstandard path. Restore the intended label rather than generating an allow rule blindly.

getenforce
sestatus
ls -Zd /srv/app /srv/app/config.yaml
matchpathcon /srv/app/config.yaml 2>/dev/null || true
ausearch -m AVC,USER_AVC -ts recent 2>/dev/null | tail -n 100

# Restore the policy-defined label when that is the confirmed cause.
# restorecon -v /srv/app/config.yaml

# Inspect service-related booleans before changing one.
# getsebool -a | grep SERVICE

# Temporary permissive testing affects security and requires authorization.
# Prefer a scoped permissive domain where supported, preserve AVC evidence,
# and restore enforcing mode immediately after the bounded test.

audit2allow translates observed denials into candidate policy; it does not know whether the attempted access is legitimate. Review application design, labeling, booleans, and least privilege before creating custom policy.

11. Hands-on lab: diagnose path traversal and service identity

The lab creates a local directory tree with an intentionally missing execute bit on one parent directory. It demonstrates why a readable file can still be inaccessible.

set -Eeuo pipefail
lab="$HOME/devops-academy/linux/chapter20/lesson02/lab"
rm -rf "$lab"
mkdir -p "$lab/private/config"
printf 'mode=production
' > "$lab/private/config/app.conf"
chmod 0755 "$lab"
chmod 0640 "$lab/private"
chmod 0644 "$lab/private/config/app.conf"

printf 'Direct file mode:
'
stat -c '%A %a %U:%G %n' "$lab/private/config/app.conf"
printf '
Path components:
'
namei -l "$lab/private/config/app.conf"

printf '
Expected failed read:
'
cat "$lab/private/config/app.conf" || true

chmod 0750 "$lab/private"
printf '
After restoring directory traversal:
'
namei -l "$lab/private/config/app.conf"
cat "$lab/private/config/app.conf"

rm -rf "$lab"

12. Recovery runbook for boot, storage, and permission incidents

1. Identify the last confirmed boot or service stage and establish console access.
2. Capture the current kernel command line, boot ID, failed units, journal interval, device map, mounts, capacity, and policy denials.
3. Protect data: stop writers, drain traffic, snapshot where valid, and verify backups.
4. Validate configuration before activation; test one corrected unit, mount, or access path.
5. For filesystem repair, boot into an appropriate offline environment and verify the target device twice.
6. Restore enforcing security controls and remove emergency permissions immediately after the bounded test.
7. Reboot only when required, then verify previous-boot evidence, all required mounts, units, transactions, and monitoring.

13. Knowledge check

14. Summary

• Boot incidents must be localized to firmware, bootloader, kernel, initramfs, mount, manager, or service stages.
• Effective systemd configuration includes fragments, drop-ins, identity, environment, dependencies, and limits.
• Storage troubleshooting maps device layers and protects data before repair.
• Capacity, inode exhaustion, deleted-open files, and read-only transitions are different failure modes.
• Access troubleshooting proceeds through identity, path traversal, DAC, ACL, mount, capabilities, and mandatory policy.

15. Further reading

• systemd bootup manual
• systemctl manual
• systemd special targets
• findmnt manual
• Red Hat SELinux troubleshooting