Log Rotation, Persistent Logs, and Boot Recovery

1. Retention is a capacity and evidence policy

Logs compete with applications for storage but may be essential for security, incident reconstruction, compliance, and debugging. A retention policy defines where records live, how much space they may consume, how long they remain, whether they leave the host, who can read them, and how integrity or privacy requirements are enforced.

flowchart TB
  A["Services and kernel emit records"] --> B["journald structured storage"]
  A --> C["Application text log files"]
  B --> D["Rotation and vacuum limits"]
  C --> E["logrotate policy and reopen signal"]
  D --> F["Local retention"]
  E --> F
  F --> G["Remote collection, archive, or deletion"]
  H["Boot failure"] --> I["Previous-boot logs and recovery target"]

2. journald can store records in volatile or persistent locations

Volatile journal files live under /run/log/journal and disappear at reboot. Persistent files live under /var/log/journal. With Storage=auto, the presence of the persistent directory normally determines behavior. Early-boot records may first be collected in runtime storage and later flushed to persistent storage.

# Effective configuration, including drop-ins
systemd-analyze cat-config systemd/journald.conf 2>/dev/null \
  || cat /etc/systemd/journald.conf

# Determine which journal directories exist
sudo du -sh /run/log/journal /var/log/journal 2>/dev/null || true
ls -ld /run/log/journal /var/log/journal 2>/dev/null || true

# Known boots and current consumption
journalctl --list-boots
journalctl --disk-usage
journalctl --verify

Size controls such as SystemMaxUse=, SystemKeepFree=, RuntimeMaxUse=, and their file-count counterparts bound storage. journald may stop growing when space is constrained, but it does not guarantee that the entire filesystem can never fill due to other writers.

# /etc/systemd/journald.conf.d/retention.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=1G
MaxRetentionSec=30day
Compress=yes

3. Rotate and vacuum deliberately

# Inspect first
journalctl --disk-usage

# Rotate active files so older data becomes archived
sudo journalctl --rotate

# Examples: remove archived files beyond a reviewed boundary
sudo journalctl --vacuum-time=30days
sudo journalctl --vacuum-size=2G
sudo journalctl --vacuum-files=20

# Re-check actual usage; active files remain
journalctl --disk-usage

Vacuum operations affect archived journal files, not the currently active files. A size vacuum can therefore leave usage above the requested value. Combine size, time, and file-count policy only after deciding which constraint has priority.

4. Text logs require a safe writer-rotation contract

logrotate renames, compresses, removes, and recreates text logs according to policy. The critical question is how the application handles the renamed file. The preferred design is to rename the current file, create a new file with correct ownership and mode, and signal the application to close and reopen its descriptor. copytruncate copies and truncates the live file, but can lose or duplicate records during the race and should be a documented fallback.

/var/log/example-api/*.log {
    daily
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    dateext
    create 0640 example-api adm
    sharedscripts
    postrotate
        /bin/systemctl kill -s HUP example-api.service >/dev/null 2>&1 || true
    endscript
}

# Debug policy without rotating
sudo logrotate --debug /etc/logrotate.conf

# Inspect service/timer scheduling on systemd distributions
systemctl status logrotate.timer --no-pager 2>/dev/null || true
systemctl list-timers logrotate.timer --all 2>/dev/null || true

# State records prevent unintended repeated rotation
sudo sed -n '1,80p' /var/lib/logrotate/status 2>/dev/null || true

Test ownership, SELinux/AppArmor labels, application reopen behavior, disk requirements for compression, and recovery when the postrotate action fails. Rotating an application log is not the same as rotating the systemd journal.

5. Boot recovery starts by identifying the last successful stage

# On a running system, gather recovery clues without changing boot
systemctl --failed --no-pager
journalctl -b -p warning..alert --no-pager
journalctl -b -1 -p warning..alert --no-pager 2>/dev/null || true
systemd-analyze critical-chain
findmnt --verify --verbose 2>&1 || true

# One-boot kernel command-line choices entered in the bootloader editor:
# systemd.unit=rescue.target
# systemd.unit=emergency.target
# systemd.log_level=debug systemd.log_target=console

Recovery authentication, SELinux relabeling, encrypted disks, initramfs shells, cloud consoles, and bootloader editing differ by distribution and platform. A hosted virtual machine may require provider serial-console access; a container usually has no independent firmware or bootloader at all.

6. Hands-on lab: audit retention and dry-run log rotation

The first half is read-only. The second creates and tests a user-owned logrotate configuration in the lab directory, without touching system logs.

lab="$HOME/devops-academy/linux/chapter11/lesson05"
mkdir -p "$lab/logs"
cd "$lab"

{
  printf '=== journal usage ===\n'
  journalctl --disk-usage 2>&1 || true
  printf '\n=== known boots ===\n'
  journalctl --list-boots 2>&1 || true
  printf '\n=== failed units ===\n'
  systemctl --failed --no-pager 2>&1 || true
  printf '\n=== fstab verification ===\n'
  findmnt --verify --verbose 2>&1 || true
} > recovery-audit.txt

for n in 1 2 3 4 5; do
  printf '%s sample event %s\n' "$(date --iso-8601=seconds)" "$n" >> logs/demo.log
done

cat > logrotate.conf <<EOF
$lab/logs/demo.log {
    size 1
    rotate 3
    missingok
    notifempty
    compress
    create 0640 $USER $(id -gn)
}
EOF

# Debug first, then force only this isolated user-owned policy.
logrotate --debug --state "$lab/logrotate.state" "$lab/logrotate.conf"
logrotate --force --state "$lab/logrotate.state" "$lab/logrotate.conf"
ls -lah logs
cat logrotate.state

7. Common retention and recovery mistakes

8. Knowledge check

9. Summary

Log retention is an evidence and capacity policy. journald persistence and vacuuming govern structured journal files; logrotate governs text logs and must coordinate safely with writers. Boot recovery identifies the failed stage, starts the smallest useful environment, preserves evidence, repairs only the confirmed fault, and verifies a normal boot afterward.

10. Further reading

• journald.conf documentation and journalctl documentation.
• logrotate.conf(5).
• systemd.special(7), distribution boot-recovery guides, filesystem repair manuals, and platform console documentation.