Auditing, auditd, and Security Event Review

1. Audit answers who attempted what, under which identity and context

The Linux Audit subsystem records security-relevant events generated by kernel audit hooks and userspace components. auditd receives records and normally writes them to disk. One logical event may contain several record types—such as syscall, path, credential, and mandatory-access-control data—connected by a common event identifier. Audit is not a full application transaction trace and not a replacement for service logs, authentication logs, EDR, or network telemetry.

flowchart TD
  E["Kernel or trusted userspace event"] --> K["Audit subsystem and rule evaluation"]
  K --> Q["Kernel audit backlog"]
  Q --> D["auditd"]
  D --> L["Local audit log"]
  D --> P["Approved plugin or forwarding path"]
  L --> S["ausearch and aureport"]
  S --> I["Investigation and evidence package"]

An investigator must distinguish login identity from effective identity. Audit records can retain the original login user ID even after privilege escalation, supporting attribution across sudo or su. Attribution still depends on trustworthy account practices, synchronized time, protected logs, and a rule set that captured the event.

# Service and kernel audit status.
systemctl status auditd --no-pager
sudo auditctl -s

# List active rules exactly as loaded in the kernel.
sudo auditctl -l

# Inspect daemon configuration and persistent rule sources.
sudo grep -Ev '^\s*(#|$)' /etc/audit/auditd.conf
sudo find /etc/audit/rules.d -maxdepth 1 -type f -name '*.rules' -print -exec sed -n '1,160p' {} \;

# Distribution helper may compile rules.d into audit.rules.
command -v augenrules >/dev/null && sudo augenrules --check || true

2. Rules should encode a security question

Control rules configure audit behavior. Filesystem rules observe operations involving selected paths. Syscall rules filter by architecture, syscall, identity, result, path fields, or other attributes. Every rule adds evaluation and record volume, so broad rules such as auditing every syscall by every process are usually operationally unsafe. Assign a key that states the control or investigation purpose.

cat > /tmp/50-devops-academy.rules <<'EOF'
# Record changes to identity and privilege policy files.
-w /etc/passwd -p wa -k identity_changes
-w /etc/group -p wa -k identity_changes
-w /etc/sudoers -p wa -k privilege_policy
-w /etc/sudoers.d/ -p wa -k privilege_policy

# Record unsuccessful open attempts by human login users.
-a always,exit -F arch=b64 -S openat -F exit=-EACCES -F auid>=1000 -F auid!=unset -k denied_file_access
-a always,exit -F arch=b64 -S openat -F exit=-EPERM  -F auid>=1000 -F auid!=unset -k denied_file_access
EOF

# Load into a disposable lab's active ruleset after review. This changes live audit rules.
sudo auditctl -R /tmp/50-devops-academy.rules
sudo auditctl -l | grep -E 'identity_changes|privilege_policy|denied_file_access' 

The example loads rules immediately for a lab; persistence requires a reviewed file under /etc/audit/rules.d/ and the distribution’s supported reload process. On 64-bit systems, architecture-specific syscall numbers matter. Some environments need corresponding arch=b32 rules for 32-bit programs. Directory watches and path rules have semantics and performance considerations that should be verified against the installed audit version.

3. Search by time, key, identity, and event interpretation

ausearch groups related audit records into events and can interpret numeric identifiers. aureport produces summaries by user, executable, file, authentication, anomaly, or other dimensions. Preserve raw records before relying on interpreted output because name resolution and local account state can change.

# Time-bounded keyed searches.
sudo ausearch -k identity_changes -ts today -i
sudo ausearch -k privilege_policy -ts recent -i
sudo ausearch -k denied_file_access -ts recent -i

# Search by original login user and executable.
sudo ausearch -ua 1000 -ts today -i
sudo ausearch -x sudo -ts today -i

# Summaries identify where to investigate; they are not conclusions.
sudo aureport --auth --summary -ts today
sudo aureport --failed --summary -ts today
sudo aureport --user --summary -ts today
sudo aureport --executable --summary -ts today

Interpret fields in context. uid and euid describe credentials at the event; auid tracks login identity; ses identifies an audit session; success and exit describe result; comm is a short command name; exe points to an executable; name and path records describe objects. A successful syscall is not necessarily authorized business activity.

4. Capacity and failure policy are security controls

Audit volume depends on event rate, records per event, average record size, retention time, and compression or forwarding behavior. A planning approximation is:

auditd.conf controls log location, rotation, maximum size, space thresholds, and actions when storage or I/O fails. A workstation, general server, and regulated high-assurance host may require different actions. Halting on audit failure can preserve assurance but also creates availability risk; silently discarding records preserves availability but weakens evidence. This is a governance decision.

# Inspect capacity, current log growth, and configured reactions.
sudo du -sh /var/log/audit 2>/dev/null || true
sudo find /var/log/audit -maxdepth 1 -type f -printf '%TY-%Tm-%Td %TH:%TM %s %p\n' 2>/dev/null | sort

grep -E '^(log_file|max_log_file|num_logs|max_log_file_action|space_left|space_left_action|admin_space_left|admin_space_left_action|disk_full_action|disk_error_action)\s*=' \
  /etc/audit/auditd.conf 2>/dev/null || true

# Kernel backlog state and lost-record counters.
sudo auditctl -s
journalctl -k --since '-24 hours' | grep -Ei 'audit.*(backlog|lost|rate limit)' || true

5. Security review correlates audit with other evidence

A disciplined review starts from a time window, asset, account, or alert. Preserve original audit logs, note host identity and clock state, then search for related sessions, processes, files, network activity, package changes, and MAC denials. Audit can show that a privileged file changed; package-manager history can show an approved update; configuration management can show intended state; a ticket can establish authorization.

case_dir="$HOME/devops-academy/linux/chapter17/lesson03/case-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -m 0700 -p "$case_dir"

{
  date -u --iso-8601=seconds
  hostnamectl 2>/dev/null || hostname
  uname -a
  timedatectl status 2>/dev/null || true
  sudo auditctl -s
} > "$case_dir/context.txt"

sudo ausearch -k privilege_policy -ts today --raw \
  > "$case_dir/privilege-policy.raw" 2> "$case_dir/ausearch-errors.txt" || true
sudo ausearch -k privilege_policy -ts today -i \
  > "$case_dir/privilege-policy.interpreted" 2>/dev/null || true

journalctl --since today -u sudo -u ssh -u sshd \
  > "$case_dir/related-journal.txt" 2>&1 || true

sha256sum "$case_dir"/* > "$case_dir/SHA256SUMS"
chmod 0600 "$case_dir"/*

Checksums detect later byte changes but do not prove who collected the evidence or whether the source was complete. Record collector identity, commands, privilege level, tool versions, transfer path, redactions, and every handoff. Move evidence to protected storage according to retention and privacy policy.

6. Tune rules from measurable signal, not convenience

High event volume can hide meaningful activity and exhaust storage. Start with explicit security requirements: identity database changes, privilege policy, time changes, kernel-module operations, audit configuration, authentication, sensitive secrets, or regulated objects. Test on representative workload. Measure event rate, latency overhead, backlog, lost records, and analyst usefulness. Exclude only after understanding what evidence is lost.

# Count records and keyed events in a bounded window.
sudo ausearch -ts recent --raw 2>/dev/null | wc -l
sudo ausearch -ts recent -k denied_file_access --raw 2>/dev/null | wc -l

# Summarize event types and executables for tuning.
sudo ausearch -ts recent --raw 2>/dev/null \
  | awk '/^type=/{print $1}' | sort | uniq -c | sort -nr | head
sudo aureport --executable --summary -ts recent 2>/dev/null | head -n 40

# Recheck health after a rule change.
sudo auditctl -s
sudo journalctl -u auditd --since '-1 hour' --no-pager

7. Hands-on lab: create and verify a keyed lab event

Run this on a disposable VM. The rule observes writes to one lab file, creates an event, searches it by key, and removes the transient rule. It does not modify persistent audit configuration.

lab="$HOME/devops-academy/linux/chapter17/lesson03"
mkdir -p "$lab"
touch "$lab/audit-target.txt"

sudo auditctl -w "$lab/audit-target.txt" -p wa -k devops_academy_lab
printf 'event at %s\n' "$(date -u --iso-8601=seconds)" >> "$lab/audit-target.txt"
sync
sleep 1

sudo ausearch -k devops_academy_lab -ts recent -i \
  | tee "$lab/audit-event.txt"

sudo auditctl -W "$lab/audit-target.txt" -k devops_academy_lab
! sudo auditctl -l | grep -q devops_academy_lab
sha256sum "$lab/audit-event.txt" > "$lab/SHA256SUMS"

8. Common audit mistakes

9. Knowledge check

10. Summary

• The kernel audit subsystem, backlog, daemon, storage, and review tools form one evidence pipeline.
• Rules must answer defined security questions and be tested for volume and overhead.
• ausearch correlates events; aureport helps identify review targets.
• Capacity, lost-record counters, rotation, and failure actions are part of audit assurance.
• Evidence handling requires raw preservation, timestamps, context, checksums, access control, and chain of custody.

11. Further reading

• auditd(8)
• auditctl(8)
• audit.rules(7)
• ausearch(8)
• aureport(8)