Journal Analysis with journalctl

1. A journal entry is a record with fields, not just a line

systemd-journald receives messages from service standard output and error, the native journal API, syslog-compatible sockets, the kernel log buffer, audit integration, and other sources. It adds trusted metadata such as boot ID, process identity, executable path, unit name, transport, timestamps, and cgroup context. journalctl

flowchart TD
  A["Service stdout \nand stderr"] --> J["systemd-journald"]
  B["Kernel messages"] --> J
  C["syslog and logger"] --> J
  D["Native journal fields"] --> J
  J --> R["Volatile or persistent journal files"]
  R --> Q["journalctl field, boot, unit, time queries"]
  Q --> E["Human view, JSON, export, follow"]

The visible MESSAGE= is only one field. Trusted fields beginning with an underscore are added by the journal and cannot normally be forged by an unprivileged sender. Application-defined fields can carry request IDs, environments, versions, and other dimensions when the logging library supports them.

2. Select the correct boot and unit before reading messages

# Enumerate boots known to the local journal
journalctl --list-boots

# Current and previous boot
journalctl -b --no-pager -n 100
journalctl -b -1 --no-pager -n 100

# One unit in the current boot
journalctl -u ssh.service -b --no-pager -n 100 2>/dev/null \
  || journalctl -u sshd.service -b --no-pager -n 100

# Kernel messages from the selected boot
journalctl -k -b --no-pager -n 120

A previous boot is queryable only if records survived reboot. On systems using volatile journal storage, -b -1 may have no data. Unit filters use journal metadata and are safer than grepping rendered text for a service name.

3. Bound the incident with time and severity

# Absolute and relative time windows
journalctl --since "2026-08-05 01:20:00" --until "2026-08-05 01:35:00"
journalctl --since "30 minutes ago" --until now

# Priority range: warning through alert
journalctl -p warning..alert -b --no-pager

# Combine unit, boot, time, and priority
journalctl -u example.service -b \
  --since "15 minutes ago" -p notice..alert --no-pager

Priority filters depend on applications assigning meaningful syslog priorities. A serious application error may still be logged as informational text, while a harmless transient warning may be high priority. Severity is one dimension, not a complete incident classifier.

4. Query fields instead of repeatedly grepping presentation text

# Inspect all fields for a few entries
journalctl -u systemd-logind.service -n 3 -o verbose

# Exact field matches
journalctl _COMM=sshd -b --no-pager 2>/dev/null || true
journalctl SYSLOG_IDENTIFIER=sudo --since today --no-pager 2>/dev/null || true
journalctl _UID="$(id -u)" --since "1 hour ago" --no-pager

# OR within the same field; AND across different fields
journalctl _SYSTEMD_UNIT=ssh.service _SYSTEMD_UNIT=sshd.service -b --no-pager

# Discover values present for a field
journalctl -F _SYSTEMD_UNIT | sort | head -n 60

Multiple matches for the same field are alternatives; matches for different fields are combined. For automation, use stable fields and document the exact command so another engineer can reproduce the result.

5. Choose an output format for the consumer

# Follow a unit during a controlled test
journalctl -fu example.service

# JSON for processing with jq
journalctl -u example.service --since "10 minutes ago" -o json \
  | jq -r '[.__REALTIME_TIMESTAMP, .PRIORITY, .MESSAGE] | @tsv'

# Disk usage and internal verification
journalctl --disk-usage
journalctl --verify

6. Hands-on lab: emit and query structured test messages

The lab uses logger and systemd-cat to create uniquely tagged messages, then captures reproducible queries. Journal visibility depends on local policy.

lab="$HOME/devops-academy/linux/chapter11/lesson04"
mkdir -p "$lab"
tag="devops-academy-$USER-$$"
start="$(date --iso-8601=seconds)"

logger -t "$tag" -p user.notice "phase=start lesson=journal-analysis"
printf 'phase=stdout status=ok\nphase=stderr status=warning\n' \
  | systemd-cat -t "$tag" -p info
logger -t "$tag" -p user.warning "phase=finish result=controlled-warning"

sleep 1
journalctl -t "$tag" --since "$start" --no-pager -o short-iso-precise \
  > "$lab/messages.txt"
journalctl SYSLOG_IDENTIFIER="$tag" --since "$start" --no-pager -o json \
  > "$lab/messages.jsonl"
journalctl SYSLOG_IDENTIFIER="$tag" --since "$start" --no-pager -o verbose \
  > "$lab/messages.verbose.txt"

wc -l "$lab"/*
head -n 5 "$lab/messages.txt"

7. Why expected records may be absent

8. Knowledge check

9. Summary

The journal is a structured evidence store. Effective analysis selects the correct boot and unit, bounds time, combines trusted fields, chooses an output format suited to humans or automation, accounts for access and retention limits, and preserves exact reproducible queries.

10. Further reading

• journalctl documentation.
• systemd.journal-fields(7), systemd-journald.service(8), and sd_journal_send(3).
• Application logging guidance for structured fields, correlation identifiers, and sensitive-data handling.