Cron, at, and systemd Timers

1. A scheduler starts work; it does not make the work reliable

Linux offers several scheduling surfaces. cron is compact and widely available for recurring calendar schedules. at queues a one-time command. A systemd timer activates a service unit and provides dependency management, structured logs, resource controls, missed-run behavior, and explicit state. The correct choice depends on the host, workload, failure model, and observability requirements—not on which syntax is shortest.

flowchart TD
  A["Work must run later"] --> R{"Recurring?"}
  R -- no --> O["Use at for a simple one-time job"]
  R -- yes --> C{"Needs dependencies, hardening, missed-run catch-up, or rich logs?"}
  C -- no --> CR["Use cron with explicit environment and logging"]
  C -- yes --> ST["Use a systemd service plus timer"]
  O --> V["Validate execution, output, and exit status"]
  CR --> V
  ST --> V

The command itself must still be safe to repeat, bounded in duration, explicit about inputs and outputs, and able to report failure. A scheduler can run a broken script perfectly on time.

2. Understand cron fields, ownership, and execution context

A common five-field crontab entry specifies minute, hour, day of month, month, and day of week. System crontabs such as /etc/crontab and files under /etc/cron.d/ add a user field. User crontabs created with crontab -e do not. Mixing these forms is a frequent source of silent failures.

# User crontab: minute hour day-of-month month day-of-week command
15 2 * * * /usr/local/sbin/nightly-report

# /etc/cron.d/nightly-report: includes the account field
15 2 * * * reportsvc /usr/local/sbin/nightly-report

Both day-of-month and day-of-week have implementation-specific matching semantics; on common cron implementations, a job can run when either restricted field matches. Verify behavior on the target distribution. Use crontab -l, inspect the installed cron daemon, and test schedules in a non-production lab.

3. Scheduled jobs run with a smaller, different environment

Cron does not start an interactive login shell. The working directory, PATH, locale, shell, credentials, agent sockets, and application variables may differ from your terminal. Use absolute paths, set a safe PATH, choose the shell explicitly when required, and change to a known working directory inside the script.

# Example user crontab header.
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=ops@example.invalid

# Redirect both streams to an append-only operational log.
15 2 * * * /usr/bin/flock -n /run/lock/nightly-report.lock \
  /usr/local/sbin/nightly-report >>/var/log/nightly-report.log 2>&1

Do not place secrets directly in a crontab, where they may be exposed through backups, support bundles, or process inspection. Use a credential mechanism appropriate to the service account and restrict file permissions. Remember that percent signs can have special meaning in traditional crontabs unless escaped.

4. Use at for controlled one-time execution

The at daemon queues commands for one future execution. It is useful for a maintenance action after a change window, a delayed cleanup, or a temporary verification. It is not a durable workflow engine: the host must remain available, the queue must be permitted, and the command still needs logging and validation.

# Queue a one-time command and preserve output.
printf '%s\n' \
  'date -u --iso-8601=seconds >> /var/log/post-change-check.log' \
  '/usr/local/sbin/post-change-check >> /var/log/post-change-check.log 2>&1' \
  | at now + 30 minutes

atq                 # list queued jobs
at -c JOB_ID        # inspect the exact queued script
atrm JOB_ID         # remove it before execution

Review /etc/at.allow and /etc/at.deny, plus the daemon status and logs. Record the job ID in the change ticket so another operator can audit or cancel it.

5. A systemd timer activates a service unit

Separate the work from the schedule. The service unit defines identity, command, dependencies, hardening, resource limits, and success semantics. The timer defines calendar or monotonic activation. This separation makes manual testing straightforward: run the service directly before enabling the timer.

# /etc/systemd/system/devops-backup.service
[Unit]
Description=Create the application backup set
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=backupsvc
Group=backupsvc
WorkingDirectory=/var/lib/devops-backup
ExecStart=/usr/local/sbin/devops-backup
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
PrivateTmp=true
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/devops-backup /var/log/devops-backup

# /etc/systemd/system/devops-backup.timer
[Unit]
Description=Run the application backup every night

[Timer]
OnCalendar=*-*-* 02:15:00
Persistent=true
RandomizedDelaySec=10m
AccuracySec=1m
Unit=devops-backup.service

[Install]
WantedBy=timers.target

sudo systemd-analyze verify \
  /etc/systemd/system/devops-backup.service \
  /etc/systemd/system/devops-backup.timer
sudo systemctl daemon-reload
sudo systemctl start devops-backup.service
sudo systemctl status devops-backup.service --no-pager
sudo systemctl enable --now devops-backup.timer
systemctl list-timers --all --no-pager | grep devops-backup

6. Calendar, monotonic, persistence, and jitter express different intent

OnCalendar= expresses wall-clock schedules. OnBootSec=, OnUnitActiveSec=, and related monotonic directives express elapsed time. Persistent=true causes a missed calendar activation to run after the timer becomes active again, which is useful for laptops and intermittently available servers but dangerous when a long outage would trigger stale work. RandomizedDelaySec= spreads load across a fleet.

# Validate expressions and inspect future activations.
systemd-analyze calendar 'Mon..Fri *-*-* 03:00:00'
systemd-analyze calendar --iterations=8 '*-*-01 04:30:00'

# Inspect timer and last service result.
systemctl show devops-backup.timer \
  -p NextElapseUSecRealtime -p LastTriggerUSec -p Result
journalctl -u devops-backup.timer -u devops-backup.service --since today

Daylight-saving transitions can skip or duplicate local wall-clock times. Use UTC where operationally appropriate, document the timezone, and test schedules around clock changes. Time synchronization is a dependency of calendar automation.

7. Design for overlap, retries, idempotency, and failure visibility

Use a lock when concurrent execution is unsafe, but do not let a lock hide chronic overrun. Emit duration and exit status, alert on skipped runs, and investigate why the previous instance is still active. Prefer an idempotent operation whose repeated execution converges to the same desired state.

#!/usr/bin/env bash
set -Eeuo pipefail
umask 077

lock=/run/lock/devops-inventory.lock
exec 9>"$lock"
if ! flock -n 9; then
  printf '%s status=skipped reason=overlap\n' "$(date -u --iso-8601=seconds)" >&2
  exit 75
fi

started=$(date +%s)
trap 'rc=$?; now=$(date +%s); printf "%s rc=%d duration_s=%d\n" \
  "$(date -u --iso-8601=seconds)" "$rc" "$((now-started))"' EXIT

tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
/usr/bin/find /srv/app -xdev -type f -printf '%P\n' | /usr/bin/sort >"$tmp"
/usr/bin/install -m 0600 "$tmp" /var/lib/devops-inventory/current.txt

Retries belong inside a bounded policy with backoff and a final nonzero result. A scheduler that retries forever can amplify an outage. For systemd services, consider RuntimeMaxSec=, TimeoutStartSec=, failure dependencies, and external alerting.

8. Troubleshoot from schedule to process to result

1. Confirm the schedule is installed for the intended account and host.
2. Validate the system clock, timezone, and next activation.
3. Verify the daemon or timer is enabled and active.
4. Inspect the exact command, environment, permissions, working directory, and interpreter.
5. Read scheduler and service logs around the expected time.
6. Run the command manually as the service identity with a minimal environment.
7. Confirm output, side effects, exit status, duration, and downstream dependencies.

# Simulate a sparse cron-like environment for diagnosis.
sudo -u backupsvc env -i \
  HOME=/var/lib/backupsvc \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  SHELL=/bin/bash \
  /usr/local/sbin/devops-backup

# Distribution log locations differ; query the journal first.
journalctl --since '2 hours ago' \
  -u cron.service -u crond.service -u devops-backup.service --no-pager

9. Hands-on lab: build an observable systemd timer

Create the following user-level service and timer so the lab requires no root privileges. It records a timestamp and a small system snapshot every two minutes.

mkdir -p "$HOME/.config/systemd/user" "$HOME/devops-academy/linux/chapter18/lesson01"

cat >"$HOME/.config/systemd/user/academy-snapshot.service" <<'EOF'
[Unit]
Description=DevOps Academy scheduling snapshot

[Service]
Type=oneshot
ExecStart=/bin/bash -lc 'printf "utc=%s load=%s\n" "$(date -u --iso-8601=seconds)" "$(cat /proc/loadavg)" >> "$HOME/devops-academy/linux/chapter18/lesson01/snapshots.log"'
NoNewPrivileges=true
PrivateTmp=true
EOF

cat >"$HOME/.config/systemd/user/academy-snapshot.timer" <<'EOF'
[Unit]
Description=Run the Academy snapshot every two minutes

[Timer]
OnBootSec=30s
OnUnitActiveSec=2m
AccuracySec=10s
Unit=academy-snapshot.service

[Install]
WantedBy=timers.target
EOF

systemd-analyze --user verify "$HOME/.config/systemd/user/academy-snapshot."{service,timer}
systemctl --user daemon-reload
systemctl --user start academy-snapshot.service
systemctl --user enable --now academy-snapshot.timer
systemctl --user list-timers --all | grep academy-snapshot
journalctl --user -u academy-snapshot.service --since today --no-pager

10. Common scheduling mistakes

11. Knowledge check

12. Summary

• Choose cron, at, or systemd timers according to recurrence, dependencies, observability, and missed-run semantics.
• Scheduled jobs require explicit environment, identity, paths, logging, and time assumptions.
• Systemd separates activation from the service that performs work.
• Locks, bounded runtime, idempotency, and alerts prevent silent schedule failure.
• Troubleshooting follows the chain from installed schedule to activation, process, result, and downstream effect.

13. Further reading

• crontab(5)
• at(1)
• systemd.timer
• systemd.time
• flock(1)