Writing and Hardening systemd Unit Files

1. A unit file is a lifecycle contract

A good service unit states what must exist before startup, which executable systemd launches, how readiness is determined, what constitutes failure, how shutdown occurs, which identity and directories are available, and how the unit joins larger boot transactions. It should not hide orchestration inside an opaque shell wrapper when systemd can model the behavior directly.

flowchart TB
  A["Unit dependencies and ordering"] --> B["Execution identity and environment"]
  B --> C["ExecStart and readiness protocol"]
  C --> D["Runtime supervision and resource limits"]
  D --> E["Stop, timeout, restart, failure result"]
  F["Sandbox and capability policy"] --> C
  G["Install relationships"] --> A

2. Choose Type= from the program's readiness behavior

Type=simple treats the started process as the service and considers it started immediately. Type=exec waits until the executable has been invoked successfully. Type=notify expects an explicit readiness message. Type=forking supports traditional daemons that fork and usually requires a reliable PID file. Type=oneshot runs finite work and may remain logically active with RemainAfterExit=yes.

[Unit]
Description=Example API service
Wants=network-online.target
After=network-online.target

[Service]
Type=exec
User=example-api
Group=example-api
WorkingDirectory=/srv/example-api
EnvironmentFile=-/etc/example-api/environment
ExecStart=/usr/local/bin/example-api --config /etc/example-api/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
TimeoutStartSec=45s
TimeoutStopSec=30s

[Install]
WantedBy=multi-user.target

systemd does not automatically invoke a shell for ExecStart=. Shell operators such as pipes, redirection, globbing, and && require an explicit shell, but a dedicated executable or script with clear quoting and error handling is usually safer.

3. Requirement and ordering are separate dimensions

Requires= and Wants= express pull-in relationships. After= and Before= express ordering only. A unit can be ordered after another without requiring it, and can require another without specifying which starts first. network.target means the networking stack is being managed; it does not guarantee a usable route or remote dependency. Even network-online.target depends on distribution-specific wait-online behavior and should not replace application retries.

# Inspect the dependency graph and ordering relationships
systemctl list-dependencies example.service
systemctl list-dependencies --reverse example.service
systemctl show example.service -p Wants -p Requires -p After -p Before

# View the merged definition and all drop-ins
systemctl cat example.service

# Create an administrator drop-in instead of editing vendor files
sudo systemctl edit example.service
sudo systemctl daemon-reload

4. Remove privileges the service does not need

Hardening options are defense in depth. They cannot repair an unsafe application, but they can reduce access after compromise. Start from the workload's actual requirements, test in a representative environment, and document every exception.

[Service]
User=example-api
Group=example-api
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/example-api /var/log/example-api
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
CapabilityBoundingSet=
AmbientCapabilities=
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
SystemCallArchitectures=native
UMask=0027
LimitNOFILE=65536
MemoryMax=1G
TasksMax=512

Some applications require writable executable memory, device access, privileged ports, kernel interfaces, home directories, or additional address families. Tighten iteratively. Use the journal to distinguish application failures from sandbox denials, and never disable a security control globally just to make one service start.

# Score the exposed surface of an installed unit
systemd-analyze security example.service

# Show effective security-related properties
systemctl show example.service \
  -p User -p Group -p NoNewPrivileges -p PrivateTmp \
  -p ProtectSystem -p ProtectHome -p CapabilityBoundingSet \
  -p MemoryMax -p TasksMax

5. Restart policies must not create failure storms

Restart=on-failure is a common default for long-running daemons. always also restarts after clean exits and can be wrong for finite jobs. Start-rate limits prevent rapid loops. Timeouts bound hung startup and shutdown. A service should respond to SIGTERM and complete cleanup before TimeoutStopSec= expires.

[Unit]
StartLimitIntervalSec=60s
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=10s
TimeoutStartSec=45s
TimeoutStopSec=30s
KillSignal=SIGTERM
SuccessExitStatus=0
RestartPreventExitStatus=64 78

A restart loop can amplify dependency outages, exhaust API quotas, rotate logs aggressively, or hide the first failure. Capture the initial result and logs before resetting the failed state.

6. Hands-on lab: author and verify a hardened unit draft

The lab creates a unit file in your home directory and asks systemd to verify it. It does not install or start a system service.

lab="$HOME/devops-academy/linux/chapter11/lesson03"
mkdir -p "$lab/state" "$lab/output"
cd "$lab"

cat > academy-report.service <<UNIT
[Unit]
Description=DevOps Academy report-unit draft
ConditionPathIsDirectory=$lab/state

[Service]
Type=oneshot
User=$USER
Group=$(id -gn)
WorkingDirectory=$lab
ExecStart=/usr/bin/find $lab/state -maxdepth 1 -type f -printf %f\\n
StandardOutput=append:$lab/output/report.txt
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=$lab/output
RestrictSUIDSGID=yes
LockPersonality=yes
UMask=0027

[Install]
WantedBy=multi-user.target
UNIT

touch state/alpha state/beta
systemd-analyze verify "$lab/academy-report.service"

# Inspect the draft and retain a checksum for review
sed -n '1,160p' academy-report.service
sha256sum academy-report.service > academy-report.service.sha256

# Optional: copy to a disposable VM's /etc/systemd/system only after review.
printf 'Unit verified; it has not been installed or started.\n' 

7. Common unit-authoring mistakes

8. Knowledge check

9. Summary

A systemd service file is an explicit lifecycle and security contract. Correct units model readiness, dependencies, identity, shutdown, failure, and enablement; avoid hidden shell orchestration; limit restart storms; and remove unnecessary privileges through tested sandbox and resource controls.

10. Further reading

• systemd.unit(5), systemd.service(5), and systemd.exec(5).
• systemd.resource-control(5), systemd.kill(5), and systemd-analyze(1).
• Application documentation for readiness, reload, shutdown, file access, capabilities, and health checks.