Kernel Parameters, sysctl, Limits, and Hardening

1. Hardening reduces attack surface while preserving required behavior

A secure baseline is not a universal list of values copied into every server. Kernel version, distribution defaults, container role, network architecture, debugging requirements, performance profile, compliance target, and application behavior all matter. A hardening control should state the threat or misuse it addresses, the scope where it applies, the compatibility risk, the owner, and the test that proves the system still works.

flowchart TD
  R["Threat, requirement, and asset role"] --> B["Capture current effective state"]
  B --> D["Read kernel and distribution documentation"]
  D --> C["Stage one narrow control"]
  C --> T["Test security effect and workload compatibility"]
  T --> O{"Expected result?"}
  O -- no --> X["Rollback and record incompatibility"]
  O -- yes --> P["Persist with ownership and rationale"]
  P --> V["Monitor drift and regressions"]

Defense in depth combines patching, least privilege, MAC, service isolation, network policy, secure boot, kernel self-protection, logging, and recovery. A sysctl cannot compensate for an exposed unpatched service, and an aggressive restriction that breaks monitoring or recovery can increase operational risk.

2. sysctl exposes selected kernel parameters at runtime

The sysctl command reads and writes values represented under /proc/sys/. Dots in a key usually map to path separators. Runtime writes normally last until reboot. Persistent configuration is loaded from files such as /etc/sysctl.conf and directories including /etc/sysctl.d/, with exact precedence and boot timing determined by the distribution and systemd tooling. Use a clearly named local file instead of editing vendor files.

# Read effective values without changing them.
sysctl kernel.randomize_va_space
sysctl kernel.kptr_restrict
sysctl kernel.dmesg_restrict
sysctl fs.protected_hardlinks
sysctl fs.protected_symlinks
sysctl net.ipv4.conf.all.accept_redirects
sysctl net.ipv4.conf.default.accept_redirects

# Show the backing procfs paths.
cat /proc/sys/kernel/randomize_va_space
cat /proc/sys/fs/protected_hardlinks

# Inspect local and vendor configuration sources.
find /etc/sysctl.d /run/sysctl.d /usr/lib/sysctl.d \
  -maxdepth 1 -type f -name '*.conf' -print 2>/dev/null | sort

Do not assume that a key exists on every kernel or is writable in every namespace. Some values are global; some are per network namespace; some are fixed by kernel configuration or lockdown; some are changed later by network managers, container runtimes, or application startup. Record both the source file and final effective value.

3. Stage and validate a small sysctl policy

The following candidate illustrates controls often evaluated on general-purpose servers. It is not a universal recommendation. IP forwarding, redirects, source routing, unprivileged namespaces, core dumps, BPF, ptrace, and kernel log access all have legitimate use cases. Confirm platform documentation and application dependencies.

cat > /tmp/60-devops-academy-hardening.conf <<'EOF'
# Protect common temporary-directory link attacks.
fs.protected_hardlinks = 1
fs.protected_symlinks = 1

# Restrict kernel pointer and unprivileged kernel-log exposure.
kernel.kptr_restrict = 2
kernel.dmesg_restrict = 1

# Use address-space randomization when supported.
kernel.randomize_va_space = 2

# Example host policy: do not accept ICMP redirects.
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
EOF

# Validate syntax and supported keys in an isolated VM first.
sudo sysctl --load=/tmp/60-devops-academy-hardening.conf

# Re-read effective values after loading.
while read -r key _; do
  [[ -z "$key" || "$key" == \#* ]] && continue
  sysctl "$key"
done < /tmp/60-devops-academy-hardening.conf

sysctl --system reloads the entire configuration set and can change unrelated values, so it is a broad production action. Loading one staged file is easier to reason about, but rollback still requires saved previous values. Network settings may need both all and default semantics understood: existing and future interfaces can differ.

# Capture exact previous values for rollback.
keys=(
  fs.protected_hardlinks
  fs.protected_symlinks
  kernel.kptr_restrict
  kernel.dmesg_restrict
  kernel.randomize_va_space
  net.ipv4.conf.all.accept_redirects
  net.ipv4.conf.default.accept_redirects
)

rollback=/root/devops-academy-sysctl-rollback.conf
sudo sh -c ': > "$1"' sh "$rollback"
for key in "${keys[@]}"; do
  value=$(sysctl -n "$key")
  printf '%s = %s\n' "$key" "$value" | sudo tee -a "$rollback" >/dev/null
done

# Restore if validation fails.
# sudo sysctl --load="$rollback"

4. Resource limits have different scopes and enforcement paths

POSIX resource limits constrain properties such as open files, process count, core-file size, locked memory, and CPU time. A shell can show and set limits for itself and descendants with ulimit. PAM can apply limits.conf and limits.d to login sessions when the PAM limits module is in the relevant stack. systemd services use unit properties such as LimitNOFILE= and LimitNPROC=. cgroups govern aggregate resources for a unit or group and are often more appropriate than per-process limits.

# Shell and process effective limits.
ulimit -a
cat /proc/$$/limits

# Service limits and cgroup controls.
systemctl show nginx.service \
  -p LimitNOFILE -p LimitNPROC -p TasksMax \
  -p MemoryMax -p CPUQuotaPerSecUSec 2>/dev/null || true

# Login-limit policy sources.
grep -RHEv '^\s*(#|$)' /etc/security/limits.conf /etc/security/limits.d 2>/dev/null || true

# Compare a real service process with unit configuration.
pid=$(systemctl show -p MainPID --value nginx.service 2>/dev/null || true)
if [[ "$pid" =~ ^[1-9][0-9]*$ ]]; then
  cat "/proc/$pid/limits"
fi

Raising limits can increase resource-exhaustion impact; lowering them can cause intermittent failures under load. Document expected concurrency and file-descriptor use. A rough requirement for a connection-oriented service might include listening sockets, active connections, upstream connections, files, pipes, logs, and safety margin—not merely the maximum client count.

5. Capabilities and systemd sandboxing reduce service privilege

Linux capabilities divide traditional root privilege into narrower units, but many capabilities remain powerful. File capabilities, process permitted/effective/inheritable sets, ambient capabilities, and the bounding set interact. systemd can drop capabilities, prevent privilege gain, isolate temporary directories, make filesystem areas read-only or inaccessible, restrict address families, filter system calls, and create namespaces. These controls must match service behavior.

# Inspect process and file capabilities.
getcap -r /usr/bin /usr/sbin 2>/dev/null | head -n 40
capsh --print 2>/dev/null || true

pid=$(systemctl show -p MainPID --value nginx.service 2>/dev/null || true)
if [[ "$pid" =~ ^[1-9][0-9]*$ ]]; then
  grep -E '^(Cap(Inh|Prm|Eff|Bnd|Amb)|NoNewPrivs|Seccomp):' "/proc/$pid/status"
fi

# Analyze a unit's available sandbox controls.
systemd-analyze security nginx.service 2>/dev/null || true
systemctl cat nginx.service 2>/dev/null || true

A staged override might add NoNewPrivileges=yes, PrivateTmp=yes, ProtectSystem=strict, ProtectHome=true, a reduced CapabilityBoundingSet=, or RestrictAddressFamilies=. Do not paste a maximal template. Use systemd-analyze security as a review aid, not a compliance verdict, and test start, reload, logging, certificate renewal, upgrades, and failure recovery.

sudo install -d -m 0755 /etc/systemd/system/example.service.d
sudo tee /etc/systemd/system/example.service.d/60-hardening.conf >/dev/null <<'EOF'
[Service]
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
EOF

sudo systemctl daemon-reload
sudo systemctl restart example.service
sudo systemctl status example.service --no-pager
systemctl show example.service -p DropInPaths -p NoNewPrivileges -p PrivateTmp

# Roll back the drop-in if compatibility tests fail.
# sudo rm /etc/systemd/system/example.service.d/60-hardening.conf
# sudo systemctl daemon-reload
# sudo systemctl restart example.service

6. Compare with a benchmark, but preserve local rationale

Benchmarks such as CIS profiles, distribution security guides, organizational standards, and government baselines provide useful control catalogs. They cannot know every host role. Mark each control applicable, not applicable with reason, implemented, compensating control, or exception with expiration. Automate evidence collection separately from automatic remediation until compatibility and rollback are proven.

evidence="$HOME/devops-academy/linux/chapter17/lesson04/baseline"
mkdir -p "$evidence"

sysctl -a 2>/dev/null | sort > "$evidence/sysctl-effective.txt"
ulimit -a > "$evidence/shell-limits.txt"
systemctl show example.service > "$evidence/example-service-effective.txt" 2>&1 || true
systemd-analyze security example.service > "$evidence/example-service-security.txt" 2>&1 || true

{
  uname -a
  systemd --version | head -n 1
  sysctl --version 2>/dev/null || true
} > "$evidence/tool-context.txt"

sha256sum "$evidence"/* > "$evidence/SHA256SUMS"

7. Hands-on lab: evaluate one hardening control safely

Select one candidate key supported by your disposable VM. Capture the current value, threat rationale, workload test, temporary change, verification, and rollback. Do not apply the whole candidate file to production as a lab.

lab="$HOME/devops-academy/linux/chapter17/lesson04"
mkdir -p "$lab"
key='kernel.dmesg_restrict'
old=$(sysctl -n "$key")
printf '%s=%s\n' "$key" "$old" > "$lab/before.txt"

sudo sysctl -w "$key=1" | tee "$lab/change.txt"
sysctl "$key" | tee "$lab/after.txt"

# Compatibility/security checks: unprivileged read should be restricted.
if dmesg >/dev/null 2> "$lab/unprivileged-dmesg-error.txt"; then
  printf 'Unexpected: unprivileged dmesg succeeded\n' | tee "$lab/result.txt"
else
  printf 'Expected: unprivileged dmesg denied\n' | tee "$lab/result.txt"
fi

# Restore exact prior state.
sudo sysctl -w "$key=$old" | tee "$lab/rollback.txt"
test "$(sysctl -n "$key")" = "$old"
sha256sum "$lab"/* > "$lab/SHA256SUMS"

8. Common hardening mistakes

9. Knowledge check

10. Summary

• Hardening is role- and threat-specific configuration, not a universal value dump.
• Capture effective sysctl state and configuration precedence before changing anything.
• Shell, PAM, systemd, cgroup, and capability controls have different scopes.
• Stage service sandbox controls incrementally and verify real operation and recovery.
• Every control needs rationale, compatibility tests, persistence ownership, rollback, and drift monitoring.

11. Further reading

• Linux kernel — Documentation for /proc/sys
• Linux kernel — Kernel self-protection
• sysctl.d(5)
• limits.conf(5)
• systemd.exec(5)