Signals, kill, pkill, and Graceful Shutdown

1. Signals are asynchronous lifecycle notifications

A signal is a small notification delivered to a process or thread. It does not carry a large payload; it tells the target that an event occurred. Each signal has a default action, and many signals can be caught by a handler, ignored, or temporarily blocked. Signal names are more readable than numbers and avoid some cross-platform differences.

SIGKILL and SIGSTOP cannot be caught, blocked, or ignored. They are kernel-enforced controls. That makes them powerful, but SIGKILL denies the target any opportunity to clean up application state.

2. Signal delivery has a target and a policy

flowchart TD
  V["Verify PID, start time, owner, and service"] --> T["Send SIGTERM"]
  T --> W["Wait for bounded grace period"]
  W -->|Exited| C["Confirm cleanup and service state"]
  W -->|Still running| D["Collect diagnostics"]
  D --> E{"Approved escalation?"}
  E -->|No| O["Continue investigation"]
  E -->|Yes| K["Send SIGKILL"]
  K --> C

Before signaling, verify that the target is the intended process instance. PIDs are reused. Record PID, parent, owner, start time, executable, arguments, service unit, and cgroup when relevant. A broad name match can affect unrelated processes.

pid=$$

# List available signals and show the numeric mapping on this host.
kill -l

# Verify a target before any action.
ps -o user,pid,ppid,lstart,etime,stat,comm,args -p "$pid"
readlink -f "/proc/$pid/exe"
cat "/proc/$pid/cgroup"

# Check whether a PID currently exists and is signalable by this user.
if kill -0 "$pid" 2>/dev/null; then
  printf 'PID %s exists and is accessible.\n' "$pid"
fi

kill -0 sends no signal; it performs existence and permission checks. It still does not prove that a reused PID identifies the same process you observed earlier.

3. kill, pgrep, and pkill

# Preview exact-name matches owned by the current user.
pgrep -a -u "$(id -u)" -x sleep || true

# Start two controlled targets.
sleep 300 & p1=$!
sleep 300 & p2=$!

# Verify explicit PIDs, then request termination.
ps -o pid,ppid,user,lstart,comm,args -p "$p1" -p "$p2"
kill -TERM "$p1" "$p2"
wait "$p1" 2>/dev/null || true
wait "$p2" 2>/dev/null || true

4. The application defines graceful behavior

The kernel delivers the signal, but the application decides what a caught signal means. A well-designed service handles termination by stopping new work, marking itself unready, completing or canceling bounded in-flight work, flushing buffers, closing listeners and files, releasing locks, writing final status, and exiting before the grace deadline.

Orchestrators and service managers often send SIGTERM, wait a configured grace period, then use SIGKILL. Align application drain time with platform timeouts.

5. Handle termination in a shell worker

Shell traps run code when selected signals or shell events occur. Keep handlers simple, idempotent, and bounded. The example writes state only inside a dedicated lab directory and terminates its current child.

lab="$HOME/devops-academy/linux/chapter08/lesson04"
mkdir -p "$lab"
worker="$lab/worker.sh"

cat > "$worker" <<'SCRIPT'
#!/usr/bin/env bash
set -u
state_dir=${1:?state directory required}
mkdir -p "$state_dir"
child=''

cleanup() {
  trap - TERM INT
  printf '%s shutdown requested\n' "$(date --iso-8601=seconds)" >> "$state_dir/events.log"
  if [[ -n $child ]] && kill -0 "$child" 2>/dev/null; then
    kill -TERM "$child" 2>/dev/null || true
    wait "$child" 2>/dev/null || true
  fi
  rm -f "$state_dir/worker.pid"
  printf '%s cleanup complete\n' "$(date --iso-8601=seconds)" >> "$state_dir/events.log"
  exit 0
}

trap cleanup TERM INT
printf '%s\n' "$$" > "$state_dir/worker.pid"
printf '%s worker started pid=%s\n' "$(date --iso-8601=seconds)" "$$" >> "$state_dir/events.log"

while :; do
  sleep 10 &
  child=$!
  wait "$child" 2>/dev/null || true
  child=''
done
SCRIPT
chmod 700 "$worker"

"$worker" "$lab/state" &
pid=$!
sleep 1
kill -TERM "$pid"
wait "$pid"
cat "$lab/state/events.log"
test ! -e "$lab/state/worker.pid"

Production shell services require deeper design around signal races, subprocess groups, traps while waiting, lock cleanup, and supervisor integration. Prefer purpose-built service code for complex concurrency.

6. Implement bounded escalation

Escalation should be explicit and observable. Send TERM once, wait for a defined interval, capture diagnostics if the target remains, then use KILL only under an approved policy.

terminate_with_timeout() {
  local pid=$1 timeout=${2:-10} started current

  ps -o user,pid,ppid,lstart,stat,comm,args -p "$pid" || return 1
  started=$(stat -c %Y "/proc/$pid" 2>/dev/null) || return 1
  kill -TERM "$pid"

  for (( current=0; current<timeout; current++ )); do
    if ! kill -0 "$pid" 2>/dev/null; then
      return 0
    fi
    # Abort if the /proc instance appears to have changed.
    [[ $(stat -c %Y "/proc/$pid" 2>/dev/null) == "$started" ]] || return 1
    sleep 1
  done

  printf 'PID %s exceeded %ss grace period; escalating.\n' "$pid" "$timeout" >&2
  kill -KILL "$pid"
}

# Test only against a process created by this shell.
sleep 300 &
pid=$!
terminate_with_timeout "$pid" 5
wait "$pid" 2>/dev/null || true

The directory timestamp check is only a lab safeguard, not a universal process-instance identifier. Production supervisors use stronger lifecycle ownership, pidfds, cgroups, or service-unit state.

7. Common mistakes

8. Knowledge check

9. Summary

Signals are precise lifecycle tools when the target and application semantics are understood. Verify identity, prefer service-manager operations, request graceful termination with SIGTERM, wait for a bounded interval, preserve diagnostics, and escalate to SIGKILL only when necessary. Robust services make shutdown an explicit, tested operating path.

10. Further reading

• signal(7), kill(1), kill(2), pgrep(1), and pkill(1).
• Bash Reference Manual sections on signals and traps.
• systemd service stop behavior and container-orchestrator termination-grace documentation.