Parallel Workers, Bounded Concurrency, and Ordering

1. Parallelism is a resource policy

Concurrent work can reduce wall-clock time, but it also increases CPU, memory, I/O, file-descriptor, network, and remote-service pressure. Worker count should be controlled.

flowchart LR
  Q["task queue"] --> W1["worker 1"]
  Q --> W2["worker 2"]
  Q --> W3["worker 3"]
  W1 --> R["results"]
  W2 --> R
  W3 --> R

2. xargs -P gives simple bounded parallelism

find artifacts -type f -print0 |
xargs -0 -n 1 -P 4 sha256sum

At most four checksum processes run simultaneously while pathname boundaries remain safe.

3. Completion order is not input order

printf '%s\n' 3 1 2 |
xargs -n 1 -P 3 bash -c '
  sleep "$1"
  printf "done=%s\n" "$1"
' _

If deterministic output order matters, capture per-task output and reorder it after workers finish.

4. Background jobs give explicit status and identity control

pids=()
names=()

run_task api &
pids+=("$!")
names+=("api")

run_task worker &
pids+=("$!")
names+=("worker")

failed=0

for i in "${!pids[@]}"; do
  if wait "${pids[$i]}"; then
    printf 'task=%s status=ok\n' "${names[$i]}"
  else
    status=$?
    printf 'task=%s status=failed code=%d\n' \
      "${names[$i]}" "$status" >&2
    ((failed += 1))
  fi
done

(( failed == 0 ))

Tracking PID and logical task name together makes failures diagnosable.

5. wait -n can power a rolling worker pool

On supported Bash versions, wait -n lets the parent start replacement work when any child completes. Provide a simpler fallback if older Bash versions are part of your support matrix.

6. The shared dependency often defines the real concurrency limit

Thirty-two CPU cores do not justify thirty-two concurrent API calls if the remote service allows five, or thirty-two writes if every worker contends for one disk.

7. Concurrency is not rate limiting

Four workers may still send hundreds of requests per second if each request is fast. Concurrency limits simultaneous in-flight work; rate limiting controls events per unit time.

8. Decide what one worker failure means

Fail-fast workflows may stop scheduling new tasks and terminate owned children. Best-effort workflows may allow all independent work to finish. The policy must be explicit.

9. Isolate worker output

mkdir -p results

run_task "$task_id" > "results/$task_id.out" 2> "results/$task_id.err"

Per-task files avoid interleaved output and make deterministic aggregation easier.

10. Hands-on lab: concurrent tasks with deterministic reporting

mkdir -p "$HOME/devops-academy/bash/chapter19/lesson04"
cd "$HOME/devops-academy/bash/chapter19/lesson04"

cat > parallel-demo.sh <<'EOF'
#!/usr/bin/env bash
set -u

run_task() {
  local id=$1
  local delay=$2
  sleep "$delay"
  printf 'task=%s delay=%s\n' "$id" "$delay"
}

pids=()
names=()

for spec in "A:2" "B:1" "C:3"; do
  id=${spec%%:*}
  delay=${spec##*:}

  run_task "$id" "$delay" > "$id.out" &
  pids+=("$!")
  names+=("$id")
done

failed=0

for i in "${!pids[@]}"; do
  if wait "${pids[$i]}"; then
    printf 'completed=%s\n' "${names[$i]}"
  else
    ((failed += 1))
  fi
done

printf '%s\n' '--- deterministic report order ---'
for id in A B C; do
  cat "$id.out"
done

(( failed == 0 ))
EOF

chmod u+x parallel-demo.sh
bash parallel-demo.sh

11. Knowledge check

12. Summary

Bound parallelism according to real resource limits, expect nondeterministic completion order, track per-task identity and status, separate rate limiting from concurrency, and define failure policy before launching workers.

13. Further reading

• GNU findutils documentation — xargs parallel execution.
• GNU Bash Reference Manual — background jobs and wait.
• Operating-system resource-limit documentation.
• Distributed-systems guidance on concurrency and rate limiting.