Retries, Backoff, and Transient Failure Handling

1. Retry only failures that may actually be transient

The first question is not “how many retries?” It is “is this failure safe and meaningful to retry?”

2. Every retry loop needs an attempt bound

max_attempts=4

for ((attempt=1; attempt<=max_attempts; attempt++)); do
  if perform_operation; then
    break
  fi

  if (( attempt == max_attempts )); then
    printf 'operation failed after %d attempts\n' "$attempt" >&2
    exit 1
  fi
done

3. Backoff reduces pressure on a failing dependency

delay=1
max_attempts=5

for ((attempt=1; attempt<=max_attempts; attempt++)); do
  if probe_endpoint; then
    exit 0
  fi

  (( attempt == max_attempts )) && break
  sleep "$delay"
  (( delay *= 2 ))
done

exit 1

Exponential backoff gives a struggling service progressively more recovery time.

4. Jitter prevents synchronized retry storms

base_delay=$(( 2 ** (attempt - 1) ))
jitter=$(( RANDOM % 1000 ))
sleep_seconds=$(awk -v b="$base_delay" -v j="$jitter" \
  'BEGIN { printf "%.3f", b + (j / 1000) }')

sleep "$sleep_seconds"

When many clients fail at once, identical retry schedules can create a thundering herd. Random jitter spreads the retries over time.

5. Retry policy should inspect the failure class

if curl --fail --silent --show-error --max-time 20 "$url"; then
  return 0
else
  status=$?
  case $status in
    6|7|28) return 75 ;;  # example transient classes
    *)      return "$status" ;;
  esac
fi

6. HTTP status and transport status are different layers

A request can fail before HTTP exists, or it can receive an HTTP response that indicates temporary or permanent failure. Production retry logic often needs both transport-level and application-level classification.

7. Use an overall time budget, not only attempt count

deadline=$((SECONDS + 60))

while (( SECONDS < deadline )); do
  if perform_operation; then
    exit 0
  fi
  sleep 2
done

printf 'retry budget exhausted\n' >&2
exit 1

Attempt count alone can be misleading when individual attempts have variable duration.

8. Retry safety depends on side effects

Reads and convergent updates are usually easier to retry. Creates, payments, message sends, deployments, and mutations may require an idempotency key, transaction identifier, or reconciliation step before another attempt.

9. Respect server-provided retry guidance

Some APIs provide a retry-after value or rate-limit reset time. Prefer server guidance over a locally invented aggressive retry schedule when the protocol defines it.

10. Log attempts without flooding output

printf 'retry attempt=%d/%d delay=%ss reason=%s\n' \
  "$attempt" "$max_attempts" "$delay" "$reason" >&2

Operators need to know that a command is retrying, why, and when it will stop.

11. Hands-on lab: reusable retry helper

mkdir -p "$HOME/devops-academy/bash/chapter12/lesson02"
cd "$HOME/devops-academy/bash/chapter12/lesson02"

retry() {
  local max_attempts=$1
  shift

  local attempt delay status
  delay=1

  for ((attempt=1; attempt<=max_attempts; attempt++)); do
    if "$@"; then
      return 0
    else
      status=$?
    fi

    if (( attempt == max_attempts )); then
      printf 'failed after %d attempts status=%d\n' \
        "$attempt" "$status" >&2
      return "$status"
    fi

    printf 'attempt=%d failed status=%d retry_in=%ss\n' \
      "$attempt" "$status" "$delay" >&2

    sleep "$delay"
    (( delay *= 2 ))
  done
}

attempt_file=./attempt
printf '0\n' > "$attempt_file"

flaky_demo() {
  n=$(cat "$attempt_file")
  ((n += 1))
  printf '%d\n' "$n" > "$attempt_file"
  printf 'demo attempt=%d\n' "$n"

  (( n >= 3 ))
}

retry 5 flaky_demo

12. Knowledge check

13. Summary

Reliable retries are classified, bounded, delayed, and observable. Use backoff and jitter, distinguish permanent from transient failure, respect remote guidance, and never retry side effects until you know repetition is safe.

14. Further reading

• curl documentation — exit codes, retry, timeout behavior.
• HTTP semantics — retryable methods and status codes.
• AWS Architecture guidance — exponential backoff and jitter.
• Google SRE guidance — handling overload and retries.