curl Fundamentals for HTTP Automation

1. curl turns HTTP into a shell command

curl is one of the most useful tools for DevOps automation because it can make HTTP requests, send headers and bodies, follow redirects, enforce timeouts, and expose transport failures through exit status.

flowchart LR
  B["Bash"] --> C["curl"]
  C --> H["HTTP server"]
  H --> R["status + headers + body"]
  R --> C
  C --> B

2. Start with a simple GET request

curl https://api.example.com/health

By default, the response body is written to stdout. Production scripts usually add explicit failure, timeout, and output options.

3. Make HTTP errors visible to shell control flow

if curl --fail --silent --show-error "$url"; then
  printf 'request succeeded\n' >&2
else
  status=$?
  printf 'curl failed status=%d\n' "$status" >&2
  exit "$status"
fi

--fail makes selected HTTP error responses produce a non-zero exit status instead of being treated as transport success.

4. Bound connection and total request time

curl \
  --connect-timeout 5 \
  --max-time 30 \
  --fail \
  --silent \
  --show-error \
  "$url"

A connect timeout limits connection establishment; a total timeout bounds the entire transfer.

5. Send request headers explicitly

curl \
  --fail \
  --silent \
  --show-error \
  --header 'Accept: application/json' \
  --header "Authorization: Bearer $API_TOKEN" \
  "$url"

6. Use the HTTP method that matches the API contract

curl --request DELETE \
  --fail --silent --show-error \
  "$url"

7. Send JSON without accidental shell interpolation

payload=$(jq -n \
  --arg service "$service" \
  --arg environment "$environment" \
  '{service: $service, environment: $environment}')

curl \
  --request POST \
  --header 'Content-Type: application/json' \
  --data "$payload" \
  --fail --silent --show-error \
  "$url"

Building JSON with jq is safer than manually concatenating quotes and escaped characters in Bash.

8. Capture response body and HTTP status separately

tmp=$(mktemp) || exit 1

http_code=$(
  curl \
    --silent \
    --show-error \
    --output "$tmp" \
    --write-out '%{http_code}' \
    "$url"
)
curl_status=$?

if (( curl_status != 0 )); then
  printf 'transport failure=%d\n' "$curl_status" >&2
  rm -f -- "$tmp"
  exit "$curl_status"
fi

printf 'http_code=%s\n' "$http_code" >&2
cat "$tmp"
rm -f -- "$tmp"

This pattern separates transport status, HTTP status, and body content instead of flattening them into one string.

9. Redirect following should be intentional

curl --location --fail --silent --show-error "$url"

--location follows redirects. Review security and credential-forwarding implications when requests cross hosts or trust boundaries.

10. Download to a file with explicit failure behavior

tmp=$(mktemp) || exit 1

if curl \
  --fail \
  --silent \
  --show-error \
  --output "$tmp" \
  "$artifact_url"; then
  mv -- "$tmp" artifact.tar.gz
else
  status=$?
  rm -f -- "$tmp"
  exit "$status"
fi

Staging the download keeps a partial transfer from appearing under the final artifact name.

11. Hands-on lab: build an HTTP helper

mkdir -p "$HOME/devops-academy/bash/chapter13/lesson01"
cd "$HOME/devops-academy/bash/chapter13/lesson01"

cat > http-get.sh <<'EOF'
#!/usr/bin/env bash
set -u
set -o pipefail

url=${1:-}
[[ -n $url ]] || {
  printf 'usage: %s URL\n' "$0" >&2
  exit 64
}

command -v curl >/dev/null 2>&1 || {
  printf 'curl is required\n' >&2
  exit 69
}

tmp=$(mktemp) || exit 1
trap 'rm -f -- "$tmp"' EXIT

if http_code=$(
  curl \
    --connect-timeout 5 \
    --max-time 20 \
    --silent \
    --show-error \
    --output "$tmp" \
    --write-out '%{http_code}' \
    "$url"
); then
  printf 'http_code=%s\n' "$http_code" >&2
  cat "$tmp"
else
  status=$?
  printf 'transport failure=%d\n' "$status" >&2
  exit "$status"
fi
EOF

chmod u+x http-get.sh
./http-get.sh https://example.com >/dev/null

12. Knowledge check

13. Summary

Reliable curl automation uses explicit timeouts, transport-status checks, HTTP-status handling, safe body construction, protected credentials, and staged file writes. Treat HTTP as a protocol with several independent failure layers.

14. Further reading

• curl documentation — exit codes, timeouts, headers, redirects, output formatting.
• HTTP semantics — methods and status codes.
• jq manual — constructing JSON values.
• OWASP guidance — secrets in logs and command execution.