Introduction to sed, awk, and Command Pipelines

1. A pipeline is a graph of concurrent processes

In A | B | C, the shell connects standard output from each command to standard input of the next. The processes normally run concurrently. Standard error is not piped unless explicitly redirected, and each process has its own exit status.

flowchart TD
  I["Input files or producer"] --> S["sed filters or rewrites records"]
  S --> A["awk selects fields and aggregates"]
  A --> O["sort, report file, or terminal"]
  S -. "exit status" .-> P["Bash PIPESTATUS and pipefail"]
  A -. "exit status" .-> P
  O -. "exit status" .-> P

2. sed applies an editing program to a stream

sed reads one record at a time into a pattern space, executes commands, and normally prints the result. The -n option suppresses automatic printing so that p can select exactly what appears.

log="$HOME/devops-academy/linux/chapter05/lesson05/deployments.tsv"

# Print records 2 through 5 only.
sed -n '2,5p' -- "$log"

# Print records containing a literal status token.
sed -n '/status=FAILED/p' -- "$log"

# Replace the first occurrence on each line; add g for all occurrences.
sed 's/environment=staging/environment=stage/' -- "$log"

# Remove blank lines and comment lines from a simple configuration stream.
sed -e '/^[[:space:]]*$/d' -e '/^[[:space:]]*#/d' -- app.conf

Without -n, combining automatic output with an explicit p command prints selected lines twice. Test transformations without -i first. In-place editing syntax and backup behavior differ among sed implementations, so production scripts should document their portability target.

Addresses and commands

3. awk is a pattern-action language for records and fields

An awk program consists of rules such as PATTERN { ACTION }. Input is split into records, normally lines, and fields, normally runs of whitespace. $0 is the complete record; $1, $2, and so on are fields; NF is the number of fields; and NR is the cumulative record number.

# Tab-separated input: print service and status fields.
awk -F '\t' '{ print $2, $3 }' -- "$log"

# Select records and format a clear result.
awk -F '\t' '$3 == "FAILED" { printf "%s service=%s duration=%ss\n", $1, $2, $4 }' -- "$log"

# Aggregate counts and averages per service.
awk -F '\t' '
  { count[$2]++; total[$2] += $4 }
  END {
    for (service in count) {
      printf "%s\tcount=%d\tavg_seconds=%.2f\n", service, count[service], total[service] / count[service]
    }
  }
' -- "$log" | LC_ALL=C sort

Use BEGIN for initialization before input and END for final reports. Pass shell values with -v name="$value" rather than constructing program text through unsafe string interpolation.

threshold=120
awk -F '\t' -v limit="$threshold" '
  $4 + 0 > limit { print $1, $2, $4 }
' -- "$log"

4. Make pipeline failure visible

By default, Bash reports a pipeline’s status as the status of its final command. An earlier stage can fail while the final consumer exits successfully. set -o pipefail makes a pipeline fail when any stage fails, using the rightmost non-zero status.

set -o pipefail

if sed -n '/status=/p' -- "$log" \
  | awk -F '\t' '$3 == "FAILED" { print $2 }' \
  | LC_ALL=C sort -u \
  > failed-services.txt; then
  printf 'Report written: %s\n' "$PWD/failed-services.txt"
else
  status=$?
  printf 'Pipeline failed with status %d\n' "$status" >&2
  exit "$status"
fi

Immediately after a pipeline, Bash’s PIPESTATUS array contains each stage’s status. Capture it before another command overwrites it.

sed -n '/status=/p' -- "$log" | awk -F '\t' '{ print $2 }' | sort -u
statuses=("${PIPESTATUS[@]}")
printf 'sed=%s awk=%s sort=%s\n' "${statuses[0]}" "${statuses[1]}" "${statuses[2]}"

5. Design pipelines as observable stages

6. Hands-on lab: build a deployment-health report

lab="$HOME/devops-academy/linux/chapter05/lesson05"
mkdir -p -- "$lab"
log="$lab/deployments.tsv"
report="$lab/health-report.txt"
tmp=$(mktemp "$lab/.health-report.XXXXXX")
trap 'rm -f -- "$tmp"' EXIT

cat > "$log" <<'EOF'
2026-08-04T12:00:00Z	orders-api	SUCCESS	82
2026-08-04T12:05:00Z	payments-api	FAILED	145
2026-08-04T12:10:00Z	orders-api	SUCCESS	75
2026-08-04T12:15:00Z	catalog-api	SUCCESS	61
2026-08-04T12:20:00Z	payments-api	FAILED	132
2026-08-04T12:25:00Z	catalog-api	SUCCESS	58
EOF

set -o pipefail
{
  printf '%s\n' 'Deployment health report'
  printf 'Generated: %s\n\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')"

  printf '%s\n' 'Failures by service:'
  awk -F '\t' '$3 == "FAILED" { failures[$2]++ } END { for (s in failures) print failures[s], s }' -- "$log" \
    | sort -nr

  printf '%s\n' '\nAverage duration by service:'
  awk -F '\t' '{ count[$2]++; total[$2]+=$4 } END { for (s in count) printf "%s %.1f seconds\n", s, total[s]/count[s] }' -- "$log" \
    | LC_ALL=C sort

  printf '%s\n' '\nSlow deployments over 120 seconds:'
  awk -F '\t' '$4 + 0 > 120 { printf "%s %s %s seconds\n", $1, $2, $4 }' -- "$log"
} > "$tmp"

[[ -s $tmp ]]
grep -q -F 'payments-api' -- "$tmp"
mv -- "$tmp" "$report"
trap - EXIT
cat -- "$report"

7. Know when to stop extending the one-liner

sed is excellent for line-oriented selection and substitution. awk is excellent for field-aware calculations and reports. Move to jq for JSON, a YAML processor for YAML, a CSV parser for quoted CSV, or Python/another language when you need nested structures, robust error objects, complex tests, reusable modules, or maintainable domain logic.

8. Common mistakes and stronger habits

9. Knowledge check

10. Chapter summary

Chapter 5 established a complete text-processing workflow: inspect bounded text, select content with grep, find filesystem objects safely, transform records with focused core utilities, and use sed and awk inside observable pipelines. The governing principles are explicit input boundaries, quoted patterns, preserved evidence, format-aware parsing, verified intermediate states, and visible failures.

11. Further reading

• GNU sed manual — addresses, commands, substitution, hold space, and portability options.
• GNU Awk User’s Guide — records, fields, patterns, actions, arrays, and numeric processing.
• GNU Bash Reference Manual — pipelines, PIPESTATUS, pipefail, redirection, and traps.
• GNU Coreutils and Grep manuals — composable filters and sorting behavior.
• POSIX specifications for sed, awk, shell pipelines, and text files.