Composing Text Pipelines without Fragile Parsing

1. Define record boundaries before building the pipeline

2. Human-readable output is not automatically an API

Headers, alignment, abbreviations, color, locale, and version-specific formatting can all make display output brittle to parse.

3. Prefer one coherent transformation over needless conversions

# Several stages:
cut -d: -f1,2 services.txt |
grep ':prod$' |
cut -d: -f1

# One field-aware stage:
awk -F: '$2 == "prod" { print $1 }' services.txt

4. Make upstream failure visible

set -o pipefail

if generate_inventory |
   normalize_records |
   validate_records > inventory.txt; then
  printf 'pipeline succeeded\n'
else
  status=$?
  printf 'pipeline failed: %d\n' "$status" >&2
  exit "$status"
fi

5. Keep diagnostics out of the data stream

producer   2> >(tee producer.log >&2) |
transformer |
consumer

stdout can remain a clean record stream while stderr remains useful to humans and CI logs.

6. Control locale when deterministic ordering matters

LC_ALL=C sort input.txt

Locale affects collation and some character classes. Set it explicitly when bytewise ordering is part of the automation contract.

7. Use NUL for arbitrary filenames

find artifacts -type f -print0 |
while IFS= read -r -d '' file; do
  sha256sum -- "$file"
done

8. Keep structured data structured

curl --fail --silent --show-error "$url" |
jq -r '.services[] | select(.enabled) | [.name, .replicas] | @tsv' |
while IFS=$'\t' read -r name replicas; do
  printf 'service=%s replicas=%s\n' "$name" "$replicas"
done

9. Temporary artifacts are sometimes more reliable than streaming

Use a temporary file when you need to validate a complete intermediate result, preserve evidence after failure, feed multiple consumers, or retry only a later stage.

tmp=$(mktemp)

if generate_inventory >"$tmp" &&
   validate_inventory "$tmp"; then
  process_inventory "$tmp"
  status=$?
else
  status=$?
fi

rm -f -- "$tmp"
exit "$status"

10. Let Bash orchestrate specialist parsers

flowchart LR
  B["Bash orchestration"] --> P["jq / awk / parser"]
  P --> R["simple records"]
  R --> B
  B --> C["CLI actions"]

11. Hands-on lab: robust production inventory pipeline

mkdir -p "$HOME/devops-academy/bash/chapter08/lesson05"
cd "$HOME/devops-academy/bash/chapter08/lesson05"

cat > services.txt <<'EOF'
api:prod:3
worker:prod:2
cache:staging:1
metrics:prod:1
EOF

set -o pipefail

if report=$(
  awk -F: '$2 == "prod" { print $1 ":" $3 }' services.txt |
  LC_ALL=C sort
); then
  printf '%s\n' '--- production inventory ---'
  printf '%s\n' "$report"
else
  status=$?
  printf 'inventory pipeline failed: %d\n' "$status" >&2
  exit "$status"
fi

printf 'prod_service_count=%s\n' \
  "$(printf '%s\n' "$report" | wc -l)"

12. Knowledge check

13. Summary

Robust pipelines are built around explicit record contracts. Prefer machine-readable producer modes, keep stderr separate, use NUL for filenames, control locale when needed, expose pipeline failure, and let structured parsers own structured formats.

14. Further reading

• GNU Coreutils and GNU findutils manuals.
• GNU Bash Reference Manual — Pipelines, Redirections, and pipefail.
• jq Manual.
• POSIX utility specifications for grep, sed, awk, sort, and related tools.