Pipelines, pipefail, and Pipeline Exit Status

1. A pipeline connects stdout to the next stdin

For a pipeline such as producer | filter | consumer, Bash creates pipes and launches commands so each stage can stream data to the next.

flowchart TD
  A["producer"] -->|"stdout"| P1["pipe"]
  P1 -->|"stdin"| B["filter"]
  B -->|"stdout"| P2["pipe"]
  P2 -->|"stdin"| C["consumer"]
  A -. "stderr" .-> T["terminal / inherited stderr"]
  B -. "stderr" .-> T
  C -. "stderr" .-> T

stderr is not part of the pipeline by default. Each stage inherits the shell's stderr unless you redirect it.

2. Pipelines stream; they do not require whole intermediate files

Pipeline stages can run concurrently. A producer can write data while the next command is already consuming it. This enables low-latency processing and avoids many temporary files.

printf '%s\n'   'INFO api ready'   'WARN cache slow'   'INFO worker ready' |
  grep '^INFO' |
  awk '{print $2}' |
  sort -u

This composability depends on each command honoring a useful stream contract. Tools that write clean records to stdout and diagnostics to stderr integrate naturally.

3. By default, pipeline status usually follows the last command

Without pipefail, Bash uses the exit status of the final command as the status of a normal foreground pipeline (subject to negation with !). That can hide an upstream failure.

set +o pipefail

bash -c 'exit 7' | cat
printf 'pipeline status without pipefail = %d\n' "$?" 

cat successfully reaches end-of-file and exits zero, so the overall pipeline can appear successful even though the producer failed.

4. PIPESTATUS preserves each stage's most recent status

Bash stores an array named PIPESTATUS containing the exit statuses of commands in the most recently executed foreground pipeline.

set +o pipefail

bash -c 'printf "data\n"; exit 7' |
  grep data |
  cat

statuses=("${PIPESTATUS[@]}")
printf 'stage statuses: %s\n' "${statuses[*]}" 

Capture PIPESTATUS immediately. Running another command—even a diagnostic printf—updates shell status state and can replace the values you intended to inspect.

5. pipefail makes upstream failure visible

With set -o pipefail, a pipeline returns zero only if all commands succeed. If one or more commands fail, the pipeline status is the value from the rightmost failing command.

set -o pipefail

if bash -c 'printf "data\n"; exit 7' | grep data | cat; then
  printf 'pipeline succeeded\n'
else
  status=$?
  printf 'pipeline failed with status %d\n' "$status" >&2
fi

6. pipefail and set -e solve different problems

pipefail changes how pipeline status is calculated. set -e (errexit) influences when Bash exits after certain failing commands. They are related but not interchangeable, and set -e has context-dependent exceptions.

set -o pipefail

if producer | transformer | consumer; then
  printf 'all pipeline stages succeeded\n'
else
  printf 'pipeline failed\n' >&2
  exit 1
fi

Explicit control flow such as if remains valuable even in scripts that use strict-mode settings. It documents which failures are expected, recoverable, or fatal.

7. |& includes stderr in the pipe

Bash provides |& as shorthand that pipes both stdout and stderr from the command on the left into the next command. It is roughly equivalent to merging stderr into stdout before the pipe.

{
  printf 'normal record\n'
  printf 'warning record\n' >&2
} |& sed 's/^/[combined] /' 

Use this only when combining the streams is intentional. If stdout carries structured data, |& can make the stream impossible for downstream parsers to consume reliably.

8. Pipeline stages can run in subshell environments

A common surprise is assigning a variable inside a loop that receives pipeline input:

count=0

printf '%s\n' a b c |
while IFS= read -r line; do
  ((count += 1))
done

printf 'count after pipeline = %d\n' "$count" 

In typical Bash behavior, the loop runs in a subshell environment because it is a pipeline element, so changes do not persist in the parent shell. A redirection avoids that problem:

count=0

while IFS= read -r line; do
  ((count += 1))
done < <(printf '%s\n' a b c)

printf 'count after redirection = %d\n' "$count" 

Bash also has a lastpipe option with specific conditions, but relying on explicit data-flow structure is usually easier to reason about.

9. Early consumers can cause SIGPIPE upstream

Commands such as head may stop after receiving enough data. An upstream writer that continues writing can then receive SIGPIPE because no reader remains.

set -o pipefail

# Depending on the producer and timing, the producer may observe a broken pipe.
yes "record" | head -n 3

statuses=("${PIPESTATUS[@]}")
printf 'statuses: %s\n' "${statuses[*]}" 

This is not always a “real error” in the business operation; it can be a normal consequence of intentionally stopping consumption early. Under pipefail, however, such a status can make the overall pipeline non-zero. Robust scripts distinguish expected early termination from unexpected failure.

10. Know when a temporary file is clearer than a pipeline

Pipelines are ideal for streaming transformations, but a temporary file may be better when you need to:

• inspect or preserve intermediate evidence after failure,
• run multiple independent consumers over the same expensive output,
• validate a complete artifact before continuing,
• retry one stage without rerunning an expensive producer.

Good shell design is not a contest to use the most pipes. Choose the data-flow structure that makes failure and recovery easiest to reason about.

11. Hands-on lab: detect an upstream failure

Create a three-stage pipeline where the producer intentionally fails after emitting valid-looking data. Compare behavior with and without pipefail.

mkdir -p "$HOME/devops-academy/bash/chapter03/lesson04"
cd "$HOME/devops-academy/bash/chapter03/lesson04"

cat > producer.sh <<'EOF'
#!/usr/bin/env bash
printf '%s\n'   'INFO api ready'   'WARN cache slow'   'INFO worker ready'
printf 'producer: source was incomplete\n' >&2
exit 23
EOF

set +o pipefail
bash producer.sh | grep '^INFO' | wc -l
printf 'without pipefail: %d\n' "$?"

set -o pipefail
if bash producer.sh | grep '^INFO' | wc -l; then
  printf 'unexpected success\n'
else
  pipeline_status=$?
  stage_statuses=("${PIPESTATUS[@]}")
  printf 'with pipefail: %d\n' "$pipeline_status"
  printf 'recent statuses snapshot: %s\n' "${stage_statuses[*]}"
fi

12. Knowledge check

13. Summary

Pipelines are concurrent stream networks. The pipe connects stdout to the next stdin; stderr remains separate unless you deliberately merge it. Bash's default pipeline status can hide upstream failures, while PIPESTATUS exposes individual stages and pipefail makes the overall result stricter. Also account for subshell scope and legitimate SIGPIPE behavior from early consumers.

14. Further reading

• GNU Bash Reference Manual — Pipelines and PIPESTATUS.
• GNU Bash Reference Manual — pipefail and shell options.
• POSIX Shell Command Language — pipelines.
• Linux pipe(7) and signal(7) manual pages.