Process Substitution and Multi-Stream Workflows

1. Process substitution turns a command into a path-like endpoint

Bash supports two process-substitution forms:

The expansion result is typically a path referring to a pipe or open file descriptor, such as /dev/fd/63. The exact implementation depends on the operating system and Bash build.

printf 'read endpoint: %s\n' <(printf 'hello\n')
printf 'write endpoint: %s\n' >(cat >/dev/null)

2. <(command) solves the multiple-input problem

A normal pipe supplies one stdin stream. Some tools need two or more independent file operands. Process substitution lets each operand come from a command.

diff -u   <(printf '%s\n' alpha beta gamma)   <(printf '%s\n' alpha BETA gamma) || true

flowchart LR
  A["producer A"] --> P1["/dev/fd endpoint A"]
  B["producer B"] --> P2["/dev/fd endpoint B"]
  P1 --> D["diff"]
  P2 --> D

No manually named temporary files are needed, and the two producers can run while the consumer reads them.

3. Generate sorted inputs for tools that require separate files

Commands such as comm expect separate sorted inputs. Process substitution makes a multi-stage preparation pipeline appear as a filename to the final command.

comm -12   <(printf '%s\n' api worker cache | sort)   <(printf '%s\n' api frontend cache | sort)

The result is the intersection of the two sorted streams. This style is concise because the preparation logic stays attached to the operand it creates.

4. >(command) routes output into a consumer

Output process substitution is commonly paired with redirection. For example, send stderr through a logging pipeline while leaving stdout untouched:

{
  printf 'machine-result\n'
  printf 'diagnostic one\n' >&2
  printf 'diagnostic two\n' >&2
} 2> >(sed 's/^/[stderr] /' >&2)

The command writes descriptor 2 to the process-substitution endpoint. The sed process reads that stream on stdin, transforms it, and writes it back to stderr.

5. Split stdout and stderr into independent processing paths

Process substitution enables multi-stream logging without merging the channels:

run_job() {
  printf 'artifact=/tmp/app.tar.gz\n'
  printf 'warning: cache miss\n' >&2
}

run_job   > >(tee stdout.log)   2> >(tee stderr.log >&2)

Here stdout is duplicated into stdout.log while continuing to the caller through tee. stderr is independently duplicated to stderr.log and then explicitly returned to descriptor 2.

6. Feed a loop without creating a pipeline subshell

Process substitution is especially useful with while read loops when loop assignments must persist in the current shell.

count=0
last=""

while IFS= read -r service; do
  ((count += 1))
  last=$service
done < <(printf '%s\n' api worker cache)

printf 'count=%d last=%s\n' "$count" "$last" 

The loop receives data through stdin redirection instead of being a pipeline stage, so the loop itself can run in the current Bash environment.

7. Compare live state with desired state

A practical DevOps pattern is comparing two command-generated inventories:

# Example data replaces real cloud/Kubernetes commands.
desired_services() {
  printf '%s\n' api cache worker | sort
}

running_services() {
  printf '%s\n' api worker metrics | sort
}

printf '%s\n' '--- difference ---'
diff -u <(desired_services) <(running_services) || true

In real automation, each producer could be a kubectl, cloud CLI, package query, or configuration parser. Preserve failure handling: a failed producer may leave the comparison consumer with incomplete input.

8. Process-substitution failures can be less obvious than pipeline failures

One important limitation is that the process running inside <(...) or >(...) is asynchronous relative to some of the surrounding shell logic. Its failure is not represented as a normal pipeline status in the same straightforward way as cmd1 | cmd2.

# This demonstrates the shape of the risk.
# diff may only see EOF from a producer that failed.
diff   <(bash -c 'printf "partial\n"; exit 9')   <(printf 'partial\n')

If producer success is mission-critical, consider capturing data into validated temporary files or explicitly structuring the workflow so statuses can be checked. Concision is not worth losing failure visibility.

9. Process substitution is a Bash feature, not portable POSIX sh

Scripts that declare #!/usr/bin/env bash can use process substitution when the target Bash environment supports it. Scripts intended for strict POSIX sh cannot assume this syntax.

# Bash:
diff <(producer_a) <(producer_b)

# Portable design often requires explicit temporary files:
tmp_a=$(mktemp)
tmp_b=$(mktemp)
# populate, validate, compare, and clean up deliberately

This course focuses on Bash, so Bash-native features are legitimate. Still, document the interpreter requirement because CI images, minimal containers, and embedded systems may provide only another shell.

10. The path is temporary and should not be persisted

The pathname produced by process substitution is meaningful only while the underlying descriptor or pipe remains available. Do not save it in a database or configuration file for later use.

endpoint=<(printf 'temporary data\n')
printf 'endpoint now: %s\n' "$endpoint"

# Treat this as an immediate-use endpoint, not a durable filename.
cat "$endpoint" 

If data must survive after the command finishes, write a real file or durable artifact instead.

11. Build a multi-stream command harness

A reusable command harness can timestamp stdout and stderr independently while preserving their destination descriptors:

timestamp_stream() {
  local label=$1
  while IFS= read -r line; do
    printf '[%(%Y-%m-%dT%H:%M:%S%z)T] [%s] %s\n' -1 "$label" "$line"
  done
}

demo_job() {
  printf 'artifact=/tmp/example.tar.gz\n'
  printf 'cache miss\n' >&2
  printf 'build complete\n' >&2
}

demo_job   > >(timestamp_stream STDOUT)   2> >(timestamp_stream STDERR >&2)

In a real command whose stdout must remain parseable, you would usually avoid timestamping stdout. The example demonstrates routing mechanics; choose transformations according to the stream contract.

12. Hands-on lab: compare desired and observed services

Create two generated inventories, compare them without temporary files, and then use a process-substitution-fed loop to count observed services without losing the count variable.

mkdir -p "$HOME/devops-academy/bash/chapter03/lesson05"
cd "$HOME/devops-academy/bash/chapter03/lesson05"

desired() {
  printf '%s\n' api cache worker | sort
}

observed() {
  printf '%s\n' api metrics worker | sort
}

printf '%s\n' '=== desired vs observed ==='
diff -u <(desired) <(observed) || true

count=0
while IFS= read -r service; do
  ((count += 1))
  printf 'observed: %s\n' "$service"
done < <(observed)

printf 'observed_count=%d\n' "$count"

printf '%s\n' '=== common services ==='
comm -12 <(desired) <(observed)

13. Knowledge check

14. Summary

Process substitution adapts command streams to interfaces that expect path-like operands. It is excellent for comparing multiple generated inputs, preserving current-shell loop state, and routing stdout and stderr through separate consumers. Its endpoints are temporary, its syntax is Bash-specific, and producer failures can be harder to observe than ordinary pipeline failures—so use it where the interface benefit is clear.

15. Further reading

• GNU Bash Reference Manual — Process Substitution.
• GNU Bash Reference Manual — Redirections and Pipelines.
• GNU Coreutils documentation for comm, diff, and tee.
• Linux pipe(7) and proc(5) manual pages for descriptor-backed endpoints.