Streaming versus Buffering Large Data

1. Streaming processes records incrementally

A streaming workflow handles records as they arrive instead of loading the entire dataset first. This lowers peak memory and can reduce latency for large files, logs, and command output.

flowchart LR
  I["input"] --> R1["record"]
  R1 --> P["process"]
  P --> O["output"]
  I --> R2["next record"]

2. Buffering is useful when the full dataset is genuinely needed

mapfile -t lines < small-input.txt
printf 'count=%d\n' "${#lines[@]}"

If later logic needs random access, repeated traversal, or full-dataset decisions and the input is small enough, buffering can simplify the design.

3. Avoid giant command substitutions

generate_large_report |
while IFS= read -r line; do
  process "$line"
done

Capturing a huge report into one variable increases memory use and delays downstream processing until generation is complete.

4. Pipelines are natural streaming boundaries

journalctl -u myapp --since today |
grep 'ERROR' |
awk '{ count[$5]++ } END { for (k in count) print k, count[k] }'

Each stage can begin work before the upstream producer finishes.

5. Some operations inherently need working storage

Sorting, deduplication, grouping, and joins may require large working sets. Use tools designed for those operations rather than building huge Bash arrays.

6. Temporary files are a legitimate scalability tool

tmp=$(mktemp) || exit 1
trap 'rm -f -- "$tmp"' EXIT

generate_large_report > "$tmp"
validate_report "$tmp"
compress_report "$tmp"

A temporary file is useful when several passes are required or downstream tools need seekable input.

7. tee observes a stream without fully buffering it

set -o pipefail

generate_data |
tee raw.log |
transform |
consume

With pipefail, logging a stream does not have to hide upstream failure.

8. Streaming safety depends on record boundaries

find artifacts -type f -print0 |
while IFS= read -r -d '' file; do
  process "$file"
done

For Unix pathnames, NUL-delimited records are robust because NUL cannot occur inside a pathname.

9. Pipes provide backpressure

When a consumer is slower than its producer, pipe buffers eventually fill and the producer blocks. This prevents unlimited memory growth and is one reason streaming pipelines scale well.

10. Hands-on lab: buffered versus streaming input

mkdir -p "$HOME/devops-academy/bash/chapter19/lesson02"
cd "$HOME/devops-academy/bash/chapter19/lesson02"

seq 1 100000 > numbers.txt

cat > buffered.sh <<'EOF'
#!/usr/bin/env bash
mapfile -t numbers < numbers.txt
sum=0
for n in "${numbers[@]}"; do
  ((sum += n))
done
printf 'sum=%d\n' "$sum"
EOF

cat > streaming.sh <<'EOF'
#!/usr/bin/env bash
sum=0
while IFS= read -r n; do
  ((sum += n))
done < numbers.txt
printf 'sum=%d\n' "$sum"
EOF

chmod u+x buffered.sh streaming.sh

time bash buffered.sh
time bash streaming.sh

11. Knowledge check

12. Summary

Stream data when you can process it incrementally, buffer when the algorithm needs the whole dataset, and use temporary files for large multi-pass workflows. Match record boundaries and buffering behavior to the data.

13. Further reading

• GNU Bash Reference Manual — pipelines and mapfile.
• GNU Coreutils documentation — tee, sort, uniq.
• Unix pipe and buffering documentation.
• GNU findutils documentation for NUL-delimited records.