CSV, Delimited Data, and Quoting Hazards

1. Delimited text is only simple under a strict contract

A format such as service:environment:replicas is easy to parse when you control the data and guarantee that the delimiter can never appear inside a field.

IFS=: read -r service environment replicas \
  <<< 'api:prod:3'

This is not equivalent to parsing CSV.

2. Real CSV has quoting rules

CSV commonly allows delimiters inside quoted fields, quote escaping, and even embedded newlines. The line "Smith, Jane",admin contains a comma that is data, not a field separator.

3. See how naive splitting fails

line='"Smith, Jane",admin,active'

IFS=, read -r name role status <<< "$line"

printf 'name=%s\n' "$name"
printf 'role=%s\n' "$role"
printf 'status=%s\n' "$status"

The fields are wrong because the shell does not implement CSV quote semantics.

4. Use a CSV-aware parser for real CSV

python3 - <<'PY'
import csv

with open("users.csv", newline="", encoding="utf-8") as f:
    for row in csv.DictReader(f):
        print(row["name"], row["role"], sep="\t")
PY

Python's standard csv module is often available in DevOps environments and correctly understands CSV quoting rules.

5. TSV can be simpler, but it still needs a contract

while IFS=$'\t' read -r service environment replicas; do
  printf 'service=%s env=%s replicas=%s\n' \
    "$service" "$environment" "$replicas"
done < services.tsv

This is safe only when tabs and record-breaking newlines are forbidden or escaped according to an agreed format.

6. NUL delimiters are ideal for Unix path records

find artifacts -type f -print0 |
while IFS= read -r -d '' file; do
  printf 'path=%q\n' "$file"
done

Use delimiter formats that match the domain. NUL is uniquely suitable for Unix filenames because the byte cannot appear inside a pathname.

7. Inventing an escaping grammar creates a new parser problem

Once you decide that delimiters may be escaped with backslashes, quotes, doubling, or percent encoding, you have created a grammar. At that point, a standard structured format is often cheaper and safer.

8. Headers are schema, not just decoration

python3 - <<'PY'
import csv

with open("services.csv", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    required = {"service", "environment", "replicas"}
    missing = required - set(reader.fieldnames or [])
    if missing:
        raise SystemExit(f"missing columns: {sorted(missing)}")
PY

Column names let a parser validate schema and avoid depending on physical column order.

9. Text fields still need semantic validation

[[ $replicas =~ ^[0-9]+$ ]] || {
  printf 'invalid replicas: %q\n' "$replicas" >&2
  exit 65
}

A correct CSV parser guarantees syntax, not business meaning.

10. Encoding and line endings can be operational issues

Files from spreadsheets or Windows systems may contain BOM markers or CRLF line endings. A format-aware parser and explicit UTF-8 policy reduce surprises compared with ad-hoc shell splitting.

11. Generate CSV with a CSV writer too

python3 - <<'PY'
import csv
import sys

writer = csv.writer(sys.stdout)
writer.writerow(["name", "role"])
writer.writerow(["Smith, Jane", "admin"])
PY

Correct output escaping matters just as much as correct input parsing.

12. Hands-on lab: compare naive and real CSV parsing

mkdir -p "$HOME/devops-academy/bash/chapter13/lesson04"
cd "$HOME/devops-academy/bash/chapter13/lesson04"

cat > users.csv <<'EOF'
name,role,note
"Smith, Jane",admin,"primary operator"
"Worker ""Blue""",viewer,"contains a quoted nickname"
EOF

printf '%s\n' '--- CSV-aware parser ---'
python3 - <<'PY'
import csv

with open("users.csv", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(
            f"name={row['name']!r} "
            f"role={row['role']!r} "
            f"note={row['note']!r}"
        )
PY

printf '%s\n' '--- naive split of first record (incorrect) ---'
line=$(sed -n '2p' users.csv)
IFS=, read -r a b c <<< "$line"
printf 'a=%s b=%s c=%s\n' "$a" "$b" "$c"

13. Knowledge check

14. Summary

Delimited text is safe only when its delimiter rules are explicit. General CSV needs a CSV parser, TSV still needs field restrictions or escaping rules, and Unix filenames need NUL-delimited records. Avoid creating undocumented mini-formats in shell code.

15. Further reading

• Python standard library — csv module.
• RFC 4180 — common CSV format conventions.
• GNU Coreutils documentation — text utilities and delimiters.
• GNU findutils — NUL-delimited pathname processing.