Transforming Text with cut, sort, uniq, tr, and wc

1. Every text transformation encodes a data model

A line-oriented pipeline assumes records are separated by newline. cut may assume fields use one delimiter. sort assumes a collation and key model. uniq assumes equal records are adjacent. tr operates on characters, while wc counts selected units.

flowchart TD
  I["Input records"] --> C["cut selects fields"]
  C --> T["tr normalizes characters"]
  T --> S["sort orders equal keys together"]
  S --> U["uniq counts adjacent duplicates"]
  U --> W["wc or sort produces a summary"]

2. Select stable fields with cut

cut can select byte positions, character positions, or delimiter-separated fields. It does not understand repeated whitespace as one separator and it does not parse quoted CSV.

data="$HOME/devops-academy/linux/chapter05/lesson04/services.tsv"

# Select tab-separated service name and status fields.
cut -f 1,3 -- "$data"

# Select colon-separated username and login shell from passwd syntax.
cut -d: -f1,7 -- /etc/passwd

# Select fixed character positions only when the layout is truly fixed.
printf '%s\n' '20260804orders-api OK' | cut -c1-8,9-18,20-

For tabular operational data you control, tabs are often safer than arbitrary runs of spaces. If input spacing is irregular, normalize it first or use awk, which can split on runs of whitespace by default.

3. Sort with explicit keys and comparison modes

sort reads all input, compares records according to the selected locale and options, and writes ordered output. Lexicographic order differs from numeric, human-size, month, and version order.

# Stable bytewise ordering for reproducible automation.
LC_ALL=C sort -- "$data"

# Sort a tab-separated table by numeric latency in field 4, descending.
sort -t $'\t' -k4,4nr -- "$data"

# Version-aware ordering.
printf '%s\n' v1.9 v1.10 v1.2 | sort -V

# Write safely to an explicit output file.
sort -t $'\t' -k1,1 -o "$data.sorted" -- "$data"

Set LC_ALL=C when bytewise reproducibility is required across hosts. Human-facing alphabetical order may benefit from the user’s locale, but automation should not silently change when deployed to a machine with different locale settings.

4. uniq operates on adjacent duplicates

uniq compares neighboring records. It does not discover duplicates scattered through unsorted input. The common pattern is therefore sort | uniq, or sort | uniq -c for counts.

printf '%s\n' api worker api scheduler worker \
  | LC_ALL=C sort \
  | uniq -c \
  | sort -nr

# Display only duplicated adjacent records after sorting.
cut -f1 -- "$data" | LC_ALL=C sort | uniq -d

# Display records that occur exactly once.
cut -f1 -- "$data" | LC_ALL=C sort | uniq -u

sort -u can combine ordering and uniqueness, but its key and comparison options determine which records are considered equal. sort | uniq and sort -u are not interchangeable in every keyed sort.

5. Translate, delete, and squeeze characters with tr

tr reads standard input and writes standard output. It does not accept ordinary input filenames. Use it to normalize case, map delimiters, delete selected characters, or squeeze repeated characters.

# Normalize ASCII lowercase to uppercase under the C locale.
printf '%s\n' 'staging-ready' | LC_ALL=C tr '[:lower:]' '[:upper:]'

# Convert commas to tabs for simple unquoted data.
printf '%s\n' 'orders,healthy,42' | tr ',' '\t'

# Squeeze repeated spaces into one space.
printf '%s\n' 'orders     healthy   42' | tr -s ' '

# Delete carriage returns from CRLF text streams.
tr -d '\r' < windows-input.txt > unix-output.txt

Character ranges and classes can be locale-sensitive. Byte-oriented cleanup should set a suitable locale and verify that multibyte text is not being damaged.

6. Count the unit you actually need

wc can report lines, words, bytes, characters, and maximum display width. A “line” count is conventionally a count of newline characters, so a final unterminated record may surprise you.

wc -l -- "$data"      # newline count
wc -w -- "$data"      # word count
wc -c -- "$data"      # byte count
wc -m -- "$data"      # character count in the current locale

# Capture a clean numeric value without the filename column.
record_count=$(wc -l < "$data")
printf 'records=%d\n' "$record_count"

Byte count and character count differ for multibyte encodings such as UTF-8. Operational limits may be defined in bytes, while user-visible length may be defined in characters.

7. Hands-on lab: summarize a service access log

lab="$HOME/devops-academy/linux/chapter05/lesson04"
mkdir -p -- "$lab"
log="$lab/access.tsv"

cat > "$log" <<'EOF'
2026-08-04T11:00:01Z	orders-api	200	42
2026-08-04T11:00:02Z	payments-api	503	870
2026-08-04T11:00:03Z	orders-api	200	38
2026-08-04T11:00:04Z	orders-api	500	410
2026-08-04T11:00:05Z	payments-api	200	95
2026-08-04T11:00:06Z	catalog-api	200	51
2026-08-04T11:00:07Z	payments-api	503	920
EOF

printf '%s\n' '=== requests per service ==='
cut -f2 -- "$log" \
  | LC_ALL=C sort \
  | uniq -c \
  | sort -nr

printf '%s\n' '=== status-code frequency ==='
cut -f3 -- "$log" \
  | LC_ALL=C sort \
  | uniq -c \
  | sort -nr

printf '%s\n' '=== slow requests, highest latency first ==='
sort -t $'\t' -k4,4nr -- "$log" | head -n 3

printf 'total_requests=%s\n' "$(wc -l < "$log")"
printf 'error_requests=%s\n' "$(cut -f3 -- "$log" | grep -Ec '^(5[0-9]{2})$')"

8. Common mistakes and stronger habits

9. Knowledge check

10. Summary

cut selects stable fields, sort establishes order, uniq collapses adjacent equality, tr normalizes characters, and wc counts explicit units. Correct pipelines make delimiters, keys, locale, and record assumptions visible.

11. Further reading

• GNU Coreutils manual — cut, sort, uniq, tr, and wc.
• GNU Grep manual — record selection and regular expressions.
• GNU Bash Reference Manual — pipelines, command substitution, quoting, and locale variables.
• POSIX utility specifications for text-processing portability.
• Unicode and locale documentation for byte, character, and collation behavior.