Archives and Compression with tar, gzip, xz, and zip

1. Archiving and compression solve different problems

An archive combines a set of files and metadata into one logical stream. Compression removes statistical redundancy from a byte stream. tar is primarily an archive format and tool; gzip and xz are compressors commonly applied to a tar stream. zip is both a container format and per-entry compression scheme. None of these is automatically a backup strategy: retention, independent storage, encryption, verification, and restore testing remain separate concerns.

flowchart TD
  S["Selected files and metadata"] --> T["tar archive stream"]
  T --> C{"Compression"}
  C --> G["gzip: fast and widely compatible"]
  C --> X["xz: smaller, slower, memory intensive"]
  C --> N["none: fastest, streamable archive"]
  G --> H["Checksum, sign, store, verify"]
  X --> H
  N --> H

2. Build tar archives from deliberate relative paths

Run tar from a controlled parent directory with -C so entries are relative. Avoid embedding absolute paths. List the archive before extraction and keep the source selection explicit. GNU tar short options are compact, but long options make automation easier to review.

work="$HOME/devops-academy/linux/chapter18/lesson02"
mkdir -p "$work" "$work/source/config" "$work/source/data"
printf 'mode=production\n' >"$work/source/config/app.conf"
printf 'record-01\nrecord-02\n' >"$work/source/data/records.txt"

# Create an uncompressed archive with relative entries.
tar --create --file="$work/application.tar" \
  --directory="$work/source" \
  config data

# Inspect without extracting.
tar --list --verbose --file="$work/application.tar"

# Test that the complete stream can be read.
tar --compare --file="$work/application.tar" \
  --directory="$work/source" 2>/dev/null || true

Shell expansion happens before tar sees arguments. Quote variables, use --null --files-from for arbitrary filenames, and place -- before operands when names could begin with a dash.

3. Choose compression according to the workflow

Gzip is generally fast and broadly supported. XZ often produces smaller archives but consumes more CPU and memory and can be much slower. Compression offers little benefit for already-compressed media, encrypted data, or many package formats. Benchmark representative data rather than relying on extension-based assumptions.

# Integrated tar compression.
tar --create --gzip --file="$work/application.tar.gz" \
  --directory="$work/source" config data

tar --create --xz --file="$work/application.tar.xz" \
  --directory="$work/source" config data

# Equivalent streaming form; pipefail preserves upstream failure.
set -o pipefail
tar --create --directory="$work/source" config data \
  | gzip -9 >"$work/application-stream.tar.gz"

# Verify compressed streams without writing extracted files.
gzip --test "$work/application.tar.gz"
xz --test "$work/application.tar.xz"

ls -lh "$work"/application.tar*

Parallel compressors such as pigz can reduce wall-clock time when CPU is available. In production, bound CPU and I/O so an archive job does not starve the workload it is protecting.

4. Use zip when ecosystem compatibility matters

ZIP is common across desktop and application ecosystems. Unix ownership, special files, ACLs, and extended attributes may not round-trip consistently across implementations. Treat ZIP as a portability format unless you have tested the exact metadata requirements.

cd "$work/source"
zip -r "$work/application.zip" config data
zipinfo "$work/application.zip"
unzip -t "$work/application.zip"

mkdir -p "$work/zip-restore"
unzip -q "$work/application.zip" -d "$work/zip-restore"
diff -ruN "$work/source" "$work/zip-restore"

Password-based ZIP encryption varies in strength and interoperability. Prefer a reviewed encryption layer or backup tool with authenticated encryption and deliberate key management rather than assuming zip -P provides adequate protection.

5. Decide which filesystem metadata must survive

Application recovery may require more than file bytes. Ownership, mode bits, timestamps, symbolic links, hard-link relationships, ACLs, extended attributes, Linux capabilities, sparse extents, device nodes, and security labels can matter. The archiving user needs permission to read and later restore that metadata.

# GNU tar options for richer Linux metadata.
sudo tar --create --file="$work/system-config.tar" \
  --acls --xattrs --selinux --numeric-owner \
  --one-file-system \
  --directory=/ etc

# Inspect numeric ownership and entry types.
tar --list --verbose --numeric-owner --file="$work/system-config.tar" | head -n 40

# Detect sparse files before choosing a strategy.
find /var/lib -xdev -type f -printf '%S %s %p\n' 2>/dev/null \
  | sort -n | head -n 20

Restoring numeric owners onto a different identity namespace can assign files to the wrong account. Preserve the identity mapping or deliberately translate it. Never extract privileged device nodes or setuid files from an untrusted archive.

6. Treat archive extraction as a write operation with attacker-controlled paths

A malicious archive can contain ../ traversal, absolute paths, symlinks that redirect later writes, hard links, device nodes, or filenames designed to confuse operators. Extract into a newly created, non-privileged directory after listing entries. Do not extract untrusted archives directly into /, a deployment directory, or a user home.

archive="$work/application.tar.gz"
restore=$(mktemp -d "$work/restore.XXXXXX")

# Reject obvious absolute and parent-traversal names before extraction.
if tar -tzf "$archive" | grep -E '(^/|(^|/)\.\.(/|$))'; then
  printf 'unsafe archive path detected\n' >&2
  exit 1
fi

# Review entry types, then extract without ownership restoration.
tar -tvzf "$archive"
tar --extract --gzip --file="$archive" \
  --directory="$restore" \
  --no-same-owner --no-same-permissions

find "$restore" -xdev -printf '%y %m %u:%g %p -> %l\n'

7. Reproducible archives remove accidental variation

CI artifacts and source releases often need identical bytes when inputs are identical. Normalize order, timestamps, owner, group, numeric IDs, locale, and compression metadata. Exclude caches and generated files deliberately. Determinism improves provenance and caching, but it does not prove the inputs were trustworthy.

export LC_ALL=C SOURCE_DATE_EPOCH=1704067200

tar --create \
  --sort=name \
  --mtime="@$SOURCE_DATE_EPOCH" \
  --owner=0 --group=0 --numeric-owner \
  --pax-option=delete=atime,delete=ctime \
  --directory="$work/source" config data \
  | gzip -n -9 >"$work/application-reproducible.tar.gz"

sha256sum "$work/application-reproducible.tar.gz" \
  >"$work/application-reproducible.tar.gz.sha256"
sha256sum --check "$work/application-reproducible.tar.gz.sha256"

Sign the checksum or artifact when authenticity matters. A checksum stored beside mutable data detects accidental corruption but not an attacker who can replace both files.

8. Split, transfer, and verify large archives carefully

For constrained channels, split a completed archive into numbered chunks and checksum both the archive and chunks. Reassemble in lexical order and verify before extraction. Prefer a transfer protocol that can resume and validate rather than creating many chunks unnecessarily.

split --bytes=500M --numeric-suffixes=0 --suffix-length=4 \
  "$work/application.tar.xz" "$work/application.tar.xz.part-"
sha256sum "$work"/application.tar.xz.part-* >"$work/parts.sha256"

# On the receiving side:
sha256sum --check "$work/parts.sha256"
cat "$work"/application.tar.xz.part-* >"$work/reassembled.tar.xz"
cmp "$work/application.tar.xz" "$work/reassembled.tar.xz"
xz --test "$work/reassembled.tar.xz"

9. Hands-on lab: create, verify, and safely restore an archive

lab="$HOME/devops-academy/linux/chapter18/lesson02/lab"
rm -rf "$lab"
mkdir -p "$lab/source/subdir" "$lab/restore"
printf 'alpha\n' >"$lab/source/a.txt"
printf 'beta\n' >"$lab/source/subdir/b.txt"
ln -s subdir/b.txt "$lab/source/latest.txt"
chmod 0640 "$lab/source/a.txt"

# Create a deterministic gzip-compressed archive.
tar --create --sort=name --mtime='UTC 2026-01-01' \
  --owner=0 --group=0 --numeric-owner \
  --directory="$lab/source" . \
  | gzip -n >"$lab/lab.tar.gz"
sha256sum "$lab/lab.tar.gz" >"$lab/lab.tar.gz.sha256"

gzip --test "$lab/lab.tar.gz"
tar -tzf "$lab/lab.tar.gz"
sha256sum --check "$lab/lab.tar.gz.sha256"
tar --extract --gzip --file="$lab/lab.tar.gz" \
  --directory="$lab/restore" --no-same-owner

diff -ruN "$lab/source" "$lab/restore"
find "$lab/restore" -printf '%y %m %p -> %l\n'

10. Common archive mistakes

11. Knowledge check

12. Summary

• Archiving, compression, backup, and artifact authenticity are distinct concerns.
• Use deliberate relative paths and inspect archives before extraction.
• Choose gzip, xz, or zip according to compatibility, speed, memory, and metadata requirements.
• Preserve only the metadata the recovery scenario actually needs.
• Deterministic archives normalize order and metadata; checksums and signatures support verification.

13. Further reading

• GNU tar manual
• GNU gzip manual
• XZ Utils documentation
• Info-ZIP project
• Reproducible Builds — archives