rsync-Based Backups and Remote Replication

1. rsync synchronizes trees; policy makes the synchronization a backup

rsync efficiently makes a destination resemble a source by comparing metadata and, when needed, transferring file data. It can operate locally, through a remote shell such as SSH, or against an rsync daemon. A mirror alone is not historical backup: deletion and corruption can be reproduced immediately. Add immutable or versioned generations, retention, independent credentials, monitoring, and restore tests.

flowchart TD
  S["Define source boundary and exclusions"] --> D["Dry-run with itemized changes"]
  D --> R["Review delete, ownership, links, and destination"]
  R --> T["Transfer through constrained identity"]
  T --> V["Verify exit status, manifest, and sample content"]
  V --> G["Publish generation atomically"]
  G --> P["Apply retention without deleting last known-good"]

2. The source trailing slash changes the destination tree

With source /srv/app, rsync copies the directory name into the destination. With source /srv/app/, it copies the contents of that directory. Both can be correct, but confusing them creates nested paths or writes files one level higher than intended. State the desired restored path before writing the command.

mkdir -p /tmp/rsync-demo/source/sub /tmp/rsync-demo/a /tmp/rsync-demo/b
printf 'data\n' >/tmp/rsync-demo/source/sub/file.txt

rsync -a /tmp/rsync-demo/source  /tmp/rsync-demo/a/
# Result: /tmp/rsync-demo/a/source/sub/file.txt

rsync -a /tmp/rsync-demo/source/ /tmp/rsync-demo/b/
# Result: /tmp/rsync-demo/b/sub/file.txt

find /tmp/rsync-demo/a /tmp/rsync-demo/b -type f -print

The same rule applies to remote paths. Quote remote operands so the local shell does not expand wildcards unexpectedly.

3. Quick checks and delta transfer are not end-to-end content proof

Rsync usually decides whether a regular file needs transfer using size and modification time. With suitable local/remote conditions it can use a block-delta algorithm to avoid sending unchanged blocks. --checksum forces content checksums for the pre-transfer decision but reads every file on both sides and is not a substitute for a protected backup manifest.

# Observe exactly what rsync believes changed.
rsync -a --dry-run --itemize-changes --stats \
  /srv/app/ /mnt/backup/app-current/

# Use checksums only when the additional full read is justified.
rsync -a --checksum --dry-run --itemize-changes \
  /srv/app/ /mnt/backup/app-current/

4. Archive mode is a starting point, not “preserve everything”

-a expands to recursive copying with links, permissions, times, group, owner, and devices/special files, but it does not include ACLs, extended attributes, hard-link preservation, or every filesystem-specific feature. Remote privilege and identity mapping determine what can actually be restored.

# Linux-oriented metadata preservation for a trusted system backup.
sudo rsync -aHAX --numeric-ids --sparse \
  --one-file-system \
  --info=stats2,progress2 \
  /srv/app/ /mnt/backup/app-current/

# Inspect metadata on a sample after transfer.
getfacl -p /srv/app/config/app.conf /mnt/backup/app-current/config/app.conf
getfattr -d -m- /srv/app/config/app.conf /mnt/backup/app-current/config/app.conf 2>/dev/null || true
stat -c '%n %a %u:%g %s %y' \
  /srv/app/config/app.conf /mnt/backup/app-current/config/app.conf

--numeric-ids preserves numeric identities rather than mapping names. That is useful for same-namespace recovery and risky across different identity domains. Document the intended restore host.

5. Filters define scope; deletion defines destructive convergence

Use ordered include and exclude rules. A parent directory must be included for a deeper include to be reachable. Store filters in a reviewed file for complex jobs. --delete removes destination entries absent from the source and can destroy the only good copy after accidental source deletion. Always dry-run and use a versioned destination.

# /etc/devops-backup/app.rules
+ /config/***
+ /data/***
- /cache/***
- /tmp/***
- *.pid
- *

rsync -aHAX --numeric-ids \
  --filter='merge /etc/devops-backup/app.rules' \
  --delete-delay \
  --dry-run --itemize-changes \
  /srv/app/ /mnt/backup/app-current/

# Review deletions explicitly before removing --dry-run.
rsync -aHAX --numeric-ids \
  --filter='merge /etc/devops-backup/app.rules' \
  --delete-delay \
  --itemize-changes --stats \
  /srv/app/ /mnt/backup/app-current/

6. Constrain remote transport and destination authority

Use a dedicated backup account, host-key verification, a narrowly authorized SSH key, and a destination not writable by production services. Do not disable host-key checking. Consider a forced command or restricted wrapper that validates module, path, and arguments. Separate the ability to append new generations from the ability to delete old ones.

# Client-side example with explicit identity and host-key policy.
rsync -aHAX --numeric-ids --partial --delay-updates \
  -e 'ssh -i /etc/devops-backup/id_ed25519 -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes' \
  /srv/app/ \
  backup@backup01.example.invalid:/srv/backup/incoming/app/

# Test connection and remote identity separately.
ssh -i /etc/devops-backup/id_ed25519 \
  -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes \
  backup@backup01.example.invalid 'id; pwd'

For root-owned metadata, avoid granting unrestricted remote root. Evaluate --fake-super, filesystem capabilities, a constrained receiver, or a purpose-built backup repository according to the threat model.

7. --link-dest can create snapshot-style generations

A hard-link-based generation stores unchanged regular files as hard links to a previous generation while changed files receive new inodes. Each generation appears as a complete tree. The reference path supplied to --link-dest is interpreted relative to the destination when it is not absolute.

#!/usr/bin/env bash
set -Eeuo pipefail
umask 077

root=/srv/backup/app
stamp=$(date -u +%Y%m%dT%H%M%SZ)
staging="$root/.staging-$stamp"
final="$root/$stamp"
latest="$root/latest"
mkdir -p "$staging"

args=(-aHAX --numeric-ids --delete-delay --itemize-changes)
if [[ -L "$latest" ]]; then
  previous=$(readlink -f "$latest")
  args+=(--link-dest="$previous")
fi

rsync "${args[@]}" /srv/app/ "$staging/"
printf '%s\n' "$stamp" >"$staging/BACKUP_GENERATION"
find "$staging" -xdev -type f -print0 | sort -z | xargs -0 sha256sum \
  >"$staging/SHA256SUMS"
mv "$staging" "$final"
ln -sfn "$stamp" "$root/.latest-new"
mv -Tf "$root/.latest-new" "$latest"

Hard-linked files must not be modified in place after publication, or previous generations change too. Make completed generations read-only and apply retention carefully.

8. Choose interruption behavior deliberately

--partial keeps partially transferred files for resume. --partial-dir isolates them. --delay-updates moves completed temporary files into place near the end. --inplace writes directly to destination files and can reduce temporary space, but it exposes partial state and can corrupt hard-linked generations. Avoid it for published snapshots unless you fully understand the consequences.

rsync -aHAX --numeric-ids \
  --partial-dir=.rsync-partial \
  --delay-updates \
  --timeout=120 \
  --contimeout=20 \
  /srv/app/ backup@backup01.example.invalid:/srv/backup/incoming/app/

Rsync exit code 24 can indicate vanished source files during an active workload; that may be expected for caches and unacceptable for a consistent dataset. Classify and alert on exit codes rather than converting all nonzero results to success.

9. Verify the replica independently

Check command exit status, destination capacity, generation publication, file counts, metadata samples, and protected manifests. Run a read-only comparison and periodic restore. A second rsync dry-run can reveal drift, but a compromised source and destination may agree on the wrong state.

# Read-only comparison after transfer.
rsync -aHAXn --numeric-ids --delete --itemize-changes \
  /srv/app/ /srv/backup/app/latest/

# Compare logical counts and bytes.
find /srv/app -xdev -type f -printf '%s\n' | awk '{n++; b+=$1} END {print n,b}'
find /srv/backup/app/latest -xdev -type f -printf '%s\n' | awk '{n++; b+=$1} END {print n,b}'

# Verify a generation manifest from protected storage.
cd /srv/backup/app/latest
sha256sum --check SHA256SUMS

10. Hands-on lab: build two local generations

lab="$HOME/devops-academy/linux/chapter18/lesson04"
source="$lab/source"
repo="$lab/repository"
rm -rf "$lab"
mkdir -p "$source" "$repo/gen1" "$repo/gen2"
printf 'one\n' >"$source/a.txt"
printf 'stable\n' >"$source/unchanged.txt"

rsync -a "$source/" "$repo/gen1/"
printf 'two\n' >"$source/a.txt"
printf 'new\n' >"$source/b.txt"
rsync -a --link-dest="$repo/gen1" "$source/" "$repo/gen2/"

# Unchanged files should share an inode; changed files should not.
ls -li "$repo/gen1/unchanged.txt" "$repo/gen2/unchanged.txt"
ls -li "$repo/gen1/a.txt" "$repo/gen2/a.txt"
diff -ruN "$source" "$repo/gen2"

11. Common rsync mistakes

12. Knowledge check

13. Summary

• Rsync synchronizes trees; versioning, retention, isolation, and testing create a backup system.
• Source trailing-slash semantics must be understood before every transfer.
• Use dry-run, itemized changes, filters, and reviewed deletion behavior.
• Archive mode does not preserve every Linux metadata class.
• Constrained SSH identities, atomic generation publication, manifests, and restores make remote replication defensible.

14. Further reading

• rsync(1)
• How rsync works
• ssh(1)
• ssh_config(5)
• sha256sum(1)