SSH Clients, Servers, and Host Keys

1. SSH establishes an authenticated encrypted channel

Secure Shell is both a protocol family and a collection of tools. The ssh client initiates a TCP connection, negotiates algorithms, verifies the server’s host identity, performs user authentication, and opens one or more logical channels inside an encrypted connection. The sshd daemon listens on the server, presents host keys, applies server policy, and creates sessions or forwarding channels for authenticated users.

These identities are separate. A host key answers “which server am I talking to?” A user password, user key, certificate, hardware authenticator, or other method answers “which user is requesting access?” Confusing the two makes it easy to accept a man-in-the-middle server or to diagnose the wrong authentication stage.

sequenceDiagram
  participant C as SSH client
  participant N as Network path
  participant S as sshd server
  participant A as Authentication policy
  C->>N: TCP connection to host:port
  N->>S: Deliver connection
  C->>S: Send client version and algorithm proposals
  S-->>C: Return server version and selected algorithms
  S-->>C: Host public key and proof
  C->>C: Verify known_hosts or trusted CA
  C->>S: Perform key exchange and start encrypted transport
  S-->>C: Confirm encrypted transport
  C->>A: Offer user authentication method
  A-->>C: Accept or reject
  C->>S: Open shell, command, SFTP, or forwarding channel
  S-->>C: Return channel data and responses

2. Client and server components have different responsibilities

Distribution naming differs. Debian-family systems often call the unit ssh.service; many RPM-family systems use sshd.service. The executable remains sshd. Diagnose the actual host rather than copying a service name blindly.

3. Connection setup is negotiated, not assumed

Client and server advertise protocol versions and supported key-exchange, host-key, encryption, and message-authentication algorithms. They select mutually supported choices according to policy. A failure before user authentication may therefore be a TCP, version, algorithm, host-key, or configuration problem—not a bad password.

# Client version and compiled feature information
ssh -V

# Supported algorithm categories on this client
ssh -Q kex
ssh -Q key
ssh -Q cipher
ssh -Q mac

# Compute the effective client configuration without connecting
ssh -G example-host | sed -n '1,120p'

# Bounded TCP reachability test before invoking SSH
nc -vz -w 3 example-host 22 2>&1 || true

ssh -G expands configuration, aliases, defaults, and command-line options into the effective client settings. It is one of the safest ways to discover which hostname, port, user, identity files, proxy, and host-key policy the client will actually use.

4. Host keys give a server a persistent cryptographic identity

During the handshake, the server proves possession of a host private key. The client compares the corresponding public key or certificate with its trust data. On a first connection, many clients use trust on first use and ask the operator to confirm a fingerprint. That confirmation is meaningful only when the fingerprint is obtained through an independent trusted channel such as a provider console, configuration-management record, deployment output, or an administrator already connected securely.

# Inspect public host keys on a server you administer
sudo find /etc/ssh -maxdepth 1 -type f -name 'ssh_host_*_key.pub' -print

# Print SHA-256 fingerprints without exposing private keys
for key in /etc/ssh/ssh_host_*_key.pub; do
  [ -r "$key" ] || continue
  ssh-keygen -lf "$key" -E sha256
 done

# Inspect local trust records for a hostname
ssh-keygen -F server.example.net

# Remove a stale entry only after independently verifying a legitimate rotation
# ssh-keygen -R server.example.net

# Hash existing clear-text hostnames in known_hosts
# ssh-keygen -H -f "$HOME/.ssh/known_hosts"

A changed host key can indicate a rebuilt server, restored image, address reassignment, key rotation, load-balancer change, DNS error, or attack. Do not solve the warning by deleting the entry until the change has been explained and verified.

5. Inspect the server before changing it

# Identify package and executable availability
command -v ssh
command -v sshd || true

# Discover service names and current state
systemctl list-unit-files 'ssh*.service' --no-pager 2>/dev/null || true
systemctl status ssh.service --no-pager 2>/dev/null || true
systemctl status sshd.service --no-pager 2>/dev/null || true

# Verify listeners and owning processes
ss -lntp 'sport = :22' 2>/dev/null || true

# Read effective daemon configuration when permitted
sudo sshd -T 2>/dev/null | sed -n '1,160p' || true

# Recent service events; unit name varies
sudo journalctl -u ssh.service -u sshd.service --since '-30 min' --no-pager 2>/dev/null || true

sshd -T prints effective configuration after defaults and includes are processed. Later lessons will use it with connection criteria to evaluate Match blocks. Never restart a remote SSH daemon merely to “see whether the file works.” Validate first, preserve the existing session, and ensure console or out-of-band access.

6. Hands-on lab: create an SSH endpoint evidence bundle

This read-only lab records local client behavior and, when a target is supplied, collects the target’s offered host keys for later independent verification. It does not connect, authenticate, or modify trust files.

lab="$HOME/devops-academy/linux/chapter13/lesson01"
mkdir -p "$lab"
cd "$lab"

{
  date --iso-8601=seconds
  printf '\n=== client version ===\n'
  ssh -V 2>&1
  printf '\n=== effective localhost client config ===\n'
  ssh -G localhost 2>/dev/null | sed -n '1,120p'
  printf '\n=== local listeners ===\n'
  ss -lntp 'sport = :22' 2>&1 || true
  printf '\n=== local host-key fingerprints ===\n'
  for key in /etc/ssh/ssh_host_*_key.pub; do
    [ -r "$key" ] && ssh-keygen -lf "$key" -E sha256
  done
} > ssh-baseline.txt

# Optional: TARGET=server.example.net PORT=22 bash this-script.sh
if [ -n "${TARGET:-}" ]; then
  port=${PORT:-22}
  ssh-keyscan -T 5 -p "$port" "$TARGET" > offered-host-keys.txt 2> keyscan-errors.txt || true
  ssh-keygen -lf offered-host-keys.txt -E sha256 > offered-fingerprints.txt 2>&1 || true
  printf 'VERIFY THESE FINGERPRINTS THROUGH AN INDEPENDENT TRUSTED CHANNEL.\n' > verification-required.txt
fi

sha256sum ./*.txt > evidence.sha256

7. Common SSH foundation mistakes

8. Knowledge check

9. Summary

SSH remote administration begins with a layered trust model. TCP delivers the connection, OpenSSH negotiates cryptographic algorithms, the server proves its host identity, the client verifies that identity, and only then does user authentication authorize a session. Reliable operators separate these stages, inspect effective configuration, preserve fingerprints, and treat host-key warnings as security evidence rather than inconvenience.

10. Further reading

• OpenSSH ssh(1) manual.
• OpenSSH sshd(8) manual.
• OpenSSH ssh-keygen(1) manual.
• ssh_config(5), sshd_config(5), and your distribution’s OpenSSH service documentation.