Hardening SSH and Troubleshooting Remote Access

1. Hardening is controlled reduction of capability

SSH hardening is not a list of copied directives. It begins with an access model: who may connect, from where, through which network path, with which authentication factors, to which accounts, for which session or forwarding capabilities, and how emergency recovery works. Each restriction should map to a requirement and have a test.

The safest sequence is: preserve a working session, obtain console or out-of-band access, record effective configuration, create a backup, edit a drop-in or managed source, validate syntax and effective policy, reload rather than blindly restart, test a new session, verify logs and required workflows, then close the original session only after success.

flowchart TB
  A["Define required access and recovery path"] --> B["Record effective sshd policy and active sessions"]
  B --> C["Back up authoritative configuration"]
  C --> D["Make one reviewed change"]
  D --> E["Validate syntax with sshd -t"]
  E --> F["Inspect effective policy with sshd -T"]
  F --> G["Reload while keeping existing session"]
  G --> H["Test new login and required channels"]
  H --> I{"Success?"}
  I -->|Yes| J["Record evidence and commit"]
  I -->|No| K["Rollback through preserved session or console"]
  K --> B

2. Inspect what sshd will actually enforce

The main configuration may include drop-in directories, distribution defaults, and conditional Match blocks. Reading one file is not enough. sshd -t checks configuration validity and key availability; sshd -T prints effective settings. The -C connection specification evaluates conditional rules for a particular user, host, and address.

# Validate current daemon configuration; no restart or reload
sudo sshd -t

# Print effective global policy
sudo sshd -T > sshd-effective-global.txt

# Evaluate Match blocks for a representative connection
sudo sshd -T \
  -C user=deploy,host=app01.example.net,addr=192.0.2.50 \
  > sshd-effective-deploy.txt

# Review security-relevant settings
for file in sshd-effective-global.txt sshd-effective-deploy.txt; do
  printf '\n=== %s ===\n' "$file"
  grep -E '^(port|listenaddress|permitrootlogin|passwordauthentication|kbdinteractiveauthentication|pubkeyauthentication|authenticationmethods|allowusers|allowgroups|denyusers|denygroups|maxauthtries|logingracetime|allowtcpforwarding|allowagentforwarding|x11forwarding|permittunnel|gatewayports|permituserenvironment|disableforwarding) ' "$file" || true
 done

# Discover includes and authoritative files
sudo grep -RInE '^[[:space:]]*(Include|Match|PasswordAuthentication|PermitRootLogin|AllowUsers|AllowGroups|AuthenticationMethods)' \
  /etc/ssh/sshd_config /etc/ssh/sshd_config.d 2>/dev/null || true

Some settings are lists, some can be reset or appended, and some are only meaningful in particular scopes. Use the manual for the installed version and test effective output rather than assuming a copied configuration behaves as expected.

3. Harden identity and session capability deliberately

# Illustrative drop-in; review against your actual access model.
# /etc/ssh/sshd_config.d/50-security-baseline.conf
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
MaxAuthTries 4
LoginGraceTime 30
X11Forwarding no
AllowAgentForwarding no
PermitUserEnvironment no

# Keep forwarding disabled by default, then allow it narrowly if required.
AllowTcpForwarding no
PermitTunnel no

Match Group ssh-forwarders
    AllowTcpForwarding local
    PermitOpen database.internal:5432

4. The daemon is only one layer of the remote-access boundary

Place SSH behind security groups, firewalls, VPNs, private networks, zero-trust gateways, or bastions where appropriate. Restricting port 22 by source can reduce attack surface, but source addresses may change, NAT can aggregate users, and IPv6 paths may bypass IPv4-only policy. Host controls and network controls should reinforce each other.

# Verify listeners and bind addresses
sudo ss -lntp | grep -E '(:22\b|sshd)' || true

# Read firewall policy without changing it
sudo nft list ruleset 2>/dev/null | sed -n '1,220p' || true
sudo firewall-cmd --list-all 2>/dev/null || true
sudo ufw status verbose 2>/dev/null || true

# Confirm both IPv4 and IPv6 routes and addresses
ip -brief address
ip route show
ip -6 route show

# Discover failed-login evidence; log locations vary
sudo journalctl -u ssh.service -u sshd.service --since '-24 hours' --no-pager 2>/dev/null | \
  grep -Ei 'failed|invalid|refused|disconnect|error' | tail -n 120 || true

5. Troubleshoot by connection stage

# Client-side evidence: avoid password prompts during diagnosis
ssh -vvv \
  -o BatchMode=yes \
  -o ConnectTimeout=8 \
  -o IdentitiesOnly=yes \
  -i "$HOME/.ssh/id_ed25519_devops_admin" \
  user@server.example.net true \
  2> ssh-client-debug.log || true

# Effective client choices
ssh -G server.example.net > ssh-client-effective.txt

# Server-side account and path checks
getent passwd user
sudo passwd -S user 2>/dev/null || true
sudo namei -l /home/user/.ssh/authorized_keys 2>/dev/null || true
sudo stat -c '%A %a %U:%G %n' \
  /home/user /home/user/.ssh /home/user/.ssh/authorized_keys 2>/dev/null || true

# Server events around the attempt
sudo journalctl -u ssh.service -u sshd.service \
  --since '-10 minutes' --no-pager > ssh-server-events.log 2>/dev/null || true

# SELinux evidence where applicable
sudo ausearch -m AVC,USER_AVC -ts recent 2>/dev/null | tail -n 120 || true

Verbose output can contain usernames, hostnames, paths, key fingerprints, proxy details, and addresses. Treat diagnostic bundles as potentially sensitive and redact carefully before sharing.

6. Reload and test without destroying the recovery path

# Example controlled sequence after an approved edit
sudo cp -a /etc/ssh/sshd_config.d/50-security-baseline.conf \
  "/etc/ssh/sshd_config.d/50-security-baseline.conf.bak.$(date +%Y%m%d%H%M%S)"

sudo sshd -t
sudo sshd -T -C user="$USER",host="$(hostname -f 2>/dev/null || hostname)",addr="${ADMIN_SOURCE_IP:?set ADMIN_SOURCE_IP}" \
  | sed -n '1,180p'

# Unit name varies; reload keeps established sessions alive in normal operation.
if systemctl list-unit-files ssh.service --no-legend | grep -q '^ssh.service'; then
  sudo systemctl reload ssh.service
else
  sudo systemctl reload sshd.service
fi

# KEEP THE ORIGINAL SESSION OPEN. From a second terminal:
# ssh -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_devops_admin server 'id; hostname'

# Verify server logs and effective policy before closing the original session.
sudo journalctl -u ssh.service -u sshd.service --since '-5 minutes' --no-pager

If validation or the new-session test fails, restore the backup through the preserved session or console, validate again, and reload. A rollback command that has never been tested is only a hope.

7. Hands-on lab: perform a no-change SSH hardening review

This lab inventories current policy and produces recommendations. It does not edit or reload the daemon.

lab="$HOME/devops-academy/linux/chapter13/lesson05"
rm -rf "$lab"
install -d -m 700 "$lab"
cd "$lab"

{
  date --iso-8601=seconds
  printf '\n=== service files ===\n'
  systemctl list-unit-files 'ssh*.service' --no-pager 2>/dev/null || true
  printf '\n=== listeners ===\n'
  sudo ss -lntp | grep -E '(:22\b|sshd)' || true
  printf '\n=== syntax validation ===\n'
  sudo sshd -t && printf 'sshd configuration syntax: valid\n'
} > baseline.txt 2>&1

sudo sshd -T > effective.txt

grep -E '^(port|listenaddress|permitrootlogin|passwordauthentication|kbdinteractiveauthentication|pubkeyauthentication|authenticationmethods|allowusers|allowgroups|maxauthtries|logingracetime|allowtcpforwarding|allowagentforwarding|x11forwarding|permittunnel|gatewayports|permituserenvironment|disableforwarding) ' \
  effective.txt > security-settings.txt

cat > review.md <<'EOF'
# SSH hardening review

## Required access paths
- [ ] Named administrators and source networks documented
- [ ] Emergency console or out-of-band recovery tested
- [ ] Human, automation, transfer, and forwarding identities separated

## Validation before any change
- [ ] Existing session remains open
- [ ] Authoritative files and Includes identified
- [ ] Backup and exact rollback command prepared
- [ ] `sshd -t` passes
- [ ] `sshd -T -C ...` matches intended user/source policy
- [ ] Second-session login and required transfer/forwarding tests defined

## Evidence after any change
- [ ] New login succeeds with the intended identity
- [ ] Prohibited authentication and forwarding methods fail
- [ ] Journals contain expected events and no new errors
- [ ] Configuration survives reboot or automated reconciliation
EOF

sha256sum baseline.txt effective.txt security-settings.txt review.md > evidence.sha256

8. Common hardening and troubleshooting mistakes

9. Knowledge check

10. Summary

SSH hardening succeeds when it removes unnecessary capability without removing required recovery. Start from an access model, inspect effective policy, protect privileged identities, narrow authentication and forwarding, layer network controls, validate before reload, and preserve a working session. Troubleshooting should follow connection stages from name and route through host trust, negotiation, user authentication, account policy, and session setup.

11. Further reading

• OpenSSH sshd_config(5) manual.
• sshd(8) validation and effective-configuration options.
• ssh(1) verbose diagnostics.
• Your distribution’s OpenSSH packaging, SELinux/AppArmor, firewall, and logging documentation.