Network and DNS Incident Scenarios

1. A connection is a path, not a single host setting

A client request depends on name resolution, source address selection, routing, neighbor discovery, firewall and NAT policy, transport handshake, server listener, TLS validation, application protocol, and downstream dependencies. “Ping works” confirms only a subset of that path.

flowchart TD
  A["Application request"] --> D["DNS and address selection"]
  D --> R["Source address and route"]
  R --> L["Link, neighbor, MTU"]
  L --> F["Firewall, NAT, policy"]
  F --> T["TCP or UDP transport"]
  T --> S["Server listener"]
  S --> C["TLS and protocol"]
  C --> U["Upstream dependency"]

State the exact source namespace, destination name and address, protocol, port, timestamp, expected policy, and observed error. A test from the wrong host or namespace can falsely prove reachability.

2. Start with local interface, address, route, and socket state

Use concise views first, then expand the relevant object. Confirm that the intended interface is operational, the address scope is correct, and the selected route matches the expected source.

ip -brief link
ip -brief address
ip route show table main
ip -6 route show table main
ip rule show

# Ask the kernel how it would route one destination.
ip route get 203.0.113.10
ip -6 route get 2001:db8::10 2>/dev/null || true

# List listening and established sockets without service-name ambiguity.
ss -lntup
ss -tan state established
ss -s

# Map a known port to process and local address.
ss -lntp 'sport = :443'
ss -lunp 'sport = :53' 

A service bound to 127.0.0.1 is not reachable through the host’s external address. A service bound to 0.0.0.0 may still be blocked by firewall, namespace, policy, or application-level authorization.

3. Route and neighbor failures produce different evidence

The routing table selects a next hop and source; ARP for IPv4 or Neighbor Discovery for IPv6 resolves the link-layer neighbor. A correct route with an incomplete neighbor entry points toward local link, VLAN, address duplication, or neighbor-discovery problems.

dst=192.0.2.25
ip route get "$dst"
ip neigh show
ip neigh get "$dst" 2>/dev/null || true

# Monitor changes during a short reproduction window.
timeout 20s ip monitor link address route neigh

# Counters expose drops, errors, and overruns.
ip -s link show
ethtool -S eth0 2>/dev/null | grep -Ei 'drop|error|miss|timeout' | head -n 50 || true

# Policy routing and multiple tables can override the main table.
ip rule show
ip route show table all

Do not flush neighbor tables or routes as a first step. Capture the failing state, compare a healthy peer, and identify who owns configuration: NetworkManager, systemd-networkd, Netplan, cloud-init, DHCP, a CNI, or automation.

4. Interpret transport errors precisely

Connection refused usually means the destination responded but no listener accepted the port, or policy actively rejected it. A timeout can mean packet loss, silent filtering, no route, dead destination, asymmetric return path, overloaded state tracking, or an application that accepts but never responds.

host=example.com
port=443

# Bounded TCP and HTTP tests.
timeout 5 bash -c "</dev/tcp/$host/$port" && echo connected
curl --connect-timeout 3 --max-time 10 -sv "https://$host/" -o /dev/null

# TLS handshake, certificate chain, SNI, and negotiated protocol.
timeout 10 openssl s_client -connect "$host:$port" -servername "$host"   -brief </dev/null

# Trace path where policy permits. TCP mode may match the application path better.
traceroute "$host" 2>/dev/null || true
tracepath "$host" 2>/dev/null || true

5. DNS incidents require the resolver path and the queried name

A Linux application may use glibc NSS, systemd-resolved, a local caching daemon, container runtime DNS, a VPN resolver, or direct DNS libraries. Search domains and ndots can transform a short name into several queries. Split DNS can return different answers per interface or namespace.

# Application-facing resolver behavior.
getent ahosts example.com
getent hosts example.com
cat /etc/nsswitch.conf
ls -l /etc/resolv.conf
cat /etc/resolv.conf

# systemd-resolved state where present.
resolvectl status 2>/dev/null || true
resolvectl query example.com 2>/dev/null || true
resolvectl statistics 2>/dev/null || true

# Direct DNS queries expose record, server, flags, and timing.
dig example.com A
dig example.com AAAA
dig +trace example.com 2>/dev/null | tail -n 40
dig @1.1.1.1 example.com A  # only when external resolvers are permitted

# Compare search-domain expansion explicitly.
dig service.example.internal A
dig service A

Do not replace /etc/resolv.conf blindly. On many systems it is a managed symlink. Identify the owner and per-link resolver state before editing NetworkManager, systemd-resolved, Netplan, DHCP, VPN, or container configuration.

6. Separate authoritative, recursive, cache, and client failures

NXDOMAIN means the responding resolver asserts that the name does not exist in the relevant view. SERVFAIL indicates the resolver could not complete validation or resolution. A timeout means no usable response arrived. Stale answers may come from positive or negative caches.

name=api.example.internal
printf 'Application view:
'
timeout 5 getent ahosts "$name" || true
printf '
Resolved view:
'
timeout 5 resolvectl query "$name" 2>/dev/null || true
printf '
Configured server view:
'
server=$(awk '/^nameserver/{print $2; exit}' /etc/resolv.conf)
[[ -n $server ]] && timeout 5 dig "@$server" "$name" A +noall +answer +authority +comments

# Flush only after capturing evidence and confirming cache ownership.
# resolvectl flush-caches

When only containers fail, run the same queries inside the workload namespace and inspect its /etc/resolv.conf, search domains, DNS service address, network policy, and node path.

7. Firewall diagnosis needs the effective ruleset and counters

nftables rules can be spread across tables, chains, priorities, hooks, sets, maps, and included files. Container engines and orchestrators may create their own policy. Inspect the effective ruleset before editing a source file.

# Requires appropriate privilege and authorization.
sudo nft list ruleset
sudo nft --handle list ruleset
sudo nft list counters 2>/dev/null || true

# Validate a candidate ruleset without applying it.
sudo nft --check --file /etc/nftables.conf

# Trace one packet only in a controlled maintenance window; tracing can be noisy.
# sudo nft add rule inet filter input meta nftrace set 1
# sudo nft monitor trace

# Legacy compatibility views may not show all native nftables semantics.
sudo iptables-save 2>/dev/null | head -n 100 || true

Record rule handle, chain, hook, priority, counter delta, and the exact packet tuple. Avoid “temporarily disabling the firewall” on a production host; use a narrow logged rule with expiration and rollback.

8. MTU and path-MTU failures often affect large or encrypted traffic only

A small ping may work while TLS handshakes, uploads, tunnels, or replication stall. Encapsulation reduces usable payload MTU. Blocked ICMP “packet too big” messages can break path-MTU discovery.

ip link show
ip route get 203.0.113.10
tracepath 203.0.113.10

# IPv4 do-not-fragment probe; adjust size and target to the approved path.
ping -M do -s 1472 -c 3 203.0.113.10
ping -M do -s 1400 -c 3 203.0.113.10

# TCP state can expose retransmissions and MSS information.
ss -ti dst 203.0.113.10

Do not lower every interface MTU permanently from one symptom. Identify the encapsulation boundary, confirm packet evidence, and set the MTU in the system that owns the interface or overlay.

9. Packet capture answers where traffic stops

Capture only the necessary interfaces, hosts, ports, directions, duration, and bytes. Payloads may contain credentials or personal data; prefer headers and metadata when sufficient.

# Capture 200 packets or 30 seconds, whichever occurs first.
sudo timeout 30 tcpdump -ni any -c 200 -s 128   'host 203.0.113.10 and tcp port 443'   -w /var/tmp/inc-2041-https.pcap
sudo chmod 0600 /var/tmp/inc-2041-https.pcap
sha256sum /var/tmp/inc-2041-https.pcap

# Human-readable bounded view without DNS or service-name resolution.
sudo tcpdump -nn -r /var/tmp/inc-2041-https.pcap -c 50

# Examples of interpretation:
# SYN repeated, no SYN-ACK: loss/filter/no listener path or return-path problem.
# SYN, SYN-ACK, ACK then RST: application or local policy closes connection.
# Handshake succeeds, encrypted pause: inspect TLS/app timing and dependencies.

10. Scenario matrix: use the symptom to pick the next test

11. Hands-on lab: separate listener and DNS behavior

This local-only lab starts an HTTP server on loopback, verifies the listener, tests the literal address and hostname path, and captures no external traffic.

set -Eeuo pipefail
lab="$HOME/devops-academy/linux/chapter20/lesson03/lab"
rm -rf "$lab"
mkdir -p "$lab/www"
printf 'network lab ok
' > "$lab/www/index.html"

python3 -m http.server 18080 --bind 127.0.0.1 --directory "$lab/www"   >"$lab/server.log" 2>&1 &
pid=$!
trap 'kill "$pid" 2>/dev/null || true; wait "$pid" 2>/dev/null || true; rm -rf "$lab"' EXIT
sleep 1

ss -lntp 'sport = :18080' || true
ip route get 127.0.0.1
getent hosts localhost
curl --connect-timeout 2 --max-time 5 -v http://127.0.0.1:18080/
curl --connect-timeout 2 --max-time 5 -v http://localhost:18080/

# Demonstrate a closed port without changing the server.
curl --connect-timeout 2 --max-time 3 -v http://127.0.0.1:18081/ || true

kill "$pid"
wait "$pid" 2>/dev/null || true
trap - EXIT
rm -rf "$lab"

12. Network and DNS incident runbook

1. Record source namespace, destination name/address, protocol, port, error, timestamp, and expected policy.
2. Resolve the name through the application path and directly through the configured resolver.
3. Verify link, address, route selection, neighbor state, and interface counters.
4. Verify the server listener address, owning process, unit identity, and local reachability.
5. Inspect effective firewall/NAT policy and counters on each relevant hop.
6. Test transport, TLS, and application protocol separately with bounded timeouts.
7. Capture a narrow packet trace at two strategic points when earlier evidence cannot localize loss.
8. After repair, verify both directions, all address families, monitoring, and configuration ownership.

13. Knowledge check

14. Summary

• Connectivity spans DNS, route, link, policy, transport, listener, TLS, application, and dependencies.
• Use the exact source namespace and kernel route selection for each test.
• Distinguish authoritative DNS, recursive resolver, local stub, cache, and application behavior.
• Use ruleset counters and narrow packet captures to localize packet loss.
• MTU, address-family, loopback binding, and asymmetric routing create selective failures.

15. Further reading

• ip-route manual
• ss socket-statistics manual
• resolvectl manual
• systemd-resolved manual
• nftables project documentation