Network Throughput, Connections, and Packet Inspection

1. Network performance spans applications, sockets, interfaces, and paths

An application may wait before it reaches the network: DNS, connection pools, TLS, server queues, and application locks all contribute. The kernel then manages socket buffers, congestion control, routing, queuing, and interface transmission. Switches, firewalls, tunnels, load balancers, remote servers, and return paths add further behavior. Host counters only describe the host’s view.

flowchart TD
  A["Application and DNS"] --> S["Socket queues and TCP/UDP"]
  S --> R["Routing, firewall, qdisc"]
  R --> I["Interface and driver"]
  I --> N["Network path"]
  N --> P["Peer service"]
  P --> O["Response latency and throughput"]

Define the direction and endpoint before collecting data: client to server, server to dependency, ingress, egress, one interface, one namespace, or one container. A host with several interfaces may route traffic differently than expected.

ip -brief address
ip route show
ip rule show

# Interface counters, drops, errors, and queue statistics.
ip -s link

# Route decision for a concrete destination without sending traffic.
ip route get 192.0.2.10

2. Throughput is only one network signal

Bytes per second describe volume; packets per second describe packet-processing demand. Latency describes completion time. Loss and retransmission indicate that work must be repeated. Drops can occur at interfaces, socket queues, firewalls, or remote devices. Connection rate and concurrent connections affect state tables and service accept queues.

# Protocol summary and socket counts.
ss -s

# Listening TCP and UDP sockets with owning processes when permitted.
ss -lntup

# Established TCP sockets, timers, queues, and process details.
ss -tanpo state established

# Kernel protocol counters; compare two samples rather than reading totals alone.
nstat -az | grep -E 'TcpRetransSegs|TcpExtListenDrops|TcpExtTCPTimeouts|IpInDiscards|IpOutDiscards' || true

Cumulative counters require deltas. A million retransmissions since boot may be harmless on a long-lived, high-volume host; a sharp increase during one incident may be decisive. Record timestamps and compute rates over the same interval as application symptoms.

3. Socket state and queues reveal connection pressure

ss exposes local and peer addresses, TCP state, receive queue, send queue, timers, and—when privileges permit—process ownership. Large persistent queues can indicate an application that is not reading, a peer that is not receiving, congestion, or a stalled service. Listening sockets can also experience accept-queue pressure.

# Count TCP states.
ss -tanH | awk '{count[$1]++} END {for (state in count) print state, count[state]}' | sort

# Find listening sockets with nonzero queued connections.
ss -lntH | awk '$2 != 0 {print}'

# Inspect sockets for one destination port.
ss -tanpo '( dport = :443 or sport = :443 )'

# Show TCP-level details such as congestion algorithm and RTT where available.
ss -tin state established | head -n 80

SYN-SENT accumulation can indicate unreachable peers or filtered traffic. SYN-RECV growth can reflect a surge or incomplete handshakes. Many TIME-WAIT sockets may be normal for short-lived outbound connections, but they can expose poor connection reuse or ephemeral-port pressure. CLOSE-WAIT accumulation often points to an application that did not close after the peer ended the connection.

4. Interface counters need driver and namespace context

RX/TX errors, dropped packets, overruns, missed packets, and carrier events can point toward physical, driver, queue, or virtualization issues. In containers, the relevant counters may live in a network namespace or on a veth peer. Offload features can also make packet counts and capture sizes look surprising.

interface=$(ip route show default | awk 'NR==1 {print $5}')
if [[ -n $interface ]]; then
  ip -s link show dev "$interface"
  if command -v ethtool >/dev/null; then
    ethtool "$interface" 2>/dev/null || true
    ethtool -S "$interface" 2>/dev/null | head -n 80 || true
  fi
fi

# Network namespaces visible to the current user.
ip netns list 2>/dev/null || true

Do not reset counters during an incident unless your operating procedure explicitly requires it; cumulative evidence can be valuable. Instead, take two timestamped samples and compute differences.

5. Packet inspection is powerful, privileged, and sensitive

A packet capture can verify whether requests leave, replies return, retransmissions occur, resets are sent, or DNS answers differ. It can also expose credentials, tokens, hostnames, payloads, and customer data. Obtain authorization, capture only the necessary interface and filter, limit duration and file size, store securely, and delete according to policy.

# Capture only localhost TCP port 8000, stop after 40 packets.
sudo tcpdump -i lo -nn -s 128 -c 40 'tcp port 8000' -w localhost-8000.pcap

# Read metadata without resolving names.
tcpdump -nn -tttt -r localhost-8000.pcap

# Bounded rotating capture: 10 MiB per file, keep three files.
sudo tcpdump -i eth0 -nn -s 128 -C 10 -W 3 \
  -w incident.pcap 'host 192.0.2.10 and tcp port 443' 

-nn avoids name and service resolution that can add traffic or obscure numeric evidence. A reduced snap length limits payload collection, but choose it carefully because headers from encapsulation and options still require space. Capture filters reduce data before it is written; display filters in analysis tools operate after capture.

6. Active throughput tests require controlled endpoints

iperf3 can measure TCP or UDP performance, but it generates load and must target a server you control. Results reflect the test path and parameters, not necessarily application behavior. Run active tests in approved windows and avoid public servers unless explicitly authorized.

# On a controlled lab server:
iperf3 --server --one-off

# On the controlled client:
iperf3 --client 192.0.2.20 --time 10 --parallel 1

# Reverse direction from server to client.
iperf3 --client 192.0.2.20 --reverse --time 10

# JSON output for later comparison.
iperf3 --client 192.0.2.20 --time 10 --json > iperf3-result.json

Compare test throughput with interface speed, RTT, retransmissions, CPU use, and socket windows. Multiple parallel streams can hide single-flow limitations, while one stream may underfill a high-bandwidth, high-latency path.

7. Hands-on lab: inspect a localhost HTTP exchange

The lab starts a Python HTTP server bound only to loopback, records socket state, makes a request, and optionally captures the exchange when tcpdump and sudo permission are available.

lab="$HOME/devops-academy/linux/chapter16/lesson03"
rm -rf "$lab"
mkdir -p "$lab/www"
printf 'network-lab\n' > "$lab/www/index.txt"
cd "$lab"

python3 -m http.server 8000 --bind 127.0.0.1 --directory "$lab/www" \
  > server.log 2>&1 &
server_pid=$!
cleanup() {
  kill "$server_pid" 2>/dev/null || true
  wait "$server_pid" 2>/dev/null || true
}
trap cleanup EXIT
sleep 1

ss -lntp '( sport = :8000 )' > listening.txt 2>&1 || true
curl --fail --silent --show-error http://127.0.0.1:8000/index.txt > response.txt
ss -tanpo '( dport = :8000 or sport = :8000 )' > sockets.txt 2>&1 || true

printf '%s\n' '--- response ---'
cat response.txt
printf '%s\n' '--- listening socket ---'
cat listening.txt
printf '%s\n' '--- observed sockets ---'
cat sockets.txt

8. Common network mistakes

9. Knowledge check

10. Summary

• Network performance includes application waits, socket queues, interfaces, paths, and peers.
• Use throughput, packet rate, latency, loss, retransmission, queue, and error evidence together.
• ss, ip, and protocol counters reveal connection and host behavior.
• Active tests generate load and must use controlled endpoints.
• Packet captures require narrow filters and data-governance discipline.

11. Further reading

• ss(8)
• ip(8)
• tcpdump(1)
• iperf3 invocation documentation
• tcp(7)