Linux Networking for Docker and Kubernetes

1. Container networking starts with ordinary Linux networking objects

A network namespace owns interfaces, addresses, routes, sockets, and protocol state. A virtual Ethernet pair acts like a cable with one endpoint in one namespace and the peer elsewhere. A Linux bridge can connect host-side endpoints, while routing and firewall rules move traffic beyond the host. Container engines automate this assembly.

flowchart TD
  P["Process socket in container netns"] --> E1["eth0 veth endpoint"]
  E1 --> E2["host-side veth peer"]
  E2 --> B["Linux bridge or CNI datapath"]
  B --> R["Host routing and policy"]
  R --> N["NAT or routed underlay/overlay"]
  N --> D["Destination Pod, container, service, or external host"]
  Q["DNS and service discovery"] --> P
  F["nftables, iptables, eBPF, NetworkPolicy"] --> R

Implementations may replace a bridge or iptables rules with routing, eBPF, an overlay, hardware offload, or cloud-native interfaces. Diagnose the actual datapath instead of assuming one engine’s default.

2. Enter the correct network namespace before trusting observations

ip addr, ip route, ss, and firewall commands report the namespace from which they run. A host can show no listening socket even though a container namespace has one, or show a published host socket that forwards elsewhere. Start by mapping workload identity to host PID and namespace inode.

# Host namespace inventory.
ip -brief link
ip -brief address
ip route show table all
bridge link 2>/dev/null || true
bridge vlan show 2>/dev/null || true
ss -lntup
lsns -t net

# Runtime PID lookup examples.
name=${1:-example}
pid=$(podman inspect -f '{{.State.Pid}}' "$name" 2>/dev/null ||       docker inspect -f '{{.State.Pid}}' "$name" 2>/dev/null || true)
if [[ $pid =~ ^[0-9]+$ && $pid -gt 0 ]]; then
  printf 'host_pid=%s netns=%s
' "$pid" "$(readlink /proc/$pid/ns/net)"
  sudo nsenter -t "$pid" -n -- ip -brief address
  sudo nsenter -t "$pid" -n -- ip route
  sudo nsenter -t "$pid" -n -- ss -lntup
fi

3. Bridge networking and port publication are separate operations

In a common local-engine design, a container receives an address on a private bridge. Outbound traffic may be source-NATed to the host address. Publishing a port adds a host listener or forwarding rule that maps host address and port to a container address and port. Exposing a port in image metadata does not itself publish it.

# Engine network and publication state.
podman network ls 2>/dev/null || docker network ls
podman network inspect NETWORK 2>/dev/null || docker network inspect NETWORK 2>/dev/null || true
podman port CONTAINER 2>/dev/null || docker port CONTAINER 2>/dev/null || true

# Host forwarding and firewall evidence.
sysctl net.ipv4.ip_forward net.ipv6.conf.all.forwarding
sudo nft list ruleset 2>/dev/null | sed -n '1,220p' || true
sudo iptables-save 2>/dev/null | sed -n '1,220p' || true

# Confirm which addresses accept the published port.
ss -lntp 'sport = :8080' 2>/dev/null || true

Binding to 0.0.0.0 or :: can expose a service on every applicable interface. Bind deliberately, apply host and upstream firewall policy, and test from an external vantage point.

4. DNS and MTU failures often imitate application failures

Container DNS may be supplied by an embedded engine resolver, host resolver integration, or Kubernetes DNS Service. Search domains and ndots can generate additional queries. Overlay encapsulation reduces usable payload size; path-MTU discovery failures can allow small packets while stalling TLS, registry pulls, or larger responses.

# Resolver and route evidence inside the target namespace.
cat /etc/resolv.conf
getent ahosts kubernetes.default.svc.cluster.local 2>/dev/null || true
resolvectl status 2>/dev/null || true

# Interface MTU and route selection.
ip -d link show
ip route get 1.1.1.1

# Bounded path-MTU probes; choose an authorized destination.
ping -c 3 -M do -s 1400 DESTINATION 2>/dev/null || true
tracepath DESTINATION 2>/dev/null || true

# Capture DNS only for a short interval and authorized interface.
sudo timeout 15 tcpdump -ni any -c 50 'port 53' 2>/dev/null || true

5. Kubernetes gives each Pod one shared network namespace

Every Pod receives a cluster-unique IP in the Kubernetes network model. Containers in the same Pod share the Pod network namespace and communicate over localhost. Pods should be able to reach other Pods according to the cluster network implementation and policy without application-managed host-port translation. On Linux, the runtime commonly invokes CNI plugins to configure the Pod network.

Kubernetes defines the model and APIs, while the selected network implementation supplies much of the datapath. A NetworkPolicy object has no enforcement effect unless the implementation supports it.

6. Services and EndpointSlices decouple clients from Pod churn

A Service selects or directly references backend endpoints and presents a stable virtual IP or name. The node dataplane translates or load-balances traffic to ready endpoints. Failures can occur in selector matching, readiness, EndpointSlice creation, Service port mapping, node proxy state, policy, or the backend itself.

# Read-only Kubernetes service path inspection.
kubectl get service -A -o wide
kubectl get endpointslice -A -o wide
kubectl describe service -n NAMESPACE SERVICE
kubectl get pod -n NAMESPACE -o wide --show-labels

# Test name resolution and direct endpoints from a permitted debug Pod.
kubectl exec -n NAMESPACE POD -- getent hosts SERVICE.NAMESPACE.svc.cluster.local
kubectl exec -n NAMESPACE POD -- sh -c 'ip addr; ip route; cat /etc/resolv.conf'

# Node-level proxy and networking components vary by cluster.
kubectl get daemonset -A
kubectl get pods -A -o wide | grep -Ei 'cni|calico|cilium|flannel|kube-proxy' || true

Prefer a progression: client namespace → DNS result → Service definition → EndpointSlices → direct backend test → dataplane rules or maps → backend listener and logs.

7. NetworkPolicy is allow-list policy selected by labels

By default, Pods are not isolated for ingress or egress. A Pod becomes isolated for a direction when at least one applicable policy selects it for that direction. Allowed flows are the union of applicable allow rules. Policy is additive; rule order is not a first-match firewall chain.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: frontend
      ports:
        - protocol: TCP
          port: 8443
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: data
      ports:
        - protocol: TCP
          port: 5432

Account for DNS egress, monitoring, identity, time, registries, and control-plane dependencies. Test policy from authorized source Pods and inspect implementation-specific enforcement evidence.

8. Stateful translation and connection tracking have finite capacity

Traditional NAT and many Service implementations rely on netfilter connection tracking. Exhaustion or aggressive timeouts can cause intermittent new-connection failures while established connections continue. Observe counts, limits, drops, retransmissions, socket queues, and node pressure together.

# Kernel counters and conntrack capacity.
sysctl net.netfilter.nf_conntrack_count        net.netfilter.nf_conntrack_max 2>/dev/null || true
conntrack -S 2>/dev/null || true
nstat -az 2>/dev/null | grep -E 'TcpRetransSegs|IpInDiscards|IpOutNoRoutes' || true

# Interface and qdisc drops.
ip -s link
for dev in $(ls /sys/class/net); do
  tc -s qdisc show dev "$dev" 2>/dev/null || true
done

# Socket pressure and summaries.
ss -s
cat /proc/net/sockstat
cat /proc/net/sockstat6

9. Hands-on lab: build two network namespaces and a veth link

This root-required lab creates two isolated namespaces connected by one veth pair. It adds no bridge, NAT, or external route. The cleanup trap removes all objects.

#!/usr/bin/env bash
set -Eeuo pipefail
left=da-left
right=da-right
cleanup() {
  sudo ip netns del "$left" 2>/dev/null || true
  sudo ip netns del "$right" 2>/dev/null || true
}
trap cleanup EXIT
cleanup

sudo ip netns add "$left"
sudo ip netns add "$right"
sudo ip link add veth-left type veth peer name veth-right
sudo ip link set veth-left netns "$left"
sudo ip link set veth-right netns "$right"

sudo ip -n "$left" addr add 192.0.2.1/30 dev veth-left
sudo ip -n "$right" addr add 192.0.2.2/30 dev veth-right
sudo ip -n "$left" link set lo up
sudo ip -n "$right" link set lo up
sudo ip -n "$left" link set veth-left up
sudo ip -n "$right" link set veth-right up

sudo ip -n "$left" -brief address
sudo ip -n "$right" -brief address
sudo ip netns exec "$left" ping -c 3 192.0.2.2

# Demonstrate separate socket tables.
sudo ip netns exec "$left" ss -lntup
sudo ip netns exec "$right" ss -lntup

10. A systematic container-network troubleshooting sequence

1. State source, destination, protocol, port, expected policy, and failure time.
2. Resolve both workload identities to nodes, host PIDs, namespace inodes, and IP addresses.
3. Verify listener, resolver, route, interface state, MTU, and local policy inside the source and destination namespaces.
4. Trace the host-side veth, bridge/routing/eBPF datapath, NAT, conntrack, and node firewall.
5. For Kubernetes, validate Service selectors, EndpointSlices, readiness, NetworkPolicy, and CNI component health.
6. Capture a bounded packet trace at two strategic points and compare timestamps.

11. Knowledge check

12. Summary

• Container datapaths are composed from network namespaces, interfaces, routing, and policy.
• Port publication is distinct from image metadata and internal listening.
• Kubernetes Pods share one network namespace; CNI implements the cluster datapath.
• Services, EndpointSlices, DNS, NetworkPolicy, MTU, and conntrack require separate evidence.
• Effective diagnosis begins in the correct namespace and follows the actual packet path.

13. Further reading

• ip-netns(8)
• Linux kernel — Ethernet bridge
• Kubernetes cluster networking
• Kubernetes NetworkPolicy
• Container Network Interface specification