Calico eBPF Source IP Preservation: The Unexpected Story of High Tail Latency
The Calico eBPF data plane is your choice if latency is your primary concern. It was very disturbing that some benchmarking brought to our attention that eBPF had higher tail latency than iptables. The 99+% percentiles were higher by as much as a few hundred milliseconds. We did a whole bunch of experiments and we could not crack the nut until we observed that there are some occasional and unexpected TCP reset (RST) packets, but no connections were reset.
We noticed that the RST belongs to a connection that was already completed and finished a while ago. That was strange, but it pointed us in the right direction. We also knew that this happens only if the benchmark uses a LoadBalancer and we have connections going through multiple nodes to the same backend pod. That was enough to get to the root cause, but let’s start at the beginning…
External Clients and Kubernetes Services
One of the shortcomings of iptables/nftables based networking in Kubernetes is that if an external client connects to your cluster via a NodePort or a LoadBalancer, you may lose its IP address along the way. The reason for this is that the client may connect to Continue reading




