Calico eBPF Source IP Preservation: The Unexpected Story of High Tail Latency

The Calico eBPF data plane is your choice if latency is your primary concern. It was very disturbing that some benchmarking brought to our attention that eBPF had higher tail latency than iptables. The 99+% percentiles were higher by as much as a few hundred milliseconds. We did a whole bunch of experiments and we could not crack the nut until we observed that there are some occasional and unexpected TCP reset (RST) packets, but no connections were reset.

We noticed that the RST belongs to a connection that was already completed and finished a while ago. That was strange, but it pointed us in the right direction. We also knew that this happens only if the benchmark uses a LoadBalancer and we have connections going through multiple nodes to the same backend pod. That was enough to get to the root cause, but let’s start at the beginning…

External Clients and Kubernetes Services

One of the shortcomings of iptables/nftables based networking in Kubernetes is that if an external client connects to your cluster via a NodePort or a LoadBalancer, you may lose its IP address along the way. The reason for this is that the client may connect to Continue reading

Introducing Low-Latency DNS Policy with eBPF in Calico Enterprise

In Kubernetes, pods often need to securely communicate with external resources, such as internet services or APIs. Traditional Kubernetes network policies use IP addresses to identify these external resources. However, managing policies with IP addresses can be challenging because IPs often change, especially when dealing with dynamic websites or APIs.

Calico Enterprise addresses this challenge by extending Kubernetes network policies to support Fully Qualified Domain Names (FQDNs). This allows users to define policies using domain names instead of IP addresses, making it easier to manage and secure egress traffic. By dynamically mapping domain names to IPs, Calico ensures that policies remain up-to-date, enabling seamless and secure connectivity to external resources.

While this approach is conceptually simple, practical implementation is tricky. DNS mappings are dynamic: domain names often resolve to different IPs with each query, and wildcard support (e.g., *.example.com) adds complexity. To address this, Calico monitors DNS traffic to create and manage domain-to-IP mappings dynamically, translating high-level DNS-based rules into efficient low-level constructs like iptables, nftables, or eBPF.

Evolution of Calico DNS policy implementation

The DNS policy implementation significantly impacts performance and reliability. Currently, Calico offers three different modes to operate the DNS Continue reading

eBPF: Enabling Security and Performance to Co-Exist

Today, most organizations and individuals use Linux and the Linux kernel with a “one-size-fits-all” approach. This differs from how Linux was used in the past–for example, 20 years ago, many users would compile their kernel and modify it to fit their specific needs, architectures and use cases. This is no longer the case, as one-size-fits-all has become good enough. But, like anything in life, “good enough” is not the best you can get.

Enter: Extended Berkeley Packet Filter (eBPF). eBPF allows users to modify one-size-fits-all to fit their specific needs. While this was not impossible before, it was cumbersome and often unsecure.

eBPF is a feature available in Linux kernels that allows users to safely load programs into the kernel, to customize its operation. With eBPF, the kernel and its behavior become highly customizable, instead of being fixed.

Utilizing eBPF, users can load a program into the kernel and instruct the kernel to execute their program if, for example, a certain packet is seen or another event occurs. eBPF lets programs run without needing to add additional modules or modify the kernel source code. Users can think of it as a lightweight, sandboxed virtual machine (VM) within the Linux kernel Continue reading