How 1&1 Mail & Media Scaled Kubernetes Networking with eBPF and Calico

“We started in 2017 with Calico and never regretted it!”
—Stefan Fudeus, Product Owner/Lead Architect, 1&1 Mail & Media

Challenge

1&1 Mail & Media, part of the IONOS group, powers popular European internet brands including GMX and Web.de, serving more than 50% of Germany’s population with critical identity and email infrastructure. With roughly 45 to 50 million users, network reliability is non-negotiable. Any downtime could affect millions.

By 2022, the company had containerized 80% of its workloads on Kubernetes across three self-managed data centers. While the platform, backed by bare metal nodes and custom network layers, was highly scalable, network throughput bottlenecks began to emerge. Pods were limited to 2.5 Gbps of bandwidth due to IP encapsulation overhead, despite 10 Gbps network interfaces.

The team needed a solution that:

• Improved pod-to-pod network performance
• Maintained strong network policy isolation across up to 40 tenants per cluster
• Scaled to millions of network connections and 1.4 million HTTP requests per second

Solution

1&1 Mail & Media had adopted Calico back in 2017, largely for its unique Kubernetes NetworkPolicy standard support. As their Kubernetes platform evolved, with clusters scaling to 300 bare metal nodes, 16,000 pods, and over 4 million Continue reading

Top 5 Kubernetes Network Issues You Can Catch Early with Calico Whisker

Kubernetes networking is deceptively simple on the surface, until it breaks, silently leaks data, or opens the door to a full-cluster compromise. As modern workloads become more distributed and ephemeral, traditional logging and metrics just can’t keep up with the complexity of cloud-native traffic flows.

That’s where Calico Whisker comes in. Whisker is a lightweight Kubernetes-native observability tool created by Tigera. It offers deep insights into real-time traffic flow patterns, without requiring you to deploy heavyweight service meshes or packet sniffer. And here’s something you won’t get anywhere else: Whisker is data plane-agnostic. Whether you run Calico eBPF data plane, nftables, or iptables, you’ll get the same high-fidelity flow logs with consistent fields, format, and visibility. You don’t have to change your data plane, Whisker fits right in and shows you the truth, everywhere.

Let’s walk through 5 network issues Whisker helps you catch early, before they turn into outages or security incidents.

1. Policy Misconfigurations

Traditional observability tools often show whether a packet was forwarded, accepted or dropped, but not why. They lack visibility into which Kubernetes network policy was responsible or if one was even applied.

With Whisker, each network flow is paired with:

• The enforced policy name Continue reading

Kubernetes Is Powerful, But Not Secure (at least not by default)

Kubernetes has transformed how we deploy and manage applications. It gives us the ability to spin up a virtual data center in minutes, scaling infrastructure with ease. But with great power comes great complexities, and in the case of Kubernetes, that complexity is security.

By default, Kubernetes permits all traffic between workloads in a cluster. This “allow by default” stance is convenient during development, and testing but it’s dangerous in production. It’s up to DevOps, DevSecOps, and cloud platform teams to lock things down.

To improve the security posture of a Kubernetes cluster, we can use microsegmentation, a practice that limits each workload’s network reach so it can only talk to the specific resources it needs. This is an essential security method in today’s cloud-native environments.

Why Is Microsegmentation So Hard?

We all understand that network policies can achieve microsegmentation; or in other words, it can divide our Kubernetes network model into isolated pieces. This is important since Kubernetes is usually used to provide multiple teams with their infrastructural needs or host multiple workloads for different tenants. With that, you would think network policies are first citizens of clusters. However, when we dig into implementing them, three operational challenges Continue reading

Dry Run: Your Kubernetes network policies with Calico staged network policies

Kubernetes Network Policies (KNP) are powerful resources that help secure and isolate workloads in a cluster. By defining what traffic is allowed to and from specific pods, KNPs provide the foundation for zero-trust networking and least-privilege access in cloud-native environments.

But there’s a problem: KNPs are risky, and applying them without a clear game plan can be potentially disruptive.

Without deep insight into existing traffic flows, applying a restrictive policy can instantly break connectivity killing live workloads, user sessions, or critical app dependencies. An even scarier scenario is when we implement policies that we think cover everything and workloads actually work, but after a restart or scaling operation we hit new problems. Kubernetes, with all of its features, has no built-in “dry run” mode for policies, and no first-class observability to show what would be blocked or allowed which is the right decision since Kubernetes is an orchestrator not an implementer.

This forces platform teams into a difficult choice, deploy permissive or no policies and weaken security, or Risk service disruption while debugging restrictive ones. As a result, many teams delay implementing network policies entirely only to regret it after a zero-day exploit like Log4Shell, XZ backdoor, or other vulnerabilities Continue reading

Calico Whisker & Staged Network Policies: Secure Kubernetes Workloads Without Downtime

Rolling out network policies in a live Kubernetes cluster can feel like swapping wings mid-flight—one typo or overly broad rule and critical traffic is grounded. Calico’s Staged Network Policies remove the turbulence by letting you deploy policies in staged mode, so you can observe their impact before enforcing anything. Add Whisker, the open-source policy enforcement and testing tool (introduced as part of Calico Open Source 3.30) that captures every flow and tags it with a policy verdict, and you’ve got a safety harness that proves your change is sound long before you flip the switch. In this post, we’ll walk you through how you can leverage these capabilities to tighten security, validate intent, and ship changes confidently—without a single packet of downtime.

Deploying a Kubernetes Cluster

Calico for Policy is a CNI agnostic tool. Refer to the Calico Open Source docs for a list of supported CNIs. The git repository for this blog post can be found here.

For this post, let’s deploy a simple AKS cluster with Azure CNI.

## Configure 
az group create --name calicooss --location eastus2

## Create a 3 node AKS cluster with Azure CNI
az aks create \
  --resource-group calicooss \
  --name  Continue reading

A Detailed Look at Calico Cloud Free Tier

Why Calico Cloud Free Tier?

As Kubernetes environments grow in scale and complexity, platform teams face increasing pressure to secure workloads without slowing down application delivery. But managing and enforcing network policies in Kubernetes is notoriously difficult—especially when visibility into pod-to-pod communication is limited or nonexistent. Teams are often forced to rely on manual traffic inspection, standalone logs, or trial-and-error policy changes, increasing the risk of misconfiguration and service disruption. Safe policy management and microsegmentation becomes a daunting task without clear knowledge or insight into which services should communicate with each other.

In this detailed look, we’ll explore how Calico Cloud Free Tier builds upon Calico Open Source, and helps platform teams visualize traffic with a dynamic service graph, simplifies policy management, and even analyzes actual traffic to recommend policies.

What is Calico Cloud Free Tier?

Calico Cloud Free Tier is a managed SaaS, no-cost offering that extends the capabilities of Calico Open Source 3.30 and higher to help Kubernetes teams improve network visibility, simplify policy management, and improve security by simplifying microsegmentation. Designed for single-cluster environments, it provides platform engineers and operators with powerful observability and policy management tools. With a seamless onboarding experience for users already Continue reading

Switching to eBPF One Step at a Time with Calico DNS Inline Policy

Calico Enterprise lets users write network policies using domain names instead of IP addresses. This is done by dynamically mapping domain names to IP addresses and matching the egress traffic against these IPs. We have discussed this feature in detail when we introduced the Inline mode for the eBPF data plane in Calico Enterprise 3.20 release! It addresses the latency and performance issues of the various modes used by Calico in iptables/nftables data planes. It is a shame that Calico users who are not yet ready to switch completely to eBPF would miss out on this big DNS policy improvement. Don’t worry! We found a way to port it to iptables to enhance our users’ experience without forcing users to make a huge leap.

In Calico Enterprise v3.21, we have extended the Inline DNS policy mode to iptables. In this mode, DNS policies are updated in real time as DNS responses are parsed by eBPF within the data plane, thus improving the performance.

Calico iptables – DNS Inline policy

In all the existing modes in the iptables data plane, the DNS response packets are sent to Felix – Calico’s userspace agent. It parses the packets and updates the Continue reading

Securing Kubernetes Traffic with Calico Ingress Gateway

If you’ve managed traffic in Kubernetes, you’ve likely navigated the world of Ingress controllers. For years, Ingress has been the standard way of getting our HTTP/S services exposed. But let’s be honest, it often felt like a compromise. We wrestled with controller-specific annotations to unlock critical features, blurred the lines between infrastructure and application concerns, and sometimes wished for richer protocol support or a more standardized approach. This “pile of vendor annotations,” while functional, highlighted the limitations of a standard that struggled to keep pace with the complex demands of modern, multi-team environments and even led to security vulnerabilities.

Wait a second, is this the ‘Ingress vs. Gateway API’ debate?

Yes, and it’s a crucial one. The Kubernetes Gateway API isn’t just an Ingress v2; it’s a fundamental redesign, the “future” of Kubernetes ingress, built by the community to address these very challenges head-on.

What makes Gateway API different?

There are three main points that I came across while evaluating GatewayAPI and Ingress controllers:

1. Standardization & Portability: It aims to provide a core, standard way to manage ingress, reducing reliance on vendor-specific hacks and making it easier to switch implementations – change the class, and it should “just work.”
2. Continue reading

Secure and Scalable Kubernetes for Multi-Cluster Management

This story is becoming more and more common in the Kubernetes world. What starts as a manageable cluster or two can quickly balloon into a sprawling, multi-cluster architecture spanning public clouds, private data centers, or a bit of both. And with that growth comes a whole new set of headaches. How do you keep tabs on compliance across wildly different configurations? When a service goes down across multiple clusters, how do you pinpoint the cause amidst the chaos? And what about those hard-to-diagnose latency issues that seem to crop up between regions?

The truth is, achieving secure and scalable multi-cluster Kubernetes isn’t about throwing more tools at the problem. It’s about having the right tools and adopting the right best practices. This is where a solution like Calico Cluster Mesh shines, offering those essential capabilities for a seamless multi-cluster experience without the complexity or overhead that you expect with traditional service meshes.

The Multi-Cluster Challenge: When Complexity Takes Over

So, why are so many organizations finding themselves in this multi-cluster maze? Often, it’s driven by solid business reasons:

• High Availability and Disaster Recovery: Spreading workloads across multiple regions or clusters means that if one goes down, your users shouldn’t notice.
• Continue reading

Is It Time to Migrate? A Practical Look at Kubernetes Ingress vs. Gateway API

If you’ve managed traffic in Kubernetes, you’ve likely navigated the world of Ingress controllers. For years, Ingress has been the standard way of getting HTTP/S services exposed. But let’s be honest, it often felt like a compromise. We wrestled with controller-specific annotations to unlock critical features, blurred the lines between infrastructure and application concerns, this complexity didn’t just make portability more difficult, it sometimes led to security vulnerabilities and other complications.

As part of Calico Open Source v3.30, we have released a free and open source Calico Ingress Gateway that implements a custom built Envoy proxy with the Kubernetes Gateway API standard to help you navigate Ingress complexities with style. This blog post is designed to get you up to speed on why such a change might be the missing link in your environment.

The Situation: The Ingress Rut

The challenge with traditional Ingress wasn’t a lack of effort, since the landscape is full of innovative solutions. However, the problem was the lack of a unified, expressive, and role-aware standard. Existing ingress controllers were capable, implemented advanced features, however at the same time tied you to a specific project/vendor.

This meant:

• Vendor Lock-In: Migrating from one ingress controller Continue reading

Why we need a unified approach to Kubernetes environments

Today, organizations struggle managing disparate technologies for their Kubernetes networking and network security needs. Leveraging multiple technologies for networking and security for in-cluster, ingress, egress, and traffic across clusters creates challenges, including operational complexities and increased costs. For example, to manage ingress traffic for Kubernetes clusters, users cobble together multiple solutions from different providers such as ingress controllers or gateways and load balancers for routing traffic, as well as Web Application Firewalls (WAFs) for enhanced security.

Despite the challenges it brings, deploying disparate technologies has been a “necessary evil” for organizations to get all the capabilities needed for holistic Kubernetes networking. Here, we’ll explore challenges this proliferation of tooling introduces, and provide actionable tips for today’s platform and security teams to overcome these issues.

Challenges Managing Multiple Technologies

The fragmented approach to networking and network security in Kubernetes leads to challenges and inefficiencies, including:

• Operational overhead: Each technology comes with its own learning curve, setup, configuration, integration, and maintenance requirements. This leads to a challenging user experience.
• Increased costs: Licensing and operational costs accumulate as more tools are deployed.
• Scaling challenges: As clusters grow or spread across diverse environments, ensuring consistent and secure networking becomes harder.
• Security gaps: Disjointed solutions Continue reading

What’s New in Calico: Spring 2025

Introducing Calico Cloud Free Tier

Calico provides a unified platform for all your Kubernetes networking, network security, and observability requirements. From ingress/egress management and east-west policy enforcement to multi-cluster connectivity, Calico delivers comprehensive capabilities. It is distribution-agnostic, preventing vendor lock-in and offering a consistent experience across popular Kubernetes distributions and managed services. Calico eliminates silos, providing seamless networking and observability for containers, VMs, and bare metal servers, and extends effortlessly to multi-cluster environments, in the cloud, on-premises, and at the edge.

With the recent release of Calico Open Source 3.30, we added:

• Improved observability to visualize and troubleshoot workload communication with Calico Whisker and the Goldmane API.
• Kubernetes Network Policies are critical for preventing ransomware, achieving microsegmentation to isolate sensitive assets for compliance, and thwarting attacks from malicious actors. However, implementing them effectively can be challenging due to the complexity of identifying, testing, and rapidly updating policies to meet evolving threats. Calico Open Source 3.30 introduces staged policies to enable teams to audit and validate policies before they are enforced, reducing the risk of misconfigured policies and improving security and compliance.
• The ability to manage Kubernetes ingress traffic with Calico Ingress Gateway, a 100% upstream, enterprise-ready implementation Continue reading

Recap: KubeCon + CloudNativeCon Europe 2025

When I got the assignment to attend KubeCon 1st of April I thought it was an April prank, but as the date got closer I realized—this is for real and I’ll be on the ground in London at the tenth anniversary of cloud native computing. I’ve seen a lot of tech events during my years in the industry while trying not to get replaced by AI and I have to say this one stands out!

Image source: CNCF YouTube Channel

Here is my recap of KubeCon + CloudNativeCon Europe 2025.

CalicoCon 2025

CalicoCon is an event that happens twice every year, as a co-located event during KubeCon NA and EU. It’s a free event that allows you to learn about Tigera’s vision for the future of networking and security in the cloud. There’s also an after-party to celebrate our community and people like you who are on this journey with us!

This year our main focus was on Calico v3.30, our upcoming release that will add a lot of anticipated features to Calico, unlocking things like observability, staged network policy, and gateway api. CalicoCon brought together cloud-native enthusiasts to explore the latest advancements in Calico and Kubernetes networking.

Continue reading

How to get started with Calico Observability features

Kubernetes, by default, adopts a permissive networking model where all pods can freely communicate unless explicitly restricted using network policies. While this simplifies application deployment, it introduces significant security risks. Unrestricted network traffic allows workloads to interact with unauthorized destinations, increasing the potential for cyberattacks such as Remote Code Execution (RCE), DNS spoofing, and privilege escalation.

To better understand these problems, let’s examine a sample Kubernetes application: ANP Demo App.

This application comprises a deployment that spawns pods and a service that exposes them to external users in a similar situation like any real word workload which you will encounter in your environment.

If you open the application service before implementing any policies, the application reports the following messages:

1. Container can reach the Internet – Without network policies, an attacker can use our container as an entry point by exploiting it with a vulnerability. This could allow them to exfiltrate data or establish remote control over the workload by leveraging its Internet access.
2. Container can reach CoreDNS Pods – Kubernetes relies heavily on DNS, with records served using CoreDNS Pods. While communication between your Pods and CoreDNS is essential and not inherently a vulnerability, pairing it with unrestricted access to Continue reading

Calico Open Source 3.30: Exploring the Goldmane API for custom Kubernetes Network Observability

Kubernetes is built on the foundation of APIs and abstraction, and Calico leverages its extensibility to deliver network security and observability in both its commercial and open source versions. APIs are the special sauce that help automate and operationalize your Kubernetes platforms as part of a CI/CD pipeline and other GitOps workflows.

Calico OSS 3.30, introduces numerous battle-tested observability and security tools from our commercial editions. This includes the following key features:

• Goldmane – A gRPC-based API for accessing and capturing flow logs and policy evaluation metrics
• Whisker – A web-based tool for viewing and filtering flow logs to troubleshoot connectivity issues and author and maintain Calico network security policies
• GlobalStagedNetworkPolicy and StagedNetworkPolicy – New custom resources that allow you to audit the behavior of a new policy before you actively enforce it
• Calico Ingress Gateway – Our 100% upstream, enterprise-ready implementation of the Gateway API that is based on Envoy Gateway
• Calico Cloud ready – Every OSS cluster includes the required components to connect to a stateless, read-only, and free version of Calico Cloud

You may know about the Calico REST API, which allows you to manage Calico resources, such as Calico network policy, Calico IPAM configurations Continue reading

Calico Whisker, Your New Ally in Network Observability

With the upcoming release of Calico v3.30 on the horizon, we are excited to introduce Calico Whisker, a simple yet powerful User Interface (UI) designed to enhance network observability and policy debugging. If you’ve ever struggled to make sense of network flow logs or troubleshoot policies in a complex Kubernetes cluster, Whisker is your friend!

Whisker is a three part deployment that holds a UI, backend and a gRPC channel to communicate with the Felix brain of Calico to gather live flow information and present it in a human readable, easy to understand way. But before we get started let’s dive into why Whisker is a must-have for your Kubernetes environment, what problems it solves, and how it can streamline your policy management.

Navigating Network Flows is Difficult

In Kubernetes environments, network flows are the backbone of communication between workloads. As clusters scale, so does the complexity of managing these flows and their security. Without clear visibility and effective observability tools, teams often struggle with:

• Diagnosing unexplained workload behavior and determining why certain applications aren’t working as expected.
• Identifying the real reason why certain workload communications are permitted or denied, which stems from understanding which policies are affecting specific Continue reading

Introducing Calico 3.30: A New Era of Open Source Network Security and Observability for Kubernetes

When we first launched Project Calico in 2016, we set out to make Kubernetes networking easy, reliable, and scalable for all organizations. Our goal was to abstract away the complexity and performance overheads of other CNI plugins while simultaneously extending Kubernetes network policy to make it easier to secure your Kubernetes workloads.

Over the last 9 years, we’ve seen our community grow alongside Calico Open Source, which has become the most widely adopted Kubernetes networking tool that now powers over 8 million nodes across more than 166 countries. We’ve seen the challenges our community has faced as more organizations adopt Kubernetes, and as the scale and complexity of these Kubernetes deployments has increased. Through our commercial offerings, we’ve helped solve networking and network security challenges for some of the world’s largest Kubernetes deployments, from financial institutions to telcos.

Calico OSS 3.30

With the release of Calico OSS 3.30 in May, we are open sourcing our battle-tested observability and security tools from our commercial editions. This includes the following key features:

• Goldmane – A gRPC-based API for accessing and capturing flow logs and policy evaluation metrics
• Whisker – A web-based tool for viewing and filtering flow logs to troubleshoot Continue reading

A Detailed Look at the Calico Ingress Gateway

Managing traffic in Kubernetes environments presents serious security and operational challenges. Traditional ingress solutions lack flexibility, rely on proprietary configurations, and offer limited traffic control, creating security gaps and inefficiencies.

What’s needed is a more flexible, scalable, and policy-driven approach to ingress traffic management. Enter Calico Ingress Gateway—built to eliminate these limitations while enhancing security, visibility, and control over ingress traffic at scale.

So Why an Ingress Gateway?

An ingress gateway serves as the first point of contact for external traffic entering a Kubernetes cluster. For most modern applications, this traffic includes API requests, user connections, or service calls, all of which need to be routed to the appropriate workloads securely and efficiently. Without a robust ingress solution, organizations face a range of challenges:

• Customization Challenges: Legacy ingress solutions provide limited flexibility, frequently requiring custom annotations to extend functionality, which increases the time and complexity of implementations.
• Operational Complexity: Traditional ingress controllers often rely on proprietary configurations, making deployments harder to manage and less portable across environments.
• Limited Traffic Control: Basic ingress controllers lack some of the advanced features needed to manage, shape, and secure traffic effectively.

What is the Calico Ingress Gateway?

The Calico Ingress Gateway is a 100% Continue reading

How Calico Network Security Works

In the rapidly evolving world of Kubernetes, network security remains one of the most challenging aspects for organizations. The shift to dynamic containerized environments brings challenges like inter-cluster communication, rapid scaling, and multi-cloud deployments. These challenges, compounded by tool sprawl and fragmented visibility, leave teams grappling with operational inefficiencies, misaligned priorities, and increasing vulnerabilities. Without a unified solution, organizations risk security breaches and compliance failures.

Calico reimagines Kubernetes security with a holistic, end-to-end approach that simplifies operations while strengthening defenses. By unifying key capabilities like ingress and egress gateways, microsegmentation, and real-time observability, Calico empowers teams to bridge the gaps between security, compliance, and operational efficiency. The result is a scalable, robust platform that addresses the unique demands of containerized environments without introducing unnecessary complexity. Let’s look at how Calico’s key network security capabilities make this possible.

Calico Ingress Gateway

The Calico Ingress Gateway is a Kubernetes-native solution, built on the Envoy Gateway, that serves as a centralized entry point for managing and securing incoming traffic to your clusters. Implementing the Kubernetes Gateway API specification, it replaces traditional ingress controllers with a more robust, scalable, and flexible architecture that is capable of more Continue reading

Calico eBPF Source IP Preservation: The Unexpected Story of High Tail Latency

The Calico eBPF data plane is your choice if latency is your primary concern. It was very disturbing that some benchmarking brought to our attention that eBPF had higher tail latency than iptables. The 99+% percentiles were higher by as much as a few hundred milliseconds. We did a whole bunch of experiments and we could not crack the nut until we observed that there are some occasional and unexpected TCP reset (RST) packets, but no connections were reset.

We noticed that the RST belongs to a connection that was already completed and finished a while ago. That was strange, but it pointed us in the right direction. We also knew that this happens only if the benchmark uses a LoadBalancer and we have connections going through multiple nodes to the same backend pod. That was enough to get to the root cause, but let’s start at the beginning…

External Clients and Kubernetes Services

One of the shortcomings of iptables/nftables based networking in Kubernetes is that if an external client connects to your cluster via a NodePort or a LoadBalancer, you may lose its IP address along the way. The reason for this is that the client may connect to Continue reading