How 1&1 Mail & Media Scaled Kubernetes Networking with eBPF and Calico
“We started in 2017 with Calico and never regretted it!”
—Stefan Fudeus, Product Owner/Lead Architect, 1&1 Mail & Media
Challenge
1&1 Mail & Media, part of the IONOS group, powers popular European internet brands including GMX and Web.de, serving more than 50% of Germany’s population with critical identity and email infrastructure. With roughly 45 to 50 million users, network reliability is non-negotiable. Any downtime could affect millions.
By 2022, the company had containerized 80% of its workloads on Kubernetes across three self-managed data centers. While the platform, backed by bare metal nodes and custom network layers, was highly scalable, network throughput bottlenecks began to emerge. Pods were limited to 2.5 Gbps of bandwidth due to IP encapsulation overhead, despite 10 Gbps network interfaces.
The team needed a solution that:
- Improved pod-to-pod network performance
- Maintained strong network policy isolation across up to 40 tenants per cluster
- Scaled to millions of network connections and 1.4 million HTTP requests per second
Solution
1&1 Mail & Media had adopted Calico back in 2017, largely for its unique Kubernetes NetworkPolicy standard support. As their Kubernetes platform evolved, with clusters scaling to 300 bare metal nodes, 16,000 pods, and over 4 million Continue reading
HS109: Is AI a Purchase or a Hire?
Is adding AI to your environment a software purchase? Or is it more like hiring an employee? Heavy Strategy’s John Burke and Johna Johnson debate whether AI should be treated as just another application you buy and use, or be handled like an employee you’re bringing on staff (complete with background and reference checks, training... Read more »
IBM Outlines Steps To Verify Claims Of Quantum Advantage
D-Wave executives stirred up some controversy earlier this year when they claimed a smaller version of its Advantage 2 annealing quantum system, armed with 1,200 qubits, had reached “quantum supremacy,” – or “quantum advantage” – that significant but ill-defined time when a quantum system is able to solve a problem in much less time, at a lower cost, or more efficiently than the most powerful classical supercomputer. …
IBM Outlines Steps To Verify Claims Of Quantum Advantage was written by Jeffrey Burt at The Next Platform.
Reducing double spend latency from 40 ms to < 1 ms on privacy proxy
One of Cloudflare’s big focus areas is making the Internet faster for end users. Part of the way we do that is by looking at the "big rocks" or bottlenecks that might be slowing things down — particularly processes on the critical path. When we recently turned our attention to our privacy proxy product, we found a big opportunity for improvement.
What is our privacy proxy product? These proxies let users browse the web without exposing their personal information to the websites they’re visiting. Cloudflare runs infrastructure for privacy proxies like Apple’s Private Relay and Microsoft’s Edge Secure Network.
Like any secure infrastructure, we make sure that users authenticate to these privacy proxies before we open up a connection to the website they’re visiting. In order to do this in a privacy-preserving way (so that Cloudflare collects the least possible information about end-users) we use an open Internet standard – Privacy Pass – to issue tokens that authenticate to our proxy service.
Every time a user visits a website via our Privacy Proxy, we check the validity of the Privacy Pass token which is included in the Proxy-Authorization header in their request. Before we cryptographically validate a user's token, we check Continue reading
Notes on Flow Control for High Speed Networks
“In theory, there is no difference between theory and practice. In practice, there is.” - Jan L. A. van de SnepscheutPerplexity is using stealth, undeclared crawlers to evade website no-crawl directives
We are observing stealth crawling behavior from Perplexity, an AI-powered answer engine. Although Perplexity initially crawls from their declared user agent, when they are presented with a network block, they appear to obscure their crawling identity in an attempt to circumvent the website’s preferences. We see continued evidence that Perplexity is repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity, as well as ignoring — or sometimes failing to even fetch — robots.txt files.
The Internet as we have known it for the past three decades is rapidly changing, but one thing remains constant: it is built on trust. There are clear preferences that crawlers should be transparent, serve a clear purpose, perform a specific activity, and, most importantly, follow website directives and preferences. Based on Perplexity’s observed behavior, which is incompatible with those preferences, we have de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling.
We received complaints from customers who had both disallowed Perplexity crawling activity in their robots.txt files and also created WAF rules to specifically block both of Perplexity’s declared crawlers: PerplexityBot and Perplexity-User. Continue reading
NB537: Palo Alto Networks IDs New Market With $25 Billion CyberArk Buy; Intel to Shed Networking Biz
Take a Network Break! Guest opinionator Tom Hollingsworth joins Johna Johnson to opine on the latest tech news. On the vulnerability front, several versions of BentoML are open to a server side request forgery. Looking at tech news, Intel will spin out its networking and edge group as it continues cost-cutting, Palo Alto Networks makes... Read more »MUST READ: Storage Devices and Latency
PlanetScale published a great article describing the high-level principles of how storage devices work and covering everything from tape drives to SSDs and network-attached storage — a must-read for anyone even remotely interested in how their data is stored.
From Classroom to Community 1 – Sort and Game
I am a former high school teacher with a passion for networking and programming, especially […]
The post From Classroom to Community 1 - Sort and Game first appeared on Brezular's Blog.
Python Scripts – From Classroom to Community – 1
I am a former high school teacher with a passion for networking and programming, especially […]
The post Python Scripts – From Classroom to Community - 1 first appeared on Brezular's Blog.
HN790: From Rule-Based to Goal-Based: Rethinking Autonomous AI Operations (Sponsored)
On Heavy Networking today, AI operations for networking. That is, how do we delegate some amount of responsibility for network operations to artificial intelligence? Cisco is our sponsor, and our guests are Omar Sultan, Director for Product Management of Automation and AI; and Javier Antich, Chief Mad Scientist for AI (yes, that’s his title!). We talk... Read more »TNO037: The Next Era of Network Management and Operations
What’s the next era of network management and operations? Total Network Operations talks to Mahesh Jethanandani, Area Director(AD) for all of Operations and Management (OPS) at IETF and Distinguished Engineer at Arrcus. Mahesh describes a workshop from December of 2024 that sought to investigate the past, present, and future of network management and operations. He... Read more »Vulnerability disclosure on SSL for SaaS v1 (Managed CNAME)
Earlier this year, a group of external researchers identified and reported a vulnerability in Cloudflare’s SSL for SaaS v1 (Managed CNAME) product offering through Cloudflare’s bug bounty program. We officially deprecated SSL for SaaS v1 in 2021; however, some customers received extensions for extenuating circumstances that prevented them from migrating to SSL for SaaS v2 (Cloudflare for SaaS). We have continually worked with the remaining customers to migrate them onto Cloudflare for SaaS over the past four years and have successfully migrated the vast majority of these customers. For most of our customers, there is no action required; for the very small number of SaaS v1 customers, we will be actively working to help migrate you to SSL for SaaS v2 (Cloudflare for SaaS).
Back in 2017, Cloudflare announced SSL for SaaS, a product that allows SaaS providers to extend the benefits of Cloudflare security and performance to their end customers. Using a “Managed CNAME” configuration, providers could bring their customer’s domain onto Cloudflare. In the first version of SSL for SaaS (v1), the traffic for Custom Hostnames is proxied to the origin based on the IP addresses assigned to the Continue reading
AI Embiggens The Big Clouds, Especially Microsoft
Building a successful cloud and growing it quarter after quarter, year after year, with first mover status or without it, is not an easy task. …
AI Embiggens The Big Clouds, Especially Microsoft was written by Timothy Prickett Morgan at The Next Platform.
IPB180: IPv6 Basics – Deployment
We’re continuing our IPv6 Basics series discussing on this episode deployment. We’ll help you sort out why you should deploy IPv6, things to consider before starting a deployment, and what approach you should take such as “inside out” vs. “outside in” and when you should deploy IPv6. Lastly we explain why you should seek out... Read more »N4N035: Well Actually . . . Listener Comments and Corrections
We ask listeners for follow up and you’ve sent it in. On today’s show we respond to listener comments and corrections on tunneling, the link aggregation control protocol, link aggregation in general, and DHCP options. We also talk about the network engineering certification journey. If you’ve got a “Well, actually” or any other follow up,... Read more »For Financial Services Firms, AI Inference Is As Challenging As Training
A decade ago, when traditional machine learning techniques were first being commercialized, training was incredibly hard and expensive, but because models were relatively small, inference – running new data through a model to cause an application to act or react – was easy. …
For Financial Services Firms, AI Inference Is As Challenging As Training was written by Timothy Prickett Morgan at The Next Platform.