Over 700 million events/second: How we make sense of too much data
Cloudflare's network provides an enormous array of services to our customers. We collect and deliver associated data to customers in the form of event logs and aggregated analytics. As of December 2024, our data pipeline is ingesting up to 706M events per second generated by Cloudflare's services, and that represents 100x growth since our 2018 data pipeline blog post.
At peak, we are moving 107 GiB/s of compressed data, either pushing it directly to customers or subjecting it to additional queueing and batching.
All of these data streams power things like Logs, Analytics, and billing, as well as other products, such as training machine learning models for bot detection. This blog post is focused on techniques we use to efficiently and accurately deal with the high volume of data we ingest for our Analytics products. A previous blog post provides a deeper dive into the data pipeline for Logs.
The pipeline can be roughly described by the following diagram.
The data pipeline has multiple stages, and each can and will naturally break or slow down because of hardware failures or misconfiguration. And when that happens, there is just too much data to be able to Continue reading
Cisco VRRPv3 IPv6 Configuration Sucks
I spent way too much time ironing out the VRRPv3 quirks on the dozen (or so) platforms supported by netlab. This is the second blog post describing some of the ridiculous stuff I had to deal with.
This is how you configure the basic VRRPv3 parameters for IPv4 on a Cisco IOS/XE device:
VRRPv3 IPv4 configuration on Cisco IOS
interface GigabitEthernet0/1
vrrp 217 address-family ipv4
address 172.16.33.42
You would expect something similar for IPv6, right? You’d be right if you were working with Arista EOS:
From Python to Go 012. Processing User Input From CLI Arguments And Standard Input.
Hello my friend,
Whenever we develop any network and IT infrastructure automation applications, we need to have some options to provide user input. In previous blog posts in these series we already covered how to provide user input via environment variables and files. Whilst these two approaches can cover majority of your use cases, especially if you develop containerized applications running in autonomy, there are still two options we would like to talk today about.
Why To Bother Learning Automation?
For many years I was doing network design and operation without automating it (or at least without structured approach to automate it). And there are still loads of such job positions out there. And I see it based on the audience of my blog: majority of people here for networking knowledge, much less are for automation topics. From pure pragmatic standpoint of writing popular blogs, I should stick to network technologies, especially something fancy as SD-WAN and others. However, from the direction of the technologies development, I see that value (including jobs) comes from intersection of domains: networking, compute, storage, software development, data bases, Kubernetes, observability, etc. I’m of a strong opinion that engineers these days must be aware of Continue reading
Palo Alto How to Configure SSL Decryption?
Most websites we access today use HTTPS, and to fully leverage a Next-Generation Firewall (NGFW) like Palo Alto, inspecting encrypted HTTPS sessions is crucial. Configuring SSL decryption isn't just a set-it-and-forget-it task. It requires careful consideration and ongoing improvements. In this blog post, we'll explore how to configure SSL decryption in Palo Alto firewalls and highlight some pitfalls to be aware of. So, let's get to it.
As always, if you find this post helpful, press the ‘clap’ button on the left. It means a lot to me and helps me know you enjoy this type of content.
Palo Alto How to Block Specific URLs?
In this blog post, we’ll explore how to block specific sites using a Palo Alto firewall. There are two ways to achieve this, and we’ll cover both options.
Suresh Vina
SSL Decryption Considerations
As I mentioned earlier, configuring SSL decryption isn’t as simple as flipping a switch. Decryption allows your firewall to inspect the contents of encrypted sessions. Normally, HTTPS traffic is encrypted from your browser to the server, ensuring the sessions are private. However, with SSL decryption, the firewall acts as a man-in-the-middle, inspecting the traffic in plain text. It’s crucial Continue reading
Brad McCredie Is The Pedal To AMD’s Datacenter GPU Metal
Brad McCredie like engines, and more importantly, he likes to make them go fast. …
Brad McCredie Is The Pedal To AMD’s Datacenter GPU Metal was written by Timothy Prickett Morgan at The Next Platform.
HN765: Telecom In the Bahamas: Lessons In Resiliency
Many of us have had network design discussions relating to natural disasters. What if a fire comes through? Or a flood? For most of us, those discussions don’t feel overly worthy of our attention. Yes, we should think about it. Yes, we should plan for it. If we’re really serious, we’ll even dust off the... Read more »Hedge 256: The Impact of Your First Language
Richard Wexelblat published an article in 1980 titled: “The consequences of one’s first programming language.” We’ve all seen C code written like Python, or Python code written like C, so it’s obvious a coder’s first language has a long lasting effect on their style. What about network engineers? Are there times and places where the first of anything a network engineers encounters has a long lasting impact on the way they think and work? In this roundtable, Tom, Eyvonne, and Russ consider different ways this might apply to network engineering.
TNO013: How Do We Apply DevOps to Traditional Networking?
Can we map DevOps principles and practices to network operations? Should we? On today’s show, we talk with Williams Collins and Eyvonne Sharp to understand DevOps both culturally and practically. We also discuss the challenges of applying DevOps to network operations. We look at lightweight options such as starting small, having a centralized source of... Read more »Use BGP Outbound Route Filters (ORF) for IP Prefixes
When a BGP router cannot fit the whole BGP table into its forwarding table (FIB), we often use inbound filters to limit the amount of information the device keeps in its BGP table. That’s usually a waste of resources:
- The BGP neighbor has to send information about all prefixes in its BGP table
- The device with an inbound filter wastes additional CPU cycles to drop many incoming updates.
Wouldn’t it be better for the device with an inbound filter to push that filter to its BGP neighbors?
GenAI Boom: Datacenter Spending Forecast Raised Again
Economic and technical forces have a kind of momentum that keeps them growing even as any new technology goes through its inevitable hype cycle from innovation to inflated expectations to disillusionment to deployment into productivity. …
GenAI Boom: Datacenter Spending Forecast Raised Again was written by Timothy Prickett Morgan at The Next Platform.
IPB167: Crossing the 50% IPv6 Adoption Threshold
IPv6 adoption should cross 50% in 2025. In today’s podcast, we discuss the implications and significance of IPv6 adoption reaching this milestone. While this transition might not be visible to average users, it does impact IT professionals. We explore varying adoption rates across industries, and talk about compliance requirements will soon necessitate IPv6. We also... Read more »How Enterprise AI Can Ease The Data Gravity Burden
COMMISSIONED: Among the many tough decisions IT leaders face is where to best host AI workloads. …
How Enterprise AI Can Ease The Data Gravity Burden was written by Timothy Prickett Morgan at The Next Platform.
N4N010: Why Was the Colon Chosen as a Delimiter in IPv6?
It’s history day on N Is For Networking! We learn about the development of IPv6 directly from Bob Hinden, one of the pioneers who made it happen. Bob discusses his journey from early work on ARPANET to his significant contributions to IPv6. We also cover the transition from IPv4, the challenges faced during IPv6’s creation,... Read more »Sturgeon’s Law, VRRPv3 Edition
I just wasted several days trying to figure out how to make the dozen (or so) platforms for which we implemented VRRPv3 in netlab work together. This is the first in a series of blog posts describing the ridiculous stuff we discovered during that journey
The idea was pretty simple:
- Create a lab with the tested device and a well-known probe connected to the same subnet.
- Disable VRRP (or interface) on the probe and check IPv4 and IPv6 connectivity through the tested device (verifying it takes over ownership of VRRP MAC and IP addresses).
- Reenable VRRP on the probe and change its VRRP priority several times to check the state transitions through INIT/BACKUP(lower priority)/MASTER(change in priority)/BACKUP(preempting after a change in priority).
OpenAI Declares Its Hardware Independence (Sort Of) With Stargate Project
The dependency dance between AI pioneer OpenAI and the Microsoft Azure cloud and the application software divisions of its parent company are fascinating to watch. …
OpenAI Declares Its Hardware Independence (Sort Of) With Stargate Project was written by Timothy Prickett Morgan at The Next Platform.