Rewinding the exponential: How we keep hardware usage down

More than 20% of the web sits behind Cloudflare’s network, and our team, the Data team, owns the data pipeline responsible for plumbing that data (and more) from our global network to applicable central data centers. The Cloudflare network is a globally distributed system supporting tens of thousands of servers. Data that power Cloudflare Logs, SQL API, observability, internal analytics, and more flow through this network and use our data pipeline to serve customers.

In May of this year, we were providing data delivery at roughly two billion events per second. This includes customer traffic as well as data sent from internal teams. Year after year, we tackle the same problem — exponential traffic growth against linear hardware growth. We have to react to this increase in order to keep resource costs under control. Over the past five years, we’ve made various improvements to compensate for the data volume increase in comparison to hardware allocation, equivalent to deferring two and a half years of hardware expansion. This post details those changes: breaking up our monolithic data schema in 2022, adjusting compression in 2024, rearchitecting pipeline replication in spring 2026, decentralizing sampling in summer 2026, and introducing aggregations into the Continue reading

Introducing notifications for HTTP Traffic Anomalies

When it comes to managing Internet properties, the difference between a small technical hiccup and major incident is often a matter of speed. Proactive alerting plays a crucial role, which is why we were excited when we released HTTP Error Rate notifications — giving administrators visibility into when end users are experiencing errors.

But what if there are issues that don't show up as errors, like a sudden drop in traffic, or a spike?

Today, we're excited to announce Traffic Anomalies notifications, available to enterprise customers. These notifications trigger when Cloudflare detects unexpected changes in traffic, giving another valuable perspective into the health of your systems.

Unexpected changes in traffic could be indicative of many things. If you run an ecommerce site and see a spike in traffic that could be great news — maybe customers are flocking to your sale, or you just had an ad run on a popular TV show. However, it could also mean that something is going wrong: maybe someone accidentally turned off a firewall rule, and now you’re seeing more malicious traffic. Either way, you might want to know that something has changed.

Similarly, a sudden drop in traffic could mean many things. Perhaps Continue reading