4.2 Tbps of bad packets and a whole lot more: Cloudflare’s Q3 DDoS report
Welcome to the 19th edition of the Cloudflare DDoS Threat Report. Released quarterly, these reports provide an in-depth analysis of the DDoS threat landscape as observed across the Cloudflare network. This edition focuses on the third quarter of 2024.
With a 296 Terabit per second (Tbps) network located in over 330 cities worldwide, Cloudflare is used as a reverse proxy by nearly 20% of all websites. Cloudflare holds a unique vantage point to provide valuable insights and trends to the broader Internet community.
The number of DDoS attacks spiked in the third quarter of 2024. Cloudflare mitigated nearly 6 million DDoS attacks, representing a 49% increase QoQ and 55% increase YoY.
Out of those 6 million, Cloudflare’s autonomous DDoS defense systems detected and mitigated over 200 hyper-volumetric DDoS attacks exceeding rates of 3 terabits per second (Tbps) and 2 billion packets per second (Bpps). The largest attack peaked at 4.2 Tbps and lasted just a minute.
The Banking & Financial Services industry was subjected to the most DDoS attacks. China was the country most targeted by DDoS attacks, and Indonesia was the largest source of DDoS attacks.
To learn more about DDoS attacks and other types Continue reading
Training a million models per day to save customers of all sizes from DDoS attacks
Our always-on DDoS protection runs inside every server across our global network. It constantly analyzes incoming traffic, looking for signals associated with previously identified DDoS attacks. We dynamically create fingerprints to flag malicious traffic, which is dropped when detected in high enough volume — so it never reaches its destination — keeping customer websites online.
In many cases, flagging bad traffic can be straightforward. For example, if we see too many requests to a destination with the same protocol violation, we can be fairly sure this is an automated script, rather than a surge of requests from a legitimate web browser.
Our DDoS systems are great at detecting attacks, but there’s a minor catch. Much like the human immune system, they are great at spotting attacks similar to things they have seen before. But for new and novel threats, they need a little help knowing what to look for, which is an expensive and time-consuming human endeavor.
Cloudflare protects millions of Internet properties, and we serve over 60 million HTTP requests per second on average, so trying to find unmitigated attacks in such a huge volume of traffic is a daunting task. In order to protect the smallest of companies, Continue reading
Is this thing on? Using OpenBMC and ACPI power states for reliable server boot
At Cloudflare, we provide a range of services through our global network of servers, located in 330 cities worldwide. When you interact with our long-standing application services, or newer services like Workers AI, you’re in contact with one of our fleet of thousands of servers which support those services.
These servers which provide Cloudflare services are managed by a Baseboard Management Controller (BMC). The BMC is a special purpose processor — different from the Central Processing Unit (CPU) of a server — whose sole purpose is ensuring a smooth operation of the server.
Regardless of the server vendor, each server has this BMC. The BMC runs independently of the CPU and has its own embedded operating system, usually referred to as firmware. At Cloudflare, we customize and deploy a server-specific version of the BMC firmware. The BMC firmware we deploy at Cloudflare is based on the Linux Foundation Project for BMCs, OpenBMC. OpenBMC is an open-sourced firmware stack designed to work across a variety of systems including enterprise, telco, and cloud-scale data centers. The open-source nature of OpenBMC gives us greater flexibility and ownership of this critical server subsystem, instead of the closed nature of proprietary firmware. Continue reading
Building Vectorize, a distributed vector database, on Cloudflare’s Developer Platform
Vectorize is a globally distributed vector database that enables you to build full-stack, AI-powered applications with Cloudflare Workers. Vectorize makes querying embeddings — representations of values or objects like text, images, audio that are designed to be consumed by machine learning models and semantic search algorithms — faster, easier and more affordable.
In this post, we dive deep into how we built Vectorize on Cloudflare’s Developer Platform, leveraging Cloudflare’s global network, Cache, Workers, R2, Queues, Durable Objects, and container platform.
A vector database is a queryable store of vectors. A vector is a large array of numbers called vector dimensions.
A vector database has a similarity search query: given an input vector, it returns the vectors that are closest according to a specified metric, potentially filtered on their metadata.
Vector databases are used to power semantic search, document classification, and recommendation and anomaly detection, as well as contextualizing answers generated by LLMs (Retrieval Augmented Generation, RAG).
Conventional data structures like B-trees, or binary search trees expect the data they index to be cheap to compare and to follow Continue reading
The story of web framework Hono, from the creator of Hono
Hono is a fast, lightweight web framework that runs anywhere JavaScript does, built with Web Standards. Of course, it runs on Cloudflare Workers.
It was three years ago, in December 2021. At that time, I wanted to create applications for Cloudflare Workers, but the code became verbose without using a framework, and couldn't find a framework that suited my needs. Itty-router was very nice but too simple. Worktop and Sunder did the same things I wanted to do, but their APIs weren't quite to my liking. I was also interested in creating a router — a program that determines which action is executed based on the HTTP method and URL path of the Request — made of a Trie tree structure because it’s fast. So, I started building a web framework with a Trie tree-based router.
“While trying to create my applications, I ended up creating my framework for them.” — a classic example of yak shaving. However, Hono is now used by many developers, including Cloudflare, which uses Hono in core products. So, this journey into the depths of yak shaving was ultimately meaningful.
Hono truly runs anywhere — not just on Cloudflare Continue reading
Analysis of the EPYC 145% performance gain in Cloudflare Gen 12 servers
Cloudflare's network spans more than 330 cities in over 120 countries, serving over 60 million HTTP requests per second and 39 million DNS queries per second on average. These numbers will continue to grow, and at an accelerating pace, as will Cloudflare’s infrastructure to support them. While we can continue to scale out by deploying more servers, it is also paramount for us to develop and deploy more performant and more efficient servers.
At the heart of each server is the processor (central processing unit, or CPU). Even though many aspects of a server rack can be redesigned to improve the cost to serve a request, CPU remains the biggest lever, as it is typically the primary compute resource in a server, and the primary enabler of new technologies.
Cloudflare’s 12th Generation server with AMD EPYC 9684-X (codenamed Genoa-X) is 145% more performant and 63% more efficient. These are big numbers, but where do the performance gains come from? Cloudflare’s hardware system engineering team did a sensitivity analysis on three variants of 4th generation AMD EPYC processor to understand the contributing factors.
For the 4th generation AMD EPYC Processors, AMD offers three architectural variants:
mainstream classic Zen 4 cores, codenamed Continue reading
Protect against identity-based attacks by sharing Cloudflare user risk scores with Okta
Cloudflare One, our secure access service edge (SASE) platform, is introducing a new integration with Okta, the identity and access management (IAM) vendor, to share risk indicators in real-time and simplify how organizations can dynamically manage their security posture in response to changes across their environments.
For many organizations, it is becoming increasingly challenging and inefficient to adapt to risks across their growing attack surface. In particular, security teams struggle with multiple siloed tools that fail to share risk data effectively with each other, leading to excessive manual effort to extract signals from the noise. To address this complexity, Cloudflare launched risk posture management capabilities earlier this year to make it easier for organizations to accomplish three key jobs on one platform:
Evaluating risk posed by people by using first-party user entity and behavior analytics (UEBA) models
Exchanging risk telemetry with best-in-class security tools, and
Enforcing risk controls based on those dynamic first- and third-party risk scores.
Today’s announcement builds on these capabilities (particularly job #2) and our partnership with Okta by enabling organizations to share Cloudflare’s real-time user risk scores with Okta, which can then automatically enforce policies based on that user’s risk. In this way, organizations can adapt Continue reading
CJ Desai: Why I joined Cloudflare as President of Product and Engineering
I am thrilled to embark on this journey to run Product and Engineering at Cloudflare, driving forward the mission of helping build a better Internet.
While I was a graduate student at University of Illinois, the university introduced the Mosaic web browser to students. In addition to being super easy to install and use, it displayed pictures next to text for the first time. This may not seem impressive today, but back then it felt like a magical step forward.
This simple but powerful upgrade opened up the once niche user base from academics to the masses, transforming the world wide web to become an Internet phenomenon. Since then, I’ve always sought to be part of teams that worked on transformational technologies, including Software-as-a-Service, cloud computing, and AI. Innovation is the life blood of every technology company. To this day, I’m inspired by building products and technology that get adopted at mass scale.
The world is in a very interesting moment for technological innovation: the AI landscape is uncharted and developing at an exponential rate; the urgency for enterprises to reduce tech debt and reliance on legacy applications is at an all Continue reading
What’s new in Cloudflare One: Digital Experience (DEX) monitoring notifications and seamless access to Cloudflare Gateway with China Express
At Cloudflare, we are constantly innovating and launching new features and capabilities across our product portfolio. We are introducing roundup blog posts to ensure that you never miss the latest updates across our platform. In this post, we are excited to share two new ways that our customers can continue to keep their web properties performant and secure with Cloudflare One: new Digital Experience Monitoring (DEX) notifications help proactively identify issues that can affect the end-user digital experience, and integration with China Express enables secure access to China-hosted sites for Cloudflare Gateway customers.
As with other notification types, DEX notifications can be configured and reviewed from Cloudflare dashboard notifications.
DEX notifications address the challenge of proactively identifying issues affecting the digital experience of your end users. By monitoring device health and conducting synthetic tests from WARP clients deployed on your fleet's end-user devices, DEX provides valuable insights. These notifications empower IT administrators to quickly identify and address connectivity and application performance problems before they impact a wide range of users.
By proactively notifying administrators when problems arise, DEX helps minimize user disruption and provides Continue reading
CJ Desai: Why I joined Cloudflare as President of Product and Engineering
I am thrilled to embark on this journey to run Product and Engineering at Cloudflare, driving forward the mission of helping build a better Internet.
While I was a graduate student at University of Illinois, the university introduced the Mosaic web browser to students. In addition to being super easy to install and use, it displayed pictures next to text for the first time. This may not seem impressive today, but back then it felt like a magical step forward.
This simple but powerful upgrade opened up the once niche user base from academics to the masses, transforming the world wide web to become an Internet phenomenon. Since then, I’ve always sought to be part of teams that worked on transformational technologies, including Software-as-a-Service, cloud computing, and AI. Innovation is the life blood of every technology company. To this day, I’m inspired by building products and technology that get adopted at mass scale.
The world is in a very interesting moment for technological innovation: the AI landscape is uncharted and developing at an exponential rate; the urgency for enterprises to reduce tech debt and reliance on legacy applications is at an all time Continue reading
What’s new in Cloudflare One: Digital Experience (DEX) monitoring notifications and seamless access to Cloudflare Gateway with China Express
At Cloudflare, we are constantly innovating and launching new features and capabilities across our product portfolio. We are introducing roundup blog posts to ensure that you never miss the latest updates across our platform. In this post, we are excited to share two new ways that our customers can continue to keep their web properties performant and secure with Cloudflare One: new Digital Experience Monitoring (DEX) notifications help proactively identify issues that can affect the end-user digital experience, and integration with China Express enables secure access to China-hosted sites for Cloudflare Gateway customers.
Digital Experience Monitoring (DEX) offers device, application, and network performance monitoring, providing IT administrators with insights to quickly identify and resolve issues. With DEX notifications , account administrators can create configurable alert rules based on available algorithms (z-score, SLO) and existing DEX filters. When notification criteria are satisfied, customers are notified via email, Pagerduty, or Webhooks
As with other notification types, DEX notifications can be configured and reviewed from Cloudflare dashboard notifications.
DEX notifications address the challenge of proactively identifying issues affecting the digital experience of your end users. Continue reading
Improving platform resilience at Cloudflare through automation
Failure is an expected state in production systems, and no predictable failure of either software or hardware components should result in a negative experience for users. The exact failure mode may vary, but certain remediation steps must be taken after detection. A common example is when an error occurs on a server, rendering it unfit for production workloads, and requiring action to recover.
When operating at Cloudflare’s scale, it is important to ensure that our platform is able to recover from faults seamlessly. It can be tempting to rely on the expertise of world-class engineers to remediate these faults, but this would be manual, repetitive, unlikely to produce enduring value, and not scaling. In one word: toil; not a viable solution at our scale and rate of growth.
In this post we discuss how we built the foundations to enable a more scalable future, and what problems it has immediately allowed us to solve.
The Cloudflare Site Reliability Engineering (SRE) team builds and manages the platform that helps product teams deliver our extensive suite of offerings to customers. One important component of this platform is the collection of servers that power critical products such as Durable Objects, Workers, Continue reading
Leveraging Kubernetes virtual machines at Cloudflare with KubeVirt
Cloudflare runs several multi-tenant Kubernetes clusters across our core data centers. These general-purpose clusters run on bare metal and power our control plane, analytics, and various engineering tools such as build infrastructure and continuous integration.
Kubernetes is a container orchestration platform. It enables software engineers to deploy containerized applications to a cluster of machines. This enables teams to build highly-available software on a scalable and resilient platform.
In this blog post we discuss our Kubernetes architecture, why we needed virtualization, and how we’re using it today.
Multi-tenancy is a concept where one system can share its resources among a wide range of customers. This model allows us to build and manage a small number of general purpose Kubernetes clusters for our internal application teams. Keeping the number of clusters small reduces our operational toil. This model shrinks costs and increases computational efficiency by sharing hardware. Multi-tenancy also allows us to scale more efficiently. Scaling is done at either a cluster or application level. Cluster operators scale the platform by adding more hardware. Teams scale their applications by updating their Kubernetes manifests. They can scale vertically by increasing their resource requests or horizontally by increasing the number of Continue reading
Cloudflare acquires Kivera to add simple, preventive cloud security to Cloudflare One
We’re excited to announce that Kivera, a cloud security, data protection, and compliance company, has joined Cloudflare. This acquisition extends our SASE portfolio to incorporate inline cloud app controls, empowering Cloudflare One customers with preventative security controls for all their cloud services.
In today’s digital landscape, cloud services and SaaS (software as a service) apps have become indispensable for the daily operation of organizations. At the same time, the amount of data flowing between organizations and their cloud providers has ballooned, increasing the chances of data leakage, compliance issues, and worse, opportunities for attackers. Additionally, many companies — especially at enterprise scale — are working directly with multiple cloud providers for flexibility based on the strengths, resiliency against outages or errors, and cost efficiencies of different clouds.
Security teams that rely on Cloud Security Posture Management (CSPM) or similar tools for monitoring cloud configurations and permissions and Infrastructure as code (IaC) scanning are falling short due to detecting issues only after misconfigurations occur with an overwhelming volume of alerts. The combination of Kivera and Cloudflare One puts preventive controls directly into the deployment process, or ‘inline’, blocking errors before they happen. This offers a proactive approach essential to Continue reading
Thermal design supporting Gen 12 hardware: cool, efficient and reliable
In the dynamic evolution of AI and cloud computing, the deployment of efficient and reliable hardware is critical. As we roll out our Gen 12 hardware across hundreds of cities worldwide, the challenge of maintaining optimal thermal performance becomes essential. This blog post provides a deep dive into the robust thermal design that supports our newest Gen 12 server hardware, ensuring it remains reliable, efficient, and cool (pun very much intended).
Generally speaking, a server has five core resources: CPU (computing power), RAM (short term memory), SSD (long term storage), NIC (Network Interface Controller, connectivity beyond the server), and GPU (for AI/ML computations). Each of these components can withstand different temperature limits based on their design, materials, location within the server, and most importantly, the power they are designed to work at. This final criteria is known as thermal design power (TDP).
The reason why TDP is so important is closely related to the first law of thermodynamics, which states that energy cannot be created or destroyed, only transformed. In semiconductors, electrical energy is converted into heat, and TDP measures the maximum heat output that needs to be managed to ensure Continue reading
Enhance your website’s security with Cloudflare’s free security.txt generator
Meet Georgia, a diligent website administrator at a growing e-commerce company. Every day, Georgia juggles multiple tasks, from managing server uptime to ensuring customer data security. One morning, Georgia receives an email from a security researcher who discovered a potential vulnerability on the website. The researcher struggled to find the right contact information, leading to delays in reporting the issue. Georgia realizes the need for a standardized way to communicate with security researchers, ensuring that vulnerabilities are reported swiftly and efficiently. This is where security.txt comes in.
Security.txt is becoming a widely adopted standard among security-conscious organizations. By providing a common location and format for vulnerability disclosure information, it helps bridge the gap between security researchers and organizations. This initiative is supported by major companies and aligns with global security best practices. By offering an automated security.txt generator for free, we aim to empower all of our users to enhance their security measures without additional costs.
In 2020, Cloudflare published the Cloudflare Worker for the security.txt generator as an open-source project on GitHub, demonstrating our commitment to enhancing web security. This tool is actively used Continue reading
How Cloudflare auto-mitigated world record 3.8 Tbps DDoS attack
Since early September, Cloudflare's DDoS protection systems have been combating a month-long campaign of hyper-volumetric L3/4 DDoS attacks. Cloudflare’s defenses mitigated over one hundred hyper-volumetric L3/4 DDoS attacks throughout the month, with many exceeding 2 billion packets per second (Bpps) and 3 terabits per second (Tbps). The largest attack peaked 3.8 Tbps — the largest ever disclosed publicly by any organization. Detection and mitigation was fully autonomous. The graphs below represent two separate attack events that targeted the same Cloudflare customer and were mitigated autonomously.
A mitigated 3.8 Terabits per second DDoS attack that lasted 65 seconds
A mitigated 2.14 billion packet per second DDoS attack that lasted 60 seconds
Cloudflare customers are protected
Cloudflare customers using Cloudflare’s HTTP reverse proxy services (e.g. Cloudflare WAF and Cloudflare CDN) are automatically protected.
Cloudflare customers using Spectrum and Magic Transit are also automatically protected. Magic Transit customers can further optimize their protection by deploying Magic Firewall rules to enforce a strict positive and negative security model at the packet layer.
Other Internet properties may not be safe
The scale and frequency of these attacks are unprecedented. Due to their sheer size and bits/packets per second rates, these Continue reading
Patent troll Sable pays up, dedicates all its patents to the public!
Back in February, we celebrated our victory at trial in the U.S. District Court for the Western District of Texas against patent trolls Sable IP and Sable Networks. This was the culmination of nearly three years of litigation against Sable, but it wasn’t the end of the story.
Today we’re pleased to announce that the litigation against Sable has finally concluded on terms that we believe send a strong message to patent trolls everywhere — if you bring meritless patent claims against Cloudflare, we will fight back and we will win.
We’re also pleased to announce additional prizes in Project Jengo, and to make a final call for submissions before we determine the winners of the Final Awards. As a reminder, Project Jengo is Cloudflare’s effort to fight back against patent trolls by flipping the incentive structure that has encouraged the growth of patent trolls who extract settlements out of companies using frivolous lawsuits. We do this by asking the public to help identify prior art that can invalidate any of the patents that a troll holds, not just the ones that are asserted against Cloudflare. We’ve already given out over $125,000 to individuals since the launch Continue reading
Impact of Verizon’s September 30 outage on Internet traffic
On Monday, September 30, customers on Verizon’s mobile network in multiple cities across the United States reported experiencing a loss of connectivity. Impacted phones showed “SOS” instead of the usual bar-based signal strength indicator, and customers complained of an inability to make or receive calls on their mobile devices.
AS6167 (CELLCO) is the autonomous system used by Verizon for its mobile network. To better understand how the outage impacted Internet traffic on Verizon’s network, we took a look at HTTP request volume from AS6167 independent of geography, as well as traffic from AS6167 in various cities that were reported to be the most significantly impacted.
Although initial reports of connectivity problems started around 09:00 ET (13:00 UTC), we didn’t see a noticeable change in request volume at an ASN level until about two hours later. Just before 12:00 ET (16:00 UTC), Verizon published a social media post acknowledging the problem, stating “We are aware of an issue impacting service for some customers. Our engineers are engaged and we are working quickly to identify and solve the issue.”
As the Cloudflare Radar graph below shows, a slight decline (-5%) in HTTP traffic as compared to traffic at the Continue reading
Wrapping up another Birthday Week celebration
2024 marks Cloudflare’s 14th birthday. Birthday Week each year is packed with major announcements and the release of innovative new offerings, all focused on giving back to our customers and the broader Internet community. Birthday Week has become a proud tradition at Cloudflare and our culture, to not just stay true to our mission, but to always stay close to our customers. We begin planning for this week of celebration earlier in the year and invite everyone at Cloudflare to participate.
Months before Birthday Week, we invited teams to submit ideas for what to announce. We were flooded with submissions, from proposals for implementing new standards to creating new products for developers. Our biggest challenge is finding space for it all in just one week — there is still so much to build. Good thing we have a birthday to celebrate each year, but we might need an extra day in Birthday Week next year!
In case you missed it, here’s everything we announced during 2024’s Birthday Week:
Monday
|
What |
In a sentence… |
|
Start auditing and controlling the AI models accessing your content |
Understand which AI-related bots and crawlers can access your website, and which content you choose to allow Continue reading |