Post Mortem on Cloudflare Control Plane and Analytics Outage
Beginning on Thursday, November 2, 2023 at 11:43 UTC Cloudflare's control plane and analytics services experienced an outage. The control plane of Cloudflare consists primarily of the customer-facing interface for all of our services including our website and APIs. Our analytics services include logging and analytics reporting.
The incident lasted from November 2 at 11:44 UTC until November 4 at 04:25 UTC. We were able to restore most of our control plane at our disaster recovery facility as of November 2 at 17:57 UTC. Many customers would not have experienced issues with most of our products after the disaster recovery facility came online. However, other services took longer to restore and customers that used them may have seen issues until we fully resolved the incident. Our raw log services were unavailable for most customers for the duration of the incident.
Services have now been restored for all customers. Throughout the incident, Cloudflare's network and security services continued to work as expected. While there were periods where customers were unable to make changes to those services, traffic through our network was not impacted.
This post outlines the events that caused this incident, the architecture we had in place to prevent issues Continue reading
Cloudflare incident on October 30, 2023
Multiple Cloudflare services were unavailable for 37 minutes on October 30, 2023. This was due to the misconfiguration of a deployment tool used by Workers KV. This was a frustrating incident, made more difficult by Cloudflare’s reliance on our own suite of products. We are deeply sorry for the impact it had on customers. What follows is a discussion of what went wrong, how the incident was resolved, and the work we are undertaking to ensure it does not happen again.
Workers KV is our globally distributed key-value store. It is used by both customers and Cloudflare teams alike to manage configuration data, routing lookups, static asset bundles, authentication tokens, and other data that needs low-latency access.
During this incident, KV returned what it believed was a valid HTTP 401 (Unauthorized) status code instead of the requested key-value pair(s) due to a bug in a new deployment tool used by KV.
These errors manifested differently for each product depending on how KV is used by each service, with their impact detailed below.
What was impacted
A number of Cloudflare services depend on Workers KV for distributing configuration, routing information, static asset serving, and authentication state globally. These services instead received Continue reading
Introducing notifications for HTTP Traffic Anomalies
When it comes to managing Internet properties, the difference between a small technical hiccup and major incident is often a matter of speed. Proactive alerting plays a crucial role, which is why we were excited when we released HTTP Error Rate notifications — giving administrators visibility into when end users are experiencing errors.
But what if there are issues that don't show up as errors, like a sudden drop in traffic, or a spike?
Today, we're excited to announce Traffic Anomalies notifications, available to enterprise customers. These notifications trigger when Cloudflare detects unexpected changes in traffic, giving another valuable perspective into the health of your systems.
Unexpected changes in traffic could be indicative of many things. If you run an ecommerce site and see a spike in traffic that could be great news — maybe customers are flocking to your sale, or you just had an ad run on a popular TV show. However, it could also mean that something is going wrong: maybe someone accidentally turned off a firewall rule, and now you’re seeing more malicious traffic. Either way, you might want to know that something has changed.
Similarly, a sudden drop in traffic could mean many things. Perhaps Continue reading
Introducing HAR Sanitizer: secure HAR sharing
On Wednesday, October 18th, 2023, Cloudflare’s Security Incident Response Team (SIRT) discovered an attack on our systems that originated from an authentication token stolen from one of Okta’s support systems. No Cloudflare customer information or systems were impacted by the incident, thanks to the real-time detection and rapid action of our Security Incident Response Team (SIRT) in tandem with our Zero Trust security posture and use of hardware keys. With that said, we’d rather not repeat the experience — and so we have built a new security tool that can help organizations render this type of attack obsolete for good.
The bad actor in the Okta breach compromised user sessions by capturing session tokens from administrators at Cloudflare and other impacted organizations. They did this by infiltrating Okta’s customer support system and stealing one of the most common mechanisms for troubleshooting — an HTTP Response Archive (HAR) file.
HAR files contain a record of a user’s browser session, a kind of step-by-step audit, that a user can share with someone like a help desk agent to diagnose an issue. However, the file can also contain sensitive information that can be used to launch an attack.
As a follow-up to the Continue reading
Email Routing subdomain support, new APIs and security protocols
It's been two years since we announced Email Routing, our solution to create custom email addresses for your domains and route incoming emails to your preferred mailbox. Since then, the team has worked hard to evolve the product and add more powerful features to meet our users' expectations. Examples include Route to Workers, which allows you to process your Emails programmatically using Workers scripts, Public APIs, Audit Logs, or DMARC Management.
We also made significant progress in supporting more email security extensions and protocols, protecting our customers from unwanted traffic, and keeping our IP space reputation for email egress impeccable to maximize our deliverability rates to whatever inbox upstream provider you chose.
Since leaving beta, Email Routing has grown into one of our most popular products; it’s used by more than one million different customer zones globally, and we forward around 20 million messages daily to every major email platform out there. Our product is mature, robust enough for general usage, and suitable for any production environment. And it keeps evolving: today, we announce three new features that will help make Email Routing more secure, flexible, and powerful than ever.
New security protocols
The SMTP email protocol Continue reading
DDoS threat report for 2023 Q3
Welcome to the third DDoS threat report of 2023. DDoS attacks, or distributed denial-of-service attacks, are a type of cyber attack that aims to disrupt websites (and other types of Internet properties) to make them unavailable for legitimate users by overwhelming them with more traffic than they can handle — similar to a driver stuck in a traffic jam on the way to the grocery store.
We see a lot of DDoS attacks of all types and sizes, and our network is one of the largest in the world spanning more than 300 cities in over 100 countries. Through this network we serve over 64 million HTTP requests per second at peak and about 2.3 billion DNS queries every day. On average, we mitigate 140 billion cyber threats each day. This colossal amount of data gives us a unique vantage point to understand the threat landscape and provide the community access to insightful and actionable DDoS trends.
In recent weeks, we've also observed a surge in DDoS attacks and other cyber attacks against Israeli newspaper and media websites, as well as financial institutions and government websites. Palestinian websites have also seen a significant increase in DDoS attacks. View Continue reading
Q3 2023 Internet disruption summary
This post is also available in 简体中文, 繁體中文, 한국어, Deutsch, Français and Español.
Cloudflare operates in more than 300 cities in over 100 countries, where we interconnect with over 12,500 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions.
We have been publishing these summaries since the first quarter of 2022, and over that time, the charts on Cloudflare Radar have evolved. Many of the traffic graphs in early editions of this summary were screenshots from the relevant traffic pages on Radar. Late last year, we launched the ability to download graphs, and earlier this year, to embed dynamic graphs, and these summaries have taken advantage of those capabilities where possible. Sharp-eyed readers may notice an additional evolution in some of the graphs below: yellow highlighting indicating an observed “traffic anomaly”. Identification of such anomalies, along with the ability to be notified about them, as well as a timeline enhancement (embedded below) to the Cloudflare Radar Outage Center, were launched as Continue reading
Cache Reserve goes GA: enhanced control to minimize egress costs
Everyone is chasing the highest cache ratio possible. Serving more content from Cloudflare’s cache means it loads faster for visitors, saves website operators money on egress fees from origins, and provides multiple layers of resiliency and protection to make sure that content is available to be served and websites scale effortlessly. A year ago we introduced Cache Reserve to help customer’s serve as much content as possible from Cloudflare’s cache.
Today, we are thrilled to announce the graduation of Cache Reserve from beta to General Availability (GA), accompanied by the introduction of several exciting new features. These new features include adding Cache Reserve into the analytics shown on the Cache overview section of the Cloudflare dashboard, giving customers the ability to see how they are using Cache Reserve over time. We have also added the ability for customers to delete all data in Cache Reserve without losing content in the edge cache. This is useful for customers who are no longer using Cache Reserve storage.
We’re also introducing new tools that give organizations more granular control over which files are saved to Cache Reserve, based on valuable feedback we received during the beta. The default configuration of Cache Reserve Continue reading
Cache Rules go GA: precision control over every part of your cache
One year ago we introduced Cache Rules, a new way to customize cache settings on Cloudflare. Cache Rules provide greater flexibility for how users cache content, offering precise controls, a user-friendly API, and seamless Terraform integrations. Since it was released in late September 2022, over 100,000 websites have used Cache Rules to fine-tune their cache settings.
Today, we're thrilled to announce that Cache Rules, along with several other Rules products, are generally available (GA). But that’s not all — we're also introducing new configuration options for Cache Rules that provide even more options to customize how you cache on Cloudflare. These include functionality to define what resources are eligible for Cache Reserve, what timeout values should be respected when receiving data from your origin server, which custom ports we should use when we cache content, and whether we should bypass Cloudflare’s cache in the absence of a cache-control header.
Cache Rules give users full control and the ability to tailor their content delivery strategy for almost any use case, without needing to write code. As Cache Rules go GA, we are incredibly excited to see how fast customers can achieve their perfect cache strategy.
History of Customizing Cache Continue reading
Cyber attacks in the Israel-Hamas war
On October 7, 2023, at 03:30 GMT (06:30 AM local time), Hamas attacked Israeli cities and fired thousands of rockets toward populous locations in southern and central Israel, including Tel Aviv and Jerusalem. Air raid sirens began sounding, instructing civilians to take cover.
Approximately twelve minutes later, Cloudflare systems automatically detected and mitigated DDoS attacks that targeted websites that provide critical information and alerts to civilians on rocket attacks. The initial attack peaked at 100k requests per second (rps) and lasted ten minutes. Forty-five minutes later, a second much larger attack struck and peaked at 1M rps. It lasted six minutes. Additional smaller DDoS attacks continued hitting the websites in the next hours.
Not just DDoS attacks
Multiple Israeli websites and mobile apps have become targets of various pro-Palestinian hacktivist groups. According to Cybernews, one of those groups, AnonGhost, exploited a vulnerability in a mobile app that alerts Israeli civilians of incoming rockets, “Red Alert: Israel”. The exploit allowed them to intercept requests, expose servers and APIs, and send fake alerts to some app users, including a message that a “nuclear bomb is coming Continue reading
How Cloudflare mitigated yet another Okta compromise
On Wednesday, October 18, 2023, we discovered attacks on our system that we were able to trace back to Okta – threat actors were able to leverage an authentication token compromised at Okta to pivot into Cloudflare’s Okta instance. While this was a troubling security incident, our Security Incident Response Team’s (SIRT) real-time detection and prompt response enabled containment and minimized the impact to Cloudflare systems and data. We have verified that no Cloudflare customer information or systems were impacted by this event because of our rapid response. Okta has now released a public statement about this incident.
This is the second time Cloudflare has been impacted by a breach of Okta’s systems. In March 2022, we blogged about our investigation on how a breach of Okta affected Cloudflare. In that incident, we concluded that there was no access from the threat actor to any of our systems or data – Cloudflare’s use of hard keys for multi-factor authentication stopped this attack.
The key to mitigating this week’s incident was our team’s early detection and immediate response. In fact, we contacted Okta about the breach of their systems before they had notified us. The attacker used an open Continue reading
Empowering our partners with the new Tenant Platform dashboard
Itching to get started? Apply to the Self Serve Partner Beta or Enterprise partner programs now.
Cloudflare has always worked closely with partners to help build a better Internet. From our earliest Hosting Partners, to our latest Cloudflare One program and Authorized Service Delivery partners, we are dedicated to supporting our peers across the networking and cybersecurity ecosystem to secure Enterprise networks, mission-critical applications, and remote employees. As part of that commitment, we are proud to announce the general availability of our first dashboard for our Tenant Platform, providing an intuitive user interface for agencies and partners to manage their client accounts.
Tenant Platform introduction
The first version of the Tenant Platform was created in 2018 to support one of our large integration partners, IBM Cloud. They needed a secure way to independently provision accounts for their clients, spin up custom subscriptions, invite service users within each new account, and begin to configure the service. This platform, although API only, worked extremely well with our OEM and integration partners that were including our solution within their current platform to support their customers.
User interface overview
As Cloudflare has expanded the type of partners and customers it works Continue reading
Network flow monitoring is GA, providing end-to-end traffic visibility
Network engineers often find they need better visibility into their network’s traffic and operations while analyzing DDoS attacks or troubleshooting other traffic anomalies. These engineers typically have some high level metrics about their network traffic, but they struggle to collect essential information on the specific traffic flows that would clarify the issue. To solve this problem, Cloudflare has been piloting a cloud network flow monitoring product called Magic Network Monitoring that gives customers end-to-end visibility into all traffic across their network.
Today, Cloudflare is excited to announce that Magic Network Monitoring (previously called Flow Based Monitoring) is now generally available to all enterprise customers. Over the last year, the Cloudflare engineering team has significantly improved Magic Network Monitoring; we’re excited to offer a network services product that will help our customers identify threats faster, reduce vulnerabilities, and make their network more secure.
Magic Network Monitoring is automatically enabled for all Magic Transit and Magic WAN enterprise customers. The product is located at the account level of the Cloudflare dashboard and can be opened by navigating to “Analytics & Logs > Magic Monitoring”. The onboarding process for Magic Network Monitoring is self-serve, and all enterprise customers with access can begin Continue reading
Introducing the Project Argus Datacenter-ready Secure Control Module design specification
Historically, data center servers have used motherboards that included all key components on a single circuit board. The DC-SCM (Datacenter-ready Secure Control Module) decouples server management and security functions from a traditional server motherboard, enabling development of server management and security solutions independent of server architecture. It also provides opportunities for reducing server printed circuit board (PCB) material cost, and allows unified firmware images to be developed.
Today, Cloudflare is announcing that it has partnered with Lenovo to design a DC-SCM for our next-generation servers. The design specification has been published to the OCP (Open Compute Project) contribution database under the name Project Argus.
A brief introduction to baseboard management controllers
A baseboard management controller (BMC) is a specialized processor that can be found in virtually every server product. It allows remote access to the server through a network connection, and provides a rich set of server management features. Some of the commonly used BMC features include server power management, device discovery, sensor monitoring, remote firmware update, system event logging, and error reporting.
In a typical server design, the BMC resides on the server motherboard, along with other key components such as the processor, memory, CPLD and so on. This Continue reading
Malicious “RedAlert – Rocket Alerts” Application Targets Israeli Phone Calls, SMS, and User Information
On October 13, 2023, Cloudflare’s Cloudforce One Threat Operations Team became aware of a website hosting a Google Android Application (APK) impersonating the legitimate RedAlert - Rocket Alerts application (https://play.google.com/store/apps/details?id=com.red.alert&hl=en&pli=1). More than 5,000 rockets have been launched into Israel since the attacks from Hamas began on October 7th 2023. RedAlert - Rocket Alerts developed by Elad Nava allows individuals to receive timely and precise alerts about incoming airstrikes. Many people living in Israel rely on these alerts to seek safety - a service which has become increasingly important given the newest escalations in the region.
Applications alerting of incoming airstrikes have become targets as only days ago, Pro-Palestinian hacktivist group AnonGhost exploited a vulnerability in another application, “Red Alert: Israel” by Kobi Snir. (https://cybernews.com/cyber-war/israel-redalert-breached-anonghost-hamas/) Their exploit allowed them to intercept requests, expose servers and APIs, and send fake alerts to some app users, including a message that a “nuclear bomb is coming”. AnonGhost also claimed they attacked other rocket alert applications, including RedAlert by Elad Nava. As of October 11, 2023, the RedAlert app was reportedly functioning normally.
In the last two days, a new malicious website (hxxps://redalerts[.]me) has Continue reading
How Prisma saved 98% on distribution costs with Cloudflare R2
The following is a guest post written by Pierre-Antoine Mills, Miguel Fernández, and Petra Donka of Prisma. Prisma provides a server-side library that helps developers read and write data to the database in an intuitive, efficient and safe way.
Prisma’s mission is to redefine how developers build data-driven applications. At its core, Prisma provides an open-source, next-generation TypeScript Object-Relational Mapping (ORM) library that unlocks a new level of developer experience thanks to its intuitive data model, migrations, type-safety, and auto-completion.
Prisma ORM has experienced remarkable growth, engaging a vibrant community of developers. And while it was a great problem to have, this growth was causing an explosion in our AWS infrastructure costs. After investigating a wide range of alternatives, we went with Cloudflare’s R2 storage — and as a result are thrilled that our engine distribution costs have decreased by 98%, while delivering top-notch performance.
It was a natural fit: Prisma is already a proud technology partner of Cloudflare’s, offering deep database integration with Cloudflare Workers. And Cloudflare products provide much of the underlying infrastructure for Prisma Accelerate and Prisma Pulse, empowering user-focused product development. In this post, we’ll dig into how we decided to extend our ongoing Continue reading
HTTP/2 Rapid Reset: deconstructing the record-breaking attack
Starting on Aug 25, 2023, we started to notice some unusually big HTTP attacks hitting many of our customers. These attacks were detected and mitigated by our automated DDoS system. It was not long however, before they started to reach record breaking sizes — and eventually peaked just above 201 million requests per second. This was nearly 3x bigger than our previous biggest attack on record.
Concerning is the fact that the attacker was able to generate such an attack with a botnet of merely 20,000 machines. There are botnets today that are made up of hundreds of thousands or millions of machines. Given that the entire web typically sees only between 1–3 billion requests per second, it's not inconceivable that using this method could focus an entire web’s worth of requests on a small number of targets.
Detecting and Mitigating
This was a novel attack vector at an unprecedented scale, but Cloudflare's existing protections were largely able to absorb the brunt of the attacks. While initially we saw some impact to customer traffic — affecting roughly 1% of requests during the initial wave of attacks — today we’ve Continue reading
HTTP/2 Zero-Day Vulnerability Results in Record-Breaking DDoS Attacks
Earlier today, Cloudflare, along with Google and Amazon AWS, disclosed the existence of a novel zero-day vulnerability dubbed the “HTTP/2 Rapid Reset” attack. This attack exploits a weakness in the HTTP/2 protocol to generate enormous, hyper-volumetric Distributed Denial of Service (DDoS) attacks. Cloudflare has mitigated a barrage of these attacks in recent months, including an attack three times larger than any previous attack we’ve observed, which exceeded 201 million requests per second (rps). Since the end of August 2023, Cloudflare has mitigated more than 1,100 other attacks with over 10 million rps — and 184 attacks that were greater than our previous DDoS record of 71 million rps.
This zero-day provided threat actors with a critical new tool in their Swiss Army knife of vulnerabilities to exploit and attack their victims at a magnitude that has never been seen before. While at times complex and challenging to combat, these attacks allowed Cloudflare the opportunity to develop purpose-built technology to mitigate the effects of the zero-day vulnerability.
If you are using Cloudflare for HTTP DDoS mitigation, you are protected. And below, we’ve included more information on this vulnerability, and Continue reading
HTTP/2 Zero-Day-Sicherheitslücke führt zu rekordverdächtigen DDoS-Angriffen
Heute hat Cloudflare zusammen mit Google und Amazon AWS die Existenz einer neuartigen Zero-Day-Schwachstelle bekannt gegeben, die als „HTTP/2 Rapid Reset“-Angriff bezeichnet wird. Dieser Angriff nutzt eine Schwachstelle im HTTP/2-Protokoll aus, um enorme, hypervolumetrische Distributed Denial of Service (DDoS)-Angriffe zu generieren. Cloudflare hat in den letzten Monaten eine Flut dieser Angriffe abgewehrt, einschließlich eines Angriffs, der dreimal so groß war wie der größte Angriff, den wir bisher jemals verzeichnet hatten, und der 201 Millionen Anfragen pro Sekunde (rps) überstieg. Seit Ende August 2023 hat Cloudflare mehr als 1.100 weitere Angriffe mit über 10 Millionen rps abgewehrt – und 184 Angriffe, die unseren bisherigen DDoS-Rekord von 71 Millionen rps übertrafen.
Diese Zero-Day-Schwachstelle gab den Bedrohungsakteuren ein wichtiges neues Werkzeug in ihrem Werkzeugkasten an Schwachstellen an die Hand, mit dem sie ihre Opfer in einem noch nie dagewesenen Ausmaß ausnutzen und angreifen können. Diese Angriffe waren mitunter komplex und schwierig zu bekämpfen. Cloudflare bot sich dadurch jedoch die Gelegenheit, eine speziell konzipierte Technologie zu entwickeln, um die Auswirkungen der Zero-Day-Schwachstelle abzuwehren.
Wenn Sie Cloudflare für die HTTP-DDoS-Abwehr nutzen, sind Sie geschützt. Im Folgenden finden Sie weitere Continue reading
Resultados de la vulnerabilidad HTTP/2 Zero-Day en ataques DDoS sin precedentes
Hoy temprano, Cloudflare, Google y Amazon AWS, divulgaron la existencia de una nueva vulnerabilidad zero-day que se conoce como ataque “HTTP/2 Rapid Reset”. Este ataque aprovecha un punto débil en el protocolo HTTP/2 para generar enormes ataques hipervolumétricos por denegación de servicio distribuido (DDoS). Cloudflare ha mitigado un aluvión de estos ataques en los últimos meses, incluso uno tres veces más grande que cualquier ataque anterior que hayamos observado, que superó las 201 millones de solicitudes por segundo (rps). Desde fines de agosto de 2023, Cloudflare ha mitigado otros más de 1100 ataques con más de 10 millones de rps — y 184 ataques fueron de una magnitud mayor a nuestro récord de ataques DDoS previos de 71 millones de rps.
Este zero-day brindó a los ciberdelincuentes una nueva herramienta fundamental en su navaja suiza de vulnerabilidades para aprovecharse de sus víctimas y atacarlas a una magnitud que nunca habíamos visto. Si bien a veces estos ataques son complejos y difíciles de combatir, brindaron a Cloudflare la oportunidad de desarrollar tecnología con el propósito de mitigar los efectos de la vulnerabilidad zero-day.