How Cloudflare erroneously throttled a customer’s web traffic
Over the years when Cloudflare has had an outage that affected our customers we have very quickly blogged about what happened, why, and what we are doing to address the causes of the outage. Today’s post is a little different. It’s about a single customer’s website not working correctly because of incorrect action taken by Cloudflare.
Although the customer was not in any way banned from Cloudflare, or lost access to their account, their website didn’t work. And it didn’t work because Cloudflare applied a bandwidth throttle between us and their origin server. The effect was that the website was unusable.
Because of this unusual throttle there was some internal confusion for our customer support team about what had happened. They, incorrectly, believed that the customer had been limited because of a breach of section 2.8 of our Self-Serve Subscription Agreement which prohibits use of our self-service CDN to serve excessive non-HTML content, such as images and video, without a paid plan that includes those services (this is, for example, designed to prevent someone building an image-hosting service on Cloudflare and consuming a huge amount of bandwidth; for that sort of use case we have paid image and video Continue reading
Get notified about the most relevant events with Advanced HTTP Alerts
Today we’re excited to be announcing more flexibility to HTTP alerting, enabling customers to customize the types of activity they’re alerted on and how those alerts are organized.
Prior to today, HTTP alerts at Cloudflare have been very generic. You could choose which Internet properties you wanted and what sensitivity you wanted to be alerted on, but you couldn’t choose anything else. You couldn’t, for example, exclude the IP addresses you use to test things. You couldn’t choose to monitor only a specific path. You couldn’t choose which HTTP statuses you wanted to be alerted on. You couldn’t even choose to monitor your entire account instead of specific zones.
Our customers leverage the Cloudflare network for a myriad of use cases ranging from decreasing bandwidth costs and accelerating asset delivery with Cloudflare CDN to protecting their applications against brute force attacks with Cloudflare Bot Management. Whether the reasons for routing traffic through the Cloudflare network are simple or complex, one powerful capability that comes for free is observability.
With traffic flowing through the network, we can monitor and alert customers about anomalous events such as spikes in origin error rates, enabling them to investigate further and mitigate any issues as Continue reading
Manage and control the use of dedicated egress IPs with Cloudflare Zero Trust
Before identity-driven Zero Trust rules, some SaaS applications on the public Internet relied on the IP address of a connecting user as a security model. Users would connect from known office locations, with fixed IP address ranges, and the SaaS application would check their address in addition to their login credentials.
Many systems still offer that second factor method. Customers of Cloudflare One can use a dedicated egress IP for this purpose as part of their journey to a Zero Trust model. Unlike other solutions, customers using this option do not need to deploy any infrastructure of their own. However, not all traffic needs to use those dedicated egress IPs.
Today, we are announcing policies that give administrators control over when Cloudflare uses their dedicated egress IPs. Specifically, administrators can use a rule builder in the Cloudflare dashboard to determine which egress IP is used and when, based on attributes like identity, application, IP address, and geolocation. This capability is available to any enterprise-contracted customer that adds on dedicated egress IPs to their Zero Trust subscription.
Why did we build this?
In today’s hybrid work environment, organizations aspire for more consistent security and IT experiences to manage their employees’ traffic Continue reading
Cloudflare’s handling of a bug in interpreting IPv4-mapped IPv6 addresses
In November 2022, our bug bounty program received a critical and very interesting report. The report stated that certain types of DNS records could be used to bypass some of our network policies and connect to ports on the loopback address (e.g. 127.0.0.1) of our servers. This post will explain how we dealt with the report, how we fixed the bug, and the outcome of our internal investigation to see if the vulnerability had been previously exploited.
RFC 4291 defines ways to embed an IPv4 address into IPv6 addresses. One of the methods defined in the RFC is to use IPv4-mapped IPv6 addresses, that have the following format:
| 80 bits | 16 | 32 bits |
+--------------------------------------+--------------------------+
|0000..............................0000|FFFF| IPv4 address |
+--------------------------------------+----+---------------------+
In IPv6 notation, the corresponding mapping for 127.0.0.1 is ::ffff:127.0.0.1 (RFC 4038)
The researcher was able to use DNS entries based on mapped addresses to bypass some of our controls and access ports on the loopback address or non-routable IPs.
This vulnerability was reported on November 27 to our bug bounty program. Our Security Incident Response Team (SIRT) was contacted, and incident response activities Continue reading
Uptick in healthcare organizations experiencing targeted DDoS attacks
Healthcare in the crosshairs
Over the past few days, Cloudflare, as well as other sources, have observed healthcare organizations targeted by a pro-Russian hacktivist group claiming to be Killnet. There has been an increase in the amount of healthcare organizations coming to us to help get out from under these types of attacks. Multiple healthcare organizations behind Cloudflare have also been targeted by HTTP DDoS attacks and Cloudflare has helped them successfully mitigate these attacks. The United States Department of Health and Human Services issued an Analyst Note detailing the threat of Killnet-related cyberattacks to the healthcare industry.
A rise in political tensions and escalation of the conflict in Ukraine are all factors that play into the current cybersecurity threat landscape. Unlike traditional warfare, the Internet has enabled and empowered groups of individuals to carry out targeted attacks regardless of their location or involvement. Distributed-denial-of-Service (DDoS) attacks have the unfortunate advantage of not requiring an intrusion or a foothold to be launched and have, unfortunately, become more accessible than ever before.
The attacks observed by the Cloudflare global network do not show a clear indication that they are originating from a single botnet and the attack methods and sources Continue reading
CVE-2022-47929: traffic control noqueue no problem?
USER namespaces power the functionality of our favorite tools such as docker, podman, and kubernetes. We wrote about Linux namespaces back in June and explained them like this:
Most of the namespaces are uncontroversial, like the UTS namespace which allows the host system to hide its hostname and time. Others are complex but straightforward - NET and NS (mount) namespaces are known to be hard to wrap your head around. Finally, there is this very special, very curious USER namespace. USER namespace is special since it allows the - typically unprivileged owner to operate as "root" inside it. It's a foundation to having tools like Docker to not operate as true root, and things like rootless containers.
Due to its nature, allowing unprivileged users access to USER namespace always carried a great security risk. With its help the unprivileged user can in fact run code that typically requires root. This code is often under-tested and buggy. Today we will look into one such case where USER namespaces are leveraged to exploit a kernel bug that can result in an unprivileged denial of service attack.
Enter Linux Traffic Control queue disciplines
In 2019, we were exploring leveraging Linux Traffic Control's queue Continue reading
Cyberattacks on Holocaust educational websites increased in 2022
Today we mark the International Holocaust Remembrance Day. We commemorate the victims that were robbed of their possessions, stripped of their rights, deported, starved, dehumanized and murdered by the Nazis and their accomplices. During the Holocaust and in the events that led to it, the Nazis exterminated one third of the European Jewish population. Six million Jews, along with countless other members of minority and disability groups, were murdered because the Nazis believed they were inferior.
Seventy eight years later, after the liberation of the infamous Auschwitz death camp, antisemitism still burns with hatred. According to a study performed by the Campaign Against Antisemitism organization on data provided by the UK Home Office, Jews are 500% more likely to be targeted by hate crime than any other faith group per capita.
Cyberattacks targeting Holocaust educational websites
From Cloudflare’s vantage point we can point to distressing findings as well. In 2021, cyberattacks on Holocaust educational websites doubled year over year. In 2021, one out of every 100 HTTP requests sent to Holocaust educational websites behind Cloudflare was part of an attack. In 2022, the share of those cyber attacks grew again by 49% YoY. Cyberattacks represented 1.6% of all Continue reading
Inside Geo Key Manager v2: re-imagining access control for distributed systems
In December 2022 we announced the closed beta of the new version of Geo Key Manager. Geo Key Manager v2 (GeoV2) is the next step in our journey to provide customers with a secure and flexible way to control the distribution of their private keys by geographic location. Our original system, Geo Key Manager v1, was launched as a research project in 2017, but as customer needs evolved and our scale increased, we realized that we needed to make significant improvements to provide a better user experience.
One of the principal challenges we faced with Geo Key Manager v1 (GeoV1) was the inflexibility of our access control policies. Customers required richer data localization, often spurred by regulatory concerns. Internally, events such as the conflict in Ukraine reinforced the need to be able to quickly restrict access to sensitive key material. Geo Key Manager v1’s underlying cryptography was a combination of identity-based broadcast encryption and identity-based revocation that simulated a subset of the functionality offered by Attribute-Based Encryption (ABE). Replacing this with an established ABE scheme addressed the inflexibility of our access control policies and provided a more secure foundation for our system.
Unlike our previous scheme, which limited future Continue reading
Towards a global framework for cross-border data flows and privacy protection
As our societies and economies rely more and more on digital technologies, there is an increased need to share and transfer data, including personal data, over the Internet. Cross-border data flows have become essential to international trade and global economic development. In fact, the digital transformation of the global economy could never have happened as it did without the open and global architecture of the Internet and the ability for data to transcend national borders. As we described in our blog post yesterday, data localization doesn’t necessarily improve data privacy. Actually, there can be real benefits to data security and - by extension - privacy if we are able to transfer data across borders. So with Data Privacy Day coming up tomorrow, we wanted to take this opportunity to drill down into the current environment for the transfer of personal data from the EU to the US, which is governed by the EU’s privacy regulation (GDPR). Looking to the future, we will make the case for a more stable, global cross-border data transfer framework, which will be critical for an open, more secure and more private Internet.
The privacy challenge to cross-border data flows
In the last decade, we have Continue reading
Navigating the changing data localization landscape with Cloudflare’s Data Localization Suite
This post is also available in Português.
At Cloudflare, we believe that deploying effective cybersecurity measures is the best way to protect the privacy of personal information and can be more effective than making sure that information stays within a particular jurisdiction. Yet, we hear from customers in Europe, India, Australia, Japan, and many other regions that, as part of their privacy programs, they need solutions to localize data in order to meet their regulatory obligations.
So as we think about Data Privacy Day, which is coming up on January 28, we are in the interesting position of disagreeing with those who believe that data localization is a proxy for better data privacy, but of also wanting to support our customers who have to comply with certain regulations.
For this reason, we introduced our Data Localization Suite (DLS) in 2020 to help customers navigate a data protection landscape that focuses more and more on data localization. With the DLS, customers can use Cloudflare’s powerful global network and security measures to protect their businesses, while keeping the data we process on their behalf local. Since its launch, we’ve had many customers adopt the Data Localization Suite. In this blog post we Continue reading
Investing in security to protect data privacy
If you’ve made it to 2023 without ever receiving a notice that your personal information was compromised in a security breach, consider yourself lucky. In a best case scenario, bad actors only got your email address and name – information that won’t cause you a huge amount of harm. Or in a worst-case scenario, maybe your profile on a dating app was breached and intimate details of your personal life were exposed publicly, with life-changing impacts. But there are also more hidden, insidious ways that your personal data can be exploited. For example, most of us use an Internet Service Provider (ISP) to connect to the Internet. Some of those ISPs are collecting information about your Internet viewing habits, your search histories, your location, etc. – all of which can impact the privacy of your personal information as you are targeted with ads based on your online habits.
You also probably haven’t made it to 2023 without hearing at least something about Internet privacy laws around the globe. In some jurisdictions, lawmakers are driven by a recognition that the right to privacy is a fundamental human right. In other locations, lawmakers are passing laws to address the harms their citizens Continue reading
Armed to Boot: an enhancement to Arm’s Secure Boot chain
Over the last few years, there has been a rise in the number of attacks that affect how a computer boots. Most modern computers use a specification called Unified Extensible Firmware Interface (UEFI) that defines a software interface between an operating system (e.g. Windows) and platform firmware (e.g. disk drives, video cards). There are security mechanisms built into UEFI that ensure that platform firmware can be cryptographically validated and boot securely through an application called a bootloader. This firmware is stored in non-volatile SPI flash memory on the motherboard, so it persists on the system even if the operating system is reinstalled and drives are replaced.
This creates a ‘trust anchor’ used to validate each stage of the boot process, but, unfortunately, this trust anchor is also a target for attack. In these UEFI attacks, malicious actions are loaded onto a compromised device early in the boot process. This means that malware can change configuration data, establish persistence by ‘implanting’ itself, and can bypass security measures that are only loaded at the operating system stage. So, while UEFI-anchored secure boot protects the bootloader from bootloader attacks, it does not protect the UEFI firmware itself.
Cloudflare Incident on January 24th, 2023
Several Cloudflare services became unavailable for 121 minutes on January 24th, 2023 due to an error releasing code that manages service tokens. The incident degraded a wide range of Cloudflare products including aspects of our Workers platform, our Zero Trust solution, and control plane functions in our content delivery network (CDN).
Cloudflare provides a service token functionality to allow automated services to authenticate to other services. Customers can use service tokens to secure the interaction between an application running in a data center and a resource in a public cloud provider, for example. As part of the release, we intended to introduce a feature that showed administrators the time that a token was last used, giving users the ability to safely clean up unused tokens. The change inadvertently overwrote other metadata about the service tokens and rendered the tokens of impacted accounts invalid for the duration of the incident.
The reason a single release caused so much damage is because Cloudflare runs on Cloudflare. Service tokens impact the ability for accounts to authenticate, and two of the impacted accounts power multiple Cloudflare services. When these accounts’ service tokens were overwritten, the services that run on these accounts began to experience Continue reading
Intelligent, automatic restarts for unhealthy Kafka consumers
At Cloudflare, we take steps to ensure we are resilient against failure at all levels of our infrastructure. This includes Kafka, which we use for critical workflows such as sending time-sensitive emails and alerts.
We learned a lot about keeping our applications that leverage Kafka healthy, so they can always be operational. Application health checks are notoriously hard to implement: What determines an application as healthy? How can we keep services operational at all times?
These can be implemented in many ways. We’ll talk about an approach that allows us to considerably reduce incidents with unhealthy applications while requiring less manual intervention.
Kafka at Cloudflare
Cloudflare is a big adopter of Kafka. We use Kafka as a way to decouple services due to its asynchronous nature and reliability. It allows different teams to work effectively without creating dependencies on one another. You can also read more about how other teams at Cloudflare use Kafka in this post.
Kafka is used to send and receive messages. Messages represent some kind of event like a credit card payment or details of a new user created in your platform. These messages can be represented in multiple ways: JSON, Protobuf, Avro and so on.
Internet disruptions overview for Q4 2022
Cloudflare operates in more than 250 cities in over 100 countries, where we interconnect with over 10,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions.
While Internet disruptions are never convenient, online interest in the 2022 World Cup in mid-November and the growth in online holiday shopping in many areas during November and December meant that connectivity issues could be particularly disruptive. Having said that, the fourth quarter appeared to be a bit quieter from an Internet disruptions perspective, although Iran and Ukraine continued to be hotspots, as we discuss below.
Government directed
Multi-hour Internet shutdowns are frequently used by authoritarian governments in response to widespread protests as a means of limiting communications among protestors, as well preventing protestors from sharing information and video with the outside world. During the fourth quarter Cuba and Sudan again implemented such shutdowns, while Iran continued the series of “Internet curfews” across mobile networks it started in mid-September, in addition to implementing several other regional Internet shutdowns.
Cuba
Introducing Waiting Room Bypass Rules
Leveraging the power and versatility of Cloudflare's Ruleset Engine, Waiting Room now offers customers more fine-tuned control over their waiting room traffic. Queue only the traffic you want to with Waiting Room Bypass Rules, now available to all Enterprise customers with an Advanced Purchase of Waiting Room.
Customers depend on Waiting Room for always-on protection from unexpected and overwhelming traffic surges that would otherwise bring their site down. Waiting Room places excess users in a fully customizable virtual waiting room, admitting new visitors dynamically as spots become available on a customer’s site. Instead of throwing error pages or delivering poorly-performing site pages, Waiting Room empowers customers to take control of their end-user experience during unmanageable traffic surges.
Additionally, customers use Waiting Room Event Scheduling to manage user flow and ensure reliable site performance before, during, and after online events such as product restocks, seasonal sales, and ticket sales. With Event Scheduling, customers schedule changes to their waiting rooms' settings and custom queuing page ahead of time, with options to pre-queue early arrivers and offload event traffic from their origins after the event has concluded.
As part of Continue reading
Three new winners of Project Jengo, and more defeats for the patent troll
Project Jengo is a Cloudflare effort to fight back against patent trolls by flipping the incentive structure that has encouraged the growth of patent trolls who extract settlements out of companies using frivolous lawsuits. We do this by asking the public to identify prior art that can invalidate any of the patents that a troll holds – not just the ones that are asserted against Cloudflare.
Since we launched Project Jengo over five years ago, we’ve given out over $135,000 to individuals who helped us find prior art to invalidate patents owned by patent trolls. By invalidating those patents – many of which are so blatantly marginal or broad that they never should have been granted in the first place – we hope to decrease the amount of harassment and frivolous lawsuits that patent trolls bring against innovative technology companies.
Today, we’re excited to announce three new Project Jengo winners. These individuals have helped us push forward our effort to take down patent trolls, and continue to fight trolling in favor of innovation.
The patent troll
The current case involves a patent troll called Sable Networks who asserted four patents that generally describe a flow-based router or a mechanism Continue reading
A debugging story: corrupt packets in AF_XDP; a kernel bug or user error?
panic: Invalid TCP packet: Truncated
A few months ago we started getting a handful of crash reports for flowtrackd, our Advanced TCP Protection system that runs on our global network. The provided stack traces indicated that the panics occurred while parsing a TCP packet that was truncated.
What was most interesting wasn’t that we failed to parse the packet. It isn’t rare that we receive malformed packets from the Internet that are (deliberately or not) truncated. Those packets will be caught the first time we parse them and won’t make it to the latter processing stages. However, in our case, the panic occurred the second time we parsed the packet, indicating it had been truncated after we received it and successfully parsed it the first time. Both parse calls were made from a single green thread and referenced the same packet buffer in memory, and we made no attempts to mutate the packet in between.
It can be easy to dread discovering a bug like this. Is there a race condition? Is there memory corruption? Is this a kernel bug? A compiler bug? Our plan to get to the root cause of this potentially complex issue was to identify symptom(s) Continue reading
Cloud CNI privately connects your clouds to Cloudflare
This post is also available in 简体中文, 日本語 and Español.
For CIOs, networking is a hard process that is often made harder. Corporate networks have so many things that need to be connected and each one of them needs to be connected differently: user devices need managed connectivity through a Secure Web Gateway, offices need to be connected using the public Internet or dedicated connectivity, data centers need to be managed with their own private or public connectivity, and then you have to manage cloud connectivity on top of it all! It can be exasperating to manage connectivity for all these different scenarios and all their privacy and compliance requirements when all you want to do is enable your users to access their resources privately, securely, and in a non-intrusive manner.
Cloudflare helps simplify your connectivity story with Cloudflare One. Today, we’re excited to announce that we support direct cloud interconnection with our Cloudflare Network Interconnect, allowing Cloudflare to be your one-stop shop for all your interconnection needs.
Customers using IBM Cloud, Google Cloud, Azure, Oracle Cloud Infrastructure, and Amazon Web Services can now open direct connections from their private cloud instances into Cloudflare. In this blog, we’re going Continue reading
CIO Week 2023 recap
This post is also available in 日本語, 简体中文, and Español.
In our Welcome to CIO Week 2023 post, we talked about wanting to start the year by celebrating the work Chief Information Officers do to keep their organizations safe and productive.
Over the past week, you learned about announcements addressing all facets of your technology stack – including new services, betas, strategic partnerships, third party integrations, and more. This recap blog summarizes each announcement and labels what capability is generally available (GA), in beta, or on our roadmap.
We delivered on critical capabilities requested by our customers – such as even more comprehensive phishing protection and deeper integrations with the Microsoft ecosystem. Looking ahead, we also described our roadmap for emerging technology categories like Digital Experience Monitoring and our vision to make it exceedingly simple to route traffic from any source to any destination through Cloudflare’s network.
Everything we launched is designed to help CIOs accelerate their pursuit of digital transformation. In this blog, we organized our announcement summaries based on the three feelings we want CIOs to have when they consider partnering with Cloudflare:
- CIOs now have a simpler roadmap to Zero Trust and SASE: We announced Continue reading