Effects of the conflict in Sudan on Internet patterns
On Saturday, April 15, 2023, an armed conflict between rival factions of the military government of Sudan began. Cloudflare observed a disruption in Internet traffic on that Saturday, starting at 08:00 UTC, which deepened on Sunday. Since then, the conflict has continued, and different ISPs have been affected, in some cases with a 90% drop in traffic. On May 2, Internet traffic is still ~30% lower than pre-conflict levels. This blog post will show what we’ve been seeing in terms of Internet disruption there.
On the day that clashes broke out, our data shows that traffic in the country dropped as much as 60% on Saturday, after 08:00 UTC, with a partial recovery on Sunday around 14:00, but it has consistently been lower than before. Although we saw outages and disruptions on major local Internet providers, the general drop in traffic could also be related to different human usage patterns because of the conflict, with people trying to leave the country. In Ukraine, we saw a clear drop in traffic, not always related to ISP outages, after the war started, when people were leaving the country.
Here’s the hourly perspective of Sudan’s Internet traffic over the past weeks as seen Continue reading
Celebrating Australia’s Privacy Awareness Week 2023
When a country throws a privacy party, Cloudflare is there! We are proud to be an official sponsor of the Australian Privacy Awareness Week 2023, and we think this year’s theme of “Privacy 101: Back to Basics” is more important now than ever. In recent months, Australians have been hit with the news of massive personal data privacy breaches where millions of Australian citizens' private and sensitive data was compromised, seemingly easily. Meanwhile, the Australian Attorney General released its Privacy Act Review Report 2022 earlier this year, calling for a number of changes to Australia’s privacy regulations.
You’re probably familiar with the old-school privacy basics of giving users notice and consent. But we think it’s time for some new “privacy basics”. Thanks to rapid developments in new technologies and new security threat vectors, notice and consent can only go so far to protect the privacy of your personal data. New challenges call for new solutions: security solutions and privacy enhancing technologies to keep personal data protected. Cloudflare is excited to play a role in building and using these technologies to help our customers keep their sensitive information private and enable individual consumers to protect themselves. Investing in and offering these Continue reading
SLP: a new DDoS amplification vector in the wild
Earlier today, April 25, 2023, researchers Pedro Umbelino at Bitsight and Marco Lux at Curesec published their discovery of CVE-2023-29552, a new DDoS reflection/amplification attack vector leveraging the SLP protocol. If you are a Cloudflare customer, your services are already protected from this new attack vector.
Service Location Protocol (SLP) is a “service discovery” protocol invented by Sun Microsystems in 1997. Like other service discovery protocols, it was designed to allow devices in a local area network to interact without prior knowledge of each other. SLP is a relatively obsolete protocol and has mostly been supplanted by more modern alternatives like UPnP, mDNS/Zeroconf, and WS-Discovery. Nevertheless, many commercial products still offer support for SLP.
Since SLP has no method for authentication, it should never be exposed to the public Internet. However, Umbelino and Lux have discovered that upwards of 35,000 Internet endpoints have their devices’ SLP service exposed and accessible to anyone. Additionally, they have discovered that the UDP version of this protocol has an amplification factor of up to 2,200x, which is the third largest discovered to-date.
Cloudflare expects the prevalence of SLP-based DDoS attacks to rise significantly in the coming weeks as malicious actors learn how to exploit Continue reading
Why I joined Cloudflare as Chief Security Officer
I am absolutely thrilled and feel incredibly blessed to have joined Cloudflare as Chief Security Officer (CSO). Cybersecurity has always been my passion and focus of my career. I am grateful to join such a dynamic and innovative team. Cloudflare is a cybersecurity industry leader and offers unmatched technology that is second to none.
A little about me
I have been a CSO for over 20 years in the financial and private sectors with SVB, HSBC, McAfee, Ameren, and Scottrade. I have been privileged to lead the security teams of some of the world's largest, most complex, and most innovative companies; however, my greatest honor has been working with and collaborating among some of the world's most amazing people. I have learned my dedication, expertise, and passion from my leaders, peers, and teams, which have taught me how to build and lead world-class security programs that protect organizations from the most sophisticated threats. Because security is constantly evolving, the key is, and always will be, to build an active, diverse community of highly empathetic people that will successfully protect the organization.
My charter
As I step into my new role as CSO at Cloudflare, I am excited to take on Continue reading
Secure by default: recommendations from the CISA’s newest guide, and how Cloudflare follows these principles to keep you secure
When you buy a new house, you shouldn’t have to worry that everyone in the city can unlock your front door with a universal key before you change the lock. You also shouldn’t have to walk around the house with a screwdriver and tighten the window locks and back door so that intruders can’t pry them open. And you really shouldn’t have to take your alarm system offline every few months to apply critical software updates that the alarm vendor could have fixed with better software practices before they installed it.
Similarly, you shouldn’t have to worry that when you buy a network discovery tool it can be accessed by any attacker until you change the password, or that your expensive hardware-based firewalls can be recruited to launch DDoS attacks or run arbitrary code without the need to authenticate.
This “default secure” posture is the focus of a recently published guide jointly authored by the Cybersecurity and Infrastructure Agency (CISA), NSA, FBI, and six other international agencies representing the United Kingdom, Australia, Canada, Germany, Netherlands, and New Zealand. In the guide, the authors implore technology vendors to follow Secure-by-Design and Secure-by-Default principles, shifting the burden of security as much Continue reading
Oxy: Fish/Bumblebee/Splicer subsystems to improve reliability
At Cloudflare, we are building proxy applications on top of Oxy that must be able to handle a huge amount of traffic. Besides high performance requirements, the applications must also be resilient against crashes or reloads. As the framework evolves, the complexity also increases. While migrating WARP to support soft-unicast (Cloudflare servers don't own IPs anymore), we needed to add different functionalities to our proxy framework. Those additions increased not only the code size but also resource usage and states required to be preserved between process upgrades.
To address those issues, we opted to split a big proxy process into smaller, specialized services. Following the Unix philosophy, each service should have a single responsibility, and it must do it well. In this blog post, we will talk about how our proxy interacts with three different services - Splicer (which pipes data between sockets), Bumblebee (which upgrades an IP flow to a TCP socket), and Fish (which handles layer 3 egress using soft-unicast IPs). Those three services help us to improve system reliability and efficiency as we migrated WARP to support soft-unicast.
Splicer
Most transmission tunnels in our proxy forward packets without making any modifications. In other words, given Continue reading
DDR4 memory organization and how it affects memory bandwidth
When shopping for DDR4 memory modules, we typically look at the memory density and memory speed. For example a 32GB DDR4-2666 memory module has 32GB of memory density, and the data rate transfer speed is 2666 mega transfers per second (MT/s).
If we take a closer look at the selection of DDR4 memories, we will then notice that there are several other parameters to choose from. One of them is rank x organization, for example 1Rx8, 2Rx4, 2Rx8 and so on. What are these and does memory module rank and organization have an effect on DDR4 module performance?
In this blog, we will study the concepts of memory rank and organization, and how memory rank and organization affect the memory bandwidth performance by reviewing some benchmarking test results.
Memory rank
Memory rank is a term that is used to describe how many sets of DRAM chips, or devices, exist on a memory module. A set of DDR4 DRAM chips is always 64-bit wide, or 72-bit wide if ECC is supported. Within a memory rank, all chips share the address, command and control signals.
The concept of memory rank is very similar to memory bank. Memory rank is a term used Continue reading
Consent management made easy and clear with Cloudflare Zaraz
Depending on where you live you may be asked to agree to the use of cookies when visiting a website for the first time. And if you've ever clicked something other than Approve you'll have noticed that the list of choices about which services should or should not be allowed to use cookies can be very, very long. That's because websites typically incorporate numerous third party tools for tracking, A/B testing, retargeting, etc. – and your consent is needed for each one of them.
For website owners it's really hard to keep track of which third party tools are used and whether they've asked end users about all of them. There are tools that help you load third-party scripts on your website, and there are tools that help you manage and gather consent. Making the former respect the choices made in the latter is often cumbersome, to say the least.
This changes with Cloudflare Zaraz, a solution that makes third-party tools secure and fast, and that now can also help you with gathering and managing consent. Using the Zaraz Consent Manager, you can easily collect users’ consent preferences on your website, using a consent modal, and apply your consent policy Continue reading
Measuring network quality to better understand the end-user experience
You’re visiting your family for the holidays and you connect to the WiFi, and then notice Netflix isn’t loading as fast as it normally does. You go to speed.cloudflare.com, fast.com, speedtest.net, or type “speed test” into Google Chrome to figure out if there is a problem with your Internet connection, and get something that looks like this:
If you want to see what that looks like for you, try it yourself here. But what do those numbers mean? How do those numbers relate to whether or not your Netflix isn’t loading or any of the other common use cases: playing games or audio/video chat with your friends and loved ones? Even network engineers find that speed tests are difficult to relate to the user experience of… using the Internet..
Amazingly, speed tests have barely changed in nearly two decades, even though the way we use the Internet has changed a lot. With so many more people on the Internet, the gaps between speed tests and the user’s experience of network quality are growing. The problem is so important that the Internet’s standards organization is paying attention, too.
From a high-level, there are three grand network Continue reading
Making home Internet faster has little to do with “speed”
More than ten years ago, researchers at Google published a paper with the seemingly heretical title “More Bandwidth Doesn’t Matter (much)”. We published our own blog showing it is faster to fly 1TB of data from San Francisco to London than it is to upload it on a 100 Mbps connection. Unfortunately, things haven’t changed much. When you make purchasing decisions about home Internet plans, you probably consider the bandwidth of the connection when evaluating Internet performance. More bandwidth is faster speed, or so the marketing goes. In this post, we’ll use real-world data to show both bandwidth and – spoiler alert! – latency impact the speed of an Internet connection. By the end, we think you’ll understand why Cloudflare is so laser focused on reducing latency everywhere we can find it.
First, we should quickly define bandwidth and latency. Bandwidth is the amount of data that can be transmitted at any single time. It’s the maximum throughput, or capacity, of the communications link between two servers that want to exchange data. Usually, the bottleneck – the place in the network where the connection is constrained by the amount of bandwidth available – is in the “last mile”, either the Continue reading
Cloudflare One named in Gartner® Magic Quadrant™ for Security Service Edge
Gartner has recognized Cloudflare in the 2023 “Gartner® Magic Quadrant™ for Security Service Edge (SSE)” report for its ability to execute and completeness of vision. We are excited to share that the Cloudflare Zero Trust solution, part of our Cloudflare One platform, is one of only ten vendors recognized in the report.
Of the 10 companies named to this year’s Gartner® Magic Quadrant™ report, Cloudflare is the only new vendor addition. You can read more about our position in the report and what customers say about using Cloudflare One here.
Cloudflare is also the newest vendor when measured by the date since our first products in the SSE space launched. We launched Cloudflare Access, our best-in-class Zero Trust access control product, a little less than five years ago. Since then, we have released hundreds of features and shipped nearly a dozen more products to create a comprehensive SSE solution that over 10,000 organizations trust to keep their organizations data, devices and teams both safe and fast. We moved that quickly because we built Cloudflare One on top of the same network that already secures and accelerates large segments of the Internet today.
We deliver our SSE services on the Continue reading
Introducing Cloudflare’s new Network Analytics dashboard
We’re pleased to introduce Cloudflare’s new and improved Network Analytics dashboard. It’s now available to Magic Transit and Spectrum customers on the Enterprise plan.
The dashboard provides network operators better visibility into traffic behavior, firewall events, and DDoS attacks as observed across Cloudflare’s global network. Some of the dashboard’s data points include:
- Top traffic and attack attributes
- Visibility into DDoS mitigations and Magic Firewall events
- Detailed packet samples including full packets headers and metadata
This dashboard was the outcome of a full refactoring of our network-layer data logging pipeline. The new data pipeline is decentralized and much more flexible than the previous one — making it more resilient, performant, and scalable for when we add new mitigation systems, introduce new sampling points, and roll out new services. A technical deep-dive blog is coming soon, so stay tuned.
In this blog post, we will demonstrate how the dashboard helps network operators:
- Understand their network better
- Respond to DDoS attacks faster
- Easily generate security reports for peers and managers
Understand your network better
One of the main responsibilities network operators bare is ensuring the operational stability Continue reading
Internet disruptions overview for Q1 2023
Cloudflare operates in more than 285 cities in over 100 countries, where we interconnect with over 11,500 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions.
We entered 2023 with Internet disruptions due to causes that ran the gamut, including several government-directed Internet shutdowns, cyclones, a massive earthquake, power outages, cable cuts, cyberattacks, technical problems, and military action. As we have noted in the past, this post is intended as a summary overview of observed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter.
Government directed
Iran
Over the last six-plus months, government-directed Internet shutdowns in Iran have largely been in response to protests over the death of Mahsa Amini while in police custody. While these shutdowns are still occurring in a limited fashion, a notable shutdown observed in January was intended to prevent cheating on academic exams. Internet shutdowns with a similar purpose have been observed across a number of other countries, and have also occurred in Continue reading
DDoS threat report for 2023 Q1
Welcome to the first DDoS threat report of 2023. DDoS attacks, or distributed denial-of-service attacks, are a type of cyber attack that aim to overwhelm Internet services such as websites with more traffic than they can handle, in order to disrupt them and make them unavailable to legitimate users. In this report, we cover the latest insights and trends about the DDoS attack landscape as we observed across our global network.
Kicking off 2023 with a bang
Threat actors kicked off 2023 with a bang. The start of the year was characterized by a series of hacktivist campaigns against Western targets including banking, airports, healthcare and universities — mainly by the pro-Russian Telegram-organized groups Killnet and more recently by AnonymousSudan.
While Killnet-led and AnonymousSudan-led cyberattacks stole the spotlight, we haven’t witnessed any novel or exceedingly large attacks by them.
Hyper-volumetric attacks
We did see, however, an increase of hyper-volumetric DDoS attacks launched by other threat actors — with the largest one peaking above 71 million requests per second (rps) — exceeding Google’s previous world record of 46M rps by 55%.
Back to Killnet and AnonymousSudan, while no noteworthy attacks were reported, we shouldn't underestimate the potential risks. Unprotected Internet Continue reading
Protect your domain with Zone Holds
Today, we are announcing Zone Holds, a new capability for enterprise customers that gives them control of if and when someone else can add the same zone to another Cloudflare account. When multiple teams at a company want to use Cloudflare, one team might accidentally step on another’s toes and try to manage the same zone in two accounts. Zone Holds ensure that this cannot happen by enforcing that only one account can contain a given domain, optionally inclusive of subdomains or custom hostnames, unless explicit permission is granted by the account owner of the zone.
What can go wrong today
Cloudflare already requires zones to be authenticated via DNS before traffic is proxied through our global network. This ensures that only domain owners can authorize traffic to be sent through and controlled with Cloudflare. However, many of our customers are large organizations with many teams all trying to protect and accelerate their web properties. In these cases, one team may not realize that a given domain is already being protected with Cloudflare. If they activate a second instance of the same domain in Cloudflare, they end up replacing the original zone that another team was already managing with Cloudflare. Continue reading
Cloudflare’s view of the Virgin Media outage in the UK
Just after midnight (UTC) on April 4, subscribers to UK ISP Virgin Media (AS5089) began experiencing an Internet outage, with subscriber complaints multiplying rapidly on platforms including Twitter and Reddit.
Cloudflare Radar data shows Virgin Media traffic dropping to near-zero around 00:30 UTC, as seen in the figure below. Connectivity showed some signs of recovery around 02:30 UTC, but fell again an hour later. Further nominal recovery was seen around 04:45 UTC, before again experiencing another complete outage between around 05:45-06:45 UTC, after which traffic began to recover, reaching expected levels around 07:30 UTC.
After the initial set of early-morning disruptions, Virgin Media experienced another round of issues in the afternoon. Cloudflare observed instability in traffic from Virgin Media’s network (called an autonomous system in Internet jargon) AS5089 starting around 15:00 UTC, with a significant drop just before 16:00 UTC. However in this case, it did not appear to be a complete outage, with traffic recovering approximately a half hour later.
Virgin Media’s Twitter account acknowledged the early morning disruption several hours after it began, posting responses stating “We’re aware of an issue that is affecting broadband services for Virgin Media customers as well as our contact centres. Our teams Continue reading
Oxy: the journey of graceful restarts
Any software under continuous development and improvement will eventually need a new version deployed to the systems running it. This can happen in several ways, depending on how much you care about things like reliability, availability, and correctness. When I started out in web development, I didn’t think about any of these qualities; I simply blasted my new code over FTP directly to my /cgi-bin/ directory, which was the style at the time. For those of us producing desktop software, often you sidestep this entirely by having the user save their work, close the program and install an update – but they usually get to decide when this happens.
At Cloudflare we have to take this seriously. Our software is in constant use and cannot simply be stopped abruptly. A dropped HTTP request can cause an entire webpage to load incorrectly, and a broken connection can kick you out of a video call. Taking away reliability creates a vacuum filled only by user frustration.
The limitations of the typical upgrade process
There is no one right way to upgrade software reliably. Some programming languages and environments make it easier than others, but in a Turing-complete language few things are impossible.
mTLS client certificate revocation vulnerability with TLS Session Resumption
On December 16, 2022, Cloudflare discovered a bug where, in limited circumstances, some users with revoked certificates may not have been blocked by Cloudflare firewall settings. Specifically, Cloudflare’s Firewall Rules solution did not block some users with revoked certificates from resuming a session via mutual transport layer security (mTLS), even if the customer had configured Firewall Rules to do so. This bug has been mitigated, and we have no evidence of this being exploited. We notified any customers that may have been impacted in an abundance of caution, so they can check their own logs to determine if an mTLS protected resource was accessed by entities holding a revoked certificate.
What happened?
One of Cloudflare Firewall Rules’ features, introduced in March 2021, lets customers revoke or block a client certificate, preventing it from being used to authenticate and establish a session. For example, a customer may use Firewall Rules to protect a service by requiring clients to provide a client certificate through the mTLS authentication protocol. Customers could also revoke or disable a client certificate, after which it would no longer be able to be used to authenticate a party initiating an encrypted session via mTLS.
When Cloudflare receives Continue reading
Introducing Rollbacks for Workers Deployments
In November, 2022, we introduced deployments for Workers. Deployments are created as you make changes to a Worker. Each one is unique. These let you track changes to your Workers over time, seeing who made the changes, and where they came from.
When we made the announcement, we also said our intention was to build more functionality on top of deployments.
Today, we’re proud to release rollbacks for deployments.
Rollbacks
As nice as it would be to know that every deployment is perfect, it’s not always possible - for various reasons. Rollbacks provide a quick way to deploy past versions of a Worker - providing another layer of confidence when developing and deploying with Workers.
Via the dashboard
In the dashboard, you can navigate to the Deployments tab. For each deployment that’s not the most recent, you should see a new icon on the far right of the deployment. Hovering over that icon will display the option to rollback to the specified deployment.
Clicking on that will bring up a confirmation dialog, where you can enter a reason for rollback. This provides another mechanism of record-keeping and helps give more context for why the rollback was necessary.
Once you enter Continue reading