One year of war in Ukraine: Internet trends, attacks, and resilience
The Internet has become a significant factor in geopolitical conflicts, such as the ongoing war in Ukraine. Tomorrow marks one year since the Russian invasion of that country. This post reports on Internet insights and discusses how Ukraine's Internet remained resilient in spite of dozens of disruptions in three different stages of the conflict.
Key takeaways:
- Internet traffic shifts in Ukraine are clearly visible from east to west as Ukrainians fled the war, with country-wide traffic dropping as much as 33% after February 24, 2022.
- Air strikes on energy infrastructure starting in October led to widespread Internet disruptions that continue in 2023.
- Application-layer cyber attacks in Ukraine rose 1,300% in early March 2022 compared to pre-war levels.
- Government administration, financial services, and the media saw the most attacks targeting Ukraine.
- Traffic from a number of networks in Kherson was re-routed through Russia between June and October, subjecting traffic to Russia’s restrictions and limitations, including content filtering. Even after traffic ceased to reroute through Russia, those Ukrainian networks saw major outages through at least the end of the year, while two networks remain offline.
- Through efforts on the ground to repair damaged fiber optics and restore electrical power, Ukraine’s networks have Continue reading
The Chief Zero Trust Officer: a new role for a new era of cybersecurity
Setting the stage for Zero Trust
Over the last few years the topic of cyber security has moved from the IT department to the board room. The current climate of geopolitical and economic uncertainty has made the threat of cyber attacks all the more pressing, with businesses of all sizes and across all industries feeling the impact. From the potential for a crippling ransomware attack to a data breach that could compromise sensitive consumer information, the risks are real and potentially catastrophic. Organizations are recognizing the need for better resilience and preparation regarding cybersecurity. It is not enough to simply react to attacks as they happen; companies must proactively prepare for the inevitable in their approach to cybersecurity.
The security approach that has gained the most traction in recent years is the concept of Zero Trust. The basic principle behind Zero Trust is simple: don't trust anything; verify everything. The impetus for a modern Zero Trust architecture is that traditional perimeter-based (castle-and-moat) security models are no longer sufficient in today's digitally distributed landscape. Organizations must adopt a holistic approach to security based on verifying the identity and trustworthiness of all users, devices, and systems that access their networks and data.
So long, and thanks for all the deployments: deprecating Wrangler v1
Cloudflare Workers allow developers to deploy code instantly across the globe. Wrangler is the CLI tool we build (and use!) to create, modify, and upload Workers. We recently announced a new version of Wrangler with a bunch of new features – including offline development, zero-config startup, and developer tools support. Since then, we’ve been working hard to make the developer experience with version 2 as smooth and enjoyable as possible. We’re confident in what we’ve built and are now planning to officially deprecate version 1.
What’s happening?
Version 1 of Wrangler (@cloudflare/wrangler on npm) is now deprecated, which means no new features or bug fixes will be published unless they’re critical. Beginning August 2023, no further updates will be provided and the Wrangler v1 GitHub repo will be archived. We strongly recommend you upgrade to version 2 (wrangler on npm) to receive continued support. We have a migration guide to make this process easy!
Why?
Our goal is to make development on the Cloudflare platform as smooth and enjoyable as possible. Whether that means simplifying common workflows, incorporating powerful tools into the Wrangler codebase, or opening up Wrangler for use as a library Continue reading
A look at Internet traffic trends during Super Bowl LVII
The Super Bowl has been happening since the end of the 1966 season, the same year that the ARPANET project, which gave birth to the Internet, was initiated. Around 20 years ago, 50% of the US population were Internet users, and that number is now around 92%. So, it's no surprise that interest in an event like Super Bowl LVII resulted in a noticeable dip in Internet traffic in the United States at the time of the game's kickoff, dropping to around 5% lower than the previous Sunday. During the game, Rihanna's halftime show also caused a significant drop in Internet traffic across most states, with Pennsylvania and New York feeling the biggest impact, but messaging and video platforms saw a surge of traffic right after her show ended.
In this blog post, we will dive into who the biggest winners were among Super Bowl advertisers, as well as examine how traffic to food delivery services, social media and sports and betting websites changed during the game. In addition, we look at traffic trends seen at city and state levels during the game, as well as email threat volume across related categories in the weeks ahead of the game.
Cloudflare mitigates record-breaking 71 million request-per-second DDoS attack
This was a weekend of record-breaking DDoS attacks. Over the weekend, Cloudflare detected and mitigated dozens of hyper-volumetric DDoS attacks. The majority of attacks peaked in the ballpark of 50-70 million requests per second (rps) with the largest exceeding 71 million rps. This is the largest reported HTTP DDoS attack on record, more than 35% higher than the previous reported record of 46M rps in June 2022.
The attacks were HTTP/2-based and targeted websites protected by Cloudflare. They originated from over 30,000 IP addresses. Some of the attacked websites included a popular gaming provider, cryptocurrency companies, hosting providers, and cloud computing platforms. The attacks originated from numerous cloud providers, and we have been working with them to crack down on the botnet.
Over the past year, we’ve seen more attacks originate from cloud computing providers. For this reason, we will be providing service providers that own their own autonomous system a free Botnet threat feed. The feed will provide service providers threat intelligence about their own IP space; attacks originating from within their autonomous system. Service providers that operate their own IP space can now sign up to the Continue reading
Welcome to Wildebeest: the Fediverse on Cloudflare
The Fediverse has been a hot topic of discussion lately, with thousands, if not millions, of new users creating accounts on platforms like Mastodon to either move entirely to "the other side" or experiment and learn about this new social network.
Today we're introducing Wildebeest, an open-source, easy-to-deploy ActivityPub and Mastodon-compatible server built entirely on top of Cloudflare's Supercloud. If you want to run your own spot in the Fediverse you can now do it entirely on Cloudflare.
The Fediverse, built on Cloudflare
Today you're left with two options if you want to join the Mastodon federated network: either you join one of the existing servers (servers are also called communities, and each one has its own infrastructure and rules), or you can run your self-hosted server.
There are a few reasons why you'd want to run your own server:
- You want to create a new community and attract other users over a common theme and usage rules.
- You don't want to have to trust third-party servers or abide by their policies and want your server, under your domain, for your personal account.
- You want complete control over your data, personal information, and content and visibility over what Continue reading
How Cloudflare erroneously throttled a customer’s web traffic
Over the years when Cloudflare has had an outage that affected our customers we have very quickly blogged about what happened, why, and what we are doing to address the causes of the outage. Today’s post is a little different. It’s about a single customer’s website not working correctly because of incorrect action taken by Cloudflare.
Although the customer was not in any way banned from Cloudflare, or lost access to their account, their website didn’t work. And it didn’t work because Cloudflare applied a bandwidth throttle between us and their origin server. The effect was that the website was unusable.
Because of this unusual throttle there was some internal confusion for our customer support team about what had happened. They, incorrectly, believed that the customer had been limited because of a breach of section 2.8 of our Self-Serve Subscription Agreement which prohibits use of our self-service CDN to serve excessive non-HTML content, such as images and video, without a paid plan that includes those services (this is, for example, designed to prevent someone building an image-hosting service on Cloudflare and consuming a huge amount of bandwidth; for that sort of use case we have paid image and video Continue reading
Get notified about the most relevant events with Advanced HTTP Alerts
Today we’re excited to be announcing more flexibility to HTTP alerting, enabling customers to customize the types of activity they’re alerted on and how those alerts are organized.
Prior to today, HTTP alerts at Cloudflare have been very generic. You could choose which Internet properties you wanted and what sensitivity you wanted to be alerted on, but you couldn’t choose anything else. You couldn’t, for example, exclude the IP addresses you use to test things. You couldn’t choose to monitor only a specific path. You couldn’t choose which HTTP statuses you wanted to be alerted on. You couldn’t even choose to monitor your entire account instead of specific zones.
Our customers leverage the Cloudflare network for a myriad of use cases ranging from decreasing bandwidth costs and accelerating asset delivery with Cloudflare CDN to protecting their applications against brute force attacks with Cloudflare Bot Management. Whether the reasons for routing traffic through the Cloudflare network are simple or complex, one powerful capability that comes for free is observability.
With traffic flowing through the network, we can monitor and alert customers about anomalous events such as spikes in origin error rates, enabling them to investigate further and mitigate any issues as Continue reading
Manage and control the use of dedicated egress IPs with Cloudflare Zero Trust
Before identity-driven Zero Trust rules, some SaaS applications on the public Internet relied on the IP address of a connecting user as a security model. Users would connect from known office locations, with fixed IP address ranges, and the SaaS application would check their address in addition to their login credentials.
Many systems still offer that second factor method. Customers of Cloudflare One can use a dedicated egress IP for this purpose as part of their journey to a Zero Trust model. Unlike other solutions, customers using this option do not need to deploy any infrastructure of their own. However, not all traffic needs to use those dedicated egress IPs.
Today, we are announcing policies that give administrators control over when Cloudflare uses their dedicated egress IPs. Specifically, administrators can use a rule builder in the Cloudflare dashboard to determine which egress IP is used and when, based on attributes like identity, application, IP address, and geolocation. This capability is available to any enterprise-contracted customer that adds on dedicated egress IPs to their Zero Trust subscription.
Why did we build this?
In today’s hybrid work environment, organizations aspire for more consistent security and IT experiences to manage their employees’ traffic Continue reading
Cloudflare’s handling of a bug in interpreting IPv4-mapped IPv6 addresses
In November 2022, our bug bounty program received a critical and very interesting report. The report stated that certain types of DNS records could be used to bypass some of our network policies and connect to ports on the loopback address (e.g. 127.0.0.1) of our servers. This post will explain how we dealt with the report, how we fixed the bug, and the outcome of our internal investigation to see if the vulnerability had been previously exploited.
RFC 4291 defines ways to embed an IPv4 address into IPv6 addresses. One of the methods defined in the RFC is to use IPv4-mapped IPv6 addresses, that have the following format:
| 80 bits | 16 | 32 bits |
+--------------------------------------+--------------------------+
|0000..............................0000|FFFF| IPv4 address |
+--------------------------------------+----+---------------------+
In IPv6 notation, the corresponding mapping for 127.0.0.1 is ::ffff:127.0.0.1 (RFC 4038)
The researcher was able to use DNS entries based on mapped addresses to bypass some of our controls and access ports on the loopback address or non-routable IPs.
This vulnerability was reported on November 27 to our bug bounty program. Our Security Incident Response Team (SIRT) was contacted, and incident response activities Continue reading
Uptick in healthcare organizations experiencing targeted DDoS attacks
Healthcare in the crosshairs
Over the past few days, Cloudflare, as well as other sources, have observed healthcare organizations targeted by a pro-Russian hacktivist group claiming to be Killnet. There has been an increase in the amount of healthcare organizations coming to us to help get out from under these types of attacks. Multiple healthcare organizations behind Cloudflare have also been targeted by HTTP DDoS attacks and Cloudflare has helped them successfully mitigate these attacks. The United States Department of Health and Human Services issued an Analyst Note detailing the threat of Killnet-related cyberattacks to the healthcare industry.
A rise in political tensions and escalation of the conflict in Ukraine are all factors that play into the current cybersecurity threat landscape. Unlike traditional warfare, the Internet has enabled and empowered groups of individuals to carry out targeted attacks regardless of their location or involvement. Distributed-denial-of-Service (DDoS) attacks have the unfortunate advantage of not requiring an intrusion or a foothold to be launched and have, unfortunately, become more accessible than ever before.
The attacks observed by the Cloudflare global network do not show a clear indication that they are originating from a single botnet and the attack methods and sources Continue reading
CVE-2022-47929: traffic control noqueue no problem?
USER namespaces power the functionality of our favorite tools such as docker, podman, and kubernetes. We wrote about Linux namespaces back in June and explained them like this:
Most of the namespaces are uncontroversial, like the UTS namespace which allows the host system to hide its hostname and time. Others are complex but straightforward - NET and NS (mount) namespaces are known to be hard to wrap your head around. Finally, there is this very special, very curious USER namespace. USER namespace is special since it allows the - typically unprivileged owner to operate as "root" inside it. It's a foundation to having tools like Docker to not operate as true root, and things like rootless containers.
Due to its nature, allowing unprivileged users access to USER namespace always carried a great security risk. With its help the unprivileged user can in fact run code that typically requires root. This code is often under-tested and buggy. Today we will look into one such case where USER namespaces are leveraged to exploit a kernel bug that can result in an unprivileged denial of service attack.
Enter Linux Traffic Control queue disciplines
In 2019, we were exploring leveraging Linux Traffic Control's queue Continue reading
Cyberattacks on Holocaust educational websites increased in 2022
Today we mark the International Holocaust Remembrance Day. We commemorate the victims that were robbed of their possessions, stripped of their rights, deported, starved, dehumanized and murdered by the Nazis and their accomplices. During the Holocaust and in the events that led to it, the Nazis exterminated one third of the European Jewish population. Six million Jews, along with countless other members of minority and disability groups, were murdered because the Nazis believed they were inferior.
Seventy eight years later, after the liberation of the infamous Auschwitz death camp, antisemitism still burns with hatred. According to a study performed by the Campaign Against Antisemitism organization on data provided by the UK Home Office, Jews are 500% more likely to be targeted by hate crime than any other faith group per capita.
Cyberattacks targeting Holocaust educational websites
From Cloudflare’s vantage point we can point to distressing findings as well. In 2021, cyberattacks on Holocaust educational websites doubled year over year. In 2021, one out of every 100 HTTP requests sent to Holocaust educational websites behind Cloudflare was part of an attack. In 2022, the share of those cyber attacks grew again by 49% YoY. Cyberattacks represented 1.6% of all Continue reading
Inside Geo Key Manager v2: re-imagining access control for distributed systems
In December 2022 we announced the closed beta of the new version of Geo Key Manager. Geo Key Manager v2 (GeoV2) is the next step in our journey to provide customers with a secure and flexible way to control the distribution of their private keys by geographic location. Our original system, Geo Key Manager v1, was launched as a research project in 2017, but as customer needs evolved and our scale increased, we realized that we needed to make significant improvements to provide a better user experience.
One of the principal challenges we faced with Geo Key Manager v1 (GeoV1) was the inflexibility of our access control policies. Customers required richer data localization, often spurred by regulatory concerns. Internally, events such as the conflict in Ukraine reinforced the need to be able to quickly restrict access to sensitive key material. Geo Key Manager v1’s underlying cryptography was a combination of identity-based broadcast encryption and identity-based revocation that simulated a subset of the functionality offered by Attribute-Based Encryption (ABE). Replacing this with an established ABE scheme addressed the inflexibility of our access control policies and provided a more secure foundation for our system.
Unlike our previous scheme, which limited future Continue reading
Towards a global framework for cross-border data flows and privacy protection
As our societies and economies rely more and more on digital technologies, there is an increased need to share and transfer data, including personal data, over the Internet. Cross-border data flows have become essential to international trade and global economic development. In fact, the digital transformation of the global economy could never have happened as it did without the open and global architecture of the Internet and the ability for data to transcend national borders. As we described in our blog post yesterday, data localization doesn’t necessarily improve data privacy. Actually, there can be real benefits to data security and - by extension - privacy if we are able to transfer data across borders. So with Data Privacy Day coming up tomorrow, we wanted to take this opportunity to drill down into the current environment for the transfer of personal data from the EU to the US, which is governed by the EU’s privacy regulation (GDPR). Looking to the future, we will make the case for a more stable, global cross-border data transfer framework, which will be critical for an open, more secure and more private Internet.
The privacy challenge to cross-border data flows
In the last decade, we have Continue reading
Navigating the changing data localization landscape with Cloudflare’s Data Localization Suite
This post is also available in Português.
At Cloudflare, we believe that deploying effective cybersecurity measures is the best way to protect the privacy of personal information and can be more effective than making sure that information stays within a particular jurisdiction. Yet, we hear from customers in Europe, India, Australia, Japan, and many other regions that, as part of their privacy programs, they need solutions to localize data in order to meet their regulatory obligations.
So as we think about Data Privacy Day, which is coming up on January 28, we are in the interesting position of disagreeing with those who believe that data localization is a proxy for better data privacy, but of also wanting to support our customers who have to comply with certain regulations.
For this reason, we introduced our Data Localization Suite (DLS) in 2020 to help customers navigate a data protection landscape that focuses more and more on data localization. With the DLS, customers can use Cloudflare’s powerful global network and security measures to protect their businesses, while keeping the data we process on their behalf local. Since its launch, we’ve had many customers adopt the Data Localization Suite. In this blog post we Continue reading
Investing in security to protect data privacy
If you’ve made it to 2023 without ever receiving a notice that your personal information was compromised in a security breach, consider yourself lucky. In a best case scenario, bad actors only got your email address and name – information that won’t cause you a huge amount of harm. Or in a worst-case scenario, maybe your profile on a dating app was breached and intimate details of your personal life were exposed publicly, with life-changing impacts. But there are also more hidden, insidious ways that your personal data can be exploited. For example, most of us use an Internet Service Provider (ISP) to connect to the Internet. Some of those ISPs are collecting information about your Internet viewing habits, your search histories, your location, etc. – all of which can impact the privacy of your personal information as you are targeted with ads based on your online habits.
You also probably haven’t made it to 2023 without hearing at least something about Internet privacy laws around the globe. In some jurisdictions, lawmakers are driven by a recognition that the right to privacy is a fundamental human right. In other locations, lawmakers are passing laws to address the harms their citizens Continue reading
Armed to Boot: an enhancement to Arm’s Secure Boot chain
Over the last few years, there has been a rise in the number of attacks that affect how a computer boots. Most modern computers use a specification called Unified Extensible Firmware Interface (UEFI) that defines a software interface between an operating system (e.g. Windows) and platform firmware (e.g. disk drives, video cards). There are security mechanisms built into UEFI that ensure that platform firmware can be cryptographically validated and boot securely through an application called a bootloader. This firmware is stored in non-volatile SPI flash memory on the motherboard, so it persists on the system even if the operating system is reinstalled and drives are replaced.
This creates a ‘trust anchor’ used to validate each stage of the boot process, but, unfortunately, this trust anchor is also a target for attack. In these UEFI attacks, malicious actions are loaded onto a compromised device early in the boot process. This means that malware can change configuration data, establish persistence by ‘implanting’ itself, and can bypass security measures that are only loaded at the operating system stage. So, while UEFI-anchored secure boot protects the bootloader from bootloader attacks, it does not protect the UEFI firmware itself.
Cloudflare Incident on January 24th, 2023
Several Cloudflare services became unavailable for 121 minutes on January 24th, 2023 due to an error releasing code that manages service tokens. The incident degraded a wide range of Cloudflare products including aspects of our Workers platform, our Zero Trust solution, and control plane functions in our content delivery network (CDN).
Cloudflare provides a service token functionality to allow automated services to authenticate to other services. Customers can use service tokens to secure the interaction between an application running in a data center and a resource in a public cloud provider, for example. As part of the release, we intended to introduce a feature that showed administrators the time that a token was last used, giving users the ability to safely clean up unused tokens. The change inadvertently overwrote other metadata about the service tokens and rendered the tokens of impacted accounts invalid for the duration of the incident.
The reason a single release caused so much damage is because Cloudflare runs on Cloudflare. Service tokens impact the ability for accounts to authenticate, and two of the impacted accounts power multiple Cloudflare services. When these accounts’ service tokens were overwritten, the services that run on these accounts began to experience Continue reading
Intelligent, automatic restarts for unhealthy Kafka consumers
At Cloudflare, we take steps to ensure we are resilient against failure at all levels of our infrastructure. This includes Kafka, which we use for critical workflows such as sending time-sensitive emails and alerts.
We learned a lot about keeping our applications that leverage Kafka healthy, so they can always be operational. Application health checks are notoriously hard to implement: What determines an application as healthy? How can we keep services operational at all times?
These can be implemented in many ways. We’ll talk about an approach that allows us to considerably reduce incidents with unhealthy applications while requiring less manual intervention.
Kafka at Cloudflare
Cloudflare is a big adopter of Kafka. We use Kafka as a way to decouple services due to its asynchronous nature and reliability. It allows different teams to work effectively without creating dependencies on one another. You can also read more about how other teams at Cloudflare use Kafka in this post.
Kafka is used to send and receive messages. Messages represent some kind of event like a credit card payment or details of a new user created in your platform. These messages can be represented in multiple ways: JSON, Protobuf, Avro and so on.