DNSSEC issues take Fiji domains offline

On the morning of March 8, a post to Hacker News stated that “All .fj domains have gone offline”, listing several hostnames in domains within the Fiji top level domain (known as a ccTLD) that had become unreachable. Commenters in the associated discussion thread had mixed results in being able to reach .fj hostnames—some were successful, while others saw failures. The fijivillage news site also highlighted the problem, noting that the issue also impacted Vodafone’s M-PAiSA app/service, preventing users from completing financial transactions.

The impact of this issue can be seen in traffic to Cloudflare customer zones in the .com.fj second-level domain. The graph below shows that HTTP traffic to these zones dropped by approximately 40% almost immediately starting around midnight UTC on March 8. Traffic volumes continued to decline throughout the rest of the morning.

Looking at Cloudflare’s 1.1.1.1 resolver data for queries for .com.fj hostnames, we can also see that error volume associated with those queries climbs significantly starting just after midnight as well. This means that our resolvers encountered issues with the answers from .fj servers.

This observation suggests that the problem was strictly DNS related, rather than connectivity related—Cloudflare Radar Continue reading

Announcing experimental DDR in 1.1.1.1

1.1.1.1 sees approximately 600 billion queries per day. However, proportionally, most queries sent to this resolver are over cleartext: 89% over UDP and TCP combined, and the remaining 11% are encrypted. We care about end-user privacy and would prefer to see all of these queries sent to us over an encrypted transport using DNS-over-TLS or DNS-over-HTTPS. Having a mechanism by which clients could discover support for encrypted protocols such as DoH or DoT will help drive this number up and lead to more name encryption on the Internet. That’s where DDR – or Discovery of Designated Resolvers – comes into play. As of today, 1.1.1.1 supports the latest version of DDR so clients can automatically upgrade non-secure UDP and TCP connections to secure connections. In this post, we’ll describe the motivations for DDR, how the mechanism works, and, importantly, how you can test it out as a client.

DNS transports and public resolvers

We initially launched our public recursive resolver service 1.1.1.1 over three years ago, and have since seen its usage steadily grow. Today, it is one of the fastest public recursive resolvers available to end-users, supporting the latest security Continue reading

CVE-2022-26143: A Zero-Day vulnerability for launching UDP amplification DDoS attacks

A zero-day vulnerability in the Mitel MiCollab business phone system has recently been discovered (CVE-2022-26143). This vulnerability, called TP240PhoneHome, which Cloudflare customers are already protected against, can be used to launch UDP amplification attacks. This type of attack reflects traffic off vulnerable servers to victims, amplifying the amount of traffic sent in the process by an amplification factor of 220 billion percent in this specific case.

Cloudflare has been actively involved in investigating the TP240PhoneHome exploit, along with other members of the InfoSec community. Read our joint disclosure here for more details. As far as we can tell, the vulnerability has been exploited as early as February 18, 2022. We have deployed emergency mitigation rules to protect Cloudflare customers against the amplification DDoS attacks.

Mitel has been informed of the vulnerability. As of February 22, they have issued a high severity security advisory advising their customers to block exploitation attempts using a firewall, until a software patch is made available. Cloudflare Magic Transit customers can use the Magic Firewall to block external traffic to the exposed Mitel UDP port 10074 by following the example in the screenshot below, or by pasting the following expression into their Magic Firewall Continue reading

CVE-2022-26143: TP240PhoneHome reflection/amplification DDoS attack vector

Beginning in mid-February 2022, security researchers, network operators, and security vendors observed a spike in DDoS attacks sourced from UDP port 10074 targeting broadband access ISPs, financial institutions, logistics companies, and organizations in other vertical markets.

Upon further investigation, it was determined that the devices abused to launch these attacks are MiCollab and MiVoice Business Express collaboration systems produced by Mitel, which incorporate TP-240 VoIP- processing interface cards and supporting software; their primary function is to provide Internet-based site-to-site voice connectivity for PBX systems.

Approximately 2600 of these systems have been incorrectly provisioned so that an unauthenticated system test facility has been inadvertently exposed to the public Internet, allowing attackers to leverage these PBX VoIP gateways as DDoS reflectors/amplifiers.

Mitel is aware that these systems are being abused to facilitate high-pps (packets-per-second) DDoS attacks, and have been actively working with customers to remediate abusable devices with patched software that disables public access to the system test facility.

In this blog, we will further explore the observed activity, explain how the driver has been abused, and share recommended mitigation steps. This research was created cooperatively among a team of researchers from Akamai SIRT, Cloudflare, Lumen Black Lotus Labs, NETSCOUT ASERT, Continue reading

International Women’s Day 2022

“I would venture to guess that Anon,
who wrote so many poems without signing them,
was often a woman.” - Virginia Woolf

Welcome to International Women’s Day 2022! Here at Cloudflare, we are happy to celebrate it with you! Our celebration is not only this blog post, but many events prepared for the month of March: our way of honoring Women’s History Month by showcasing women’s empowerment. We want to celebrate the achievements, ideas, passion and work that women bring to the world. We want to advocate for equality and to achieve gender parity. And we want to highlight the brilliant work that our women colleagues do every day. Welcome!

This is a time of celebration but also one to reflect on the current state. The global gender gap is not expected to close for another 136 years. This gap has also worsened due to the COVID-19 pandemic, which has negatively impacted the lives of women and girls by deepening pre-existing inequalities. Improving this state is a collective effort—we all need to get involved!

Who are we? Womenflare!

First, let’s introduce ourselves. We are Womenflare—Cloudflare’s Employee Resource Group (ERG) for all who identify as and advocate for Continue reading

Cloudflare, CrowdStrike, and Ping Identity launch the Critical Infrastructure Defense Project

Today, in partnership with CrowdStrike and Ping Identity, Cloudflare is launching the Critical Infrastructure Defense Project (CriticalInfrastructureDefense.org). The Project was born out of conversations with cybersecurity and government experts concerned about potential retaliation to the sanctions that resulted from the Russian invasion of Ukraine.

In particular, there is a fear that critical United States infrastructure will be targeted with cyber attacks. While these attacks may target any industry, the experts we consulted with were particularly concerned about three areas that were often underprepared and could cause significant disruption: hospitals, energy, and water.

To help address that need, Cloudflare, CrowdStrike, and Ping Identity have committed under the Critical Infrastructure Defense Project to offer a broad suite of our products for free for at least the next four months to any United States-based hospital, or energy or water utility. You can learn more at: www.CriticalInfrastructureDefense.org.

We are not powerless against hackers. Organizations that have adopted a Zero Trust approach to security have been successful at mitigating even determined attacks. There are three core components to any Zero Trust security approach: 1) Network Security, 2) Endpoint Security; and 3) Identity.

Cloudflare, CrowdStrike, and Ping Identity are three of Continue reading

Steps we’ve taken around Cloudflare’s services in Ukraine, Belarus, and Russia

At Cloudflare, we've watched in horror the Russian invasion of Ukraine. As the possibility of war looked more likely, we began to carefully monitor the situation on the ground, with the goal of keeping our employees, our customers, and our network safe.

Helping protect Ukraine against cyberattacks

Attacks against the Internet in Ukraine began even before the start of the invasion. Those attacks—and the steady stream of DDoS attacks we’ve seen in the days since—prompted us to extend our services to Ukrainian government and telecom organizations at no cost in order to ensure they can continue to operate and deliver critical information to their citizens as well as to the rest of the world about what is happening to them.

Going beyond that, under Project Galileo, we are expediting onboarding of any Ukrainian entities for our full suite of protections. We are currently assisting more than sixty organizations in Ukraine and the region—with about 25% of those organizations coming aboard during the current crisis. Many of the new organizations are groups coming together to assist refugees, share vital information, or members of the Ukrainian diaspora in nearby countries looking to organize and help. Any Ukrainian organizations that are facing Continue reading

Shields up: free Cloudflare services to improve your cyber readiness

Since our founding, Cloudflare's mission has been to "help build a better Internet," and we take it to heart. It used to be that the services required to adequately secure an online presence were only available to the largest of enterprises — organizations big enough to afford both the technology itself and the teams to manage it.

We've worked hard over the years to level the playing field. This has meant making more and more of the essential tools for protecting an online presence available to as many people as possible. Cloudflare offers unmetered DDoS protection — for free. We were the first to introduce SSL at scale — for free. And it’s not just protection for your external-facing infrastructure: we have a free Zero Trust plan that enables teams to protect their internal-facing infrastructure, too.

These types of tools have always been important for the billions of people on the Internet. But perhaps never as important as they've become this week.

Concurrent with the Russian invasion of Ukraine, we've seen increasing cyberattacks on the Internet, too. Governments around the world are encouraging organizations to go “shields up” — with warnings coming from the United States’ Cybersecurity & Infrastructure Security Continue reading

Internet traffic patterns in Ukraine since February 21, 2022

Cloudflare operates in more than 250 cities worldwide where we connect our equipment to the Internet to provide our broad range of services. We have data centers in Ukraine, Belarus and Russia and across the world. To operate our service we monitor traffic trends, performance and errors seen at each data center, aggregate data about DNS, and congestion and packet loss on Internet links.

Internet Traffic

For reference, here is a map of Ukraine showing its major cities. Note that whenever we talk about dates and times in this post, we are using UTC. Ukraine’s current time zone is UTC+2.

Internet traffic in Ukraine generally follows a pretty predictable pattern based on day and night. Lowest in the hours after local midnight and picking up as people wake up. It’s not uncommon to see a dip around lunchtime and a peak when people go home in the evening. That pattern is clearly visible in this chart of overall Internet traffic seen by Cloudflare for Ukrainian networks on Monday, Tuesday, and Wednesday prior to the invasion.

Starting Thursday, traffic was significantly lower. On Thursday, we saw about 70% of our normal request volume and about 60% on Friday. Continue reading

iCloud Private Relay: information for Cloudflare customers

iCloud Private Relay is a new Internet privacy service from Apple that allows users with iOS 15, iPadOS 15, or macOS Monterey on their devices and an iCloud+ subscription, to connect to the Internet and browse with Safari in a more secure and private way. Cloudflare is proud to work with Apple to operate portions of Private Relay infrastructure.

In this post, we’ll explain how website operators can ensure the best possible experience for end users using iCloud Private Relay. Additional material is available from Apple, including “Set up iCloud Private Relay on all your devices”, and “Prepare Your Network or Web Server for iCloud Private Relay” which covers network operator scenarios in detail.

How browsing works using iCloud Private Relay

The design of the iCloud Private Relay system ensures that no single party handling user data has complete information on both who the user is and what they are trying to access.

To do this, Private Relay uses modern encryption and transport mechanisms to relay traffic from user devices through Apple and partner infrastructure before sending traffic to the destination website.

Here’s a diagram depicting what connection metadata is available to who when not using Private Relay Continue reading

The post-quantum future: challenges and opportunities

“People ask me to predict the future, when all I want to do is prevent it. Better yet, build it. Predicting the future is much too easy, anyway. You look at the people around you, the street you stand on, the visible air you breathe, and predict more of the same. To hell with more. I want better.”
— Ray Bradbury, from Beyond 1984: The People Machines

The story and the path are clear: quantum computers are coming that will have the ability to break the cryptographic mechanisms we rely on to secure modern communications, but there is hope! The cryptographic community has designed new mechanisms to safeguard against this disruption. There are challenges: will the new safeguards be practical? How will the fast-evolving Internet migrate to this new reality? In other blog posts in this series, we have outlined some potential solutions to these questions: there are new algorithms for maintaining confidentiality and authentication (in a “post-quantum” manner) in the protocols we use. But will they be fast enough to deploy at scale? Will they provide the required properties and work in all protocols? Are they easy to use?

Adding post-quantum cryptography into architectures and networks Continue reading

Post-quantumify internal services: Logfwrdr, Tunnel, and gokeyless

Theoretically, there is no impediment to adding post-quantum cryptography to any system. But the reality is harder. In the middle of last year, we posed ourselves a big challenge: to change all internal connections at Cloudflare to use post-quantum cryptography. We call this, in a cheeky way, “post-quantum-ifying” our services. Theoretically, this should be simple: swap algorithms for post-quantum ones and move along. But with dozens of different services in various programming languages (as we have at Cloudflare), it is not so simple. The challenge is big but we are here and up for the task! In this blog post, we will look at what our plan was, where we are now, and what we have learned so far. Welcome to the first announcement of a post-quantum future at Cloudflare: our connections are going to be quantum-secure!

What are we doing?

The life of most requests at Cloudflare begins and ends at the edge of our global network. Not all requests are equal and on their path they are transmitted by several protocols. Some of those protocols provide security properties whilst others do not. For the protocols that do, for context, Cloudflare uses: TLS, QUIC, WireGuard, DNSSEC Continue reading

HPKE: Standardizing public-key encryption (finally!)

For the last three years, the Crypto Forum Research Group of the Internet Research Task Force (IRTF) has been working on specifying the next generation of (hybrid) public-key encryption (PKE) for Internet protocols and applications. The result is Hybrid Public Key Encryption (HPKE), published today as RFC 9180.

HPKE was made to be simple, reusable, and future-proof by building upon knowledge from prior PKE schemes and software implementations. It is already in use in a large assortment of emerging Internet standards, including TLS Encrypted Client Hello and Oblivious DNS-over-HTTPS, and has a large assortment of interoperable implementations, including one in CIRCL. This article provides an overview of this new standard, going back to discuss its motivation, design goals, and development process.

A primer on public-key encryption

Public-key cryptography is decades old, with its roots going back to the seminal work of Diffie and Hellman in 1976, entitled “New Directions in Cryptography.” Their proposal – today called Diffie-Hellman key exchange – was a breakthrough. It allowed one to transform small secrets into big secrets for cryptographic applications and protocols. For example, one can bootstrap a secure channel for exchanging messages with confidentiality and integrity using a key exchange Continue reading

Cloudflare re-enforces commitment to security in Germany via BSIG audit

As a large data processing country, Germany is at the forefront of security and privacy regulation in Europe and sets the tone for other countries to follow. Analyzing and meeting the requirements to participate in Germany’s cloud security industry requires adherence to international, regional, and country-specific standards. Cloudflare is pleased to announce that we have taken appropriate organizational and technical precautions to prevent disruptions to the availability, integrity, authenticity, and confidentiality of Cloudflare’s production systems in accordance with BSI-KritisV. TÜViT, the auditing body tasked with auditing Cloudflare and providing the evidence to BSI every two years. Completion of this audit allows us to comply with the NIS Directive within Germany.

Why do cloud companies operating in Germany need to go through a BSI audit?

In 2019, Cloudflare registered as an Operator of Essential Services’ under the EU Directive on Security of Network and Information Systems (NIS Directive). The NIS Directive is cybersecurity legislation with the goal to enhance cybersecurity across the EU. Every member state has started to adopt national legislation for the NIS Directive and the criteria for compliance is set individually by each country. As an ‘Operator of Essential Services’ in Germany, Cloudflare is regulated by the Federal Continue reading

Building Confidence in Cryptographic Protocols

An introduction to formal analysis and our proof of the security of KEMTLS

Good morning everyone, and welcome to another Post-Quantum–themed blog post! Today we’re going to look at something a little different. Rather than look into the past or future quantum we’re going to look as far back as the ‘80s and ‘90s, to try and get some perspective on how we can determine whether a protocol is or is not secure. Unsurprisingly, this question comes up all the time. Cryptographers like to build fancy new cryptosystems, but just because we, the authors, can’t break our own designs, it doesn’t mean they are secure: it just means we are not smart enough to break them.

One might at this point wonder why in a post-quantum themed blog post we are talking about security proofs. The reason is simple: the new algorithms that claim to be safe against quantum threats need proofs showing that they actually are safe. In this blog post, not only are we going to introduce how we go about proving a protocol is secure, we’re going to introduce the security proofs of KEMTLS, a version of TLS designed to be more secure against quantum computers, and Continue reading

Using EasyCrypt and Jasmin for post-quantum verification

Cryptographic code is everywhere: it gets run when we connect to the bank, when we send messages to our friends, or when we watch cat videos. But, it is not at all easy to take a cryptographic specification written in a natural language and produce running code from it, and it is even harder to validate both the theoretical assumptions and the correctness of the implementation itself. Mathematical proofs, as we talked about in our previous blog post, and code inspection are simply not enough. Testing and fuzzing can catch common or well-known bugs or mistakes, but might miss rare ones that can, nevertheless, be triggered by an attacker. Static analysis can detect mistakes in the code, but cannot check whether the code behaves as described by the specification in natural-language (for functional correctness). This gap between implementation and validation can have grave consequences in terms of security in the real world, and we need to bridge this chasm.

In this blog post, we will be talking about ways to make this gap smaller by making the code we deploy better through analyzing its security properties and its implementation. This blog post continues our work on high assurance Continue reading

Why we are acquiring Area 1

This post is also available in Français and Español.

Cloudflare’s mission is to help build a better Internet. We’ve invested heavily in building the world’s most powerful cloud network to deliver a faster, safer and more reliable Internet for our users. Today, we’re taking a big step towards enhancing our ability to secure our customers.

Earlier today we announced that Cloudflare has agreed to acquire Area 1 Security. Area 1’s team has built exceptional cloud-native technology to protect businesses from email-based security threats. Cloudflare will integrate Area 1’s technology with our global network to give customers the most complete Zero Trust security platform available.

Why Email Security?

Back at the turn of the century I was involved in the fight against email spam. At the time, before the mass use of cloud-based email, spam was a real scourge. Clogging users’ inboxes, taking excruciatingly long to download, and running up people’s Internet bills. The fight against spam involved two things, one technical and one architectural.

Technically, we figured out how to use machine-learning to successfully differentiate between spam and genuine. And fairly quickly email migrated to being largely cloud-based. But together these changes didn’t kill spam, but they relegated to a Continue reading

Making protocols post-quantum

Ever since the (public) invention of cryptography based on mathematical trap-doors by Whitfield Diffie, Martin Hellman, and Ralph Merkle, the world has had key agreement and signature schemes based on discrete logarithms. Rivest, Shamir, and Adleman invented integer factorization-based signature and encryption schemes a few years later. The core idea, that has perhaps changed the world in ways that are hard to comprehend, is that of public key cryptography. We can give you a piece of information that is completely public (the public key), known to all our adversaries, and yet we can still securely communicate as long as we do not reveal our piece of extra information (the private key). With the private key, we can then efficiently solve mathematical problems that, without the secret information, would be practically unsolvable.

In later decades, there were advancements in our understanding of integer factorization that required us to bump up the key sizes for finite-field based schemes. The cryptographic community largely solved that problem by figuring out how to base the same schemes on elliptic curves. The world has since then grown accustomed to having algorithms where public keys, secret keys, and signatures are just a handful of Continue reading

BGP security and confirmation biases

This is not what I imagined my first blog article would look like, but here we go.

On February 1, 2022, a configuration error on one of our routers caused a route leak of up to 2,000 Internet prefixes to one of our Internet transit providers. This leak lasted for 32 seconds and at a later time 7 seconds. We did not see any traffic spikes or drops in our network and did not see any customer impact because of this error, but this may have caused an impact to external parties, and we are sorry for the mistake.

Timeline

All timestamps are UTC.

As part of our efforts to build the best network, we regularly update our Internet transit and peering links throughout our network. On February 1, 2022, we had a “hot-cut” scheduled with one of our Internet transit providers to simultaneously update router configurations on Cloudflare and ISP routers to migrate one of our existing Internet transit links in Newark to a link with more capacity. Doing a “hot-cut” means that both parties will change cabling and configuration at the same time, usually while being on a conference call, to reduce downtime and impact on the network. Continue reading

Deep Dive Into a Post-Quantum Key Encapsulation Algorithm

The Internet is accustomed to the fact that any two parties can exchange information securely without ever having to meet in advance. This magic is made possible by key exchange algorithms, which are core to certain protocols, such as the Transport Layer Security (TLS) protocol, that are used widely across the Internet.

Key exchange algorithms are an elegant solution to a vexing, seemingly impossible problem. Imagine a scenario where keys are transmitted in person: if Persephone wishes to send her mother Demeter a secret message, she can first generate a key, write it on a piece of paper and hand that paper to her mother, Demeter. Later, she can scramble the message with the key, and send the scrambled result to her mother, knowing that her mother will be able to unscramble the message since she is also in possession of the same key.

But what if Persephone is kidnapped (as the story goes) and cannot deliver this key in person? What if she can no longer write it on a piece of paper because someone (by chance Hades, the kidnapper) might read that paper and use the key to decrypt any messages between them? Key exchange algorithms Continue reading