Investigating threats using the Cloudflare Security Center

Cloudflare blocks a lot of diverse security threats, with some of the more interesting attacks targeting the “long tail” of the millions of Internet properties we protect. The data we glean from these attacks trains our machine learning models and improves the efficacy of our network and application security products, but historically hasn’t been available to query directly. This week, we’re changing that.

All customers will soon be granted access to our new threat investigations portal, Investigate, in the Cloudflare Security Center (first launched in December 2021). Additionally, we’ll be annotating threats across our analytics platform with this intelligence to streamline security workflows and tighten feedback loops.

What sorts of data might you want to look up here? Let’s say you’re seeing an IP address in your logs and want to learn which hostnames have pointed to it via DNS, or you’re seeing a cluster of attacks come from an autonomous system (AS) you’re not familiar with. Or maybe you want to investigate a domain name to see how it’s been categorized from a threat perspective. Simply enter any of those items into the omni search box, and we’ll tell you everything we know.

IPs and hostnames will be Continue reading

Get full observability into your Cloudflare logs with New Relic

Building a great customer experience is at the heart of any business. Building resilient products is half the battle — teams also need observability into their applications and services that are running across their stack.

Cloudflare provides analytics and logs for our products in order to give our customers visibility to extract insights. Many of our customers use Cloudflare along with other applications and network services and want to be able to correlate data through all of their systems.

Understanding normal traffic patterns, causes of latency and errors can be used to improve performance and ultimately the customer experience. For example, for websites behind Cloudflare, analyzing application logs and origin server logs along with Cloudflare’s HTTP request logs give our customers an end-to-end visibility about the journey of a request.

We’re excited to have partnered with New Relic to create a direct integration that provides this visibility. The direct integration with our logging product, Logpush, means customers no longer need to pay for middleware to get their Cloudflare data into New Relic. The result is a faster log delivery and fewer costs for our mutual customers!

We’ve invited the New Relic team to dig into how New Relic One can Continue reading

Leverage IBM QRadar SIEM to get insights from Cloudflare logs

It’s just gone midnight, and you’ve just been notified that there is a malicious IP hitting your servers. You need to triage the situation; find the who, what, where, when, why as fast and in as much detail as possible.

Based on what you find out, your next steps could fall anywhere between classifying the alert as a false positive, to escalating the situation and alerting on-call staff from around your organization with a middle of the night wake up.

For anyone that’s gone through a similar situation, you’re aware that the security tools you have on hand can make the situation infinitely easier. It’s invaluable to have one platform that provides complete visibility of all the endpoints, systems and operations that are running at your company.

Cloudflare protects customers’ applications through application services: DNS, CDN and WAF to name a few. We also have products that protect corporate applications, like our Zero Trust offerings Access and Gateway. Each of these products generates logs that provide customers visibility into what’s happening in their environments. Many of our customers use Cloudflare’s services along with other network or application services, such as endpoint management, containerized systems and their own servers.

We’re excited Continue reading

Introducing: Backup Certificates

At Cloudflare, we pride ourselves in giving every customer the ability to provision a TLS certificate for their Internet application — for free. Today, we are responsible for managing the certificate lifecycle for almost 45 million certificates from issuance to deployment to renewal. As we build out the most resilient, robust platform, we want it to be “future-proof” and resilient against events we can’t predict.

Events that cause us to re-issue certificates for our customers, like key compromises, vulnerabilities, and mass revocations require immediate action. Otherwise, customers can be left insecure or offline. When one of these events happens, we want to be ready to mitigate impact immediately. But how?

By having a backup certificate ready to deploy — wrapped with a different private key and issued from a different Certificate Authority than the primary certificate that we serve.

Events that lead to certificate re-issuance

Cloudflare re-issues certificates every day — we call this a certificate renewal. Because certificates come with an expiration date, when Cloudflare sees that a certificate is expiring soon, we initiate a new certificate renewal order. This way, by the time the certificate expires, we already have an updated certificate deployed and ready to use for Continue reading

Welcome to Security Week 2022!

Recent events are bringing cybersecurity to the forefront of many conversations.

Governments around the world are encouraging businesses to go “shields up” following Ukraine’s invasion. The current threat is significantly higher than before and any organization with Internet-facing infrastructure should put security as a top priority for the year.

To help keep services online, Cloudflare is also participating in the Critical Infrastructure Defense Project ensuring teams can get the best help to secure networks and applications more vulnerable to cyber threats, such as those in the medical, water and energy sectors.

As another example, not too long ago, Log4J, a high-severity vulnerability affecting many Java-based applications, also highlighted how important good security is on the Internet as attackers immediately started scanning for vulnerable applications within hours of the attack vector becoming public.

Unfortunately, these events are almost certainly not going to be our last reminders.

Over the next six days, we intend to tackle the broad topic of cyber security with a simple goal: ensure security is no longer an afterthought.

Security, however, is also hard, and you never know when “you’ve done enough”. The importance of good security practices should never be underestimated. Reliable and secure Continue reading

Stream now supports SRT as a drop-in replacement for RTMP

SRT is a new and modern live video transport protocol. It features many improvements to the incumbent popular video ingest protocol, RTMP, such as lower latency, and better resilience against unpredictable network conditions on the public Internet. SRT supports newer video codecs and makes it easier to use accessibility features such as captions and multiple audio tracks. While RTMP development has been abandoned since at least 2012, SRT development is maintained by an active community of developers.

We don’t see RTMP use going down anytime soon, but we can do something so authors of new broadcasting software, as well as video streaming platforms, can have an alternative.

Starting today, in open beta, you can use Stream Connect as a gateway to translate SRT to RTMP or RTMP to SRT with your existing applications. This way, you can get the last-mile reliability benefits of SRT and can continue to use the RTMP service of your choice. It’s priced at $1 per 1,000 minutes, regardless of video encoding parameters.

You can also use SRT to go live on Stream Live, our end-to-end live streaming service to get HLS and DASH manifest URLs from your SRT input, and do simulcasting to multiple Continue reading

How Cloudflare verifies the code WhatsApp Web serves to users

How do you know the code your web browser downloads when visiting a website is the code the website intended you to run? In contrast to a mobile app downloaded from a trusted app store, the web doesn’t provide the same degree of assurance that the code hasn’t been tampered with. Today, we’re excited to be partnering with WhatsApp to provide a system that assures users that the code run when they visit WhatsApp on the web is the code that WhatsApp intended.

With WhatsApp usage in the browser growing, and the increasing number of at-risk users — including journalists, activists, and human rights defenders — WhatsApp wanted to take steps to provide assurances to browser-based users. They approached us to help dramatically raise the bar for third-parties looking to compromise or otherwise tamper with the code responsible for end-to-end encryption of messages between WhatsApp users.

So how will this work? Cloudflare holds a hash of the code that WhatsApp users should be running. When users run WhatsApp in their browser, the WhatsApp Code Verify extension compares a hash of that code that is executing in their browser with the hash that Cloudflare has — enabling them to easily see Continue reading

DNSSEC issues take Fiji domains offline

On the morning of March 8, a post to Hacker News stated that “All .fj domains have gone offline”, listing several hostnames in domains within the Fiji top level domain (known as a ccTLD) that had become unreachable. Commenters in the associated discussion thread had mixed results in being able to reach .fj hostnames—some were successful, while others saw failures. The fijivillage news site also highlighted the problem, noting that the issue also impacted Vodafone’s M-PAiSA app/service, preventing users from completing financial transactions.

The impact of this issue can be seen in traffic to Cloudflare customer zones in the .com.fj second-level domain. The graph below shows that HTTP traffic to these zones dropped by approximately 40% almost immediately starting around midnight UTC on March 8. Traffic volumes continued to decline throughout the rest of the morning.

Looking at Cloudflare’s 1.1.1.1 resolver data for queries for .com.fj hostnames, we can also see that error volume associated with those queries climbs significantly starting just after midnight as well. This means that our resolvers encountered issues with the answers from .fj servers.

This observation suggests that the problem was strictly DNS related, rather than connectivity related—Cloudflare Radar Continue reading

Announcing experimental DDR in 1.1.1.1

1.1.1.1 sees approximately 600 billion queries per day. However, proportionally, most queries sent to this resolver are over cleartext: 89% over UDP and TCP combined, and the remaining 11% are encrypted. We care about end-user privacy and would prefer to see all of these queries sent to us over an encrypted transport using DNS-over-TLS or DNS-over-HTTPS. Having a mechanism by which clients could discover support for encrypted protocols such as DoH or DoT will help drive this number up and lead to more name encryption on the Internet. That’s where DDR – or Discovery of Designated Resolvers – comes into play. As of today, 1.1.1.1 supports the latest version of DDR so clients can automatically upgrade non-secure UDP and TCP connections to secure connections. In this post, we’ll describe the motivations for DDR, how the mechanism works, and, importantly, how you can test it out as a client.

DNS transports and public resolvers

We initially launched our public recursive resolver service 1.1.1.1 over three years ago, and have since seen its usage steadily grow. Today, it is one of the fastest public recursive resolvers available to end-users, supporting the latest security Continue reading

CVE-2022-26143: A Zero-Day vulnerability for launching UDP amplification DDoS attacks

A zero-day vulnerability in the Mitel MiCollab business phone system has recently been discovered (CVE-2022-26143). This vulnerability, called TP240PhoneHome, which Cloudflare customers are already protected against, can be used to launch UDP amplification attacks. This type of attack reflects traffic off vulnerable servers to victims, amplifying the amount of traffic sent in the process by an amplification factor of 220 billion percent in this specific case.

Cloudflare has been actively involved in investigating the TP240PhoneHome exploit, along with other members of the InfoSec community. Read our joint disclosure here for more details. As far as we can tell, the vulnerability has been exploited as early as February 18, 2022. We have deployed emergency mitigation rules to protect Cloudflare customers against the amplification DDoS attacks.

Mitel has been informed of the vulnerability. As of February 22, they have issued a high severity security advisory advising their customers to block exploitation attempts using a firewall, until a software patch is made available. Cloudflare Magic Transit customers can use the Magic Firewall to block external traffic to the exposed Mitel UDP port 10074 by following the example in the screenshot below, or by pasting the following expression into their Magic Firewall Continue reading

CVE-2022-26143: TP240PhoneHome reflection/amplification DDoS attack vector

Beginning in mid-February 2022, security researchers, network operators, and security vendors observed a spike in DDoS attacks sourced from UDP port 10074 targeting broadband access ISPs, financial institutions, logistics companies, and organizations in other vertical markets.

Upon further investigation, it was determined that the devices abused to launch these attacks are MiCollab and MiVoice Business Express collaboration systems produced by Mitel, which incorporate TP-240 VoIP- processing interface cards and supporting software; their primary function is to provide Internet-based site-to-site voice connectivity for PBX systems.

Approximately 2600 of these systems have been incorrectly provisioned so that an unauthenticated system test facility has been inadvertently exposed to the public Internet, allowing attackers to leverage these PBX VoIP gateways as DDoS reflectors/amplifiers.

Mitel is aware that these systems are being abused to facilitate high-pps (packets-per-second) DDoS attacks, and have been actively working with customers to remediate abusable devices with patched software that disables public access to the system test facility.

In this blog, we will further explore the observed activity, explain how the driver has been abused, and share recommended mitigation steps. This research was created cooperatively among a team of researchers from Akamai SIRT, Cloudflare, Lumen Black Lotus Labs, NETSCOUT ASERT, Continue reading

International Women’s Day 2022

“I would venture to guess that Anon,
who wrote so many poems without signing them,
was often a woman.” - Virginia Woolf

Welcome to International Women’s Day 2022! Here at Cloudflare, we are happy to celebrate it with you! Our celebration is not only this blog post, but many events prepared for the month of March: our way of honoring Women’s History Month by showcasing women’s empowerment. We want to celebrate the achievements, ideas, passion and work that women bring to the world. We want to advocate for equality and to achieve gender parity. And we want to highlight the brilliant work that our women colleagues do every day. Welcome!

This is a time of celebration but also one to reflect on the current state. The global gender gap is not expected to close for another 136 years. This gap has also worsened due to the COVID-19 pandemic, which has negatively impacted the lives of women and girls by deepening pre-existing inequalities. Improving this state is a collective effort—we all need to get involved!

Who are we? Womenflare!

First, let’s introduce ourselves. We are Womenflare—Cloudflare’s Employee Resource Group (ERG) for all who identify as and advocate for Continue reading

Cloudflare, CrowdStrike, and Ping Identity launch the Critical Infrastructure Defense Project

Today, in partnership with CrowdStrike and Ping Identity, Cloudflare is launching the Critical Infrastructure Defense Project (CriticalInfrastructureDefense.org). The Project was born out of conversations with cybersecurity and government experts concerned about potential retaliation to the sanctions that resulted from the Russian invasion of Ukraine.

In particular, there is a fear that critical United States infrastructure will be targeted with cyber attacks. While these attacks may target any industry, the experts we consulted with were particularly concerned about three areas that were often underprepared and could cause significant disruption: hospitals, energy, and water.

To help address that need, Cloudflare, CrowdStrike, and Ping Identity have committed under the Critical Infrastructure Defense Project to offer a broad suite of our products for free for at least the next four months to any United States-based hospital, or energy or water utility. You can learn more at: www.CriticalInfrastructureDefense.org.

We are not powerless against hackers. Organizations that have adopted a Zero Trust approach to security have been successful at mitigating even determined attacks. There are three core components to any Zero Trust security approach: 1) Network Security, 2) Endpoint Security; and 3) Identity.

Cloudflare, CrowdStrike, and Ping Identity are three of Continue reading

Steps we’ve taken around Cloudflare’s services in Ukraine, Belarus, and Russia

At Cloudflare, we've watched in horror the Russian invasion of Ukraine. As the possibility of war looked more likely, we began to carefully monitor the situation on the ground, with the goal of keeping our employees, our customers, and our network safe.

Helping protect Ukraine against cyberattacks

Attacks against the Internet in Ukraine began even before the start of the invasion. Those attacks—and the steady stream of DDoS attacks we’ve seen in the days since—prompted us to extend our services to Ukrainian government and telecom organizations at no cost in order to ensure they can continue to operate and deliver critical information to their citizens as well as to the rest of the world about what is happening to them.

Going beyond that, under Project Galileo, we are expediting onboarding of any Ukrainian entities for our full suite of protections. We are currently assisting more than sixty organizations in Ukraine and the region—with about 25% of those organizations coming aboard during the current crisis. Many of the new organizations are groups coming together to assist refugees, share vital information, or members of the Ukrainian diaspora in nearby countries looking to organize and help. Any Ukrainian organizations that are facing Continue reading

Shields up: free Cloudflare services to improve your cyber readiness

Since our founding, Cloudflare's mission has been to "help build a better Internet," and we take it to heart. It used to be that the services required to adequately secure an online presence were only available to the largest of enterprises — organizations big enough to afford both the technology itself and the teams to manage it.

We've worked hard over the years to level the playing field. This has meant making more and more of the essential tools for protecting an online presence available to as many people as possible. Cloudflare offers unmetered DDoS protection — for free. We were the first to introduce SSL at scale — for free. And it’s not just protection for your external-facing infrastructure: we have a free Zero Trust plan that enables teams to protect their internal-facing infrastructure, too.

These types of tools have always been important for the billions of people on the Internet. But perhaps never as important as they've become this week.

Concurrent with the Russian invasion of Ukraine, we've seen increasing cyberattacks on the Internet, too. Governments around the world are encouraging organizations to go “shields up” — with warnings coming from the United States’ Cybersecurity & Infrastructure Security Continue reading

Internet traffic patterns in Ukraine since February 21, 2022

Cloudflare operates in more than 250 cities worldwide where we connect our equipment to the Internet to provide our broad range of services. We have data centers in Ukraine, Belarus and Russia and across the world. To operate our service we monitor traffic trends, performance and errors seen at each data center, aggregate data about DNS, and congestion and packet loss on Internet links.

Internet Traffic

For reference, here is a map of Ukraine showing its major cities. Note that whenever we talk about dates and times in this post, we are using UTC. Ukraine’s current time zone is UTC+2.

Internet traffic in Ukraine generally follows a pretty predictable pattern based on day and night. Lowest in the hours after local midnight and picking up as people wake up. It’s not uncommon to see a dip around lunchtime and a peak when people go home in the evening. That pattern is clearly visible in this chart of overall Internet traffic seen by Cloudflare for Ukrainian networks on Monday, Tuesday, and Wednesday prior to the invasion.

Starting Thursday, traffic was significantly lower. On Thursday, we saw about 70% of our normal request volume and about 60% on Friday. Continue reading

iCloud Private Relay: information for Cloudflare customers

iCloud Private Relay is a new Internet privacy service from Apple that allows users with iOS 15, iPadOS 15, or macOS Monterey on their devices and an iCloud+ subscription, to connect to the Internet and browse with Safari in a more secure and private way. Cloudflare is proud to work with Apple to operate portions of Private Relay infrastructure.

In this post, we’ll explain how website operators can ensure the best possible experience for end users using iCloud Private Relay. Additional material is available from Apple, including “Set up iCloud Private Relay on all your devices”, and “Prepare Your Network or Web Server for iCloud Private Relay” which covers network operator scenarios in detail.

How browsing works using iCloud Private Relay

The design of the iCloud Private Relay system ensures that no single party handling user data has complete information on both who the user is and what they are trying to access.

To do this, Private Relay uses modern encryption and transport mechanisms to relay traffic from user devices through Apple and partner infrastructure before sending traffic to the destination website.

Here’s a diagram depicting what connection metadata is available to who when not using Private Relay Continue reading

The post-quantum future: challenges and opportunities

“People ask me to predict the future, when all I want to do is prevent it. Better yet, build it. Predicting the future is much too easy, anyway. You look at the people around you, the street you stand on, the visible air you breathe, and predict more of the same. To hell with more. I want better.”
— Ray Bradbury, from Beyond 1984: The People Machines

The story and the path are clear: quantum computers are coming that will have the ability to break the cryptographic mechanisms we rely on to secure modern communications, but there is hope! The cryptographic community has designed new mechanisms to safeguard against this disruption. There are challenges: will the new safeguards be practical? How will the fast-evolving Internet migrate to this new reality? In other blog posts in this series, we have outlined some potential solutions to these questions: there are new algorithms for maintaining confidentiality and authentication (in a “post-quantum” manner) in the protocols we use. But will they be fast enough to deploy at scale? Will they provide the required properties and work in all protocols? Are they easy to use?

Adding post-quantum cryptography into architectures and networks Continue reading

Post-quantumify internal services: Logfwrdr, Tunnel, and gokeyless

Theoretically, there is no impediment to adding post-quantum cryptography to any system. But the reality is harder. In the middle of last year, we posed ourselves a big challenge: to change all internal connections at Cloudflare to use post-quantum cryptography. We call this, in a cheeky way, “post-quantum-ifying” our services. Theoretically, this should be simple: swap algorithms for post-quantum ones and move along. But with dozens of different services in various programming languages (as we have at Cloudflare), it is not so simple. The challenge is big but we are here and up for the task! In this blog post, we will look at what our plan was, where we are now, and what we have learned so far. Welcome to the first announcement of a post-quantum future at Cloudflare: our connections are going to be quantum-secure!

What are we doing?

The life of most requests at Cloudflare begins and ends at the edge of our global network. Not all requests are equal and on their path they are transmitted by several protocols. Some of those protocols provide security properties whilst others do not. For the protocols that do, for context, Cloudflare uses: TLS, QUIC, WireGuard, DNSSEC Continue reading

HPKE: Standardizing public-key encryption (finally!)

For the last three years, the Crypto Forum Research Group of the Internet Research Task Force (IRTF) has been working on specifying the next generation of (hybrid) public-key encryption (PKE) for Internet protocols and applications. The result is Hybrid Public Key Encryption (HPKE), published today as RFC 9180.

HPKE was made to be simple, reusable, and future-proof by building upon knowledge from prior PKE schemes and software implementations. It is already in use in a large assortment of emerging Internet standards, including TLS Encrypted Client Hello and Oblivious DNS-over-HTTPS, and has a large assortment of interoperable implementations, including one in CIRCL. This article provides an overview of this new standard, going back to discuss its motivation, design goals, and development process.

A primer on public-key encryption

Public-key cryptography is decades old, with its roots going back to the seminal work of Diffie and Hellman in 1976, entitled “New Directions in Cryptography.” Their proposal – today called Diffie-Hellman key exchange – was a breakthrough. It allowed one to transform small secrets into big secrets for cryptographic applications and protocols. For example, one can bootstrap a secure channel for exchanging messages with confidentiality and integrity using a key exchange Continue reading