Cloudflare’s 2024 Annual Founders’ Letter
This week Cloudflare will celebrate the fourteenth anniversary of our launch. We think of it as our birthday. As is our tradition ever since our first anniversary, we use our Birthday Week each year to launch new products that we think of as gifts back to the Internet. For the last five years, we also take this time to write our annual Founders’ Letter reflecting on our business and the state of the Internet. This year is no different.
That said, one thing that is different is you may have noticed we've actually had fewer public innovation weeks over the last year than usual. That's been because a couple of incidents nearly a year ago caused us to focus on improving our internal systems over releasing new features. We're incredibly proud of our team's focus to make security, resilience, and reliability the top priorities for the last year. Today, Cloudflare's underlying platform, and the products that run on top of it, are significantly more robust than ever before.
With that work largely complete, and our platform in its strongest shape ever, we plan to pick back up the usual cadence of new product launches that we're known for. This Continue reading
Removing uncertainty through “what-if” capacity planning
Infrastructure planning for a network serving more than 81 million requests at peak and which is globally distributed across more than 330 cities in 120+ countries is complex. The capacity planning team at Cloudflare ensures there is enough capacity in place all over the world so that our customers have one less thing to worry about - our infrastructure, which should just work. Through our processes, the team puts careful consideration into “what-ifs”. What if something unexpected happens and one of our data centers fails? What if one of our largest customers triples, or quadruples their request count? Across a gamut of scenarios like these, the team works to understand where traffic will be served from and how the Cloudflare customer experience may change.
This blog post gives a look behind the curtain of how these scenarios are modeled at Cloudflare, and why it's so critical for our customers.
Scenario planning and our customers
Cloudflare customers rely on the data centers that Cloudflare has deployed all over the world, placing us within 50 ms of approximately 95% of the Internet-connected population globally. But round-trip time to our end users means little if those data centers don’t have the capacity Continue reading
Cloudflare incident on September 17, 2024
On September 17, 2024, during routine maintenance, Cloudflare inadvertently stopped announcing fifteen IPv4 prefixes, affecting some Business plan websites for approximately one hour. During this time, IPv4 traffic for these customers would not have reached Cloudflare, and users attempting to connect to websites assigned addresses within those prefixes would have received errors.
We’re very sorry for this outage.
This outage was the result of an internal software error and not the result of an attack. In this blog post, we’re going to talk about what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again.
Background
Cloudflare assembled a dedicated Addressing team in 2019 to simplify the ways that IP addresses are used across Cloudflare products and services. The team builds and maintains systems that help Cloudflare conserve and manage its own network resources. The Addressing team also manages periodic changes to the assignment of IP addresses across infrastructure and services at Cloudflare. In this case, our goal was to reduce the number of IPv4 addresses used for customer websites, allowing us to free up addresses for other purposes, like deploying infrastructure in new locations. Since IPv4 addresses are a finite Continue reading
How Cloudflare is helping domain owners with the upcoming Entrust CA distrust by Chrome and Mozilla
Chrome and Mozilla announced that they will stop trusting Entrust’s public TLS certificates issued after November 12, 2024 and December 1, 2024, respectively. This decision stems from concerns related to Entrust’s ability to meet the CA/Browser Forum’s requirements for a publicly trusted certificate authority (CA). To prevent Entrust customers from being impacted by this change, Entrust has announced that they are partnering with SSL.com, a publicly trusted CA, and will be issuing certs from SSL.com’s roots to ensure that they can continue to provide their customers with certificates that are trusted by Chrome and Mozilla.
We’re excited to announce that we’re going to be adding SSL.com as a certificate authority that Cloudflare customers can use. This means that Cloudflare customers that are currently relying on Entrust as a CA and uploading their certificate manually to Cloudflare will now be able to rely on Cloudflare’s certificate management pipeline for automatic issuance and renewal of SSL.com certificates.
CA distrust: responsibilities, repercussions, and responses
With great power comes great responsibility Every publicly trusted certificate authority (CA) is responsible for maintaining a high standard of security and compliance to ensure that the certificates they issue are trustworthy. Continue reading
Protecting APIs from abuse using sequence learning and variable order Markov chains
Consider the case of a malicious actor attempting to inject, scrape, harvest, or exfiltrate data via an API. Such malicious activities are often characterized by the particular order in which the actor initiates requests to API endpoints. Moreover, the malicious activity is often not readily detectable using volumetric techniques alone, because the actor may intentionally execute API requests slowly, in an attempt to thwart volumetric abuse protection. To reliably prevent such malicious activity, we therefore need to consider the sequential order of API requests. We use the term sequential abuse to refer to malicious API request behavior. Our fundamental goal thus involves distinguishing malicious from benign API request sequences.
In this blog post, you’ll learn about how we address the challenge of helping customers protect their APIs against sequential abuse. To this end, we’ll unmask the statistical machine learning (ML) techniques currently underpinning our Sequence Analytics product. We’ll build on the high-level introduction to Sequence Analytics provided in a previous blog post.
API sessions
Introduced in the previous blog post, let’s consider the idea of a time-ordered series of HTTP API requests initiated by a specific user. These occur as the user interacts with a service, such as while browsing Continue reading
How the Harris-Trump US presidential debate influenced Internet traffic
Much has changed in the 2024 United States presidential election since the June 27 debate between Donald Trump and Joe Biden, then the presumptive nominees for the November election. Now, over two months later, on September 10, the debate was between Kamala Harris, the Democratic nominee, and Donald Trump, the Republican nominee. In this post, we will explore the event's impact on Internet traffic in specific states where there was a bigger impact than during the Biden-Trump debate, as well as examine cyberattacks, email phishing trends, and general DNS data on candidates, news, and election-related activity.
We’ve been tracking the 2024 elections globally through our blog and election report on Cloudflare Radar, covering some of the more than 60 national elections this year. Regarding the US elections, we have previously reported on trends surrounding the first Biden vs. Trump debate, the attempted assassination of Trump, the Republican National Convention, and the Democratic National Convention.
Typically, we have observed that election days don’t come with significant changes to Internet traffic, and the same is true for debates. Yet, debates can also draw attention that impacts traffic, especially when there is heightened anticipation. The 2024 debates were not only Continue reading
Customers get increased integration with Cloudflare Email Security and Zero Trust through expanded partnership with CrowdStrike
Today, we’re excited to expand our recent Unified Risk Posture announcement with more information on our latest integrations with CrowdStrike. We previously shared that our CrowdStrike Falcon Next-Gen SIEM integration allows for deeper analysis and further investigations by unifying first- and third-party data, native threat intelligence, AI, and workflow automation to allow your security teams to focus on work that matters.
This post explains how Falcon Next-Gen SIEM allows customers to identify and investigate risky user behavior and analyze data combined with other log sources to uncover hidden threats. By combining Cloudflare and CrowdStrike, organizations are better equipped to manage risk and decisively take action to stop cyberattacks.
By leveraging the combined capabilities of Cloudflare and CrowdStrike, organizations combine Cloudflare’s email security and zero trust logging capabilities with CrowdStrike’s dashboards and custom workflows to get better visibility into their environments and remediate potential threats. Happy Cog, a full-service digital agency, currently leverages the integration. Co-Founder and President Matthew Weinberg said:
'The integration of Cloudflare’s robust Zero Trust capabilities with CrowdStrike Falcon Next-Gen SIEM enables organizations to gain a more comprehensive view of the threat landscape and take action to mitigate both internal and external risks posed by today’s security Continue reading
A good day to trie-hard: saving compute 1% at a time
Cloudflare’s global network handles a lot of HTTP requests – over 60 million per second on average. That in and of itself is not news, but it is the starting point to an adventure that started a few months ago and ends with the announcement of a new open-source Rust crate that we are using to reduce our CPU utilization, enabling our CDN to handle even more of the world’s ever-increasing Web traffic.
Motivation
Let’s start at the beginning. You may recall a few months ago we released Pingora (the heart of our Rust-based proxy services) as an open-source project on GitHub. I work on the team that maintains the Pingora framework, as well as Cloudflare’s production services built upon it. One of those services is responsible for the final step in transmitting users’ (non-cached) requests to their true destination. Internally, we call the request’s destination server its “origin”, so our service has the (unimaginative) name of “pingora-origin”.
One of the many responsibilities of pingora-origin is to ensure that when a request leaves our infrastructure, it has been cleaned to remove the internal information we use to route, measure, and optimize traffic for our customers. This has to be Continue reading
More NPM packages on Cloudflare Workers: Combining polyfills and native code to support Node.js APIs
Today, we are excited to announce a preview of improved Node.js compatibility for Workers and Pages. Broader compatibility lets you use more NPM packages and take advantage of the JavaScript ecosystem when writing your Workers.
Our newest version of Node.js compatibility combines the best features of our previous efforts. Cloudflare Workers have supported Node.js in some form for quite a while. We first announced polyfill support in 2021, and later built-in support for parts of the Node.js API that has expanded over time.
The latest changes make it even better:
You can use far more NPM packages on Workers.
You can use packages that do not use the
node: prefix to import Node.js APIsYou can use more Node.js APIs on Workers, including most methods on
async_hooks,buffer,dns,os, andevents. Many more, such asfsorprocessare importable with mocked methods.
To give it a try, add the following flag to wrangler.toml, and deploy your Worker with Wrangler:
compatibility_flags = ["nodejs_compat_v2"]
Packages that could not be imported with nodejs_compat, even as a dependency of another package, will now load. This Continue reading
Bringing insights into TCP resets and timeouts to Cloudflare Radar
Cloudflare handles over 60 million HTTP requests per second globally, with approximately 70% received over TCP connections (the remaining are QUIC/UDP). Ideally, every new TCP connection to Cloudflare would carry at least one request that results in a successful data exchange, but that is far from the truth. In reality, we find that, globally, approximately 20% of new TCP connections to Cloudflare’s servers time out or are closed with a TCP “abort” message either before any request can be completed or immediately after an initial request.
This post explores those connections that, for various reasons, appear to our servers to have been halted unexpectedly before any useful data exchange occurs. Our work reveals that while connections are normally ended by clients, they can also be closed due to third-party interference. Today we’re excited to launch a new dashboard and API endpoint on Cloudflare Radar that shows a near real-time view of TCP connections to Cloudflare’s network that terminate within the first 10 ingress packets due to resets or timeouts, which we’ll refer to as anomalous TCP connections in this post. Analyzing this anomalous behavior provides insights into scanning, connection tampering, DoS attacks, connectivity issues, and other behaviors.
A global assessment of third-party connection tampering
Have you ever made a phone call, only to have the call cut as soon as it is answered, with no obvious reason or explanation? This analogy is the starting point for understanding connection tampering on the Internet and its impact.
We have found that 20 percent of all Internet connections are abruptly closed before any useful data can be exchanged. Essentially, every fifth call is cut before being used. As with a phone call, it can be challenging for one or both parties to know what happened. Was it a faulty connection? Did the person on the other end of the line hang up? Did a third party intervene to stop the call?
On the Internet, Cloudflare is in a unique position to help figure out when a third party may have played a role. Our global network allows us to identify patterns that suggest that an external party may have intentionally tampered with a connection to prevent content from being accessed. Although they are often hard to decipher, the ways connections are abruptly closed give clues to what might have happened. Sources of tampering generally do not try to hide their actions, which leaves hints of Continue reading
Making progress on routing security: the new White House roadmap
The Internet can feel like magic. When you load a webpage in your browser, many simultaneous requests for data fly back and forth to remote servers. Then, often in less than one second, a website appears. Many people know that DNS is used to look up a hostname, and resolve it to an IP address, but fewer understand how data flows from your home network to the network that controls the IP address of the web server.
The Internet is an interconnected network of networks, operated by thousands of independent entities. To allow these networks to communicate with each other, in 1989, on the back of two napkins, three network engineers devised the Border Gateway Protocol (BGP). It allows these independent networks to signal directions for IP prefixes they own, or that are reachable through their network. At that time, Internet security wasn’t a big deal — SSL, initially developed to secure websites, wasn’t developed until 1995, six years later. So BGP wasn’t originally built with security in mind, but over time, security and availability concerns have emerged.
Today, the White House Office of the National Cyber Director issued the Roadmap to Enhancing Internet Routing Security, and Continue reading
Exploring Internet and security trends during the 2024 U.S. Democratic National Convention
The 2024 Democratic National Convention (DNC) wrapped up on Thursday, August 22, in Chicago, Illinois. Since our blog post about Internet trends during the first presidential debate between President Joe Biden and former President Donald Trump on June 27, the presidential race has fundamentally changed. We experienced the attempted assassination of Trump, the Republican National Convention (RNC), Biden’s late July withdrawal from the race, and Vice President Kamala Harris being selected as the Democratic nominee and participating in her party’s convention this week. Here, we’ll examine trends more focused on DNS traffic to news and candidate-related sites, cyberattacks targeting politically-related organizations, and spam and malicious emails mentioning the candidates’ names.
Over 60 more national elections are scheduled to take place across the world this year, and we have been monitoring them as they occur. Our goal is to provide a neutral analysis of their impact on Internet behavior, which often mirrors human activities. Significant events, such as the total eclipse in Mexico, the United States, and Canada, and the Paris 2024 Olympics, have had an impact on Internet traffic. Our ongoing election report on Cloudflare Radar includes updates from recent elections in the European Union, France, Continue reading
Go wild: Wildcard support in Rules and a new open-source wildcard crate
Back in 2012, we introduced Page Rules, a pioneering feature that gave Cloudflare users unprecedented control over how their web traffic was managed. At the time, this was a significant leap forward, enabling users to define patterns for specific URLs and adjust Cloudflare features on a page-by-page basis. The ability to apply such precise configurations through a simple, user-friendly interface was a major advancement, establishing Page Rules as a cornerstone of our platform.
Page Rules allowed users to implement a variety of actions, including redirects, which automatically send visitors from one URL to another. Redirects are crucial for maintaining a seamless user experience on the Internet, whether it's guiding users from outdated links to new content or managing traffic during site migrations.
As the Internet has evolved, so too have the needs of our users. The demand for greater flexibility, higher performance, and more advanced capabilities led to the development of the Ruleset Engine, a powerful framework designed to handle complex rule evaluations with unmatched speed and precision.
In September 2022, we announced and released Single Redirects as a modern replacement for the URL Forwarding feature of Page Rules. Built on top of the Ruleset Engine, this Continue reading
NIST’s first post-quantum standards
On August 13th, 2024, the US National Institute of Standards and Technology (NIST) published the first three cryptographic standards designed to resist an attack from quantum computers: ML-KEM, ML-DSA, and SLH-DSA. This announcement marks a significant milestone for ensuring that today’s communications remain secure in a future world where large-scale quantum computers are a reality.
In this blog post, we briefly discuss the significance of NIST’s recent announcement, how we expect the ecosystem to evolve given these new standards, and the next steps we are taking. For a deeper dive, see our March 2024 blog post.
Why are quantum computers a threat?
Cryptography is a fundamental aspect of modern technology, securing everything from online communications to financial transactions. For instance, when visiting this blog, your web browser used cryptography to establish a secure communication channel to Cloudflare’s server to ensure that you’re really talking to Cloudflare (and not an impersonator), and that the conversation remains private from eavesdroppers.
Much of the cryptography in widespread use today is based on mathematical puzzles (like factoring very large numbers) which are computationally out of reach for classical (non-quantum) computers. We could likely continue to use traditional cryptography for decades to Continue reading
NIST’s first post-quantum standards
On August 13th, 2024, the US National Institute of Standards and Technology (NIST) published the first three cryptographic standards designed to resist an attack from quantum computers: ML-KEM, ML-DSA, and SLH-DSA. This announcement marks a significant milestone for ensuring that today’s communications remain secure in a future world where large-scale quantum computers are a reality.
In this blog post, we briefly discuss the significance of NIST’s recent announcement, how we expect the ecosystem to evolve given these new standards, and the next steps we are taking. For a deeper dive, see our March 2024 blog post.
Why are quantum computers a threat?
Cryptography is a fundamental aspect of modern technology, securing everything from online communications to financial transactions. For instance, when visiting this blog, your web browser used cryptography to establish a secure communication channel to Cloudflare’s server to ensure that you’re really talking to Cloudflare (and not an impersonator), and that the conversation remains private from eavesdroppers.
Much of the cryptography in widespread use today is based on mathematical puzzles (like factoring very large numbers) which are computationally out of reach for classical (non-quantum) computers. We could likely continue to use traditional cryptography for decades to Continue reading
Summer 2024 weather report: Cloudflare with a chance of Intern-ets
During the summer of 2024, Cloudflare welcomed approximately 60 Intern-ets from all around the globe on a mission to #HelpBuildABetterInternet. Over the course of their internships, our wonderful interns tackled real-world challenges from different teams all over the company and contributed to cutting-edge projects. As returning interns, we – Shaheen, Aaron, and Jada – would like to show off the great work our cohort has done and experiences we’ve had throughout our time here.
Austin Interns after volunteering at the Central Texas Food Bank.
Putting the SHIP in internSHIP
Cloudflare interns take pride in driving high-impact initiatives, playing a vital role in advancing Cloudflare's mission. With our diverse roles and projects this summer, we'd love to highlight some of the exciting work we've been involved in:
Rahul, a Software Engineer intern, built a system to autograde intern application assignments for future students looking to join Cloudflare. It was built entirely on the Cloudflare Developer Platform, using Cloudflare Access, Browser Rendering, D1, Durable Objects, R2, and Workers!
Jessica, a Software Engineer intern, created a new threads api for the Workers AI team that automatically recalls past messages when running inference, helping developers to Continue reading
Summer 2024 weather report: Cloudflare with a chance of Intern-ets
During the summer of 2024, Cloudflare welcomed approximately 60 Intern-ets from all around the globe on a mission to #HelpBuildABetterInternet. Over the course of their internships, our wonderful interns tackled real-world challenges from different teams all over the company and contributed to cutting-edge projects. As returning interns, we – Shaheen, Aaron, and Jada – would like to show off the great work our cohort has done and experiences we’ve had throughout our time here.
Putting the SHIP in internSHIP
Cloudflare interns take pride in driving high-impact initiatives, playing a vital role in advancing Cloudflare's mission. With our diverse roles and projects this summer, we'd love to highlight some of the exciting work we've been involved in:
Rahul, a Software Engineer intern, built a system to autograde intern application assignments for future students looking to join Cloudflare. It was built entirely on the Cloudflare Developer Platform, using Cloudflare Access, Browser Rendering, D1, Durable Objects, R2, and Workers!
Jessica, a Software Engineer intern, created a new threads api for the Workers AI team that automatically recalls past messages when running inference, helping developers to Continue reading
A wild week in phishing, and what it means for you
Being a bad guy on the Internet is a really good business. In more than 90% of cybersecurity incidents, phishing is the root cause of the attack, and during this third week of August phishing attacks were reported against the U.S. elections, in the geopolitical conflict between the U.S., Israel, and Iran, and to cause $60M in corporate losses.
You might think that after 30 years of email being the top vector for attack and risk we are helpless to do anything about it, but that would be giving too much credit to bad actors, and a misunderstanding of how defenders focused on detections can take control and win.
Phishing isn’t about email exclusively, or any specific protocol for that matter. Simply put, it is an attempt to get a person, like you or me, to take an action that unwittingly leads to damages. These attacks work because they appear to be authentic, visually or organizationally, such as pretending to be the CEO or CFO of your company, and when you break it down they are three main attack vectors that Cloudflare has seen most impactful from the bad emails we protect our customers Continue reading
A wild week in phishing, and what it means for you
This post is also available in 한국어, 日本語, 简体中文, 繁體中文, Français, Deutsch and Español.
Being a bad guy on the Internet is a really good business. In more than 90% of cybersecurity incidents, phishing is the root cause of the attack, and during this third week of August phishing attacks were reported against the U.S. elections, in the geopolitical conflict between the U.S., Israel, and Iran, and to cause $60M in corporate losses.
You might think that after 30 years of email being the top vector for attack and risk we are helpless to do anything about it, but that would be giving too much credit to bad actors, and a misunderstanding of how defenders focused on detections can take control and win.
Phishing isn’t about email exclusively, or any specific protocol for that matter. Simply put, it is an attempt to get a person, like you or me, to take an action that unwittingly leads to damages. These attacks work because they appear to be authentic, visually or organizationally, such as pretending to be the CEO or CFO of your company, and when you break it down they are three Continue reading