Major data center power failure (again): Cloudflare Code Orange tested

This post is also available in Français, Español.

Here's a post we never thought we'd need to write: less than five months after one of our major data centers lost power, it happened again to the exact same data center. That sucks and, if you're thinking "why do they keep using this facility??," I don't blame you. We're thinking the same thing. But, here's the thing, while a lot may not have changed at the data center, a lot changed over those five months at Cloudflare. So, while five months ago a major data center going offline was really painful, this time it was much less so.

This is a little bit about how a high availability data center lost power for the second time in five months. But, more so, it's the story of how our team worked to ensure that even if one of our critical data centers lost power it wouldn't impact our customers.

On November 2, 2023, one of our critical facilities in the Portland, Oregon region lost power for an extended period of time. It happened because of a cascading series of faults that appears to have been caused by maintenance by the Continue reading

Post Mortem on Cloudflare Control Plane and Analytics Outage

Beginning on Thursday, November 2, 2023 at 11:43 UTC Cloudflare's control plane and analytics services experienced an outage. The control plane of Cloudflare consists primarily of the customer-facing interface for all of our services including our website and APIs. Our analytics services include logging and analytics reporting.

The incident lasted from November 2 at 11:44 UTC until November 4 at 04:25 UTC. We were able to restore most of our control plane at our disaster recovery facility as of November 2 at 17:57 UTC. Many customers would not have experienced issues with most of our products after the disaster recovery facility came online. However, other services took longer to restore and customers that used them may have seen issues until we fully resolved the incident. Our raw log services were unavailable for most customers for the duration of the incident.

Services have now been restored for all customers. Throughout the incident, Cloudflare's network and security services continued to work as expected. While there were periods where customers were unable to make changes to those services, traffic through our network was not impacted.

This post outlines the events that caused this incident, the architecture we had in place to prevent issues Continue reading

Cloudflare’s 2023 Annual Founders’ Letter

Cloudflare is officially a teenager. We launched on September 27, 2010. Today we celebrate our thirteenth birthday. As is our tradition, we use the week of our birthday to launch products that we think of as our gift back to the Internet. More on some of the incredible announcements in a second, but we wanted to start by talking about something more fundamental: our identity.

Like many kids, it took us a while to fully understand who we are. We chafed at being put in boxes. People would describe Cloudflare as a security company, and we'd say, "That's not all we do." They'd say we were a network, and we'd object that we were so much more. Worst of all, they'd sometimes call us a "CDN," and we'd remind them that caching is a part of any sensibly designed system, but it shouldn't be a feature unto itself. Thank you very much.

And so, yesterday, the day before our thirteenth birthday, we announced to the world finally what we realized we are: a connectivity cloud.

The connectivity cloud

What does that mean? "Connectivity" means we measure ourselves by connecting people and things together. Our job isn't to be the Continue reading

Welcome to Cloudflare’s Impact Week

In the early days of Cloudflare, we made it a policy that every new hire had to interview with either me or my co-founder Michelle. It’s still the case today, though we now have more than 3,000 employees, continue to hire great people as we find them, and, because there are only so many hours in the day, have had to enlist a few more senior executives to help with these final calls.

At first, these calls were about helping screen for new members of our small team. But, as our team grew, the purpose of these calls changed. Today, by the time I do the final call with someone we’ve made the decision to hire them, so it’s rarely about screening. Instead, the primary purpose is to make sure everyone joining has had a positive conversation with a senior member of our team, so if in the future they ever see something going wrong they’ll hopefully feel a bit more comfortable letting one of us know. I think because of that these calls are some of the most important work I do.

But, for me, there’s another purpose. I get to hear first-hand why people chose to apply. That’s Continue reading

Adjusting pricing, introducing annual plans, and accelerating innovation

This post is also available in 繁體中文, 简体中文, 日本語, 한국어, Deutsch, Français, Pусский, Español, Português.

Cloudflare is raising prices for the first time in the last 12 years. Beginning January 15, 2023, new sign ups will be charged \$25 per month for our Pro Plan (up from \$20 per month) and \$250 per month for our Business Plan (up from \$200 per month). Any paying customers who sign up before January 15, 2023, including any currently paying customers who signed up at any point over the last 12 years, will be grandfathered at the old monthly price until May 14, 2023.

We are also introducing an option to pay annually, rather than monthly, that we hope most customers will choose to switch to. Annual plans are available today and discounted from the new monthly rate to \$240 per year for the Pro Plan (the equivalent of \$20 per month, saving \$60 per year) and \$2,400 per year for the Business Plan (the equivalent of \$200 per month, saving \$600 per year). In other words, if you choose to pay annually for Cloudflare you can lock in our old monthly prices.

After not Continue reading

Cloudflare’s 2022 Annual Founders’ Letter

Cloudflare launched on September 27, 2010. This week we'll celebrate our 12th birthday. As has become our tradition, we'll be announcing a series of products that we think of as our gifts back to the Internet. In previous years, these have included products and initiatives like Universal SSL, Cloudflare Workers, our Zero Markup Registrar, the Bandwidth Alliance, and R2 — our zero egress fee object store — which went GA last week.

We're really excited for what we'll be announcing this year and hope to surprise and delight all of you over the course of the week with the products and features we believe live up to our mission of helping build a better Internet.

Founders' letter

While this will be our 12th Birthday Week of product announcements, for the last two years, as the cofounders of the company, we've also taken this time as an opportunity to write a letter publicly reflecting on the previous year and what's on our minds as we go into the year ahead.

Since our last birthday, it's been a tale of two halves of a very different year. At the end of 2021 and into the first two months Continue reading

Blocking Kiwifarms

We have blocked Kiwifarms. Visitors to any of the Kiwifarms sites that use any of Cloudflare's services will see a Cloudflare block page and a link to this post. Kiwifarms may move their sites to other providers and, in doing so, come back online, but we have taken steps to block their content from being accessed through our infrastructure.

This is an extraordinary decision for us to make and, given Cloudflare's role as an Internet infrastructure provider, a dangerous one that we are not comfortable with. However, the rhetoric on the Kiwifarms site and specific, targeted threats have escalated over the last 48 hours to the point that we believe there is an unprecedented emergency and immediate threat to human life unlike we have previously seen from Kiwifarms or any other customer before.

Escalating threats

Kiwifarms has frequently been host to revolting content. Revolting content alone does not create an emergency situation that necessitates the action we are taking today. Beginning approximately two weeks ago, a pressure campaign started with the goal to deplatform Kiwifarms. That pressure campaign targeted Cloudflare as well as other providers utilized by the site.

Cloudflare provides security services to Kiwifarms, protecting them from DDoS and Continue reading

Cloudflare’s abuse policies & approach

Cloudflare launched nearly twelve years ago. We’ve grown to operate a network that spans more than 275 cities in over 100 countries. We have millions of customers: from small businesses and individual developers to approximately 30 percent of the Fortune 500. Today, more than 20 percent of the web relies directly on Cloudflare’s services.

Over the time since we launched, our set of services has become much more complicated. With that complexity we have developed policies around how we handle abuse of different Cloudflare features. Just as a broad platform like Google has different abuse policies for search, Gmail, YouTube, and Blogger, Cloudflare has developed different abuse policies as we have introduced new products.

We published our updated approach to abuse last year at:

https://www.cloudflare.com/trust-hub/abuse-approach/

However, as questions have arisen, we thought it made sense to describe those policies in more detail here.  

The policies we built reflect ideas and recommendations from human rights experts, activists, academics, and regulators. Our guiding principles require abuse policies to be specific to the service being used. This is to ensure that any actions we take both reflect the ability to address the harm and minimize unintended consequences. We believe that Continue reading

The mechanics of a sophisticated phishing scam and how we stopped it

Yesterday, August 8, 2022, Twilio shared that they’d been compromised by a targeted phishing attack. Around the same time as Twilio was attacked, we saw an attack with very similar characteristics also targeting Cloudflare’s employees. While individual employees did fall for the phishing messages, we were able to thwart the attack through our own use of Cloudflare One products, and physical security keys issued to every employee that are required to access all our applications.

We have confirmed that no Cloudflare systems were compromised. Our Cloudforce One threat intelligence team was able to perform additional analysis to further dissect the mechanism of the attack and gather critical evidence to assist in tracking down the attacker.

This was a sophisticated attack targeting employees and systems in such a way that we believe most organizations would be likely to be breached. Given that the attacker is targeting multiple organizations, we wanted to share here a rundown of exactly what we saw in order to help other companies recognize and mitigate this attack.

Targeted Text Messages

On July 20, 2022, the Cloudflare Security team received reports of employees receiving legitimate-looking text messages pointing to what appeared to be a Cloudflare Okta login Continue reading

What Cloudflare is Doing to Keep the Open Internet Flowing into Russia and Keep Attacks from Getting Out

Following Russia’s unjustified and tragic invasion of Ukraine in late February, the world has watched closely as Russian troops attempted to advance across Ukraine, only to be resisted and repelled by the Ukrainian people. Similarly, we’ve seen a significant amount of cyber attack activity in the region. We continue to work to protect an increasing number of Ukrainian government, media, financial, and nonprofit websites, and we protected the Ukrainian top level domain (.ua) to help keep Ukraine’s presence on the Internet operational.

At the same time, we’ve closely watched significant and unprecedented activity on the Internet in Russia. The Russian government has taken steps to tighten its control over both the technical components and the content of the Russian Internet. For their part, the people in Russia are doing something very different. They have been adopting tools to maintain access to the global Internet, and they have been seeking out non-Russian media sources. This blog post outlines what we’ve observed.

The Russian Government asserts control over the Internet

Over the last five years, the Russian government has taken steps to tighten its control of a sovereign Internet within Russia’s borders, including laws requiring Russian ISPs to install equipment allowing Continue reading

Cloudflare, CrowdStrike, and Ping Identity launch the Critical Infrastructure Defense Project

Today, in partnership with CrowdStrike and Ping Identity, Cloudflare is launching the Critical Infrastructure Defense Project (CriticalInfrastructureDefense.org). The Project was born out of conversations with cybersecurity and government experts concerned about potential retaliation to the sanctions that resulted from the Russian invasion of Ukraine.

In particular, there is a fear that critical United States infrastructure will be targeted with cyber attacks. While these attacks may target any industry, the experts we consulted with were particularly concerned about three areas that were often underprepared and could cause significant disruption: hospitals, energy, and water.

To help address that need, Cloudflare, CrowdStrike, and Ping Identity have committed under the Critical Infrastructure Defense Project to offer a broad suite of our products for free for at least the next four months to any United States-based hospital, or energy or water utility. You can learn more at: www.CriticalInfrastructureDefense.org.

We are not powerless against hackers. Organizations that have adopted a Zero Trust approach to security have been successful at mitigating even determined attacks. There are three core components to any Zero Trust security approach: 1) Network Security, 2) Endpoint Security; and 3) Identity.

Cloudflare, CrowdStrike, and Ping Identity are three of Continue reading

Steps we’ve taken around Cloudflare’s services in Ukraine, Belarus, and Russia

At Cloudflare, we've watched in horror the Russian invasion of Ukraine. As the possibility of war looked more likely, we began to carefully monitor the situation on the ground, with the goal of keeping our employees, our customers, and our network safe.

Helping protect Ukraine against cyberattacks

Attacks against the Internet in Ukraine began even before the start of the invasion. Those attacks—and the steady stream of DDoS attacks we’ve seen in the days since—prompted us to extend our services to Ukrainian government and telecom organizations at no cost in order to ensure they can continue to operate and deliver critical information to their citizens as well as to the rest of the world about what is happening to them.

Going beyond that, under Project Galileo, we are expediting onboarding of any Ukrainian entities for our full suite of protections. We are currently assisting more than sixty organizations in Ukraine and the region—with about 25% of those organizations coming aboard during the current crisis. Many of the new organizations are groups coming together to assist refugees, share vital information, or members of the Ukrainian diaspora in nearby countries looking to organize and help. Any Ukrainian organizations that are facing Continue reading

Why Cloudflare Bought Zaraz

Today we're excited to announce that Cloudflare has acquired Zaraz. The Zaraz value proposition aligns with Cloudflare's mission. They aim to make the web more secure, more reliable, and faster. And they built their solution on Cloudflare Workers. In other words, it was a no-brainer that we invite them to join our team.

Be Careful Who Takes Out the Trash

To understand Zaraz's value proposition, you need to understand one of the biggest risks to most websites that people aren't paying enough attention to. And, to understand that, let me use an analogy.

Imagine you run a business. Imagine that business is, I don't know, a pharmacy. You have employees. They have a process and way they do things. They're under contract, and you conduct background checks before you hire them. They do their jobs well and you trust them. One day, however, you realize that no one is emptying the trash. So you ask your team to find someone to empty the trash regularly.

Your team is busy and no one has the time to add this to their regular duties. But one plucky employee has an idea. He goes out on the street and hails down a relative Continue reading

Cloudflare’s Annual Founders’ Letter

This week we celebrate Cloudflare's birthday. We launched the company 11 years ago tomorrow: September 27, 2010. It has been our tradition, since our first birthday, to use this week to launch innovative new products that we think of as our gift back to the Internet.

Since going public, it's also been an opportunity for us to update our Annual Founders' Letter and share what's on our mind. Recently we've been thinking about three things: team, the Internet, and innovation.

Team

When anyone asks us the key to Cloudflare's success, we always say the same thing: the team we've been able to attract to help us achieve our mission of helping build a better Internet. In the last year we've had more than 250,000 people apply to work for us and extended offers to less than one half of one percent of them. We continue to attract great people.

It's incredible to realize that more than half of Cloudflare's team today started since March 13, 2020, when we closed all our physical offices due to the pandemic. In the last several months, as we've started to see a light at the end of the COVID tunnel, we've been hosting what Continue reading

The Future of Work at Cloudflare

During Impact Week, we've shared how Cloudflare is providing tools for our customers to minimize their environmental impact as well as what we, as a company, are doing to help society at large. But some critical stakeholders we haven’t talked much about yet are Cloudflare's more than 2,000 employees: who build our services, support and educate our customers, keep our finances in order, work through difficult policy issues, and empower us to accomplish everything we have.

Over the last year and a half, we've all challenged a lot of the assumptions about what it means to "work." Prior to the start of the pandemic, Cloudflare was very much a work-from-office culture. And so when, on March 13, 2020, we closed all our offices and asked everyone to work from home, the two of us were extremely nervous.

And then something unexpected happened: a lot of things got better.

As a company, productivity increased — when measured by our success selling our products, our pace of shipping new products, and even things like the time it takes for our finance team to close our books.

Other day-to-day things got better, too. We noticed a marked increase in participation in Continue reading

Helping build a green Internet

When we started Cloudflare, we weren't thinking about minimizing the environmental impact of the Internet. Frankly, I didn't really think of the Internet as having much of an environmental impact. It was just this magical resource that gave access to information and services from anywhere.

But that was before I started racking servers in hyper-cooled data centers. Before Cloudflare started paying the bills to keep those servers powered up and cooled down. Before we became obsessed with maximizing the number of requests we could process per watt of power. And long before we started buying directly from renewable power suppliers to drive down the cost of electricity across our network.

Today, I have a very good understanding of how much power it takes to run the Internet. It therefore wasn't surprising to read the Boston Consulting Group study which found that 2% of all carbon output, about 1 billion metric tons per year, is attributable to the Internet. That’s the equivalent of the entire aviation industry.

Cloudflare: Accidentally Environmentally Friendly By Design

While we didn't set out to reduce the environmental impact of the Internet, Cloudflare has always had efficiency at its core. It comes from our ongoing fight with Continue reading

Welcome to Cloudflare Impact Week

If I'm completely honest, Cloudflare didn't start out as a mission-driven company. When Lee, Michelle, and I first started thinking about starting a company in 2009 we saw an opportunity as the world was shifting from on-premise hardware and software to services in the cloud. It seemed inevitable to us that the same shift would come to security, performance, and reliability services. And, getting ahead of that trend, we could build a great business.

One problem we had was that we knew in order to have a great business we needed to win large organizations with big IT budgets as customers. And, in order to do that, we needed to have the data to build a service that would keep them safe. But we only could get data on security threats once we had customers. So we had a chicken and egg problem.

Our solution was to provide a basic version of Cloudflare's services for free. We reasoned that individual developers and small businesses would sign up for the free service. We'd learn a lot about security threats and performance and reliability opportunities based on their traffic data. And, Continue reading

AWS’s Egregious Egress

When web hosting services first emerged in the mid-1990s, you paid for everything on a separate meter: bandwidth, storage, CPU, and memory. Over time, customers grew to hate the nickel-and-dime nature of these fees. The market evolved to a fixed-fee model. Then came Amazon Web Services.

AWS was a huge step forward in terms of flexibility and scalability, but a massive step backward in terms of pricing. Nowhere is that more apparent than with their data transfer (bandwidth) pricing. If you look at the (ironically named) AWS Simple Monthly Calculator you can calculate the price they charge for bandwidth for their typical customer. The price varies by region, which shouldn't surprise you because the cost of transit is dramatically different in different parts of the world.

Charging for Stocks, Paying for Flows

AWS charges customers based on the amount of data delivered — 1 terabyte (TB) per month, for example. To visualize that, imagine data is water. AWS fills a bucket full of water and then charges you based on how much water is in the bucket. This is known as charging based on “stocks.”

On the other hand, AWS pays for bandwidth based on the capacity of their Continue reading

Welcome to Cloudflare Security Week 2021!

Today kicks off Cloudflare's 2021 Security Week. Like all innovation weeks at Cloudflare, we'll be announcing a dizzying number of new products, opening products that have been in beta to general availability, and talking to customers and through use cases on how to use our network to fulfill our mission of helping build a better Internet.

In Cloudflare's early days, I resisted the label of being a "security company." It seemed overly limiting. Instead, we were setting out to fix the underlying "bugs" of the Internet. The Internet was never built for what it's become. We started Cloudflare to fix that. Being more secure was table stakes, but we also wanted to make the Internet faster, more reliable, and more efficient.

But a lot of what we do is about security. Approximately half our products are security related. And that makes sense because some of the Internet's deepest flaws are that it specifically did not engineer in security from the beginning.

Security: The Internet’s Afterthought

John Graham-Cumming, Cloudflare's CTO, gives a terrific talk about how the Internet we all have come to rely on wasn’t designed to have the security we all need. In Tim Berners-Lee's original proposal for Continue reading

Introducing Project Fair Shot: Ensuring COVID-19 Vaccine Registration Sites Can Keep Up With Demand

Around the world government and medical organizations are struggling with one of the most difficult logistics challenges in history: equitably and efficiently distributing the COVID-19 vaccine. There are challenges around communicating who is eligible to be vaccinated, registering those who are eligible for appointments, ensuring they show up for their appointments, transporting the vaccine under the required handling conditions, ensuring that there are trained personnel to administer the vaccine, and then doing it all over again as most of the vaccines require two doses.

Cloudflare can't help with most of that problem, but there is one key part that we realized we could help facilitate: ensuring that registration websites don't crash under load when they first begin scheduling vaccine appointments. Project Fair Shot provides Cloudflare's new Waiting Room service for free for any government, municipality, hospital, pharmacy, or other organization responsible for distributing COVID-19 vaccines. It is open to eligible organizations around the world and will remain free until at least July 1, 2021 or longer if there is still more demand for appointments for the vaccine than there is supply.

Crashing Registration Websites

The problem of vaccine scheduling registration websites crashing under load isn't theoretical: it is happening over Continue reading