Author Archives: Matthew Prince
Author Archives: Matthew Prince
We just turned up Salt Lake City, Utah — Cloudflare's 120th data center. Salt Lake holds a special place in Cloudflare's history. I grew up in the region and still have family there. Back in 2004, Lee Holloway and I lived just up into the mountains in Park City when we built Project Honey Pot, the open source project that inspired the original idea for Cloudflare.
Salt Lake also holds a special place in the history of the Internet. The University of Utah, based there, was one of the original four Arpanet locations (along with UCLA, UC Santa Barbara, and the Stanford Research Institute). The school also educated the founders of great technology companies like Silicon Graphics, Adobe, Atari, Netscape, and Pixar. Many were graduates of the computer graphics department lead by Professors Ivan Sutherland and David Evans.
In 1980, when I was seven years old, my grandmother, who lived a few blocks from the University, gave me an Apple II+ for Christmas. I took to it like a duck to water. My mom enrolled in a continuing education computer course at the University of Utah teaching BASIC programming. I went with her to the classes. Unbeknownst to the Continue reading
Cloudflare turns seven years old today. We launched on September 27, 2010.
It was only a few days after our launch that we got our first request to support video streaming. Yet, until today, we'd avoided it.
Why? Simply put: the video streaming market is screwed up. While there's a lot of money spent on video, there are only really about 1,000 customers that do any meaningful level of streaming.
This is in large part because it's technically far too complicated. If you want to move beyond just uploading your videos to a consumer service like YouTube, then you have to use at least three different services. You need someone to encode your video into a streamable format, you need someone else to act as the content delivery network delivering the bytes, and you need someone else still to provide the player code that runs on the client device. Further, since video encoding standards keep evolving and vary across generations of devices, it becomes challenging to ensure a consistently high quality experience for all visitors.
And if that sounds like a technical mess, the business side is even worse. Encoding companies charge based on CPU usage, which is driven by Continue reading
This is the week of Cloudflare's seventh birthday. It's become a tradition for us to announce a series of products each day of this week and bring major new benefits to our customers. We're beginning with one I'm especially proud of: Unmetered Mitigation.
Cloudflare runs one of the largest networks in the world. One of our key services is DDoS mitigation and we deflect a new DDoS attack aimed at our customers every three minutes. We do this with over 15 terabits per second of DDoS mitigation capacity. That's more than the publicly announced capacity of every other DDoS mitigation service we're aware of combined. And we're continuing to invest in our network to expand capacity at an accelerating rate.
Virtually every Cloudflare competitor will send you a bigger bill if you are unlucky enough to get targeted by an attack. We've seen examples of small businesses that survive massive attacks to then be crippled by the bills other DDoS mitigation vendors sent them. From the beginning of Cloudflare's history, it never felt right that you should have to pay more if you came under an attack. That feels barely a Continue reading
Jango Fett by Brickset (Flickr)
When Blackbird Tech, a notorious patent troll, sued us earlier this year for patent infringement, we discovered quickly that the folks at Blackbird were engaged in what appeared to be the broad and unsubstantiated assertion of patents -- filing about 115 lawsuits in less than 3 years, and have not yet won a single one of those cases on the merits in court. Cloudflare felt an appropriate response would be to review all of Blackbird Tech’s patents, not just the one it asserted against Cloudflare, to determine if they are invalid or should be limited in scope. We enlisted your help in this endeavor by placing a $50,000 bounty on prior art that proves the Blackbird Tech patents are invalid or overbroad, an effort we dubbed Project Jengo.
Since its inception, Project Jengo has doubled in size and provided us with a good amount of high quality prior art submissions. We have received more than 230 submissions so far, and have only just begun to scratch the surface. We have already come across a number of standouts that appear to be strong contenders for invalidating many of the Blackbird Tech patents. This means it is Continue reading
Earlier today, Cloudflare terminated the account of the Daily Stormer. We've stopped proxying their traffic and stopped answering DNS requests for their sites. We've taken measures to ensure that they cannot sign up for Cloudflare's services ever again.
Our terms of service reserve the right for us to terminate users of our network at our sole discretion. The tipping point for us making this decision was that the team behind Daily Stormer made the claim that we were secretly supporters of their ideology.
Our team has been thorough and have had thoughtful discussions for years about what the right policy was on censoring. Like a lot of people, we’ve felt angry at these hateful people for a long time but we have followed the law and remained content neutral as a network. We could not remain neutral after these claims of secret support by Cloudflare.
Now, having made that decision, let me explain why it's so dangerous.
There are a number of different organizations that work in concert to bring you the Internet. They include:
When we started Cloudflare we had no idea if anyone would validate our core idea. Our idea was what that everyone should have the ability to be as fast and secure as the Internet giants like Google, Facebook, and Microsoft. Six years later, it's incredible how far that core idea has taken us.
Today, Cloudflare runs one of the largest global networks. We have data centers in 115 cities around the world and continue to expand. We've built a core service that delivers performance, security, availability, and insight to more than 6 million users.
From the beginning, our goal has been to democratize the Internet. Today we're taking another step toward that goal with the launch of the Cloudflare Apps Platform and the Cloudflare Developer Fund. To understand that, you have to understand where we started.
When we started Cloudflare we needed two things: a collection of users for the service, and finances to help us fund our development. In both cases, people were taking a risk on Cloudflare. Our first users came from Project Honey Pot, which Lee Holloway and I created back in 2004. Members Continue reading
Jengo Fett by Brickset (Flickr)
We knew the case against patent trolls was the right one, but we have been overwhelmed by the response to our blog posts on patent trolls and our program for finding prior art on the patents held by Blackbird Tech, which we’ve dubbed Project Jengo. As we discuss in this post, your comments and contributions have allowed us to expand and intensify our efforts to challenge the growing threat that patent trolls pose to innovative tech companies.
In a little over a week since we started the program, we’ve received 141 separate prior art submissions. But we know there’s an opportunity to find a lot more.
We’ve been impressed with the exceptionally high quality of the submissions. The Cloudflare community of users and readers of our blog are an accomplished bunch, so we have a number of searches that were done by expert engineers and programmers. In one case that stood out to us, someone wrote in about a project they personally had worked on as an engineer back in 1993, which they are convinced is conclusive prior art Continue reading
On March 20th, Cloudflare received our first patent infringement claim: Blackbird Tech LLC v. Cloudflare, Inc. Today we’re filing our Answer to that claim in a federal court in Delaware. We have very strong arguments we will present in the litigation, mostly because the patent asserted against us does not have anything to do with our technology.
The infringement claim is not a close one. The asserted patent, US 6453335 (‘335 patent) was filed in 1998, and describes a system for monitoring an existing data channel and inserting error pages when transmission rates fall below a certain level. Nothing from 1998—including the ’335 patent—comes close to describing our state-of-the-art product that is provisioned via DNS, speeds up internet content delivery, and protects against malicious attackers. Our technology is innovative and different, and Cloudflare’s technology has about 150 patents issued or in process.
We also expect to show that the patent itself is invalid. For example, if the ’335 patent is read broadly enough to cover our system (which shouldn’t happen), it would also cover any system where electronic communications are examined and redacted or modified. But this is not new. Filtering products performing similar functions were around long before Continue reading
Jengo Fett by Brickset (Flickr)
As readers of this blog likely know, especially if you read this post, Cloudflare has been sued by a dangerous new breed of patent troll, Blackbird Technologies, asserting a very old and very vague patent. And we know we are not alone in being frustrated about the way that such patent trolls inhibit the growth of innovative companies. Cloudflare is asking for your help in this effort, and we’re putting our money where our mouth is.
Patent trolls take advantage of a system they assume is tilted in their favor, where they can take vague technology patents issued years ago and apply them as broadly as imaginable to the latest technology. And they do this without the limitations of having to show the original patent holder would have actually exercised the patent, because most of them don’t, at all. Patent trolls think they can sit back and pick off settlements from companies because their lawsuits are a nuisance and the costs of defending those suits are considerable.
Changing this dynamic and leveling the playing field is going to require an entirely new approach. Fighting such strong, though perverse, economic incentives is going Continue reading
Last Thursday, ProPublica published an article critiquing our handling of some abuse reports that we receive. Feedback from the article caused us to reevaluate how we handle abuse reports. As a result, we've decided to update our abuse reporting system to allow individuals reporting threats and child sexual abuse material to do so anonymously. We are rolling this change out and expect it to be available by the end of the week.
I appreciate the feedback we received. How we handle abuse reports has evolved over the last six and a half years of Cloudflare's history. I wanted to take this opportunity to walk through some of the rationale that got us to this point and caused us to have a blindspot to the case that was highlighted in the article.
Cloudflare is not a hosting provider. We do not store the definitive copy of any of the content that someone may want to file an abuse claim about. If we terminate a customer it doesn’t make the content go away. Instead, we are more akin to a specialized network. One of the functions of the network that we provide is to add security to the content Continue reading
Last Thursday we released details on a bug in Cloudflare's parser impacting our customers. It was an extremely serious bug that caused data flowing through Cloudflare's network to be leaked onto the Internet. We fully patched the bug within hours of being notified. However, given the scale of Cloudflare, the impact was potentially massive.
The bug has been dubbed “Cloudbleed.” Because of its potential impact, the bug has been written about extensively and generated a lot of uncertainty. The burden of that uncertainty has been felt by our partners, customers, our customers’ customers. The question we’ve been asked the most often is: what risk does Cloudbleed pose to me?
We've spent the last twelve days using log data on the actual requests we’ve seen across our network to get a better grip on what the impact was and, in turn, provide an estimate of the risk to our customers. This post outlines our initial findings.
The summary is that, while the bug was very bad and had the potential to be much worse, based on our analysis so far: 1) we have found no evidence based on our logs that the bug was maliciously exploited before it was patched; Continue reading
In 2017, we've predicted that more than half of the traffic to Cloudflare's network will come from mobile devices. Even if they are formatted to be displayed on a small screen, the mobile web is built on traditional web protocols and technologies that were designed for desktop CPUs, network connections, and displays. As a result, browsing the mobile web feels sluggish compared with using native mobile apps.
In October 2015, the team at Google announced Accelerated Mobile Pages (AMP), a new, open technology to make the mobile web as fast as native apps. Since then, a large number of publishers have adopted AMP. Today, 600 million pages across 700,000 different domains are available in the AMP format.
The majority of traffic to this AMP content comes from people running searches on Google.com. If a visitor finds content through some source other than a Google search, even if the content can be served from AMP, it typically won't be. As a result, the mobile web continues to be slower than it needs to be.
Cloudflare's Accelerated Mobile Links helps solve this problem, making content, regardless of how it's discovered, app-quick. Once enabled, Accelerated Mobile Continue reading
In 2011 we launched the Cloudflare Apps platform in an article that described Cloudflare as “not ... the sexiest business in the world.” Sexy or not, Cloudflare has since grown from the 3.5 billion pageviews a month we were doing then to over 1.3 trillion per month today. Along the way, we’ve powered more than a million app installations onto our customer’s websites.
For the last 6 years Cloudflare has been focused on building one of the world’s largest networks. The importance of that work has not left as much time as we would have liked to improve our app platform. With just 21 apps, we knew we were not delivering all that our marketplace could offer.
About six months ago, we were introduced to the team at Eager. Eager was building its own app store for installation onto any website. They impressed us with their ability to enable even the most non-technical website owner to install powerful tools to improve their sites through a slick interface. Eager’s platform included the features we wanted in our marketplace, like the ability to preview an app on a user's site before installing it. Even better, Eager had a powerful app Continue reading
The last few weeks have seen several high-profile outages in legacy DNS and DDoS-mitigation services due to large scale attacks. Cloudflare's customers have, understandably, asked how we are positioned to handle similar attacks.
While there are limits to any service, including Cloudflare, we are well architected to withstand these recent attacks and continue to scale to stop the larger attacks that will inevitably come. We are, multiple times per day, mitigating the very botnets that have been in the news. Based on the attack data that has been released publicly, and what has been shared with us privately, we have been successfully mitigating attacks of a similar scale and type without customer outages.
I thought it was a good time to talk about how Cloudflare's architecture is different than most legacy DNS and DDoS-mitigation services and how that's helped us keep our customers online in the face of these extremely high volume attacks.
Before delving into our architecture, it's worth taking a second to think about another analogous technology problem that is better understood: scaling databases. From the mid-1980s, when relational databases started taking off, through the early 2000s the way companies thought of scaling Continue reading
The mission of the United State's Government's Consumer Product Safety Commission (CPSC) is to protect consumers from injury by products. It's ironic then that the CPSC is playing an unwitting role in most of the largest DDoS attacks seen on the Internet. To understand how, you need to understand a bit about how you launch a high volume DDoS.
Logo of the Consumer Product Safety Commission
DDoS attacks are inherently about an attacker sending more traffic to a victim than the victim can handle. The challenge for an attacker is to find a way to generate a large amount of traffic. Launching a DDoS attack is a criminal act, so an attacker can't simply go sign up for large transit contracts. Instead, attackers find ways to leverage other people's resources.
One of the most effective strategies is known as an amplification attack. In these attacks, an attacker can amplify their resources by reflecting them off other resources online that magnify the level of traffic. The most popular amplification vector is known as DNS reflection.
Beginning in March 2016, we began hearing reports of a gang of cybercriminals once again calling themselves the Armada Collective. The calling card of the gang was an extortion email sent to a wide variety of online businesses threatening to launch DDoS attacks if they weren't paid in Bitcoin.
From The Wizard of Oz (1939)
We heard from more than 100 existing and prospective CloudFlare customers who had received the Armada Collective's emailed threats. We've also compared notes with other DDoS mitigation vendors with customers that had received similar threats.
Our conclusion was a bit of a surprise: we've been unable to find a single incident where the current incarnation of the Armada Collective has actually launched a DDoS attack. In fact, because the extortion emails reuse Bitcoin addresses, there's no way the Armada Collective can tell who has paid and who has not. In spite of that, the cybercrooks have collected hundreds of thousands of dollars in extortion payments.
The extortion emails sent by the Armada Collective have been remarkably consistent over the last two months. Here's an example:
The Tor Project makes a browser that allows anyone to surf the Internet anonymously. Tor stands for "the Onion router" and that describes how the service works. Traffic is routed through a number of relays run across the Internet where each relay only knows the next hop (because each hop is enclosed in a cryptographic envelope), not the ultimate destination, until the traffic gets to the final exit node which connects to the website — like peeling the layers of an onion.
Think of it like a black box: traffic goes into the box, is bounced around between a random set of relays, and ultimately comes out to connect to the requested site. Anonymity is assured because anyone monitoring the network would have a difficult time tying the individuals making the requests going into the black box with the requests coming out.
Anonymity online is important for a number of reasons we at CloudFlare believe in. For instance, Tor is instrumental in ensuring that individuals living in repressive regimes can access information that may otherwise be blocked or illegal. We this is so important that we offer Continue reading
At CloudFlare, we’ve constructed one of the world’s largest networks purpose-built to protect our customers from a wide range of attacks. We’re so good at it that attackers increasingly look for ways to go around us, rather than go through us. One of the biggest risks for high-profile customers has been having their domain stolen at the registrar.
In 2013, we became intimately familiar with this problem when domains for the New York Times were hijacked and the newspaper’s CTO reached out to us to help get it back. We were able to assist, but the newspaper had its web and email traffic rerouted for hours.
Since the New York Times domain hijack, a number of other sites have had their domains stolen. We ourselves have seen multiple attempts to take control of CloudFlare’s registrar account. Thankfully, none have been successful—but some have gotten closer than we were comfortable with. Given the risk, we began looking for a registrar with security protocols that we could trust.
In the early days of the Internet, domain registration was free. As the Internet began to take off, demand for domain registrations exploded. In 1993, unable to Continue reading
After December 31, 2015, SSL certificates that use the SHA-1 hash algorithm for their signature will be declared technology non grata on the modern Internet. Google's Chrome browser has already begun displaying a warning for SHA-1 based certs that expire after 2015. Other browsers are mirroring Google and, over the course of 2016, will begin issuing warnings and eventually completely distrust connections to sites using SHA-1 signed certs. And, starting January 1, 2016, you will no longer be able to get a new SHA-1 certificate from most certificate authorities.
For the most part, that's a good thing. Prohibitively difficult to forge certificate signatures are part of what keeps encryption systems secure. As computers get faster, the risk that, for any given hashing algorithm, you can forge a certificate with the same signature increases. If an attacker can forge a certificate then they could potentially impersonate the identity of a real site and intercept its encrypted traffic or masquerade as that site.
This isn't the first time we've been through this exercise. The original hashing algorithm used for most certificate signatures in the early days of the web was MD5. In 2008, researchers demonstrated they were able to Continue reading
Today is September 27, 2015. It's a rare Super Blood Moon. And it's also CloudFlare's birthday. CloudFlare launched 5 years ago today. It was a Monday. While Michelle, Lee, and I had high expectations, we would never have imagined what's happened since then.
In the last five years we've stopped 7 trillion cyber attacks, saved more than 94,116 years worth of time, and served 99.4 trillion requests — nearly half of those in the last 6 months. You can learn more from this timeline of the last five years.
Every year we like to celebrate our birthday by giving something seemingly impossible back to our users. Two years ago we enabled on our Automatic IPv6 Gateway, allowing our users to support IPv6 without having to update their own servers. Last year we made Universal SSL support available to all our customers, even those on our free plan. And this year, we announced the expansion across Mainland China, building the first truly global performance and security platform.
We celebrated in San Francisco last week with CloudFlare's first Internet Summit Continue reading