Author Archives: Matthew Prince
Author Archives: Matthew Prince
In 2017, we've predicted that more than half of the traffic to Cloudflare's network will come from mobile devices. Even if they are formatted to be displayed on a small screen, the mobile web is built on traditional web protocols and technologies that were designed for desktop CPUs, network connections, and displays. As a result, browsing the mobile web feels sluggish compared with using native mobile apps.
In October 2015, the team at Google announced Accelerated Mobile Pages (AMP), a new, open technology to make the mobile web as fast as native apps. Since then, a large number of publishers have adopted AMP. Today, 600 million pages across 700,000 different domains are available in the AMP format.
The majority of traffic to this AMP content comes from people running searches on Google.com. If a visitor finds content through some source other than a Google search, even if the content can be served from AMP, it typically won't be. As a result, the mobile web continues to be slower than it needs to be.
Cloudflare's Accelerated Mobile Links helps solve this problem, making content, regardless of how it's discovered, app-quick. Once enabled, Accelerated Mobile Continue reading
In 2011 we launched the Cloudflare Apps platform in an article that described Cloudflare as “not ... the sexiest business in the world.” Sexy or not, Cloudflare has since grown from the 3.5 billion pageviews a month we were doing then to over 1.3 trillion per month today. Along the way, we’ve powered more than a million app installations onto our customer’s websites.
For the last 6 years Cloudflare has been focused on building one of the world’s largest networks. The importance of that work has not left as much time as we would have liked to improve our app platform. With just 21 apps, we knew we were not delivering all that our marketplace could offer.
About six months ago, we were introduced to the team at Eager. Eager was building its own app store for installation onto any website. They impressed us with their ability to enable even the most non-technical website owner to install powerful tools to improve their sites through a slick interface. Eager’s platform included the features we wanted in our marketplace, like the ability to preview an app on a user's site before installing it. Even better, Eager had a powerful app Continue reading
The last few weeks have seen several high-profile outages in legacy DNS and DDoS-mitigation services due to large scale attacks. Cloudflare's customers have, understandably, asked how we are positioned to handle similar attacks.
While there are limits to any service, including Cloudflare, we are well architected to withstand these recent attacks and continue to scale to stop the larger attacks that will inevitably come. We are, multiple times per day, mitigating the very botnets that have been in the news. Based on the attack data that has been released publicly, and what has been shared with us privately, we have been successfully mitigating attacks of a similar scale and type without customer outages.
I thought it was a good time to talk about how Cloudflare's architecture is different than most legacy DNS and DDoS-mitigation services and how that's helped us keep our customers online in the face of these extremely high volume attacks.
Before delving into our architecture, it's worth taking a second to think about another analogous technology problem that is better understood: scaling databases. From the mid-1980s, when relational databases started taking off, through the early 2000s the way companies thought of scaling Continue reading
The mission of the United State's Government's Consumer Product Safety Commission (CPSC) is to protect consumers from injury by products. It's ironic then that the CPSC is playing an unwitting role in most of the largest DDoS attacks seen on the Internet. To understand how, you need to understand a bit about how you launch a high volume DDoS.
Logo of the Consumer Product Safety Commission
DDoS attacks are inherently about an attacker sending more traffic to a victim than the victim can handle. The challenge for an attacker is to find a way to generate a large amount of traffic. Launching a DDoS attack is a criminal act, so an attacker can't simply go sign up for large transit contracts. Instead, attackers find ways to leverage other people's resources.
One of the most effective strategies is known as an amplification attack. In these attacks, an attacker can amplify their resources by reflecting them off other resources online that magnify the level of traffic. The most popular amplification vector is known as DNS reflection.
We've written about DNS reflection attacks in detail before. The basics are that an attacker generates DNS requests from a network that allows Continue reading
Beginning in March 2016, we began hearing reports of a gang of cybercriminals once again calling themselves the Armada Collective. The calling card of the gang was an extortion email sent to a wide variety of online businesses threatening to launch DDoS attacks if they weren't paid in Bitcoin.
From The Wizard of Oz (1939)
We heard from more than 100 existing and prospective CloudFlare customers who had received the Armada Collective's emailed threats. We've also compared notes with other DDoS mitigation vendors with customers that had received similar threats.
Our conclusion was a bit of a surprise: we've been unable to find a single incident where the current incarnation of the Armada Collective has actually launched a DDoS attack. In fact, because the extortion emails reuse Bitcoin addresses, there's no way the Armada Collective can tell who has paid and who has not. In spite of that, the cybercrooks have collected hundreds of thousands of dollars in extortion payments.
The extortion emails sent by the Armada Collective have been remarkably consistent over the last two months. Here's an example:
The Tor Project makes a browser that allows anyone to surf the Internet anonymously. Tor stands for "the Onion router" and that describes how the service works. Traffic is routed through a number of relays run across the Internet where each relay only knows the next hop (because each hop is enclosed in a cryptographic envelope), not the ultimate destination, until the traffic gets to the final exit node which connects to the website — like peeling the layers of an onion.
Think of it like a black box: traffic goes into the box, is bounced around between a random set of relays, and ultimately comes out to connect to the requested site. Anonymity is assured because anyone monitoring the network would have a difficult time tying the individuals making the requests going into the black box with the requests coming out.
Anonymity online is important for a number of reasons we at CloudFlare believe in. For instance, Tor is instrumental in ensuring that individuals living in repressive regimes can access information that may otherwise be blocked or illegal. We this is so important that we offer Continue reading
At CloudFlare, we’ve constructed one of the world’s largest networks purpose-built to protect our customers from a wide range of attacks. We’re so good at it that attackers increasingly look for ways to go around us, rather than go through us. One of the biggest risks for high-profile customers has been having their domain stolen at the registrar.
In 2013, we became intimately familiar with this problem when domains for the New York Times were hijacked and the newspaper’s CTO reached out to us to help get it back. We were able to assist, but the newspaper had its web and email traffic rerouted for hours.
Since the New York Times domain hijack, a number of other sites have had their domains stolen. We ourselves have seen multiple attempts to take control of CloudFlare’s registrar account. Thankfully, none have been successful—but some have gotten closer than we were comfortable with. Given the risk, we began looking for a registrar with security protocols that we could trust.
In the early days of the Internet, domain registration was free. As the Internet began to take off, demand for domain registrations exploded. In 1993, unable to Continue reading
After December 31, 2015, SSL certificates that use the SHA-1 hash algorithm for their signature will be declared technology non grata on the modern Internet. Google's Chrome browser has already begun displaying a warning for SHA-1 based certs that expire after 2015. Other browsers are mirroring Google and, over the course of 2016, will begin issuing warnings and eventually completely distrust connections to sites using SHA-1 signed certs. And, starting January 1, 2016, you will no longer be able to get a new SHA-1 certificate from most certificate authorities.
For the most part, that's a good thing. Prohibitively difficult to forge certificate signatures are part of what keeps encryption systems secure. As computers get faster, the risk that, for any given hashing algorithm, you can forge a certificate with the same signature increases. If an attacker can forge a certificate then they could potentially impersonate the identity of a real site and intercept its encrypted traffic or masquerade as that site.
This isn't the first time we've been through this exercise. The original hashing algorithm used for most certificate signatures in the early days of the web was MD5. In 2008, researchers demonstrated they were able to Continue reading
Today is September 27, 2015. It's a rare Super Blood Moon. And it's also CloudFlare's birthday. CloudFlare launched 5 years ago today. It was a Monday. While Michelle, Lee, and I had high expectations, we would never have imagined what's happened since then.
In the last five years we've stopped 7 trillion cyber attacks, saved more than 94,116 years worth of time, and served 99.4 trillion requests — nearly half of those in the last 6 months. You can learn more from this timeline of the last five years.
Every year we like to celebrate our birthday by giving something seemingly impossible back to our users. Two years ago we enabled on our Automatic IPv6 Gateway, allowing our users to support IPv6 without having to update their own servers. Last year we made Universal SSL support available to all our customers, even those on our free plan. And this year, we announced the expansion across Mainland China, building the first truly global performance and security platform.
We celebrated in San Francisco last week with CloudFlare's first Internet Summit Continue reading
CloudFlare launched five years ago. Within a year of our launch, the biggest surprise was the strong global demand for our service. From nearly the beginning, China was the second largest source of traffic by country to our network, behind only the United States.
In retrospect, that shouldn't have been a surprise. In 2010, the year we launched, 34% of China's population, or 450 million people, were online. Today, nearly half the country is online. To put it another way, with 700 million people online, China represents a quarter of all Internet users. If your mission is to help build a better Internet, like CloudFlare's is, then China is a country you cannot ignore.
Consequently, starting in 2011, we began to investigate how CloudFlare could bring our service to the Chinese Internet. Four years later, we're excited to announce the extension of CloudFlare's performance and security platform across mainland China. This is the story of how we did it.
The Challenges
There are three major challenges to extending a service like CloudFlare's across mainland China: technical, economic, and regulatory.
Technical
From a technical perspective, the Chinese Internet, despite its many similarities, is different than the rest of the world. Unlike Continue reading
CloudFlare turns 5 years old this September. It's been an amazing ride since our launch. Before we launched at TechCrunch Disrupt on September 27, 2010, we'd signed up about 1,000 beta customers. It took us nine months to get those first customers. (By comparison, today we typically sign up 1,000 customers every 3 hours.)
Those first beta customers were instrumental. They put up with us when we were had only one data center (in Chicago). They put up with us as we brought traffic online in our next facilities in Ashburn, Virginia and San Jose, California — and had the routing challenges that came along with running a distributed network for the first time. They sent us bug reports, provided us feature requests, and were instrumental to building the foundation that grew into what is CloudFlare today.
Archival Footage
When we launched, we wanted to feature their stories and experience about CloudFlare so we had them submit their stories by video. Here's the video we included as part of our launch presentation.
I'm proud of the fact that more than 80% of those original 1,000 customers are still using CloudFlare five years later.
Send Us Your Stories
As we Continue reading
When we started CloudFlare, we thought we were building a service to make websites faster and more secure, and we wanted to make the service as easy and accessible as possible. As a result, we built the CloudFlare interface to put basic functions front and center and designed it to look more like a consumer app than the UI for the powerful network it controlled.
Over time, we realized there was a lot more to CloudFlare. In 2011, we added the concept of Apps, and a myriad of additional performance and security features from Rocket Loader to Railgun were added too. All these additional settings got buried under a lowly gear menu next to each site in a customer's account.
While still easier to navigate than the average enterprise app, using our UI could be a frustrating experience. For instance, imagine you wanted to turn on Rocket Loader for multiple sites. You'd have to go to My Websites, click the gear menu next to one of your domains, navigate to CloudFlare Settings, select the Performance Settings tab, scroll to Rocket Loader, then toggle it on. Then you had to go back to My Websites and repeat the process again for Continue reading
Today the United States Federal Communications Commission (FCC) voted to extend the rules that previously regulated the telephone industry to now regulate Internet Service Providers (ISPs). The Commission did this in order to preserve the principle of network neutrality. Broadly stated, this principle is that networks should not discriminate against content that passes through them.
At CloudFlare, we are strong proponents of network neutrality. My co-founder, Michelle Zatlyn, sat on the FCC's Open Internet Advisory Committee. The work of that committee played a role in guiding today's vote. So there is a large part of us that is celebrating today.
At the same time, I have deep concerns that proponents of a free and open Internet may look back on today not as a great victory, but as the first step in what may turn out to be a devastating loss. The Internet has largely been governed from the bottom up by technologists seeking rough consensus and running code. Today's action by the FCC may mark the beginning of a new era where the Internet is regulated by lawyers from the top down. As a technologist and recovering lawyer, that worries me.
If you think Continue reading
I'm excited to announce that today kicks off SSL Week at CloudFlare. Over the course of this week, we'll make a series of announcements on what we're doing to improve encryption on the Internet.
Inherently, for encryption to be the most effective, it has to meet three criteria: 1) it needs to be easy and inexpensive to use; 2) it needs to be fast so it doesn't tax performance; and 3) it needs to be up to date and ahead of the latest vulnerabilities.
Throughout CloudFlare's history, these priorities have guided our approach to encryption. Last September, we announced Universal SSL and brought world class encryption to every CloudFlare customer, even those on our Free service plan. While that effort doubled the size of the encrypted web, our work is far from done. This week we're announcing a series of initiatives that further our efforts to ensure we provide the easiest, fastest, and most secure encryption.
While Universal SSL made it easy to ensure that the connection from a device to CloudFlare was secure, this week we're going to begin the process of making it easy (and free) to ensure the connection from CloudFlare back to Continue reading
For the last week we've been tracking rumors about a new vulnerability in SSL. This specific vulnerability, which was just announced, targets SSLv3. The vulnerability allows an attacker to add padding to a request in order to then calculate the plaintext of encryption using the SSLv3 protocol. Effectively, this allows an attacker to compromise the encryption when using the SSLv3 protocol. Full details have been published by Google in a paper which dubs the bug POODLE (PDF).
Generally, modern browsers will default to a more modern encryption protocol (e.g., TLSv1.2). However, it's possible for an attacker to simulate conditions in many browsers that will cause them to fall back to SSLv3. The risk from this vulnerability is that if an attacker could force a downgrade to SSLv3 then any traffic exchanged over an encrypted connection using that protocol could be intercepted and read.
In response, CloudFlare has disabled SSLv3 across our network by default for all customers. This will have an impact on some older browsers, resulting in an SSL connection error. The biggest impact is Internet Explorer 6 running on Windows XP or older. To quantify this, we've been tracking SSLv3 usage.
It turns out it takes a while to deploy SSL certificates for 2 million websites. :-) Even longer when you get a flood of new sign ups. While we'd hoped to have the deployment complete within 24 hours of the announcement, it now looks like it's going to take a bit longer. We now expect that the full deployment will be complete about 48 hours from now (0700 UTC). Beyond that, nothing about the plan for Universal SSL has changed and hundreds of thousands of sites are already active.
In order to get through the highest priority sites first, we've prioritized provisioning the sites with the most traffic.
While you wait for your site to get provisioned, you may see a certificate mismatch error if you try and visit it over HTTPS. (Rest assured, there are no errors if you visit over HTTP.) The errors over HTTPS are expected and normal during the provisioning process. Examples of what these error looks like in various browsers )Chrome, Safari, Firefox, and Internet Explorer) are below:
To give you a sense of our progress provisioning Universal SSL for your sites, we've updated the alert that Continue reading
The team at CloudFlare is excited to announce the release of Universal SSL™. Beginning today, we will support SSL connections to every CloudFlare customer, including the 2 million sites that have signed up for the free version of our service.
This morning we began rolling out the Universal SSL across all our current customers. We expect this process to be complete for all current customers before the end of the day. Yesterday, there were about 2 million sites active on the Internet that supported encrypted connections. By the end of the day today, we'll have doubled that.
For new customers who sign up for CloudFlare's free plan, after we get through provisioning existing customers, it will take up to 24 hours to activate Universal SSL. As always, SSL for paid plans will be provisioned instantly upon signup.
For all customers, we will now automatically provision a SSL certificate on CloudFlare's network that will accept HTTPS connections for a customer's domain and subdomains. Those certificates include an entry for the root domain (e.g., example.com) as well as a wildcard entry for all first-level subdomains (e.g., www.example.com, blog.example.com, etc. Continue reading
I wanted to write one more thing about Keyless SSL, our announcement from last week, before attention shifts to what we'll be announcing on Monday. Keyless allows us to provide CloudFlare's service without having private SSL keys stored locally on our edge servers. The news last week focused on how this could allow very large customers, like major financial institutions, to use CloudFlare without trusting us with their private keys.
But there's another use that will benefit the entire CloudFlare userbase, not just our largest enterprise customers, and it's this: Keyless SSL is a key part of our strategy to continue to expand CloudFlare's global network.
CloudFlare's network today consists of 28 edge data centers that span much of the globe. We have technical and security requirements for these facilities in order to ensure that the equipment they house remains secure. Generally, we're in Tier III or IV data center facilities with the highest level of security. In our San Jose facility, for instance, you have to pass through 5 biometric scans, in addition to multiple 24x7 manned guard check points, before you can get to the electronically locked cabinets housing our servers.
There Continue reading
Since CloudFlare launched to the public four years ago today, we've always considered September 27th our birthday. We like to celebrate by doing something nice for our team and also for our customers. Two years ago, for example, we brought a cake into the office and then enabled free IPv6 support for all our customers.
Saturday is our birthday this year, so we decided to celebrate it a few days later when we'd all be back in the office on Monday, September 29th. That actually corresponds to the day we presented at the finals of the TechCrunch Disrupt startup contest where we launched. We ended up coming in second. Mike Arrington, the founder of TechCrunch, said we were basically "muffler repair for the Internet."
Looking back, that's actually not a bad description. At core, CloudFlare's mission is to help build a better Internet by fixing its biggest problems -- its metaphorical rusty mufflers. This year, we thought it would be great to repair a big, ugly muffler that should have been fixed a long time ago.
This Monday, we'll bring a cake into the office. (It'll have to be a lot bigger as our team has grown substantially.) Continue reading
CloudFlare is an engineering-driven company. This is a story we're proud of because it embodies the essence of who we are: when faced with a problem, we found a novel solution. Technical details to follow but, until then, welcome to the no hardware world.
The story begins on a Saturday morning, in the Fall of 2012, almost exactly two years ago. I got a call on my cell phone that woke me. It was a man who introduced himself as the Chief Information Security Officer (CISO) at one of the world's largest banks.
"I got your number from a reporter," he said. "We have an incident. Could you and some of your team be in New York Monday morning? We'd value your advice." We were a small startup. Of course we were going to drop everything and fly across the country to see if we could help.
I called John Roberts and Sri Rao, two members of CloudFlare's team. John had an air of calm about him and owned more khaki pants than any of the rest of us. Sri was a senior member of our technical operations team and could, already at that point, Continue reading