Phoenix, AZ: CloudFlare’s 64th data center
Three years and 46 data centers later our expansion returns to the United States. Phoenix, the latest addition to the CloudFlare network, is our 10th point of presence in North America, and the start of our effort to further regionalize traffic across the continent. This means faster page loads and transaction speeds for your sites and applications, as well as for the 6 million Internet users throughout the Southwestern US that use them.
Eat Surf local
The vast majority of Internet traffic in the US is exchanged in only a small handful of cities: Los Angeles, the San Francisco Bay Area, Dallas, Chicago, Miami, Ashburn (Virginia) and New York. These locations evolved into key interconnection points largely as a result of their status as population and economic centers. However, if you're one of the 236 million Americans that live outside of these metro areas, you have to hike quite a bit further to access your favorite content on the Internet.
To illustrate this, we measured the level of local interconnection between a handful of our Tier 1 Internet providers—NTT, TeliaSonera, Tata Communications and Cogent—in different metro areas. For the uninitiated, Tier 1 networks are the group of networks that Continue reading
CloudFlare is now PCI 3.1 certified
The Payment Card Industry Data Security Standard (PCI DSS) is a global financial information security standard that keeps credit card holders safe. It ensures that any company processing credit card transactions adheres to the highest technical standards.
PCI certification has several levels. Level one (the highest level) is reserved for those companies that handle the greatest numbers of credit cards. Companies at level one PCI compliance are subject to the most stringent checks.
CloudFlare’s mission leads it to provide security for some of the most important companies in the world. This is why CloudFlare chose to be audited as a level one service provider. By adhering to PCI’s rigorous financial security controls, CloudFlare ensures that security is held to the highest standard and that those controls are validated independently by a recognised body.
If you are interested in learning more, see these details about the Payment Card Industry Data Security Standard.
This year’s update from PCI 2.0 to 3.1 was long overdue. PCI DSS 2.0 was issued in October 2010, and the information security threat landscape does not stand still—especially when it comes to industries that deal with financial payments or credit cards. New attacks are almost Continue reading
Creative foot-shooting with Go RWMutex
Hi, I'm Filippo and today I managed to surprise myself! (And not in a good way.)
I'm developing a new module ("filter" as we call them) for RRDNS, CloudFlare's Go DNS server. It's a rewrite of the authoritative module, the one that adds the IP addresses to DNS answers.
It has a table of CloudFlare IPs that looks like this:
type IPMap struct {
sync.RWMutex
M map[string][]net.IP
}
It's a global filter attribute:
type V2Filter struct {
name string
IPTable *IPMap
// [...]
}
CC-BY-NC-ND image by Martin SoulStealer
The table changes often, so a background goroutine periodically reloads it from our distributed key-value store, acquires the lock (f.IPTable.Lock()), updates it and releases the lock (f.IPTable.Unlock()). This happens every 5 minutes.
Everything worked in tests, including multiple and concurrent requests.
Today we deployed to an off-production test machine and everything worked. For a few minutes. Then RRDNS stopped answering queries for the beta domains served by the new code.
What. That worked on my laptop™.
Here's the IPTable consumer function. You can probably spot the bug.
func (f *V2Filter) getCFAddr(...) (result []dns.RR) {
f. Continue readingBen Fathi: Why I Joined CloudFlare
I’m sure some of you are scratching your head right about now wondering why I would join an Internet security and optimization company. But, Ben, this is not even close to your passion: operating systems.
I had the same reaction when I first saw the CloudFlare website. I wasn’t even sure it made sense for me to go interview here. After taking a closer look, however, I realized that it would be the perfect new home for me. Take a look at this page for a brief introduction to what CloudFlare does and how we do it.
Interviewing at CloudFlare
If you know me, you know that I'm a sucker for distributed systems. I fall for a hard computer science problem every time. So, it shouldn’t be a surprise to you that CloudFlare’s John Graham-Cumming, had me at “hello” when he nonchalantly described one of the company's projects: a globally distributed key value store with sub-second consistency guarantees! Ho hum! No big deal.
As the interview process progressed, the team graciously spent several hours walking me through the architecture as well as future plans and product roadmaps. These discussions and email exchanges were frequently interrupted by my cries of protest: Continue reading
Results of experimenting with Brotli for dynamic web content
Compression is one of the most important tools CloudFlare has to accelerate website performance. Compressed content takes less time to transfer, and consequently reduces load times. On expensive mobile data plans, compression even saves money for consumers. However, compression is not free—it comes at a price. It is one of the most compute expensive operations our servers perform, and the better the compression rate we want, the more effort we have to spend.
The most popular compression format on the web is gzip. We put a great deal of effort into improving the performance of the gzip compression, so we can perform compression on the fly with fewer CPU cycles. Recently a potential replacement for gzip, called Brotli, was announced by Google. Being early adopters for many technologies, we at CloudFlare want to see for ourselves if it is as good as claimed.
This post takes a look at a bit of history behind gzip and Brotli, followed by a performance comparison.
Compression 101
Many popular lossless compression algorithms rely on LZ77 and Huffman coding, so it’s important to have a basic understanding of these two techniques before getting into gzip or Brotli.
LZ77
LZ77 is a simple technique developed Continue reading
DNSSEC is Open for Beta
Since January, CloudFlare has been running a small, private beta for DNSSEC. Starting today, the DNSSEC beta is open for everyone. To request access, email [email protected].
A Background on DNS and DNSSEC
DNS is the system that lets your browser know which web server to connect to when you request to visit a website. It’s the underlying backbone of the usable internet, and yet, is vulnerable to man in the middle attacks.
In DNS, an attacker sitting in the middle of your connection to the internet can tell your browser to connect to any web server they’d like. Browsers trust any DNS records they receive as a response to a DNS query, because DNS, invented in 1983 before the public consumption of the Internet, does not perform any authentication.
There is a solution. It’s called DNSSEC and it adds cryptographic hashes and signatures for authenticating DNS records. You can read more about DNSSEC and how it works in a previous blog post.
The DNSSEC beta is open to all websites that use CloudFlare for DNS. If you want to be a part of our beta and be one of the first CloudFlare websites with DNSSEC, email us for beta Continue reading
A Look at the New WordPress Brute Force Amplification Attack
Recently, a new brute force attack method for WordPress instances was identified by Sucuri. This latest technique allows attackers to try a large number of WordPress username and password login combinations in a single HTTP request.
The vulnerability can easily be abused by a simple script to try a significant number of username and password combinations with a relatively small number of HTTP requests. The following diagram shows a 4-fold increase in login attempts to HTTP requests, but this can trivially be expanded to a thousand logins.
This form of brute force attack is harder to detect, since you won’t necessarily see a flood of requests. Fortunately, all CloudFlare paid customers have the option to enable a Web Application Firewall ruleset to stop this new attack method.
What is XML-RPC?
To understand the vulnerability, it’s important to understand the basics of the XML remote procedure protocol (XML-RPC).
XML-RPC uses XML encoding over HTTP to provide a remote procedure call protocol. It’s commonly used to execute various functions in a WordPress instance for APIs and other automated tasks. Requests that modify, manipulate, or view data using XML-RPC require user credentials with sufficient permissions.
Here is an example that requests a list Continue reading
Single RX queue kernel bypass in Netmap for high packet rate networking
In a previous post we discussed the performance limitations of the Linux kernel network stack. We detailed the available kernel bypass techniques allowing user space programs to receive packets with high throughput. Unfortunately, none of the discussed open source solutions supported our needs. To improve the situation we decided to contribute to the Netmap project. In this blog post we'll describe our proposed changes.
CC BY-SA 2.0 image by Binary Koala
Our needs
At CloudFlare we are constantly dealing with large packet floods. Our network constantly receives a large volume of packets, often coming from many, simultaneous attacks. In fact, it is entirely possible that the server which just served you this blog post is dealing with a many-million packets per second flood right now.
Since the Linux Kernel can't really handle a large volume of packets, we need to work around it. During packet floods we offload selected network flows (belonging to a flood) to a user space application. This application filters the packets at very high speed. Most of the packets are dropped, as they belong to a flood. The small number of "valid" packets are injected back to the kernel and handled in the same way Continue reading
Doubling the speed of jpegtran with SIMD
It is no secret that at CloudFlare we put a great effort into accelerating our customers' websites. One way to do it is to reduce the size of the images on the website. This is what our Polish product is for. It takes various images and makes them smaller using open source tools, such as jpegtran, gifsicle and pngcrush.
However those tools are computationally expensive, and making them go faster, makes our servers go faster, and subsequently our customers' websites as well.
Recently, I noticed that we spent ten times as much time "polishing" jpeg images as we do when polishing pngs.
We already improved the performance of pngcrush by using our supercharged version of zlib. So it was time to look what can be done for jpegtran (part of the libjpeg distribution).
Quick profiling
To get fast results I usually use the Linux perf utility. It gives a nice, if simple, view of the hotspots in the code. I used this image for my benchmark.
perf record ./jpegtran -outfile /dev/null -progressive -optimise -copy none test.jpeg
And we get:
perf report
54.90% lt-jpegtran libjpeg.so.9.1.0 [.] encode_mcu_AC_refine
Continue reading
Manchester, UK: CloudFlare’s 63rd data center
Our new point of presence in Manchester, UK brings the CloudFlare network to 63 points of presence across 33 countries. In other words, the sun never sets across the CloudFlare network. Our data center in Manchester also admits the United Kingdom into a small club of countries with more than one CloudFlare data center, including the US, China, Japan, Australia, Germany, and France.
As of yesterday, traffic from the majority of Internet users in Northern England is now mere milliseconds away. More importantly, our Manchester and London data centers allow for redundancy and content localization within the UK for all of our customers.
In homage
The city of Manchester has made more than its fair share of technical contributions over the years. It is the city where Rolls met Royce (their first car drove off the line of their Manchester factory in 1904), and is also home to the first modern computer. The computer, nicknamed "Baby", was built at The University of Manchester using technology developed for WWII communications equipment, and ran the world's first stored program at 11am on Monday 21st June, 1948.
It is fitting, then, that in the last three years there has been a concerted effort Continue reading
Happy 5th Birthday, CloudFlare!
CloudFlare customers recorded videos to celebrate our first five years
Today is September 27, 2015. It's a rare Super Blood Moon. And it's also CloudFlare's birthday. CloudFlare launched 5 years ago today. It was a Monday. While Michelle, Lee, and I had high expectations, we would never have imagined what's happened since then.
In the last five years we've stopped 7 trillion cyber attacks, saved more than 94,116 years worth of time, and served 99.4 trillion requests — nearly half of those in the last 6 months. You can learn more from this timeline of the last five years.
Celebrating by doing the impossible
Every year we like to celebrate our birthday by giving something seemingly impossible back to our users. Two years ago we enabled on our Automatic IPv6 Gateway, allowing our users to support IPv6 without having to update their own servers. Last year we made Universal SSL support available to all our customers, even those on our free plan. And this year, we announced the expansion across Mainland China, building the first truly global performance and security platform.
Internet Summit & Party
We celebrated in San Francisco last week with CloudFlare's first Internet Summit Continue reading
Mobile Ad Networks as DDoS Vectors: A Case Study
CloudFlare servers are constantly being targeted by DDoS'es. We see everything from attempted DNS reflection attacks to L7 HTTP floods involving large botnets.
Recently an unusual flood caught our attention. A site reliability engineer on call noticed a large number of HTTP requests being issued against one of our customers.
The request
Here is one of the requests:
POST /js/404.js HTTP/1.1
Host: www.victim.com
Connection: keep-alive
Content-Length: 426
Origin: http://attacksite.com
User-Agent: Mozilla/5.0 (Linux; U; Android 4.4.4; zh-cn; MI 4LTE Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/42.0.0.0 Mobile Safari/537.36 XiaoMi/MiuiBrowser/2.1.1
Content-Type: application/x-www-form-urlencoded
Accept: */*
Referer: http://attacksite.com/html/part/86.html
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,en-US;q=0.8
id=datadatadasssssssssssssssssssssssssssssssssssssssssssassssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssadatadata
We received millions of similar requests, clearly suggesting a flood. Let's take a deeper look at this request.
First, let's note that the headers look legitimate. We often see floods issued by Python or Ruby scripts, with weird Accept-Language or User-Agent headers. But this one doesn't look like it. This request is a proper request issued by a real browser.
Next, notice the request is a POST and contains an Origin header — it was issued by an Ajax (XHR) cross Continue reading
iOS 9 — How did the launch really go?
On September 16 2015 at 10:00AM PST, Apple released their latest update to the iPhone: iOS 9. For several days after the announcement, ISPs and customers reported problems downloading iOS 9 due to overloaded servers. Obviously, the demand for iOS 9 was higher than even Apple anticipated, but how much higher? Few organizations outside of Apple have any idea what the actual iOS 9 adoption rates look like.
By analyzing the user agent strings of requests passing through the CloudFlare network, we were able to piece together a pretty good picture of iOS 9 uptake. Here’s an hour-by-hour look at requests from iOS 8 devices (blue) and iOS 9 devices (orange) for the first 24 hours after the announcement.
We started seeing small amounts of iOS 9 usage before it was officially released, followed by a spike immediately after the launch (times are shown in UTC, so the 10:00AM announcement shows up as hour 18). You can also see a second spike at 10:00 UTC when Europe started waking up.
What about iOS 9.1?
Even though the official release was for iOS 9.0, we also found beta iOS 9.1 in the wild. Curious about the comparative traffic Continue reading
Why we raised $110m from Fidelity, Google, Microsoft, Baidu and Qualcomm
The past few years have been marked by tremendous growth for CloudFlare. At the time of our last fundraising in December 2012, CloudFlare was a team of 37 operating a network in 23 cities and 15 countries—today we number over 200 with a presence in 62 cities and 33 countries. We’ve grown from delivering 85 billion page views per month for 500 thousand customers to nearly 1 trillion each month across 4 million Internet properties, all the while protecting our customers from hundreds of billions of cyber threats. The growth and resonance of our service since CloudFlare’s founding 5 years ago is beyond our wildest of expectations, but it is only in the coming years that our scale and efforts to build a better Internet will become visible.
In 2016 alone we will more than double our global presence, increase the size of our network by an order of magnitude, and with that allow millions of new businesses and online publishers to accelerate and secure their online applications and harness the growing power of the Internet economy. Our service is built on the simple premise that any individual or business should be able to quickly and easily ensure the global Continue reading
Announcing CloudFlare’s Internet Summit – And How to Get an Invitation
Five years ago next week, CloudFlare launched its service to the public. We’re celebrating our birthday in a variety of ways, including holding our first-ever Internet Summit on Thursday, September 24th. As part of the Internet Summit, we’re bringing together policymakers, business executives, cybersecurity experts, and academics from all over the world to discuss the threats and opportunities for the Internet over the next five years.
Through a series of fireside chats and panel discussions, featured speakers will discuss the top technology trends shaping business today, including cyber security, mobility, and the Internet of Things. These compelling perspectives will offer insights into the future of the global Internet and its implications on society.
Featured speakers include:
- Toomas Hendrik Ilves, The President of Estonia
- Alex Stamos, chief security officer of Facebook
- Matt Grob, chief technology officer of Qualcomm
- Andrew Ng, chief scientist at Baidu
- Nicole Wong, former US deputy CTO & legal director for products at Twitter
- Andy McAfee, MIT professor & author of "The Second Machine Age"
- Cindy Cohn, executive director of the Electronic Frontier Foundation
- Adam Langley, security engineer at Google
- David Brin, scientist, best-selling author & tech-futurist
- Rajiv Pant, former chief technology officer of The New York Times
- Continue reading
CloudFlare + WHMCS: Faster Websites For Your Customers
We’re at the cPanel Conference in Denver this week, so feel free to drop by our booth and say hello. It’s a great opportunity to connect with our partners and better understand their needs. We’re always trying to streamline our partners’ user experience, and we thought it would be a fitting time to walk through our recently updated WHMCS integration.
CloudFlare’s WHMCS 6.0 plugin lets hosting providers and registrars extend all the benefits of CloudFlare directly to their customers. You can offer your entire user base a global CDN with 62 points of presence, automatic web content optimization, basic DDoS protection, reputation-based threat protection, and much more with virtually no extra work.
These benefits are seamlessly integrated into your WHMCS client. All your customers need to do is click a button, and a new CloudFlare account will be configured for them.
While signing up for an account on www.cloudflare.com only takes a few minutes, users do need to point the relevant DNS records to CloudFlare’s nameservers. Offerring a one-click solution via our WHMCS integration is a great opportunity for hosting providers and registrars to streamline the process for their customers.
Universal SSL with WHMCS
CloudFlare’s Universal Continue reading
How We Extended CloudFlare’s Performance and Security Into Mainland China
CloudFlare launched five years ago. Within a year of our launch, the biggest surprise was the strong global demand for our service. From nearly the beginning, China was the second largest source of traffic by country to our network, behind only the United States.
In retrospect, that shouldn't have been a surprise. In 2010, the year we launched, 34% of China's population, or 450 million people, were online. Today, nearly half the country is online. To put it another way, with 700 million people online, China represents a quarter of all Internet users. If your mission is to help build a better Internet, like CloudFlare's is, then China is a country you cannot ignore.
Consequently, starting in 2011, we began to investigate how CloudFlare could bring our service to the Chinese Internet. Four years later, we're excited to announce the extension of CloudFlare's performance and security platform across mainland China. This is the story of how we did it.
The Challenges
There are three major challenges to extending a service like CloudFlare's across mainland China: technical, economic, and regulatory.
Technical
From a technical perspective, the Chinese Internet, despite its many similarities, is different than the rest of the world. Unlike Continue reading
Kuala Lumpur, Malaysia: CloudFlare’s 45th data center
Kuala Lumpur joins the CloudFlare network as our 45th global point of presence (PoP). While this latest deployment comes only a day after the announcement of our expansion in Berlin (back-to-back!), it's been a long three years since we last crossed a new border in Asia. Kuala Lumpur expands our presence in the Asia-Pacific region to nine PoPs: Kuala Lumpur, Seoul, Tokyo, Osaka, Hong Kong, Singapore, Sydney, Melbourne and Auckland.
No boomerangs allowed
One of the difficulties of delivering content locally in certain parts of Asia (and in many other parts of the world for that matter) is that certain ISPs only connect with other ISPs in the same local Internet ecosystem outside of their national borders. In the absence of domestic interconnection, a request (e.g. an e-mail or web request) from one local ISP to another "boomerangs" outside of the national border before it is ultimately delivered to another local ISP. If you live or travel in certain parts of Asia, this is one of the leading culprits for why the web feels slow. The lack of a domestic and central interconnection point also makes it challenging for networks like CloudFlare, both Continue reading
Berlin, Germany: CloudFlare’s 44th data center
Our data center in Berlin is our 3rd in Germany following Frankfurt and Düsseldorf, 14th in Europe, and 44th globally. Berlin is of considerable importance not just because it is the capital of Europe's most populous country, but also because it is the 2nd largest city in the European Union by population* trailing only London. As of this moment, CloudFlare has a point of presence (PoP) in 7 out of Europe's 10 most populous cities, and we're headed for a perfect 10-for-10.
Ich bin ein Berliner
"I am one with the people of Berlin," best expresses our sentiments following this latest launch, but is more famously a reference to U.S. President John F. Kennedy's June 26th, 1963 speech in West Berlin (and also the source of an amusing urban legend). The story goes that Kennedy should have said "Ich bin Berliner" ("I am a citizen of Berlin"), but instead remarked "Ich bin ein Berliner" which translates as "I am a jelly doughnut."
The Berliner: we treated ourselves to one a few in celebration of the launch
As it turns out, and despite decades of misinformation, Kennedy was linguistically correct. While in proper German an actual Berliner Continue reading
Weird bug of the day: Twitter in-app browser can’t visit site
We keep a close eye on tweets that mention CloudFlare because sometimes we get early warning about odd errors that we are not seeing ourselves through our monitoring systems.
Towards the end of August we saw a small number of tweets like this one:
indicating that trying to browse to a CloudFlare customer web site using the Twitter in-app browser was resulting in an error page. Which was very odd because it was clearly only happening occasionally: very occasionally.
Luckily, the person who tweeted that was in the same timezone as me and able to help debug together (thanks James White!); we discovered that the following sequence of events was necessary to reproduce the bug:
Click on a link in a tweet to a web site that is using an https URL and open in the Twitter in-app browser (not mobile Safari). This site may or may not be a CloudFlare customer.
Then click on a link on that page to a site over an http URL. This site must be on CloudFlare.
BOOM
That explained why this happened very rarely, but the question became... why did it happen at all? After some debugging it appeared to happen in Continue reading