SHA-1 Deprecation: No Browser Left Behind
After December 31, 2015, SSL certificates that use the SHA-1 hash algorithm for their signature will be declared technology non grata on the modern Internet. Google's Chrome browser has already begun displaying a warning for SHA-1 based certs that expire after 2015. Other browsers are mirroring Google and, over the course of 2016, will begin issuing warnings and eventually completely distrust connections to sites using SHA-1 signed certs. And, starting January 1, 2016, you will no longer be able to get a new SHA-1 certificate from most certificate authorities.
For the most part, that's a good thing. Prohibitively difficult to forge certificate signatures are part of what keeps encryption systems secure. As computers get faster, the risk that, for any given hashing algorithm, you can forge a certificate with the same signature increases. If an attacker can forge a certificate then they could potentially impersonate the identity of a real site and intercept its encrypted traffic or masquerade as that site.
Deprecating Old Standards
This isn't the first time we've been through this exercise. The original hashing algorithm used for most certificate signatures in the early days of the web was MD5. In 2008, researchers demonstrated they were able to Continue reading
What is Internet Goverance and Why Does it Matter?
Last month, CloudFlare participated the tenth annual Internet Governance Forum (IGF) in Joao Pessoa, Brazil. Since it was launched at the United Nations’ World Summit on the Information Society (WSIS) in 2005, the IGF has provided valuable opportunities for thousands of representatives of non-profit groups, businesses, governments, and others to debate decisions that will affect the future of the Internet. While the Forum does not negotiate any treaties or other agreements, what participants learn there can influence corporate strategies, standards proposals, and national government policies. Even more importantly, discussions in the hallways (or in the bar or on the beach) can lead to new projects, new thinking, and new collaborations.
The range of issues and the diversity of speakers on panels and at the podium was even greater this year than at previous IGFs. Issues ranged from the need for strong encryption to whether net neutrality regulations are needed—from countering the abuse of women online to how to foster deployment of IPv6 and Internet Exchange Points. You can watch all 167 IGF sessions, which were webcast and archived. I represent CloudFlare as a member of the Multistakeholder Advisory Group (MAG), which organizes the IGF program. Together with the other MAG Continue reading
Tools for debugging, testing and using HTTP/2
With CloudFlare's release of HTTP/2 for all our customers the web suddenly has a lot of HTTP/2 connections. To get the most out of HTTP/2 you'll want to be using an up to date web browser (all the major browsers support HTTP/2).
But there are some non-browser tools that come in handy when working with HTTP/2. This blog post starts with a useful browser add-on, and then delves into command-line tools, load testing, conformance verification, development libraries and packet decoding for HTTP/2.
If you know of something that I've missed please write a comment.
Browser Indicators
For Google Chrome there's a handy HTTP/2 and SPDY Indicator extension that adds a colored lightning bolt to the browser bar showing the protocol being used when a web page is viewed.
The blue lightning bolt shown here indicates that the CloudFlare home page was served using HTTP/2:
A green lightning bolt indicates the site was served using SPDY and gives the SPDY version number. In this case SPDY/3.1:
A grey lightning bolt indicates that neither HTTP/2 no SPDY were used. Here the web page was served using HTTP/1.1.
There's a similar extension for Firefox.
Online testing
There's also a handy online Continue reading
HTTP/2 is here! Goodbye SPDY? Not quite yet
Why choose, if you can have both? Today CloudFlare is introducing HTTP/2 support for all customers using SSL/TLS connections, while still supporting SPDY. There is no need to make a decision between SPDY or HTTP/2. Both are automatically there for you and your customers.
Enabling HTTP/2
If you are a customer on the Free or Pro plan, there is no need to do anything at all. Both SPDY and HTTP/2 are already enabled for you. With this improvement, your website’s audience will always use the fastest protocol version when accessing your site over TLS/SSL.
Customers on Business and Enterprise plans may enable HTTP/2 within the "Network" application of the CloudFlare Dashboard.
HTTP/2 is here!
In February of 2015, the IETF’s steering group for publication as standards-track RFCs approved the HTTP/2 and associated HPACK specifications.
After more than 15 years, the Hypertext Transfer Protocol (HTTP) received a long-overdue upgrade. HTTP/2 is largely based on Google's experimental SPDY protocol, which was first announced in November 2009 as an internal project to increase the speed of the web.
Benefits of HTTP/2 and SPDY
The main focus of both SPDY and HTTP/2 is on performance, especially latency as perceived by the end-user while using Continue reading
Zürich, Switzerland: CloudFlare’s 69th data center
Grüetzi Zürich, our 5th point of presence (PoP) to be announced this week, and 69th globally! Located at the northern tip of Lake Zürich in Switzerland, the city of Zürich, often referred to as "Downtown Switzerland," is the largest city in the country. Following this expansion, traffic from Switzerland's seven million internet users to sites and apps using CloudFlare is now mere milliseconds away. Although best known to some for its chocolate and banks, Switzerland is home to many of the most significant developments prefacing the modern internet.
Vague but interesting
It was in 1989 that Tim Berners-Lee, a British scientist at CERN, the large particle physics laboratory near Geneva, Switzerland, invented the World Wide Web (WWW). Tim laid out his vision to meet the demand for automatic information-sharing between scientists in universities and institutes around the world in a memo titled, "Information Management: a Proposal". Amusingly, his initial proposal wasn't immediately accepted. In fact, his boss at the time noted that the proposal was, "vague but exciting" on the cover page.
At least one VC may have said something similar to us once upon a time
The first website at CERN—and in the world—was dedicated to the Continue reading
CloudFlare launches India data centers in Mumbai, Chennai and New Delhi
India is home to 400 million Internet users, second only to China, and will add more new users this year than any other country in the world. CloudFlare protects and accelerates 4 million websites, mobile apps and APIs, and is trusted by over 10,000 new customers each day. Combine these forces, and we are positioned to connect hundreds of millions of Indian users with the millions of internet applications they use each day.
Today, we accelerate this momentum with the announcement of three new points of presence (PoPs) in Mumbai, Chennai and New Delhi. These new sites represent the 66th, 67th and 68th data centers respectively across our global network.
We’ve come a long way
The beginnings of the “internet” in India as we know it started in 1986 when the country launched ERNET (the Education and Research Network). Six years later, a 64 Kbps digital leased line was commissioned from the National Centre for Software Technology in Mumbai to UUNet in Virginia to connect India with the rest of the internet. By comparison, a single port on our router in each of Mumbai, Chennai and New Delhi has nearly 160,000 times the capacity today.
The pace of progress has Continue reading
The story of one latency spike
A customer reported an unusual problem with our CloudFlare CDN: our servers were responding to some HTTP requests slowly. Extremely slowly. 30 seconds slowly. This happened very rarely and wasn't easily reproducible. To make things worse all our usual monitoring hadn't caught the problem. At the application layer everything was fine: our NGINX servers were not reporting any long running requests.
Time to send in The Wolf.
He solves problems.
Following the evidence
First, we attempted to reproduce what the customer reported—long HTTP responses. Here is a chart of of test HTTP requests time measured against our CDN:
We ran thousands of HTTP queries against one server over a couple of hours. Almost all the requests finished in milliseconds, but, as you can clearly, see 5 requests out of thousands took as long as 1000ms to finish. When debugging network problems the delays of 1s, 30s are very characteristic. They may indicate packet loss since the SYN packets are usually retransmitted at times 1s, 3s, 7s, 15, 31s.
Blame the network
At first we thought the spikes in HTTP load times might indicate some sort of network problem. To be sure we ran ICMP pings against two IPs over many Continue reading
Copenhagen, Denmark: CloudFlare’s 65th data center
To get the week started it's our distinct pleasure to introduce CloudFlare's latest PoP (point of presence) in Copenhagen, Denmark. Our Copenhagen data center extends the CloudFlare network to 65 PoPs across 34 countries, with 17 in Europe alone. The CloudFlare network, including all of the Internet applications and content of our users, is now delivered with a median latency of under 40ms throughout the entire continent—by comparison, it takes 300-400ms to blink one's eyes!
Danish traffic, previously served from Stockholm and Amsterdam, shifts into Copenhagen
As can be seen above, traffic has already started to reach Copenhagen, with steady increases over the course of the day (all times in UTC). The new site is also already mitigating cyber attacks launched against our customers. The spike in traffic around 08:46 UTC is a modest portion of a globally distributed denial of service (DDoS) attack targeted at CloudFlare. By distributing the attack across an ever growing footprint of data centers, mitigation is made easy (and our site reliability engineers can sleep soundly!).
The week's not over
In December 2014 we announced our intention to launch one data center per week throughout 2015. It's an ambitious goal, but we're well on Continue reading
Announcing Universal DNSSEC: Secure DNS for Every Domain
CloudFlare launched just five years ago with the goal of building a better Internet. That’s why we are excited to announce that beginning today, anyone on CloudFlare can secure their traffic with DNSSEC in just one simple step.
This follows one year after we made SSL available for free, and in one week, more than doubled the size of the encrypted web. Today we will do the same with DNSSEC, and this year, we’ll double the size of the DNSSEC-enabled web, bringing DNSSEC to millions of websites, for free.
If DNS is the phone book of the Internet, DNSSEC is the unspoofable caller ID. DNSSEC ensures that a website’s traffic is safely directed to the correct servers, so that a connection to a website is not intercepted by a man-in-the-middle.
Solving A Decades-Old Vulnerability In DNS
Every website visit begins with a DNS query. When I visit cloudflare.com, my browser first needs to find the IP address:
cloudflare.com. 272 IN A 198.41.215.163
When DNS was invented in 1983, the Internet was used by only a handful of professors and researchers, and no one imagined that there could be foul play. Thus, DNS relies on Continue reading
Phoenix, AZ: CloudFlare’s 64th data center
Three years and 46 data centers later our expansion returns to the United States. Phoenix, the latest addition to the CloudFlare network, is our 10th point of presence in North America, and the start of our effort to further regionalize traffic across the continent. This means faster page loads and transaction speeds for your sites and applications, as well as for the 6 million Internet users throughout the Southwestern US that use them.
Eat Surf local
The vast majority of Internet traffic in the US is exchanged in only a small handful of cities: Los Angeles, the San Francisco Bay Area, Dallas, Chicago, Miami, Ashburn (Virginia) and New York. These locations evolved into key interconnection points largely as a result of their status as population and economic centers. However, if you're one of the 236 million Americans that live outside of these metro areas, you have to hike quite a bit further to access your favorite content on the Internet.
To illustrate this, we measured the level of local interconnection between a handful of our Tier 1 Internet providers—NTT, TeliaSonera, Tata Communications and Cogent—in different metro areas. For the uninitiated, Tier 1 networks are the group of networks that Continue reading
CloudFlare is now PCI 3.1 certified
The Payment Card Industry Data Security Standard (PCI DSS) is a global financial information security standard that keeps credit card holders safe. It ensures that any company processing credit card transactions adheres to the highest technical standards.
PCI certification has several levels. Level one (the highest level) is reserved for those companies that handle the greatest numbers of credit cards. Companies at level one PCI compliance are subject to the most stringent checks.
CloudFlare’s mission leads it to provide security for some of the most important companies in the world. This is why CloudFlare chose to be audited as a level one service provider. By adhering to PCI’s rigorous financial security controls, CloudFlare ensures that security is held to the highest standard and that those controls are validated independently by a recognised body.
If you are interested in learning more, see these details about the Payment Card Industry Data Security Standard.
This year’s update from PCI 2.0 to 3.1 was long overdue. PCI DSS 2.0 was issued in October 2010, and the information security threat landscape does not stand still—especially when it comes to industries that deal with financial payments or credit cards. New attacks are almost Continue reading
Creative foot-shooting with Go RWMutex
Hi, I'm Filippo and today I managed to surprise myself! (And not in a good way.)
I'm developing a new module ("filter" as we call them) for RRDNS, CloudFlare's Go DNS server. It's a rewrite of the authoritative module, the one that adds the IP addresses to DNS answers.
It has a table of CloudFlare IPs that looks like this:
type IPMap struct {
sync.RWMutex
M map[string][]net.IP
}
It's a global filter attribute:
type V2Filter struct {
name string
IPTable *IPMap
// [...]
}
CC-BY-NC-ND image by Martin SoulStealer
The table changes often, so a background goroutine periodically reloads it from our distributed key-value store, acquires the lock (f.IPTable.Lock()), updates it and releases the lock (f.IPTable.Unlock()). This happens every 5 minutes.
Everything worked in tests, including multiple and concurrent requests.
Today we deployed to an off-production test machine and everything worked. For a few minutes. Then RRDNS stopped answering queries for the beta domains served by the new code.
What. That worked on my laptop™.
Here's the IPTable consumer function. You can probably spot the bug.
func (f *V2Filter) getCFAddr(...) (result []dns.RR) {
f. Continue readingBen Fathi: Why I Joined CloudFlare
I’m sure some of you are scratching your head right about now wondering why I would join an Internet security and optimization company. But, Ben, this is not even close to your passion: operating systems.
I had the same reaction when I first saw the CloudFlare website. I wasn’t even sure it made sense for me to go interview here. After taking a closer look, however, I realized that it would be the perfect new home for me. Take a look at this page for a brief introduction to what CloudFlare does and how we do it.
Interviewing at CloudFlare
If you know me, you know that I'm a sucker for distributed systems. I fall for a hard computer science problem every time. So, it shouldn’t be a surprise to you that CloudFlare’s John Graham-Cumming, had me at “hello” when he nonchalantly described one of the company's projects: a globally distributed key value store with sub-second consistency guarantees! Ho hum! No big deal.
As the interview process progressed, the team graciously spent several hours walking me through the architecture as well as future plans and product roadmaps. These discussions and email exchanges were frequently interrupted by my cries of protest: Continue reading
Results of experimenting with Brotli for dynamic web content
Compression is one of the most important tools CloudFlare has to accelerate website performance. Compressed content takes less time to transfer, and consequently reduces load times. On expensive mobile data plans, compression even saves money for consumers. However, compression is not free—it comes at a price. It is one of the most compute expensive operations our servers perform, and the better the compression rate we want, the more effort we have to spend.
The most popular compression format on the web is gzip. We put a great deal of effort into improving the performance of the gzip compression, so we can perform compression on the fly with fewer CPU cycles. Recently a potential replacement for gzip, called Brotli, was announced by Google. Being early adopters for many technologies, we at CloudFlare want to see for ourselves if it is as good as claimed.
This post takes a look at a bit of history behind gzip and Brotli, followed by a performance comparison.
Compression 101
Many popular lossless compression algorithms rely on LZ77 and Huffman coding, so it’s important to have a basic understanding of these two techniques before getting into gzip or Brotli.
LZ77
LZ77 is a simple technique developed Continue reading
DNSSEC is Open for Beta
Since January, CloudFlare has been running a small, private beta for DNSSEC. Starting today, the DNSSEC beta is open for everyone. To request access, email [email protected].
A Background on DNS and DNSSEC
DNS is the system that lets your browser know which web server to connect to when you request to visit a website. It’s the underlying backbone of the usable internet, and yet, is vulnerable to man in the middle attacks.
In DNS, an attacker sitting in the middle of your connection to the internet can tell your browser to connect to any web server they’d like. Browsers trust any DNS records they receive as a response to a DNS query, because DNS, invented in 1983 before the public consumption of the Internet, does not perform any authentication.
There is a solution. It’s called DNSSEC and it adds cryptographic hashes and signatures for authenticating DNS records. You can read more about DNSSEC and how it works in a previous blog post.
The DNSSEC beta is open to all websites that use CloudFlare for DNS. If you want to be a part of our beta and be one of the first CloudFlare websites with DNSSEC, email us for beta Continue reading
A Look at the New WordPress Brute Force Amplification Attack
Recently, a new brute force attack method for WordPress instances was identified by Sucuri. This latest technique allows attackers to try a large number of WordPress username and password login combinations in a single HTTP request.
The vulnerability can easily be abused by a simple script to try a significant number of username and password combinations with a relatively small number of HTTP requests. The following diagram shows a 4-fold increase in login attempts to HTTP requests, but this can trivially be expanded to a thousand logins.
This form of brute force attack is harder to detect, since you won’t necessarily see a flood of requests. Fortunately, all CloudFlare paid customers have the option to enable a Web Application Firewall ruleset to stop this new attack method.
What is XML-RPC?
To understand the vulnerability, it’s important to understand the basics of the XML remote procedure protocol (XML-RPC).
XML-RPC uses XML encoding over HTTP to provide a remote procedure call protocol. It’s commonly used to execute various functions in a WordPress instance for APIs and other automated tasks. Requests that modify, manipulate, or view data using XML-RPC require user credentials with sufficient permissions.
Here is an example that requests a list Continue reading
Single RX queue kernel bypass in Netmap for high packet rate networking
In a previous post we discussed the performance limitations of the Linux kernel network stack. We detailed the available kernel bypass techniques allowing user space programs to receive packets with high throughput. Unfortunately, none of the discussed open source solutions supported our needs. To improve the situation we decided to contribute to the Netmap project. In this blog post we'll describe our proposed changes.
CC BY-SA 2.0 image by Binary Koala
Our needs
At CloudFlare we are constantly dealing with large packet floods. Our network constantly receives a large volume of packets, often coming from many, simultaneous attacks. In fact, it is entirely possible that the server which just served you this blog post is dealing with a many-million packets per second flood right now.
Since the Linux Kernel can't really handle a large volume of packets, we need to work around it. During packet floods we offload selected network flows (belonging to a flood) to a user space application. This application filters the packets at very high speed. Most of the packets are dropped, as they belong to a flood. The small number of "valid" packets are injected back to the kernel and handled in the same way Continue reading
Doubling the speed of jpegtran with SIMD
It is no secret that at CloudFlare we put a great effort into accelerating our customers' websites. One way to do it is to reduce the size of the images on the website. This is what our Polish product is for. It takes various images and makes them smaller using open source tools, such as jpegtran, gifsicle and pngcrush.
However those tools are computationally expensive, and making them go faster, makes our servers go faster, and subsequently our customers' websites as well.
Recently, I noticed that we spent ten times as much time "polishing" jpeg images as we do when polishing pngs.
We already improved the performance of pngcrush by using our supercharged version of zlib. So it was time to look what can be done for jpegtran (part of the libjpeg distribution).
Quick profiling
To get fast results I usually use the Linux perf utility. It gives a nice, if simple, view of the hotspots in the code. I used this image for my benchmark.
perf record ./jpegtran -outfile /dev/null -progressive -optimise -copy none test.jpeg
And we get:
perf report
54.90% lt-jpegtran libjpeg.so.9.1.0 [.] encode_mcu_AC_refine
Continue reading
Manchester, UK: CloudFlare’s 63rd data center
Our new point of presence in Manchester, UK brings the CloudFlare network to 63 points of presence across 33 countries. In other words, the sun never sets across the CloudFlare network. Our data center in Manchester also admits the United Kingdom into a small club of countries with more than one CloudFlare data center, including the US, China, Japan, Australia, Germany, and France.
As of yesterday, traffic from the majority of Internet users in Northern England is now mere milliseconds away. More importantly, our Manchester and London data centers allow for redundancy and content localization within the UK for all of our customers.
In homage
The city of Manchester has made more than its fair share of technical contributions over the years. It is the city where Rolls met Royce (their first car drove off the line of their Manchester factory in 1904), and is also home to the first modern computer. The computer, nicknamed "Baby", was built at The University of Manchester using technology developed for WWII communications equipment, and ran the world's first stored program at 11am on Monday 21st June, 1948.
It is fitting, then, that in the last three years there has been a concerted effort Continue reading
Happy 5th Birthday, CloudFlare!
CloudFlare customers recorded videos to celebrate our first five years
Today is September 27, 2015. It's a rare Super Blood Moon. And it's also CloudFlare's birthday. CloudFlare launched 5 years ago today. It was a Monday. While Michelle, Lee, and I had high expectations, we would never have imagined what's happened since then.
In the last five years we've stopped 7 trillion cyber attacks, saved more than 94,116 years worth of time, and served 99.4 trillion requests — nearly half of those in the last 6 months. You can learn more from this timeline of the last five years.
Celebrating by doing the impossible
Every year we like to celebrate our birthday by giving something seemingly impossible back to our users. Two years ago we enabled on our Automatic IPv6 Gateway, allowing our users to support IPv6 without having to update their own servers. Last year we made Universal SSL support available to all our customers, even those on our free plan. And this year, we announced the expansion across Mainland China, building the first truly global performance and security platform.
Internet Summit & Party
We celebrated in San Francisco last week with CloudFlare's first Internet Summit Continue reading