CloudFlare in 2014: Bigger, Faster, Securer

At the end of 2013 we posted a blog article titled 2013: Rebuild the Engine; 2014: Step on the Gas which explained how in 2013 we had been rebuilding the engine that powers CloudFlare and how we expected 2014 to be when we stepped on the gas.

In that blog post, we said that we'd be expanding our network to betters serve customers in China and Latin America (as well as continuing other global expansions), and that we'd be making a big announcement around SSL.

CC BY-ND 2.0 image by Do Hyun-Kim

Looking back at 2014, we did a whole lot more and many of those changes had a meaningful impact well beyond CloudFlare. Now when we make a change, the needles on the Internet's dials move: when we roll out support for new protocols, sites tracking those protocols see a sudden jump in usage.

Here's a month by month review of CloudFlare's 2014:

January 8: keeping our promise to Latin America, we opened our first data center there in Chile.

January 27: we published our first transparency report covering National Security Orders on the first day it became legal to discuss them.

February 13: we Continue reading

Kyoto Tycoon Secure Replication

Kyoto Tycoon is a distributed key-value store written by FAL Labs, and it is used extensively at CloudFlare. Like many popular key-value stores, Kyoto Tycoon uses timestamp-based replication to ensure eventual consistency and guarantee ordering. Kyoto Tycoon is an open source project, and in the spirit of the holidays, we’re contributing our internal changes back to the open source project.

_{CC BY-ND 2.0 image by Moyan Brenn}

CloudFlare uses Kyoto Tycoon to replicate data from a Postgres Database to our 30 data centers around the world. In practice, it takes around 3 seconds for full propagation in normal conditions. This is our pipeline for distributing sensitive data like our session ticket keys and DNS data to the CloudFlare edge.

Protecting data in transit

If the Internet is not a dangerous place, it at least has dangerous neighborhoods. To move from one datacenter to another, data has to pass through the public Internet. Data could end up going though some network with a wire-tap in place, or through a network with an unscrupulous network operator.

Datacenter-to-datacenter encryption has been brought into the international spotlight since the surveillance revelations. One of the leaked slides contained the expression “SSL added Continue reading

Improving PicoHTTPParser further with AVX2

Vlad Krasnov recently joined CloudFlare to work on low level optimization of CloudFlare's servers. This is the first of a number of blog posts that will include code he's optimized and open sourced.

In a recent post, Kazuho's Weblog describes an improvement to PicoHTTPParser. This improvement utilizes the SSE4.2 instruction PCMPESTRI in order to find the delimiters in a HTTP request/response and parse them accordingly. This update, compared to the previous version of the code, is impressive.

CC BY-SA 2.0 image by Intel Free Press

PCMPESTRI is a versatile instruction that allows scanning of up to 16 bytes at once for occurrences of up to 16 distinct characters (bytes), or up to 8 ranges of characters (bytes). It can also be used for string and substring comparison. However, there are a few drawbacks: the instruction has a high latency of 11 cycles, and is limited to 16 bytes per instruction. It's also under utilized for range comparison in PicoHTTPParser, because it only tests two or three ranges per invocation (out of eight it is capable of). Furthermore, some simple math (16 bytes / 11 cycles) shows that using this instruction limits the parser to 1.45 bytes/cycle throughput.

Continue reading

Johannesburg: CloudFlare’s 30th data center

Fire up the celebration braai, Jozi! CloudFlare is here, and it’s a big one. An important milestone (our 30th data center) calls for an equally important new location: Johannesburg, South Africa, our first data center in Africa.

For the local audience: Steek aan 'n braai ter viering, Jozi! CloudFlare is hier en dis 'n groot een. 'n Belangrike mylpaal (ons 30ste datasentrum), vra vir ewe belangrike en nuwe ligging: Johannesburg, Suid-Afrika, ons eerste datasentrum in Afrika.

Now serving Southern Africa

Prior to now nearly all CloudFlare traffic delivered to Africa was served from our London, Amsterdam and Hong Kong data centers with round trip latency of 200-350ms. Bandwidth in the region is notoriously expensive (it would make even the Australians blush) making it prohibitive to enter into the continent. That is, before now. Just a few months ago we were fortunate to enter into discussions with a number of partners in the region that share CloudFlare’s vision to help build a better Internet.

Our Johannesburg data center will not only make sites on CloudFlare more performant for Internet users in South Africa, but also for Internet users across all of Southern Africa (and beyond). From Botswana to Kenya, users Continue reading

Lima, Peru: CloudFlare’s 29th data center

Just when you thought we’d reached the end, CloudFlare’s Latin America data center expansion continues. Hot on the heels of our recent expansion into Santiago, São Paulo, and Medellin, this holiday season commences in Lima with our 29th data center globally, and our fourth in Latin America.

Latin America is the fastest growing source of traffic to CloudFlare's network, with nearly 10x growth in just the last twelve months. Our new data center in Lima reduces the latency to access any site using CloudFlare, increases web performance for users in the region from Iquitos to Tacna, and adds another point of redundancy. It also increases the capacity and surface area of the CloudFlare network to absorb massive cyber attacks. This is of particular benefit to CloudFlare customers the Presidency of Peru and the ONPE, Peru’s National Election Office. In the lead up to the Peruvian elections this month, CloudFlare partnered with the Government of Peru to ensure that local elections go off without a hitch — no easy feat when voter turnout is expected to reach nearly 90%. Whether you are running a site, mobile app, or national election we have an offering for you.

Coming Continue reading

Prepare Your Site for Traffic Spikes this Holiday Season

The holiday season is approaching, and everyone is thinking about gifts for their friends and family. As people increasingly shop online, this means huge spikes in traffic for web sites---especially ecommerce sites. We want you to get the most out of this year’s surge in web traffic, so we’ve created a list of tips to help you prepare your site to ensure your visitors have a reliable and fast experience.

Make sure your site can handle traffic spikes:

1) Contact your hosting provider to understand the limits of your hosting plan

Even though CloudFlare offsets most of the load to your website via caching and request filtering, a certain amount of traffic will still pass through to your host. Knowing the limits of your plan can help prevent a bottleneck from your hosting plan.

2) Reduce the number of unwanted requests to your infrastructure

CloudFlare allows you to block IP address individually or IPs from entire regions. If you don’t want or need traffic from certain IPs or regions, you can block them using your Threat Control panel. This is useful for sites who know where their visitors usually come from.

For example, if you run an ecommerce site with Continue reading

Migrating to the Ghost Blogging Platform

For those of you that follow the CloudFlare blog, you’ll know that we try to be prolific. We have industry leaders like Matthew Prince, John Graham-Cumming, Nick Sullivan, and others publishing pieces weekly from the front lines of internet performance and security. We’re also big fans of open source software, which is used in almost everything we do.

A little over a year ago we watched as a brand new independent open source blogging platform called Ghost started making waves, raising over $300,000 on Kickstarter. A little later, we reached out to the team to see if CloudFlare could help make the lightning-fast Node.js platform even faster and more secure on the Ghost(Pro) hosted service.

In March, Ghost announced that their entire Pro network was powered by CloudFlare, and today we’re pleased to announce that the CloudFlare blog is now running on Ghost.

While things look largely the same, you’ll find new and improved RSS feeds as well as tag and author archives to allow you to browse through our backlog of content more easily. The biggest improvement by far, though, is in the writing tools which we now have available to us—meaning our team is Continue reading

CloudFlare and SHA-1 Certificates

At CloudFlare, we’re dedicated to ensuring sites are not only secure, but also available to the widest audience. In the coming months, both Google’s Chrome browser and Mozilla’s Firefox browser are changing their policy with respect to certain web site certificates. We are aware of these changes, and we have modified our SSL offerings to ensure customer sites continue to be secure and available to all visitors.

Chrome (and Firefox) and SHA-1

Google will be making changes to its Chrome browser in upcoming versions to change the way they treat certain web site certificates based on their digital signature. These changes affect over 80% of websites.

As described in our blog post on CFSSL, web site certificates are organized using a chain of trust. Digital signatures are the glue that connects the certificates in the chain. Each certificate is digitally signed by its issuer using a digital signature algorithm defined by the type of key and a cryptographic hash function (such as MD5, SHA-1, SHA-256).

Starting in Chrome 39 (to be released this month, November 2014), certificates signed with a SHA-1 signature algorithm will be considered less trusted than those signed with a more modern SHA-2 algorithm. This change Continue reading

DNSSEC: Complexities and Considerations

This blog post is a follow-up to our previous introduction to DNSSEC. Read that first if you are not familiar with DNSSEC.

DNSSEC is an extension to DNS: it provides a system of trust for DNS records. It’s a major change to one of the core components of the Internet. In this post we examine some of the complications of DNSSEC, and what CloudFlare plans to do to reduce any negative impact they might have. The main issues are zone content exposure, key management, and the impact on DNS reflection/amplification attacks.

Zone content exposure

DNS is split into smaller pieces called zones. A zone typically starts at a domain name, and contains all records pertaining to the subdomains. Each zone is managed by a single manager. For example, cloudflare.com is a zone containing all DNS records for cloudflare.com and its subdomains (e.g. www.cloudflare.com, api.cloudflare.com).

There is no directory service for subdomains in DNS so if you want to know if api.cloudflare.com exists, you have to ask a DNS server and that DNS server will end up asking cloudflare.com whether api.cloudflare.com exists. This is not true with DNSSEC. In Continue reading

Cloudflare – An Awesome IPv6 Move – Thank you!

Recently Cloudflare made a pretty cool move, and made their IPv6 services available to all of their customers – even the free ones, like me! So first things first, huge kudos to Cloudflare for offering this up; it has offered … Continue reading →

If you liked this post, please do click through to the source at Cloudflare – An Awesome IPv6 Move – Thank you! and give me a share/like. Thank you!

Take a break and watch two recent engineering talks

Recently, I spoke at the dotGo 2014 conference in Paris and my colleague (and creator of OpenResty) Yichun Zhang spoke at the first NGINX conference in San Francisco.

If you need to take a break, go grab a drink and enjoy one of these two talks.

The Latest and Greatest from ngx_lua: New Features & Tools

Tired of writing NGINX C-modules or setting-up back-end application servers? The ngx_lua module was created to save time and pain, while opening up new possibilities in the world of NGINX. The ngx_lua module embeds the Lua dynamic language into the NGINX core, turning NGINX into a highly scriptable proxy server. Many use it as a non-blocking full-stack web application server as well--also known as OpenResty.

Led by ngx_lua co-creator and sole-maintainer, CloudFlare’s Yichun Zhang, this presentation will introduce all the latest features implemented in the ngx_lua module as well as other new tools. Yichun will focus on features including: light threads, websockets, timers, NGINX worker initialization hooks, SSL/TLS coroutine-based sockets (or “cosockets”), full-duplex cosockets and more.

. .

The session wraps-up covering new advanced tools to troubleshoot and profile ngx_lua-based systems including dynamic tracing utilities based on Systemtap and GDB extension commands.

I came for the Continue reading

Drupal 7 SA-CORE-2014-005 SQL Injection Protection

Yesterday the Drupal Security Team released a critical security patch for Drupal 7 that fixes a very serious SQL injection vulnerability. At the same time we pushed an update to our Drupal WAF rules to mitigate this problem. Any customer using the WAF and with the Drupal ruleset enabled will have received automatic protection.

Rule D0002 provides protection against this vulnerability. If you do not have that ruleset enabled and are using Drupal clicking the ON button next to CloudFlare Drupal in the WAF Settings will enable protection immediately.

CloudFlare WAF protection can help mitigate vulnerabilities like this, but it is vital that Drupal 7 users upgrade to the safe version of Drupal immediately.

SSLv3 Support Disabled By Default Due to POODLE Vulnerability

For the last week we've been tracking rumors about a new vulnerability in SSL. This specific vulnerability, which was just announced, targets SSLv3. The vulnerability allows an attacker to add padding to a request in order to then calculate the plaintext of encryption using the SSLv3 protocol. Effectively, this allows an attacker to compromise the encryption when using the SSLv3 protocol. Full details have been published by Google in a paper which dubs the bug POODLE (PDF).

Generally, modern browsers will default to a more modern encryption protocol (e.g., TLSv1.2). However, it's possible for an attacker to simulate conditions in many browsers that will cause them to fall back to SSLv3. The risk from this vulnerability is that if an attacker could force a downgrade to SSLv3 then any traffic exchanged over an encrypted connection using that protocol could be intercepted and read.

In response, CloudFlare has disabled SSLv3 across our network by default for all customers. This will have an impact on some older browsers, resulting in an SSL connection error. The biggest impact is Internet Explorer 6 running on Windows XP or older. To quantify this, we've been tracking SSLv3 usage.

SSLv3 Continue reading

Automatic protection for common web platforms

If you are a CloudFlare Pro or above customer you enjoy the protection of the CloudFlare WAF. If you use one of the common web platforms, such as WordPress, Drupal, Plone, WHMCS, or Joomla, then it's worth checking if the relevant CloudFlare WAF ruleset is enabled.

That's because CloudFlare pushes updates to these rules automatically when new vulnerabilities are found. If you enable the relevant ruleset for your technology then you'll be protected the moment new rules are published.

For example, here's a screenshot of the WAF Settings for a customer who uses WordPress (but doesn't use Joomla). If CloudFlare pushes rules to the WordPress set then they'll be protected automatically.

Enabling a ruleset is simple. Just click the ON/OFF button and make sure it's set to ON.

Here's a customer with the Drupal ruleset disabled. Clicking the ON/OFF button would enable that ruleset and provide protection from existing vulnerabilities and automatic protection if new rules are deployed.

For common problems we've rolled out protection across the board. For example, we rolled out Heartbleed protection and Shellshock automatically, but for technology-specific updates it's best to enable the appropriate ruleset in the WAF Settings.

CloudFlare Publishes Semiannual Transparency Report:

Painting by Rene Margritte

Today CloudFlare is publishing its third Transparency Report covering the first half of 2014. This report covers government information requests from January 1, 2013 to June 30, 2014, and updates our two existing transparency reports: partial January 2013 Transparency Report and complete 2013 Transparency Report.

CloudFlare’s Transparency Reports shows how many subpoenas, court orders, search warrants, pen register/trap and trace (PRTT) orders, and national security orders CloudFlare received during the reporting period. In this current Transparency Report, we have also added a separate category for wiretap orders CloudFlare received. CloudFlare’s Transparency Reports also shows how many domains and accounts were affected by our response to those requests during the reporting period. CloudFlare’s Transparency Reports do not include non-governmental requests.

We will continue to update this report on a semiannual basis at Transparency Report.

Special thanks to our legal intern, Murtaza Sajjad, for helping to compile this report.

DNSSEC: An Introduction

At CloudFlare our mission is to help build a better Internet. Part of this effort includes making web sites faster, more reliable, and more trustworthy. The obvious first choice in protocols to help make websites more secure is HTTPS. CloudFlare’s latest product—Universal SSL—helps web site operators provide a trustworthy browsing experience for their site visitors by giving their site HTTPS support for free. In this blog post we look at another protocol, DNS, and explore one proposal to improve its trustworthiness: DNSSEC.

DNS is one of the pillars of authority on the Internet. DNS is used to translate domain names (like www.cloudflare.com) to numeric Internet addresses (like 198.41.214.163)—it’s often referred to as the “phone book of the Internet”.

DNSSEC is a set of security extensions to DNS that provides the means for authenticating DNS records. CloudFlare is planning to introduce DNSSEC in the next six months, and has brought Olafur Gudmundsson, one of the co-inventors of DNSSEC, on board to help lead the project.

CC BY 2.0 by Eric Fischer

Introduction

The Domain Name System (DNS) is one of the oldest and most fundamental components of the modern Internet. As the Continue reading

The little extra that comes with Universal SSL

CC BY 2.0 by JD Hancock

Last Monday we announced our SSL for Free plan users called Universal SSL. Universal SSL means that any site running on CloudFlare gets a free SSL certificate, and is automatically secured over HTTPS.

Using SSL for a web site helps make the site more secure, but there's another benefit: it can also make the site faster. That's because the SPDY protocol, created by Google to speed up the web, actually requires SSL and only web sites that support HTTPS can use SPDY.

CloudFlare has long supported SPDY, and kept up to date with improvements in the protocol. We currently support the most recent version of SPDY: 3.1.

CloudFlare's mission to bring the tools of the Internet giants to everyone is two fold: security and performance. As part of the Universal SSL launch, we also rolled out SPDY for everyone. Many of the web's largest sites use SPDY; now all sites that use CloudFlare are in the same league.

If your site is on CloudFlare, and you use a modern browser that supports SPDY, you'll find that the HTTPS version of your site is now served over SPDY. SPDY allows the Continue reading

Route leak incident on October 2, 2014

Today, CloudFlare suffered downtime which caused customers’ sites to be inaccessible in certain parts of the world. We take the availability of our customers’ web properties very seriously. Incidents like this get the absolute highest priority, attention, and follow up. The pain felt by our customers is also felt deeply by the CloudFlare team in London and San Francisco.

This downtime was the result of a BGP route leak by Internexa, an ISP in Latin America. Internexa accidentally directed large amounts of traffic destined for CloudFlare data centers around the world to a single data center in Medellín, Colombia. At the same time Internexa also leaked routes belonging to Telecom Argentina causing disruption in Argentina. This was the result of Internexa announcing via BGP that their network, instead of ours, handled traffic for CloudFlare. This miscommunication caused a flood of traffic to quickly overwhelm the data center in Medellín. The incident lasted 49 minutes, from 15:08UTC to 15:57UTC.

The exact impact of the route leak to our customers’ visitors depended on the geography of the Internet. Traffic to CloudFlare’s customers sites dropped by 50% in North America and 12% in Europe. The impact on our network in Asia was isolated Continue reading

Universal SSL: How It Scales

On Monday, we announced Universal SSL, enabling HTTPS for all websites using CloudFlare’s Free plan. Universal SSL represents a massive increase in the number of sites we serve over HTTPS—from tens of thousands, to millions. People have asked us, both in comments and in person, how our servers handle this extra load. The answer, in a nutshell, is this: we found that with the right hardware, software, and configuration, the cost of SSL on web servers can be reduced to almost nothing.

Modern Hardware

CloudFlare’s entire infrastructure is built on modern commodity hardware. Specifically, our web servers are running on CPUs manufactured by Intel that were designed with cryptography in mind.

All Intel CPUs based on the Westmere CPU microarchitecture (introduced in 2010) and later have specialized cryptographic instructions. Important for CloudFlare’s Universal SSL rollout are the AES-NI instructions which speed up the Advanced Encryption Standard (AES) algorithm. There’s also a set of instructions called Carry-less Multiplication (CLMUL) that computes mathematical operations binary finite fields. CLMUL can be used to speed up AES in Galois Counter-mode (GCM): our preferred mode of encryption due to its resistance against recent attacks like BEAST.

As we described in our primer on TLS Continue reading

Inside Shellshock: How hackers are using it to exploit systems

On Wednesday of last week details, of the Shellshock bash bug emerged. This bug started a scramble to patch computers, servers, routers, firewalls, and other computing appliances using vulnerable versions of bash.

CloudFlare immediately rolled out protection for Pro, Business, and Enterprise customers through our Web Application Firewall. On Sunday, after studying the extent of the problem, and looking at logs of attacks stopped by our WAF, we decided to roll out protection for our Free plan customers as well.

Since then we've been monitoring attacks we've stopped in order to understand what they look like, and where they come from. Based on our observations, it's clear that hackers are exploiting Shellshock worldwide.

(CC BY 2.0 aussiegall)

Eject

The Shellshock problem is an example of an arbitrary code execution (ACE) vulnerability. Typically, ACE vulnerability attacks are executed on programs that are running, and require a highly sophisticated understanding the internals of code execution, memory layout, and assembly language—in short, this type of attack requires an expert.

Attacker will also use an ACE vulnerability to upload or run a program that gives them a simple way of controlling the targeted machine. This is often achieved by running a "shell". Continue reading