Take a break and watch two recent engineering talks

Recently, I spoke at the dotGo 2014 conference in Paris and my colleague (and creator of OpenResty) Yichun Zhang spoke at the first NGINX conference in San Francisco.

If you need to take a break, go grab a drink and enjoy one of these two talks.

The Latest and Greatest from ngx_lua: New Features & Tools

Tired of writing NGINX C-modules or setting-up back-end application servers? The ngx_lua module was created to save time and pain, while opening up new possibilities in the world of NGINX. The ngx_lua module embeds the Lua dynamic language into the NGINX core, turning NGINX into a highly scriptable proxy server. Many use it as a non-blocking full-stack web application server as well--also known as OpenResty.

Led by ngx_lua co-creator and sole-maintainer, CloudFlare’s Yichun Zhang, this presentation will introduce all the latest features implemented in the ngx_lua module as well as other new tools. Yichun will focus on features including: light threads, websockets, timers, NGINX worker initialization hooks, SSL/TLS coroutine-based sockets (or “cosockets”), full-duplex cosockets and more.

. .

The session wraps-up covering new advanced tools to troubleshoot and profile ngx_lua-based systems including dynamic tracing utilities based on Systemtap and GDB extension commands.

I came for the Continue reading

Drupal 7 SA-CORE-2014-005 SQL Injection Protection

Yesterday the Drupal Security Team released a critical security patch for Drupal 7 that fixes a very serious SQL injection vulnerability. At the same time we pushed an update to our Drupal WAF rules to mitigate this problem. Any customer using the WAF and with the Drupal ruleset enabled will have received automatic protection.

Rule D0002 provides protection against this vulnerability. If you do not have that ruleset enabled and are using Drupal clicking the ON button next to CloudFlare Drupal in the WAF Settings will enable protection immediately.

CloudFlare WAF protection can help mitigate vulnerabilities like this, but it is vital that Drupal 7 users upgrade to the safe version of Drupal immediately.

SSLv3 Support Disabled By Default Due to POODLE Vulnerability

For the last week we've been tracking rumors about a new vulnerability in SSL. This specific vulnerability, which was just announced, targets SSLv3. The vulnerability allows an attacker to add padding to a request in order to then calculate the plaintext of encryption using the SSLv3 protocol. Effectively, this allows an attacker to compromise the encryption when using the SSLv3 protocol. Full details have been published by Google in a paper which dubs the bug POODLE (PDF).

Generally, modern browsers will default to a more modern encryption protocol (e.g., TLSv1.2). However, it's possible for an attacker to simulate conditions in many browsers that will cause them to fall back to SSLv3. The risk from this vulnerability is that if an attacker could force a downgrade to SSLv3 then any traffic exchanged over an encrypted connection using that protocol could be intercepted and read.

In response, CloudFlare has disabled SSLv3 across our network by default for all customers. This will have an impact on some older browsers, resulting in an SSL connection error. The biggest impact is Internet Explorer 6 running on Windows XP or older. To quantify this, we've been tracking SSLv3 usage.

SSLv3 Continue reading

Automatic protection for common web platforms

If you are a CloudFlare Pro or above customer you enjoy the protection of the CloudFlare WAF. If you use one of the common web platforms, such as WordPress, Drupal, Plone, WHMCS, or Joomla, then it's worth checking if the relevant CloudFlare WAF ruleset is enabled.

That's because CloudFlare pushes updates to these rules automatically when new vulnerabilities are found. If you enable the relevant ruleset for your technology then you'll be protected the moment new rules are published.

For example, here's a screenshot of the WAF Settings for a customer who uses WordPress (but doesn't use Joomla). If CloudFlare pushes rules to the WordPress set then they'll be protected automatically.

Enabling a ruleset is simple. Just click the ON/OFF button and make sure it's set to ON.

Here's a customer with the Drupal ruleset disabled. Clicking the ON/OFF button would enable that ruleset and provide protection from existing vulnerabilities and automatic protection if new rules are deployed.

For common problems we've rolled out protection across the board. For example, we rolled out Heartbleed protection and Shellshock automatically, but for technology-specific updates it's best to enable the appropriate ruleset in the WAF Settings.

CloudFlare Publishes Semiannual Transparency Report:

Painting by Rene Margritte

Today CloudFlare is publishing its third Transparency Report covering the first half of 2014. This report covers government information requests from January 1, 2013 to June 30, 2014, and updates our two existing transparency reports: partial January 2013 Transparency Report and complete 2013 Transparency Report.

CloudFlare’s Transparency Reports shows how many subpoenas, court orders, search warrants, pen register/trap and trace (PRTT) orders, and national security orders CloudFlare received during the reporting period. In this current Transparency Report, we have also added a separate category for wiretap orders CloudFlare received. CloudFlare’s Transparency Reports also shows how many domains and accounts were affected by our response to those requests during the reporting period. CloudFlare’s Transparency Reports do not include non-governmental requests.

We will continue to update this report on a semiannual basis at Transparency Report.

Special thanks to our legal intern, Murtaza Sajjad, for helping to compile this report.

DNSSEC: An Introduction

At CloudFlare our mission is to help build a better Internet. Part of this effort includes making web sites faster, more reliable, and more trustworthy. The obvious first choice in protocols to help make websites more secure is HTTPS. CloudFlare’s latest product—Universal SSL—helps web site operators provide a trustworthy browsing experience for their site visitors by giving their site HTTPS support for free. In this blog post we look at another protocol, DNS, and explore one proposal to improve its trustworthiness: DNSSEC.

DNS is one of the pillars of authority on the Internet. DNS is used to translate domain names (like www.cloudflare.com) to numeric Internet addresses (like 198.41.214.163)—it’s often referred to as the “phone book of the Internet”.

DNSSEC is a set of security extensions to DNS that provides the means for authenticating DNS records. CloudFlare is planning to introduce DNSSEC in the next six months, and has brought Olafur Gudmundsson, one of the co-inventors of DNSSEC, on board to help lead the project.

CC BY 2.0 by Eric Fischer

Introduction

The Domain Name System (DNS) is one of the oldest and most fundamental components of the modern Internet. As the Continue reading

The little extra that comes with Universal SSL

CC BY 2.0 by JD Hancock

Last Monday we announced our SSL for Free plan users called Universal SSL. Universal SSL means that any site running on CloudFlare gets a free SSL certificate, and is automatically secured over HTTPS.

Using SSL for a web site helps make the site more secure, but there's another benefit: it can also make the site faster. That's because the SPDY protocol, created by Google to speed up the web, actually requires SSL and only web sites that support HTTPS can use SPDY.

CloudFlare has long supported SPDY, and kept up to date with improvements in the protocol. We currently support the most recent version of SPDY: 3.1.

CloudFlare's mission to bring the tools of the Internet giants to everyone is two fold: security and performance. As part of the Universal SSL launch, we also rolled out SPDY for everyone. Many of the web's largest sites use SPDY; now all sites that use CloudFlare are in the same league.

If your site is on CloudFlare, and you use a modern browser that supports SPDY, you'll find that the HTTPS version of your site is now served over SPDY. SPDY allows the Continue reading

Route leak incident on October 2, 2014

Today, CloudFlare suffered downtime which caused customers’ sites to be inaccessible in certain parts of the world. We take the availability of our customers’ web properties very seriously. Incidents like this get the absolute highest priority, attention, and follow up. The pain felt by our customers is also felt deeply by the CloudFlare team in London and San Francisco.

This downtime was the result of a BGP route leak by Internexa, an ISP in Latin America. Internexa accidentally directed large amounts of traffic destined for CloudFlare data centers around the world to a single data center in Medellín, Colombia. At the same time Internexa also leaked routes belonging to Telecom Argentina causing disruption in Argentina. This was the result of Internexa announcing via BGP that their network, instead of ours, handled traffic for CloudFlare. This miscommunication caused a flood of traffic to quickly overwhelm the data center in Medellín. The incident lasted 49 minutes, from 15:08UTC to 15:57UTC.

The exact impact of the route leak to our customers’ visitors depended on the geography of the Internet. Traffic to CloudFlare’s customers sites dropped by 50% in North America and 12% in Europe. The impact on our network in Asia was isolated Continue reading

Universal SSL: How It Scales

On Monday, we announced Universal SSL, enabling HTTPS for all websites using CloudFlare’s Free plan. Universal SSL represents a massive increase in the number of sites we serve over HTTPS—from tens of thousands, to millions. People have asked us, both in comments and in person, how our servers handle this extra load. The answer, in a nutshell, is this: we found that with the right hardware, software, and configuration, the cost of SSL on web servers can be reduced to almost nothing.

Modern Hardware

CloudFlare’s entire infrastructure is built on modern commodity hardware. Specifically, our web servers are running on CPUs manufactured by Intel that were designed with cryptography in mind.

All Intel CPUs based on the Westmere CPU microarchitecture (introduced in 2010) and later have specialized cryptographic instructions. Important for CloudFlare’s Universal SSL rollout are the AES-NI instructions which speed up the Advanced Encryption Standard (AES) algorithm. There’s also a set of instructions called Carry-less Multiplication (CLMUL) that computes mathematical operations binary finite fields. CLMUL can be used to speed up AES in Galois Counter-mode (GCM): our preferred mode of encryption due to its resistance against recent attacks like BEAST.

As we described in our primer on TLS Continue reading

Inside Shellshock: How hackers are using it to exploit systems

On Wednesday of last week details, of the Shellshock bash bug emerged. This bug started a scramble to patch computers, servers, routers, firewalls, and other computing appliances using vulnerable versions of bash.

CloudFlare immediately rolled out protection for Pro, Business, and Enterprise customers through our Web Application Firewall. On Sunday, after studying the extent of the problem, and looking at logs of attacks stopped by our WAF, we decided to roll out protection for our Free plan customers as well.

Since then we've been monitoring attacks we've stopped in order to understand what they look like, and where they come from. Based on our observations, it's clear that hackers are exploiting Shellshock worldwide.

(CC BY 2.0 aussiegall)

Eject

The Shellshock problem is an example of an arbitrary code execution (ACE) vulnerability. Typically, ACE vulnerability attacks are executed on programs that are running, and require a highly sophisticated understanding the internals of code execution, memory layout, and assembly language—in short, this type of attack requires an expert.

Attacker will also use an ACE vulnerability to upload or run a program that gives them a simple way of controlling the targeted machine. This is often achieved by running a "shell". Continue reading

Universal SSL: Be just a bit more patient

It turns out it takes a while to deploy SSL certificates for 2 million websites. :-) Even longer when you get a flood of new sign ups. While we'd hoped to have the deployment complete within 24 hours of the announcement, it now looks like it's going to take a bit longer. We now expect that the full deployment will be complete about 48 hours from now (0700 UTC). Beyond that, nothing about the plan for Universal SSL has changed and hundreds of thousands of sites are already active.

Errors you may see

In order to get through the highest priority sites first, we've prioritized provisioning the sites with the most traffic.

While you wait for your site to get provisioned, you may see a certificate mismatch error if you try and visit it over HTTPS. (Rest assured, there are no errors if you visit over HTTP.) The errors over HTTPS are expected and normal during the provisioning process. Examples of what these error looks like in various browsers )Chrome, Safari, Firefox, and Internet Explorer) are below:

Tracking our progress

To give you a sense of our progress provisioning Universal SSL for your sites, we've updated the alert that Continue reading

Origin Server Connection Security with Universal SSL

Earlier today, CloudFlare enabled Universal SSL: HTTPS support for all sites by default. Universal SSL provides state-of-the-art encryption between browsers and CloudFlare’s edge servers keeping web traffic private and secure from tampering.

CloudFlare’s Flexible SSL mode is the default for CloudFlare sites on the Free plan. Flexible SSL mode means that traffic from browsers to CloudFlare will be encrypted, but traffic from CloudFlare to a site's origin server will not be. To take advantage of our Full and Strict SSL mode—which encrypts the connection between CloudFlare and the origin server—it’s necessary to install a certificate on the origin server.

We made Universal SSL free so that everyone can use modern, strong encryption tools to protect their web traffic. More encrypted traffic helps build a safer, better Internet. In keeping with CloudFlare’s goal to help build a better Internet, we have some tips on how to upgrade your site from Flexible SSL to Full or Strict SSL.

Option 1: Full SSL: create a self-signed certificate

Dealing with Certificate Authorities (CAs) can be frustrating, and the process of obtaining a certificate can be time consuming. In the meantime, you can get started by installing a self-signed certificate on your origin server. This Continue reading

Introducing Universal SSL

The team at CloudFlare is excited to announce the release of Universal SSL™. Beginning today, we will support SSL connections to every CloudFlare customer, including the 2 million sites that have signed up for the free version of our service.

This morning we began rolling out the Universal SSL across all our current customers. We expect this process to be complete for all current customers before the end of the day. Yesterday, there were about 2 million sites active on the Internet that supported encrypted connections. By the end of the day today, we'll have doubled that.

For new customers who sign up for CloudFlare's free plan, after we get through provisioning existing customers, it will take up to 24 hours to activate Universal SSL. As always, SSL for paid plans will be provisioned instantly upon signup.

How does it work?

For all customers, we will now automatically provision a SSL certificate on CloudFlare's network that will accept HTTPS connections for a customer's domain and subdomains. Those certificates include an entry for the root domain (e.g., example.com) as well as a wildcard entry for all first-level subdomains (e.g., www.example.com, blog.example.com, etc. Continue reading

Shellshock protection enabled for all customers

On Thursday, we rolled out protection against the Shellshock bash vulnerability for all paying customers through the CloudFlare WAF. This protection was enabled automatically and immediately starting blocking malicious requests.

We had a number of requests for protection from Shellshock for all our customers, including those on the Free plan.

After observing the actual Shellshock traffic across our network and after seeing the true severity of the vulnerability become clear, we've built and tested a special Basic ShellShock Protection for all customers.

That protection is now operating and enabled for every CloudFlare customer (Free, Pro, Business and Enterprise). Paying customers have the additional protection of more complex Shellshock rules in the CloudFlare WAF.

Every CloudFlare customer is now being protected from the most common attack vectors based on the Shellshock problem and paying customers continue to have the more advanced protection that was rolled out yesterday.

One More Thing: Keyless SSL and CloudFlare’s Growing Network

I wanted to write one more thing about Keyless SSL, our announcement from last week, before attention shifts to what we'll be announcing on Monday. Keyless allows us to provide CloudFlare's service without having private SSL keys stored locally on our edge servers. The news last week focused on how this could allow very large customers, like major financial institutions, to use CloudFlare without trusting us with their private keys.

But there's another use that will benefit the entire CloudFlare userbase, not just our largest enterprise customers, and it's this: Keyless SSL is a key part of our strategy to continue to expand CloudFlare's global network.

CloudFlare's Global Network Today

CloudFlare's network today consists of 28 edge data centers that span much of the globe. We have technical and security requirements for these facilities in order to ensure that the equipment they house remains secure. Generally, we're in Tier III or IV data center facilities with the highest level of security. In our San Jose facility, for instance, you have to pass through 5 biometric scans, in addition to multiple 24x7 manned guard check points, before you can get to the electronically locked cabinets housing our servers.

There Continue reading

Celebrating CloudFlare’s 4th Birthday

Since CloudFlare launched to the public four years ago today, we've always considered September 27th our birthday. We like to celebrate by doing something nice for our team and also for our customers. Two years ago, for example, we brought a cake into the office and then enabled free IPv6 support for all our customers.

Saturday is our birthday this year, so we decided to celebrate it a few days later when we'd all be back in the office on Monday, September 29th. That actually corresponds to the day we presented at the finals of the TechCrunch Disrupt startup contest where we launched. We ended up coming in second. Mike Arrington, the founder of TechCrunch, said we were basically "muffler repair for the Internet."

Looking back, that's actually not a bad description. At core, CloudFlare's mission is to help build a better Internet by fixing its biggest problems -- its metaphorical rusty mufflers. This year, we thought it would be great to repair a big, ugly muffler that should have been fixed a long time ago.

This Monday, we'll bring a cake into the office. (It'll have to be a lot bigger as our team has grown substantially.) Continue reading

Bash vulnerability CVE-2014-6271 patched

This morning, Stephane Chazelas disclosed a vulnerability in the program bash, the GNU Bourne-Again-Shell. This software is widely used, especially on Linux servers, such as the servers used to provide CloudFlare’s performance and security cloud services.

This vulnerability is a serious risk to Internet infrastructure, as it allows remote code execution in many common configurations, and the severity is heightened due to bash being in the default configuration of most Linux servers. While bash is not directly used by remote users, it is used internally by popular software packages such as web, mail, and administration servers. In the case of a web server, a specially formatted web request, when passed by the web server to the bash application, can cause the bash software to run commands on the server for the attacker. More technical information was posted on the oss-sec mailing list.

The security community has assigned this bash vulnerability the ID CVE-2014-6271.

As soon as we became aware of this vulnerability, CloudFlare’s engineering and operations teams tested a patch to protect our servers, and deployed it across our infrastructure. As of now, all CloudFlare servers are protected against CVS-2014-6271.

Everyone who is using the bash software package should upgrade Continue reading

Keyless SSL: The Nitty Gritty Technical Details

We announced Keyless SSL yesterday to an overwhelmingly positive response. We read through the comments on this blog, Reddit, Hacker News, and people seem interested in knowing more and getting deeper into the technical details. In this blog post we go into extraordinary detail to answer questions about how Keyless SSL was designed, how it works, and why it’s secure. Before we do so, we need some background about how encryption works on the Internet. If you’re already familiar, feel free to skip ahead.

TLS

Transport Layer Security (TLS) is the workhorse of web security. It lets websites prove their identity to web browsers, and protects all information exchanged from prying eyes using encryption. The TLS protocol has been around for years, but it’s still mysterious to even hardcore tech enthusiasts. Understanding the fundamentals of TLS is the key to understanding Keyless SSL.

Dual goals

TLS has two main goals: confidentiality and authentication. Both are critically important to securely communicating on the Internet.

Communication is considered confidential when two parties are confident that nobody else can understand their conversation. Confidentiality can be achieved using symmetric encryption: use a key known only to the two parties involved to encrypt Continue reading

Announcing Keyless SSL™: All the Benefits of CloudFlare Without Having to Turn Over Your Private SSL Keys

CloudFlare is an engineering-driven company. This is a story we're proud of because it embodies the essence of who we are: when faced with a problem, we found a novel solution. Technical details to follow but, until then, welcome to the no hardware world.

Fall in San Francisco

The story begins on a Saturday morning, in the Fall of 2012, almost exactly two years ago. I got a call on my cell phone that woke me. It was a man who introduced himself as the Chief Information Security Officer (CISO) at one of the world's largest banks.

"I got your number from a reporter," he said. "We have an incident. Could you and some of your team be in New York Monday morning? We'd value your advice." We were a small startup. Of course we were going to drop everything and fly across the country to see if we could help.

I called John Roberts and Sri Rao, two members of CloudFlare's team. John had an air of calm about him and owned more khaki pants than any of the rest of us. Sri was a senior member of our technical operations team and could, already at that point, Continue reading

How Stacks are Handled in Go.

At CloudFlare, We use Go for a variety of services and applications. In this blog post, We're going to take a deep dive into some of the technical intricacies of Go.

One of the more important features of Go is goroutines. They are cheap, cooperatively scheduled threads of execution that are used for a variety of operations, like timeouts, generators and racing multiple backends against each other. To make goroutines suitable for as many tasks as possible, we have to make sure that each goroutine takes up the least amount of memory, but also have people be able to start them up with a minimal amount of configuration.

To achieve this, Go manages stacks in way that behaves like any other language, but is quite different in how they're implemented.

An introduction to thread stacks

Before we look at Go, let's look at how stacks are managed in a traditional language like C.

When you start up a thread in C, the standard library is responsible for allocating a block of memory to be used as that thread's stack. It will allocate this block, tell the kernel where it is and let the kernel handle the execution of the thread. Continue reading