John Graham-Cumming

Author Archives: John Graham-Cumming

First Bay Area OpenResty Meetup

On March 9, 章亦春, known to most of us as agentzh, organized the first Bay Area OpenResty Meetup at CloudFlare's San Francisco office.

CloudFlare is a big user of Lua, LuaJIT, NGINX and OpenResty and happy to be able to sponsor Yichun's work on this fast, flexible platform.

The slides and videos from the meetup are now available for viewing by people who were unable to be there in person.

abode.io by Dragos Dascalita of Adobe

   

The slides are here.

KONG by Marco Palladino from Mashape

       

The slides can be found here

What's new in OpenResty for 2016 by Yichun Zhang of CloudFlare

       

Yichun's slides are here

If you are interested in being present at the next OpenResty Meetup by sure to follow the meetup itself.

Staying afloat: the DROWN Attack and CloudFlare

CloudFlare customers are automatically protected against the recently disclosed DROWN Attack. We do not have SSLv2 enabled on our servers.

We publish our SSL configuration here so that others can use it. We currently accept TLS 1.0, 1.1 and 1.2.

We are proactively testing our customers' origin web servers to detect vulnerable servers and will be reaching out to any that have a server that is vulnerable to DROWN.

In the interim, ensure that SSLv2 is fully disabled and/or that private keys are not shared with servers that still need to have SSLv2.

Tools for debugging, testing and using HTTP/2

With CloudFlare's release of HTTP/2 for all our customers the web suddenly has a lot of HTTP/2 connections. To get the most out of HTTP/2 you'll want to be using an up to date web browser (all the major browsers support HTTP/2).

But there are some non-browser tools that come in handy when working with HTTP/2. This blog post starts with a useful browser add-on, and then delves into command-line tools, load testing, conformance verification, development libraries and packet decoding for HTTP/2.

If you know of something that I've missed please write a comment.

Browser Indicators

For Google Chrome there's a handy HTTP/2 and SPDY Indicator extension that adds a colored lightning bolt to the browser bar showing the protocol being used when a web page is viewed.

The blue lightning bolt shown here indicates that the CloudFlare home page was served using HTTP/2:

A green lightning bolt indicates the site was served using SPDY and gives the SPDY version number. In this case SPDY/3.1:

A grey lightning bolt indicates that neither HTTP/2 no SPDY were used. Here the web page was served using HTTP/1.1.

There's a similar extension for Firefox.

Online testing

There's also a handy online Continue reading

Weird bug of the day: Twitter in-app browser can’t visit site

We keep a close eye on tweets that mention CloudFlare because sometimes we get early warning about odd errors that we are not seeing ourselves through our monitoring systems.

Towards the end of August we saw a small number of tweets like this one:

indicating that trying to browse to a CloudFlare customer web site using the Twitter in-app browser was resulting in an error page. Which was very odd because it was clearly only happening occasionally: very occasionally.

Luckily, the person who tweeted that was in the same timezone as me and able to help debug together (thanks James White!); we discovered that the following sequence of events was necessary to reproduce the bug:

  1. Click on a link in a tweet to a web site that is using an https URL and open in the Twitter in-app browser (not mobile Safari). This site may or may not be a CloudFlare customer.

  2. Then click on a link on that page to a site over an http URL. This site must be on CloudFlare.

  3. BOOM

That explained why this happened very rarely, but the question became... why did it happen at all? After some debugging it appeared to happen in Continue reading

Test all the things: IPv6, HTTP/2, SHA-2

CloudFlare constantly tries to stay on the leading edge of Internet technologies so that our customers' web sites use the latest, fastest, most secure protocols. For example, in the past we've enabled IPv6 and SPDY/3.1.

Today we've switched on a test server that is open for people to test compatibility of web clients. It's a mirror of this blog and is served from https://http2.cloudflare.com/. The server uses three technologies that it may be helpful to test with: IPv4/IPv6, HTTP/2 and an SSL certificate that uses SHA-2 for its signature.

The server has both IPv4 and IPv6 addresses.

$ dig +short http2.cloudflare.com A
45.55.83.207  
$ dig +short http2.cloudflare.com AAAA
2604:a880:800:10:5ca1:ab1e:f4:e001

The certificate is based on SHA-2 (in this case SHA-256). This is important because SHA-1 is being deprecated by some browsers very soon. On a recent browser the connection will also be secured using ECDHE (for forward secrecy).

And, finally, the server uses HTTP/2 if the browser is capable. For example, in Google Chrome, with the HTTP/2 and SPDY indicator extension the blue lightning bolt indicates that the page was served using HTTP/2:

This server isn't on the normal CloudFlare Continue reading

Railgun v5 has landed: better, faster, lighter

Three years ago we launched Railgun, CloudFlare's origin network optimizer. Railgun allows us to cache the uncacheable to accelerate the connection between CloudFlare and our customers' origin servers. That brings the benefit of a CDN to even dynamic content with no need for 'fast purging' or other tricks. With Railgun even dynamic, ever-changing pages benefit from caching.

CC BY 2.0 image by Nathan E Photography

Over those three years Railgun has been deployed widely by our customers to accelerate the delivery of their web sites and lower their bandwidth costs.

Today we're announcing the availability of Railgun v5 with a number of significant improvements:

We've substantially reduced memory utilization and CPU requirements

Railgun performs delta compression on every request/response requiring CPU (to perform the compression) and memory (to keep a cache of pages to delta against). Version 5 has undergone extensive optimization based on the performance of Railgun on large web sites and at hosting providers. Version 5 requires much less memory and lower CPU.

A new, lighter weight, faster wire protocol

The original Railgun wire protocol that transfer requests and compressed responses between the customer server and CloudFlare's infrastructure has been completely replaced with a new, lighter-weight Continue reading

Ensuring the web is for everyone

This is the text of an internal email I sent at CloudFlare that we thought worth sharing more widely. I annotated it a bit with links that weren't in the original.

"Tim Berners-Lee- Mosaic by Sue Edkins at Sheen Lane Centre" by Robert Smith - Own work. Licensed under CC BY-SA 4.0 via Commons

Subject: Days of future past

Folks,

One of the exciting things about working at CloudFlare is our continual push to stay on top of what's new for our customers. We've pushed things like IPv6 and SPDY in the past; and we'll soon be giving the world DNSSEC and HTTP/2. In the world of SSL we've stayed on top of changes in recommended cipher suites and offer the latest signature algorithms SHA-2 to our customers.

But as we do this we must not forget the old protocols. Because we serve a truly global audience we serve everyone on the planet. It's easy inside a Silicon Valley bubble to think that everyone is on 1Gbps Internet connection with the latest version of Chrome on a new Mac, but the worldwide reality is far different.

We see every type of machine and browser out there. And Continue reading

The oldest trick in the ASCII book

If you're old enough (or interested enough) to have spent a lot of time messing around with the ASCII table then you might have run into a strange fact: it's possible to uppercase ASCII text using just bitwise AND.

And it turns out that in some situations this isn't just a curiosity, but actually useful. Here are the ASCII characters 0x20 (space) to 0x7E (tilde).

     0123456789ABCDEF0123456789ABCDEF
    +--------------------------------
0x20| !"#$%&'()*+,-./0123456789:;<=>?
0x40|@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_
0x60|`abcdefghijklmnopqrstuvwxyz{|}~

It's immediately obvious that each lowercase letter has an ASCII code 0x20 more than the corresponding uppercase letter. For example, lowercase m is 0x6D and uppercase M is 0x4D. And since 0x20 is a single bit then it's possible to uppercase an ASCII letter by taking its code and applying AND 0xDF (masking out the 0x20 bit).

Performing AND 0xDF has no effect on the first two rows above: they, including the uppercase letters, are unchanged. Only the third row is affected. There the lowercase letters get uppercased but there's some collateral damage: ` { | } ~ change to @ [ ] ^.

But if you know that a string has a limited character set then this trick can come in handy. Lots of old protocols Continue reading

A Go Gotcha: When Closures and Goroutines Collide

Here's a small Go gotcha that it's easy to fall into when using goroutines and closures. Here's a simple program that prints out the numbers 0 to 9:

(You can play with this in the Go Playground here)

package main

import "fmt"

func main() {  
    for i := 0; i < 10; i++ {
        fmt.Printf("%d ", i)
    }
}

It's output is easy to predict: 

0 1 2 3 4 5 6 7 8 9

If you decided that it would be nice to run those fmt.Printfs concurrently using goroutines you might be surprised by the result. Here's a version of the code that runs each fmt.Printf in its own goroutine and uses a sync.WaitGroup to wait for the goroutines to terminate. 

package main

import (  
    "fmt"
    "runtime"
    "sync"
)

func main() {  
    runtime.GOMAXPROCS(runtime.NumCPU())

    var wg sync.WaitGroup
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            fmt.Printf("%d ", i)
            wg.Done()
        }()
    }

    wg.Wait()
}

(This code is in the Go Playground here). If you're thinking concurrently then you'll likely predict that the output will be the numbers 0 to 9 in some Continue reading

No upgrade needed: CloudFlare sites already protected from FREAK

The newly announced FREAK vulnerability is not a concern for CloudFlare's SSL customers. We do not support 'export grade' cryptography (which, by its nature, is weak) and we upgraded to the non-vulnerable version of OpenSSL the day it was released in early January.

CC BY 2.0 image by Stuart Heath

Our OpenSSL configuration is freely available on our Github account here as are our patches to OpenSSL 1.0.2.

We strive to stay on top of vulnerabilities as they are announced; in this case no action was necessary as we were already protected by decisions to eliminate cipher suites and upgrade software.

We are also pro-active about disabling protocols and ciphers that are outdated (such as SSLv3, RC4) and keep up to date with the latest and most secure ciphers (such as ChaCha-Poly, forward secrecy and elliptic curves).

End of the road for RC4

Today, we completely disabled the RC4 encryption algorithm for all SSL/TLS connections to CloudFlare sites. It's no longer possible to connect to any site that uses CloudFlare using RC4.

Over a year ago, we disabled RC4 for connections for TLS 1.1 and above because there were more secure algorithms available. In May 2014, we deprecated RC4 by moving it to the lowest priority in our list of cipher suites. That forced any browser that had a good alternative to RC4 to use it. Those two changes meant that almost everyone who was using RC4 to connect to CloudFlare sites switched to a more secure protocol.

Back in May, we noted that some people still needed RC4, particularly people using old mobile phones and some Windows XP users. At the time, 4% of requests using RC4 came from a single phone type: the Nokia 6120.

At the time, we noted that roughly 0.000002% of requests to CloudFlare were using the RC4 protocol. In the last 9 months, that number is halved and so, although some people are still using RC4, we have decided to turn off the protocol. It's simply no longer secure.

The remaining users are almost Continue reading

DDoS Packet Forensics: Take me to the hex!

A few days ago, my colleague Marek sent an email about a DDoS attack against one of our DNS servers that we'd been blocking with our BPF rules. He noticed that there seemed to be a strange correlation between the TTL field in the IP header and the IPv4 source address.

CC BY 2.0 image by Jeremy Keith

The source address was being spoofed, as usual, and apparently chosen randomly, but something else was going on. He offered a bottle of Scotch to the first person to come up with a satisfactory solution.

Here's what some of the packets looked like: 

$ tcpdump -ni eth0 -c 10 "ip[8]=40 and udp and port 53"
1.181.207.7.46337 > x.x.x.x.53: 65098+  
1.178.97.141.45569 > x.x.x.x.53: 65101+  
1.248.136.142.63489 > x.x.x.x.53: 65031+  
1.207.241.195.52993 > x.x.x.x.53: 65072+

$ tcpdump -ni eth0 -c 10 "ip[8]=41 and udp and port 53"
2.10.30.2.2562 > x.x.x.x.53: 65013+  
2.4.9.36.1026 > x.x.x.x.53: 65019+  
2.98. Continue reading

CloudFlare in 2014: Bigger, Faster, Securer

At the end of 2013 we posted a blog article titled 2013: Rebuild the Engine; 2014: Step on the Gas which explained how in 2013 we had been rebuilding the engine that powers CloudFlare and how we expected 2014 to be when we stepped on the gas.

In that blog post, we said that we'd be expanding our network to betters serve customers in China and Latin America (as well as continuing other global expansions), and that we'd be making a big announcement around SSL.

CC BY-ND 2.0 image by Do Hyun-Kim

Looking back at 2014, we did a whole lot more and many of those changes had a meaningful impact well beyond CloudFlare. Now when we make a change, the needles on the Internet's dials move: when we roll out support for new protocols, sites tracking those protocols see a sudden jump in usage.

Here's a month by month review of CloudFlare's 2014:

January 8: keeping our promise to Latin America, we opened our first data center there in Chile.

January 27: we published our first transparency report covering National Security Orders on the first day it became legal to discuss them.

February 13: we Continue reading

Take a break and watch two recent engineering talks

Recently, I spoke at the dotGo 2014 conference in Paris and my colleague (and creator of OpenResty) Yichun Zhang spoke at the first NGINX conference in San Francisco.

If you need to take a break, go grab a drink and enjoy one of these two talks.

The Latest and Greatest from ngx_lua: New Features & Tools

Tired of writing NGINX C-modules or setting-up back-end application servers? The ngx_lua module was created to save time and pain, while opening up new possibilities in the world of NGINX. The ngx_lua module embeds the Lua dynamic language into the NGINX core, turning NGINX into a highly scriptable proxy server. Many use it as a non-blocking full-stack web application server as well--also known as OpenResty.

Led by ngx_lua co-creator and sole-maintainer, CloudFlare’s Yichun Zhang, this presentation will introduce all the latest features implemented in the ngx_lua module as well as other new tools. Yichun will focus on features including: light threads, websockets, timers, NGINX worker initialization hooks, SSL/TLS coroutine-based sockets (or “cosockets”), full-duplex cosockets and more.

. .

The session wraps-up covering new advanced tools to troubleshoot and profile ngx_lua-based systems including dynamic tracing utilities based on Systemtap and GDB extension commands.

I came for the Continue reading

Drupal 7 SA-CORE-2014-005 SQL Injection Protection

Yesterday the Drupal Security Team released a critical security patch for Drupal 7 that fixes a very serious SQL injection vulnerability. At the same time we pushed an update to our Drupal WAF rules to mitigate this problem. Any customer using the WAF and with the Drupal ruleset enabled will have received automatic protection.

Rule D0002 provides protection against this vulnerability. If you do not have that ruleset enabled and are using Drupal clicking the ON button next to CloudFlare Drupal in the WAF Settings will enable protection immediately.

CloudFlare WAF protection can help mitigate vulnerabilities like this, but it is vital that Drupal 7 users upgrade to the safe version of Drupal immediately.

Automatic protection for common web platforms

If you are a CloudFlare Pro or above customer you enjoy the protection of the CloudFlare WAF. If you use one of the common web platforms, such as WordPress, Drupal, Plone, WHMCS, or Joomla, then it's worth checking if the relevant CloudFlare WAF ruleset is enabled.

That's because CloudFlare pushes updates to these rules automatically when new vulnerabilities are found. If you enable the relevant ruleset for your technology then you'll be protected the moment new rules are published.

For example, here's a screenshot of the WAF Settings for a customer who uses WordPress (but doesn't use Joomla). If CloudFlare pushes rules to the WordPress set then they'll be protected automatically.

Enabling a ruleset is simple. Just click the ON/OFF button and make sure it's set to ON.

Here's a customer with the Drupal ruleset disabled. Clicking the ON/OFF button would enable that ruleset and provide protection from existing vulnerabilities and automatic protection if new rules are deployed.

For common problems we've rolled out protection across the board. For example, we rolled out Heartbleed protection and Shellshock automatically, but for technology-specific updates it's best to enable the appropriate ruleset in the WAF Settings.

The little extra that comes with Universal SSL

CC BY 2.0 by JD Hancock

Last Monday we announced our SSL for Free plan users called Universal SSL. Universal SSL means that any site running on CloudFlare gets a free SSL certificate, and is automatically secured over HTTPS.

Using SSL for a web site helps make the site more secure, but there's another benefit: it can also make the site faster. That's because the SPDY protocol, created by Google to speed up the web, actually requires SSL and only web sites that support HTTPS can use SPDY.

CloudFlare has long supported SPDY, and kept up to date with improvements in the protocol. We currently support the most recent version of SPDY: 3.1.

CloudFlare's mission to bring the tools of the Internet giants to everyone is two fold: security and performance. As part of the Universal SSL launch, we also rolled out SPDY for everyone. Many of the web's largest sites use SPDY; now all sites that use CloudFlare are in the same league.

If your site is on CloudFlare, and you use a modern browser that supports SPDY, you'll find that the HTTPS version of your site is now served over SPDY. SPDY allows the Continue reading

Route leak incident on October 2, 2014

Today, CloudFlare suffered downtime which caused customers’ sites to be inaccessible in certain parts of the world. We take the availability of our customers’ web properties very seriously. Incidents like this get the absolute highest priority, attention, and follow up. The pain felt by our customers is also felt deeply by the CloudFlare team in London and San Francisco.

This downtime was the result of a BGP route leak by Internexa, an ISP in Latin America. Internexa accidentally directed large amounts of traffic destined for CloudFlare data centers around the world to a single data center in Medellín, Colombia. At the same time Internexa also leaked routes belonging to Telecom Argentina causing disruption in Argentina. This was the result of Internexa announcing via BGP that their network, instead of ours, handled traffic for CloudFlare. This miscommunication caused a flood of traffic to quickly overwhelm the data center in Medellín. The incident lasted 49 minutes, from 15:08UTC to 15:57UTC.

The exact impact of the route leak to our customers’ visitors depended on the geography of the Internet. Traffic to CloudFlare’s customers sites dropped by 50% in North America and 12% in Europe. The impact on our network in Asia was isolated Continue reading

Inside Shellshock: How hackers are using it to exploit systems

On Wednesday of last week details, of the Shellshock bash bug emerged. This bug started a scramble to patch computers, servers, routers, firewalls, and other computing appliances using vulnerable versions of bash.

CloudFlare immediately rolled out protection for Pro, Business, and Enterprise customers through our Web Application Firewall. On Sunday, after studying the extent of the problem, and looking at logs of attacks stopped by our WAF, we decided to roll out protection for our Free plan customers as well.

Since then we've been monitoring attacks we've stopped in order to understand what they look like, and where they come from. Based on our observations, it's clear that hackers are exploiting Shellshock worldwide.

(CC BY 2.0 aussiegall)

Eject

The Shellshock problem is an example of an arbitrary code execution (ACE) vulnerability. Typically, ACE vulnerability attacks are executed on programs that are running, and require a highly sophisticated understanding the internals of code execution, memory layout, and assembly language—in short, this type of attack requires an expert.

Attacker will also use an ACE vulnerability to upload or run a program that gives them a simple way of controlling the targeted machine. This is often achieved by running a "shell". Continue reading

Shellshock protection enabled for all customers

On Thursday, we rolled out protection against the Shellshock bash vulnerability for all paying customers through the CloudFlare WAF. This protection was enabled automatically and immediately starting blocking malicious requests.

We had a number of requests for protection from Shellshock for all our customers, including those on the Free plan.

After observing the actual Shellshock traffic across our network and after seeing the true severity of the vulnerability become clear, we've built and tested a special Basic ShellShock Protection for all customers.

That protection is now operating and enabled for every CloudFlare customer (Free, Pro, Business and Enterprise). Paying customers have the additional protection of more complex Shellshock rules in the CloudFlare WAF.

Every CloudFlare customer is now being protected from the most common attack vectors based on the Shellshock problem and paying customers continue to have the more advanced protection that was rolled out yesterday.

1 … 4 5 6 7