Ensuring the web is for everyone

This is the text of an internal email I sent at CloudFlare that we thought worth sharing more widely. I annotated it a bit with links that weren't in the original.

"Tim Berners-Lee- Mosaic by Sue Edkins at Sheen Lane Centre" by Robert Smith - Own work. Licensed under CC BY-SA 4.0 via Commons

Subject: Days of future past

Folks,

One of the exciting things about working at CloudFlare is our continual push to stay on top of what's new for our customers. We've pushed things like IPv6 and SPDY in the past; and we'll soon be giving the world DNSSEC and HTTP/2. In the world of SSL we've stayed on top of changes in recommended cipher suites and offer the latest signature algorithms SHA-2 to our customers.

But as we do this we must not forget the old protocols. Because we serve a truly global audience we serve everyone on the planet. It's easy inside a Silicon Valley bubble to think that everyone is on 1Gbps Internet connection with the latest version of Chrome on a new Mac, but the worldwide reality is far different.

We see every type of machine and browser out there. And Continue reading

Dear Internet, Send Us Your Videos

CloudFlare turns 5 years old this September. It's been an amazing ride since our launch. Before we launched at TechCrunch Disrupt on September 27, 2010, we'd signed up about 1,000 beta customers. It took us nine months to get those first customers. (By comparison, today we typically sign up 1,000 customers every 3 hours.)

Those first beta customers were instrumental. They put up with us when we were had only one data center (in Chicago). They put up with us as we brought traffic online in our next facilities in Ashburn, Virginia and San Jose, California — and had the routing challenges that came along with running a distributed network for the first time. They sent us bug reports, provided us feature requests, and were instrumental to building the foundation that grew into what is CloudFlare today.

Archival Footage

When we launched, we wanted to feature their stories and experience about CloudFlare so we had them submit their stories by video. Here's the video we included as part of our launch presentation.

I'm proud of the fact that more than 80% of those original 1,000 customers are still using CloudFlare five years later.

Send Us Your Stories

As we Continue reading

Now serving the Middle East: 4 new data centers, partnerships

Our last embarkation into a new geography coincided with a significant milestone: our 30th data center (and first in Africa) in Johannesburg, South Africa. And as we march past number 40, we’re proud to announce yet another. Introducing CloudFlare’s latest points of presence (PoPs) in Doha, Qatar; Dubai, United Arab Emirates; Kuwait City, Kuwait; and Muscat, Oman. These data centers are the first wave in our MENA (Middle East/North Africa) expansion, and the 39th, 40th, 41st and 42nd data centers, respectively, to join our global network.

Up to this point all CloudFlare traffic delivered to the MENA region was served from our London, Frankfurt, Marseille, Paris and/or Singapore data centers, with round trip latency of up to 200-350ms. As in Africa, local bandwidth in MENA is notoriously expensive making it cost prohibitive to deliver content locally. That is (once again), until now! We're proud to announce the first of a series of agreements with regional carriers including Etisalat, Omantel, Ooredoo, and Zain to help build a better Internet in the region.

How to build a better Internet

A few of the necessary ingredients to build a better Internet include international connectivity (often in Continue reading

DNS parser, meet Go fuzzer

Here at CloudFlare we are heavy users of the github.com/miekg/dns Go DNS library and we make sure to contribute to its development as much as possible. Therefore when Dmitry Vyukov published go-fuzz and started to uncover tens of bugs in the Go standard library, our task was clear.

Hot Fuzz

Fuzzing is the technique of testing software by continuously feeding it inputs that are automatically mutated. For C/C++, the wildly successful afl-fuzz tool by Michał Zalewski uses instrumented source coverage to judge which mutations pushed the program into new paths, eventually hitting many rarely-tested branches.

go-fuzz applies the same technique to Go programs, instrumenting the source by rewriting it (like godebug does). An interesting difference between afl-fuzz and go-fuzz is that the former normally operates on file inputs to unmodified programs, while the latter asks you to write a Go function and passes inputs to that. The former usually forks a new process for each input, the latter keeps calling the function without restarting often.

There is no strong technical reason for this difference (and indeed afl recently gained the ability to behave like go-fuzz), but it's likely due to the different ecosystems in which they Continue reading

Dublin, Ireland: CloudFlare’s 38th data center

Top of the morning to our users and readers from Ireland! Our latest PoP in Dublin is our 38th globally, and 14th in Europe following our Bucharest deployment last week. As of yesterday, traffic from Ireland's 3.6 million Internet users will now be routed through Dublin as opposed to our London PoP (which will still serve as a point of redundancy).

Silicon Docks

By now you've heard of Silicon Valley, Silicon Alley, and possibly even Silicon Prairie, but across the pond there's another tech hub making quite a name for itself. Silicon Docks, the Dublin neighborhood bordering the Grand Canal Docks, is home to the European headquarters of Google, Facebook, Twitter, Dropbox, AirBnb, LinkedIn and CloudFlare customer, Yelp, just to name a few. While our own European headquarters is in London, Dublin's exploding tech scene made it an obvious choice for a new PoP.

Dublin is also near to our hearts as the home of CloudFlare customers Web Summit and F.ounders, two of the world's premier tech conferences. Visitors to the 2012 Web Summit and F.ounders events may even remember being greeted Continue reading

Up and to the Right: Forrester Research Ranks CloudFlare as a “Leader” for DDoS Services Providers

Forrester Research, Inc. has released The Forrester Wave™: DDoS Services Providers, Q3 2015 report which ranks CloudFlare as a leader. How do you get placed “up and to the right”? The leaders in this Wave, including CloudFlare, demonstrated effective portals, good client and revenue growth, and a focus on customer service. They also all have the ability to defend against the largest amplification attacks and the most pernicious application attacks.

Here’s some of the criteria CloudFlare received the highest possible scores for:

• Attack types defended
• Data/scrubbing center geographic presence
• Detection tactics
• SSL traffic inspection

The DDoS Services Providers Wave also notes that CloudFlare boasts fast mitigation times, and that our customers gave us high marks for service delivery. The report cited CloudFlare’s excellent capabilities to deliver hybrid DDoS solutions as well.

So how does the report evaluate vendors? It evaluates vendors based on three major categories, each with specific criteria:

• Current offering: The strength of vendors’ current DDoS product offering is based on evaluation categories including: business description, amplification attack defense, attack types defended, customer portal features, customer references, data/scrubbing center geographic presence, SSL traffic inspection, and standard mitigation times.

• Strategy: Vendors’ position on the horizontal axis of Continue reading

A deep look at CVE-2015-5477 and how CloudFlare Virtual DNS customers are protected

Last week ISC published a patch for a critical remotely exploitable vulnerability in the BIND9 DNS server capable of causing a crash with a single packet.

CC BY 2.0 image by Ralph Aversen

The public summary tells us that a mistake in handling of queries for the TKEY type causes an assertion to fail, which in turn crashes the server. Since the assertion happens during the query parsing, there is no way to avoid it: it's the first thing that happens on receiving a packet, before any decision is made about what to do with it.

TKEY queries are used in the context of TSIG, a protocol DNS servers can use to authenticate to each other. They are special in that unlike normal DNS queries they include a “meta” record (of type TKEY) in the EXTRA/ADDITIONAL section of the message.

CC BY 2.0 image by Ralph Aversen

Since the exploit packet is now public, I thought we might take a dive and look at the vulnerable code. Let's start by taking a look at the output of a crashing instance:

03-Aug-2015 16:38:55.509 message.c:2352: REQUIRE(*name == ((void*)0)) failed, back trace  
03-Aug-2015 16:38:55.510 #0 0x10001510d in  Continue reading

Quick and dirty annotations for Go stack traces

CloudFlare’s DNS server, RRDNS, is entirely written in Go and typically runs tens of thousands goroutines. Since goroutines are cheap and Go I/O is blocking we run one goroutine per file descriptor we listen on and queue new packets for processing.

CC BY-SA 2.0 image by wiredforlego

When there are thousands of goroutines running, debug output quickly becomes difficult to interpret. For example, last week I was tracking down a problem with a file descriptor and wanted to know what its listening goroutine was doing. With 40k stack traces, good luck figuring out which one is having trouble.

Go stack traces include parameter values, but most Go types are (or are implemented as) pointers, so what you will see passed to the goroutine function is just a meaningless memory address.

We have a couple options to make sense of the addresses: get a heap dump at the same time as the stack trace and cross-reference the pointers, or have a debug endpoint that prints a goroutine/pointer -> IP map. Neither are seamless.

Underscore to the rescue

However, we know that integers are shown in traces, so what we did is first convert IPv4 addresses to their uint32 Continue reading

Bucharest, Romania: CloudFlare’s 37th data center

Our global expansion continues in Bucharest, Romania, the 6th largest city in the European Union* following London, Berlin, Madrid, Rome, and Paris (nearly all of which feature a CloudFlare PoP!). From Bucharest, our latest data center will serve all 11 million Romanian Internet users, as well as users throughout the Balkans and Eastern Europe.

In good company

Romania is geographically situated between Bulgaria, Hungary, Moldova, Serbia, and Ukraine, making it an ideal destination to attract additional Internet traffic throughout much of Eastern Europe. Of course, geographic reality is rarely a mirror of Internet reality. Adding a new point of presence doesn't automatically mean that traffic from surrounding areas (or even traffic in the very same country) will route to that particular data center. This entirely depends on the interconnection of International carriers with local Internet service providers (ISPs) and large networks like CloudFlare.

It is for this precise reason that we place even more emphasis on our interconnection within a particular PoP as opposed to the absolute number of dots we add to our network map. Of course, the combination of the two (expanding wide and deep) is even better, and is why CloudFlare is blazing fast Continue reading

CloudFlare headed to HostingCon 2015. Thanks for the memories and let’s create some more!

The CloudFlare team is heading to HostingCon 2015 in San Diego next week. We are excited to meet colleagues from the industry, reconnect with partners, and make new friends.

This year’s conference marks a milestone of sorts. It’s our fifth time at HostingCon and we’ve come full circle - our first HostingCon took place in San Diego. Here are some fun facts on what we’ve accomplished since our first HostingCon in 2011:

• 25 new data centers expanded our network to a total of 36 worldwide
• 2M+ customers served
• 800+ conference attendees transported in our signature limo service. If you haven’t already, sign up to arrive in style.
• 2,500+ Nerf guns delivered. Check out the new models this year in celebration of Railgun 5.0 launch
• 3,000+ CloudFlare t-shirts bringing smiles to our partners

Today, CloudFlare is trusted by over 5,000 partners who offer performance and security to millions of customers accelerating and protecting websites, APIs, and mobile apps. We work hard to deliver real savings for our partners. For example, over the past month we saved our partners more than 25 petabytes in aggregate bandwidth (roughly equivalent to 350 hours of HDTV video); stopped 65 billion+ malicious attacks that would Continue reading

Introducing Partner Analytics

CloudFlare has over 5,000 partner hosting providers. Every day, thousands of our partners' customers take advantage of CloudFlare to help them be faster and more secure. The benefits to our partners aren't just happier customers, they also translate into real savings. In the last month, for instance, we saved our partners more than 25 Petabytes in aggregate bandwidth. In addition to bandwidth savings, in that same period, we stopped more than 65 billion malicious requests that would have otherwise impacted our partners' infrastructure. Now we've broken out the bandwidth and performance data by partners so they can see the savings and protection we're delivering.

Back when we launched the CloudFlare Partner Program four years ago, we periodically distributed these figures as high level summaries of bandwidth saved, threats blocked, and number of domains protected and accelerated via each partnership. Our partners knew anecdotally from their own logs and operating expenditures that CloudFlare was reducing their costs and greatly improving their customers’ experiences, but we did not yet have the tools to help demonstrate these benefits on a repeatable and granular basis.

It wasn’t that we didn’t want to provide this data, it was that our tremendous growth rate had stretched Continue reading

Célébrer le 14 Juillet avec Marseille, le 36ème point de présence de CloudFlare

What better day than the 14th of July (Bastille Day) to announce the latest addition to our network in Marseille, France? Our data center in the southern city of Marseille is our 2nd in France, 12th in Europe and 36th globally.

Pourquoi Marseille?

Marseille, France’s second largest city following Paris, is home to 2 million Internet users across the surrounding metropolitan area. It also serves as another point of redundancy to our Paris data center, one of our most trafficked facilities in the whole of Europe.

However, the true importance of Marseille is not just redundancy or its size. Marseille’s southern location makes it a major Internet gateway for networks throughout the Mediterranean, including many African and Middle Eastern countries. This is reflected by the fact that a substantial number of undersea submarine cables carrying Internet traffic are routed through Marseille (7 to be exact, and for those fastidious followers of our blog).

These undersea cables are the principal means by which many countries are able to access the rest of the Internet—that is to say, access all of the other global networks that make up this big Continue reading

CloudFlare Lands a New Office in Singapore

After months of preparation, my teammates Algin, Marty, Adam, Jono and I touched down in Singapore and were greeted by skyscrapers, malls, Singlish, chili crab, and Marty’s special sweet and sour chicken. It immediately hit us that we were no longer in San Francisco.

The Internet never sleeps, which means it is crucial for us to have a presence in Asia to operate our globally distributed network. Singapore was a natural choice for us given the thriving tech community, the business friendliness of the country, the delicious hawker stalls, and our harbor view rooftop hangout:

Since we are new in town, if there are meetups or groups in Singapore that you think we should be part of (or any good restaurants we should try) – let us know. We will be at RSA Asia Pacific & Japan on Friday July 24 here in Singapore. Come meet us in person and learn more about CloudFlare during Nick Sullivan’s session on The New Key Management - Unlocking the Safeguards of Keeping Keys Private.

As one global company, we took team members from both our San Francisco and London offices to be the foundation for the local team. We are actively looking to Continue reading

Fighting Cancer: The Unexpected Benefit Of Open Sourcing Our Code

Recently I was contacted by Dr. Igor Kozin from The Institute of Cancer Research in London. He asked about the optimal way to compile CloudFlare's open source fork of zlib. It turns out that zlib is widely used to compress the SAM/BAM files that are used for DNA sequencing. And it turns out our zlib fork is the best open source solution for that file format.

CC BY-SA 2.0 image by Shaury Nash

The files used for this kind of research reach hundreds of gigabytes and every time they are compressed and decompressed with our library many important seconds are saved, bringing the cure for cancer that much closer. At least that's what I am going to tell myself when I go to bed.

This made me realize that the benefits of open source go much farther than one can imagine, and you never know where a piece of code may end up. Open sourcing makes sophisticated algorithms and software accessible to individuals and organizations that would not have the resources to develop them on their own, or the money pay for a proprietary solution.

It also made me wonder exactly what we did to zlib that makes it Continue reading

Increasing Cache Hit Rates with Query String Sort

Optimized Performance: Increasing Cache Hit Rate

At CloudFlare, we care a lot about serving requests as fast as possible. Files can be served much faster when already in CloudFlare’s cache. Skipping the trip to the customer’s web server eliminates the latency of that connection and saves bandwidth from the connection between CloudFlare and the customer’s origin, and allows us to utilize the full speed of our ultra-fast servers.

By default, CloudFlare only caches static files. However, Page Rules can be utilized to set more files as cacheable. For more information on Page Rules, please see the Page Rules section of our knowledge base.

Items are cached by their full URL, including the query string. However, due to the details of how query strings work, this can lead to some cache misses. There is no RFC which defines that the order of query strings arguments matter, but in some (rare) cases they do. Thus, by default, CloudFlare caches the following two requests separately:

https://example.com/a?color=red&word=hi https://example.com/a?word=hi&color=red

Introducing Query String Sort

With a newly available Enterprise-level feature called Query String Sort, CloudFlare will first sort the query strings in a URL into a deterministic order before checking cache Continue reading

The Internet is a cooperative system: CNAME to Dyn DNS outage of 6 July 2015

Today, shortly after 21:00 UTC, on our internal operations chat there was a scary message from one of our senior support staff: "getting DNS resolution errors on support.cloudflare.com", at the same time as automated monitoring indicated a problem. Shortly thereafter, we saw alarms and feedback from a variety of customers (but not everyone) reporting "1001 errors", which indicated a DNS resolution error on the CloudFlare backend. Needless to say, this got an immediate and overwhelming response from our operations and engineering teams, as we hadn't changed anything and had no other indications of anomaly.

In the course of debugging, we were able to identify common characteristics of affected sites—CNAME-based users of CloudFlare, rather than complete domain hosted entirely on CloudFlare, which, ironically, included our own support site, support.cloudflare.com. When users point (via CNAME) to a domain instead of providing us with an IP address, our network resolves that name —- and is obviously unable to connect if the DNS provider has issues. (Our status page https://www.cloudflarestatus.com/ is off-network and was unaffected). Then, we were investigating why only certain domains were having issues—was the issue with the upstream DNS? Testing whether their domains were resolvable Continue reading

Blue Light Special: Ensuring fast global configuration changes

CloudFlare operates a huge global network of servers that proxy our customers' web sites, operate as caches, inspect requests to ensure they are not malicious, deflect DDoS attacks and handle one of the largest authoritative DNS systems in the world. And where there's software there's configuration information.

CloudFlare is highly customisable. Each customer has a unique configuration consisting of DNS records, all manner of settings (such as minification, image recompression, IP-based blocking, which individual WAF rules to execute) and per-URL rules. And the configuration changes constantly.

Warp speed configuration

We offer almost instant configuration changes. If a user adds a DNS record it should be globally resolvable in seconds. If a user enables a CloudFlare WAF rule it should happen very, very fast to protect a site. This presents a challenge because those configuration changes need to be pushed across the globe very quickly.

We've written in the past about the underlying technology we use: Kyoto Tycoon and how we secured it from eavesdroppers. We also monitor its performance.

DNS records are currently changing at a rate of around 40 per second, 24 hours a day. All those changes need to be propagated in seconds.

So we take propagation times Continue reading

Welcome UK2 Group!

Today we are thrilled to welcome UK2 Group as a CloudFlare partner. Customers of UK2 Group (including its brands UK2.net, Midphase, and Westhost) are now able to access CloudFlare’s web performance and security solutions with a single click. Backed by CloudFlare, UK2 Group’s customers can now protect their websites against security threats, ensure only clean traffic gets served, and speed up site performance no matter where visitors are located. Customers in need of advanced features, and even more performance and security, can sign up for CloudFlare Plus—a plan only offered through our reseller partners.

UK2 Group is one of the innovators of the hosting industry and operates globally. While its name points to its roots (located just down the road from the CloudFlare office in London), it also has an extensive presence in the US. We’re excited to partner with UK2 Group to provide the best web performance and security to its numerous customers.

Click here to learn more.

Setting Go variables from the outside

CloudFlare's DNS server, RRDNS, is written in Go and the DNS team used to generate a file called version.go in our Makefile. version.go looked something like this:

// THIS FILE IS AUTOGENERATED BY THE MAKEFILE. DO NOT EDIT.

// +build    make

package version

var (  
    Version   = "2015.6.2-6-gfd7e2d1-dev"
    BuildTime = "2015-06-16-0431 UTC"
)

and was used to embed version information in RRDNS. It was built inside the Makefile using sed and git describe from a template file. It worked, but was pretty ugly.

Today we noticed that another Go team at CloudFlare, the Data team, had a much smarter way to bake version numbers into binaries using the -X linker option.

The -X Go linker option, which you can set with -ldflags, sets the value of a string variable in the Go program being linked. You use it like this: -X main.version 1.0.0.

A simple example: let's say you have this source file saved as hello.go.

package main

import "fmt"

var who = "World"

func main() {  
    fmt.Printf("Hello, %s.n", who)
}

Then you can use go run (or other build commands like go build or go install Continue reading

How to achieve low latency with 10Gbps Ethernet

Good morning!

In a recent blog post we explained how to tweak a simple UDP application to maximize throughput. This time we are going to optimize our UDP application for latency. Fighting with latency is a great excuse to discuss modern features of multiqueue NICs. Some of the techniques covered here are also discussed in the scaling.txt kernel document.

CC BY-SA 2.0 image by Xiaojun Deng

Our experiment will be setup up as follows:

• We will have two physical Linux hosts: the 'client' and the 'server'. They communicate with a simple UDP echo protocol.
• Client sends a small UDP frame (32 bytes of payload) and waits for the reply, measuring the round trip time (RTT). Server echoes back the packets immediately after they are received.
• Both hosts have 2GHz Xeon CPU's, with two sockets of 6 cores and Hyper Threading (HT) enabled - so 24 CPUs per host.
• The client has a Solarflare 10Gb NIC, the server has an Intel 82599 10Gb NIC. Both cards have fiber connected to a 10Gb switch.
• We're going to measure the round trip time. Since the numbers are pretty small, there is a lot of jitter when counting the averages. Instead, it Continue reading