Dublin, Ireland: CloudFlare’s 38th data center

Top of the morning to our users and readers from Ireland! Our latest PoP in Dublin is our 38th globally, and 14th in Europe following our Bucharest deployment last week. As of yesterday, traffic from Ireland's 3.6 million Internet users will now be routed through Dublin as opposed to our London PoP (which will still serve as a point of redundancy).

Silicon Docks

By now you've heard of Silicon Valley, Silicon Alley, and possibly even Silicon Prairie, but across the pond there's another tech hub making quite a name for itself. Silicon Docks, the Dublin neighborhood bordering the Grand Canal Docks, is home to the European headquarters of Google, Facebook, Twitter, Dropbox, AirBnb, LinkedIn and CloudFlare customer, Yelp, just to name a few. While our own European headquarters is in London, Dublin's exploding tech scene made it an obvious choice for a new PoP.

Dublin is also near to our hearts as the home of CloudFlare customers Web Summit and F.ounders, two of the world's premier tech conferences. Visitors to the 2012 Web Summit and F.ounders events may even remember being greeted Continue reading

Up and to the Right: Forrester Research Ranks CloudFlare as a “Leader” for DDoS Services Providers

Forrester Research, Inc. has released The Forrester Wave™: DDoS Services Providers, Q3 2015 report which ranks CloudFlare as a leader. How do you get placed “up and to the right”? The leaders in this Wave, including CloudFlare, demonstrated effective portals, good client and revenue growth, and a focus on customer service. They also all have the ability to defend against the largest amplification attacks and the most pernicious application attacks.

Here’s some of the criteria CloudFlare received the highest possible scores for:

• Attack types defended
• Data/scrubbing center geographic presence
• Detection tactics
• SSL traffic inspection

The DDoS Services Providers Wave also notes that CloudFlare boasts fast mitigation times, and that our customers gave us high marks for service delivery. The report cited CloudFlare’s excellent capabilities to deliver hybrid DDoS solutions as well.

So how does the report evaluate vendors? It evaluates vendors based on three major categories, each with specific criteria:

• Current offering: The strength of vendors’ current DDoS product offering is based on evaluation categories including: business description, amplification attack defense, attack types defended, customer portal features, customer references, data/scrubbing center geographic presence, SSL traffic inspection, and standard mitigation times.

• Strategy: Vendors’ position on the horizontal axis of Continue reading

A deep look at CVE-2015-5477 and how CloudFlare Virtual DNS customers are protected

Last week ISC published a patch for a critical remotely exploitable vulnerability in the BIND9 DNS server capable of causing a crash with a single packet.

CC BY 2.0 image by Ralph Aversen

The public summary tells us that a mistake in handling of queries for the TKEY type causes an assertion to fail, which in turn crashes the server. Since the assertion happens during the query parsing, there is no way to avoid it: it's the first thing that happens on receiving a packet, before any decision is made about what to do with it.

TKEY queries are used in the context of TSIG, a protocol DNS servers can use to authenticate to each other. They are special in that unlike normal DNS queries they include a “meta” record (of type TKEY) in the EXTRA/ADDITIONAL section of the message.

CC BY 2.0 image by Ralph Aversen

Since the exploit packet is now public, I thought we might take a dive and look at the vulnerable code. Let's start by taking a look at the output of a crashing instance:

03-Aug-2015 16:38:55.509 message.c:2352: REQUIRE(*name == ((void*)0)) failed, back trace  
03-Aug-2015 16:38:55.510 #0 0x10001510d in  Continue reading

Quick and dirty annotations for Go stack traces

CloudFlare’s DNS server, RRDNS, is entirely written in Go and typically runs tens of thousands goroutines. Since goroutines are cheap and Go I/O is blocking we run one goroutine per file descriptor we listen on and queue new packets for processing.

CC BY-SA 2.0 image by wiredforlego

When there are thousands of goroutines running, debug output quickly becomes difficult to interpret. For example, last week I was tracking down a problem with a file descriptor and wanted to know what its listening goroutine was doing. With 40k stack traces, good luck figuring out which one is having trouble.

Go stack traces include parameter values, but most Go types are (or are implemented as) pointers, so what you will see passed to the goroutine function is just a meaningless memory address.

We have a couple options to make sense of the addresses: get a heap dump at the same time as the stack trace and cross-reference the pointers, or have a debug endpoint that prints a goroutine/pointer -> IP map. Neither are seamless.

Underscore to the rescue

However, we know that integers are shown in traces, so what we did is first convert IPv4 addresses to their uint32 Continue reading

Bucharest, Romania: CloudFlare’s 37th data center

Our global expansion continues in Bucharest, Romania, the 6th largest city in the European Union* following London, Berlin, Madrid, Rome, and Paris (nearly all of which feature a CloudFlare PoP!). From Bucharest, our latest data center will serve all 11 million Romanian Internet users, as well as users throughout the Balkans and Eastern Europe.

In good company

Romania is geographically situated between Bulgaria, Hungary, Moldova, Serbia, and Ukraine, making it an ideal destination to attract additional Internet traffic throughout much of Eastern Europe. Of course, geographic reality is rarely a mirror of Internet reality. Adding a new point of presence doesn't automatically mean that traffic from surrounding areas (or even traffic in the very same country) will route to that particular data center. This entirely depends on the interconnection of International carriers with local Internet service providers (ISPs) and large networks like CloudFlare.

It is for this precise reason that we place even more emphasis on our interconnection within a particular PoP as opposed to the absolute number of dots we add to our network map. Of course, the combination of the two (expanding wide and deep) is even better, and is why CloudFlare is blazing fast Continue reading

CloudFlare headed to HostingCon 2015. Thanks for the memories and let’s create some more!

The CloudFlare team is heading to HostingCon 2015 in San Diego next week. We are excited to meet colleagues from the industry, reconnect with partners, and make new friends.

This year’s conference marks a milestone of sorts. It’s our fifth time at HostingCon and we’ve come full circle - our first HostingCon took place in San Diego. Here are some fun facts on what we’ve accomplished since our first HostingCon in 2011:

• 25 new data centers expanded our network to a total of 36 worldwide
• 2M+ customers served
• 800+ conference attendees transported in our signature limo service. If you haven’t already, sign up to arrive in style.
• 2,500+ Nerf guns delivered. Check out the new models this year in celebration of Railgun 5.0 launch
• 3,000+ CloudFlare t-shirts bringing smiles to our partners

Today, CloudFlare is trusted by over 5,000 partners who offer performance and security to millions of customers accelerating and protecting websites, APIs, and mobile apps. We work hard to deliver real savings for our partners. For example, over the past month we saved our partners more than 25 petabytes in aggregate bandwidth (roughly equivalent to 350 hours of HDTV video); stopped 65 billion+ malicious attacks that would Continue reading

Introducing Partner Analytics

CloudFlare has over 5,000 partner hosting providers. Every day, thousands of our partners' customers take advantage of CloudFlare to help them be faster and more secure. The benefits to our partners aren't just happier customers, they also translate into real savings. In the last month, for instance, we saved our partners more than 25 Petabytes in aggregate bandwidth. In addition to bandwidth savings, in that same period, we stopped more than 65 billion malicious requests that would have otherwise impacted our partners' infrastructure. Now we've broken out the bandwidth and performance data by partners so they can see the savings and protection we're delivering.

Back when we launched the CloudFlare Partner Program four years ago, we periodically distributed these figures as high level summaries of bandwidth saved, threats blocked, and number of domains protected and accelerated via each partnership. Our partners knew anecdotally from their own logs and operating expenditures that CloudFlare was reducing their costs and greatly improving their customers’ experiences, but we did not yet have the tools to help demonstrate these benefits on a repeatable and granular basis.

It wasn’t that we didn’t want to provide this data, it was that our tremendous growth rate had stretched Continue reading

Célébrer le 14 Juillet avec Marseille, le 36ème point de présence de CloudFlare

What better day than the 14th of July (Bastille Day) to announce the latest addition to our network in Marseille, France? Our data center in the southern city of Marseille is our 2nd in France, 12th in Europe and 36th globally.

Pourquoi Marseille?

Marseille, France’s second largest city following Paris, is home to 2 million Internet users across the surrounding metropolitan area. It also serves as another point of redundancy to our Paris data center, one of our most trafficked facilities in the whole of Europe.

However, the true importance of Marseille is not just redundancy or its size. Marseille’s southern location makes it a major Internet gateway for networks throughout the Mediterranean, including many African and Middle Eastern countries. This is reflected by the fact that a substantial number of undersea submarine cables carrying Internet traffic are routed through Marseille (7 to be exact, and for those fastidious followers of our blog).

These undersea cables are the principal means by which many countries are able to access the rest of the Internet—that is to say, access all of the other global networks that make up this big Continue reading

CloudFlare Lands a New Office in Singapore

After months of preparation, my teammates Algin, Marty, Adam, Jono and I touched down in Singapore and were greeted by skyscrapers, malls, Singlish, chili crab, and Marty’s special sweet and sour chicken. It immediately hit us that we were no longer in San Francisco.

The Internet never sleeps, which means it is crucial for us to have a presence in Asia to operate our globally distributed network. Singapore was a natural choice for us given the thriving tech community, the business friendliness of the country, the delicious hawker stalls, and our harbor view rooftop hangout:

Since we are new in town, if there are meetups or groups in Singapore that you think we should be part of (or any good restaurants we should try) – let us know. We will be at RSA Asia Pacific & Japan on Friday July 24 here in Singapore. Come meet us in person and learn more about CloudFlare during Nick Sullivan’s session on The New Key Management - Unlocking the Safeguards of Keeping Keys Private.

As one global company, we took team members from both our San Francisco and London offices to be the foundation for the local team. We are actively looking to Continue reading

Fighting Cancer: The Unexpected Benefit Of Open Sourcing Our Code

Recently I was contacted by Dr. Igor Kozin from The Institute of Cancer Research in London. He asked about the optimal way to compile CloudFlare's open source fork of zlib. It turns out that zlib is widely used to compress the SAM/BAM files that are used for DNA sequencing. And it turns out our zlib fork is the best open source solution for that file format.

CC BY-SA 2.0 image by Shaury Nash

The files used for this kind of research reach hundreds of gigabytes and every time they are compressed and decompressed with our library many important seconds are saved, bringing the cure for cancer that much closer. At least that's what I am going to tell myself when I go to bed.

This made me realize that the benefits of open source go much farther than one can imagine, and you never know where a piece of code may end up. Open sourcing makes sophisticated algorithms and software accessible to individuals and organizations that would not have the resources to develop them on their own, or the money pay for a proprietary solution.

It also made me wonder exactly what we did to zlib that makes it Continue reading

Increasing Cache Hit Rates with Query String Sort

Optimized Performance: Increasing Cache Hit Rate

At CloudFlare, we care a lot about serving requests as fast as possible. Files can be served much faster when already in CloudFlare’s cache. Skipping the trip to the customer’s web server eliminates the latency of that connection and saves bandwidth from the connection between CloudFlare and the customer’s origin, and allows us to utilize the full speed of our ultra-fast servers.

By default, CloudFlare only caches static files. However, Page Rules can be utilized to set more files as cacheable. For more information on Page Rules, please see the Page Rules section of our knowledge base.

Items are cached by their full URL, including the query string. However, due to the details of how query strings work, this can lead to some cache misses. There is no RFC which defines that the order of query strings arguments matter, but in some (rare) cases they do. Thus, by default, CloudFlare caches the following two requests separately:

https://example.com/a?color=red&word=hi https://example.com/a?word=hi&color=red

Introducing Query String Sort

With a newly available Enterprise-level feature called Query String Sort, CloudFlare will first sort the query strings in a URL into a deterministic order before checking cache Continue reading

The Internet is a cooperative system: CNAME to Dyn DNS outage of 6 July 2015

Today, shortly after 21:00 UTC, on our internal operations chat there was a scary message from one of our senior support staff: "getting DNS resolution errors on support.cloudflare.com", at the same time as automated monitoring indicated a problem. Shortly thereafter, we saw alarms and feedback from a variety of customers (but not everyone) reporting "1001 errors", which indicated a DNS resolution error on the CloudFlare backend. Needless to say, this got an immediate and overwhelming response from our operations and engineering teams, as we hadn't changed anything and had no other indications of anomaly.

In the course of debugging, we were able to identify common characteristics of affected sites—CNAME-based users of CloudFlare, rather than complete domain hosted entirely on CloudFlare, which, ironically, included our own support site, support.cloudflare.com. When users point (via CNAME) to a domain instead of providing us with an IP address, our network resolves that name —- and is obviously unable to connect if the DNS provider has issues. (Our status page https://www.cloudflarestatus.com/ is off-network and was unaffected). Then, we were investigating why only certain domains were having issues—was the issue with the upstream DNS? Testing whether their domains were resolvable Continue reading

Blue Light Special: Ensuring fast global configuration changes

CloudFlare operates a huge global network of servers that proxy our customers' web sites, operate as caches, inspect requests to ensure they are not malicious, deflect DDoS attacks and handle one of the largest authoritative DNS systems in the world. And where there's software there's configuration information.

CloudFlare is highly customisable. Each customer has a unique configuration consisting of DNS records, all manner of settings (such as minification, image recompression, IP-based blocking, which individual WAF rules to execute) and per-URL rules. And the configuration changes constantly.

Warp speed configuration

We offer almost instant configuration changes. If a user adds a DNS record it should be globally resolvable in seconds. If a user enables a CloudFlare WAF rule it should happen very, very fast to protect a site. This presents a challenge because those configuration changes need to be pushed across the globe very quickly.

We've written in the past about the underlying technology we use: Kyoto Tycoon and how we secured it from eavesdroppers. We also monitor its performance.

DNS records are currently changing at a rate of around 40 per second, 24 hours a day. All those changes need to be propagated in seconds.

So we take propagation times Continue reading

Welcome UK2 Group!

Today we are thrilled to welcome UK2 Group as a CloudFlare partner. Customers of UK2 Group (including its brands UK2.net, Midphase, and Westhost) are now able to access CloudFlare’s web performance and security solutions with a single click. Backed by CloudFlare, UK2 Group’s customers can now protect their websites against security threats, ensure only clean traffic gets served, and speed up site performance no matter where visitors are located. Customers in need of advanced features, and even more performance and security, can sign up for CloudFlare Plus—a plan only offered through our reseller partners.

UK2 Group is one of the innovators of the hosting industry and operates globally. While its name points to its roots (located just down the road from the CloudFlare office in London), it also has an extensive presence in the US. We’re excited to partner with UK2 Group to provide the best web performance and security to its numerous customers.

Click here to learn more.

Setting Go variables from the outside

CloudFlare's DNS server, RRDNS, is written in Go and the DNS team used to generate a file called version.go in our Makefile. version.go looked something like this:

// THIS FILE IS AUTOGENERATED BY THE MAKEFILE. DO NOT EDIT.

// +build    make

package version

var (  
    Version   = "2015.6.2-6-gfd7e2d1-dev"
    BuildTime = "2015-06-16-0431 UTC"
)

and was used to embed version information in RRDNS. It was built inside the Makefile using sed and git describe from a template file. It worked, but was pretty ugly.

Today we noticed that another Go team at CloudFlare, the Data team, had a much smarter way to bake version numbers into binaries using the -X linker option.

The -X Go linker option, which you can set with -ldflags, sets the value of a string variable in the Go program being linked. You use it like this: -X main.version 1.0.0.

A simple example: let's say you have this source file saved as hello.go.

package main

import "fmt"

var who = "World"

func main() {  
    fmt.Printf("Hello, %s.n", who)
}

Then you can use go run (or other build commands like go build or go install Continue reading

How to achieve low latency with 10Gbps Ethernet

Good morning!

In a recent blog post we explained how to tweak a simple UDP application to maximize throughput. This time we are going to optimize our UDP application for latency. Fighting with latency is a great excuse to discuss modern features of multiqueue NICs. Some of the techniques covered here are also discussed in the scaling.txt kernel document.

CC BY-SA 2.0 image by Xiaojun Deng

Our experiment will be setup up as follows:

• We will have two physical Linux hosts: the 'client' and the 'server'. They communicate with a simple UDP echo protocol.
• Client sends a small UDP frame (32 bytes of payload) and waits for the reply, measuring the round trip time (RTT). Server echoes back the packets immediately after they are received.
• Both hosts have 2GHz Xeon CPU's, with two sockets of 6 cores and Hyper Threading (HT) enabled - so 24 CPUs per host.
• The client has a Solarflare 10Gb NIC, the server has an Intel 82599 10Gb NIC. Both cards have fiber connected to a 10Gb switch.
• We're going to measure the round trip time. Since the numbers are pretty small, there is a lot of jitter when counting the averages. Instead, it Continue reading

Check out these brand new videos on how to optimize CloudFlare

Someone once said that the best things in life are free and I can’t agree more. I want to draw the attention of the CloudFlare community to a great resource that helps maximize the value of our product. Troy Hunt, an experienced trainer and blogger, has produced a video course on using CloudFlare. The video series is available through Pluralsight, an online training site for developers.

Because the folks at Pluralsight think that this is a great resource, the video tutorials are being offered to everyone for a week absolutely for free.

So what can you expect to learn? The course kicks off by explaining what CloudFlare brings to the table, and then sets up a site on CloudFlare, including configuring the name server records with your DNS provider. All of this helps get things up and running quickly. Then it gets deeper.

One module of the course is devoted to understanding more about SSL and further strengthening the implementation. For example, CloudFlare’s SSL rates high on the Qualys SSL Labs Test and scores an “A” right out of the box. But you can make it better – an “A+” – just by enabling HSTS. However, you really want to Continue reading

How to build your own public key infrastructure

A major part of securing a network as geographically diverse as CloudFlare’s is protecting data as it travels between datacenters. Customer data and logs are important to protect but so is all the control data that our applications use to communicate with each other. For example, our application servers need to securely communicate with our new datacenter in Osaka, Japan.

CC BY-SA 2.0 image by kris krüg

Great security architecture requires a defense system with multiple layers of protection. As CloudFlare’s services have grown, the need to secure application-to-application communication has grown with it. As a result, we needed a simple and maintainable way to ensure that all communication between CloudFlare’s internal services stay protected, so we built one based on known and reliable protocols.

Our system of trust is based on a Public Key Infrastructure (PKI) using internally-hosted Certificate Authorities (CAs). In this post we will describe how we built our PKI, how we use it internally, and how to run your own with our open source software. This is a long post with lots of information, grab a coffee!

Protection at the application layer

Most reasonably complex modern web services are not made up of one monolithic Continue reading

Osaka, Japan: CloudFlare’s 35th data center

Move over Jurassic World, the long awaited sequel to our Tokyo deployment is here. Our Osaka data center is our 2nd in Japan, 5th in Asia (following deployments in Hong Kong, Tokyo, Singapore, and Seoul), and 35th globally. This latest deployment serves not only Osaka, Japan's second largest city, but also Nagoya, the 3rd largest, and the entire Keihanshin metropolitan area including Kyoto and Kobe. This means faster application delivery to the area's 30 million inhabitants, and full redundancy of traffic with our Tokyo facility. CloudFlare is now mere milliseconds away from all 110 million Internet users in Japan.

The Internet in Asia

Even though Asia is home to many of the most technologically advanced nations in the world, the delivery of Internet traffic across the region is hardly seamless. Many of the incumbent telecommunications providers in the region (e.g. NTT, Tata, PCCW, Hinet, Singtel, among many others) do not interconnect with one another locally. This is another way of saying that traffic is routed poorly in the region. Traffic sent from one network in Japan, for example, may have to pass through an entirely different country or, in some instances, even the United States, Continue reading

EFF, CloudFlare Ask Federal Court Not To Force Internet Companies To Enforce Music Labels’ Trademarks

This blog was originally posted by the Electronic Frontier Foundation who is represents CloudFlare in this case.

JUNE 18, 2015 | BY MITCH STOLTZ

This month, CloudFlare and EFF pushed back against major music labels’ latest strategy to force Internet infrastructure companies like CloudFlare to become trademark and copyright enforcers, by challenging a broad court order that the labels obtained in secret. Unfortunately, the court denied CloudFlare’s challenge and ruled that the secretly-obtained order applied to CloudFlare. This decision, and the strategy that led to it, present a serious problem for Internet infrastructure companies of all sorts, and for Internet users, because they lay out a blueprint for quick, easy, potentially long-lasting censorship of expressive websites with little or no court review. The fight’s not over for CloudFlare, though. Yesterday, CloudFlare filed a motion with the federal court in Manhattan, asking Judge Alison J. Nathan to modify the order and put the responsibility of identifying infringing domain names back on the music labels.

We’ve reported recently about major entertainment companies’ quest to make websites disappear from the Internet at their say-so. The Internet blacklist bills SOPA and PIPA were part of that strategy, along with the Department of Homeland Security’s Continue reading