Fighting Cancer: The Unexpected Benefit Of Open Sourcing Our Code
Recently I was contacted by Dr. Igor Kozin from The Institute of Cancer Research in London. He asked about the optimal way to compile CloudFlare's open source fork of zlib. It turns out that zlib is widely used to compress the SAM/BAM files that are used for DNA sequencing. And it turns out our zlib fork is the best open source solution for that file format.
CC BY-SA 2.0 image by Shaury Nash
The files used for this kind of research reach hundreds of gigabytes and every time they are compressed and decompressed with our library many important seconds are saved, bringing the cure for cancer that much closer. At least that's what I am going to tell myself when I go to bed.
This made me realize that the benefits of open source go much farther than one can imagine, and you never know where a piece of code may end up. Open sourcing makes sophisticated algorithms and software accessible to individuals and organizations that would not have the resources to develop them on their own, or the money pay for a proprietary solution.
It also made me wonder exactly what we did to zlib that makes it Continue reading
Increasing Cache Hit Rates with Query String Sort
Optimized Performance: Increasing Cache Hit Rate
At CloudFlare, we care a lot about serving requests as fast as possible. Files can be served much faster when already in CloudFlare’s cache. Skipping the trip to the customer’s web server eliminates the latency of that connection and saves bandwidth from the connection between CloudFlare and the customer’s origin, and allows us to utilize the full speed of our ultra-fast servers.
By default, CloudFlare only caches static files. However, Page Rules can be utilized to set more files as cacheable. For more information on Page Rules, please see the Page Rules section of our knowledge base.
Items are cached by their full URL, including the query string. However, due to the details of how query strings work, this can lead to some cache misses. There is no RFC which defines that the order of query strings arguments matter, but in some (rare) cases they do. Thus, by default, CloudFlare caches the following two requests separately:
https://example.com/a?color=red&word=hi
https://example.com/a?word=hi&color=red
Introducing Query String Sort
With a newly available Enterprise-level feature called Query String Sort, CloudFlare will first sort the query strings in a URL into a deterministic order before checking cache Continue reading
The Internet is a cooperative system: CNAME to Dyn DNS outage of 6 July 2015
Today, shortly after 21:00 UTC, on our internal operations chat there was a scary message from one of our senior support staff: "getting DNS resolution errors on support.cloudflare.com", at the same time as automated monitoring indicated a problem. Shortly thereafter, we saw alarms and feedback from a variety of customers (but not everyone) reporting "1001 errors", which indicated a DNS resolution error on the CloudFlare backend. Needless to say, this got an immediate and overwhelming response from our operations and engineering teams, as we hadn't changed anything and had no other indications of anomaly.
In the course of debugging, we were able to identify common characteristics of affected sites—CNAME-based users of CloudFlare, rather than complete domain hosted entirely on CloudFlare, which, ironically, included our own support site, support.cloudflare.com. When users point (via CNAME) to a domain instead of providing us with an IP address, our network resolves that name —- and is obviously unable to connect if the DNS provider has issues. (Our status page https://www.cloudflarestatus.com/ is off-network and was unaffected). Then, we were investigating why only certain domains were having issues—was the issue with the upstream DNS? Testing whether their domains were resolvable Continue reading
Blue Light Special: Ensuring fast global configuration changes
CloudFlare operates a huge global network of servers that proxy our customers' web sites, operate as caches, inspect requests to ensure they are not malicious, deflect DDoS attacks and handle one of the largest authoritative DNS systems in the world. And where there's software there's configuration information.
CloudFlare is highly customisable. Each customer has a unique configuration consisting of DNS records, all manner of settings (such as minification, image recompression, IP-based blocking, which individual WAF rules to execute) and per-URL rules. And the configuration changes constantly.
Warp speed configuration
We offer almost instant configuration changes. If a user adds a DNS record it should be globally resolvable in seconds. If a user enables a CloudFlare WAF rule it should happen very, very fast to protect a site. This presents a challenge because those configuration changes need to be pushed across the globe very quickly.
We've written in the past about the underlying technology we use: Kyoto Tycoon and how we secured it from eavesdroppers. We also monitor its performance.
DNS records are currently changing at a rate of around 40 per second, 24 hours a day. All those changes need to be propagated in seconds.
So we take propagation times Continue reading
Welcome UK2 Group!
Today we are thrilled to welcome UK2 Group as a CloudFlare partner. Customers of UK2 Group (including its brands UK2.net, Midphase, and Westhost) are now able to access CloudFlare’s web performance and security solutions with a single click. Backed by CloudFlare, UK2 Group’s customers can now protect their websites against security threats, ensure only clean traffic gets served, and speed up site performance no matter where visitors are located. Customers in need of advanced features, and even more performance and security, can sign up for CloudFlare Plus—a plan only offered through our reseller partners.
UK2 Group is one of the innovators of the hosting industry and operates globally. While its name points to its roots (located just down the road from the CloudFlare office in London), it also has an extensive presence in the US. We’re excited to partner with UK2 Group to provide the best web performance and security to its numerous customers.
Setting Go variables from the outside
CloudFlare's DNS server, RRDNS, is written in Go and the DNS team used to generate a file called version.go in our Makefile. version.go looked something like this:
// THIS FILE IS AUTOGENERATED BY THE MAKEFILE. DO NOT EDIT.
// +build make
package version
var (
Version = "2015.6.2-6-gfd7e2d1-dev"
BuildTime = "2015-06-16-0431 UTC"
)
and was used to embed version information in RRDNS. It was built inside the Makefile using sed and git describe from a template file. It worked, but was pretty ugly.
Today we noticed that another Go team at CloudFlare, the Data team, had a much smarter way to bake version numbers into binaries using the -X linker option.
The -X Go linker option, which you can set with -ldflags, sets the value of a string variable in the Go program being linked. You use it like this: -X main.version 1.0.0.
A simple example: let's say you have this source file saved as hello.go.
package main
import "fmt"
var who = "World"
func main() {
fmt.Printf("Hello, %s.n", who)
}
Then you can use go run (or other build commands like go build or go install Continue reading
How to achieve low latency with 10Gbps Ethernet
Good morning!
In a recent blog post we explained how to tweak a simple UDP application to maximize throughput. This time we are going to optimize our UDP application for latency. Fighting with latency is a great excuse to discuss modern features of multiqueue NICs. Some of the techniques covered here are also discussed in the scaling.txt kernel document.
CC BY-SA 2.0 image by Xiaojun Deng
Our experiment will be setup up as follows:
- We will have two physical Linux hosts: the 'client' and the 'server'. They communicate with a simple UDP echo protocol.
- Client sends a small UDP frame (32 bytes of payload) and waits for the reply, measuring the round trip time (RTT). Server echoes back the packets immediately after they are received.
- Both hosts have 2GHz Xeon CPU's, with two sockets of 6 cores and Hyper Threading (HT) enabled - so 24 CPUs per host.
- The client has a Solarflare 10Gb NIC, the server has an Intel 82599 10Gb NIC. Both cards have fiber connected to a 10Gb switch.
- We're going to measure the round trip time. Since the numbers are pretty small, there is a lot of jitter when counting the averages. Instead, it Continue reading
Check out these brand new videos on how to optimize CloudFlare
Someone once said that the best things in life are free and I can’t agree more. I want to draw the attention of the CloudFlare community to a great resource that helps maximize the value of our product. Troy Hunt, an experienced trainer and blogger, has produced a video course on using CloudFlare. The video series is available through Pluralsight, an online training site for developers.
Because the folks at Pluralsight think that this is a great resource, the video tutorials are being offered to everyone for a week absolutely for free.
So what can you expect to learn? The course kicks off by explaining what CloudFlare brings to the table, and then sets up a site on CloudFlare, including configuring the name server records with your DNS provider. All of this helps get things up and running quickly. Then it gets deeper.
One module of the course is devoted to understanding more about SSL and further strengthening the implementation. For example, CloudFlare’s SSL rates high on the Qualys SSL Labs Test and scores an “A” right out of the box. But you can make it better – an “A+” – just by enabling HSTS. However, you really want to Continue reading
How to build your own public key infrastructure
A major part of securing a network as geographically diverse as CloudFlare’s is protecting data as it travels between datacenters. Customer data and logs are important to protect but so is all the control data that our applications use to communicate with each other. For example, our application servers need to securely communicate with our new datacenter in Osaka, Japan.
CC BY-SA 2.0 image by kris krüg
Great security architecture requires a defense system with multiple layers of protection. As CloudFlare’s services have grown, the need to secure application-to-application communication has grown with it. As a result, we needed a simple and maintainable way to ensure that all communication between CloudFlare’s internal services stay protected, so we built one based on known and reliable protocols.
Our system of trust is based on a Public Key Infrastructure (PKI) using internally-hosted Certificate Authorities (CAs). In this post we will describe how we built our PKI, how we use it internally, and how to run your own with our open source software. This is a long post with lots of information, grab a coffee!
Protection at the application layer
Most reasonably complex modern web services are not made up of one monolithic Continue reading
Osaka, Japan: CloudFlare’s 35th data center
Move over Jurassic World, the long awaited sequel to our Tokyo deployment is here. Our Osaka data center is our 2nd in Japan, 5th in Asia (following deployments in Hong Kong, Tokyo, Singapore, and Seoul), and 35th globally. This latest deployment serves not only Osaka, Japan's second largest city, but also Nagoya, the 3rd largest, and the entire Keihanshin metropolitan area including Kyoto and Kobe. This means faster application delivery to the area's 30 million inhabitants, and full redundancy of traffic with our Tokyo facility. CloudFlare is now mere milliseconds away from all 110 million Internet users in Japan.
The Internet in Asia
Even though Asia is home to many of the most technologically advanced nations in the world, the delivery of Internet traffic across the region is hardly seamless. Many of the incumbent telecommunications providers in the region (e.g. NTT, Tata, PCCW, Hinet, Singtel, among many others) do not interconnect with one another locally. This is another way of saying that traffic is routed poorly in the region. Traffic sent from one network in Japan, for example, may have to pass through an entirely different country or, in some instances, even the United States, Continue reading
EFF, CloudFlare Ask Federal Court Not To Force Internet Companies To Enforce Music Labels’ Trademarks
This blog was originally posted by the Electronic Frontier Foundation who is represents CloudFlare in this case.
JUNE 18, 2015 | BY MITCH STOLTZ
This month, CloudFlare and EFF pushed back against major music labels’ latest strategy to force Internet infrastructure companies like CloudFlare to become trademark and copyright enforcers, by challenging a broad court order that the labels obtained in secret. Unfortunately, the court denied CloudFlare’s challenge and ruled that the secretly-obtained order applied to CloudFlare. This decision, and the strategy that led to it, present a serious problem for Internet infrastructure companies of all sorts, and for Internet users, because they lay out a blueprint for quick, easy, potentially long-lasting censorship of expressive websites with little or no court review. The fight’s not over for CloudFlare, though. Yesterday, CloudFlare filed a motion with the federal court in Manhattan, asking Judge Alison J. Nathan to modify the order and put the responsibility of identifying infringing domain names back on the music labels.
We’ve reported recently about major entertainment companies’ quest to make websites disappear from the Internet at their say-so. The Internet blacklist bills SOPA and PIPA were part of that strategy, along with the Department of Homeland Security’s Continue reading
Go has a debugger—and it’s awesome!
Something that often, uh... bugs1 Go developers is the lack of a proper debugger. Sure, builds are ridiculously fast and easy, and println(hex.Dump(b)) is your friend, but sometimes it would be nice to just set a breakpoint and step through that endless if chain or print a bunch of values without recompiling ten times.
CC BY 2.0 image by Carl Milner
You could try to use some dirty gdb hacks that will work if you built your binary with a certain linker and ran it on some architectures when the moon was in a waxing crescent phase, but let's be honest, it isn't an enjoyable experience.
Well, worry no more! godebug is here!
godebug is an awesome cross-platform debugger created by the Mailgun team. You can read their introduction for some under-the-hood details, but here's the cool bit: instead of wrestling with half a dozen different ptrace interfaces that would not be portable, godebug rewrites your source code and injects function calls like godebug.Line on every line, godebug.Declare at every variable declaration, and godebug.SetTrace for breakpoints (i.e. wherever you type _ = "breakpoint").
I find this solution brilliant. What you get out Continue reading
How to receive a million packets per second
Last week during a casual conversation I overheard a colleague saying: "The Linux network stack is slow! You can't expect it to do more than 50 thousand packets per second per core!"
That got me thinking. While I agree that 50kpps per core is probably the limit for any practical application, what is the Linux networking stack capable of? Let's rephrase that to make it more fun:
On Linux, how hard is it to write a program that receives 1 million UDP packets per second?
Hopefully, answering this question will be a good lesson about the design of a modern networking stack.
CC BY-SA 2.0 image by Bob McCaffrey
First, let us assume:
Measuring packets per second (pps) is much more interesting than measuring bytes per second (Bps). You can achieve high Bps by better pipelining and sending longer packets. Improving pps is much harder.
Since we're interested in pps, our experiments will use short UDP messages. To be precise: 32 bytes of UDP payload. That means 74 bytes on the Ethernet layer.
For the experiments we will use two physical servers: "receiver" and "sender".
They both have two six core 2GHz Xeon processors. With hyperthreading (HT) enabled Continue reading
iOS Developers — Migrate to iOS 9 with CloudFlare
Thousands of developers use CloudFlare to accelerate and secure the backend of their mobile applications and websites. This week is Apple’s Worldwide Developers Conference (WWDC), where thousands of Apple developers come to San Francisco to talk, learn and share best practices for developing software for Apple platforms. New announcements from Apple this week make CloudFlare an even more obvious choice for application developers.
New operating systems, new application requirements
The flagship announcement of WWDC 2015 was a new version of Apple’s mobile operating system, iOS 9, to be released in September with a developer preview available now. They also announced a new Mac operating system, OS X El Capitan, launching in the fall. Apple has a track record of developing and supporting technologies that enhance user privacy and security with iMessage and Facetime and the trend is continuing with these new operating systems. In both cases, Apple is requiring application developers to make use of two network technologies that CloudFlare is big fan of: HTTPS and IPv6.
For iOS 9 and El Capitan, all applications submitted to the iOS and Mac App Stores must work over IPv6. In previous versions, applications were allowed that only worked with IPv4.
From Continue reading
Four years later and CloudFlare is still doing IPv6 automatically
Over the past four years CloudFlare has helped well over two million websites join the modern web, making us one of the fastest growing providers of IPv6 web connectivity on the Internet. CloudFlare's Automatic IPv6 Gateway allows IPv4-only websites to support IPv6-only clients with zero clicks. No hardware. No software. No code changes. And no need to change your hosting provider.
A Four Year Story
The story of IPv6 support for customers of CloudFlare is about as long as the story of CloudFlare itself. June 6th, 2011 (four years ago) was the original World IPv6 Day, and CloudFlare participated. Each year since, the global Internet community has pushed forward with additional IPv6 deployment. Now, four years later, CloudFlare is celebrating June 6th knowing that our customers are being provided with a solid IPv6 offering that requires zero configuration to enable. CloudFlare is the only global CDN that provides IPv4/IPv6 delivery of content by default and at scale.
IPv6 has been featured in our blog various times over the last four years. We have provided support for legacy logging systems to handle IPv6 addresses, provided DDoS protection on IPv6 alongside classic IPv4 address space, and provided Continue reading
Welcome Acquia!
We’ve had the good fortune to share many great experiences with the Acquia team over the last few years. From breaking bread with founder and CTO Dries Buytaert at SXSW, to skiing the slopes of Park City with the company’s CEO Tom Erickson, to staying up late with their incredible team onboarding a joint customer under a DDoS attack. It’s always a pleasure to spend time with the Acquia team.
Today we are thrilled to welcome Acquia as a CloudFlare Partner. Together we developed Acquia Cloud Edge powered by CloudFlare making it easier for any of their customers to access CloudFlare’s web performance and security solutions. The Acquia Cloud Edge is a family of products that protects websites against security threats, ensures only clean traffic get served, and speeds up site performance no matter where visitors are located.
Acquia Cloud Edge powered by CloudFlare comes as Edge Protect and Edge CDN. Edge Protect defends against DDoS and other network-level attacks. CloudFlare sits on the network edge in front of Acquia web servers, allowing early identification of attack patterns and questionable visitors, and mitigating attacks before they reach a user’s site. Edge CDN accelerates the delivery of digital experiences through CloudFlare’s Continue reading
Logjam: the latest TLS vulnerability explained
Yesterday, a group from INRIA, Microsoft Research, Johns Hopkins, the University of Michigan, and the University of Pennsylvania published a deep analysis of the Diffie-Hellman algorithm as used in TLS and other protocols. This analysis included a novel downgrade attack against the TLS protocol itself called Logjam, which exploits EXPORT cryptography (just like FREAK).
First, let me start by saying that CloudFlare customers are not and were never affected. We don’t support non-EC Diffie-Hellman ciphersuites on either the client or origin side. We also won't touch EXPORT-grade cryptography with a 20ft stick.
But why are CloudFlare customers safe, and how does Logjam work anyway?
The image is "Logjam" as interpreted by @0xabad1dea.
Diffie-Hellman and TLS
This is a detailed technical introduction to how DH works and how it’s used in TLS—if you already know this and want to read about the attack, skip to “Enter export crypto, enter Logjam” below. If, instead, you are not interested in the nuts and bolts and want to know who’s at risk, skip to “So, what’s affected?”
To start a TLS connection, the two sides—client (the browser) and server (CloudFlare)—need to agree securely on a secret key. This process is called Continue reading
CloudFlare Supports the Passage of the USA Freedom Act
Earlier today, the lower house in the U.S. Congress (the House of Representatives) passed the USA FREEDOM Act. The Act, if passed by the Senate and signed by the President, would seek to sunset the National Security Agency’s bulk collection and mass surveillance programs, which may or may not be authorized by Section 215 of the PATRIOT Act. Under this authority the U.S. government has established its broad surveillance programs to indiscriminately collect information. Other governments have followed this lead to create additional surveillance capabilities—most recently, the French Parliament has moved a bill that would allow broad surveillance powers with little judicial oversight.
Restricting routine bulk collection is important: it’s not the government’s job to collect everything that passes over the Internet. The new version of the USA FREEDOM Act keeps useful authorities but ends bulk collection of private data under the PATRIOT Act. It also increases the transparency of the secret FISA court, which reviews surveillance programs—a key start to understanding and fixing broken policies around surveillance. The Act would also allow companies to be more transparent in their reporting related to FISA orders.
To be clear, we continue to be supportive of law enforcement and work Continue reading
CloudFlare “Interview Questions”
For quite some time we've been grilling our candidates about dirty corners of TCP/IP stack. Every engineer here must prove his/her comprehensive understanding of the full network stack. For example: what are the differences in checksumming algorithms between IPv4 and IPv6 stacks?
I'm joking of course, but in the spirit of the old TCP/IP pub game I want to share some of the amusing TCP/IP quirks I've bumped into over the last few months while working on CloudFlare's automatic attack mitigation systems.
CC BY-SA 2.0 image by Daan Berg
Don't worry if you don't know the correct answer: you may always come up with a funny one!
Some of the questions are fairly obvious, some don't have a direct answer and are supposed to provoke a longer discussion. The goal is to encourage our readers to review the dusty RFCs, get interested in the inner workings of the network stack and generally spread the knowledge about the protocols we rely on so much.
Don't forget to add a comment below if you want to share a response!
You think you know all about TCP/IP? Let's find out.
Archaeology
1) What is the lowest TCP port number?
2) The TCP Continue reading
Google PageSpeed Service customers: migrate to CloudFlare for acceleration
This week, Google announced that its hosted PageSpeed Service will be shut down. Everyone using the hosted service needs to move their site elsewhere before August 3 2015 to avoid breaking their website.
We're inviting these hosted customers: don't wait...migrate your site to CloudFlare for global acceleration (and more) right now.
CC BY 2.0 image by Roger
As TechCrunch wrote: "In many ways, PageSpeed Service is similar to what CloudFlare does but without the focus on security."
What is PageSpeed?
PageSpeed started as — and continues — as a Google-created software module for the Apache webserver to rewrite webpages to reduce latency and bandwidth, to help make the web faster.
Google introduced their hosted PageSpeed Service in July 2011, to save webmasters the hassle of installing the software module.
It's the hosted service that is being discontinued.
CloudFlare performance
CloudFlare provides similar capabilities to PageSpeed, such as minification, image compression, and asynchronous loading.
Additionally, CloudFlare offers more performance gains through a global network footprint, Railgun for dynamic content acceleration, built-in SPDY support, and more.
Not just speed
PageSpeed Service customers care about performance, and CloudFlare delivers. CloudFlare also includes security, SSL, Continue reading