Change the (S)Channel! Deconstructing the Microsoft TLS Session Resumption bug

Initial Problem Report

Several months ago we started hearing occasional reports from .NET developers that they were having trouble maintaining HTTPS sessions with one of our customer’s websites. Establishing connections worked just fine but they would periodically get disconnected, resulting in an exception that crashed their application. Around the same time, we also started hearing reports that two other Microsoft products—Internet Explorer and its heir-apparent, Edge—were also having trouble with our edge.

Just a few weeks prior, we had updated our handling of TLS session tickets to be more performant and more secure. We suspected these improvements were the trigger and focused our investigation there. What we learned was that the problem ran much deeper than .NET or IE. It went all the way down to the SChannel security package, which provides TLS functionality for a vast array of Microsoft applications.

TLS Session Tickets

Before diving into the story further, however, it’s helpful to understand exactly what TLS session tickets are, how they’re beneficial to HTTPS, and what optimizations CloudFlare does to use them at scale. (If you’d like to skip over the primer and jump right to the punchline, go ahead and click here.)

Overview

First introduced in Continue reading

CloudFlare’s Impact On The HTTP/2 “Universe”

CloudFlare released HTTP/2 support for all customers on December 3rd, 2015. Now, two months later, it's time to take a look at the impact of this release on the HTTP/2 "universe" and also at what has changed from a HTTP/2 vs. SPDY vs. HTTP 1.1 traffic ratio perspective.

HTTP/2 Usage

Previously, we showcased browser market share data from our own website. Using these numbers, we predicted the ratio of HTTP/2 traffic that we expected to see once enabled. Now, we can compare this original data set with updated data from the last 48 hours.

Below is the market share of HTTP/2 capable browsers that we saw on our website during a 48 hour period. The first one was before our HTTP/2 launch, the other one was last week. Both data sets were pulled from Google Analytics, and user agents were analyzed for HTTP/2 support.

Advanced Technical “Hacks” for your site’s SEO

Improving your site’s SEO is probably top of mind for you, but doing so takes a lot of hard work and the rules of the game are constantly changing. On Tuesday, January 26th at 10am PT/1pm ET, CloudFlare is hosting a live discussion with some of the leading experts in technical SEO. They will share advanced technical hacks to help you reap the benefits of higher search rankings. In the live discussion, Martin Woods, Reza Moaiandin, and Patrick Stox will cover:

• Tangible tips about on-page code excellency
• Semantic markup
• Web server optimization with GZIP and HTTP/2
• Web content optimization
• Site security with malware and DDoS prevention

In addition to the webinar, Reza and Martin from SALT.agency have offered a free 30 minute technical SEO consult on your website. Consults are limited to the first 50 people who signup here and also attend the live webinar event on January 26th at 10am PT. Be sure to register for the webinar, too.

CloudFlare launches new data centers in Oslo and Minneapolis

Four thousand miles (6,400 kilometers) separate CloudFlare’s latest two data centers: Oslo (#75) and Minneapolis (#76).

Oslo

In Oslo, we have now built our third data center in Scandinavia. This joins our existing facilities in Stockholm and Copenhagen. With a data center in Norway, we recognize an important country that stands above others with a staggering 95.05% of the population having Internet connectivity. This Internet penetration rate is the fourth best in the world. For reference, the Internet penetration rate in the US is 84%, the UK is 90% and Egypt, where we deployed our last data center it is only 50%

At 59.9500° N, Oslo is also the “northernmost” CloudFlare data center on our network map.

Oslo, according to the Norwegian Sagas is over 1,000 years old. CloudFlare has built itself into a facility just a handful of years old and while we respect all the wonderful history and tradition associated with Norway, we hope the locals appreciate our 21st century choice.

Norway has a very important position within the history of the Internet (well the ARPANET actually). In June 1973, the Royal Radar Establishment in Norway became one of the first international connections to Continue reading

Preventing Malicious Request Loops

The web is an collaborative ecosystem. Web standards exist to ensure that participants of the network behave in a predictable way. If network participants deviate from the established standards then there can be unintended consequences. This blog post is about one of these unintended consequences.

A group of researchers recently published a paper "Forwarding Loop Attacks in the Content Delivery Networks" describing what can happen when web services interact in a non-compliant way. They describe an attack where a malicious user can force multiple service providers to send each other an unending stream of requests in a loop. This request loop can result in resource exhaustion and denial of service at the service provider. This paper also demonstrated that the attack is practical, and can be performed using a large list of service providers.

CloudFlare's service has been modified to be standards-compliant with respect to HTTP proxying. However, fixing the vulnerability that enables this attack requires all proxy services to conform to the same standards. If even one service provider is non-compliant, the attack can still be carried out against compliant services. In this post, we will describe the attack and explain how a proxy services can go from being Continue reading

We’re Hosting a Go Hackathon!

CloudFlare is excited to partner with Women Who Go to host Gopher Gala—the first distributed Go(lang) hackathon—in our San Francisco office!

Gopher Gala is a chance to showcase your skills and compete against the best Go developers from around the world.

While the hackathon is distributed globally, CloudFlare is welcoming teams to use our new office space in SOMA this Saturday and Sunday from 9am-5pm. There will be food, drinks, and plenty of space to spread out and work with your teammates. Some of CloudFlare’s top Go developers will be participating as well.

If you’d like to sign up for the event, you can do so here: http://www.meetup.com/Women-Who-Go/events/227017435/

So, come join Women Who Go and CloudFlare and build something in a weekend:

When January 23rd: 9am-5pm
January 24th: 9am-5pm

Where CloudFlare Headquarters
101 Townsend Street
San Francisco, CA 94107

(Registration is required)

Go coverage with external tests

The Go test coverage implementation is quite ingenious: when asked to, the Go compiler will preprocess the source so that when each code portion is executed a bit is set in a coverage bitmap. This is integrated in the go test tool: go test -cover enables it and -coverprofile= allows you to write a profile to then inspect with go tool cover.

This makes it very easy to get unit test coverage, but there's no simple way to get coverage data for tests that you run against the main version of your program, like end-to-end tests.

The proper fix would involve adding -cover preprocessing support to go build, and exposing the coverage profile maybe as a runtime/pprof.Profile, but as of Go 1.6 there’s no such support. Here instead is a hack we've been using for a while in the test suite of RRDNS, our custom Go DNS server.

We create a dummy test that executes main(), we put it behind a build tag, compile a binary with go test -c -cover and then run only that test instead of running the regular binary.

Here's what the rrdns_test.go file looks like:

// +build  Continue reading

Think Global, Peer Local. Peer with CloudFlare at 100 Internet Exchange Points

Internet Exchange Points (IXPs) or Network Access Points (NAPs) facilities are where networks meet, participating in what's known as peering, which interconnects various parts of the global Internet.

At CloudFlare we are dedicated to peering. So much so that we just joined our 100th Internet Exchange point!

Image courtesy of Martin Levy

What is peering?

According to Wikipedia:

“In computer networking, peering is a voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the users of each network”

In reality this normally means a physical place where two different networks (they could be backbones, CDNs, mobile networks or broadband ISPs) connect their respective networks together to exchange traffic. Over the last fifteen years, there has been a major expansion in network interconnections, running parallel to the enormous expansion of the global Internet. This expansion includes new data centre facilities being developed to house network equipment. Some of those data centres have attracted massive numbers of networks, in no small part due to the thriving Internet Exchanges Points (both new and existing) that operate within them. London with the LINX and LONAP exchanges, Amsterdam with AMS-IX and NL-IX exchanges, Frankfurt with DE-CIX and ECIX exchanges Continue reading

Flexible, secure SSH with DNSSEC

If you read this blog on a regular basis, you probably use the little tool called SSH, especially its ubiquitous and most popular implementation OpenSSH.

Maybe you’re savvy enough to only use it with public/private keys, and therefore protect yourself from dictionary attacks. If you do then you know that in order to configure access to a new host, you need to make a copy of a public key available to that host (usually by writing it to its disk). Managing keys can be painful if you have many hosts, especially when you need to renew one of the keys. What if DNSSEC could help?

CC BY 2.0 image by William Neuheisel

With version 6.2 of OpenSSH came a feature that allows the remote host to retrieve a public key in a customised way, instead of the typical authorized_keys file in the ~/.ssh/ directory. For example, you can gather the keys of a group of users that require access to a number of machines on a single server (for example, an LDAP server), and have all the hosts query that server when they need the public key of the user attempting to log in. This saves Continue reading

Cairo, Egypt: CloudFlare’s 74th Data Center

It’s been a big year of expansion for CloudFlare’s global network as we added new data centers across six continents, and we’re certainly not done. Today we announce the launch of our newest data center in Cairo, Egypt and a partnership with Telecom Egypt. This marks our third data center in Africa, after Johannesburg and Mombasa, and our 74th data center globally.

Faster performance across Egypt

For many years, CloudFlare has been trusted by Egyptian websites to be protected from attacks.

Over half of the 20 most popular websites in Egypt already use CloudFlare to be safe, and are now seeing a 2x improvement in performance.

Reduced latency to Egypt's largest network, Telecom Egypt

Local Deployments

Just like in Egypt, we partner with ISPs globally by deploying caches directly into their facilities. These points of presence help major networks improve the performance of millions of websites, reduce their costs and capacity used in accessing our customers' content, and provide a direct local interconnect with critical Internet infrastructure. If you are a carrier or Internet service provider in Egypt, elsewhere in Africa or anywhere around the world that would like to request a CloudFlare cache deployment, please reach out to Continue reading

How to Talk to Your Parents About Encryption

It’s December 25th, which means most of you are probably at home visiting with family. I asked a few of the security engineers here at CloudFlare how they explain their jobs when they’re home for the holidays, and most of them responded with something along the lines of, "Oh, I stopped trying to do that a long time ago." Apparently, working in the cryptography field doesn’t exactly make it easy to talk about work with your parents.

After chatting with our crypto experts some more, we figured out a decent way to explain the general idea of encryption and why it’s a critical part of the Internet. While this post may not explain exactly what security engineers do on a day-to-day basis, hopefully it will help you at least tell your parents why you have a job in the first place.

Banks and Their Big Fancy Buildings

To explain encryption to your parents, I’d start by asking them why they trust their bank. Let’s say they have some cash to deposit. They drive to their bank’s local branch, walk through a big fancy lobby, wait in line for a teller, and hand them their money. It may seem like Continue reading

Zdrasti, Sofia! CloudFlare’s 73rd Data Center Now Live

Only days after the launch of our Hamburg data center, CloudFlare is excited to announce yet another European data center - this time in Sofia, Bulgaria. With over 1.2 million people, Sofia is a city with rich history tracing back over 7,000 years.

We were fascinated to note the coincidence that even as 1 in 73 of CloudFlare team members is Bulgarian, now 1 in 73 of CloudFlare data centers is in Bulgaria!

Localizing European traffic

Sofia expands the CloudFlare global network to span 20 European data centers - joining Amsterdam, Frankfurt, Paris, London, Vienna, Prague, Stockholm, Warsaw, Madrid, Milan, Dusseldorf, Marseille, Bucharest, Dublin, Manchester, Zurich, Copenhagen, Berlin and Hamburg.

Each time we launch a new data center, we improve the performance of millions of websites, expand the surface area available to fight attacks, and provide an additional point of redundancy to support our existing data centers.

Until today, many Bulgarian networks were served out of Frankfurt, over 1,000 miles away, based on their interconnection there with our tier one providers. Our newest deployment eliminates that distance, and improves the web Continue reading

Why it’s harder to forge a SHA-1 certificate than it is to find a SHA-1 collision

It’s well known that SHA-1 is no longer considered a secure cryptographic hash function. Researchers now believe that finding a hash collision (two values that result in the same value when SHA-1 is applied) is inevitable and likely to happen in a matter of months. This poses a potential threat to trust on the web, as many websites use certificates that are digitally signed with algorithms that rely on SHA-1. Luckily for everyone, finding a hash collision is not enough to forge a digital certificate and break the trust model of the Internet.

We’ll explore how hash collisions have been used to forge digital signatures in the past. We’ll also discuss how certificate authorities can make this significantly harder for attackers in the future by including randomness in certificate serial numbers.

Digital signatures are the bedrock of trust

The Internet relies on trust. Whether it’s logging in to your bank or reading Reddit, HTTPS protects you by encrypting the data you exchange with a site and authenticating the site's identity with a digital certificate. Browsers visually display the added security of HTTPS as a padlock in the address bar.

HTTPS can prove a site’s authenticity to a browser when a Continue reading

What’s inside net/http? Late binding in the Go standard library

It's well known that we're heavy users of the Go programming language at CloudFlare. Our work often involves delving into the standard library source code to understand internal code paths, error handling and performance characteristics.

Recently, I looked at how the standard library's built-in HTTP client handles connections to remote servers in order to provide minimal roundtrip latency.

CC By 2.0 Image by Dean Hochman

Connection pooling

A common pattern that aims to avoid connection setup costs (such as the TCP handshake and TLS setup) and confer control over the number of concurrently established connections is to pool them. net/http maintains a pool of connections to each remote host which supports Connection: keep-alive. The default size of the pool is two idle connections per remote host.

More interestingly, when you make a request with net/http, a race happens. Races in code are often an unwanted side effect, but in this case it's intentional. Two goroutines operate in parallel: one that tries to dial a connection to the remote host, and another which tries to retrieve an idle connection from the connection pool. The fastest goroutine wins.

To illustrate, let's look at the code executed when transport.RoundTrip(req) is Continue reading

Unser neues 72. Rechenzentrum: Hamburg

Moin Hamburg! Ensconced alongside the Elbe River, Hamburg, a major port city in northern Germany, is the second largest city in the country, and the eight largest in the European Union. Our data center in Hamburg is our 4th in Germany following deployments in Frankfurt, Düsseldorf and Berlin, our 19th in Europe, and 72nd globally. This means not only better performance in Germany, but additional redundancy for our 3 other data centers throughout the country. As of this moment, CloudFlare has a point of presence (PoP) in 8 out of Europe's 10 most populous* cities, and we're headed for a perfect 10-for-10 (look out Budapest...).

For the local audience: Liebe Freunde in Hamburg, Euer Internetanschluss ist schneller geworden und ihr könnt jetzt sicherer surfen. Viel Spaß.

Frohe Festtage!

Yesterday we announced new points of presence (PoPs) in Montreal and Vancouver. Today: Hamburg. However, the holidays are hardly over, and we have lots more cheer to spread. We've sent planes sleighs full of servers, switches, routers and PDUs to many corners of the globe. And to cap it off, we'll gift some CloudFlare gear Continue reading

Vancouver & Montreal, Canada: CloudFlare’s latest data centers

With the holiday season in full swing, it's only fitting that we continue to spread cheer, joy and a faster Internet around the world. To start the season we begin in Canada with NHL rivals Montreal and Vancouver, our 70th and 71st points of presence (PoPs) globally. Montreal and Vancouver, the 2nd and 3rd largest Canadian metropolitan areas, respectively, join our existing PoP in Canada's largest, Toronto.

Together, CloudFlare's network in Canada is now milliseconds away from the country's 31 million Internet users. As of now, the web sites, mobile apps and APIs of all CloudFlare customers are delivered at a cool 6.1 million times the speed of the fastest slapshot (for the curious, the current NHL speed record belongs to Zdeno Chára of the Boston Bruins, whose slapshot clocked 108.8 miles per hour / 175.1 kilometers per hour).

Latency matters

Canada is not just one of the most wired countries in the world, with nearly 87 per cent of Canadian households connected to the Internet, but also one of the largest as measured by e-commerce transaction volume. According to Statistics Canada, Canadian enterprises sold more than US$100 billion in goods and services over the Internet in Continue reading

A Different Kind of POP: The Joomla Unserialize Vulnerability

At CloudFlare, we spend a lot of time talking about the PoPs (Points of Presence) we have around the globe, however, on December 14th, another kind of POP came to the world: a vulnerability being exploited in the wild against Joomla’s Content Management System. This is known as a zero day attack, where it has been zero days since a patch has been released for that bug. A CVE ID has been issued for this particular vulnerability as CVE-2015-8562. Jaime Cochran and I decided to take a closer look.

In this blog post we’ll explain what the vulnerability is, give examples of actual attack payloads we’ve seen, and show how CloudFlare automatically protects Joomla users. If you are using Joomla with CloudFlare today and have our WAF enabled, you are already protected.

The Joomla Web Application Firewall rule set is enabled by default for CloudFlare customers with a Pro or higher plan, which blocks this attack. You can find it in the Joomla section of the CloudFlare Rule Set in the WAF Dashboard.

What is Joomla?

Joomla is an open source Content Management System which allows you to build web applications and control every aspect of the content of your Continue reading

Partial kernel bypass merged into netmap master

In a previous post we described our work on a new netmap mode called single-rx-queue.

After submitting the pull request, the netmap maintainers told us that the patch was interesting, but they would prefer something more configurable instead of a tailored custom mode.

After an exchange of ideas and some more work, our patch just got merged to mainline netmap.

Meet the new netmap

Before our patch netmap used to be an all-or-nothing deal. That is: there was no way to put a network adapter partially in netmap mode. All of the queues would have to be detached from the host network stack. Even a netmap mode called “single ring pair” didn't help.

Our final patch is extended and more generic, while still supporting the simple functionality of our original single-rx-queue mode.

First we modified netmap to leave queues that are not explicitly requested to be in netmap mode attached to the host stack. In this way, if a user requests a pair of rings (for example using nm_open(“netmap:eth0-4”)) it will actually get a reference to both the number 4 RX and TX rings, while keeping the other rings attached to the kernel stack.

But since the NIC is Continue reading

HTTP/2 Demo: Under the Hood

At first glance, the potential performance improvements of HTTP/1.1 versus HTTP/2 on our demo page may seem a bit hard to believe. So, we put together a technical explanation of how this demo actually works. We’d also like to credit the Gophertiles demo, which served as a basis for our own HTTP/2 demo.

Overview

A web page can only be served over either HTTP/1.1 or HTTP/2—mixing protocols is not allowed. Our demo page is HTTP/2-enabled, so there’s no way to load HTTP/1.1 content directly on the same page. Inline frames (iframes) can be used to solve this issue. We embedded two iframes on our demo page, both containing the same source code. The key difference is that one iframe loads over an HTTP/1.1 CDN while the other loads over an HTTP/2 CDN.

We chose Amazon CloudFront for the HTTP/1.1 CDN because it can only serve content over HTTP/1.1. For the HTTP/2 CDN, we’re using our own HTTP/2-enabled network. You can take a look at the individual HTTP/1.1 and HTTP/2 iframe content, which should have similar load times to the side-by-side example on our demo page.

Iframe Content

So, what is contained in Continue reading

HTTP/2 For Web Developers

HTTP/2 changes the way web developers optimize their websites. In HTTP/1.1, it’s become common practice to eek out an extra 5% of page load speed by hacking away at your TCP connections and HTTP requests with techniques like spriting, inlining, domain sharding, and concatenation.

Life’s a little bit easier in HTTP/2. It gives the typical website a 30% performance gain without a complicated build and deploy process. In this article, we’ll discuss the new best practices for website optimization in HTTP/2.

Web Optimization in HTTP/1.1

Most of the website optimization techniques in HTTP/1.1 revolved around minimizing the number of HTTP requests to an origin server. A browser can only open a limited number of simultaneous TCP connections to an origin, and downloading assets over each of those connections is a serial process: the response for one asset has to be returned before the next one can be sent. This is called head-of-line blocking.

As a result, web developers began squeezing as many assets as they could into a single connection and finding other ways to trick browsers into avoiding head-of-line blocking. In HTTP/2, some of these practices can actually hurt page load times.

The New Web Optimization Continue reading