How to Talk to Your Parents About Encryption
It’s December 25th, which means most of you are probably at home visiting with family. I asked a few of the security engineers here at CloudFlare how they explain their jobs when they’re home for the holidays, and most of them responded with something along the lines of, "Oh, I stopped trying to do that a long time ago." Apparently, working in the cryptography field doesn’t exactly make it easy to talk about work with your parents.
After chatting with our crypto experts some more, we figured out a decent way to explain the general idea of encryption and why it’s a critical part of the Internet. While this post may not explain exactly what security engineers do on a day-to-day basis, hopefully it will help you at least tell your parents why you have a job in the first place.
Banks and Their Big Fancy Buildings
To explain encryption to your parents, I’d start by asking them why they trust their bank. Let’s say they have some cash to deposit. They drive to their bank’s local branch, walk through a big fancy lobby, wait in line for a teller, and hand them their money. It may seem like Continue reading
Zdrasti, Sofia! CloudFlare’s 73rd Data Center Now Live
Only days after the launch of our Hamburg data center, CloudFlare is excited to announce yet another European data center - this time in Sofia, Bulgaria. With over 1.2 million people, Sofia is a city with rich history tracing back over 7,000 years.
We were fascinated to note the coincidence that even as 1 in 73 of CloudFlare team members is Bulgarian, now 1 in 73 of CloudFlare data centers is in Bulgaria!
Localizing European traffic
Sofia expands the CloudFlare global network to span 20 European data centers - joining Amsterdam, Frankfurt, Paris, London, Vienna, Prague, Stockholm, Warsaw, Madrid, Milan, Dusseldorf, Marseille, Bucharest, Dublin, Manchester, Zurich, Copenhagen, Berlin and Hamburg.
Each time we launch a new data center, we improve the performance of millions of websites, expand the surface area available to fight attacks, and provide an additional point of redundancy to support our existing data centers.
Until today, many Bulgarian networks were served out of Frankfurt, over 1,000 miles away, based on their interconnection there with our tier one providers. Our newest deployment eliminates that distance, and improves the web Continue reading
Why it’s harder to forge a SHA-1 certificate than it is to find a SHA-1 collision
It’s well known that SHA-1 is no longer considered a secure cryptographic hash function. Researchers now believe that finding a hash collision (two values that result in the same value when SHA-1 is applied) is inevitable and likely to happen in a matter of months. This poses a potential threat to trust on the web, as many websites use certificates that are digitally signed with algorithms that rely on SHA-1. Luckily for everyone, finding a hash collision is not enough to forge a digital certificate and break the trust model of the Internet.
We’ll explore how hash collisions have been used to forge digital signatures in the past. We’ll also discuss how certificate authorities can make this significantly harder for attackers in the future by including randomness in certificate serial numbers.
Digital signatures are the bedrock of trust
The Internet relies on trust. Whether it’s logging in to your bank or reading Reddit, HTTPS protects you by encrypting the data you exchange with a site and authenticating the site's identity with a digital certificate. Browsers visually display the added security of HTTPS as a padlock in the address bar.
HTTPS can prove a site’s authenticity to a browser when a Continue reading
What’s inside net/http? Late binding in the Go standard library
It's well known that we're heavy users of the Go programming language at CloudFlare. Our work often involves delving into the standard library source code to understand internal code paths, error handling and performance characteristics.
Recently, I looked at how the standard library's built-in HTTP client handles connections to remote servers in order to provide minimal roundtrip latency.
CC By 2.0 Image by Dean Hochman
Connection pooling
A common pattern that aims to avoid connection setup costs (such as the TCP handshake and TLS setup) and confer control over the number of concurrently established connections is to pool them. net/http maintains a pool of connections to each remote host which supports Connection: keep-alive. The default size of the pool is two idle connections per remote host.
More interestingly, when you make a request with net/http, a race happens. Races in code are often an unwanted side effect, but in this case it's intentional. Two goroutines operate in parallel: one that tries to dial a connection to the remote host, and another which tries to retrieve an idle connection from the connection pool. The fastest goroutine wins.
To illustrate, let's look at the code executed when transport.RoundTrip(req) is Continue reading
Unser neues 72. Rechenzentrum: Hamburg
Moin Hamburg! Ensconced alongside the Elbe River, Hamburg, a major port city in northern Germany, is the second largest city in the country, and the eight largest in the European Union. Our data center in Hamburg is our 4th in Germany following deployments in Frankfurt, Düsseldorf and Berlin, our 19th in Europe, and 72nd globally. This means not only better performance in Germany, but additional redundancy for our 3 other data centers throughout the country. As of this moment, CloudFlare has a point of presence (PoP) in 8 out of Europe's 10 most populous* cities, and we're headed for a perfect 10-for-10 (look out Budapest...).
For the local audience: Liebe Freunde in Hamburg, Euer Internetanschluss ist schneller geworden und ihr könnt jetzt sicherer surfen. Viel Spaß.
Frohe Festtage!
Be sure to have some Glühwein if you visit the Christkindlmärkte this holiday season
Yesterday we announced new points of presence (PoPs) in Montreal and Vancouver. Today: Hamburg. However, the holidays are hardly over, and we have lots more cheer to spread. We've sent planes sleighs full of servers, switches, routers and PDUs to many corners of the globe. And to cap it off, we'll gift some CloudFlare gear Continue reading
Vancouver & Montreal, Canada: CloudFlare’s latest data centers
With the holiday season in full swing, it's only fitting that we continue to spread cheer, joy and a faster Internet around the world. To start the season we begin in Canada with NHL rivals Montreal and Vancouver, our 70th and 71st points of presence (PoPs) globally. Montreal and Vancouver, the 2nd and 3rd largest Canadian metropolitan areas, respectively, join our existing PoP in Canada's largest, Toronto.
Together, CloudFlare's network in Canada is now milliseconds away from the country's 31 million Internet users. As of now, the web sites, mobile apps and APIs of all CloudFlare customers are delivered at a cool 6.1 million times the speed of the fastest slapshot (for the curious, the current NHL speed record belongs to Zdeno Chára of the Boston Bruins, whose slapshot clocked 108.8 miles per hour / 175.1 kilometers per hour).
Latency matters
Canada is not just one of the most wired countries in the world, with nearly 87 per cent of Canadian households connected to the Internet, but also one of the largest as measured by e-commerce transaction volume. According to Statistics Canada, Canadian enterprises sold more than US$100 billion in goods and services over the Internet in Continue reading
A Different Kind of POP: The Joomla Unserialize Vulnerability
At CloudFlare, we spend a lot of time talking about the PoPs (Points of Presence) we have around the globe, however, on December 14th, another kind of POP came to the world: a vulnerability being exploited in the wild against Joomla’s Content Management System. This is known as a zero day attack, where it has been zero days since a patch has been released for that bug. A CVE ID has been issued for this particular vulnerability as CVE-2015-8562. Jaime Cochran and I decided to take a closer look.
In this blog post we’ll explain what the vulnerability is, give examples of actual attack payloads we’ve seen, and show how CloudFlare automatically protects Joomla users. If you are using Joomla with CloudFlare today and have our WAF enabled, you are already protected.
The Joomla Web Application Firewall rule set is enabled by default for CloudFlare customers with a Pro or higher plan, which blocks this attack. You can find it in the Joomla section of the CloudFlare Rule Set in the WAF Dashboard.
What is Joomla?
Joomla is an open source Content Management System which allows you to build web applications and control every aspect of the content of your Continue reading
Partial kernel bypass merged into netmap master
In a previous post we described our work on a new netmap mode called single-rx-queue.
After submitting the pull request, the netmap maintainers told us that the patch was interesting, but they would prefer something more configurable instead of a tailored custom mode.
After an exchange of ideas and some more work, our patch just got merged to mainline netmap.
Meet the new netmap
Before our patch netmap used to be an all-or-nothing deal. That is: there was no way to put a network adapter partially in netmap mode. All of the queues would have to be detached from the host network stack. Even a netmap mode called “single ring pair” didn't help.
Our final patch is extended and more generic, while still supporting the simple functionality of our original single-rx-queue mode.
First we modified netmap to leave queues that are not explicitly requested to be in netmap mode attached to the host stack. In this way, if a user requests a pair of rings (for example using nm_open(“netmap:eth0-4”)) it will actually get a reference to both the number 4 RX and TX rings, while keeping the other rings attached to the kernel stack.
But since the NIC is Continue reading
HTTP/2 Demo: Under the Hood
At first glance, the potential performance improvements of HTTP/1.1 versus HTTP/2 on our demo page may seem a bit hard to believe. So, we put together a technical explanation of how this demo actually works. We’d also like to credit the Gophertiles demo, which served as a basis for our own HTTP/2 demo.
Overview
A web page can only be served over either HTTP/1.1 or HTTP/2—mixing protocols is not allowed. Our demo page is HTTP/2-enabled, so there’s no way to load HTTP/1.1 content directly on the same page. Inline frames (iframes) can be used to solve this issue. We embedded two iframes on our demo page, both containing the same source code. The key difference is that one iframe loads over an HTTP/1.1 CDN while the other loads over an HTTP/2 CDN.
We chose Amazon CloudFront for the HTTP/1.1 CDN because it can only serve content over HTTP/1.1. For the HTTP/2 CDN, we’re using our own HTTP/2-enabled network. You can take a look at the individual HTTP/1.1 and HTTP/2 iframe content, which should have similar load times to the side-by-side example on our demo page.
Iframe Content
So, what is contained in Continue reading
HTTP/2 For Web Developers
HTTP/2 changes the way web developers optimize their websites. In HTTP/1.1, it’s become common practice to eek out an extra 5% of page load speed by hacking away at your TCP connections and HTTP requests with techniques like spriting, inlining, domain sharding, and concatenation.
Life’s a little bit easier in HTTP/2. It gives the typical website a 30% performance gain without a complicated build and deploy process. In this article, we’ll discuss the new best practices for website optimization in HTTP/2.
Web Optimization in HTTP/1.1
Most of the website optimization techniques in HTTP/1.1 revolved around minimizing the number of HTTP requests to an origin server. A browser can only open a limited number of simultaneous TCP connections to an origin, and downloading assets over each of those connections is a serial process: the response for one asset has to be returned before the next one can be sent. This is called head-of-line blocking.
As a result, web developers began squeezing as many assets as they could into a single connection and finding other ways to trick browsers into avoiding head-of-line blocking. In HTTP/2, some of these practices can actually hurt page load times.
The New Web Optimization Continue reading
SHA-1 Deprecation: No Browser Left Behind
After December 31, 2015, SSL certificates that use the SHA-1 hash algorithm for their signature will be declared technology non grata on the modern Internet. Google's Chrome browser has already begun displaying a warning for SHA-1 based certs that expire after 2015. Other browsers are mirroring Google and, over the course of 2016, will begin issuing warnings and eventually completely distrust connections to sites using SHA-1 signed certs. And, starting January 1, 2016, you will no longer be able to get a new SHA-1 certificate from most certificate authorities.
For the most part, that's a good thing. Prohibitively difficult to forge certificate signatures are part of what keeps encryption systems secure. As computers get faster, the risk that, for any given hashing algorithm, you can forge a certificate with the same signature increases. If an attacker can forge a certificate then they could potentially impersonate the identity of a real site and intercept its encrypted traffic or masquerade as that site.
Deprecating Old Standards
This isn't the first time we've been through this exercise. The original hashing algorithm used for most certificate signatures in the early days of the web was MD5. In 2008, researchers demonstrated they were able to Continue reading
What is Internet Goverance and Why Does it Matter?
Last month, CloudFlare participated the tenth annual Internet Governance Forum (IGF) in Joao Pessoa, Brazil. Since it was launched at the United Nations’ World Summit on the Information Society (WSIS) in 2005, the IGF has provided valuable opportunities for thousands of representatives of non-profit groups, businesses, governments, and others to debate decisions that will affect the future of the Internet. While the Forum does not negotiate any treaties or other agreements, what participants learn there can influence corporate strategies, standards proposals, and national government policies. Even more importantly, discussions in the hallways (or in the bar or on the beach) can lead to new projects, new thinking, and new collaborations.
The range of issues and the diversity of speakers on panels and at the podium was even greater this year than at previous IGFs. Issues ranged from the need for strong encryption to whether net neutrality regulations are needed—from countering the abuse of women online to how to foster deployment of IPv6 and Internet Exchange Points. You can watch all 167 IGF sessions, which were webcast and archived. I represent CloudFlare as a member of the Multistakeholder Advisory Group (MAG), which organizes the IGF program. Together with the other MAG Continue reading
Tools for debugging, testing and using HTTP/2
With CloudFlare's release of HTTP/2 for all our customers the web suddenly has a lot of HTTP/2 connections. To get the most out of HTTP/2 you'll want to be using an up to date web browser (all the major browsers support HTTP/2).
But there are some non-browser tools that come in handy when working with HTTP/2. This blog post starts with a useful browser add-on, and then delves into command-line tools, load testing, conformance verification, development libraries and packet decoding for HTTP/2.
If you know of something that I've missed please write a comment.
Browser Indicators
For Google Chrome there's a handy HTTP/2 and SPDY Indicator extension that adds a colored lightning bolt to the browser bar showing the protocol being used when a web page is viewed.
The blue lightning bolt shown here indicates that the CloudFlare home page was served using HTTP/2:
A green lightning bolt indicates the site was served using SPDY and gives the SPDY version number. In this case SPDY/3.1:
A grey lightning bolt indicates that neither HTTP/2 no SPDY were used. Here the web page was served using HTTP/1.1.
There's a similar extension for Firefox.
Online testing
There's also a handy online Continue reading
HTTP/2 is here! Goodbye SPDY? Not quite yet
Why choose, if you can have both? Today CloudFlare is introducing HTTP/2 support for all customers using SSL/TLS connections, while still supporting SPDY. There is no need to make a decision between SPDY or HTTP/2. Both are automatically there for you and your customers.
Enabling HTTP/2
If you are a customer on the Free or Pro plan, there is no need to do anything at all. Both SPDY and HTTP/2 are already enabled for you. With this improvement, your website’s audience will always use the fastest protocol version when accessing your site over TLS/SSL.
Customers on Business and Enterprise plans may enable HTTP/2 within the "Network" application of the CloudFlare Dashboard.
HTTP/2 is here!
In February of 2015, the IETF’s steering group for publication as standards-track RFCs approved the HTTP/2 and associated HPACK specifications.
After more than 15 years, the Hypertext Transfer Protocol (HTTP) received a long-overdue upgrade. HTTP/2 is largely based on Google's experimental SPDY protocol, which was first announced in November 2009 as an internal project to increase the speed of the web.
Benefits of HTTP/2 and SPDY
The main focus of both SPDY and HTTP/2 is on performance, especially latency as perceived by the end-user while using Continue reading
Zürich, Switzerland: CloudFlare’s 69th data center
Grüetzi Zürich, our 5th point of presence (PoP) to be announced this week, and 69th globally! Located at the northern tip of Lake Zürich in Switzerland, the city of Zürich, often referred to as "Downtown Switzerland," is the largest city in the country. Following this expansion, traffic from Switzerland's seven million internet users to sites and apps using CloudFlare is now mere milliseconds away. Although best known to some for its chocolate and banks, Switzerland is home to many of the most significant developments prefacing the modern internet.
Vague but interesting
It was in 1989 that Tim Berners-Lee, a British scientist at CERN, the large particle physics laboratory near Geneva, Switzerland, invented the World Wide Web (WWW). Tim laid out his vision to meet the demand for automatic information-sharing between scientists in universities and institutes around the world in a memo titled, "Information Management: a Proposal". Amusingly, his initial proposal wasn't immediately accepted. In fact, his boss at the time noted that the proposal was, "vague but exciting" on the cover page.
At least one VC may have said something similar to us once upon a time
The first website at CERN—and in the world—was dedicated to the Continue reading
CloudFlare launches India data centers in Mumbai, Chennai and New Delhi
India is home to 400 million Internet users, second only to China, and will add more new users this year than any other country in the world. CloudFlare protects and accelerates 4 million websites, mobile apps and APIs, and is trusted by over 10,000 new customers each day. Combine these forces, and we are positioned to connect hundreds of millions of Indian users with the millions of internet applications they use each day.
Today, we accelerate this momentum with the announcement of three new points of presence (PoPs) in Mumbai, Chennai and New Delhi. These new sites represent the 66th, 67th and 68th data centers respectively across our global network.
We’ve come a long way
The beginnings of the “internet” in India as we know it started in 1986 when the country launched ERNET (the Education and Research Network). Six years later, a 64 Kbps digital leased line was commissioned from the National Centre for Software Technology in Mumbai to UUNet in Virginia to connect India with the rest of the internet. By comparison, a single port on our router in each of Mumbai, Chennai and New Delhi has nearly 160,000 times the capacity today.
The pace of progress has Continue reading
The story of one latency spike
A customer reported an unusual problem with our CloudFlare CDN: our servers were responding to some HTTP requests slowly. Extremely slowly. 30 seconds slowly. This happened very rarely and wasn't easily reproducible. To make things worse all our usual monitoring hadn't caught the problem. At the application layer everything was fine: our NGINX servers were not reporting any long running requests.
Time to send in The Wolf.
He solves problems.
Following the evidence
First, we attempted to reproduce what the customer reported—long HTTP responses. Here is a chart of of test HTTP requests time measured against our CDN:
We ran thousands of HTTP queries against one server over a couple of hours. Almost all the requests finished in milliseconds, but, as you can clearly, see 5 requests out of thousands took as long as 1000ms to finish. When debugging network problems the delays of 1s, 30s are very characteristic. They may indicate packet loss since the SYN packets are usually retransmitted at times 1s, 3s, 7s, 15, 31s.
Blame the network
At first we thought the spikes in HTTP load times might indicate some sort of network problem. To be sure we ran ICMP pings against two IPs over many Continue reading
Copenhagen, Denmark: CloudFlare’s 65th data center
To get the week started it's our distinct pleasure to introduce CloudFlare's latest PoP (point of presence) in Copenhagen, Denmark. Our Copenhagen data center extends the CloudFlare network to 65 PoPs across 34 countries, with 17 in Europe alone. The CloudFlare network, including all of the Internet applications and content of our users, is now delivered with a median latency of under 40ms throughout the entire continent—by comparison, it takes 300-400ms to blink one's eyes!
Danish traffic, previously served from Stockholm and Amsterdam, shifts into Copenhagen
As can be seen above, traffic has already started to reach Copenhagen, with steady increases over the course of the day (all times in UTC). The new site is also already mitigating cyber attacks launched against our customers. The spike in traffic around 08:46 UTC is a modest portion of a globally distributed denial of service (DDoS) attack targeted at CloudFlare. By distributing the attack across an ever growing footprint of data centers, mitigation is made easy (and our site reliability engineers can sleep soundly!).
The week's not over
In December 2014 we announced our intention to launch one data center per week throughout 2015. It's an ambitious goal, but we're well on Continue reading
Announcing Universal DNSSEC: Secure DNS for Every Domain
CloudFlare launched just five years ago with the goal of building a better Internet. That’s why we are excited to announce that beginning today, anyone on CloudFlare can secure their traffic with DNSSEC in just one simple step.
This follows one year after we made SSL available for free, and in one week, more than doubled the size of the encrypted web. Today we will do the same with DNSSEC, and this year, we’ll double the size of the DNSSEC-enabled web, bringing DNSSEC to millions of websites, for free.
If DNS is the phone book of the Internet, DNSSEC is the unspoofable caller ID. DNSSEC ensures that a website’s traffic is safely directed to the correct servers, so that a connection to a website is not intercepted by a man-in-the-middle.
Solving A Decades-Old Vulnerability In DNS
Every website visit begins with a DNS query. When I visit cloudflare.com, my browser first needs to find the IP address:
cloudflare.com. 272 IN A 198.41.215.163
When DNS was invented in 1983, the Internet was used by only a handful of professors and researchers, and no one imagined that there could be foul play. Thus, DNS relies on Continue reading
Phoenix, AZ: CloudFlare’s 64th data center
Three years and 46 data centers later our expansion returns to the United States. Phoenix, the latest addition to the CloudFlare network, is our 10th point of presence in North America, and the start of our effort to further regionalize traffic across the continent. This means faster page loads and transaction speeds for your sites and applications, as well as for the 6 million Internet users throughout the Southwestern US that use them.
Eat Surf local
The vast majority of Internet traffic in the US is exchanged in only a small handful of cities: Los Angeles, the San Francisco Bay Area, Dallas, Chicago, Miami, Ashburn (Virginia) and New York. These locations evolved into key interconnection points largely as a result of their status as population and economic centers. However, if you're one of the 236 million Americans that live outside of these metro areas, you have to hike quite a bit further to access your favorite content on the Internet.
To illustrate this, we measured the level of local interconnection between a handful of our Tier 1 Internet providers—NTT, TeliaSonera, Tata Communications and Cogent—in different metro areas. For the uninitiated, Tier 1 networks are the group of networks that Continue reading