The Internet is a cooperative system: CNAME to Dyn DNS outage of 6 July 2015

Today, shortly after 21:00 UTC, on our internal operations chat there was a scary message from one of our senior support staff: "getting DNS resolution errors on support.cloudflare.com", at the same time as automated monitoring indicated a problem. Shortly thereafter, we saw alarms and feedback from a variety of customers (but not everyone) reporting "1001 errors", which indicated a DNS resolution error on the CloudFlare backend. Needless to say, this got an immediate and overwhelming response from our operations and engineering teams, as we hadn't changed anything and had no other indications of anomaly.

In the course of debugging, we were able to identify common characteristics of affected sites—CNAME-based users of CloudFlare, rather than complete domain hosted entirely on CloudFlare, which, ironically, included our own support site, support.cloudflare.com. When users point (via CNAME) to a domain instead of providing us with an IP address, our network resolves that name —- and is obviously unable to connect if the DNS provider has issues. (Our status page https://www.cloudflarestatus.com/ is off-network and was unaffected). Then, we were investigating why only certain domains were having issues—was the issue with the upstream DNS? Testing whether their domains were resolvable Continue reading

Contributing back to the security community

This Friday at the RSA Conference in San Francisco, along with Marc Rogers, Principal Security Researcher at CloudFlare, I'm speaking about a version of The Grugq's PORTAL, an open source network security device designed to make life easier and safer for anyone traveling, especially internationally, with phones, tablets, laptops, and other network-connected devices.

Portal uses open-source software and services to take inexpensive, commodity travel routers and turn them into powerful security devices. Since this is pretty far from CloudFlare's core business, it warrants a brief digression into why we support projects like this.

Computer security was for a very long time only of interest to hobbyists, academics, and obscure government agencies. Cryptography was an interesting offshoot of number theory, a foundational but very abstract part of mathematics, and many of the early infrastructure components of the Internet didn't include security at all -- there was an assumption that anyone who could gain access would be responsible and well-intentioned, a consequence of the academic origins; after all, why would they want to break or steal things which were freely available.

Before the "cambrian explosion" of commercial computer security, there was still a lot of great security research -- it Continue reading

OpenSSL Security Advisory of 19 March 2015

Today there were multiple vulnerabilities released in OpenSSL, a cryptographic library used by CloudFlare (and most sites on the Internet). There has been advance notice that an announcement would be forthcoming, although the contents of the vulnerabilities were kept closely controlled and shared only with major operating system vendors until this notice.

Based on our analysis of the vulnerabilities and how CloudFlare uses the OpenSSL library, this batch of vulnerabilties primarily affects CloudFlare as a "Denial of Service" possibility (it can cause CloudFlare's proxy servers to crash), rather than as an information disclosure vulnerability. Customer traffic and customer SSL keys continue to be protected.

As is good security practice, we have quickly tested the patched version and begun a push to our production environment, to be completed within the hour. We encourage all customers to upgrade to the latest patched versions of OpenSSL on their own servers, particularly if they are using the 1.0.2 branch of the OpenSSL library.

The individual vulnerabilities included in this announcement are:

• OpenSSL 1.0.2 ClientHello sigalgs DoS (CVE-2015-0291)
• Reclassified: RSA silently downgrades to EXPORT_RSA [Client] (CVE-2015-0204)
• Multiblock corrupted pointer (CVE-2015-0290)
• Segmentation fault in DTLSv1_listen (CVE-2015-0207)
• Segmentation fault in ASN1TYPE Continue reading

Enforce Web Policy with HTTP Strict Transport Security (HSTS)

HTTP Strict Transport Security (HSTS, RFC 6797) is a web security policy technology designed to help secure HTTPS web servers against downgrade attacks. HSTS is a powerful technology which is not yet widely adopted. CloudFlare aims to change this.

Downgrade attacks (also known as SSL stripping attacks) are a serious threat to web applications. This type of attack is a form of man-in-the-middle attack in which an attacker can redirect web browsers from a correctly configured HTTPS web server to an attacker controlled server. Once the attacker has successfully redirected a user, user data, including cookies, can be compromised. Unfortunately, this attack is outside the realm of pure SSL to prevent. This is why HSTS was created.

These attacks are very real: many major websites have been attacked through SSL stripping. They are a particularly powerful attack against otherwise well secured sites, as they bypass the protections of SSL.

HSTS headers consists of an HTTP header with several parameters -- including a configurable duration for client web browsers to cache and continue to enforce policy even if the site itself changes. Through CloudFlare, it is easy to configure on a per-domain basis with standard settings.

HSTS causes compliant browsers Continue reading

Bash vulnerability CVE-2014-6271 patched

This morning, Stephane Chazelas disclosed a vulnerability in the program bash, the GNU Bourne-Again-Shell. This software is widely used, especially on Linux servers, such as the servers used to provide CloudFlare’s performance and security cloud services.

This vulnerability is a serious risk to Internet infrastructure, as it allows remote code execution in many common configurations, and the severity is heightened due to bash being in the default configuration of most Linux servers. While bash is not directly used by remote users, it is used internally by popular software packages such as web, mail, and administration servers. In the case of a web server, a specially formatted web request, when passed by the web server to the bash application, can cause the bash software to run commands on the server for the attacker. More technical information was posted on the oss-sec mailing list.

The security community has assigned this bash vulnerability the ID CVE-2014-6271.

As soon as we became aware of this vulnerability, CloudFlare’s engineering and operations teams tested a patch to protect our servers, and deployed it across our infrastructure. As of now, all CloudFlare servers are protected against CVS-2014-6271.

Everyone who is using the bash software package should upgrade Continue reading