CloudFlare Crypto Meetup: April 21, 2016
CloudFlare Crypto Meetup Teaser.
Now back in HD: the CloudFlare Cryptography Meetup series. A while back, CloudFlare hosted a pair of Meetups focused on encryption and cryptographic technology. Now that CloudFlare HQ has moved into our beautiful new home at 101 Townsend in San Francisco, we’ve decided to bring the crypto back.
In this series, we’ve invited experts from academia and industry to talk about the cryptographic protocols they are working on and to share experiences around deploying cryptographic applications in the real world. This is the place to geek out on crypto!
These talks are intended to explore interesting new crypto topics in an accessible way. It aims to be informative and thought provoking, and practical examples are encouraged.
We’ll start the evening at 6:00p.m. with time for networking, followed up with short talks by leading experts. Pizza and beer are provided!
Whether you're a cryptography hobbyist, an industry expert or just interested in the subject, come visit CloudFlare’s world headquarters at 6:00pm on April 21st.
RSVP here on Meetup.com.
Speakers
The confirmed speakers for April 21st are Brian Warner, Zakir Durumeric and Amine Kamel.
Brian Warner
magic-wormhole
"magic-wormhole" is a simple tool to move files from Continue reading
The revenge of the listening sockets
Back in November we wrote a blog post about one latency spike. Today I'd like to share a continuation of that story. As it turns out, the misconfigured rmem setting wasn't the only source of added latency.
It looked like Mr Wolf hadn't finished his job.
After adjusting the previously discussed rmem sysctl we continued monitoring our systems' latency. Among other things we measured ping times to our edge servers. While the worst case improved and we didn't see 1000ms+ pings anymore, the line still wasn't flat. Here's a graph of ping latency between an idling internal machine and a production server. The test was done within the datacenter, the packets never went to the public internet. The Y axis of the chart shows ping times in milliseconds, the X axis is the time of the measurement. Measurements were taken every second for over 6 hours:
As you can see most pings finished below 1ms. But out of 21,600 measurements about 20 had high latency of up to 100ms. Not ideal, is it?
System tap
The latency occurred within our datacenter and the packets weren't lost. This suggested a kernel issue again. Linux responds to ICMP pings from its soft Continue reading
It takes two to ChaCha (Poly)
Not long ago we introduced support for TLS cipher suites based on the ChaCha20-Poly1305 AEAD, for all our customers. Back then those cipher suites were only supported by the Chrome browser and Google's websites, but were in the process of standardization. We introduced these cipher suites to give end users on mobile devices the best possible performance and security.
CC BY-ND 2.0 image by Edwin Lee
Today the standardization process is all but complete and implementations of the most recent specification of the cipher suites have begun to surface. Firefox and OpenSSL have both implemented the new cipher suites for upcoming versions, and Chrome updated its implementation as well.
We, as pioneers of ChaCha20-Poly1305 adoption on the web, also updated our open sourced patch for OpenSSL. It implements both the older "draft" version, to keep supporting millions of users of the existing versions of Chrome, and the newer "RFC" version that supports the upcoming browsers from day one.
In this blog entry I review the history of ChaCha20-Poly1305, its standardization process, as well as its importance for the future of the web. I will also take a peek at its performance, compared to the other standard AEAD.
What Continue reading
Come Geek Out With The Original Inventor of DNS at CloudFlare
We like DNS, we think you might too.
CloudFlare and Gandi are hosting a three-part series on DNS. Our first event will be at the CloudFlare office with Paul Mockapetris, the original inventor of the Domain Name System.
Beyond inventing DNS, Paul built the first ever SMTP server. He ran networking at ARPA, served as the chair of the IETF, and is a honored member of the Internet Hall of Fame. He is currently the Chief Scientist at Threatstop, and the visiting scholar at the Universite de Pierre et Marie Curie in Paris.
The event is on Tuesday, April 12th, 2016 at 6 PM PST at our office in San Francisco, 101 Townsend Street (RSVP here). We’ll be covering the early days of DNS, DNS and security, the commercialization of DNS (what Paul famously calls DN$), and the future of DNS.
So come, grab some beer, and hang out with people who like DNS as much as you do.
Introducing CFSSL 1.2
Continuing our commitment to high quality open-source software, we’re happy to announce release 1.2 of CFSSL, our TLS/PKI Swiss Army knife. We haven’t written much about CFSSL here since we originally open sourced the project in 2014, so we thought we’d provide an update. In the last 20 months, we have added a ton of great features, and CFSSL has attracted an active community of users and contributors. Users range from large SaaS providers (Heroku) to game companies (Riot Games) and the newest Certificate Authority (Let’s Encrypt). For them and for CloudFlare, CFSSL has become a core tool for automating certificates and TLS configurations. With added support for configuration scanning, automated provisioning via the transport package, revocation, certificate transparency and PKCS#11, CFSSL is now even more powerful.
We’re also happy to announce CFSSL’s new home: cfssl.org. From there you can try out CFSSL’s user interface, download binaries, and test some of its features.
Motivation
This 2013 National Security Agency (NSA) slide describing how data from Google’s internal network was collected by intelligence agencies was eye-opening—and shocking—to many technology companies. The idea that an attacker could read messages passed between services wasn’t technically groundbreaking, but it Continue reading
The Trouble with Tor
The Tor Project makes a browser that allows anyone to surf the Internet anonymously. Tor stands for "the Onion router" and that describes how the service works. Traffic is routed through a number of relays run across the Internet where each relay only knows the next hop (because each hop is enclosed in a cryptographic envelope), not the ultimate destination, until the traffic gets to the final exit node which connects to the website — like peeling the layers of an onion.
Think of it like a black box: traffic goes into the box, is bounced around between a random set of relays, and ultimately comes out to connect to the requested site. Anonymity is assured because anyone monitoring the network would have a difficult time tying the individuals making the requests going into the black box with the requests coming out.
Importance and Challenges of Anonymity
Anonymity online is important for a number of reasons we at CloudFlare believe in. For instance, Tor is instrumental in ensuring that individuals living in repressive regimes can access information that may otherwise be blocked or illegal. We this is so important that we offer Continue reading
Going to IETF 95? Join the TLS 1.3 hackathon
If you’re in Buenos Aires on April 2-3 and are interested in building, come join the IETF Hackathon. CloudFlare and Mozilla will be working on TLS 1.3, the first new version of TLS in eight years!
At the hackathon we’ll be focusing on implementing the latest draft of TLS 1.3 and testing interoperability between existing implementations written in C, Go, OCaml, JavaScript and F*. If you have experience with network programming and cryptography, come hack on the latest and greatest protocol and help find problems before it is finalized. If you’re planning on attending, add your name to the Hackathon wiki. If you can’t make it, but implementing cryptographic protocols is your cup of tea, apply to join the CloudFlare team!
We’re very excited about TLS 1.3, which brings both security and performance improvements to HTTPS. In fact, if you have a client that speaks TLS 1.3 draft 10, you can read this blog on our TLS 1.3 mirror: tls13.cloudflare.com.
We hope to see you there!
TLS Certificate Optimization: The Technical Details behind “No Browser Left Behind”
Overview
Back in early December we announced our "no browser left behind" initiative to the world. Since then, we have served well over 500 billion SHA-1 certificates to visitors that otherwise would not have been able to communicate securely with our customers’ sites using HTTPS. All the while, we’ve continued to present newer SHA-2 certificates to modern browsers using the latest in elliptic curve cryptography, demonstrating that one does not have to sacrifice security to accommodate all the world’s Internet users. (If you weren’t able to acquire a SHA-1 certificate before CAs ceased issuing them on 2015/12/31, you can still sign up for a paid plan and we will immediately generate one to serve to your legacy visitors.)
Shortly after we announced these new benefits for our paid Universal SSL customers, we started hearing from other technology leaders who were implementing (or already had implemented) similar functionality. At first glance, the logic to identify incoming connections that only support SHA-1 seems straightforward, but as we spoke with our friends at Facebook, Twitter, and Mozilla, I realized that everyone was taking a slightly different approach. Complicating the matter even further was the fact that at CloudFlare we not only Continue reading
A Deep Dive Into DNS Packet Sizes: Why Smaller Packet Sizes Keep The Internet Safe
Yesterday we wrote about the 400 gigabit per second attacks we see on our network.
One way that attackers DDoS websites is by repeatedly doing DNS lookups that have small queries, but large answers. The attackers spoof their IP address so that the DNS answers are sent to the server they are attacking, this is called a reflection attack.
Domains with DNSSEC, because of the size of some responses, are usually ripe for this type of abuse, and many DNS providers struggle to combat DNSSEC-based DDoS attacks. Just last month, Akamai published a report on attacks using DNS lookups against their DNSSEC-signed .gov domains to DDoS other domains. They say they have seen 400 of these attacks since November.
To prevent any domain on CloudFlare being abused for a DNS amplification attack in this way, we took precautions to make sure most DNS answers we send fit in a 512 byte UDP packet, even when the zone is signed with DNSSEC. To do this, we had to be creative in our DNSSEC implementation. We chose a rarely-used-for-DNSSEC signature algorithm and even deprecated a DNS record type along the way.
Elliptic Curves: Keeping It Tight
Dutch mathematician Arjen Lenstra famously talks Continue reading
400Gbps: Winter of Whopping Weekend DDoS Attacks
Over the last month, we’ve been watching some of the largest distributed denial of service (DDoS) attacks ever seen unfold. As CloudFlare has grown we've brought on line systems capable of absorbing and accurately measuring attacks. Since we don't need to resort to crude techniques to block traffic we can measure and filter attacks with accuracy. Our systems sort bad packets from good, keep websites online and keep track of attack packet rates and bits per second.
The current spate of large attacks are all layer 3 (L3) DDoS. Layer 3 attacks consist of a large volume of packets hitting the target network, and the aim is usually to overwhelm the target network hardware or connectivity.
L3 attacks are dangerous because most of the time the only solution is to acquire large network capacity and buy beefy networking hardware, which is simply not an option for most independent website operators. Or, faced with huge packet rates, some providers simply turn off connections or entirely block IP addresses.
A Typical Day At CloudFlare
Historically, L3 attacks were the biggest headache for CloudFlare. Over the last two years, we’ve automated almost all of our L3 attack handling and these automatic systems protect Continue reading
Staying afloat: the DROWN Attack and CloudFlare
CloudFlare customers are automatically protected against the recently disclosed DROWN Attack. We do not have SSLv2 enabled on our servers.
We publish our SSL configuration here so that others can use it. We currently accept TLS 1.0, 1.1 and 1.2.
We are proactively testing our customers' origin web servers to detect vulnerable servers and will be reaching out to any that have a server that is vulnerable to DROWN.
In the interim, ensure that SSLv2 is fully disabled and/or that private keys are not shared with servers that still need to have SSLv2.
A tale of a DNS exploit: CVE-2015-7547
This post was written by Marek Vavruša and Jaime Cochran, who found out they were both independently working on the same glibc vulnerability attack vectors at 3am last Tuesday.
A buffer overflow error in GNU libc DNS stub resolver code was announced last week as CVE-2015-7547. While it doesn't have any nickname yet (last year's Ghost was more catchy), it is potentially disastrous as it affects any platform with recent GNU libc—CPEs, load balancers, servers and personal computers alike. The big question is: how exploitable is it in the real world?
It turns out that the only mitigation that works is patching. Please patch your systems now, then come back and read this blog post to understand why attempting to mitigate this attack by limiting DNS response sizes does not work.
But first, patch!
Man in the middle attack (MitM)
Let's start with the PoC from Google, it uses the first attack vector described in the vulnerability announcement. First, a 2048-byte UDP response forces buffer allocation, then a failure response forces a retry, and finally the last two answers smash the stack.
$ echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf
$ sudo python poc. Continue readingIntroducing CloudFlare Registrar: Designed for Security, Not the Masses
At CloudFlare, we’ve constructed one of the world’s largest networks purpose-built to protect our customers from a wide range of attacks. We’re so good at it that attackers increasingly look for ways to go around us, rather than go through us. One of the biggest risks for high-profile customers has been having their domain stolen at the registrar.
In 2013, we became intimately familiar with this problem when domains for the New York Times were hijacked and the newspaper’s CTO reached out to us to help get it back. We were able to assist, but the newspaper had its web and email traffic rerouted for hours.
Since the New York Times domain hijack, a number of other sites have had their domains stolen. We ourselves have seen multiple attempts to take control of CloudFlare’s registrar account. Thankfully, none have been successful—but some have gotten closer than we were comfortable with. Given the risk, we began looking for a registrar with security protocols that we could trust.
A Brief History of Registries and Registrars
In the early days of the Internet, domain registration was free. As the Internet began to take off, demand for domain registrations exploded. In 1993, unable to Continue reading
We’re hosting a Null Singapore meetup!
We're happy to announce that next week CloudFlare is hosting the Null Security meetup in Singapore. You are invited!
Null is a community for hackers and security enthusiasts. Monthly meetups are organized in a number of Asian cities. Read more at http://null.co.in/.
The lineup for the February meetup:
- All you ever wanted to know about DDoS attacks Marek Majkowski
- Security News Bytes Drupan Chandarana
- DNS Hijacking Michael Smith
If you’d like to sign up for the event, you can do so here:
What: Null Singapore - The Open Security Community meetup
When: February 24th: 6:45pm-8:45pm
Where: The Working Capitol, "The Commons" Room, 1 Keong Saik Road, Singapore 089109
CloudFlare is actively hiring in Singapore!
Padding oracles and the decline of CBC-mode cipher suites
At CloudFlare, we’re committed to making sure the encrypted web is available to everyone, even those with older browsers. At the same time, we want to make sure that as many people as possible are using the most modern and secure encryption available to them. Improving the cryptography used by the majority requires a coordinated effort between the organizations building web browsers and API clients and those working on web services like CloudFlare. Cryptography is a two-way street. Even if we support the most secure cryptographic algorithms for our customers, web visitors won’t get the benefit unless their web client supports the same algorithms.
In this blog post we explore the history of one widely used cryptographic mode that continues to cause problems: cipher block chaining (CBC). We’ll explain why CBC has proven difficult to use safely, and how recent trends in the adoption of secure ciphers by web clients have helped reduce the web’s reliance on this technology. From CloudFlare’s own data, we’ve seen the percentage of web clients that support safer cipher modes (such as AEAD) rise from under 50% to over 70% in six months, a good sign for the Internet.
What’s in a block cipher?
Ciphers Continue reading
Change the (S)Channel! Deconstructing the Microsoft TLS Session Resumption bug
Initial Problem Report
Several months ago we started hearing occasional reports from .NET developers that they were having trouble maintaining HTTPS sessions with one of our customer’s websites. Establishing connections worked just fine but they would periodically get disconnected, resulting in an exception that crashed their application. Around the same time, we also started hearing reports that two other Microsoft products—Internet Explorer and its heir-apparent, Edge—were also having trouble with our edge.
Just a few weeks prior, we had updated our handling of TLS session tickets to be more performant and more secure. We suspected these improvements were the trigger and focused our investigation there. What we learned was that the problem ran much deeper than .NET or IE. It went all the way down to the SChannel security package, which provides TLS functionality for a vast array of Microsoft applications.
TLS Session Tickets
Before diving into the story further, however, it’s helpful to understand exactly what TLS session tickets are, how they’re beneficial to HTTPS, and what optimizations CloudFlare does to use them at scale. (If you’d like to skip over the primer and jump right to the punchline, go ahead and click here.)
Overview
First introduced in Continue reading
CloudFlare’s Impact On The HTTP/2 “Universe”
CloudFlare released HTTP/2 support for all customers on December 3rd, 2015. Now, two months later, it's time to take a look at the impact of this release on the HTTP/2 "universe" and also at what has changed from a HTTP/2 vs. SPDY vs. HTTP 1.1 traffic ratio perspective.
HTTP/2 Usage
Previously, we showcased browser market share data from our own website. Using these numbers, we predicted the ratio of HTTP/2 traffic that we expected to see once enabled. Now, we can compare this original data set with updated data from the last 48 hours.
Below is the market share of HTTP/2 capable browsers that we saw on our website during a 48 hour period. The first one was before our HTTP/2 launch, the other one was last week. Both data sets were pulled from Google Analytics, and user agents were analyzed for HTTP/2 support.
| HTTP/2 capable browser | Global Market Share Late Nov 2015 | Global Market Share Late Jan 2016 |
|---|---|---|
| IE 11 on Windows 10 | 0.14% | 0.34% |
| Edge 12, and 13 | 0.35% | 0.48% |
| Firefox 36 - 45 | 5.09% | 11.05% |
| Chrome 41 - 49 | 15.06% | 38.86% |
| Safari 9 | 0.91% | 2.69% |
| Opera Continue reading |
Advanced Technical “Hacks” for your site’s SEO
Improving your site’s SEO is probably top of mind for you, but doing so takes a lot of hard work and the rules of the game are constantly changing. On Tuesday, January 26th at 10am PT/1pm ET, CloudFlare is hosting a live discussion with some of the leading experts in technical SEO. They will share advanced technical hacks to help you reap the benefits of higher search rankings. In the live discussion, Martin Woods, Reza Moaiandin, and Patrick Stox will cover:
- Tangible tips about on-page code excellency
- Semantic markup
- Web server optimization with GZIP and HTTP/2
- Web content optimization
- Site security with malware and DDoS prevention
In addition to the webinar, Reza and Martin from SALT.agency have offered a free 30 minute technical SEO consult on your website. Consults are limited to the first 50 people who signup here and also attend the live webinar event on January 26th at 10am PT. Be sure to register for the webinar, too.
CloudFlare launches new data centers in Oslo and Minneapolis
Four thousand miles (6,400 kilometers) separate CloudFlare’s latest two data centers: Oslo (#75) and Minneapolis (#76).
Oslo
In Oslo, we have now built our third data center in Scandinavia. This joins our existing facilities in Stockholm and Copenhagen. With a data center in Norway, we recognize an important country that stands above others with a staggering 95.05% of the population having Internet connectivity. This Internet penetration rate is the fourth best in the world. For reference, the Internet penetration rate in the US is 84%, the UK is 90% and Egypt, where we deployed our last data center it is only 50%
At 59.9500° N, Oslo is also the “northernmost” CloudFlare data center on our network map.
Oslo, according to the Norwegian Sagas is over 1,000 years old. CloudFlare has built itself into a facility just a handful of years old and while we respect all the wonderful history and tradition associated with Norway, we hope the locals appreciate our 21st century choice.
Norway has a very important position within the history of the Internet (well the ARPANET actually). In June 1973, the Royal Radar Establishment in Norway became one of the first international connections to Continue reading
Preventing Malicious Request Loops
The web is an collaborative ecosystem. Web standards exist to ensure that participants of the network behave in a predictable way. If network participants deviate from the established standards then there can be unintended consequences. This blog post is about one of these unintended consequences.
A group of researchers recently published a paper "Forwarding Loop Attacks in the Content Delivery Networks" describing what can happen when web services interact in a non-compliant way. They describe an attack where a malicious user can force multiple service providers to send each other an unending stream of requests in a loop. This request loop can result in resource exhaustion and denial of service at the service provider. This paper also demonstrated that the attack is practical, and can be performed using a large list of service providers.
CloudFlare's service has been modified to be standards-compliant with respect to HTTP proxying. However, fixing the vulnerability that enables this attack requires all proxy services to conform to the same standards. If even one service provider is non-compliant, the attack can still be carried out against compliant services. In this post, we will describe the attack and explain how a proxy services can go from being Continue reading