SAD DNS Explained

This week, at the ACM CCS 2020 conference, researchers from UC Riverside and Tsinghua University announced a new attack against the Domain Name System (DNS) called SAD DNS (Side channel AttackeD DNS). This attack leverages recent features of the networking stack in modern operating systems (like Linux) to allow attackers to revive a classic attack category: DNS cache poisoning. As part of a coordinated disclosure effort earlier this year, the researchers contacted Cloudflare and other major DNS providers and we are happy to announce that 1.1.1.1 Public Resolver is no longer vulnerable to this attack.

In this post, we’ll explain what the vulnerability was, how it relates to previous attacks of this sort, what mitigation measures we have taken to protect our users, and future directions the industry should consider to prevent this class of attacks from being a problem in the future.

DNS Basics

The Domain Name System (DNS) is what allows users of the Internet to get around without memorizing long sequences of numbers. What’s often called the “phonebook of the Internet” is more like a helpful system of translators that take natural language domain names (like blog.cloudflare.com or gov.uk) and Continue reading

How Cloudflare analyzes 1M DNS queries per second

On Friday, we announced DNS analytics for all Cloudflare customers. Because of our scale –– by the time you’ve finished reading this, Cloudflare DNS will have handled millions of DNS queries –– we had to be creative in our implementation. In this post, we’ll describe the systems that make up DNS Analytics which help us comb through trillions of these logs each month.

How logs come in from the edge

Cloudflare already has a data pipeline for HTTP logs. We wanted to utilize what we could of that system for the new DNS analytics. Every time one of our edge services gets an HTTP request, it generates a structured log message in the Cap’n Proto format and sends it to a local multiplexer service. Given the volume of the data, we chose not to record the full DNS message payload, only telemetry data we are interested in such as response code, size, or query name, which has allowed us to keep only ~150 bytes on average per message. It is then fused with processing metadata such as timing information and exceptions triggered during query processing. The benefit of fusing data and metadata at the edge is that we can spread Continue reading

Want to see your DNS analytics? We have a Grafana plugin for that

Curious where your DNS traffic is coming from, how much DNS traffic is on your domain, and what records people are querying for that don’t exist? We now have a Grafana plugin for you.

Grafana is an open source data visualization tool that you can use to integrate data from many sources into one cohesive dashboard, and even use it to set up alerts. We’re big Grafana fans here - we use Grafana internally for our ops metrics dashboards.

In the Cloudflare Grafana plugin, you can see the response code breakdown of your DNS traffic. During a random prefix flood, a common type of DNS DDoS attack where an attacker queries random subdomains to bypass DNS caches and overwhelm the origin nameservers, you will see the number of NXDOMAIN responses increase dramatically. It is also common during normal traffic to have a small amount of negative answers due to typos or clients searching for missing records.

You can also see the breakdown of queries by data center and by query type to understand where your traffic is coming from and what your domains are being queried for. This is very useful to identify localized issues, and to see how your Continue reading

A tale of a DNS exploit: CVE-2015-7547

This post was written by Marek Vavruša and Jaime Cochran, who found out they were both independently working on the same glibc vulnerability attack vectors at 3am last Tuesday.

A buffer overflow error in GNU libc DNS stub resolver code was announced last week as CVE-2015-7547. While it doesn't have any nickname yet (last year's Ghost was more catchy), it is potentially disastrous as it affects any platform with recent GNU libc—CPEs, load balancers, servers and personal computers alike. The big question is: how exploitable is it in the real world?

It turns out that the only mitigation that works is patching. Please patch your systems now, then come back and read this blog post to understand why attempting to mitigate this attack by limiting DNS response sizes does not work.

But first, patch!

Man in the middle attack (MitM)

Let's start with the PoC from Google, it uses the first attack vector described in the vulnerability announcement. First, a 2048-byte UDP response forces buffer allocation, then a failure response forces a retry, and finally the last two answers smash the stack.

$ echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf
$ sudo python poc. Continue reading