Sebastiaan Neuteboom
Author Archives: Sebastiaan Neuteboom
- Home → Author's archive:
Sebastiaan Neuteboom
When DNSSEC goes wrong: how we responded to the .de TLD outage
On May 5, 2026, at roughly 19:30 UTC, DENIC, the registry operator for the .de country-code top-level domain (TLD), started publishing incorrect DNSSEC signatures for the .de zone. Any validating DNS resolver receiving these signatures was required by the DNSSEC specification to reject them and return SERVFAIL to clients, including 1.1.1.1, the public DNS resolver operated by Cloudflare.
The country-code top-level domain for Germany, .de, is one of the largest on the Internet. On Cloudflare Radar, it consistently ranks among the most broadly queried TLDs globally. An outage at this level of the DNS hierarchy has the potential to make millions of domains unreachable.
In this post, we’ll walk through what we saw, the impact of these events, and how we applied temporary mitigations while DENIC resolved the issue.
DNSSEC (Domain Name System Security Extensions) adds cryptographic authentication to DNS. When a zone is signed with DNSSEC, each set of records is accompanied by a digital signature known as an RRSIG record that lets a resolver verify the records haven’t been tampered with. Unlike encrypted DNS protocols, such as DNS over TLS (DoT) and DNS over HTTPs (DoH), DNSSEC is about Continue reading
What came first: the CNAME or the A record?
On January 8, 2026, a routine update to 1.1.1.1 aimed at reducing memory usage accidentally triggered a wave of DNS resolution failures for users across the Internet. The root cause wasn't an attack or an outage, but a subtle shift in the order of records within our DNS responses.
While most modern software treats the order of records in DNS responses as irrelevant, we discovered that some implementations expect CNAME records to appear before everything else. When that order changed, resolution started failing. This post explores the code change that caused the shift, why it broke specific DNS clients, and the 40-year-old protocol ambiguity that makes the "correct" order of a DNS response difficult to define.
All timestamps referenced are in Coordinated Universal Time (UTC).
Time | Description |
|---|---|
2025-12-02 | The record reordering is introduced to the 1.1.1.1 codebase |
2025-12-10 | The change is released to our testing environment |
2026-01-07 23:48 | A global release containing the change starts |
2026-01-08 17:40 | The release reaches 90% of servers |
2026-01-08 18:19 | Incident is declared |
2026-01-08 18:27 | The release is reverted |
2026-01-08 19:55 | Revert is completed. Impact ends |
While making some improvements to lower the memory usage of Continue reading