Author Archives: Sebastiaan Neuteboom
Author Archives: Sebastiaan Neuteboom
On January 8, 2026, a routine update to 1.1.1.1 aimed at reducing memory usage accidentally triggered a wave of DNS resolution failures for users across the Internet. The root cause wasn't an attack or an outage, but a subtle shift in the order of records within our DNS responses.
While most modern software treats the order of records in DNS responses as irrelevant, we discovered that some implementations expect CNAME records to appear before everything else. When that order changed, resolution started failing. This post explores the code change that caused the shift, why it broke specific DNS clients, and the 40-year-old protocol ambiguity that makes the "correct" order of a DNS response difficult to define.
All timestamps referenced are in Coordinated Universal Time (UTC).
Time | Description |
|---|---|
2025-12-02 | The record reordering is introduced to the 1.1.1.1 codebase |
2025-12-10 | The change is released to our testing environment |
2026-01-07 23:48 | A global release containing the change starts |
2026-01-08 17:40 | The release reaches 90% of servers |
2026-01-08 18:19 | Incident is declared |
2026-01-08 18:27 | The release is reverted |
2026-01-08 19:55 | Revert is completed. Impact ends |
While making some improvements to lower the memory usage of Continue reading