Building a new IMDB: Internet Mince Pie Database

CC-BY-SA 2.0 image by Phil! Gold

Since joining Cloudflare I’ve always known that as we grew, incredible things would be possible. It’s been a long held ambition to work in an organisation with the scale to answer a very controversial and difficult question. To do so would require a collection of individuals with a depth of experience, passion, dedication & above all collaborative spirit.

As Cloudflare’s London office has grown in the last 4 years I believe 2017 is the year we reach the tipping point where this is possible. A paradigm-shift in the type of challenges Cloudflare is able to tackle. We could finally sample every commercially available mince pie in existence before the 1st of December. In doing so, we would know conclusively which mince pie we should all be buying over Christmas to share with our friends & families.

What is a mince pie?

For the uninitiated, a Mince Pie is “a sweet pie of British origin, filled with a mixture of dried fruits and spices called mincemeat, that is traditionally served during the Christmas season in the English world.” - Wikipedia for Mince Pie

The original Mince Pie was typically filled with a mixture Continue reading

How we made our DNS stack 3x faster

Cloudflare is now well into its 6th year and providing authoritative DNS has been a core part of infrastructure from the start. We’ve since grown to be the largest and one of the fastest managed DNS services on the Internet, hosting DNS for nearly 100,000 of the Alexa top 1M sites and over 6 million other web properties – or DNS zones.

CC-BY 2.0 image by Steve Jurvetson

Today Cloudflare’s DNS service answers around 1 million queries per second – not including attack traffic – via a global anycast network. Naturally as a growing startup, the technology we used to handle tens or hundreds of thousands of zones a few years ago became outdated over time, and couldn't keep up with the millions we have today. Last year we decided to replace two core elements of our DNS infrastructure: the part of our DNS server that answers authoritative queries and the data pipeline which takes changes made by our customers to DNS records and distributes them to our edge machines across the globe.

The rough architecture of the system can be seen above. We store customer DNS records and other origin server information in a central database, convert the Continue reading