How Cloudflare analyzes 1M DNS queries per second
On Friday, we announced DNS analytics for all Cloudflare customers. Because of our scale –– by the time you’ve finished reading this, Cloudflare DNS will have handled millions of DNS queries –– we had to be creative in our implementation. In this post, we’ll describe the systems that make up DNS Analytics which help us comb through trillions of these logs each month.
How logs come in from the edge
Cloudflare already has a data pipeline for HTTP logs. We wanted to utilize what we could of that system for the new DNS analytics. Every time one of our edge services gets an HTTP request, it generates a structured log message in the Cap’n Proto format and sends it to a local multiplexer service. Given the volume of the data, we chose not to record the full DNS message payload, only telemetry data we are interested in such as response code, size, or query name, which has allowed us to keep only ~150 bytes on average per message. It is then fused with processing metadata such as timing information and exceptions triggered during query processing. The benefit of fusing data and metadata at the edge is that we can spread Continue reading
Colombo, Sri Lanka: Six million Internet properties now faster for six million Internet users
We are excited to add four new data centers this week to Cloudflare's growing network, beginning with Colombo, Sri Lanka. This deployment is our 112th data center globally, and our 38th in Asia.
Faster Performance
CC BY-NC-ND 2.0 image by Pavel Dobrovsky
Six million Internet properties using Cloudflare are now even faster across the island country of Sri Lanka. Previously, local visitors to Cloudflare customers were served out of our Singapore or Dubai data centers.
Latency (ms) decreases 4x to Cloudflare customers. Source: Cedexis
Sri Lanka added over one million Internet users in the past year alone. At ~30% Internet penetration, there is considerable room to grow.
Next Three Cities
Our deployments to be revealed later this week will provide additional redundancy to existing facilities in North America and Africa.
If you enjoy the idea of helping build one of the world's largest networks, come join our team!
Anonymity and Abuse Reports
Last Thursday, ProPublica published an article critiquing our handling of some abuse reports that we receive. Feedback from the article caused us to reevaluate how we handle abuse reports. As a result, we've decided to update our abuse reporting system to allow individuals reporting threats and child sexual abuse material to do so anonymously. We are rolling this change out and expect it to be available by the end of the week.
I appreciate the feedback we received. How we handle abuse reports has evolved over the last six and a half years of Cloudflare's history. I wanted to take this opportunity to walk through some of the rationale that got us to this point and caused us to have a blindspot to the case that was highlighted in the article.
What Is Cloudflare?
Cloudflare is not a hosting provider. We do not store the definitive copy of any of the content that someone may want to file an abuse claim about. If we terminate a customer it doesn’t make the content go away. Instead, we are more akin to a specialized network. One of the functions of the network that we provide is to add security to the content Continue reading
Meet The Brand New DNS Analytics Dashboard
Have you noticed something new in your Cloudflare analytics dashboard this morning? You can now see detailed DNS analytics for your domains on Cloudflare.
If you want to skip to the punch and start exploring, go check it out here. Otherwise, hop on the DNS magic school bus - and let us show you all the neat stats in your now-available DNS analytics.
DNS analytics dashboard: What does it know? Does it know things? Let’s find out.
At the top of the DNS analytics dashboard you can see your DNS traffic health. This “Queries by Response Codes” graph breaks down queries by what response code Cloudflare DNS answered to the visitor. Like HTTP response codes, DNS response codes give an indication of what is happening behind the scenes. Mostly you will just see NOERROR, the HTTP 200 of DNS response codes, and NXDOMAIN, the HTTP 404 of DNS response codes. NXDOMAIN is particularly interesting - what are people querying for that doesn’t exist?
If you are an enterprise customer and you want to know what all the NXDOMAIN queries are, just scroll down a little bit where we show you the top queries for your domain and top queries for Continue reading
Introducing the new Cloudflare Community Forum
Cloudflare’s community of users is vast. With more than 6 million domains registered, our users come in all shapes and sizes and are located all over the world. They can also frequently be found hanging out all around the web, from social media platforms, to Q&A sites, to any number of personal interest forums. Cloudflare users have questions to ask and an awful lot of expertise to share.
It’s with that in mind that we wanted to give Cloudflare users a more centralized location to gather, and to discuss all things Cloudflare. So we have launched a new Cloudflare Community at community.cloudflare.com.
Who is this community for?
It's for anyone and everyone who uses Cloudflare. Whether you are adding your first domain and don’t know what a name server is, or you are managing 1,000s of domains via API, or you are somewhere in between. In the Cloudflare Community you will be able to find tips, tricks, troubleshooting guidance, and recommendations.
We also think this will be a great way to get feedback from users on what’s working for them, what isn’t, and ways that we can make Cloudflare better. There will even be opportunities to Continue reading
How eero mesh WiFi routers connect to the cloud
This is a guest post by Gabe Kassel, Product Manager for Embedded Software at eero.
Relying on a single wireless router to provide internet in every room of the home is like expecting a single light bulb to illuminate the entire house. It’s physics - WiFi radio waves don’t travel through walls or objects easily. The eero Home WiFi System is a new take on home connectivity, bucking the trend of one high-powered device in the center of the home. Instead, eero uses multiple access points that talk to each other via our proprietary mesh technology -- TrueMesh -- to spread coverage and a high throughput connection throughout a home.
eero’s hardware - its distributed access point system - solves the problem of spreading a consistent, stable WiFi signal in a home. But hardware is only part of the puzzle. On the backend of eero’s technology, we face different challenges: how do we build a highly available, high performance infrastructure that’s able to communicate with each eero device? We’ve discussed parts of our architecture previously, but we haven’t yet explored into how we use Cloudflare to eliminate one “single-point-of-failure” in our architecture.
How eeros interact with the cloud
eero Continue reading
IoT Security Anti-Patterns
From security cameras to traffic lights, an increasing amount of appliances we interact with on a daily basis are internet connected. A device can be considered IoT-enabled when the functionality offered by its Embedded System is exposed through an internet connected API.
Internet-of-Things technologies inherit many attack vectors that appear in other internet connected devices, however low-powered hardware-centric nature of embedded systems presents them with unique security threats. Engineers building Internet-of-Things devices must take additional precautions to ensure they do not implement security anti-patterns when addressing new problems, this blog post will investigate four such anti-patterns that have been used by real Internet-of-Things devices.
Atmel ATMEGA8 Microcontroller Wikimedia Commons - CC BY-SA 3.0
HTTP Pub/Sub
Every time your IoT-enabled alarm clock sounds, you may want it to tell your coffee machine to brew some coffee. In order to do this, your coffee machine may subscribe to messages published by your alarm clock. One such way of doing this is to implement the Publish/Subscribe Pattern within the API of the IoT devices, for this example let's assume our alarm clock and coffee machine communicate through HTTP.
In order to subscribe to messages from the alarm clock, the coffee machine sends Continue reading
Introducing TLS with Client Authentication
In a traditional TLS handshake, the client authenticates the server, and the server doesn’t know too much about the client. However, starting now, Cloudflare is offering enterprise customers TLS with client authentication, meaning that the server additionally authenticates that the client connecting to it is authorized to connect.
TLS Client Authentication is useful in cases where a server is keeping track of hundreds of thousands or millions of clients, as in IoT, or in a mobile app with millions of installs exchanging secure information. For example, an IoT company can issue a unique client certificate per device, and then limit connections to their IoT infrastructure to only their devices by blocking connections where the client doesn’t present a certificate signed by the company’s certificate authority.
Or in the case of a mobile banking app, where the bank wants to ensure customers’ secure financial data doesn’t get stolen by bots spoofing their mobile app, they can issue a unique certificate to every app install and in the TLS handshake validate requests are coming from their mobile app. Client authentication is also useful for VPNs, enterprise networks or staging sites, where corporations and developers need to lock down connections to only laptops Continue reading
Upcoming Cloudflare events: Berlin May 5-7, Austin & Portland May 11
Attending JS Conf EU, CSS Conf, or OSCON in the next couple of weeks? Live in Berlin or Austin or Portland? Come over and join Cloudflare devs in the area at our upcoming events.
JS Conf EU 2016. Photo by Holger Blank.
In Berlin? Attending JS Conf EU or CSS Conf EU?
If you’re at JS Conf EU (May 6-7) or CSS Conf EU (May 5):
- Be sure to make it to John Fawcett’s session, “Invisible Code: Building JavaScript Libraries For Non-Technical People.”
- Made a Cloudflare App? Drop by the Cloudflare Hacker Lounge to show me and I have a special gift for you.
- Started working on a Cloudflare App and have questions? Drop by the Cloudflare Hacker Lounge and get them answered.
Just happen to be in Berlin? Tweet @qiqing and @IcyApril to come hang out with us in person.
Cloudflare Apps Preview. Visualize your app here.
In Austin? Attending OSCON?
Join the core developers of Cloudflare Apps for the inaugural Cloudflare meetup in Austin. It will feature an introduction by Zack Bloom (tech lead) to the new Cloudflare Apps including details of how apps get created, moderated, installed, and served to millions of users Continue reading
Introducing Cloudflare Orbit: A Private Network for IoT Devices
In October, we wrote about a 1.75M rps DDoS attack we mitigated on our network, launched by 52,467 unique IP’s, mostly hacked CCTV cameras.
We continued to see more IoT devices in DDoS attacks, and so we started to put together a security solution to protect the devices from becoming part of the botnet in the first place. Today we’re announcing it: Cloudflare Orbit.
PC-era security doesn’t work in IoT-era computing
As we talked to IoT companies, over and over again we heard the same thing. In the consumer electronics space, IoT manufacturers were telling us that they were shipping patches to their devices, but their end users didn’t always download and install them. (Reserve your judgment, how many times have you pressed ignore when your phone asked you to update its operating system?) In the industrial control, medical and automotive spaces, where devices are used in life-critical functions, we heard a different story. Even if someone wanted to apply a patch, it just wasn’t that easy. For example, even if the manager of a nuclear power plant wants to update software on their thermostats, shutting down operations long enough to do that means the update has to Continue reading
AES-CBC is going the way of the dodo
A little over a year ago, Nick Sullivan talked about the beginning of the end for AES-CBC cipher suites, following a plethora of attacks on this cipher mode.
Today we can safely confirm that this prediction is coming true, as for the first time ever the share of AES-CBC cipher suites on Cloudflare’s edge network dropped below that of ChaCha20-Poly1305 suites, and is fast approaching the 10% mark.
Over the course of the last six months, AES-CBC shed more than 33% of its “market” share, dropping from 20% to just 13.4%.
All of that share, went to AES-GCM, that currently encrypts over 71.2% of all connections. ChaCha20-Poly1305 is stable, with 15.3% of all connections opting for that cipher. Surprisingly 3DES is still around, with 0.1% of the connections.
The internal AES-CBC cipher suite breakdown as follows:
The majority of AES-CBC connections use ECDHE-RSA or RSA key exchange, and not ECDHE-ECDSA, which implies that we mostly deal with older clients.
RSA is also dying
In other good new, the use of ECDSA surpassed that of RSA at the beginning of the year. Currently more than 60% of all connections use Continue reading
Introducing SSL for SaaS
If you’re running a SaaS company, you know how important it is that your application is performant, highly available, and hardened against attack. Your customers—and your revenue stream—depend on it. Putting your app behind a solution such as Cloudflare is an obvious move for your own infrastructure, but how do you securely (and easily) extend these benefits to your customers?
If your customers interact with your app on your domain and don’t care about branding under their custom or “vanity” domain (or aren’t paying you for the ability to do so), the solution is straightforward: onboard your domain to Cloudflare and serve the app at either https://app.yourcompany.ltd or https://yourcustomer.yourcompany.ltd. But if your customers want to host your application, portal, content management solution, etc. on their own domain for improved SEO and discoverability, e.g., https://app.yourcustomer.site the solution is not so easy.
Easily extend the benefits of Cloudflare to your customers, one hostname at a time
Until today, your best bet was to ask them to CNAME over to your infrastructure, have them generate a private key and CSR, send the latter to a CA for signing, and then securely provide you with the Continue reading
Budapest (Hungary): Cloudflare Data Center #111
Hot on the heels of several recent data center additions in Yerevan, Quito, Rome, Kansas City, Belgrade, Curacao, Djibouti and Munich, we are delighted to announce our newest deployment in Budapest, making six million websites even faster and safer across Hungary.
Until today, Hungarian visitors to these Internet properties were principally served out of our Frankfurt data center 1,000 km away, or from Vienna. We are happy to further reduce their latency to over 8 million Internet users.
CC BY 2.0 image by Moyann Brenn
Beautiful Budapest
Budapest is one of the most beautiful cities in the world, with must-see sites such as Halászbástya (Fisherman's Bastion), Az Országház (House of the Nation - The Hungarian Parliament), and the Széchenyi Chain Bridge by the Danube. We love this aerial video with breathtaking views of the city created by Milan Heal (Drone Travel Guides).
We have new facilities in the works across five continents. Watch out for even more additions to our growing network.
-The Cloudflare Team
The Cloudflare network today
Ecommerce websites on Cloudflare: best practices
Cloudflare provides numerous benefits to ecommerce sites, including advanced DDOS protection and an industry-leading Web Application Firewall (WAF) that helps secure your transactions and protect customers’ private data.
A key Cloudflare feature is caching, which allows content to be served closer to the end user from our global network of data centers. Doing so improves the user's shopping experience and contributes to increasing the proportion of people completing a purchase (conversion rate).
For example:
- Walmart found improving page load time by 1 second increased their conversion rate by 2%
- Research for Amazon showed every 0.1 second of delay costs 1% of sales
- The Barack Obama campaign website saw an 80% page load time boost resulted in a 14% increase in donations
What is caching?
Cloudflare operates over 110 data centers around the world. When a website implements Cloudflare, visitor requests for the site will proxy through the nearest Cloudflare data center instead of connecting directly to the webserver hosting the site (origin). This means Cloudflare can store content such as images, JavaScript, CSS and HTML on our servers, speeding up access to those resources for end-users.
Most ecommerce websites rely on a backend database containing product descriptions and metadata Continue reading
Understanding Our Cache and the Web Cache Deception Attack
About a month ago, security researcher Omer Gil published the details of an attack that he calls the Web Cache Deception attack. It works against sites that sit behind a reverse proxy (like Cloudflare) and are misconfigured in a particular way. Unfortunately, the definition of "misconfigured" for the purposes of this attack changes depending on how the cache works. In this post, we're going to explain the attack and then describe the algorithm that our cache uses to decide whether or not to cache a given piece of content so that customers can be sure that they are secure against this attack.
The Attack
First, we'll explain the basics of the Web Cache Deception attack. For those who want a more in-depth explanation, Omer's original post is a great resource.
Imagine that you run the social media website example.com, and that each of your users has a newsfeed at example.com/newsfeed. When a user navigates to their newsfeed, the HTTP request generated by their browser might look something like this:
GET /newsfeed HTTP/1.1
Host: example.com
...
If you use Cloudflare, you don't want us to cache this request because if Continue reading
Cloudflare Rate Limiting – Insight, Control, and Mitigation against Layer 7 DDoS Attacks
Today, Cloudflare is extending its Rate Limiting service by allowing any of our customers to sign up. Our Enterprise customers have enjoyed the benefits of Cloudflare’s Rate Limiting offering for the past several months. As part of our mission to build a better internet, we believe that everyone should have the ability to sign up for the service to protect their websites and APIs.
CC-BY 2.0 image by Benjamin Child
Rate Limiting is one more feature in our arsenal of tools that help to protect our customers against denial-of-service attacks, brute-force password attempts, and other types of abusive behavior targeting the application layer. Application layer attacks are usually a barrage of HTTP/S requests which may look like they originate from real users, but are typically generated by machines (or bots). As a result, application layer attacks are often harder to detect and can more easily bring down a site, application, or API. Rate Limiting complements our existing DDoS protection services by providing control and insight into Layer 7 DDoS attacks.
Rate Limiting is now available to all customers across all plans as an optional paid feature. The first 10,000 qualifying requests are free, which allows customers to start using Continue reading
Data Transfers Post-Brexit: Smoothing the Transition
The average internet user consumes vast amounts of data on a daily basis but rarely – unless an avid follower of Max Schrems - thinks about how the data flows or the mechanisms and legal arrangements in place to make it all happen. If companies like Cloudflare are doing their job well behind the scenes, you really shouldn’t have to – it just works, and so you can busy yourself emailing, communicating, transacting and sharing information.
Users benefit enormously from the free movement of data, and it is a highly regarded feature of living and doing business within the European Union. With the appropriate legal protections in place, scientific and societal benefits also flow along with the data, and the quality of our lives is improved immensely.
And the internet is an increasingly busy place:
Image courtesy of @LoriLewis
and @OfficiallyChadd
Let it flow, let it flow...
The European Commission reported in a communication earlier this year that the European Data Economy – i.e the marketplace where digital data is exchanged as products or services derived from raw data – was estimated at EUR 272 billion in 2015, and that the value is expected to increase to EUR Continue reading
Munich: Cloudflare’s fifth data center in Germany
Oktoberfest has come early this year! We just turned up our latest data center in Munich: our 110th data center globally, and our fifth in Germany, joining Frankfurt, Düsseldorf, Berlin and Hamburg. Just over a mile away from the historic Hofbräuhaus, our newest deployment helps make six million websites faster across Bavaria, while providing additional redundancy and capacity to withstand attacks.
Oktoberfest kommt früh in diesem Jahr! Wir haben gerade unser 110. Rechenzentrum in München angeschaltet. Wir haben somit fünf Rechenzentren in Deutschland (Frankfurt, Düsseldorf, Berlin, Hamburg und München). Nur ca. 2 Kilometer entfernt vom historischen Hofbräuhaus, wird unser neustes Rechenzentrum sechs Millionen Webseiten im Raum Bayern schneller machen, und zugleich größere Kapazitäten gegen DDoS Attacken bereitstellen.
As TripAdvisor put it: Germany’s third largest city reflects the character of a city and a post card village at the same time.
Wie TripAdvisor es ausdrückt: Deutschlands drittgrößte Stadt spiegelt zugleich den Charakter einer Stadt und eines Dorfes wider.
CC BY-NC-ND 2.0 image by Werner Nieke
Continued Expansion
We have several additional locations being added to our fast expanding network that spans 55 countries. If you have a new city in mind, let us know, Continue reading
Changing Internet Standards to Build A Secure Internet
We’ve been working with registrars and registries in the IETF on making DNSSEC easier for domain owners, and over the next two weeks we’ll be starting out by enabling DNSSEC automatically for .dk domains.
DNSSEC: A Primer
Before we get into the details of how we've improved the DNSSEC experience, we should explain why DNSSEC is important and the function it plays in keeping the web safe.
DNSSEC’s role is to verify the integrity of DNS answers. When DNS was written in the early 1980’s, it was only a few researchers and academics on the internet. They all knew and trusted each other, and couldn’t imagine a world in which someone malicious would try to operate online. As a result, DNS relies on trust to operate. When a client asks for the address of a hostname like www.cloudflare.com, without DNSSEC it will trust basically any server that returns the response, even if it wasn’t the same server it originally asked. With DNSSEC, every DNS answer is signed so clients can verify answers haven’t been manipulated over transit.
The Trouble With DNSSEC
If DNSSEC is so important, why do so few domains support it? First, for a domain to Continue reading
How we made our DNS stack 3x faster
Cloudflare is now well into its 6th year and providing authoritative DNS has been a core part of infrastructure from the start. We’ve since grown to be the largest and one of the fastest managed DNS services on the Internet, hosting DNS for nearly 100,000 of the Alexa top 1M sites and over 6 million other web properties – or DNS zones.
CC-BY 2.0 image by Steve Jurvetson
Today Cloudflare’s DNS service answers around 1 million queries per second – not including attack traffic – via a global anycast network. Naturally as a growing startup, the technology we used to handle tens or hundreds of thousands of zones a few years ago became outdated over time, and couldn't keep up with the millions we have today. Last year we decided to replace two core elements of our DNS infrastructure: the part of our DNS server that answers authoritative queries and the data pipeline which takes changes made by our customers to DNS records and distributes them to our edge machines across the globe.
The rough architecture of the system can be seen above. We store customer DNS records and other origin server information in a central database, convert the Continue reading