How to Build a Global Network that Complies with Local Law
We’ve spent a lot of time over the course of this week talking about Cloudflare engineers building technical solutions to improve privacy, increase control over data, and thereby, help our customers address regulatory challenges. But not all challenges can be solved with engineering. We sometimes have to build policies and procedures that anticipate our customers’ concerns. That has been an approach we’ve used to address government and other legal requests for data throughout the years.
Governments around the world have long had an interest in getting access to online records. Sometimes law enforcement is looking for evidence relevant to criminal investigations. Sometimes intelligence agencies are looking to learn more about what foreign governments or actors are doing. And online service providers of all kinds often serve as an access point for those electronic records.
For service providers like Cloudflare, though, those requests can be fraught. The work that law enforcement and other government authorities do is important. At the same time, the data that law enforcement and other government authorities are seeking does not belong to us. By using our services, our customers have put us in a position of trust over that data. Maintaining that trust is fundamental to Continue reading
Securing the post-quantum world
Quantum computing is inevitable; cryptography prepares for the future
Quantum computing began in the early 1980s. It operates on principles of quantum physics rather than the limitations of circuits and electricity, which is why it is capable of processing highly complex mathematical problems so efficiently. Quantum computing could one day achieve things that classical computing simply cannot.
The evolution of quantum computers has been slow. Still, work is accelerating, thanks to the efforts of academic institutions such as Oxford, MIT, and the University of Waterloo, as well as companies like IBM, Microsoft, Google, and Honeywell. IBM has held a leadership role in this innovation push and has named optimization the most likely application for consumers and organizations alike. Honeywell expects to release what it calls the “world’s most powerful quantum computer” for applications like fraud detection, optimization for trading strategies, security, machine learning, and chemistry and materials science.
In 2019, the Google Quantum Artificial Intelligence (AI) team announced that their 53-qubit (analogous to bits in classical computing) machine had achieved “quantum supremacy.” This was the first time a quantum computer was able to solve a problem faster than any classical computer in existence. This was considered a significant milestone.
Announcing Workplace Records for Cloudflare for Teams
We wanted to close out Privacy & Compliance Week by talking about something universal and certain: taxes. Businesses worldwide pay employment taxes based on where their employees do work. For most businesses and in normal times, where employees do work has been relatively easy to determine: it's where they come into the office. But 2020 has made everything more complicated, even taxes.
As businesses worldwide have shifted to remote work, employees have been working from "home" — wherever that may be. Some employees have taken this opportunity to venture further from where they usually are, sometimes crossing state and national borders.
In a lot of ways, it's gone better than expected. We're proud of helping provide technology solutions like Cloudflare for Teams that allow employees to work from anywhere and ensure they still have a fast, secure connection to their corporate resources. But increasingly we've been hearing from the heads of the finance, legal, and HR departments of our customers with a concern: "If I don't know where my employees are, I have no idea where I need to pay taxes."
Today we're announcing the beta of a new feature for Cloudflare for Teams to help solve this problem: Continue reading
Cloudflare Certifications
At Cloudflare, we prioritize initiatives that improve the security and privacy of our products and services. The security organization believes trust and transparency are foundational principles that are ingrained in what we build, the policies we set, and the data we protect. Many of our enterprise customers have stringent regulatory compliance obligations and require their cloud service providers like ourselves to provide assurance that we meet and exceed industry security standards. In the last couple of years, we’ve decided to invest in ways to make the evaluation of our security posture easier. We did so not only by obtaining recognized security certifications and reports in an aggressive timeline, but we also built a team that partners with our customers to provide transparency into our security and privacy practices.
Security Certifications & Reports
We understand the importance of providing transparency into our security processes, controls, and how our customers can continuously rely on them to operate effectively. Cloudflare complies with and supports the following standards:
SOC-2 Type II / SOC 3 (Service Organizations Controls) - Cloudflare maintains SOC reports that include the security, confidentiality, and availability trust principles. The SOC-2 report provides assurance that our products and underlying infrastructure are secure Continue reading
Cloudflare’s privacy-first Web Analytics is now available for everyone
In September, we announced that we’re building a new, free Web Analytics product for the whole web. Today, I’m excited to announce that anyone can now sign up to use our new Web Analytics — even without changing your DNS settings. In other words, Cloudflare Web Analytics can now be deployed by adding an HTML snippet (in the same way many other popular web analytics tools are) making it easier than ever to use privacy-first tools to understand visitor behavior.
Why does the web need another analytics service?
Popular analytics vendors have business models driven by ad revenue. Using them implies a bargain: they track visitor behavior and create buyer profiles to retarget your visitors with ads; in exchange, you get free analytics.
At Cloudflare, our mission is to help build a better Internet, and part of that is to deliver essential web analytics to everyone with a website, without compromising user privacy. For free. We’ve never been interested in tracking users or selling advertising. We don’t want to know what you do on the Internet — it’s not our business.
Our customers have long relied on Cloudflare’s Analytics because we’re accurate, fast, and privacy-first. In September we released a Continue reading
Deprecating the __cfduid cookie
Cloudflare is deprecating the __cfduid cookie. Starting on 10 May 2021, we will stop adding a “Set-Cookie” header on all HTTP responses. The last __cfduid cookies will expire 30 days after that.
We never used the __cfduid cookie for any purpose other than providing critical performance and security services on behalf of our customers. Although, we must admit, calling it something with “uid” in it really made it sound like it was some sort of user ID. It wasn't. Cloudflare never tracks end users across sites or sells their personal data. However, we didn't want there to be any questions about our cookie use, and we don’t want any customer to think they need a cookie banner because of what we do.
So why did we use the __cfduid cookie before, and why can we remove it now?
The primary use of the cookie is for detecting bots on the web. Malicious bots may disrupt a service that has been explicitly requested by an end user (through DDoS attacks) or compromise the security of a user's account (e.g. through brute force password cracking or credential stuffing, among others). We use many signals to build machine learning models that can Continue reading
Improving DNS Privacy with Oblivious DoH in 1.1.1.1
Today we are announcing support for a new proposed DNS standard — co-authored by engineers from Cloudflare, Apple, and Fastly — that separates IP addresses from queries, so that no single entity can see both at the same time. Even better, we’ve made source code available, so anyone can try out ODoH, or run their own ODoH service!
But first, a bit of context. The Domain Name System (DNS) is the foundation of a human-usable Internet. It maps usable domain names, such as cloudflare.com, to IP addresses and other information needed to connect to that domain. A quick primer about the importance and issues with DNS can be read in a previous blog post. For this post, it’s enough to know that, in the initial design and still dominant usage of DNS, queries are sent in cleartext. This means anyone on the network path between your device and the DNS resolver can see both the query that contains the hostname (or website) you want, as well as the IP address that identifies your device.
To safeguard DNS from onlookers and third parties, the IETF standardized DNS encryption with DNS over HTTPS (DoH) and DNS over TLS (DoT). Both protocols Continue reading
OPAQUE: The Best Passwords Never Leave your Device
Passwords are a problem. They are a problem for reasons that are familiar to most readers. For us at Cloudflare, the problem lies much deeper and broader. Most readers will immediately acknowledge that passwords are hard to remember and manage, especially as password requirements grow increasingly complex. Luckily there are great software packages and browser add-ons to help manage passwords. Unfortunately, the greater underlying problem is beyond the reaches of software to solve.
The fundamental password problem is simple to explain, but hard to solve: A password that leaves your possession is guaranteed to sacrifice security, no matter its complexity or how hard it may be to guess. Passwords are insecure by their very existence.
You might say, “but passwords are always stored in encrypted format!” That would be great. More accurately, they are likely stored as a salted hash, as explained below. Even worse is that there is no way to verify the way that passwords are stored, and so we can assume that on some servers passwords are stored in cleartext. The truth is that even responsibly stored passwords can be leaked and broken, albeit (and thankfully) with enormous effort. An increasingly pressing problem stems from the Continue reading
Good-bye ESNI, hello ECH!
Most communication on the modern Internet is encrypted to ensure that its content is intelligible only to the endpoints, i.e., client and server. Encryption, however, requires a key and so the endpoints must agree on an encryption key without revealing the key to would-be attackers. The most widely used cryptographic protocol for this task, called key exchange, is the Transport Layer Security (TLS) handshake.
In this post we'll dive into Encrypted Client Hello (ECH), a new extension for TLS that promises to significantly enhance the privacy of this critical Internet protocol. Today, a number of privacy-sensitive parameters of the TLS connection are negotiated in the clear. This leaves a trove of metadata available to network observers, including the endpoints' identities, how they use the connection, and so on.
ECH encrypts the full handshake so that this metadata is kept secret. Crucially, this closes a long-standing privacy leak by protecting the Server Name Indication (SNI) from eavesdroppers on the network. Encrypting the SNI secret is important because it is the clearest signal of which server a given client is communicating with. However, and perhaps more significantly, ECH also lays the groundwork for adding future security features and performance Continue reading
Helping build the next generation of privacy-preserving protocols
Over the last ten years, Cloudflare has become an important part of Internet infrastructure, powering websites, APIs, and web services to help make them more secure and efficient. The Internet is growing in terms of its capacity and the number of people using it and evolving in terms of its design and functionality. As a player in the Internet ecosystem, Cloudflare has a responsibility to help the Internet grow in a way that respects and provides value for its users. Today, we’re making several announcements around improving Internet protocols with respect to something important to our customers and Internet users worldwide: privacy.
These initiatives are:
- Fixing one of the last information leaks in HTTPS through Encrypted Client Hello (ECH), previously known as Encrypted SNI
- Making DNS even more private by supporting Oblivious DNS-over-HTTPS (ODoH)
- Developing a superior protocol for password authentication, OPAQUE, that makes password breaches less likely to occur
Each of these projects impacts an aspect of the Internet that influences our online lives and digital footprints. Whether we know it or not, there is a lot of private information about us and our lives floating around online. This is something we can help fix.
For over Continue reading
Privacy needs to be built into the Internet
The first phase of the Internet lasted until the early 1990s. During that time it was created and debugged, and grew globally. Its growth was not hampered by concerns about data security or privacy. Until the 1990s the race was for connectivity.
Connectivity meant that people could get online and use the Internet wherever they were. Because the “inter” in Internet implied interoperability the network was able to grow rapidly using a variety of technologies. Think dialup modems using ordinary phones lines, cable modems sending the Internet over coax originally designed for television, Ethernet, and, later, fibre optic connections and WiFi.
By the 1990s, the Internet was being used widely and for uses far beyond its academic origins. Early web pioneers, like Netscape, realized that the potential for e-commerce was gigantic but would be held back if people couldn’t have confidence in the security of online transactions.
Thus, with the introduction of SSL in 1994, the Internet moved to a second phase where security became paramount. Securing the web, and the Internet more generally, helped create the dotcom rush and the secure, online world we live in today. But this security was misunderstood by some as providing guarantees about privacy Continue reading
Introducing the Cloudflare Data Localization Suite
Today we’re excited to announce the Cloudflare Data Localization Suite, which helps businesses get the performance and security benefits of Cloudflare’s global network, while making it easy to set rules and controls at the edge about where their data is stored and protected.
The Data Localization Suite is available now as an add-on for Enterprise customers.
Cloudflare’s network is private and compliant by design. Preserving end-user privacy is core to our mission of helping to build a better Internet; we’ve never sold personal data about customers or end users of our network. We comply with laws like GDPR and maintain certifications such as ISO-27001.
Today, we're announcing tools that make it simple for our customers to build the same rigor into their own applications. In this post, I'll explain the different types of data that we process and how the Data Localization Suite keeps this data local.
We’ll also talk about how Cloudflare makes it possible to build applications that comply with data locality laws, while remaining fast, secure and scalable.
Why keep data local?
Cloudflare's customers have increasing desire or face legal requirements for data locality: they want to control the geographic location where their data is handled. Continue reading
Welcome to Privacy & Compliance Week: Reflecting Values at Cloudflare’s Core
Tomorrow kicks off Cloudflare's Privacy & Compliance Week. Over the course of the week, we'll be announcing ways that our customers can use our service to ensure they are in compliance with an increasingly complicated set of rules and laws around the world.
Early in Cloudflare's history, when Michelle, Lee, and I were talking about the business we wanted to build, we kept coming back to the word trust. We realized early on that if we were not trustworthy then no one would ever choose to route their Internet traffic through us. Above all else, we are in the trust business.
Every employee at Cloudflare goes through orientation. I teach one of the sessions titled "What Is Cloudflare?" I fill several white boards with notes and diagrams talking about where we fit in to the market. But I leave one for the end so I can write the word TRUST, in capital letters, and underline it three times. Trust is the foundation of our business.
Standing Up For Our Customers from Our Early Days
That's why we've made decisions that other companies may not have. In January 2013 the FBI showed up at our door with a National Security Continue reading
An introduction to three-phase power and PDUs
Our fleet of over 200 locations comprises various generations of servers and routers. And with the ever changing landscape of services and computing demands, it’s imperative that we manage power in our data centers right. This blog is a brief Electrical Engineering 101 session going over specifically how power distribution units (PDU) work, along with some good practices on how we use them. It appears to me that we could all use a bit more knowledge on this topic, and more love and appreciation of something that’s critical but usually taken for granted, like hot showers and opposable thumbs.
A PDU is a device used in data centers to distribute power to multiple rack-mounted machines. It’s an industrial grade power strip typically designed to power an average consumption of about seven US households. Advanced models have monitoring features and can be accessed via SSH or webGUI to turn on and off power outlets. How we choose a PDU depends on what country the data center is and what it provides in terms of voltage, phase, and plug type.
For each of our racks, all of our dual power-supply (PSU) servers are cabled to one of the two vertically mounted PDUs. Continue reading
Diversity and The Digital Divide: Thoughts From Tech Leaders
Leaders from across the tech industry and beyond recently joined us for Cloudflare’s Birthday Week, helping us celebrate Cloudflare’s 10th birthday. Many of them touched on the importance of diversity and making the Internet accessible to everyone.
Here are some of the highlights.
On the value of soliciting feedback
Selina Tobaccowala
Chief Digital Officer at Openfit, Co-Founder of Gixo
Former President & CTO of SurveyMonkey
When you think about diversity and inclusion, unfortunately, it's often only the loudest voice, the squeakiest wheel [who gets heard]. And what a survey allows you to do is let people's voices be heard who are not always willing to raise their hand or speak the loudest.
So at SurveyMonkey, we always made sure that when we were thinking about user testing and we were thinking about usability testing — that it was that broad swath of the customer because you wanted people across all different segments to submit their opinion.
I think that collecting data in a way that can be anonymized, collecting data in a way that lets people have a thoughtful versus always off the cuff conversation is really important. And what we also provided was a benchmarking product, because if you Continue reading
Improving the Resiliency of Our Infrastructure DNS Zone
In this blog post we will discuss how we made our infrastructure DNS zone more reliable by using multiple primary nameservers to leverage our own DNS product running on our edge as well as a third-party DNS provider.
Authoritative Nameservers
You can think of an authoritative nameserver as the source of truth for the records of a given DNS zone. When a recursive resolver wants to look up a record, it will eventually need to talk to the authoritative nameserver(s) for the zone in question. If you’d like to read more on the topic, our learning center provides some additional information.
Here’s an example of our authoritative nameservers (replacing our actual domain with example.com):
~$ dig NS example.com +short
ns1.example.com.
ns2.example.com.
ns3.example.com.
As you can see, there are three nameservers listed. You’ll notice that the nameservers happen to reside in the same zone, but they don’t have to. Those three nameservers point to six anycasted IP addresses (3 x IPv4, 3 x IPv6) announced from our edge, comprising data centers from 200+ cities around the world.
The Problem
We store the hostnames for all of our machines, both the ones at the Continue reading
A Thanksgiving 2020 Reading List
While our colleagues in the US are celebrating Thanksgiving this week and taking a long weekend off, there is a lot going on at Cloudflare. The EMEA team is having a full day on CloudflareTV with a series of live shows celebrating #CloudflareCareersDay.
So if you want to relax in an active and learning way this weekend, here are some of the topics we’ve covered on the Cloudflare blog this past week that you may find interesting.
Improving Performance and Search Rankings with Cloudflare for Fun and Profit
Making things fast is one of the things we do at Cloudflare. More responsive websites, apps, APIs, and networks directly translate into improved conversion and user experience. On November 10, Google announced that Google Search will directly take web performance and page experience data into account when ranking results on their search engine results pages (SERPs), beginning in May 2021.
Rustam Lalkaka and Rita Kozlov explain in this blog post how Google Search will prioritize results based on how pages score on Core Web Vitals, a measurement methodology Cloudflare has worked closely with Google to establish, and we have implemented support for in our analytics tools. Read the full blog post.
Getting Continue reading
ASICs at the Edge
At Cloudflare we pride ourselves in our global network that spans more than 200 cities in over 100 countries. To handle all the traffic passing through our network, there are multiple technologies at play. So let’s have a look at one of the cornerstones that makes all of this work… ASICs. No, not the running shoes.
What's an ASIC?
ASIC stands for Application Specific Integrated Circuit. The name already says it, it's a chip with a very narrow use case, geared towards a single application. This is in stark contrast to a CPU (Central Processing Unit), or even a GPU (Graphics Processing Unit). A CPU is designed and built for general purpose computation, and does a lot of things reasonably well. A GPU is more geared towards graphics (it's in the name), but in the last 15 years, there's been a drastic shift towards GPGPU (General Purpose GPU), in which technologies such as CUDA or OpenCL allow you to use the highly parallel nature of the GPU to do general purpose computing. A good example of GPU use is video encoding, or more recently, computer vision, used in applications such as self-driving cars.
Unlike CPUs or GPUs, ASICs are built Continue reading
A Byzantine failure in the real world
An analysis of the Cloudflare API availability incident on 2020-11-02
When we review design documents at Cloudflare, we are always on the lookout for Single Points of Failure (SPOFs). Eliminating these is a necessary step in architecting a system you can be confident in. Ironically, when you’re designing a system with built-in redundancy, you spend most of your time thinking about how well it functions when that redundancy is lost.
On November 2, 2020, Cloudflare had an incident that impacted the availability of the API and dashboard for six hours and 33 minutes. During this incident, the success rate for queries to our API periodically dipped as low as 75%, and the dashboard experience was as much as 80 times slower than normal. While Cloudflare’s edge is massively distributed across the world (and kept working without a hitch), Cloudflare’s control plane (API & dashboard) is made up of a large number of microservices that are redundant across two regions. For most services, the databases backing those microservices are only writable in one region at a time.
Each of Cloudflare’s control plane data centers has multiple racks of servers. Each of those racks has two switches that operate as a pair—both Continue reading
Building Black Friday e-commerce experiences with JAMstack and Cloudflare Workers
The idea of serverless is to allow developers to focus on writing code rather than operations — the hardest of which is scaling applications. A predictably great deal of traffic that flows through Cloudflare's network every year is Black Friday. As John wrote at the end of last year, Black Friday is the Internet's biggest online shopping day. In a past case study, we talked about how Cordial, a marketing automation platform, used Cloudflare Workers to reduce their API server latency and handle the busiest shopping day of the year without breaking a sweat.
The ability to handle immense scale is well-trodden territory for us on the Cloudflare blog, but scale is not always the first thing developers think about when building an application — developer experience is likely to come first. And developer experience is something Workers does just as well; through Wrangler and APIs like Workers KV, Workers is an awesome place to hack on new projects.
Over the past few weeks, I've been working on a sample open-source e-commerce app for selling software, educational products, and bundles. Inspired by Humble Bundle, it's built entirely on Workers, and it integrates powerfully with all kinds of first-class modern Continue reading