Cloudflare’s COVID-19 FAQs
As the status of COVID-19 continues to impact people and businesses around the world, Cloudflare is committed to providing awareness and transparency to our customers, employees, and partners about how we are responding. We do not anticipate any significant disruptions in Cloudflare services.
Our Business Continuity Team is monitoring the situation closely and all company personnel are kept up to date via multiple internal communication channels including a live chat room. Customers and the public are encouraged to visit this blog post for the latest information.
You can check the status of our network at www.cloudflarestatus.com. For COVID-19-related questions that aren’t answered below, please contact our Customer Support Team.
Does Cloudflare have a Business Continuity Team (BCT)?
Yes, Cloudflare’s Business Continuity Team is a cross-functional, geographically diverse group dedicated to navigating through a health crisis like COVID-19 as well as a variety of other scenarios that may impact employee safety and business continuity.
What is Cloudflare’s Business Continuity Plan in the light of COVID-19?
In addition to Cloudflare’s existing Disaster Recovery Plan we have implemented the following strategies:
- Daily Business Continuity Team meetings to determine if updates, changes, or communication need to be provided to customers, partners, and Continue reading
Cloudflare During the Coronavirus Emergency
This email was sent to all Cloudflare customers a short while ago
From: Matthew Prince
Date: Thu, Mar 12, 2020 at 4:20 PM
Subject: Cloudflare During the Coronavirus Emergency
We know that organizations and individuals around the world depend on Cloudflare and our network. I wanted to send you a personal note to let you know how Cloudflare is dealing with the Coronavirus emergency.
First, the health and safety of our employees and customers is our top priority. We have implemented a number of sensible policies to this end, including encouraging many employees to work from home. This, however, hasn't slowed our operations. Our network operations center (NOC), security operations center (SOC), and customer support teams will remain fully operational and can do their jobs entirely remote as needed.
Second, we are tracking Internet usage patterns globally. As more people work from home, peak traffic in impacted regions has increased, on average, approximately 10%. In Italy, which has imposed a nationwide quarantine, peak Internet traffic is up 30%. Traffic patterns have also shifted so peak traffic is occurring earlier in the day in impacted regions. None of these traffic changes raise any concern for us. Cloudflare's network is well provisioned Continue reading
Protect your team with Cloudflare Gateway
On January 7th, we announced Cloudflare for Teams, a new way to protect organizations and their employees globally, without sacrificing performance. Cloudflare for Teams centers around two core products - Cloudflare Access and Cloudflare Gateway. Cloudflare Access is already available and used by thousands of teams around the world to secure internal applications. Cloudflare Gateway solves the other end of the problem by protecting those teams from security threats without sacrificing performance.
Today, we’re excited to announce new secure DNS filtering capabilities in Cloudflare Gateway. Cloudflare Gateway protects teams from threats like malware, phishing, ransomware, crypto-mining and other security threats. You can start using Cloudflare Gateway at dash.teams.cloudflare.com. Getting started takes less than five minutes.
Why Cloudflare Gateway?
We built Cloudflare Gateway to address key challenges our customers experience with managing and securing global networks. The root cause of these challenges is architecture and inability to scale. Legacy network security models solved problems in the 1990s, but teams have continued to attempt to force the Internet of the 2020s through them.
Historically, branch offices sent all of their Internet-bound traffic to one centralized data center at or near corporate headquarters. Administrators configured that to make sure all Continue reading
Open sourcing our Sentry SSO plugin
Cloudflare Access, part of Cloudflare for Teams, replaces legacy corporate VPNs with Cloudflare’s global network. Using your existing identity provider, Access enables your end users to login from anywhere — without a clunky agent or traffic backhaul through a centralized appliance or VPN.
Today, we are open sourcing a plugin that continues to improve that experience by making it easier for teams to use Cloudflare Access with one of the software industry’s most popular engineering tools, Sentry.
What is Sentry?
Sentry is an application that helps software teams find and diagnose errors in their products. We use Sentry here at Cloudflare. When you encounter an error when using a Cloudflare product, like our dashboard, we log that event. We then use Sentry to determine what went wrong.
Sentry can categorize and roll up errors, making it easy to identify new problems before investigating them with the tool’s event logging. Engineering managers here can use the dashboards to monitor the health of a new release. Product managers often use those reports as part of prioritizing what to fix next. Engineers on our team can dig into the individual errors as they release a fix.
Sentry is available in two forms: Continue reading
How Replicated Developers Develop Remotely
This is a guest post by Marc Campbell and Grant Miller, co-founders of Replicated.
Replicated is a 5-year old infrastructure software company working to make it easy for businesses to install and operate third party software. We don’t want you to have to send your data to a multi-tenant SaaS provider just to use their services. Our team is made up of twenty-two people distributed throughout the US. One thing that’s different about Replicated is our developers don’t actually store or execute code on their laptops; all of our development happens on remote instances in the cloud.
Our product, KOTS, runs in Kubernetes and manages the lifecycle of 3rd-party applications in the Kubernetes cluster. Building and validating the product requires a developer to have access to a cluster. But as we started to hire more and more engineers it became ridiculous to ask everyone to run their own local Kubernetes cluster. We needed to both simplify and secure our setup to allow every engineer to run their environment in the cloud, and we needed to do it in a way which was seamless and secure.
Previous Dev Environments with Docker for Mac
We started with each developer building Continue reading
How to Build a Highly Productive Remote Team (or Team of Contractors) with Cloudflare for Teams
Much of IT has been built on two outdated assumptions about how work is done. First, that employees all sit in the same building or branch offices. Second, that those employees will work full-time at the same company for years.
Both of these assumptions are no longer true.
Employees now work from anywhere. In the course of writing this blog post, I opened review tickets in our internal JIRA from my dining table at home. I reviewed internal wiki pages on my phone during my commute on the train. And I spent time reviewing some marketing materials in staging in our CMS.
In a past job, I would have suffered trying to connect to these tools through a VPN. That would have slowed down my work on a laptop and made it nearly impossible to use a phone to catch up on my commute.
The second challenge is ramp-up. I joined Cloudflare a few months ago. As a member of the marketing team, I work closely with our product organization and there are several dozen tools that I need to do that.
I’m hardly alone. The rise of SaaS and custom internal applications means that employees need access to all Continue reading
Cloudflare for Teams Free for Small Businesses During Coronavirus Emergency
There are a lot of people and businesses worldwide that are currently suffering, so I don't want to waste any time in getting to the point.
Beginning today, we are making our Cloudflare for Teams products free to small businesses around the world. Teams enables remote workers to operate securely and easily. We will continue this policy for at least the next 6 months. We're doing this to help ensure that small businesses that implement work from home policies in order to combat the spread of the virus can ensure business continuity. You can learn more and apply at: https://www.cloudflare.com/smallbusiness
We've also helped launch an online hub where small businesses can see technology services available to them for free or a substantial discount from multiple companies, during the Coronavirus Emergency: https://openforbusiness.org
To understand more about why we're doing this, read on.
The IT Strain of WFH
We have a team at Cloudflare carefully monitoring the spread of the SARS-Coronavirus-2, which is responsible for the COVID-19 respiratory disease. Like at many other companies, we have heeded the advice of medical professionals and government agencies and are increasingly allowing employees to work from home in impacted regions in order Continue reading
Why the 100th Anniversary of Women’s Right to Vote in the U.S. is Important to Celebrate on International Women’s Day
Seven months ago, I joined Cloudflare to work on the Public Policy Team focusing on our democracy projects such as Project Galileo, Athenian Project and Cloudflare for Campaigns. Since I joined the team, I have learned a lot about how important cybersecurity protections are for organizations that are the target of sophisticated cyberattacks, while also learning about the complex election security environment in the United States and abroad.
It seems fitting that on International Women’s Day, a day people throughout the world are celebrating the achievements of women, we also celebrate the Centennial Anniversary of the Women’s Suffrage Movement which was the tipping point that gave many women voting rights in the United States.
Since I have been working on Cloudflare’s election security projects, this day means something extra special to me and many of my colleagues who believe that voting is the cornerstone of democracy and that having access to information regarding voting and elections is essential.
Here are five reflections that I want to share on International Women’s Day and the Centennial Anniversary of the Nineteenth Amendment which granted women the right to vote in the United States:
1. The Women’s Suffrage Movement in the United States was Continue reading
How Cloudflare keeps employees productive from any location
Cloudflare employs more than 1,200 people in 13 different offices and maintains a network that operates in 200 cities. To do that, we used to suffer through a traditional corporate VPN that backhauled traffic through a physical VPN appliance. It was, frankly, horrible to work with as a user or IT person.
With today’s mix of on-prem, public cloud and SaaS and a workforce that needs to work from anywhere, be it a coffee shop or home, that model is no longer sustainable. As we grew in headcount, we were spending too much time resolving VPN helpdesk tickets. As offices around the world opened, we could not ask our workforce to sit as every connection had to go back through a central location.
We also had to be ready to scale. Some organizations are currently scrambling to load test their own VPN in the event that their entire workforce needs to work remotely during the COVID-19 outbreak. We could not let a single physical appliance constrain our ability to deliver 26M Internet properties to audiences around the world.
To run a network like Cloudflare, we needed to use Cloudflare’s network to stay fast and secure.
We built Cloudflare Access, part Continue reading
International Women’s Day 2020: Building a Modern Security Team
When we started at Cloudflare in the summer of 2018, we joined a small security team intent on helping it grow quickly. Cloudflare was already a successful “unicorn” startup and its profile was changing fast, providing cyber security protection for millions of Internet-facing properties and moving towards becoming a public company. We were excited to help build the team that would ensure the security of Cloudflare’s systems and the sensitive customer data that flows through them.
Competing for security talent in the tech industry - where every company is investing heavily on security - isn't easy. But, in 18 months, we have grown our team 400% from under 10 people to almost 50 (and still hiring). We are proud that 40% of our team are women and 25% are from an under-represented minority. We believe from experience, and the research shows, that more diverse teams drive better business results and can be a better place to work.
In honor of International Women’s Day this Sunday, we wanted to share some of our lessons learned on how to build a diverse team and inclusive culture on a modern security team.
Lessons Learned Building a Diverse Team
- Our effort to Continue reading
The History of the URL
This issue was occuring now because the ARPANET was on the verge of switching from its original NCP protocol, to the TCP/IP protocol which powers what we now call the Internet. With that switch suddenly there would be a multitude of interconnected networks (an ‘Inter... net’) requiring a more ‘hierarchical’ domain system where ARPANET could resolve its own domains while the other networks resolved theirs.
Other networks at the time had great names like “COMSAT”, “CHAOSNET”, “UCLNET” and “INTELPOSTNET” and were maintained by groups of universities and companies all around the US who wanted to be able to communicate, and could afford to lease 56k lines from the phone company and buy the requisite PDP-11s to handle routing.
In the original ARPANET design, a central Network Information Center Continue reading
Pwned Passwords Padding (ft. Lava Lamps and Workers)
The Pwned Passwords API (part of Troy Hunt’s Have I Been Pwned service) is used tens of millions of times each day, to alert users if their credentials are breached in a variety of online services, browser extensions and applications. Using Cloudflare, the API cached around 99% of requests, making it very efficient to run.
From today, we are offering a new security advancement in the Pwned Passwords API - API clients can receive responses padded with random data. This exists to effectively protect from any potential attack vectors which seek to use passive analysis of the size of API responses to identify which anonymised bucket a user is querying. I am hugely grateful to security researcher Matt Weir who I met at PasswordsCon in Stockholm and has explored proof-of-concept analysis of unpadded API responses in Pwned Passwords and has driven some of the work to consider the addition of padded responses.
Now, by passing a header of “Add-Padding” with a value of “true”, Pwned Passwords API users are able to request padded API responses (to a minimum of 800 entries with additional padding of a further 0-200 entries). The padding consists of randomly generated hash suffixes with the usage Continue reading
Why I’m Excited to Join Cloudflare as its First CIO
I am delighted to share that I have joined Cloudflare as its first Chief Information Officer to help scale the company in this new phase of its business. It’s an incredibly exciting time to be joining Cloudflare, and I am grateful for the opportunity to do my part to help build a better Internet.
At one of my previous companies, I made a bet on Cloudflare to equip us with security and performance solutions across a very decentralized global set of products and services. This is something that would have been very difficult without a cloud solution like Cloudflare’s. Since then I’ve been watching Cloudflare grow, and have always been very impressed by the speed of innovation and transparency, but also how Cloudflare operates: doing the right thing, with integrity, and above all building trust with customers and partners. The “do the right thing, even if it’s hard” mentality that I saw from Cloudflare since I started doing business with them as a customer, was key for me. When I heard that Cloudflare was looking for its first CIO I was excited to have a discussion to see if I could help.
During the interview process I got a sense Continue reading
RPKI and the RTR protocol
Today’s Internet requires stronger protection within its core routing system and as we have already said: it's high time to stop BGP route leaks and hijacks by deploying operationally-excellent RPKI!
Luckily, over the last year plus a lot of good work has happened in this arena. If you’ve been following the growth of RPKI’s validation data, then you’ll know that more and more networks are signing their routes and creating ROA’s or Route Origin Authorizations. These are cryptographically-signed assertions of the validity of an announced IP block and contribute to the further securing of the global routing table that makes for a safer Internet.
The protocol that we have not written much about is RTR. The Resource Public Key Infrastructure (RPKI) to Router Protocol - or RTR Protocol for short. Today we’re fixing that.
RPKI rewind
We have written a few times about RPKI (here and here). We have written about how Cloudflare both signs its announced routes and filters its routing inbound from other networks (both transits and peers) using RPKI data. We also added our efforts in the open-source software space with the release of the Cloudflare RPKI Toolkit.
The primary part of the RPKI (Resource Continue reading
Addressing the Web’s Client-Side Security Challenge
Modern web architecture relies heavily on JavaScript and enabling third-party code to make client-side network requests. These innovations are built on client-heavy frameworks such as Angular, Ember, React, and Backbone that leverage the processing power of the browser to enable the execution of code directly on the client interface/web browser. These third-party integrations provide richness (chat tools, images, fonts) or extract analytics (Google Analytics). Today, up to 70% of the code executing and rendering on your customer’s browser comes from these integrations. All of these software integrations provide avenues for potential vulnerabilities.
Unfortunately, these unmanaged, unmonitored integrations operate without security consideration, providing an expansive attack surface that attackers have routinely exploited to compromise websites. Today, only 2% of the Alexa 1000 global websites were found to deploy client-side security measures to protect websites and web applications against attacks such as Magecart, XSS, credit card skimming, session redirects and website defacement.
Improving website security and ensuring performance with Cloudflare Workers
In this post, we focus on how Cloudflare Workers can be used to improve security and ensure the high performance of web applications. Tala has joined Cloudflare’s marketplace to further our common goals of ensuring website security, preserving data privacy and Continue reading
When Bloom filters don’t bloom
I've known about Bloom filters (named after Burton Bloom) since university, but I haven't had an opportunity to use them in anger. Last month this changed - I became fascinated with the promise of this data structure, but I quickly realized it had some drawbacks. This blog post is the tale of my brief love affair with Bloom filters.
While doing research about IP spoofing, I needed to examine whether the source IP addresses extracted from packets reaching our servers were legitimate, depending on the geographical location of our data centers. For example, source IPs belonging to a legitimate Italian ISP should not arrive in a Brazilian datacenter. This problem might sound simple, but in the ever-evolving landscape of the internet this is far from easy. Suffice it to say I ended up with many large text files with data like this:
This reads as: the IP 192.0.2.1 was recorded reaching Cloudflare data center number 107 with a legitimate request. This data came from many sources, including our active and passive probes, logs of certain domains we own (like cloudflare.com), public sources (like BGP table), etc. The same line would usually be repeated across multiple Continue reading
Technical Details of Why Cloudflare Chose AMD EPYC for Gen X Servers
From the very beginning Cloudflare used Intel CPU-based servers (and, also, Intel components for things like NICs and SSDs). But we're always interested in optimizing the cost of running our service so that we can provide products at a low cost and high gross margin.
We're also mindful of events like the Spectre and Meltdown vulnerabilities and have been working with outside parties on research into mitigation and exploitation which we hope to publish later this year.
We looked very seriously at ARM-based CPUs and continue to keep our software up to date for the ARM architecture so that we can use ARM-based CPUs when the requests per watt is interesting to us.
In the meantime, we've deployed AMD's EPYC processors as part of Gen X server platform and for the first time are not using any Intel components at all. This week, we announced details of this tenth generation of servers. Below is a recap of why we're excited about the design, specifications, and performance of our newest hardware.
Servers for an Accelerated Future
Every server can run every service. This architectural decision has helped us achieve higher efficiency across the Cloudflare network. It has also given us more Continue reading
Going Beyond Black History Month
Around this time of year in the United States, African-Americans are often tasked with explaining why we spend 28 (or in the case of a leap year 29) days celebrating the contributions our ancestors made to this country. It may come in the form of responding to ignorant questions posed in learning environments or expressed in well-crafted articles lauding the relevancy of Black history in our modern time.
Black history is not only relevant, it is how we ensure that our heroes are not forgotten and that we have a viable future in our respective industries. As Carter G. Woodson famously said, “If a race has no history, if it has no worthwhile tradition, it becomes a negligible factor in the thought of the world, and it stands in danger of being exterminated.”
As the US leaders of Afroflare, Cloudflare’s employee resource group (ERG) for employees of African descent, we made a personal commitment this month and beyond to effectively represent, build, and grow at Cloudflare and in the tech industry.
To honor that commitment, we decided to tackle some commonly asked questions about the state of African-Americans in tech.
How many African-Americans work in tech?
The latest Continue reading
Securing Memory at EPYC Scale
Security is a serious business, one that we do not take lightly at Cloudflare. We have invested a lot of effort into ensuring that our services, both external and internal, are protected by meeting or exceeding industry best practices. Encryption is a huge part of our strategy as it is embedded in nearly every process we have. At Cloudflare, we encrypt data both in transit (on the network) and at rest (on the disk). Both practices address some of the most common vectors used to exfiltrate information and these measures serve to protect sensitive data from attackers but, what about data currently in use?
Can encryption or any technology eliminate all threats? No, but as Infrastructure Security, it’s our job to consider worst-case scenarios. For example, what if someone were to steal a server from one of our data centers? How can we leverage the most reliable, cutting edge, innovative technology to secure all data on that host if it were in the wrong hands? Would it be protected? And, in particular, what about the server’s RAM?
Data in random access memory (RAM) is usually stored in the clear. This can leave data vulnerable to software or hardware probing by Continue reading
Gen X Performance Tuning
We are using AMD 2nd Gen EPYC 7642 for our tenth generation “Gen X” servers. We found many aspects of this processor compelling such as its increase in performance due to its frequency bump and cache-to-core ratio. We have partnered with AMD to get the best performance out of this processor and today, we are highlighting our tuning efforts that led to an additional 6% performance.
Thermal Design Power & Dynamic Power
Thermal design power (TDP) and dynamic power, amongst others, play a critical role when tuning a system. Many share a common belief that thermal design power is the maximum or average power drawn by the processor. The 48-core AMD EPYC 7642 has a TDP rating of 225W which is just as high as the 64-core AMD EPYC 7742. It comes to mind that fewer cores should translate into lower power consumption, so why is the AMD EPYC 7642 expected to draw just as much power as the AMD EPYC 7742?
Let’s take a step back and understand that TDP does not always mean the maximum or average power that the processor will draw. At a glance, Continue reading