Cloudflare Bot Management: machine learning and more
Introduction
Building Cloudflare Bot Management platform is an exhilarating experience. It blends Distributed Systems, Web Development, Machine Learning, Security and Research (and every discipline in between) while fighting ever-adaptive and motivated adversaries at the same time.
This is the ongoing story of Bot Management at Cloudflare and also an introduction to a series of blog posts about the detection mechanisms powering it. I’ll start with several definitions from the Bot Management world, then introduce the product and technical requirements, leading to an overview of the platform we’ve built. Finally, I’ll share details about the detection mechanisms powering our platform.
Let’s start with Bot Management’s nomenclature.
Some Definitions
Bot - an autonomous program on a network that can interact with computer systems or users, imitating or replacing a human user's behavior, performing repetitive tasks much faster than human users could.
Good bots - bots which are useful to businesses they interact with, e.g. search engine bots like Googlebot, Bingbot or bots that operate on social media platforms like Facebook Bot.
Bad bots - bots which are designed to perform malicious actions, ultimately hurting businesses, e.g. credential stuffing bots, third-party scraping bots, spam bots and sneakerbots.
Bot Management - blocking Continue reading
Cinco de Mayo – What are we celebrating anyway?
Greetings from Latinflare, Cloudflare’s LatinX Employee Resource Group, with members all over the US, the UK, and Portugal. Today is Cinco de Mayo! Americans everywhere will be drinking margaritas and eating chips and salsa. But what is this Mexican holiday really about and what exactly are we celebrating?
About Cinco de Mayo
Cinco de Mayo, Spanish for "Fifth of May", is an annual celebration held in Mexico on May 5th. The date is observed to commemorate the Mexican Army's victory over the French Empire at the Battle of Puebla, on May 5, 1862, under the leadership of General Ignacio Zaragoza. The victory of the smaller Mexican force against a larger French force was a boost to morale for the Mexicans. Zaragoza died months after the battle due to illness. A year after the battle, a larger French force defeated the Mexican army at the Second Battle of Puebla, and Mexico City soon fell to the invaders.
In the United States, Cinco de Mayo has taken on a significance beyond that in Mexico. More popularly celebrated in the United States than Mexico, the date has become associated with the celebration of Continue reading
Setting up Cloudflare for Teams as a Start-Up Business
Earlier this year, Cloudflare acquired S2 Systems. We were a start-up in Kirkland, Washington and now we are home to Cloudflare’s Seattle-area office.
Our team developed a new approach to remote browser isolation (RBI), a technology that runs your web browser in a cloud data center, stopping threats on the Internet from executing any code on your machine. The closer we can bring that data center to the user, the faster we can make that experience. Since the acquisition, we have been focused on running our RBI platform in every one of Cloudflare’s data centers in 200 cities around the world.
The RBI solution will join a product suite that we call Cloudflare for Teams, which consists of two products: Access and Gateway.
Those two products solve a number of problems that companies have with securing users, devices, and data. As a start-up, we struggled with a few of these challenges in really painful ways:
- How do we let prospects securely trial our RBI platform?
- How do we keep our small office secure without an IT staff?
- How can we connect to the powerful, but physically clunky and heavy development machines, when we are not in that office?
Dogfooding Continue reading
A single dashboard for Cloudflare for Teams
Starting today, Cloudflare Access can now be used in the Cloudflare for Teams dashboard. You can manage security policies for your people and devices in the same place that you build zero-trust rules to protect your applications and resources. Everything is now in one place in a single dashboard.
We are excited to launch a new UI that can be used across the entire Teams platform, but we didn’t build this dashboard just for the sake of a new look-and-feel. While migrating the Access dashboard, we focused on solving one of the largest sources of user confusion in the product.
This post breaks down why the original UI caused some headaches, how we think about objects in Cloudflare for Teams, and how we set out to fix the way we display that to our users.
Cloudflare Access
Cloudflare Access is one-half of Cloudflare for Teams, a security platform that runs on Cloudflare’s network. Teams protects users, devices and data without compromising experience or performance. We built Cloudflare Access to solve our own headaches with private networks as we grew from a team concentrated in a single office to a globally distributed organization.
Cloudflare Access replaces corporate VPNs with Cloudflare’s Continue reading
When people pause the Internet goes quiet
Recent news about the Internet has mostly been about the great increase in usage as those workers who can have been told to work from home. I've written about this twice recently, first in early March and then last week look at how Internet use has risen to a new normal.
As human behaviour has changed in response to the pandemic, it's left a mark on the charts that network operators look at day in, day out to ensure that their networks are running correctly.
Most Internet traffic has a fairly simple rhythm to it. Here, for example, is daily traffic seen on the Amsterdam Internet Exchange. It's a pattern that's familiar to most network operators. People sleep at night, and there's a peak of usage in the early evening when people get home and perhaps stream a movie, or listen to music or use the web for things they couldn't do during the workday.
But sometimes that rhythm get broken. Recently we've seen the evening peak by joined by morning peaks as well. Here's a graph from the Milan Internet Exchange. There are three peaks: morning, afternoon and evening. These peaks seem to be caused by people working from Continue reading
DDoS attacks have evolved, and so should your DDoS protection
The proliferation of DDoS attacks of varying size, duration, and persistence has made DDoS protection a foundational part of every business and organization’s online presence. However, there are key considerations including network capacity, management capabilities, global distribution, alerting, reporting and support that security and risk management technical professionals need to evaluate when selecting a DDoS protection solution.
Gartner’s view of the DDoS solutions; How did Cloudflare fare?
Gartner recently published the report Solution Comparison for DDoS Cloud Scrubbing Centers (ID G00467346), authored by Thomas Lintemuth, Patrick Hevesi and Sushil Aryal. This report enables customers to view a side-by-side solution comparison of different DDoS cloud scrubbing centers measured against common assessment criteria. If you have a Gartner subscription, you can view the report here. Cloudflare has received the greatest number of ‘High’ ratings as compared to the 6 other DDoS vendors across 23 assessment criteria in the report.
The vast landscape of DDoS attacks
From our perspective, the nature of DDoS attacks has transformed, as the economics and ease of launching a DDoS attack has changed dramatically. With a rise in cost-effective capabilities of launching a DDoS attack, we have observed a rise in the number of under 10 Gbps DDoS Continue reading
Empowering our Customers and Service Partners
Last year, Cloudflare announced the planned expansion of our partner program to help managed and professional service partners efficiently engage with Cloudflare and join us in our mission to help build a better Internet. Today, we want to highlight some of those amazing partners and our growing support and training for MSPs around the globe. We want to make sure service partners have the enablement and resources they need to bring a more secure and performant Internet experience to their customers.
This partner program tier is specifically designed for professional service firms and Managed Service Providers (MSPs and MSSPs) that want to build value-added services and support Cloudflare customers. While Cloudflare is hyper-focused on building highly scalable and easy to use products, we recognize that some customers may want to engage with a professional services firm to assist them in maximizing the value of our offerings. From building Cloudflare Workers, implementing multi-cloud load balancing, or managing WAF and DDoS events, our partner training and support enables sales and technical teams to position and support the Cloudflare platform as well as enhance their services businesses.
Training
Our training and certification is meant to help partners through each stage of Cloudflare adoption, Continue reading
Doubling the intern class – and making it all virtual
Earlier this month, we announced our plans to relaunch our intern hiring and double our intern class this summer to support more students who may have lost their internships due to COVID-19. You can find that story here. We’ve had interns joining us over the last few summers - students were able to find their way to us by applying to full-time roles and sometimes through Twitter. But, it wasn’t until last summer, in 2019, when we officially had our first official Summer Internship Program. And this year, we are doubling down.
Why do we invest in interns?
We have found interns to be invaluable. Not only do they bring an electrifying new energy over the summer, but they also come with their curiosity to help solve problems, contribute to major projects, and bring refreshing perspectives to the company.
- Ship projects: Our interns are matched with a team and work on real and meaningful projects. They are expected to ramp up, contribute like other members of the team and ship by the end of their internship.
- Hire strong talent: The internship is the “ultimate interview” that allows us to better assess new grad talent. The 12 weeks they spend with Continue reading
Creating a True One-Stop Solution for Companies to Go Global: Announcing a Partnership Between Cloudflare and JD Cloud & AI
It’s well known that global companies can face challenges doing business in and out of China due to the country’s unique rules, regulations, and norms, not to mention recent political and trade complications. Less well known is that China’s logistical and technical network infrastructure is also quite different from the rest of the world’s. With global Internet traffic up 30% over the past month due to the pandemic, these logistical and technical hurdles are increasing the burden for global businesses at exactly the wrong time. It’s now not unusual for someone based in China to have to wait extended periods and often be unable to access applications hosted elsewhere, or vice-versa, due to the lower performance of international Internet traffic to and from China. This affects global companies with customers, suppliers or employees in China, and Chinese companies who are trying to reach global users.
Our mission is to help build a better Internet, for everyone, everywhere. So, today we’re excited to announce a significant strategic partnership with JD Cloud & AI, the cloud and intelligent technology business unit of Chinese Internet giant JD.com. Through this partnership, we’ll be adding 150 data centers in mainland China, an increase in Continue reading
Releasing kubectl support in Access
Starting today, you can use Cloudflare Access and Argo Tunnel to securely manage your Kubernetes cluster with the kubectl command-line tool.
We built this to address one of the edge cases that stopped all of Cloudflare, as well as some of our customers, from disabling the VPN. With this workflow, you can add SSO requirements and a zero-trust model to your Kubernetes management in under 30 minutes.
Once deployed, you can migrate to Cloudflare Access for controlling Kubernetes clusters without disrupting your current kubectl workflow, a lesson we learned the hard way from dogfooding here at Cloudflare.
What is kubectl?
A Kubernetes deployment consists of a cluster that contains nodes, which run the containers, as well as a control plane that can be used to manage those nodes. Central to that control plane is the Kubernetes API server, which interacts with components like the scheduler and manager.
kubectl is the Kubernetes command-line tool that developers can use to interact with that API server. Users run kubectl commands to perform actions like starting and stopping the nodes, or modifying other elements of the control plane.
In most deployments, users connect to a VPN that allows them to run commands against that Continue reading
Stream Firewall Events directly to your SIEM
The highest trafficked sites using Cloudflare receive billions of requests per day. But only about 5% of those requests typically trigger security rules, whether they be “managed” rules such as our WAF and DDoS protections, or custom rules such as those configured by customers using our powerful Firewall Rules and Rate Limiting engines.
When enforcement is taken on a request that interrupts the flow of malicious traffic, a Firewall Event is logged with detail about the request including which rule triggered us to take action and what action we took, e.g., challenged or blocked outright.
Previously, if you wanted to ingest all of these events into your SIEM or logging platform, you had to take the whole firehose of requests—good and bad—and then filter them client side. If you’re paying by the log line or scaling your own storage solution, this cost can add up quickly. And if you have a security team monitoring logs, they’re being sent a lot of extraneous data to sift through before determining what needs their attention most.
As of today, customers using Cloudflare Logs can create Logpush jobs that send only Firewall Events. These events arrive much faster than our existing HTTP Continue reading
Internet performance during the COVID-19 emergency
A month ago I wrote about changes in Internet traffic caused by the COVID-19 emergency. At the time I wrote:
Cloudflare is watching carefully as Internet traffic patterns around the world alter as people alter their daily lives through home-working, cordon sanitaire, and social distancing. None of these traffic changes raise any concern for us. Cloudflare's network is well provisioned to handle significant spikes in traffic. We have not seen, and do not anticipate, any impact on our network's performance, reliability, or security globally.
That holds true today; our network is performing as expected under increased load. Overall the Internet has shown that it was built for this: designed to handle huge changes in traffic, outages, and a changing mix of use. As we are well into April I thought it was time for an update.
Growth
Here's a chart showing the relative change in Internet use as seen by Cloudflare since the beginning of the year. I've calculated moving average of the trailing seven days for each country and use December 29, 2019 as the reference point.
On this chart the highest growth in Internet use has been in Portugal: it's currently running at about a 50% increase Continue reading
Deploying Gateway using a Raspberry Pi, DNS over HTTPS and Pi-hole
Like many who are able, I am working remotely and in this post, I describe some of the ways to deploy Cloudflare Gateway directly from your home. Gateway’s DNS filtering protects networks from malware, phishing, ransomware and other security threats. It’s not only for corporate environments - it can be deployed on your browser or laptop to protect your computer or your home WiFi. Below you will learn how to deploy Gateway, including, but not limited to, DNS over HTTPS (DoH) using a Raspberry Pi, Pi-hole and DNSCrypt.
We recently launched Cloudflare Gateway and shortly thereafter, offered it for free until at least September to any company in need. Cloudflare leadership asked the global Solutions Engineering (SE) team, amongst others, to assist with the incoming onboarding calls. As an SE at Cloudflare, our role is to learn new products, such as Gateway, to educate, and to ensure the success of our prospects and customers. We talk to our customers daily, understand the challenges they face and consult on best practices. We were ready to help!
One way we stay on top of all the services that Cloudflare provides, is by using them ourselves. In this blog, I'll talk about Continue reading
Helping with COVID-19 Projects: Cloudflare Workers now free through Project Galileo
The Internet has been vital to our response to the COVID-19 crisis: enabling researchers to communicate with the rest of the world, connecting resources with people who need them, and sharing data about the spread.
It’s been amazing to see some of the projects people have stood up on Cloudflare Workers to assist during this crisis. Workers allows you to get set up in minutes, it’s fast and scalable out of the box, and there’s no infrastructure to maintain or scale, which is great if you want to create a project quickly.
To support critical web projects that help in the fight against the COVID-19 pandemic, we’re giving free access to our Cloudflare Workers compute platform through Project Galileo. We believe sites, apps, APIs, and tools that can help people with COVID-19 are exactly the type of critically important projects that Project Galileo was designed to support.
Free Cloudflare Workers
One of the earliest impacts of the COVID-19 crisis was the switch that many organizations made to a fully remote model. As that happened, and we realized that many organization’s VPNs were not up to the task of scaling to support this increased load, Cloudflare made Cloudflare for Continue reading
Is BGP Safe Yet? No. But we are tracking it carefully
BGP leaks and hijacks have been accepted as an unavoidable part of the Internet for far too long. We relied on protection at the upper layers like TLS and DNSSEC to ensure an untampered delivery of packets, but a hijacked route often results in an unreachable IP address. Which results in an Internet outage.
The Internet is too vital to allow this known problem to continue any longer. It's time networks prevented leaks and hijacks from having any impact. It's time to make BGP safe. No more excuses.
Border Gateway Protocol (BGP), a protocol to exchange routes has existed and evolved since the 1980s. Over the years it has had security features. The most notable security addition is Resource Public Key Infrastructure (RPKI), a security framework for routing. It has been the subject of a few blog posts following our deployment in mid-2018.
Today, the industry considers RPKI mature enough for widespread use, with a sufficient ecosystem of software and tools, including tools we've written and open sourced. We have fully deployed Origin Validation on all our BGP sessions with our peers and signed our prefixes.
However, the Internet can only be safe if the major network operators deploy Continue reading
Time-Based One-Time Passwords for Phone Support
As part of Cloudflare’s support offering, we provide phone support to Enterprise customers who are experiencing critical business issues.
For account security, specific account settings and sensitive details are not discussed via phone. From today, we are providing Enterprise customers with the ability to configure phone authentication to allow for greater support to be offered over the phone without need to perform validation through support tickets.
After providing your email address to a Cloudflare Support representative, you can now provide a token generated from the Cloudflare dashboard or via a 2FA app like Google Authenticator. So, a customer is able to prove over the phone that they are who they say they are.
Configuring Phone Authentication
If you are an existing Enterprise customer interested in phone support, please contact your Customer Success Manager for eligibility information and set-up. If you are interested in our Enterprise offering, please get in contact via our Enterprise plan page.
If you already have phone support eligibility, you can generate single-use tokens from the Cloudflare dashboard or configure an authenticator app to do the same remotely.
On the support page, you will see a card called “Emergency Phone Support Hotline – Authentication”. From here you Continue reading
Cloudflare Workers Now Support COBOL
Recently, COBOL has been in the news as the State of New Jersey has asked for help with a COBOL-based system for unemployment claims. The system has come under heavy load because of the societal effects of the SARS-CoV-2 virus. This appears to have prompted IBM to offer free online COBOL training.
As old as COBOL is (60 years old this month), it is still heavily used in information management systems and pretty much anywhere there’s an IBM mainframe around. Three years ago Thomson Reuters reported that COBOL is used in 43% of banking systems, is behind 80% of in-person financial transactions and 95% of times an ATM card is used. They also reported 100s of billions of lines of running COBOL.
COBOL is often a source of amusement for programmers because it is seen as old, verbose, clunky, and difficult to maintain. And it’s often the case that people making the jokes have never actually written any COBOL. We plan to give them a chance: COBOL can now be used to write code for Cloudflare’s serverless platform Workers.
Here’s a simple “Hello, World!” program written in COBOL and accessible at https://hello-world.cobol.workers.dev/. It doesn’t do much--it Continue reading
Cloudflare Dashboard and API Outage on April 15, 2020
Starting at 1531 UTC and lasting until 1952 UTC, the Cloudflare Dashboard and API were unavailable because of the disconnection of multiple, redundant fibre connections from one of our two core data centers.
This outage was not caused by a DDoS attack, or related to traffic increases caused by the COVID-19 crisis. Nor was it caused by any malfunction of software or hardware, or any misconfiguration.
What happened
As part of planned maintenance at one of our core data centers, we instructed technicians to remove all the equipment in one of our cabinets. That cabinet contained old inactive equipment we were going to retire and had no active traffic or data on any of the servers in the cabinet. The cabinet also contained a patch panel (switchboard of cables) providing all external connectivity to other Cloudflare data centers. Over the space of three minutes, the technician decommissioning our unused hardware also disconnected the cables in this patch panel.
This data center houses Cloudflare’s main control plane and database and as such, when we lost connectivity, the Dashboard and API became unavailable immediately. The Cloudflare network itself continued to operate normally and proxied customer websites and applications continued to operate. As Continue reading
Offer of Assistance to Governments During COVID-19
As the COVID-19 emergency continues to affect countries and territories around the world, the Internet has been a key factor in providing information to the public. As businesses, organizations and government agencies adjust to this new normal, we recognize the strain that this pandemic has put on the groups working to assist in virus mitigation and provide accurate information to the general public on the state of the pandemic.
At Cloudflare, this means ensuring that these entities have the necessary tools and resources available to them in these extenuating circumstances. On March 13, we announced our Cloudflare for Teams products will be free until September 1, 2020, to ensure Cloudflare users and prospective users have the tools they need to support secure and efficient remote work. Additionally, we have removed usage caps for existing Cloudflare for Teams users and are also providing onboarding sessions so these groups can continue business in this new normal.
As a company, we believe we can do more and have been thinking about ways we can support organizations and businesses that are at the forefront of the pandemic such as health officials and those providing relief to the public. Many organizations have reached out to Continue reading
Comparing HTTP/3 vs. HTTP/2 Performance
We announced support for HTTP/3, the successor to HTTP/2 during Cloudflare’s birthday week last year. Our goal is and has always been to help build a better Internet. Collaborating on standards is a big part of that, and we're very fortunate to do that here.
Even though HTTP/3 is still in draft status, we've seen a lot of interest from our users. So far, over 113,000 zones have activated HTTP/3 and, if you are using an experimental browser those zones can be accessed using the new protocol! It's been great seeing so many people enable HTTP/3: having real websites accessible through HTTP/3 means browsers have more diverse properties to test against.
When we launched support for HTTP/3, we did so in partnership with Google, who simultaneously launched experimental support in Google Chrome. Since then, we've seen more browsers add experimental support: Firefox to their nightly builds, other Chromium-based browsers such as Opera and Microsoft Edge through the underlying Chrome browser engine, and Safari via their technology preview. We closely follow these developments and partner wherever we can help; having a large network with many sites that have HTTP/3 enabled gives browser implementers an excellent testbed against which to Continue reading