Lessons Learned from Scaling Up Cloudflare’s Anomaly Detection Platform

Introduction to Anomaly Detection for Bot Management

Cloudflare’s Bot Management platform follows a “defense in depth” model. Although each layer of Bot Management has its own strengths and weaknesses, the combination of many different detection systems — including Machine Learning, rule-based heuristics, JavaScript challenges, and more — makes for a robust platform in which different detection systems compensate for each other’s weaknesses.

One of these systems is Anomaly Detection, a platform motivated by a simple idea: because bots are made to accomplish specific goals, such as credential stuffing or content scraping, they interact with websites in distinct and difficult-to-disguise ways. Over time, the actions of a bot are likely to differ from those of a real user. Anomaly detection aims to model the characteristics of legitimate user traffic as a healthy baseline. Then, when automated bot traffic is set against this baseline, the bots appear as outlying anomalies that can be targeted for mitigation.

An anomaly detection approach is:

• Resilient against bots that try to circumvent protections by spoofing request metadata (e.g., user agents)
• Able to catch previously unseen bots without being explicitly trained against them.

So, how well does this work?

Today, Anomaly Detection processes more than Continue reading