Jeffrey Tang

Author Archives: Jeffrey Tang

Lessons Learned from Scaling Up Cloudflare’s Anomaly Detection Platform

Introduction to Anomaly Detection for Bot Management

Lessons Learned from Scaling Up Cloudflare’s Anomaly Detection Platform

Cloudflare’s Bot Management platform follows a “defense in depth” model. Although each layer of Bot Management has its own strengths and weaknesses, the combination of many different detection systems — including Machine Learning, rule-based heuristics, JavaScript challenges, and more — makes for a robust platform in which different detection systems compensate for each other’s weaknesses.

One of these systems is Anomaly Detection, a platform motivated by a simple idea: because bots are made to accomplish specific goals, such as credential stuffing or content scraping, they interact with websites in distinct and difficult-to-disguise ways. Over time, the actions of a bot are likely to differ from those of a real user. Anomaly detection aims to model the characteristics of legitimate user traffic as a healthy baseline. Then, when automated bot traffic is set against this baseline, the bots appear as outlying anomalies that can be targeted for mitigation.

An anomaly detection approach is:

  • Resilient against bots that try to circumvent protections by spoofing request metadata (e.g., user agents)
  • Able to catch previously unseen bots without being explicitly trained against them.

So, how well does this work?

Today, Anomaly Detection processes more than Continue reading