Don’t roll your own high cardinality analytics, use Workers Analytics Engine

Workers Analytics Engine (or for short, Analytics Engine) is a new way for developers to store and analyze time series analytics about anything using Cloudflare Workers, and it’s now in open beta! Analytics Engine is really good at gathering time-series data for really high cardinality and high-volume data sets from Cloudflare Workers. At Cloudflare, we use Analytics Engine to provide insight into how our customers use Cloudflare products.

Log, log, logging!

As an example, Analytics Engine is used to observe the backend that powers Instant Logs. Instant Logs allows Cloudflare customers to stream a live session of the HTTP logs for their domain to the Cloudflare dashboard. The backend for Instant Logs is built on Cloudflare Workers.

Briefly, the Instant Logs backend works by receiving requests from each Cloudflare server that processes a customer's HTTP traffic. These requests contain the HTTP logs for the customer’s HTTP traffic. The Instant Logs backend then forwards these HTTP logs to the customer’s browser via a WebSocket.

In order to ensure that the HTTP logs are being delivered smoothly to a customer's browser, we need to track the request rates across all active Instant Logs sessions. We also need to track the request rates Continue reading