Filter, reshape, and redact logs before export with Transformers

Logpush delivers logs in batches as quickly as possible, enabling Cloudflare to provide information almost in real time. During our recent Birthday Week, we made Logpush available on all self-serve plans and Transformers generally available.

Those changes were among eight major updates to Cloudflare Observability and reflect our commitment to make every Cloudflare feature available to everyone. This post takes a deeper look at how Transformers work inside Logpush's export pipeline. Observability teams use Logpush to export Cloudflare logs to security information and event management systems (SIEMs), analytics platforms, and object storage. Before those logs reach their destination, teams may need to remove health checks, omit sensitive fields, or reshape records for dashboards and investigations. Today, this often requires a separate processing service that teams must deploy, scale, monitor, and maintain.

For example, a team that collects logs from multiple CDNs can add a static provider field to normalize records before they reach its analytics platform. Teams also pay to export, ingest, and store records they discard as soon as they arrive, so filtering them before delivery can reduce downstream data volume and cost.

With Transformers, that processing moves into the Logpush export workflow.Teams define a SQL query Continue reading