Billions and billions (of logs): scaling AI Gateway with the Cloudflare Developer Platform

With the rapid advancements occurring in the AI space, developers face significant challenges in keeping up with the ever-changing landscape. New models and providers are continuously emerging, and understandably, developers want to experiment and test these options to find the best fit for their use cases. This creates the need for a streamlined approach to managing multiple models and providers, as well as a centralized platform to efficiently monitor usage, implement controls, and gather data for optimization.

AI Gateway is specifically designed to address these pain points. Since its launch in September 2023, AI Gateway has empowered developers and organizations by successfully proxying over 2 billion requests in just one year, as we highlighted during September’s Birthday Week. With AI Gateway, developers can easily store, analyze, and optimize their AI inference requests and responses in real time.

With our initial architecture, AI Gateway faced a significant challenge: the logs, those critical trails of data interactions between applications and AI models, could only be retained for 30 minutes. This limitation was not just a minor inconvenience; it posed a substantial barrier for developers and businesses needing to analyze long-term patterns, ensure compliance, or simply debug over more extended periods.

In Continue reading

Durable Objects aren’t just durable, they’re fast: a 10x speedup for Cloudflare Queues

Cloudflare Queues let a developer decouple their Workers into event-driven services. Producer Workers write events to a Queue, and consumer Workers are invoked to take actions on the events. For example, you can use a Queue to decouple an e-commerce website from a service which sends purchase confirmation emails to users. During 2024’s Birthday Week, we announced that Cloudflare Queues is now Generally Available, with significant performance improvements that enable larger workloads. To accomplish this, we switched to a new architecture for Queues that enabled the following improvements:

  • Median latency for sending messages has dropped from ~200ms to ~60ms

  • Maximum throughput for each Queue has increased over 10x, from 400 to 5000 messages per second

  • Maximum Consumer concurrency for each Queue has increased from 20 to 250 concurrent invocations

Median latency drops from ~200ms to ~60ms as Queues are migrated to the new architecture

In this blog post, we'll share details about how we built Queues using Durable Objects and the Cloudflare Developer Platform, and how we migrated from an initial Beta architecture to a geographically-distributed, horizontally-scalable architecture for General Availability.

v1 Beta architecture

When initially designing Cloudflare Queues, we decided to build something simple that we could get Continue reading

How Does Netlab Deal with Server Reboots?

Now and then, someone asks how netlab deals with reboots (or power failures or crashes) of the server it’s running on.

TL&DR: It doesn’t. However…

netlab is a CLI command that acts as an umbrella orchestration layer for Vagrant and Containerlab. It does not run as a cron job, init script, or service and thus cannot be invoked when a server is booted.

Read more …

NAN077: Network Observability: Tools, Automation, and Insights

Network optimization starts with observing, but how are networks observed and what tools are used? Joining the podcast today are the authors behind the book “Modern Network Observability.” Eric Chou welcomes David Flores, Christian Adell, and Josh VanDeraa to help uncover practical strategies and real-world case studies for network observability. Episode Guests: David Flores, Christian... Read more »

D2DO254: Intelligent Data Infrastructure: How NetApp Builds for Performance, Security, and Visibility across Clouds (Sponsored)

Enterprise data is everywhere: on prem, at the edge, and across the public cloud. Storing and managing that data involves more than just SSDs and spinning disks; it’s about building and operating a data infrastructure. On today’s Day Two DevOps podcast, host Ned Bellavance explores the idea of data infrastructure with Jeff Baxter, VP of... Read more »

Management Toolkit: SMART Goals

In the last post we discussed Management vs Leadership, now let’s start getting into some useful tools, traits, and behaviors for both managers and leaders. Starting us off will be SMART goals. SMART is an acronym for Specific Measurable Achievable Realistic and Time-based. Once you hear about SMART goals and what SMART goals represent it […]

Training a million models per day to save customers of all sizes from DDoS attacks

Our always-on DDoS protection runs inside every server across our global network.  It constantly analyzes incoming traffic, looking for signals associated with previously identified DDoS attacks. We dynamically create fingerprints to flag malicious traffic, which is dropped when detected in high enough volume — so it never reaches its destination — keeping customer websites online.

In many cases, flagging bad traffic can be straightforward. For example, if we see too many requests to a destination with the same protocol violation, we can be fairly sure this is an automated script, rather than a surge of requests from a legitimate web browser.

Our DDoS systems are great at detecting attacks, but there’s a minor catch. Much like the human immune system, they are great at spotting attacks similar to things they have seen before. But for new and novel threats, they need a little help knowing what to look for, which is an expensive and time-consuming human endeavor.

Cloudflare protects millions of Internet properties, and we serve over 60 million HTTP requests per second on average, so trying to find unmitigated attacks in such a huge volume of traffic is a daunting task. In order to protect the smallest of companies, Continue reading

4.2 Tbps of bad packets and a whole lot more: Cloudflare’s Q3 DDoS report

Welcome to the 19th edition of the Cloudflare DDoS Threat Report. Released quarterly, these reports provide an in-depth analysis of the DDoS threat landscape as observed across the Cloudflare network. This edition focuses on the third quarter of 2024.

With a 296 Terabit per second (Tbps) network located in over 330 cities worldwide, Cloudflare is used as a reverse proxy by nearly 20% of all websites. Cloudflare holds a unique vantage point to provide valuable insights and trends to the broader Internet community.

Key insights 

  • The number of DDoS attacks spiked in the third quarter of 2024. Cloudflare mitigated nearly 6 million DDoS attacks, representing a 49% increase QoQ and 55% increase YoY.

  • Out of those 6 million, Cloudflare’s autonomous DDoS defense systems detected and mitigated over 200 hyper-volumetric DDoS attacks exceeding rates of 3 terabits per second (Tbps) and 2 billion packets per second (Bpps). The largest attack peaked at 4.2 Tbps and lasted just a minute.

  • The Banking & Financial Services industry was subjected to the most DDoS attacks. China was the country most targeted by DDoS attacks, and Indonesia was the largest source of DDoS attacks.

To learn more about DDoS attacks and other types Continue reading

Introducing Access for Infrastructure: SSH

BastionZero joined Cloudflare in May 2024. We are thrilled to announce Access for Infrastructure as BastionZero’s native integration into our SASE platform, Cloudflare One. Access for Infrastructure will enable organizations to apply Zero Trust controls in front of their servers, databases, network devices, Kubernetes clusters, and more. Today, we’re announcing short-lived SSH access as the first available feature. Over the coming months we will announce support for other popular infrastructure access target types like Remote Desktop Protocol (RDP), Kubernetes, and databases.

Applying Zero Trust principles to infrastructure

Organizations have embraced Zero Trust initiatives that modernize secure access to web applications and networks, but often the strategies they use to manage privileged access to their infrastructure can be siloed, overcomplicated, or ineffective. When we speak to customers about their infrastructure access solution, we see common themes and pain points:

  • Too risky: Long-lived credentials and shared keys get passed around and inflate the risk of compromise, excessive permissions, and lateral movement

  • Too clunky: Manual credential rotations and poor visibility into infrastructure access slow down incident response and compliance efforts

Some organizations have dealt with the problem of privileged access to their infrastructure by purchasing a Privileged Access Management (PAM) solution Continue reading

Per-Prefix and Per-VRF MPLS/VPN and EVPN Labels/VNIs

Long long time ago1, in an ancient town far far away2, an old-school networking Jeddi3 was driving us toward a convent4 where we had an SDN workshop5. While we were stuck in the morning traffic jam, an enthusiastic engineer sitting beside me wanted to know my opinion about per-prefix and per-VRF MPLS/VPN label allocation.

At that time, I had lived in a comfortable Cisco IOS bubble for way too long, so my answer was along the lines of “Say what???” Nicola Modena6 quickly expanded my horizons, and I said, “Gee, I have to write a blog post about that!” As you can see, it took me over a decade.

Read more …

PP036: News Roundup – NIST Nixes Password Resets, Cargo Crane Espionage Risks, Municipal Govs Targeted, and More

Today’s Packet Protector rounds up recent security news, including revised password guidelines from NIST, a White House push to help fill infosec jobs, and potential espionage risks from Chinese-made cranes being used at US ports. We also cover a hospital data breach that leaked nude patient photos, discuss why municipal governments are rich targets for... Read more »

Leveraging open technologies to monitor packet drops in AI cluster fabrics

In this talk from the recent OCP Global Summit, Aldrin Isaac, eBay, describes the challenge, AI clusters operate most efficiently over lossless networks for optimum job completion times which can be significantly impacted by dropped packets. Although networks can be designed to minimize packet loss by choosing the right network topology, optimizing network devices and protocols, an effective monitoring and troubleshooting network performance tool is still required. Such tool should capture packet drops, raise notifications and identify various drop reasons and pin point where the drops caused congestions. In turn, it allows the governing management application to tune configurations of relevant infrastructure components, including switches, NICs and GPU servers.

The talk shares the results and best practices of a TAM (Telemetry and Monitoring) solution being prepared for deployment at eBay. It leverages OCP’s SAI and open sFlow drop notification technologies as part of eBay’s ongoing initiatives to adopt open networking hardware and community SONiC for its data centers.

The sFlow Dropped Packet Notification Structures extension mentioned in the talk adds real-time packet drop notifications (including dropped packet header and drop reason) as part of an industry standard sFlow telemetry feed, making the data available to open source and commercial sFlow analytics Continue reading

Building Vectorize, a distributed vector database, on Cloudflare’s Developer Platform

Vectorize is a globally distributed vector database that enables you to build full-stack, AI-powered applications with Cloudflare Workers. Vectorize makes querying embeddings — representations of values or objects like text, images, audio that are designed to be consumed by machine learning models and semantic search algorithms — faster, easier and more affordable.

In this post, we dive deep into how we built Vectorize on Cloudflare’s Developer Platform, leveraging Cloudflare’s global network, Cache, Workers, R2, Queues, Durable Objects, and container platform.

What is a vector database?

A vector database is a queryable store of vectors. A vector is a large array of numbers called vector dimensions.

A vector database has a similarity search query: given an input vector, it returns the vectors that are closest according to a specified metric, potentially filtered on their metadata.

Vector databases are used to power semantic search, document classification, and recommendation and anomaly detection, as well as contextualizing answers generated by LLMs (Retrieval Augmented Generation, RAG).

Why do vectors require special database support?

Conventional data structures like B-trees, or binary search trees expect the data they index to be cheap to compare and to follow Continue reading

Is this thing on? Using OpenBMC and ACPI power states for reliable server boot

Introduction

At Cloudflare, we provide a range of services through our global network of servers, located in 330 cities worldwide. When you interact with our long-standing application services, or newer services like Workers AI, you’re in contact with one of our fleet of thousands of servers which support those services.

These servers which provide Cloudflare services are managed by a Baseboard Management Controller (BMC). The BMC is a special purpose processor  — different from the Central Processing Unit (CPU) of a server — whose sole purpose is ensuring a smooth operation of the server.

Regardless of the server vendor, each server has this BMC. The BMC runs independently of the CPU and has its own embedded operating system, usually referred to as firmware. At Cloudflare, we customize and deploy a server-specific version of the BMC firmware. The BMC firmware we deploy at Cloudflare is based on the Linux Foundation Project for BMCs, OpenBMC. OpenBMC is an open-sourced firmware stack designed to work across a variety of systems including enterprise, telco, and cloud-scale data centers. The open-source nature of OpenBMC gives us greater flexibility and ownership of this critical server subsystem, instead of the closed nature of proprietary firmware. Continue reading

Caddy Reverse Proxy With Docker

Caddy Reverse Proxy With Docker

I currently run multiple Docker containers across two hosts, each hosting various applications on different ports. UniFi Controller exposes the web GUI on 8443, Pi-hole on 8080, and Memos on 5230. Remembering each port number for every application started to become a hassle.

Additionally, most of these applications, like Pi-hole and Memos, do not support HTTPS out of the box. After searching for a solution to simplify this setup, I found that Caddy Reverse Proxy offers one of the simplest and most effective ways to manage these services. In this blog post, we’ll look at how to use Caddy Reverse Proxy with my Docker containers running across two hosts.

What is a Reverse Proxy?

A reverse proxy is a server that sits in front of one or more web servers and forwards client requests to them. It acts as an intermediary, handling incoming traffic and distributing it to the appropriate server. This setup can help improve security, manage SSL/TLS encryption, and simplify network traffic management by consolidating multiple services under a single domain.

What is Caddy?

Caddy is an open-source web server and reverse proxy software that is known for its simplicity and ease of use. It automatically handles HTTPS Continue reading

Lab: Configure IS-IS on Point-to-Point Links

From a very high-level perspective, OSPF and IS-IS are quite similar. Both were created in the Stone Age of networking, and both differentiate between multi-access LAN segments and point-to-point serial interfaces. Unfortunately, that approach no longer works in the Ethernet Everywhere world where most of the point-to-point links look like LAN segments, so we always have to change the default settings to make an IGP work better.

That’s what you’ll do in today’s lab exercise, which also explains the behind-the-scenes differences between point-to-point and multi-access links and the intricate world of three-way handshake.

ADCS Cert Templates for ISE Lab

In my ISE lab I’m going to be using EAP-TLS and TEAP, which means I’ll be needing user and computer certificates. The goal is to be able to enable the 802.1X supplicant via GPO and to distribute certificates automatically without requiring any user input. Another post will cover GPO, in this post I’ll cover creating the certificate templates in ADCS.

When opening the CA app, there are a number of templates provided by default:

There are already templates for User and Computer, but it’s better to leave the default templates alone and create new ones. First, we’ll create a template for user certificates. Start by right clicking Certificate Templates and selecting Manage:

Then we’re going to right click the User template and select Duplicate Template:

This is going to open up a new window with properties of the template:

Go to General and give the template a name:

Don’t select the Do not automatically reenroll option or it won’t be possible to renew certs before they expire.

Then go to Request Handling. We’re going to uncheck the Allow private key to be exported option as this is considered more secure:

Make sure Enroll subject without requiring any Continue reading

1 … 110 111 112 113 114 … 3,853