Fabric Topologies for GPU Clusters

In part five of a series on AI data center networking, various interconnected networks are discussed, highlighting the differences between traditional and AI clusters. Key topics include back-end and front-end networks, the folded-Clos topology, adaptive routing, and the significance of switch design in addressing network congestion. Dragonfly topology is also mentioned for hyperscale systems.

The post Fabric Topologies for GPU Clusters appeared first on /overlaid.

Booth Demo

Quickly engaging an audience is a challenge at a trade show - all Ethernet switches look pretty much the same and they don't do much other than blink LEDs. Real-time sFlow telemetry is a core feature of switches that can be used to create intriguing animated dashboards that draw customers into the booth by demonstrating how the switch addresses their needs.

The example above uses the open source sFlow-RT booth-demo application to demonstrate a 400G data center switch carrying RoCEv2 test traffic. The demo shows a series of RDMA write operations in the Traffic Trend chart. The Switch Port Traffic Flows diagram shows the paths traffic takes as it flows through the switch. In this example, a NCCL ring topology results traffic flowing from port to port as it goes around the logical ring of GPUs.

The quickest way to set up the demo is to run the a pre-built sflow/booth-demo image on a laptop connected to the booth display screen. The Traffic Trend chart is easily customizable to show different flow attributes that are relevant to the demo and the title can be changed to include branding and messaging. If more extensive customization is required, clone the GitHub project Continue reading

Validating Your Network with Ansible Assert

Validating Your Network with Ansible Assert

This post is part of an ongoing series - see all the posts below.

Ansible Course

In the previous part, we built an OSPF network from scratch using Ansible, and then logged in to the routers to check that everything was working. We looked at the OSPF neighbors, checked the routing table, and ran a few pings. That's fine for six routers. But imagine doing it for fifty, before and after every change, at 2 AM during a maintenance window. By router number twelve, you're not really reading the output anymore, you're just looking for the word FULL and hoping for the best.

In this part, we'll let Ansible do the checking for us. We'll write a few simple tests that check whether every router has the right number of OSPF neighbors, and whether every router can reach every other router. If something is wrong, the playbook fails and tells us exactly where the problem is. We'll also break the network on purpose, so we can see what happens when a check fails.

We're continuing with the same lab and project from the previous part, so if you haven't read it yet, I recommend starting there.

What Is assert?

Continue reading

Fixing SCHED_IDLE for real

I’ve wondered for years why SCHED_IDLE tasks are allowed to preempt other tasks. Nobody seems to know, and I’ve not had the time to dig in and understand the Linux kernel scheduler better. But now I finally have my answer.

The problem

I want to be able to run a CPU-heavy task in the background, without it stealing any CPU, any at all, from other processes.

If you read various documentation, you’d be forgiven if this is as simple as schedtool -D -e ./my-background-thing.

SCHED_FIFO can be used only with static priorities higher than 0,
which means that when a SCHED_FIFO thread becomes runnable, it
will always immediately preempt any currently running SCHED_OTHER,
SCHED_BATCH, or SCHED_IDLE thread.

(and similar statements, e.g. from sched.7)

But if you try that, you’ll find that it’ll actually preempt other processes, taking little but nonzero CPU. There’s a test program, if you want to try it out.

I literally want it to take zero CPU. And not just against SCHED_FIFO/RR, but everything.

The existing settings

There appear to be two places that minmax guarantees can show up.

Settings with default values:

$ sudo sysctl -a | grep sched_rt_
kernel. Continue reading

Global Infrastructure at NANOG 98

The most recent North American Network Operators Group (NANOG) meeting was held in mid-October 2026 in Miami. The opening keynote presentation was by Telegeography's Paul Brodsky on the topic of global Internet Infrastructure. His presentation highlighted a structural shift in global Internet infrastructure, with visible concentration of investment and control in a small number of hyperscalers.

Introducing the Cloudflare PHP Capture the Flag challenge

Last week was one for the Internet security history books. First, eight eMudhra intermediate certificates were compromised, creating the possibility of forged certificates that could be used to impersonate servers. Then came php2shell: serious vulnerabilities in PHP, the runtime behind a large portion of the Web.

Cloudflare worked around the clock as these events unfolded to protect our customers and help protect the Internet. We shipped client updates and multiple mitigations, updated our web application firewall (WAF) with new rules to block attackers from exploiting the PHP vulnerabilities, and introduced a new feature called full transaction detection.

As always in situations like this, these new capabilities were made available on all plans, including Free.

Put our WAF to the test

Protecting systems from vulnerabilities like these is always an ongoing process, not a one-time event. The widespread adoption of AI allows attackers to discover new vectors with unprecedented speed and efficiency, so collecting real-world data to test our assumptions and strengthen defenses is essential.

That’s why today we're launching the Cloudflare PHP Capture the Flag challenge, run through our Bug Bounty program. We've put a deliberately vulnerable PHP installation behind our WAF, and we're inviting security researchers Continue reading

What took the Internet offline in Q3 2026

The Internet can feel invisible when everything is working normally. But when connectivity breaks, the many systems it depends upon become evident in an instant. Because Cloudflare’s global network handles more than 20% of the web’s traffic, it offers a unique perspective of how Internet traffic is behaving. When failures in infrastructure, power, policy, or security disrupt access to the Internet, Cloudflare Radar can observe the impact by monitoring traffic to our customers. Each quarter, we review the disruptions detected and annotated on Radar, highlighting what happened, where it happened, and the broader conditions that shaped each event.

In this post, we examine the major categories of Internet disruptions observed during the third quarter of 2026, including government-directed shutdowns, power outages and natural disasters.

Disruptions at a glance

Between July and September, Internet disruptions were largely driven by failures in the systems the Internet depends on. Power grids failed in Cuba, Honduras, Guam, Ukraine, and several other countries. Natural disasters damaged electricity and telecommunications infrastructure from Nepal to Colombia. Government-directed shutdowns remained the most frequent source of disruption, driven largely by exam-related shutdowns in Iraq, while renewed conflict in Ethiopia’s Tigray region showed how quickly connectivity can become part of Continue reading

HTTP/3 load balancer connection offloading

In a previous post I described load balancing for TCP-based HTTP with connection offloading, and mentioned that HTTP/3 can do even more. By the power of AI writing a HTTP/3 version of both the load balancer (sni-router-h3) and the webserver (tarweb-h3), I have experimented enough to have more to say about how that works.

To be clear: this post is 100% human-written.

In this post I will use this terminology:

  • client: E.g. a web browser.
  • router: A load balancer / traffic router, that inspects SNI and forwards to matching backends. Mine’s called sni-router.
  • backend: A webserver. Mine’s called tarweb.

First, the TCP story

While it is possible to use TCP_REPAIR to transfer an established connection between machines (from router to backend), it requires some more infrastructure. The kernel on the chosen backend must be synced up on the connection state (sequence numbers, etc) that TCP_REPAIR provides, but that’s actually just the easy part. It must also be able to send packets to the client with the original server’s IP address as source, and packets going from the client to the backend must somehow find their way to the same backend, in a network-efficient way.

There are other problems Continue reading

Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang

On September 8, 2026, we were alerted by a security researcher, Gadi Evron from Knostic, to a tenant isolation issue affecting the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3) starting on August 27, 2026. We have remediated the issue, and we restored the model on September 9. We have confirmed that no other model hosted on Workers AI was affected.

Between August 27 and September 9, a small number of customers using the GLM 5.3 Workers AI model received responses that contained data from other customers' requests made to the same model shortly before theirs. In total, 155 responses were affected, all to prompts longer than 681,280 tokens.

After excluding our own testing and the researcher's, the requests we identified are consistent in timing and volume with ordinary long-context use, not with someone repeatedly exploiting the bug. We found no evidence of deliberate exploitation.

Today we are publishing this report alongside a post from SGLang, the open-source inference engine that we use, as a coordinated disclosure. This post describes the bug that caused this issue, our response timeline, and the subsequent hardening we have conducted on our production SGLang deployments. We also detail our continued efforts with Continue reading

Building an OSPF Network from Scratch with Ansible

Building an OSPF Network from Scratch with Ansible

This post is part of an ongoing series - see all the posts below.

Ansible Course

So far in this series, we've been working with pretty much the same set of devices, core1, core2, access1, access2, and access3. We've used them to configure VLANs, NTP, syslog, and interface descriptions, and to learn the core Ansible concepts along the way.

For this part, we're going to start with a new topology and put everything we've learned into practice with a real-world example. The lab has six Cisco routers, split across three OSPF areas. I created it using Containerlab, and I've included the topology file below, so you should be able to build the same lab and follow along.

name: ospf_lab

mgmt:
  network: mgmt
  ipv4-subnet: 192.168.202.0/24

topology:
  kinds:
    cisco_iol:
      image: cisco-iol-17.12.01

  nodes:
    r1:
      kind: cisco_iol
      mgmt-ipv4: 192.168.202.101
    r2:
      kind: cisco_iol
      mgmt-ipv4: 192.168.202.102
    r3:
      kind: cisco_iol
      mgmt-ipv4: 192.168.202.103
    r4:
      kind: cisco_iol
      mgmt-ipv4: 192.168.202.104
    r5:
      kind: cisco_iol
      mgmt-ipv4: 192.168.202.105
    r6:
      kind: cisco_iol
      mgmt-ipv4: 192.168.202.106

  links:
    # Area 0
    - endpoints: ["r1:Ethernet0/1", "r2:Ethernet0/1"]   # 10.100.12. Continue reading

Useful Ansible Filters for Network Engineers

Useful Ansible Filters for Network Engineers

This post is part of an ongoing series - see all the posts below.

Ansible Course

In the previous part of this series, we looked at what filters are and covered some of the most commonly used ones, such as selectattr, map, and regex_replace. We also used filters to solve a real problem in the previous part by sharing the same syslog servers between Cisco and Arista.

Ansible has a huge number of filters, and it's impossible to cover all of them. So in this part, I want to focus on a handful that I find especially useful as a network engineer. We'll work with IP addresses and subnets, compare lists to check whether a device's config is what it should be, merge dictionaries together, and sort lists. Along the way, we'll also build a simple compliance check that you can use in your own network right away.

This post is part of a series on Ansible for network engineers. If you're new to the series, I recommend following the parts in order, since each one builds on the previous one. Here's a quick overview of the setup we're using. The lab has five switches, core1, core2 Continue reading

1 2 3 … 3,916