Real-time elephant flow monitoring in AI / ML fabrics

The diagram shows a Probability Density Function (PDF) of the time taken to fill a default 1,400 byte sFlow datagram for a 400G switch port, randomly sampling 1-in-100,000 packets, handling a line rate RoCEv2 RDMA transfer using 4096 byte packets.

Calculations:

  • Packet Size (4KB): 4,096 bytes × 8 bits/byte = 32,768 bits
  • Line Rate Packet Speed: 400,000,000,000 bits/sec ÷ 32,768 bits = 12,207,031.25 packets per second
  • Average Time per Packet: 1 ÷ 12,207,031.25 pps = 81.92 nanoseconds
  • Average Sample Interval: 81.92 nanoseconds × 100,000 = 0.008192 seconds (8.192 ms)
  • Default 1400 byte sFlow datagrams fills with 6 packet samples (each sample containing 128 bytes of packet header + metadata)
  • Time taken to fill the datagram with 6 samples given by the Erlang-6 distribution
  • Mean: 49.15ms, Median: 46.45ms, Standard Deviation: 20.07ms
  • sFlow agent required to send UDP datagram as soon as buffer fills
  • Network transit delay is less than 1ms and can be ignored
  • Maximum delay is 1000ms. sFlow agent required to send datagram within 1 second of the first record being added

The net result is that an sFlow analyzer will receive streaming telemetry describing an elephant flow within 100ms of the Continue reading

Bot-free self-hosted analytics with GoatCounter on NixOS

In 2016, I removed Google Analytics from this blog to avoid being complicit in feeding the biggest machine for harvesting personal data. Instead, I relied on GoAccess to analyze my server logs.1 For the past couple of years, the statistics have made no sense, despite my attempts to filter bots: AI scrapers inflate the number of visitors to around 2,000 per day. Eventually, I settled on GoatCounter, an open-source, privacy-friendly web analytics platform. I replaced the JavaScript client to filter bots more aggressively and added a CSS fallback. To improve reliability, I implemented a local proxy running on each of the five web servers serving this blog. The rest of this post details how these pieces fit together and how I deploy them on NixOS. ❄️

Why GoatCounter?

GoatCounter does not collect personal data: instead of storing the reader’s IP address or relying on cookies, it creates a session identifier valid for 8 hours from the user agent and the IP address. Its feature set is modest but sufficient for a blog. If Continue reading

The magic of OFDM

I’ve been playing a bit with OFDM in software defined radio lately. And after watching a video that wasn’t even close to correct, I thought I’d write down my understanding in a blog post. I wouldn’t call myself an expert, so worst case someone will correct me, and I’m happy to be corrected.

In my experiments with OFDM my initial goal is understanding, so e.g. I won’t blindly use some hierarchical GNU Radio block.

I/Q and the actual analog signal

I think that in order to internalize OFDM, we first need to remind ourselves how a purely “real” wave becomes a complex number, at a high level.

The real signal is a value that’s a function of time. Glossing over details, this is not feasible for most of our signals, as we would need to provide a new value to the transmitter at a rate twice as high as the highest frequency we care about. For 2.4GHz experiments, we’d need 4.8 billion samples per second.

Quadrature sampling allows us to represent a slice of bandwidth at a manageable rate. We can then provide the I/Q data to hardware, instructing it to hold that “until further notice”. Continue reading

Hedge 319: Using AI

AI is everywhere. While there are many questions about whether AI is a bubble, the ethics of AI, and the cultural impact of AI, in this episode of the Hedge Mike Bushong joins Eyvonne, Tom, and Russ to talk about how network engineers can use AI. Are some use cases better than others? Are there pitfalls network engineers should avoid? Listen in and find out.
 

 
download

TNO073: Network Automation Forum: From Simple Survey to Global Community

In this crossover episode, Scott invites William Collins and Eric Chou to do the hosting so he and Network Automation Forum (NAF) cofounder Chris Grundemann can discuss how their community emerged from a simple question: Why haven’t we seen full adoption of network automation? They discuss the growth of AutoCon and how its practitioner-focused, vendor-neutral... Read more »

Why networking is different for AI Data Centers

This article discusses the necessity for rethinking data center networking in the context of AI. Traditional designs based on traffic averages and oversubscription do not apply, as AI workloads produce massive, synchronized "elephant" flows, necessitating non-blocking architectures and emphasizing metrics like job completion time and GPU utilization over conventional measures.

The post Why networking is different for AI Data Centers appeared first on /overlaid.

IPB208: IPv6 Address Management Is Broken

Traditional IPAM and DDI tools struggle to handle IPv6 at scale because they were originally designed around IPv4. Today Ed and Tom examine the visibility challenges posed by SLAAC  addresses, broken DNS integrations, and massive prefix allocations. They also share potential solutions and strategies for enterprise IPv6 deployment. AdSpot Sponsor: Megaport If you’re supporting cloud... Read more »

Sunsetting netlab Vagrant/libvirt provider

When I started the netlab project, Vagrant was the go-to tool if you wanted to build a virtual environment described in a text configuration file (an idea popularized as infrastructure-as-code). It wasn’t ideal for what we were doing, but a tool rarely does a great job when used far away from its intended use case.

netlab initially supported Vagrant with VirtualBox, quickly adding support for KVM/libvirt through the vagrant-libvirt plugin. Life was good… until it wasn’t.

HITL for autonomous agents: Where does the human go?

Human approval is easy when you are sitting in front of the agent. For an agent running by itself in a cluster, almost none of that holds.

You’re in a meeting and your agent is running in a cluster. It has a service account, it has been asked to keep a service healthy, and it has just worked out that the right fix is to roll back a database migration. Nobody is watching it. That was rather the point of deploying it. You want to get notified to approve such an important action.

This is a different problem from the one most people picture when they hear human in the loop. If you use a coding assistant, you already have human oversight and it costs almost nothing: the agent shows you a diff, you read it, you approve. That works because you are already there — at a keyboard, with the context in front of you, in the same second the agent needs an answer.

An autonomous agent has none of that. There is no session to interrupt. The person who should decide is asleep, or in a meeting, or on a plane. The approval has to travel out of Continue reading

D2DO313: Operations is a Reconciliation Loop (Sponsored)

In this sponsored episode, Paul Delory of Gartner joins Kyler and Ned to discuss the reconciler pattern for operations and the shift toward continuous operations. Together they discuss how immutable infrastructure, policy as code, and platform engineering help bridge the gap between traditional IT and modern automation. They also highlight Gartner’s upcoming Infrastructure, Operations &... Read more »

Strata Cloud Manager – Worth the Hype?

Strata Cloud Manager - Worth the Hype?

If you use Palo Alto firewalls, typically you manage them in one of two ways. You either log into each firewall directly and manage it on its own, or you use Panorama, which lets you manage multiple firewalls from a single place.

Most of us were used to this setup, myself included. If you had a handful of firewalls, logging into each one individually is okay. But once you started scaling up to tens or hundreds of firewalls across different sites, Panorama became the go-to option. It let you centrally manage the configurations and monitor logs centrally instead of touching every single firewall by hand.

Strata Cloud Manager (SCM)

However, Palo Alto released yet another option called Strata Cloud Manager (SCM) in 2023, which is hosted by Palo Alto themselves, unlike Panorama, which you manage internally. The idea at a very high level is that the firewalls connect to Strata Cloud Manager (outbound only) through the Internet, and you log into SCM to manage the configurations and push them to the firewalls.

From my experience and speaking to other people, Palo Alto is pushing SCM to customers and adding new features to it. So the unspoken assumption is that any Continue reading

TCG084: Is AI Going to Kill Us All? Separating Frontier Risk From Frontier Theater

Is AI going to kill us all? Frontier systems are producing real security failures, but extinction forecasts are being used to support policies that could consolidate control over AI. Eyvonne and William discuss Jacob Coxon’s high-profile resignation from Anthropic, the gap between real technical risks and speculative extinction narratives, and the commercial incentives driving calls... Read more »
1 … 8 9 10 11 12 … 3,913