Detecting incast congestion of RDMA flows
docker run -p 6343:6343/udp -p 8008:8008 sflow/ai-monitorThe pre-built sflow/ai-monitor image is the quickest way to try out the software. The diagram at the top of this page shows test RDMA flows crossing a 4 leaf / 2 spine test network to generate an incast collision. The flows are displayed using the trace-flow application that is included in the image.
The ai-monitor application uses packet samples to detect when GPU port is oversubscribed:
- Detect when an leaf switch egress port utilization exceeds 80%
- Count the number of sources sending traffic to the port - single flows are ignored
- Trace each of the flows back across the topology to its source GPU
The instant the threshold is crossed, the following incast_congestion event is generated by querying information maintained within the sFlow-RT analytics engine. Real-time elephant flow monitoring in AI / ML fabrics describes how standard sFlow from data center switches streams real-time telemetry that can be used to detect elephant flows with a few milliseconds.
{
"event": Continue reading
Booth Demo
The example above uses the open source sFlow-RT booth-demo application to demonstrate a 400G data center switch carrying RoCEv2 test traffic. The demo shows a series of RDMA write operations in the Traffic Trend chart. The Switch Port Traffic Flows diagram shows the paths traffic takes as it flows through the switch. In this example, a NCCL ring topology results traffic flowing from port to port as it goes around the logical ring of GPUs.
The quickest way to set up the demo is to run the a pre-built sflow/booth-demo image on a laptop connected to the booth display screen. The Traffic Trend chart is easily customizable to show different flow attributes that are relevant to the demo and the title can be changed to include branding and messaging. If more extensive customization is required, clone the GitHub project Continue reading
How vagrant-libvirt Plugin Deals with Duplicate Subnets
TL&DR: Badly. The vagrant-libvirt plugin mysteriously crashes when an existing virtual network (with a different libvirt name) uses the same IP subnet as the desired management network.
Background: netlab is using the vagrant-libvirt plugin to manage libvirt/KVM virtual machines with Vagrant. As I already have that infrastructure, I use it to start standalone virtual machines (usually to test various Ubuntu releases) on my Linux server. Things work great… until they don’t.
Here’s how I managed to waste half a day chasing imaginary gremlins caused by a simple error.
Networking Aspects of Running VMs in Containers
The vrnetlab project and its containerlab fork implement a wonderful idea: let’s package virtual machines (primarily network devices that cannot be containerized) as containers to use reliable orchestration tools like containerlab to provision network topologies.
That approach might have a few drawbacks (depending on how the container images are built), but the obvious elephant in the room is: how do you make the virtual network plumbing work?
Installing Cisco Nexus Dashboard on Proxmox
Important disclaimerThis guide is intended strictly for lab, testing, and learning purposes. Cisco does not officially support running Nexus Dashboard (ND) on Proxmox.For production deployments, always use Cisco-supported platforms such as VMware ESXi or bare-metal / supported KVM environments. VM Creation on Proxmox Create a new VM in Proxmox with the following characteristics: One critical […]
<p>The post Installing Cisco Nexus Dashboard on Proxmox first appeared on IPNET.</p>
Cloonix Network Emulator Updated to Version 53
The Cloonix network emulator continues to be actively developed in 2026. In version 53, it provides container support, a rootless installation option, experimental support for WiFi emulation, and a simplified user experience.
In this post, I will review Cloonix v53 and walk through some hands-on exercises with the new features.
What’s New in Cloonix v53
The Cloonix network emulator has evolved significantly since I last wrote about it in 2017.
I list the major changes, since I last looked at Cloonix, below:
Container Support
One of the most significant additions is native container support, using crun. Cloonix feature support for containers is, however, limited. For example, one cannot save a container’s filesystem and containers do not support Spice connections so they can’t run a desktop environment.
Cloonix containers start faster and use fewer resources than KVM virtual machines, or even Docker containers, making them ideal for scenarios where you need many lightweight nodes. I imagine containers in Cloonix will be used to create many clients in a network simulation scenario.
Cloonix continues to support virtual machines using qemu-kvm, and you can mix containers and KVM VMs in the same Cloonix topology.
Rootless Installation Option
Cloonix v53 introduces a “cloonix-toy” installation Continue reading
Spaghetti Pasta Networking
Here’s an interesting data point in case you ever wondered why things are getting slower, even though the CPU performance is supposedly increasing. Albert Siersema sent me a link to a confusing implementation of spaghetti networking.
It looks like they’re trying to solve the how do I connect two containers (network namespaces) without having the privilege to create a vEth pair challenge with plenty of chewing gum and duct tape tap interfaces 🤦♂️
Linux For Network Engineers (LFNE) – AlmaLinux & Alpine Editions
After the release of the Ubuntu 24.04 edition of Linux For Network Engineers (LFNE) I’ve got some questions from the community. Here are two new flavors of LFNE based on your requests. LFNE AlmaLinux 10 OS For Red Hat fans who prefer a RHEL-style environment. Since CentOS is no longer maintained, AlmaLinux is the closest […]
<p>The post Linux For Network Engineers (LFNE) – AlmaLinux & Alpine Editions first appeared on IPNET.</p>
Linux For Network Engineers (LFNE) – Now on Ubuntu 24.04
The Linux For Network Engineers (LFNE) Docker container has been refreshed and is now built on Ubuntu 24.04 LTS. For those new to it, LFNE is a ready-to-use Linux environment preloaded with the most popular tools used by network engineers—from packet capture and traffic analysis utilities to configuration helpers and scripting support. Instead of spending […]
<p>The post Linux For Network Engineers (LFNE) – Now on Ubuntu 24.04 first appeared on IPNET.</p>
Testing Arista AVD with GNS3 and EOS
Arista AVD (Architect, Validate, Deploy) – https://avd.arista.com – is a powerful tool that brings network architecture into the world of Infrastructure-as-Code. I wanted to try it out in a lab setting and see how it works in a non-standard environment. Since my go-to lab tool is GNS3 with Arista cEOS images — while the AVD […]
<p>The post Testing Arista AVD with GNS3 and EOS first appeared on IPNET.</p>
Deploying ELK Stack with Docker Compose (2025 Edition)
This guide walks you through installing and configuring the ELK Stack (Elasticsearch, Logstash, Kibana, and Filebeat) using Docker Compose. It is fully updated for Elasticsearch 9.0.2 and explains the necessary changes for versions 8+ and above, including the required security setup and user permissions. Prerequisites Ensure you have the following installed on your system: The […]
<p>The post Deploying ELK Stack with Docker Compose (2025 Edition) first appeared on IPNET.</p>
Weird: Ports on Linux Bridge Are Stuck
Just when you thought you got used to the weirdnesses in the networking implementations, you get a curveball like this one. Life is never dull if you test network devices.
Before releasing netlab release 2.0, I ran the full suite of integration tests for all devices for which I have the images. Interestingly, most VXLAN tests failed for Cumulus Linux 4.x even though we haven’t touched that code for ages.
Next step: trying to figure out what changed. The configuration changes were minimal. Even worse, the failure was non-deterministic. Somehow, we managed to transform a Cumulus Linux 4.x VM into a Heisenberg switch.
Taking On VMware, HPE Mashes Up VM Essentials With Morpheus Cloud Controller
The rapid changes Broadcom instituted after buying virtualization stalwart VMware for $61 billion in late 2023 continue to shape the virtualization and cloud spaces, with some enterprises facing significant higher pricing, new licensing plans, and bunding options looking for alternatives, vendors offering them alternatives, and companies rolling out plans to help with the migrations. …
Taking On VMware, HPE Mashes Up VM Essentials With Morpheus Cloud Controller was written by Jeffrey Burt at The Next Platform.
Arista EOS Spooky Action at a Distance
This blog post describes yet another bizarre behavior discovered during the netlab integration testing.
It started innocently enough: I was working on the VRRP integration test and wanted to use Arista EOS as the second (probe) device in the VRRP cluster because it produces nice JSON-formatted results that are easy to use in validation tests.
Everything looked great until I ran the test on all platforms on which netlab configures VRRP, and all of them passed apart from Arista EOS (that was before we figured out how Sturgeon’s Law applies to VRRPv3) – a “That’s funny” moment that was directly responsible for me wasting a few hours chasing white rabbits down this trail.
The Linux Bridge MTU Hell
It all started with an innocuous article describing the MTU basics. As the real purpose of the MTU is to prevent packet drops due to fixed-size receiver buffers, and I waste spend most of my time in virtual labs, I wanted to check how various virtual network devices react to incoming oversized packets.
As the first step, I created a simple netlab topology in which a single link had a slightly larger than usual MTU… and then all hell broke loose.
Capturing Traffic in Virtual Networking Labs
When I announced the Stub Networks in Virtual Labs blog post on LinkedIn, I claimed it was the last chapter in the “links in virtual labs” saga. I was wrong; here comes the fourth part of the virtual links trilogy – capturing “on the wire” traffic in virtual networking labs.
While network devices provide traffic capture capabilities (usually tcpdump in disguise generating a .pcap file), it’s often better to capture the traffic outside of the device to see what the root cause of the problems you’re experiencing might be.
Stub Networks in Virtual Labs
The previous blog posts described how virtualization products create LAN segments and point-to-point links.
However, sometimes we need stub segments – segments connected to a single router or switch – because we don’t want to waste resources creating hosts attached to a network device, but would still prefer a more realistic mechanism than static routes to inject IP subnets into routing protocols.
Point-to-Point Links in Virtual Labs
In the previous blog post, I described the usual mechanisms used to connect virtual machines or containers in a virtual lab, and the drawbacks of using Linux bridges to connect virtual network devices.
In this blog post, we’ll see how KVM/QEMU/libvirt/Vagrant use UDP tunnels to connect virtual machines, and how containerlab creates point-to-point vEth links between Linux containers.
Please Wait While We’re Preparing Your Interfaces
Once a virtual machine running a network operating system boots, you’d expect its data-plane interfaces to be operational, right? Some vendors disagree. It takes over a minute for some network operating systems to figure out they have this thing called interfaces.1
I would love to figure out what takes them so long (a minute is an eternity on modern CPUs), but I guess we’ll never know.
Behind the Scenes
netlab uses two device provisioning mechanisms: it can start virtual machines with Vagrant or containers with containerlab. Some of those containers might use KVM/QEMU to run a hidden virtual machine (see also: RFC 1925 rule 6a).
Links in Virtual Labs
There are three major ways to connect network devices in the physical world:
- Point-to-point links between devices (usually using some variant of Ethernet)
- Multi-access layer-1 networks running some IEEE 802.x encapsulation on top of that (GPON, WiFi, Ethernet hubs)
- Multi-access switched layer-2 network (dumb switches, hopefully running some STP variant)
Implementing these connections in virtual labs is a bit harder than one might think, as all virtualization solutions assume you plan to run virtual servers connected to Ethernet segments.