Public Videos: Bridging with EVPN
The EVPN in the Data Center and Bridging with EVPN parts of the EVPN webinar, featuring Dinesh Dutt, are now available without a valid ipSpace.net account. Enjoy!
IPB188: IPv6 Adoption for an Entire Country
What does it take for an entire country to adopt IPv6? Our guest today is Tenanoia (Noia) Simona, CEO of Tuvalu Telecommunications Corporation, the country’s sole telecommunications provider. She’s here to walk us through the difficulties of connecting the many islands of Tuvalu and their journey to achieving one of the world’s highest IPv6 adoption... Read more »N4N043: Redundancy vs. High Availability Part 1
In today’s chat, Holly and Ethan consider a question from listener Douglas who asks, “How do you approach designing a network for high availability and redundancy?” They start by defining differences between redundancy and high availability, and talk about Holly’s experience with her own customers. Then they share examples of how to achieve redundancy in... Read more »Building VXLAN/EVPN Data Center Lab with netlab
Dmitry Klepcha published an excellent document describing how you can use netlab to build a series of data center fabric labs, starting from a simple IP network (without routing) and finishing with a complex EVPN/VXLAN network using symmetric IRB and MLAG toward hosts.
But wait, there’s more: all the lab topologies he used in his exercises are available on GitHub, which means that you could just clone the repo and start using them (I also “borrowed” some of his ideas as future netlab improvements).
Finally, thanks a million to Roman Pomazanov for bringing Dmitry’s work to my attention (and for the quote at the end of his post ;).
D2DO287: Leveling Up in Data Science
Ever wonder what it takes to level up your career in data science? Senior Data Scientist Darya Petrashka joins Ned and Kyler to share her personal journey from management and linguistics into data science, the real difference between a junior and a senior role, and helps us get under the “data science umbrella” to see... Read more »Interesting: an MCP Agent for Link-State Routing Protocols
Vadim Semenov created a nice demo that allows you to use an LLM to query the collected link-state graphs through an MCP agent (SuzieQ would probably be faster and easier to deploy, but hey, AI).
If you want to kick the tires, you’ll find the source code on GitHub (Network AI assistant, MCP server for Topolograph service). You’ll also need Vadim’s previous projects: Topolograph and OSPF watcher or IS-IS watcher.
HW065: Using Orb to Monitor the Quality of Your Internet Connections
With the help of Josh Hardy, Co-Founder and CTO of Orb, we introduce Orb. Orb is a suite app for OS, Android, macOS, Windows, Linux that is a new way to look at your internet connectivity. Josh gives us a little background on why and how Orb was created, He then goes into more detail... Read more »PP087: Why SBOMs Are Cooler and More Useful Than You Think
Just what’s inside that commercial software you bought? Does it contain open-source components, NPM packages, or other third-party code? How could you find out? The answer is a Software Bill of Materials, or SBOM, a machine-readable inventory of a finished piece of software. Why should you care about SBOMs? Our guest, Natalie Somersall, is here... Read more »HS117: Environmental, Social and Governance Initiatives: What That Means for Your Organization (Sponsored)
Environmental, Social, Governance (ESG) initiatives aren’t just “the right thing to do”, they can also save companies real dollars, particularly if they’re investing in data centers and other infrastructure. Join Jonathan Ciccio, Continuous Improvement Manager for The Siemon Company, as we discuss The Siemon Company’s ESG initiatives. The Siemon Company has been in business for... Read more »
Multi-Pod EVPN Troubleshooting: Fixing Next Hops
Last month, I wrote about the specifics of troubleshooting multi-pod EVPN designs. Today, I’d like to start a journey through an example in which (channeling my inner CCIE preparation lab instructor) I broke as many things as I could think of.
Here’s the lab topology we’ll use (and as usual, the corresponding netlab topology file and device configurations are on GitHub). Our network has two sites (pods), each with a spine switch, a leaf switch, and a host attached to the leaf switch. The inter-pod link is connected to the spine switches to minimize the number of devices.
Cloudflare outage on November 18, 2025
On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures to deliver core network traffic. This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network.
The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.
The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.
After we initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack, we correctly identified the core issue and were able Continue reading
NB552: Nokia Switches On AI Ops; IBM Seeks a Quantum of Advantage
Take a Network Break! Red Hat Samba server has a remote command execution vulnerability, and we cover some follow-up on fusion as a viable energy source (still a work in progress). On the news front, we search for signs in SoftBank’s sale of its Nividia stake, Mplify debuts a new certificate on carrier Ethernet for... Read more »Tech Bytes: Monitoring AI Agents and MCP Servers with Cisco ThousandEyes (Sponsored)
If you’re building, testing, or using AI agents and services, how can you get network visibility into, and ensure performance of, these tools? On today’s Tech Bytes podcast, we talk with sponsor Cisco ThousandEyes about how to monitor and measure performance of AI agents, AI models and services, and Model Context Protocol servers. We’ll get... Read more »UET Data Transfer Operation: Introduction
Introduction
[Updated 22 November 2025: Handoff Section]
The previous chapter described how an application gathers information about available hardware resources and uses that information to initialize the job environment. During this initialization, hardware resources are abstracted and made accessible to the UET provider as objects.
This chapter explains the data transport process, using gradient synchronization as an example.
Figure 5-1 depicts two GPUs—Rank 0 and Rank 2—participating in the same training job (JobID: 101). Both GPUs belong to the same NCCL topology and are connected to the Scale-Out Backend Network’s rail0.
Because the training model is large, each layer of neural network is split across two GPUs using tensor parallelism, meaning that the computations of a single layer are distributed between GPUs.
During the first forward-pass training iteration, the predicted model output does not match the expected result. This triggers the backward pass process, in which gradients—values indicating how much each weight parameter should be adjusted to improve the next forward-pass prediction—are computed.
Rank 0 computes its gradients, which in Figure 5-1 are stored as a 2D matrix with 3 rows and 1024 columns. The results are stored in a memory space registered for the process in local VRAM. Continue reading
Replicate is joining Cloudflare
We have some big news to share today: Replicate, the leading platform for running AI models, is joining Cloudflare.
We first started talking to Replicate because we shared a lot in common beyond just a passion for bright color palettes. Our mission for Cloudflare’s Workers developer platform has been to make building and deploying full-stack applications as easy as possible. Meanwhile, Replicate has been on a similar mission to make deploying AI models as easy as writing a single line of code. And we realized we could build something even better together by integrating the Replicate platform into Cloudflare directly.
We are excited to share this news and even more excited for what it will mean for customers. Bringing Replicate’s tools into Cloudflare will continue to make our Developer Platform the best place on the Internet to build and deploy any AI or agentic workflow.
Before we spend more time talking about the future of AI, we want to answer the questions that are top of mind for Replicate and Cloudflare users. In short:
For existing Replicate users: Your APIs and workflows will continue to work without interruption. You will soon benefit from the Continue reading
AI Enshittification: Swiss Airlines Edition
Remember the vendor consultants who persuasively told you how to use their gear to build a disaster recovery solution with stretched VLANs, even though the only disaster recovery they ever experienced was the frantic attempt to restart their PowerPoint slide deck? Fortunately, I was only involved in the aftermath of their activity when the laws of physics reasserted themselves, and I helped the poor victims rearchitect their network into a somewhat saner state.
There’s another batch of snake-oil salesmen consultants peddling their warez to the gullible incompetent managers: the AI preachers promising reduction in support costs. Like the other group of consultants, they have never worked in support and have never implemented a working AI solution in their lives, but that never bothered them or their audience.
Unfortunately, this time I had the unfortunate “privilege” of having the painful front-row seat.
SC25: SDSC Expanse cluster live AI/ML metrics
By default, the dashboard shows the Last 24 Hours of traffic. Explore the data: select Last 30 Days to get a long term view, select Last 5 Minutes to get an up to the second view, click on items in a chart legend to show selected metric, drag to select an interval and zoom in.
The SDSC Expanse cluster live AI/ML metrics dashboard displays real-time metrics for workloads running on the cluster:
- Total Traffic Total traffic entering fabric
- Cluster Services Traffic associated with Lustre, Ceph and NFS storage, and Slurm workload management
- Core Link Traffic Histogram of load on fabric links
- Edge Link Traffic Histogram of load on access ports
- RDMA Operations Total RDMA operations
- Continue reading