AI Enshittification: Swiss Airlines Edition
Remember the vendor consultants who persuasively told you how to use their gear to build a disaster recovery solution with stretched VLANs, even though the only disaster recovery they ever experienced was the frantic attempt to restart their PowerPoint slide deck? Fortunately, I was only involved in the aftermath of their activity when the laws of physics reasserted themselves, and I helped the poor victims rearchitect their network into a somewhat saner state.
There’s another batch of snake-oil salesmen consultants peddling their warez to the gullible incompetent managers: the AI preachers promising reduction in support costs. Like the other group of consultants, they have never worked in support and have never implemented a working AI solution in their lives, but that never bothered them or their audience.
Unfortunately, this time I had the unfortunate “privilege” of having the painful front-row seat.
SC25: SDSC Expanse cluster live AI/ML metrics
By default, the dashboard shows the Last 24 Hours of traffic. Explore the data: select Last 30 Days to get a long term view, select Last 5 Minutes to get an up to the second view, click on items in a chart legend to show selected metric, drag to select an interval and zoom in.
The SDSC Expanse cluster live AI/ML metrics dashboard displays real-time metrics for workloads running on the cluster:
- Total Traffic Total traffic entering fabric
- Cluster Services Traffic associated with Lustre, Ceph and NFS storage, and Slurm workload management
- Core Link Traffic Histogram of load on fabric links
- Edge Link Traffic Histogram of load on access ports
- RDMA Operations Total RDMA operations
- Continue reading
HN805: The Past, Present, and Future of NANOG
NANOG, or the North American Network Operation Group, is an organization committed to the continuing advancement of an open, secure, and robust Internet. At the NANOG Conference 95 in late October 2025, Ethan Banks chatted with Steve Feldman, a member of NANOG’s Board of Directors. Steve has been involved with NANOG since the very first... Read more »Worth Reading 111225
If you’re using IP address truncation thinking it makes data “anonymous” or “non-personal,” you’re creating a false sense of security.
Using netlab for Classroom Training with Sander Steffann
In March 2024, I received my first PR from an airplane: Sander Steffann was flying to South Africa to deliver an Ansible training and fixed a minor annoyance in the then-new multilab feature.
Of course, I wanted to know more about his setup, but it took us over a year and a half till we managed to sit down (virtually) and chat about it, the state of IPv6, the impact of CG-NAT on fraud prevention, and why digital twins don’t make sense in large datacenter migrations.
For more details, listen to Episode 202 of Software Gone Wild.
Hedge 288: Loneliness
Sometimes we just like to talk about “life issues” as they relate to network engineering and technology career fields. Loneliness seems to especially plague network engineering and other “small” IT fields, where communities are small, change is rapid, and stress is high. Loneliness expert Lucy Rose joins Eyvonne, Tom, and Russ to discuss loneliness.
LIU004: From Fast Food to Leading Operations at an ISP
Think you need a degree or a ton of certificates to succeed in tech? Think again. Matthew Oborne joins our hosts Alexis Bertholf and Kevin Nanns to discuss how he went from working fast food to leading operations at an ISP. Your starting point doesn’t define your ceiling; resilience, adaptability, and a willingness to learn... Read more »How We Measure: IPv6
At APNIC Labs we publish a number of measurements of the deployment of various technologies that are being adopted on the Internet. Here I will look at how we measure the adoption of IPv6.Finding the grain of sand in a heap of Salt
How do you find the root cause of a configuration management failure when you have a peak of hundreds of changes in 15 minutes on thousands of servers?
That was the challenge we faced as we built the infrastructure to reduce release delays due to failures of Salt, a configuration management tool. (We eventually reduced such failures on the edge by over 5%, as we’ll explain below.) We’ll explore the fundamentals of Salt, and how it is used at Cloudflare. We then describe the common failure modes and how they delay our ability to release valuable changes to serve our customers.
By first solving an architectural problem, we provided the foundation for self-service mechanisms to find the root cause of Salt failures on servers, datacenters and groups of datacenters. This system is able to correlate failures with git commits, external service failures and ad hoc releases. The result of this has been a reduction in the duration of software release delays, and an overall reduction in toilsome, repetitive triage for SRE.
To start, we will go into the basics of the Cloudflare network and how Salt operates within it. And then we’ll get to how we solved the challenge Continue reading
netlab 25.11: SRv6 on IOS/XE, Streamlined Graphs and Reports
I managed to push out netlab release 25.11 yesterday. Here are the highlights:
- SRv6 on IOS/XE. It works with Catalyst 8000v, IOL, and IOL layer-2 image, and can be used to build L3VPNs (the IOS/XE image I have supports no other service on top of SRv6)
- RIPv2/RIPng on OpenBSD
- A more streamlined way to create reports and graphs
- The netlab graph command creates the SVG/PNG/JPEG/PDF graph instead of a graph description file if you’ve installed D2/Graphviz on your system.
We also had to make a few potentially-breaking changes, fixed a bunch of bugs, and added over a dozen small improvements.
You’ll find all the details in the release notes.
TCG062: The Tech Expertise Deficit with Russ White
Today we delve into the tech expertise deficit and why technical depth and decades of doing the work matter more than social media followers and content creation hype. Our guest is Russ White, engineer, author, teacher, and certification developer. We begin with current events in AI, and then investigate the differences between career and influence... Read more »NAN106: Unimus: Network Automation By and For Network Engineers (Sponsored)
Tomas Kirnak, CEO of Unimus, joins Eric Chou in this sponsored episode to introduce Unimus, an on-premise network configuration management system built by network engineers to solve real-world problems. In this deep dive they discuss Unimus’ proprietary “Behavioral Tree” for automatic device discovery, the platform’s vendor support, the 70/30 rule, and lowering the barrier for... Read more »Connecting to production: the architecture of remote bindings
Remote bindings are bindings that connect to a deployed resource on your Cloudflare account instead of a locally simulated resource – and recently, we announced that remote bindings are now generally available.
With this launch, you can now connect to deployed resources like R2 buckets and D1 databases while running Worker code on your local machine. This means you can test your local code changes against real data and services, without the overhead of deploying for each iteration.
In this blog post, we’ll dig into the technical details of how we built it, creating a seamless local development experience.
A key part of the Cloudflare Workers platform has been the ability to develop your code locally without having to deploy it every time you wanted to test something – though the way we’ve supported this has changed greatly over the years.
We started with wrangler dev running in remote mode. This works by deploying and connecting to a preview version of your Worker that runs on Cloudflare’s network every time you make a change to your code, allowing you to test things out as you develop. However, remote mode isn’t perfect — Continue reading
Worth Reading: The Majority AI View
Many engineers who tried out (or use) various AI products would agree that they’re useful when used correctly, but way overhyped. However, as Anil Dash explains in his Majority AI View article, we rarely hear that opinion:
What’s amazing is the reality that virtually 100% of tech experts I talk to in the industry feel this way, yet nobody outside of that cohort will mention this reality.
HS116: Nth-Party Risk May Put You on the (Block) Chain Gang
The evolution of the modern, Internet-driven economy has created the conditions for essentially unbounded Nth-party risks (that is, risks from your suppliers, and risks from your suppliers’ suppliers, and risks from your suppliers’ suppliers’ suppliers, ad infinitum). Nth party risks exist in public clouds, SaaS, software and hardware supply chains, and now in the form... Read more »
PP086: Using Let’s Encrypt and the ACME Protocol for Domain Validation Certificates
Certificates are the socks of IT—everyone needs them, and you always lose track of a few. On today’s show we dive into the ACME protocol, an IETF standard to help automate how a domain owner gets a domain validation certificate from a Certificate Authority (CA). Our guest, Ed Harmoush, a former network engineer with AWS... Read more »One-Arm Hub-and-Spoke VPN on Arista EOS
In September 2024, I described how you can build One-Arm Hub-and-Spoke VPN with MPLS/VPN. In that blog post, I mentioned that the solution doesn’t work on Arista EOS because it allocates MPLS labels to whole VRFs (per-VRF label allocation).
In early September, I received an email from Daniel Blažek telling me that Arista fixed this particular annoyance in the EOS release 4.34.2F. It still uses per-VRF label allocation, but now, you can assign a different label to the default route. Let’s see how that works with our one-arm hub-and-spoke topology:
NB551: Cisco Adds AI to Tech Support; Cryptography Hits a Post-Quantum Milestone
Take a Network Break! We start with a critical vulnerability in Cisco’s Unified Contact Center Express. On the news front it’s a Cisco triple play: the company brings AI to professional services and tech support with Cisco IQ, debuts converged infrastructure for the AI edge, and launches a new cert geared for running AI data... Read more »UET Data Transport Part I: Introduction
[Figure updated 13 November 2025]
My previous UET posts explained how an application uses libfabric function API calls to discover available hardware resources and how this information is used to create a hardware abstraction layer composed of Fabric, Domain, and Endpoint objects, along with their child objects — Event Queues, Completion Queues, Completion Counters, Address Vectors, and Memory Regions.
This chapter explains how these objects are used during data transfer operations. It also describes how information is encoded into UET protocol headers, including the Semantic Sublayer (SES) and Packet Delivery Sublayer (PDC). In addition, the chapter covers how the Congestion Management Sublayer (CMS) monitors and controls send queue rates to prevent egress buffer overflows.
Note: In this book, libfabric API calls are divided into two categories for clarity. Functions are used to create and configure fabric objects such as fabrics, domains, endpoints, and memory regions (for example, fi_fabric(), fi_domain(), and fi_mr_reg()). Operations, on the other hand, perform actual data transfer or synchronization between processes (for example, fi_write(), fi_read(), and fi_send()).
Figure 5-1 provides a high-level overview of a libfabric Remote Memory Access (RMA) operation using the fi_write function call. When an application needs to transfer data, such as gradients, from Continue reading