SC25: SDSC Expanse cluster live AI/ML metrics

The SDSC Expanse cluster live AI/ML metrics dashboard is a joint InMon / San Diego Supercomputer Center (SDSC) demonstration at The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC25) conference being held this week in St. Louis, November 16-21. Click on the dashboard link during the show to see live traffic.

By default, the dashboard shows the Last 24 Hours of traffic. Explore the data: select Last 30 Days to get a long term view, select Last 5 Minutes to get an up to the second view, click on items in a chart legend to show selected metric, drag to select an interval and zoom in.

The Expanse cluster at the San Diego Supercomputer Center is a batch-oriented science computing gateway serving thousands of users and a wide range of research projects, see Google News for examples.

The SDSC Expanse cluster live AI/ML metrics dashboard displays real-time metrics for workloads running on the cluster:

  • Total Traffic Total traffic entering fabric
  • Cluster Services Traffic associated with Lustre, Ceph and NFS storage, and Slurm workload management
  • Core Link Traffic Histogram of load on fabric links
  • Edge Link Traffic Histogram of load on access ports
  • RDMA Operations Total RDMA operations
  • Continue reading

Worth Reading 111425


An Autonomous System (AS) can protect itself against DDoS attacks by rerouting incoming DDoS traffic through a ‘DDoS scrubber’, a process that is typically implemented using the Border Gateway Protocol (BGP). While BGP-based scrubbing is a useful service, its adoption on the global Internet is unknown.


Until recently, many professionals, including many doctors, thought they needed to become more machine-like to keep their jobs. Such worry contributed to the specialization trend of the last century.


In an era of growing cyber threats, traditional defensive measures can be insufficient in the face of sophisticated or novel tactics. As a result, the question of whether private or public entities should engage in active defense or offensive “hack back” tactics has taken on greater urgency.


In this episode of PING, APNIC Chief Scientist Geoff Huston explores the complex landscape of undersea cables.


By 1990 it was clear that IP had a problem. It was still a tiny Internet at the time, but the growth patterns were exponential, doubling in size every 12 months. We were stressing out the pool of Class B IPv4 addresses and in the absence of any corrective measures this address pool would be fully depleted in 1994.

Worth Reading 111225

If you’re using IP address truncation thinking it makes data “anonymous” or “non-personal,” you’re creating a false sense of security.


With distressingly typical Silicon Valley fake-it-till-you make-it bravado, LLM creators have been telling investors that artificial general intelligence (AGI) is just around the corner (or has already been achieved!). The problem the promoters blithely ignore is that LLMs do not know how the words they input and output relate to the real world.


Many organizations rushing to cut staff in the name of AI efficiency are expected to quietly rehire those roles – often “offshore or at lower salary.”


While Silicon Valley is investing tens of billions of dollars chasing the artificial general intelligence dream, academic computing research in the U.S. is facing a severe drought.


There are two competing forces in IT, and they are at play during the GenAI era as much as they have ever been during prior eras in the datacenter.

Using netlab for Classroom Training with Sander Steffann

In March 2024, I received my first PR from an airplane: Sander Steffann was flying to South Africa to deliver an Ansible training and fixed a minor annoyance in the then-new multilab feature.

Of course, I wanted to know more about his setup, but it took us over a year and a half till we managed to sit down (virtually) and chat about it, the state of IPv6, the impact of CG-NAT on fraud prevention, and why digital twins don’t make sense in large datacenter migrations.

For more details, listen to Episode 202 of Software Gone Wild.

Listen to the podcast

Hedge 288: Loneliness

Sometimes we just like to talk about “life issues” as they relate to network engineering and technology career fields. Loneliness seems to especially plague network engineering and other “small” IT fields, where communities are small, change is rapid, and stress is high. Loneliness expert Lucy Rose joins Eyvonne, Tom, and Russ to discuss loneliness.

Check out the Cost of Loneliness here.

download

How We Measure: IPv6

At APNIC Labs we publish a number of measurements of the deployment of various technologies that are being adopted on the Internet. Here I will look at how we measure the adoption of IPv6.

Finding the grain of sand in a heap of Salt

How do you find the root cause of a configuration management failure when you have a peak of hundreds of changes in 15 minutes on thousands of servers?

That was the challenge we faced as we built the infrastructure to reduce release delays due to failures of Salt, a configuration management tool. (We eventually reduced such failures on the edge by over 5%, as we’ll explain below.) We’ll explore the fundamentals of Salt, and how it is used at Cloudflare. We then describe the common failure modes and how they delay our ability to release valuable changes to serve our customers.

By first solving an architectural problem, we provided the foundation for self-service mechanisms to find the root cause of Salt failures on servers, datacenters and groups of datacenters. This system is able to correlate failures with git commits, external service failures and ad hoc releases. The result of this has been a reduction in the duration of software release delays, and an overall reduction in toilsome, repetitive triage for SRE.

To start, we will go into the basics of the Cloudflare network and how Salt operates within it. And then we’ll get to how we solved the challenge Continue reading

netlab 25.11: SRv6 on IOS/XE, Streamlined Graphs and Reports

I managed to push out netlab release 25.11 yesterday. Here are the highlights:

  • SRv6 on IOS/XE. It works with Catalyst 8000v, IOL, and IOL layer-2 image, and can be used to build L3VPNs (the IOS/XE image I have supports no other service on top of SRv6)
  • RIPv2/RIPng on OpenBSD
  • A more streamlined way to create reports and graphs
  • The netlab graph command creates the SVG/PNG/JPEG/PDF graph instead of a graph description file if you’ve installed D2/Graphviz on your system.

We also had to make a few potentially-breaking changes, fixed a bunch of bugs, and added over a dozen small improvements.

You’ll find all the details in the release notes.

Read more …

NAN106: Unimus: Network Automation By and For Network Engineers (Sponsored)

Tomas Kirnak, CEO of Unimus, joins Eric Chou in this sponsored episode to introduce Unimus, an on-premise network configuration management system built by network engineers to solve real-world problems. In this deep dive they discuss Unimus’ proprietary “Behavioral Tree” for automatic device discovery, the platform’s vendor support, the 70/30 rule, and lowering the barrier for... Read more »

Connecting to production: the architecture of remote bindings

Remote bindings are bindings that connect to a deployed resource on your Cloudflare account instead of a locally simulated resource – and recently, we announced that remote bindings are now generally available. 

With this launch, you can now connect to deployed resources like R2 buckets and D1 databases while running Worker code on your local machine. This means you can test your local code changes against real data and services, without the overhead of deploying for each iteration. 

In this blog post, we’ll dig into the technical details of how we built it, creating a seamless local development experience.

Developing on the Workers platform

A key part of the Cloudflare Workers platform has been the ability to develop your code locally without having to deploy it every time you wanted to test something – though the way we’ve supported this has changed greatly over the years. 

We started with wrangler dev running in remote mode. This works by deploying and connecting to a preview version of your Worker that runs on Cloudflare’s network every time you make a change to your code, allowing you to test things out as you develop. However, remote mode isn’t perfect — Continue reading

Worth Reading: The Majority AI View

Many engineers who tried out (or use) various AI products would agree that they’re useful when used correctly, but way overhyped. However, as Anil Dash explains in his Majority AI View article, we rarely hear that opinion:

What’s amazing is the reality that virtually 100% of tech experts I talk to in the industry feel this way, yet nobody outside of that cohort will mention this reality.

Keep reading

HS116: Nth-Party Risk May Put You on the (Block) Chain Gang

The evolution of the modern, Internet-driven economy has created the conditions for essentially unbounded Nth-party risks (that is, risks from your suppliers, and risks from your suppliers’ suppliers, and risks from your suppliers’ suppliers’ suppliers, ad infinitum). Nth party risks exist in public clouds, SaaS, software and hardware supply chains, and now in the form... Read more »

One-Arm Hub-and-Spoke VPN on Arista EOS

In September 2024, I described how you can build One-Arm Hub-and-Spoke VPN with MPLS/VPN. In that blog post, I mentioned that the solution doesn’t work on Arista EOS because it allocates MPLS labels to whole VRFs (per-VRF label allocation).

In early September, I received an email from Daniel Blažek telling me that Arista fixed this particular annoyance in the EOS release 4.34.2F. It still uses per-VRF label allocation, but now, you can assign a different label to the default route. Let’s see how that works with our one-arm hub-and-spoke topology:

Read more …

UET Data Transport Part I: Introduction

[Figure updated 13 November 2025]

My previous UET posts explained how an application uses libfabric function API calls to discover available hardware resources and how this information is used to create a hardware abstraction layer composed of Fabric, Domain, and Endpoint objects, along with their child objects — Event Queues, Completion Queues, Completion Counters, Address Vectors, and Memory Regions.

This chapter explains how these objects are used during data transfer operations. It also describes how information is encoded into UET protocol headers, including the Semantic Sublayer (SES) and Packet Delivery Sublayer (PDC). In addition, the chapter covers how the Congestion Management Sublayer (CMS) monitors and controls send queue rates to prevent egress buffer overflows.

Note: In this book, libfabric API calls are divided into two categories for clarity. Functions are used to create and configure fabric objects such as fabrics, domains, endpoints, and memory regions (for example, fi_fabric(), fi_domain(), and fi_mr_reg()). Operations, on the other hand, perform actual data transfer or synchronization between processes (for example, fi_write(), fi_read(), and fi_send()).

Figure 5-1 provides a high-level overview of a libfabric Remote Memory Access (RMA) operation using the fi_write function call. When an application needs to transfer data, such as gradients, from Continue reading

A closer look at Python Workflows, now in beta

Developers can already use Cloudflare Workflows to build long-running, multi-step applications on Workers. Now, Python Workflows are here, meaning you can use your language of choice to orchestrate multi-step applications.

With Workflows, you can automate a sequence of idempotent steps in your application with built-in error handling and retry behavior. But Workflows were originally supported only in TypeScript. Since Python is the de facto language of choice for data pipelines, artificial intelligence/machine learning, and task automation – all of which heavily rely on orchestration – this created friction for many developers.

Over the years, we’ve been giving developers the tools to build these applications in Python, on Cloudflare. In 2020, we brought Python to Workers via Transcrypt before directly integrating Python into workerd in 2024. Earlier this year, we built support for CPython along with any packages built in Pyodide, like matplotlib and pandas, in Workers. Now, Python Workflows are supported as well, so developers can create robust applications using the language they know best.

Why Python for Workflows?

Imagine you’re training an LLM. You need to label the dataset, feed data, wait for the model to run, evaluate the loss, adjust the model, and repeat. Without automation, Continue reading

1 … 14 15 16 17 18 … 3,841