A deep dive into Cloudflare’s September 12, 2025 dashboard and API outage

What Happened

We had an outage in our Tenant Service API which led to a broad outage of many of our APIs and the Cloudflare Dashboard. 

The incident’s impact stemmed from several issues, but the immediate trigger was a bug in the dashboard. This bug caused repeated, unnecessary calls to the Tenant Service API. The API calls were managed by a React useEffect hook, but we mistakenly included a problematic object in its dependency array. Because this object was recreated on every state or prop change, React treated it as “always new,” causing the useEffect to re-run each time. As a result, the API call executed many times during a single dashboard render instead of just once. This behavior coincided with a service update to the Tenant Service API, compounding instability and ultimately overwhelming the service, which then failed to recover.

When the Tenant Service became overloaded, it had an impact on other APIs and the dashboard because Tenant Service is part of our API request authorization logic.  Without Tenant Service, API request authorization can not be evaluated.  When authorization evaluation fails, API requests return 5xx status codes.

We’re very sorry about the disruption.  The rest Continue reading

TNO041: From Ansible to AI: Jeremy Schulman on the Evolution of Network Automation

Jeremy Schulman has been working at network automation for much of his professional life. On today’s Total Network Operations, host Scott Robohn talks with Jeremy about his ongoing quest to get the network engineering bottleneck out of production. They discuss the early days of network automation when engineers tried to adopt tools from the compute... Read more »

HN796: The Why and How of Making Your Infrastructure Quantum-Safe (Sponsored)

Your production IT operations are almost certainly using cryptography libraries that are not quantum-safe, and the time to begin planning a cryptography overhaul is now. But this is likely to be a daunting project because it touches everything: clients, servers, apps, network devices, middleboxes, and so on. Daunting, but doable. We talk with Richu Channakeshava, Principal... Read more »

Lab: Running IS-IS over IPv4 Unnumbered and IPv6 LLA Interfaces

IS-IS does not use IPv4 or IPv6, so it should be a no-brainer to run it over IPv4 unnumbered or IPv6 LLA interfaces. The latter is true; the former is smack in the middle of the It Depends™ territory.

Want to know more or test the devices you’re usually working with? The Running IS-IS Over Unnumbered/LLA-only Interfaces lab exercise is just what you need.

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to basic/7-unnumbered and execute netlab up.

AUSNOG 2025

The Australian Network Operators' Group, AUSNOG, held its 19th meeting at the start of September. Rather than simply relate the content of the presentations I'd like to take a few presentations and place them into a broader context to show how such topics fit today's networked environment.

Securing AI Workloads in Kubernetes: Why Traditional Network Security Isn’t Enough

The AI revolution is here, and it’s running on Kubernetes. From fraud detection systems to generative AI platforms, AI-powered applications are no longer experimental projects; they’re mission-critical infrastructure. But with great power comes great responsibility, and for Kubernetes platform teams, that means rethinking security.

But this rapid adoption comes with a challenge: 13% of organizations have already reported breaches of AI models or applications, while another 8% don’t even know if they’ve been compromised. Even more concerning, 97% of breached organizations reported that they lacked proper AI access controls. To address this, we must recognize that AI architectures introduce entirely new attack vectors that traditional security models aren’t equipped to handle.

AI Architectures Introduce New Attack Vectors

AI workloads running in Kubernetes environments introduce a new set of security challenges. Traditional security models often fall short in addressing the unique complexities of AI pipelines, specifically related to The Multi-Cluster Problem, The East-West Traffic Dilemma, and Egress Control Complexity. Let’s explore each of these critical attack vectors in detail.

The Multi-Cluster Problem

Most enterprise AI deployments don’t run in a single cluster. Instead, they typically follow this pattern:

Training Infrastructure (GPU-Heavy)

The Curious Case of ‘ip host’ Configuration Command

Since time immemorial, I have used the ip host router configuration command to get host-to-IP mappings in networking labs without going through the hassle of setting up a DNS server. Some devices even accepted multiple IP addresses in the ip host command, allowing you to list all router interfaces in a single command and get reverse (IP-to-host) mapping working like a charm. Or so I thought 🤦‍♂️

It turns out I’m too old, and what I know is sometimes no longer true. It seems that the last implementation working as I expected is Cisco IOS Classic ☹️

Read more …

Packet trimming

The latest version of the AI Metrics dashboard uses industry standard sFlow telemetry from network switches to monitor the number of trimmed packets to use as a congestion metric.

Ultra Ethernet Specification Update describes how the Ultra Ethernet Transport (UET) Protocol has the ability to leverage optional “packet trimming” in network switches, which allows packets to be truncated rather than dropped in the fabric during congestion events. As packet spraying causes reordering, it becomes more complicated to detect loss. Packet trimming gives the receiver and the sender an early explicit indication of congestion, allowing immediate loss recovery in spite of reordering, and is a critical feature in the low-RTT environments where UET is designed to operate.

cumulus@switch:~$ nv set system forwarding packet-trim profile packet-trim-default
cumulus@switch:~$ nv config apply

NVIDIA Cumulus Linux release 5.14 for NVIDA Spectrum Ethernet Switches includes support for Packet Trimming. The above command enables packet trimming, sets the DSCP remark value to 11, sets the truncation size to 256 bytes, sets the switch priority to 4, and sets the eligibility to all ports on the switch with traffic class 1, 2, and 3. NVIDA BlueField host adapters respond to trimmed packets to ensure fast congestion recovery.

Continue reading

Labbing Network Technology Details with netlab

It’s been over four years since I published the last Software Gone Wild episode. In the meantime, I spent most of my time developing an open-source labbing tool, so it should be no surprise that the first post-hiatus episode focused on a netlab use case: how Ethan Banks (of the PacketPushers fame) is using the tool to quickly check the technology details for his N is for Networking podcast.

As expected, our discussion took us all over the place, including (according to Riverside AI):

Read more …

Why IT-Site1 Can’t Ping OT_Site1R – Show and Tell Time #1

In my earlier blog post, Troubleshooting OT Security: Why IT-Site1 Can’t Ping OT_Site1R, we discovered the reason for this issue. Our “who done it” is simple. For security reasons, we are using Cisco TrustSec to keep them from communicating. Which... Read More ›

The post Why IT-Site1 Can’t Ping OT_Site1R – Show and Tell Time #1 appeared first on Networking with FISH.

Navigating DORA with Calico: Strengthening Kubernetes Operational Resilience in Financial Services

A single cyberattack or system outage can threaten not just one financial institution, but the stability of a vast portion of the entire financial sector. For today’s financial enterprises, securing dynamic infrastructure like Kubernetes is a core operational and regulatory challenge. The solution lies in achieving DORA compliance for Kubernetes, which transforms your cloud-native infrastructure into a resilient, compliant, and secure backbone for critical financial services.

The Challenge DORA Seeks to Solve

Before DORA (Digital Operational Resilience Act), rules for financial companies primarily focused on making sure they had enough financial capital to cover losses. But what if a cyberattack or tech failure brought a large part of the financial system down? Even with plenty of financial capital, a major outage could stop most operations and cause big problems for the whole financial market. DORA steps in to fix this. It’s all about making sure financial firms can withstand, respond to, and recover quickly from cyberattacks and other digital disruptions.

What is DORA?

The Digital Operational Resilience Act (DORA) is a European Union (EU) regulation that came into effect on January 17, 2025 and is designed to strengthen the security of financial entities. It establishes uniform requirements across the financial Continue reading

HW060: CNAE – A New Wired Networking Certification From the CWNP

The CWNP offers vendor-neutral certifications for wireless networking professionals. This summer, the organization rolled out a wired certification, the Certified Network Administrator and Engineer (CNAE). This cert is aimed at wired and wireless network engineers to ensure they have a solid grounding in switching, routing, cabling, and wired protocols. The CWNP says the cert isn’t... Read more »

Worth Reading 090925


The arrangements regarding the composition and organisation of the provision and operation of authoritative root servers are one of the more long-lasting aspects of the public Internet.


Microsoft has achieved a breakthrough in the hollow core fiber technology, reducing data transmission loss to just 0.091 dB per kilometer, the lowest ever achieved and significantly below the 0.14 dB/km limit that has constrained fiber networks for decades.


China has pursued a strategy of competition among government-owned organizations, and it initiated two government-owned constellation projects, Hongyun and Hongyan, in 2018.


There is no question that one of the smartest things that chip designer, packager, and manufacturing process manager Marvell Technology did was to shell out $650 million in May 2019 to buy Avera Semiconductor.


In early 2024, a Hong Kong-based clerk at multinational consulting engineering firm Arup was duped into transferring about $25 million to scammers who used AI to impersonate the company’s CFO, and other senior executives, in a live video meeting.

PP077: News Roundup–Drift Breach Has Long Reach; FCC Investigates Its Own IoT Security Program

Is any publicity good publicity? On today’s News Roundup we talk about how Salesloft, which makes the Drift chat agent that’s been used as a jumping-off point for credential harvesting and data breach attacks against a bunch of big-name companies, is testing that proposition. We also discuss bugs affecting industrial refrigeration controllers, and Microsoft making... Read more »
1 … 24 25 26 27 28 … 3,836