A deep dive into Cloudflare’s September 12, 2025 dashboard and API outage
We had an outage in our Tenant Service API which led to a broad outage of many of our APIs and the Cloudflare Dashboard.
The incident’s impact stemmed from several issues, but the immediate trigger was a bug in the dashboard. This bug caused repeated, unnecessary calls to the Tenant Service API. The API calls were managed by a React useEffect hook, but we mistakenly included a problematic object in its dependency array. Because this object was recreated on every state or prop change, React treated it as “always new,” causing the useEffect to re-run each time. As a result, the API call executed many times during a single dashboard render instead of just once. This behavior coincided with a service update to the Tenant Service API, compounding instability and ultimately overwhelming the service, which then failed to recover.
When the Tenant Service became overloaded, it had an impact on other APIs and the dashboard because Tenant Service is part of our API request authorization logic. Without Tenant Service, API request authorization can not be evaluated. When authorization evaluation fails, API requests return 5xx status codes.
We’re very sorry about the disruption. The rest Continue reading
TNO041: From Ansible to AI: Jeremy Schulman on the Evolution of Network Automation
Jeremy Schulman has been working at network automation for much of his professional life. On today’s Total Network Operations, host Scott Robohn talks with Jeremy about his ongoing quest to get the network engineering bottleneck out of production. They discuss the early days of network automation when engineers tried to adopt tools from the compute... Read more »HN796: The Why and How of Making Your Infrastructure Quantum-Safe (Sponsored)
Your production IT operations are almost certainly using cryptography libraries that are not quantum-safe, and the time to begin planning a cryptography overhaul is now. But this is likely to be a daunting project because it touches everything: clients, servers, apps, network devices, middleboxes, and so on. Daunting, but doable. We talk with Richu Channakeshava, Principal... Read more »Hedge 280: Networking Careers
When you think of careers in network engineering, you probably think of being a console jockey, racking and stacking and configuring boxes. There is, however, much larger than just these things. Ethan Banks joins Eyvonne and Russ to talk about the broader world of careers in network engineering.
download
Lab: Running IS-IS over IPv4 Unnumbered and IPv6 LLA Interfaces
IS-IS does not use IPv4 or IPv6, so it should be a no-brainer to run it over IPv4 unnumbered or IPv6 LLA interfaces. The latter is true; the former is smack in the middle of the It Depends™ territory.
Want to know more or test the devices you’re usually working with? The Running IS-IS Over Unnumbered/LLA-only Interfaces lab exercise is just what you need.
Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to basic/7-unnumbered and execute netlab up.
AUSNOG 2025
The Australian Network Operators' Group, AUSNOG, held its 19th meeting at the start of September. Rather than simply relate the content of the presentations I'd like to take a few presentations and place them into a broader context to show how such topics fit today's networked environment.Securing AI Workloads in Kubernetes: Why Traditional Network Security Isn’t Enough
The AI revolution is here, and it’s running on Kubernetes. From fraud detection systems to generative AI platforms, AI-powered applications are no longer experimental projects; they’re mission-critical infrastructure. But with great power comes great responsibility, and for Kubernetes platform teams, that means rethinking security.
But this rapid adoption comes with a challenge: 13% of organizations have already reported breaches of AI models or applications, while another 8% don’t even know if they’ve been compromised. Even more concerning, 97% of breached organizations reported that they lacked proper AI access controls. To address this, we must recognize that AI architectures introduce entirely new attack vectors that traditional security models aren’t equipped to handle.
AI Architectures Introduce New Attack Vectors
AI workloads running in Kubernetes environments introduce a new set of security challenges. Traditional security models often fall short in addressing the unique complexities of AI pipelines, specifically related to The Multi-Cluster Problem, The East-West Traffic Dilemma, and Egress Control Complexity. Let’s explore each of these critical attack vectors in detail.
The Multi-Cluster Problem
Most enterprise AI deployments don’t run in a single cluster. Instead, they typically follow this pattern:
Training Infrastructure (GPU-Heavy)
- Dedicated clusters with high-memory GPU nodes
- Batch job Continue reading
The Curious Case of ‘ip host’ Configuration Command
Since time immemorial, I have used the ip host router configuration command to get host-to-IP mappings in networking labs without going through the hassle of setting up a DNS server. Some devices even accepted multiple IP addresses in the ip host command, allowing you to list all router interfaces in a single command and get reverse (IP-to-host) mapping working like a charm. Or so I thought 🤦♂️
It turns out I’m too old, and what I know is sometimes no longer true. It seems that the last implementation working as I expected is Cisco IOS Classic ☹️
Packet trimming
The latest version of the AI Metrics dashboard uses industry standard sFlow telemetry from network switches to monitor the number of trimmed packets to use as a congestion metric.
Ultra Ethernet Specification Update describes how the Ultra Ethernet Transport (UET) Protocol has the ability to leverage optional “packet trimming” in network switches, which allows packets to be truncated rather than dropped in the fabric during congestion events. As packet spraying causes reordering, it becomes more complicated to detect loss. Packet trimming gives the receiver and the sender an early explicit indication of congestion, allowing immediate loss recovery in spite of reordering, and is a critical feature in the low-RTT environments where UET is designed to operate.
cumulus@switch:~$ nv set system forwarding packet-trim profile packet-trim-default cumulus@switch:~$ nv config apply
NVIDIA Cumulus Linux release 5.14 for NVIDA Spectrum Ethernet Switches includes support for Packet Trimming. The above command enables packet trimming, sets the DSCP remark value to 11, sets the truncation size to 256 bytes, sets the switch priority to 4, and sets the eligibility to all ports on the switch with traffic class 1, 2, and 3. NVIDA BlueField host adapters respond to trimmed packets to ensure fast congestion recovery.
D2DO282: Simplifying Complex Kubernetes Deployments With kro
Kubernetes is flexible and customizable, but it can also be notoriously complex and difficult to deploy to. On today’s Day Two DevOps we learn about kro (Kube Resource Operator), an open-source tool that helps simplify complex application deployments. Our guest is Islam Mahgoub, a Solutions Architect at AWS focused on building kro. We talk about... Read more »Labbing Network Technology Details with netlab
It’s been over four years since I published the last Software Gone Wild episode. In the meantime, I spent most of my time developing an open-source labbing tool, so it should be no surprise that the first post-hiatus episode focused on a netlab use case: how Ethan Banks (of the PacketPushers fame) is using the tool to quickly check the technology details for his N is for Networking podcast.
As expected, our discussion took us all over the place, including (according to Riverside AI):
Why IT-Site1 Can’t Ping OT_Site1R – Show and Tell Time #1
In my earlier blog post, Troubleshooting OT Security: Why IT-Site1 Can’t Ping OT_Site1R, we discovered the reason for this issue. Our “who done it” is simple. For security reasons, we are using Cisco TrustSec to keep them from communicating. Which... Read More ›
The post Why IT-Site1 Can’t Ping OT_Site1R – Show and Tell Time #1 appeared first on Networking with FISH.
Navigating DORA with Calico: Strengthening Kubernetes Operational Resilience in Financial Services
A single cyberattack or system outage can threaten not just one financial institution, but the stability of a vast portion of the entire financial sector. For today’s financial enterprises, securing dynamic infrastructure like Kubernetes is a core operational and regulatory challenge. The solution lies in achieving DORA compliance for Kubernetes, which transforms your cloud-native infrastructure into a resilient, compliant, and secure backbone for critical financial services.
The Challenge DORA Seeks to Solve
Before DORA (Digital Operational Resilience Act), rules for financial companies primarily focused on making sure they had enough financial capital to cover losses. But what if a cyberattack or tech failure brought a large part of the financial system down? Even with plenty of financial capital, a major outage could stop most operations and cause big problems for the whole financial market. DORA steps in to fix this. It’s all about making sure financial firms can withstand, respond to, and recover quickly from cyberattacks and other digital disruptions.
What is DORA?
The Digital Operational Resilience Act (DORA) is a European Union (EU) regulation that came into effect on January 17, 2025 and is designed to strengthen the security of financial entities. It establishes uniform requirements across the financial Continue reading
HW060: CNAE – A New Wired Networking Certification From the CWNP
The CWNP offers vendor-neutral certifications for wireless networking professionals. This summer, the organization rolled out a wired certification, the Certified Network Administrator and Engineer (CNAE). This cert is aimed at wired and wireless network engineers to ensure they have a solid grounding in switching, routing, cabling, and wired protocols. The CWNP says the cert isn’t... Read more »PP077: News Roundup–Drift Breach Has Long Reach; FCC Investigates Its Own IoT Security Program
Is any publicity good publicity? On today’s News Roundup we talk about how Salesloft, which makes the Drift chat agent that’s been used as a jumping-off point for credential harvesting and data breach attacks against a bunch of big-name companies, is testing that proposition. We also discuss bugs affecting industrial refrigeration controllers, and Microsoft making... Read more »SwiNOG 40: Reliability of High-Speed Transceivers
Whenever you see Gerhard Stein and Thomas Weible from Flexoptix in a list of presenters, three things immediately become obvious:
- It will be about transceivers
- It will be fun
- It will include some crazy stuff
Their SwiNOG 40 presentation (video) met all three expectation. We learned how well transceivers cope with high temperatures and what happens when you try to melt them with a heat gun.
NB542: Hollow Core Fiber Outshines Glass; Broadcom Bags Big AI Bucks
Take a Network Break! We shine a red light on an AnyShare Service Agent API vulnerability and an active exploit against FreePBX. SASE vendor Cato Networks makes first-ever acquisition with purchase of AI security startup AIM, Microsoft researchers tout hollow core fiber tests that out-perform glass core fiber optics, and Wi-Fi 7 helps drive up... Read more »Bringing Node.js HTTP servers to Cloudflare Workers
We’re making it easier to run your Node.js applications on Cloudflare Workers by adding support for the node:http client and server APIs. This significant addition brings familiar Node.js HTTP interfaces to the edge, enabling you to deploy existing Express.js, Koa, and other Node.js applications globally with zero cold starts, automatic scaling, and significantly lower latency for your users — all without rewriting your codebase. Whether you're looking to migrate legacy applications to a modern serverless platform or build new ones using the APIs you already know, you can now leverage Workers' global network while maintaining your existing development patterns and frameworks.
Cloudflare Workers operate in a unique serverless environment where direct tcp connection isn't available. Instead, all networking operations are fully managed by specialized services outside the Workers runtime itself — systems like our Open Egress Router (OER) and Pingora that handle connection pooling, keeping connections warm, managing egress IPs, and all the complex networking details. This means as a developer, you don't need to worry about TLS negotiation, connection management, or network optimization — it's all handled for you automatically.
This fully-managed approach is actually why Continue reading