Hedge 264: Documentation and Tech Debt
On this episode of the Hedge, Eyvonne, Tom, and Russ talk about topics near and dear to every network engineer’s heart–documentation, legacy, and tech debt. What should our philosophy of documentation be? What are legacy, end of life, and tech debt, really?
Passive BGP Sessions
The Dynamic BGP Peers lab exercise gave you the opportunity to build a large-scale environment in which routers having an approved source IP addresses (usually matching an ACL/prefix list) can connect to a BGP route reflector or route server.
In a more controlled environment, you’d want to define BGP neighbors on the BGP RR/RS but not waste CPU cycles trying to establish BGP sessions with unreachable neighbors. Welcome to the world of passive BGP sessions.
Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to session/8-passive and execute netlab up.
Future Proofing Inference Servers With PCI-Express Switches
At this point in the history of datacenter systems, there can be no higher praise than to be chosen by Nvidia as a component supplier for its AI systems. …
Future Proofing Inference Servers With PCI-Express Switches was written by Timothy Prickett Morgan at The Next Platform.
TL010: Leading With Influence Rather Than Mandating
How do you lead with influence rather than mandate? On today’s show, we talk with JJ Asghar from IBM. JJ shares his extensive experience in managing open-source namespaces like GitHub and npm for IBM. He discusses the challenges of influencing decisions without formal authority and tailoring communication styles for different audiences. JJ also advocates for... Read more »N4N019: Howdy, Neighbor! And Other Routing Stuff
In today’s episode, we continue the discussion about routing and routing protocols by focusing on commonalities rather than differences among protocols such as OSPF, RIP, EIGRP, or BGP. We explain how, in general, routing protocols discover each other, communicate, maintain relationships, and exchange routing information. Next, we explore the topics of selecting best paths in... Read more »Response: Any-to-Any Connectivity in the Internet
Bob left a lengthy comment arguing with the (somewhat black-and-white) claims I made in the Rise of NAT podcast. Let’s start with the any-to-any connectivity:
From my young millennial point of view, the logic is reversed: it is because of NATs and firewalls that the internet became so asymmetrical (client/server) just like the Minitel was designed (yes, I am French), whereas the Internet (and later the web, although a client/server protocol, was meant for everyone to be a client and a server) was designed to be more balanced.
Let’s start with the early Internet. It had no peer-to-peer applications. It connected a few large computers (mainframes) that could act as servers but also allowed terminal-based user access and thus ran per-user clients.
AI Reshapes The Ethernet Datacenter Switch Market
Two decades ago, the hyperscalers and cloud builders started remaking the Ethernet switch market in the datacenter in their own image, and now it looks like AI training and inference is going to morph Ethernet switching in the datacenter once again. …
AI Reshapes The Ethernet Datacenter Switch Market was written by Timothy Prickett Morgan at The Next Platform.
Three chapters at Cloudflare: Programmer to CTO to Board of Directors
Today, after more than 13 years at the company, I am joining Cloudflare’s board of directors and retiring from my full-time position as CTO.
Back in 2012 I wrote a short post on my personal site simply titled: Programmer. The post announced that I’d recently joined a company called CloudFlare (still sporting that capital “F”) with the job title Programmer. I’d chosen that title in part because it was the very first title I’d ever had, and because it would reflect what I’d be doing at Cloudflare.
I had spent a lot of time working at startups—in technical and then management roles—and wanted to go back to the really technical part that I loved most. Cloudflare gave me that opportunity, and I worked on a lot of systems that make up the Cloudflare that so many people around the world use today.
Looking back on my time at the company it’s really, really hard to pick my top highlights. In 2019 I wrote 6,000 words on the experience of helping build Cloudflare. But here are five that stand out:
The night we finished the preparation to launch Universal SSL sticks in my memory. We set out to Continue reading
NAN088: See Something, Improve Something – An Iterative Approach to Automation Success
On today’s Network Automation Nerds, industry veteran Michael Bushong talks about lessons learned from failure. As the network industry grapples with automation and network engineers confront yet another cycle of upskilling and grinding out new certs, he warns against executives and practice leads aiming for the biggest, shiniest project. His advice? Find something that matters... Read more »Worth Reading: On Writing
One of the most significant problems engineers face when trying to improve their online presence is the “How do I start writing?” roadblock (hint: publishing bland AI-generated slop won’t get you far unless you aim to become a Thought Leader).
Zvi Mowshowitz collected links to over a dozen different writing styles, starting with JRR Tolkien. I’m pretty sure you’ll find something useful in that vast collection.
Project Jengo for Sable — final winners!
With Cloudflare’s victory against patent trolls Sable IP and Sable Networks in the books, it’s time to close out the case’s Project Jengo competition.
In our last update, we talked about the conclusion of Sable’s 3+ year campaign to extort a payment from Cloudflare based on meritless patent infringement claims. After Cloudflare’s victory at trial in February 2024, Sable finally — and fully — capitulated, agreeing to: (1) pay Cloudflare $225,000, (2) grant Cloudflare a royalty-free license to Sable’s entire patent portfolio, and (3) dedicate all of Sable’s patents to the public.
With the fight against Sable ended, we announced the Conclusion of the Case under the Project Jengo Sable Rules. Now that the Grace Period has passed, we are pleased to announce the final winners of Project Jengo for the Sable case!
Read on for background on the case, details on the Project Jengo final winners, and other patent troll-related updates.
For anyone unfamiliar with the Sable case, the story can be traced back all the way to 2006, when patent troll Sable bought patents from a company going out of business. In 2021, fifteen years after buying the patents, Sable filed suit Continue reading
Upstart Xsight Labs Raises Up The Programmable Switch Banner High
The best minds in networking spent the better part of two decades wrenching the control planes of switches and routers out of network devices and putting them into external controllers. …
Upstart Xsight Labs Raises Up The Programmable Switch Banner High was written by Timothy Prickett Morgan at The Next Platform.
PP055: News Roundup – BotNet Targets TP-Link, Threat Hunting In the Electric Grid, Apple Vs. UK Snoops, and More
This week we dive into security headlines including a botnet bonanza that includes TP-Link routers, Chinese attackers targeting Juniper and Fortinet, and a case study of nation-state actors penetrating the operator of a small US electric utility. We also discuss ransomware attacks targeting critical infrastructure, a backdoor in an Android variant used in streaming devices,... Read more »Scaling The Storage Pyramid With Micron’s Datacenter Memory And Flash
SPONSORED CONTENT Consider, for a moment, the current state of AI accelerators and datacenter GPUs. …
Scaling The Storage Pyramid With Micron’s Datacenter Memory And Flash was written by Timothy Prickett Morgan at The Next Platform.
HS098: Just Following Orders?
What do you do when your colleagues or senior leaders ask you to do something illegal? It’s hardly hypothetical; recent years have seen high-profile firings and convictions of CIOs and CISOs who’ve been ordered to break the law. John and Johna discuss steps that tech leaders can take if they’re put on the spot. Episode... Read more »
Build and deploy Remote Model Context Protocol (MCP) servers to Cloudflare
It feels like almost everyone building AI applications and agents is talking about the Model Context Protocol (MCP), as well as building MCP servers that you install and run locally on your own computer.
You can now build and deploy remote MCP servers to Cloudflare. We’ve added four things to Cloudflare that handle the hard parts of building remote MCP servers for you:
workers-oauth-provider — an OAuth Provider that makes authorization easy
McpAgent — a class built into the Cloudflare Agents SDK that handles remote transport
mcp-remote — an adapter that lets MCP clients that otherwise only support local connections work with remote MCP servers
AI playground as a remote MCP client — a chat interface that allows you to connect to remote MCP servers, with the authentication check included
The button below, or the developer docs, will get you up and running in production with this example MCP server in less than two minutes:
Unlike the local MCP servers you may have previously used, remote MCP servers are accessible on the Internet. People simply sign in and grant permissions to MCP clients using familiar authorization flows. We think this is going to be a massive deal — connecting coding agents Continue reading
Open-sourcing OpenPubkey SSH (OPKSSH): integrating single sign-on with SSH
OPKSSH makes it easy to SSH with single sign-on technologies like OpenID Connect, thereby removing the need to manually manage and configure SSH keys. It does this without adding a trusted party other than your identity provider (IdP).
We are excited to announce OPKSSH (OpenPubkey SSH) has been open-sourced under the umbrella of the OpenPubkey project. While the underlying protocol OpenPubkey became an open source Linux foundation project in 2023, OPKSSH was closed source and owned by BastionZero (now Cloudflare). Cloudflare has gifted this code to the OpenPubkey project, making it open source.
In this post, we describe what OPKSSH is, how it simplifies SSH management, and what OPKSSH being open source means for you.
A cornerstone of modern access control is single sign-on (SSO), where a user authenticates to an identity provider (IdP), and in response the IdP issues the user a token. The user can present this token to prove their identity, such as “Google says I am Alice”. SSO is the rare security technology that both increases convenience — users only need to sign in once to get access to many different systems — and increases security.
netlab 1.9.6: Static Routes to Default Gateways
Last week, I had to push out netlab release 1.9.6 to address a particularly nasty Python dependency hell to make netlab work (again) on Ubuntu 24.04 (more details). The release also brought these goodies (and a bunch of bug fixes):
- Add default gateway (including anycast- and VRRP gateway )as a valid next-hop for static routes
- Rewrite the default gateway processing and add IPv6 default gateways on links without anycast or VRRP gateways
- Set libvirt MTU to 9500 on bridge-based networks to avoid the “transparent fragmentation” on Linux bridges.
- Use device- or node variables to specify the Juniper vMX license file.
Cloudflare incident on March 21, 2025
Multiple Cloudflare services, including R2 object storage, experienced an elevated rate of errors for 1 hour and 7 minutes on March 21, 2025 (starting at 21:38 UTC and ending 22:45 UTC). During the incident window, 100% of write operations failed and approximately 35% of read operations to R2 failed globally. Although this incident started with R2, it impacted other Cloudflare services including Cache Reserve, Images, Log Delivery, Stream, and Vectorize.
While rotating credentials used by the R2 Gateway service (R2's API frontend) to authenticate with our storage infrastructure, the R2 engineering team inadvertently deployed the new credentials (ID and key pair) to a development instance of the service instead of production. When the old credentials were deleted from our storage infrastructure (as part of the key rotation process), the production R2 Gateway service did not have access to the new credentials. This ultimately resulted in R2’s Gateway service not being able to authenticate with our storage backend. There was no data loss or corruption that occurred as part of this incident: any in-flight uploads or mutations that returned successful HTTP status codes were persisted.
Once the root cause was identified and we realized we hadn’t deployed Continue reading