Hedge 264: Documentation and Tech Debt
On this episode of the Hedge, Eyvonne, Tom, and Russ talk about topics near and dear to every network engineer’s heart–documentation, legacy, and tech debt. What should our philosophy of documentation be? What are legacy, end of life, and tech debt, really?
Passive BGP Sessions
The Dynamic BGP Peers lab exercise gave you the opportunity to build a large-scale environment in which routers having an approved source IP addresses (usually matching an ACL/prefix list) can connect to a BGP route reflector or route server.
In a more controlled environment, you’d want to define BGP neighbors on the BGP RR/RS but not waste CPU cycles trying to establish BGP sessions with unreachable neighbors. Welcome to the world of passive BGP sessions.
Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to session/8-passive and execute netlab up.
TL010: Leading With Influence Rather Than Mandating
How do you lead with influence rather than mandate? On today’s show, we talk with JJ Asghar from IBM. JJ shares his extensive experience in managing open-source namespaces like GitHub and npm for IBM. He discusses the challenges of influencing decisions without formal authority and tailoring communication styles for different audiences. JJ also advocates for... Read more »N4N019: Howdy, Neighbor! And Other Routing Stuff
In today’s episode, we continue the discussion about routing and routing protocols by focusing on commonalities rather than differences among protocols such as OSPF, RIP, EIGRP, or BGP. We explain how, in general, routing protocols discover each other, communicate, maintain relationships, and exchange routing information. Next, we explore the topics of selecting best paths in... Read more »Response: Any-to-Any Connectivity in the Internet
Bob left a lengthy comment arguing with the (somewhat black-and-white) claims I made in the Rise of NAT podcast. Let’s start with the any-to-any connectivity:
From my young millennial point of view, the logic is reversed: it is because of NATs and firewalls that the internet became so asymmetrical (client/server) just like the Minitel was designed (yes, I am French), whereas the Internet (and later the web, although a client/server protocol, was meant for everyone to be a client and a server) was designed to be more balanced.
Let’s start with the early Internet. It had no peer-to-peer applications. It connected a few large computers (mainframes) that could act as servers but also allowed terminal-based user access and thus ran per-user clients.
Three chapters at Cloudflare: Programmer to CTO to Board of Directors
Today, after more than 13 years at the company, I am joining Cloudflare’s board of directors and retiring from my full-time position as CTO.
Back in 2012 I wrote a short post on my personal site simply titled: Programmer. The post announced that I’d recently joined a company called CloudFlare (still sporting that capital “F”) with the job title Programmer. I’d chosen that title in part because it was the very first title I’d ever had, and because it would reflect what I’d be doing at Cloudflare.
I had spent a lot of time working at startups—in technical and then management roles—and wanted to go back to the really technical part that I loved most. Cloudflare gave me that opportunity, and I worked on a lot of systems that make up the Cloudflare that so many people around the world use today.
Looking back on my time at the company it’s really, really hard to pick my top highlights. In 2019 I wrote 6,000 words on the experience of helping build Cloudflare. But here are five that stand out:
The night we finished the preparation to launch Universal SSL sticks in my memory. We set out to Continue reading
Introducing Calico 3.30: A New Era of Open Source Network Security and Observability for Kubernetes
When we first launched Project Calico in 2016, we set out to make Kubernetes networking easy, reliable, and scalable for all organizations. Our goal was to abstract away the complexity and performance overheads of other CNI plugins while simultaneously extending Kubernetes network policy to make it easier to secure your Kubernetes workloads.
Over the last 9 years, we’ve seen our community grow alongside Calico Open Source, which has become the most widely adopted Kubernetes networking tool that now powers over 8 million nodes across more than 166 countries. We’ve seen the challenges our community has faced as more organizations adopt Kubernetes, and as the scale and complexity of these Kubernetes deployments has increased. Through our commercial offerings, we’ve helped solve networking and network security challenges for some of the world’s largest Kubernetes deployments, from financial institutions to telcos.
Calico OSS 3.30
With the release of Calico OSS 3.30 in May, we are open sourcing our battle-tested observability and security tools from our commercial editions. This includes the following key features:
- Goldmane – A gRPC-based API for accessing and capturing flow logs and policy evaluation metrics
- Whisker – A web-based tool for viewing and filtering flow logs to troubleshoot Continue reading
NAN088: See Something, Improve Something – An Iterative Approach to Automation Success
On today’s Network Automation Nerds, industry veteran Michael Bushong talks about lessons learned from failure. As the network industry grapples with automation and network engineers confront yet another cycle of upskilling and grinding out new certs, he warns against executives and practice leads aiming for the biggest, shiniest project. His advice? Find something that matters... Read more »Worth Reading: On Writing
One of the most significant problems engineers face when trying to improve their online presence is the “How do I start writing?” roadblock (hint: publishing bland AI-generated slop won’t get you far unless you aim to become a Thought Leader).
Zvi Mowshowitz collected links to over a dozen different writing styles, starting with JRR Tolkien. I’m pretty sure you’ll find something useful in that vast collection.
Project Jengo for Sable — final winners!
With Cloudflare’s victory against patent trolls Sable IP and Sable Networks in the books, it’s time to close out the case’s Project Jengo competition.
In our last update, we talked about the conclusion of Sable’s 3+ year campaign to extort a payment from Cloudflare based on meritless patent infringement claims. After Cloudflare’s victory at trial in February 2024, Sable finally — and fully — capitulated, agreeing to: (1) pay Cloudflare $225,000, (2) grant Cloudflare a royalty-free license to Sable’s entire patent portfolio, and (3) dedicate all of Sable’s patents to the public.
With the fight against Sable ended, we announced the Conclusion of the Case under the Project Jengo Sable Rules. Now that the Grace Period has passed, we are pleased to announce the final winners of Project Jengo for the Sable case!
Read on for background on the case, details on the Project Jengo final winners, and other patent troll-related updates.
For anyone unfamiliar with the Sable case, the story can be traced back all the way to 2006, when patent troll Sable bought patents from a company going out of business. In 2021, fifteen years after buying the patents, Sable filed suit Continue reading
PP055: News Roundup – BotNet Targets TP-Link, Threat Hunting In the Electric Grid, Apple Vs. UK Snoops, and More
This week we dive into security headlines including a botnet bonanza that includes TP-Link routers, Chinese attackers targeting Juniper and Fortinet, and a case study of nation-state actors penetrating the operator of a small US electric utility. We also discuss ransomware attacks targeting critical infrastructure, a backdoor in an Android variant used in streaming devices,... Read more »HS098: Just Following Orders?
What do you do when your colleagues or senior leaders ask you to do something illegal? It’s hardly hypothetical; recent years have seen high-profile firings and convictions of CIOs and CISOs who’ve been ordered to break the law. John and Johna discuss steps that tech leaders can take if they’re put on the spot. Episode... Read more »
Build and deploy Remote Model Context Protocol (MCP) servers to Cloudflare
It feels like almost everyone building AI applications and agents is talking about the Model Context Protocol (MCP), as well as building MCP servers that you install and run locally on your own computer.
You can now build and deploy remote MCP servers to Cloudflare. We’ve added four things to Cloudflare that handle the hard parts of building remote MCP servers for you:
workers-oauth-provider — an OAuth Provider that makes authorization easy
McpAgent — a class built into the Cloudflare Agents SDK that handles remote transport
mcp-remote — an adapter that lets MCP clients that otherwise only support local connections work with remote MCP servers
AI playground as a remote MCP client — a chat interface that allows you to connect to remote MCP servers, with the authentication check included
The button below, or the developer docs, will get you up and running in production with this example MCP server in less than two minutes:
Unlike the local MCP servers you may have previously used, remote MCP servers are accessible on the Internet. People simply sign in and grant permissions to MCP clients using familiar authorization flows. We think this is going to be a massive deal — connecting coding agents Continue reading
Open-sourcing OpenPubkey SSH (OPKSSH): integrating single sign-on with SSH
OPKSSH makes it easy to SSH with single sign-on technologies like OpenID Connect, thereby removing the need to manually manage and configure SSH keys. It does this without adding a trusted party other than your identity provider (IdP).
We are excited to announce OPKSSH (OpenPubkey SSH) has been open-sourced under the umbrella of the OpenPubkey project. While the underlying protocol OpenPubkey became an open source Linux foundation project in 2023, OPKSSH was closed source and owned by BastionZero (now Cloudflare). Cloudflare has gifted this code to the OpenPubkey project, making it open source.
In this post, we describe what OPKSSH is, how it simplifies SSH management, and what OPKSSH being open source means for you.
A cornerstone of modern access control is single sign-on (SSO), where a user authenticates to an identity provider (IdP), and in response the IdP issues the user a token. The user can present this token to prove their identity, such as “Google says I am Alice”. SSO is the rare security technology that both increases convenience — users only need to sign in once to get access to many different systems — and increases security.
netlab 1.9.6: Static Routes to Default Gateways
Last week, I had to push out netlab release 1.9.6 to address a particularly nasty Python dependency hell to make netlab work (again) on Ubuntu 24.04 (more details). The release also brought these goodies (and a bunch of bug fixes):
- Add default gateway (including anycast- and VRRP gateway )as a valid next-hop for static routes
- Rewrite the default gateway processing and add IPv6 default gateways on links without anycast or VRRP gateways
- Set libvirt MTU to 9500 on bridge-based networks to avoid the “transparent fragmentation” on Linux bridges.
- Use device- or node variables to specify the Juniper vMX license file.
Cloudflare incident on March 21, 2025
Multiple Cloudflare services, including R2 object storage, experienced an elevated rate of errors for 1 hour and 7 minutes on March 21, 2025 (starting at 21:38 UTC and ending 22:45 UTC). During the incident window, 100% of write operations failed and approximately 35% of read operations to R2 failed globally. Although this incident started with R2, it impacted other Cloudflare services including Cache Reserve, Images, Log Delivery, Stream, and Vectorize.
While rotating credentials used by the R2 Gateway service (R2's API frontend) to authenticate with our storage infrastructure, the R2 engineering team inadvertently deployed the new credentials (ID and key pair) to a development instance of the service instead of production. When the old credentials were deleted from our storage infrastructure (as part of the key rotation process), the production R2 Gateway service did not have access to the new credentials. This ultimately resulted in R2’s Gateway service not being able to authenticate with our storage backend. There was no data loss or corruption that occurred as part of this incident: any in-flight uploads or mutations that returned successful HTTP status codes were persisted.
Once the root cause was identified and we realized we hadn’t deployed Continue reading
NB519: Google Bids $32 Billion for Cloud Security Startup; NVIDIA Makes Nice With Quantum Computing
Take a Network Break! This week we cover Google’s $32 billion acquisition of CNAPP provider Wiz, Cloudflare offerings for AI security and support for post-quantum encryption, and NVIDIA’s pledge to open a quantum research center in Boston. NVIDIA has also announced new switch platforms with co-packaged optics for greater efficiency, Cisco shares details on its... Read more »0324 Tech Byte: Tech Bytes: How Fortinet Unified SASE Secures Hybrid Workers for Customer Liquid Networx (Sponsored)
Today on the Tech Bytes podcast, sponsored by Fortinet, we get a customer view of Fortinet’s SASE offering from Liquid Networx. Liquid Networx isn’t just a Fortinet customer; it also provides professional services for other customers of FortiSASE. We’ll talk about why Liquid Networx decided to adopt SASE, its evolution from on-prem to cloud-based security,... Read more »Security Week 2025: in review
Thank you for following along with another Security Week at Cloudflare. We’re extremely proud of the work our team does to make the Internet safer and to help meet the challenge of emerging threats. As our CISO Grant Bourzikas outlined in his kickoff post this week, security teams are facing a landscape of rapidly increasing complexity introduced by vendor sprawl, an “AI Boom”, and an ever-growing surface area to protect.
As we continuously work to meet new challenges, Innovation Weeks like Security Week give us an invaluable opportunity to share our point of view and engage with the wider Internet community. Cloudflare’s mission is to help build a better Internet. We want to help safeguard the Internet from the arrival of quantum supercomputers, help protect the livelihood of content creators from unauthorized AI scraping, help raise awareness of the latest Internet threats, and help find new ways to help reduce the reuse of compromised passwords. Solving these challenges will take a village. We’re grateful to everyone who has engaged with us on these issues via social media, contributed to our open source repositories, and reached out through our technology partner program to work with us on the issues most Continue reading
New URLPattern API brings improved pattern matching to Node.js and Cloudflare Workers
Today, we are excited to announce that we have contributed an implementation of the URLPattern API to Node.js, and it is available starting with the v23.8.0 update. We've done this by adding our URLPattern implementation to Ada URL, the high-performance URL parser that now powers URL handling in both Node.js and Cloudflare Workers. This marks an important step toward bringing this API to the broader JavaScript ecosystem.
Cloudflare Workers has, from the beginning, embraced a standards-based JavaScript programming model, and Cloudflare was one of the founding companies for what has evolved into ECMA's 55th Technical Committee, focusing on interoperability between Web-interoperable runtimes like Workers, Node.js, Deno, and others. This contribution highlights and marks our commitment to this ongoing philosophy. Ensuring that all the JavaScript runtimes work consistently and offer at least a minimally consistent set of features is critical to ensuring the ongoing health of the ecosystem as a whole.
URLPattern API contribution is just one example of Cloudflare’s ongoing commitment to the open-source ecosystem. We actively contribute to numerous open-source projects including Node.js, V8, and Ada URL, while also maintaining our own open-source initiatives like workerd and wrangler. By upstreaming improvements Continue reading