Worth Reading: Securing NTP and the Origins of Time

Geoff Huston published an article supposedly describing the challenge of securing NTP, but as is usually the case, he couldn’t skip the prior art going all the way back (almost) to the formation of Earth.

Before coming to the how do we secure NTP section, you’ll learn everything about the wobbly Earth rotation, the changes in the Earth’s angular speed, the impact of tides, the smearing of leap seconds, the differences between UT1 and UTC, why we use quasars to measure time, and everything there is to know about NTP. Have fun!

Worth Reading: Why We’ve Tried to Replace Developers Every Decade

The never-ending “we will replace developers” (or networking engineers) pipe dream didn’t start with the latest bout of AI hype (or SDN). As Stephan Schwab explains in his Why We’ve Tried to Replace Developers Every Decade article, it started with COBOL, the magic high-level programming language that businesspeople would use to write their own programs.

At least some of us know how well that ended. I was also unfortunate to be there for the 5GL hype, the forms-driven programming hype, the “everyone will solve every problem out there with Excel macros” (it does work for networking inventory, doesn’t it?), and a few others. So please excuse me if I remain a bit skeptical about the latest fad, even though I find it (like all the previous ones) very useful when used conservatively in limited domains.

netlab: Switch to Lab Directory After an SSH Session Loss

I work on a laptop that loves to power down when not used (the right thing to do), which often breaks the SSH session to my netlab server (not so good).

Reconnecting is trivial. Figuring out which lab I was working on and where it lives on the disk after a few hours? That’s the annoying part.

We solved most of that ages ago with the netlab status --all command. It shows all running labs¹ and their directories, so you can quickly jump back to where you were. However, even that gets tedious the 100th time you have to do it.

Lab: Anycast Gateways on VXLAN Segments

Most vendors “discovered” anycast gateways when they tried implementing routing between MAC-VRFs in an EVPN environment and hit all the usual tripwires (more about that later). A few exceptions (like Arista) supported them on VLAN segments for over a decade, and it was a no-brainer to extend that support to VXLAN segments.

Want to try out how that works? The Anycast Gateways on VXLAN Segments lab exercise is just what you need.

You can run the lab on your own netlab-enabled infrastructure (more details), but also within a free GitHub Codespace or even on your Apple-silicon Mac (installation, using Arista cEOS container, using VXLAN/EVPN labs).

Arista EOS MPLS P/PE-router Behavior

Something didn’t feel right as I tried to check whether the IPv4 ECMP I observed in the latest version of Arista cEOS containers works with my MPLS/anycast scenario. The forwarding tables seemed OK, but I wasn’t getting MPLS labels in the ICMP replies (see RFC 4950 for details), even though I know Arista EOS can generate them.

I decided to go down that rabbit hole and built the simplest possible BGP-free core (the addition of BGP will become evident in a few seconds) to investigate PE/P-router behavior:

BGP Labs: Goodbye, Cumulus Linux

When I started the Online BGP Labs project in mid-2023, Cumulus Linux still seemed like a good platform to use. You could run devices as virtual machines (we were still supporting VirtualBox) or in containers (containerlab was improving with every release), and it looked more polished than bare-bones FRRouting.

Things only went downhill from there (from the perspective of offering a free and easy-to-use solution with a CLI resembling commonly-used devices):

MANRS for Enterprise Customers

In October 2023, I was talking about Internet routing security at the DEEP conference in Zadar, Croatia. After explaining the (obvious) challenges and the initiatives aimed at making Internet routing more secure (MANRS), I made my usual recommendation: vote with your wallet. However, if you’re a company in Croatia (or Slovenia, or a number of other countries), you’re stuck.

While ISPs in Croatia might be doing a great job, none of them is a MANRS participant¹, so we don’t know how good they are. The situation is not much better in Slovenia; the only ISPs claiming to serve Slovenia are Anexia (a cloud provider) and Go6 Institute, the small network operated by my good friend (and True Believer in IPv6 and MANRS) Jan Žorž. Moving further north, I was unable to get any useful data for Austria, as its country code (AT) also matches “No Data” string in MANRS table, resulting in over 500 hits.

netlab Graphs with Multi-Access Links

A netlab user wanted to create a nice-looking topology graph from a simple topology connecting a few devices to a broadcast (multi-access) link. I don’t have his exact topology, so we’ll use this one (skipping the details like setting device types)

nodes: [ r1, r2, h1, h2 ]

links:
- r1-r2
- interfaces: [ r1, r2, h1, h2 ]

This is what GraphViz generates based on netlab’s description of the lab topology:

Network Device Telemetry Protocols with Dinesh Dutt

Whenever I’m ranting about vendors changing their data models or APIs with every other release, there is inevitably a vendor engineer chiming in, saying, “Life would be so much better if the customers wouldn’t insist on doing screen scraping for the last 50 years.”

While some of that screen scraping is pure inertia, we sometimes have good reasons to do it rather than use protocols like NETCONF, gNMI, or protobufs. In Episode 205 of Software Gone Wild, I’m discussing some of those reasons and exploring the gap between vendor theory and reality with Dinesh Dutt, who is unlucky enough to have become the world’s foremost expert on crappy network telemetry.

IPv4 ECMP Works on Arista cEOS Release 4.35.2F

When I wrote about the anycast-ECMP-in-MPLS behavior in 2011, I had to use Cisco IOS to prove that ECMP worked, since Arista cEOS (running the Linux kernel for IP forwarding) didn’t install more than one equal-cost path into the Linux forwarding table.

Arista cEOS got better in the meantime; IPv4 ECMP works like a charm on cEOS release 4.35.02F. With the same lab topology I’d used in 2021, I was able to see the traffic spread across multiple nodes:

Dynamic Path MTU Discovery in Cloudflare One Client

Here’s an interesting tidbit from the what took them so long department: Cloudflare One Client continuously measures end-to-end MTU and adjusts the local tunnel interface MTU size accordingly (warning: there’s a fair amount of dubious handwaving over the interesting details), generating ICMP packet-too-big messages as close to the source as possible.

I managed to avoid VPN clients most of my life, so I have no idea whether this is a “finally someone figured that out 🎉” moment or a late catch-up to what other VPN clients have been doing for ages. Feedback (in comments or otherwise) would be most welcome!

netlab 26.03: EVPN/MPLS, IOS XR Features

netlab release 26.03 is out. Here are the highlights:

• New Cisco IOS XR features: SR-MPLS, MPLS/VPN, MPLS 6PE, EVPN over MPLS, VRRP, BGP session parameters, routing policies, prefix filters, AS-path filters, community filters, and static routes
• Juniper cRPD support (mostly by @leec-666)
• EVPN/VXLAN on Juniper vPTX and vJunos-router by @rickycraft
• OpenBSD BGP support by @remilocherer
• SR-MPLS support for OSPFv2 on EOS, FRR, IOS XE, IOS XR
• EVPN/MPLS improvements, including using SR-MPLS transport

For even more details, check the release notes.

Automating netlab-Based Cisco SD-WAN Deployment

We haven’t implemented support for Cisco SD-WAN in netlab yet, and we might never do so; after all, netlab isn’t meant to be a kitchen sink of vendor-specific features. However, having an open-source tool that uses input and output files with standardized encoding (JSON or YAML) makes it easy to develop an independent solution that adds functionality.

That’s exactly what Sebastien d’Argoeuves did: he developed a solution that automates Cisco SD-WAN deployment after the corresponding netlab lab is started, and published it in a GitHub repo. If you’re an SD-WAN fan, you must give it a try ;)

Presentation: netlab Overview and Use Cases

Yesterday, I had a short presentation on netlab use cases during the NetBCN event. It covered a dozen examples, from rapid prototyping to testing network automation software and arguing with vendor TAC. I added the “use cases” part of the presentation to the standard netlab presentation; you can view the results on ipSpace.net (no account or registration required).

The Tale of Two EVPN/MPLS Encapsulations

I decided it was high time to create EVPN/MPLS netlab integration tests and wanted to use the same approach I used for the EVPN/VXLAN ones:

• One of the PE-devices is the device we want to test
• The other PE-device is a device that is known to work (ideally, an FRRouting container).
• Bonus points if the other PE-device can generate operational data in JSON format. Using a device for which we already have a validation plugin is close to perfection.
• Add a P-router in the middle because MPLS.
• Attach some hosts to the two PE-devices (we’re testing two MAC-VRFs in the final version of the test)
• After validating everything that can reasonably be validated (OSPF session, IBGP session, EVPN AF on IBGP session), do the end-to-end pings and hope for the best.

This is the graph netlab created from the lab topology:

Worth Reading: Faster than Dijkstra?

Bruce Davie published a nice article explaining why it makes little sense to use an algorithm that’s supposedly faster than Dijkstra’s in link-state routing protocols.

Other interesting data points from the article (and linked presentations):

• People are running (a few) thousands of routers in a single area
• Running Dijkstra’s algorithm on an emulated network with 2000 nodes took 100 msec… in 2003 (page 18 of this NANOG presentation).

It turns out (as I expected) that all the noise about the need for new routing protocols we were experiencing a few years ago was either due to bad implementations or coming from nerds looking for new toys to play with.

netlab: Using L3VPN (MPLS/VPN) with SR-MPLS Core

Someone recently asked me whether it’s possible to use netlab to build an MPLS/VPN (technically, BGP/MPLS IP VPN) lab with SR-MPLS core. Of course, let’s build a simple lab using Arista EOS and Linux containers to implement this topology:

Here’s the lab topology we’ll use (also available on GitHub):

Lab: More Complex EVPN/VXLAN Bridging Scenario

In the first EVPN/VXLAN lab, we added the EVPN control plane to bridging over VXLAN. Now, let’s try out a more complex scenario: several EVPN MAC-VRFs mapped to different VLAN segments on individual PE-devices.

You can run the lab on your own netlab-enabled infrastructure (more details), but also within a free GitHub Codespace or even on your Apple-silicon Mac (installation, using Arista cEOS container, using VXLAN/EVPN labs).

Configuring 6PE Route Reflector on Cisco IOS

Imagine you want to deploy a BGP route reflector for MPLS 6PE or L3VPN service. Both services run over MPLS LSPs, use IPv4 BGP sessions, and use IPv4 next hops for BGP routes. There’s absolutely no reason to need IPv6 routing on a node that handles solely the control-plane activity (it never appears as a BGP next hop anywhere), right? Cisco IOS disagrees, as I discovered when running route reflector integration tests for netlab 6PE and (MPLS) L3VPN functionality.

Most platforms failed those tests because we forgot to configure route-reflector-clients in labeled IPv6 and VPNv4/VPNv6 address families¹. That was easy to fix, but the IOS-based devices were still failing the tests, with nothing in the toolchain ever complaining about configuration problems.

On AI Agents Speaking BGP

I guess your LinkedIn feed is as full of AI nonsense as mine is, so I usually just skip all that posturing. However, every now and then, I stumble upon an idea that makes sense… until you start to dig deeper into it.

There was this post about AI agents speaking BGP with an associated GitHub repo, so I could go take a look at what it’s all about.

The proof-of-concept (so the post author) has two components: