Worth Reading: Expert Generalists

Martin Fowler published an interesting article about Expert Generalists. Straight from the abstract:

As computer systems get more sophisticated we’ve seen a growing trend to value deep specialists. But we’ve found that our most effective colleagues have a skill in spanning many specialties.

Also:

There are two sides to real expertise. The first is the familiar depth: a detailed command of one domain’s inner workings. The second, crucial in our fast-moving field is the ability to learn quickly, spot the fundamentals that run beneath shifting tools and trends, and apply them wherever we land.

Remember how I told you to focus on the fundamentals? 😎

Molly-Guard: a Lifesaver on a Ubuntu Server

Have you ever managed to type reload in the wrong terminal window and brought down a core switch (I probably did)? I managed to do the Ubuntu equivalent of that stupidity: I told my main Ubuntu server to sudo poweroff instead of doing that to a Vagrant VM.

Fortunately, the open-source world doesn’t have to rely on the roadmaps created by networking vendors’ product managers; if there’s a big enough pain, someone will solve it.

IS-IS 3-Way Handshake and the Power of SHOULD

Yesterday, I mentioned that a Cisco router running pre-standard IS-IS 3-way handshake (this is why you need it) interoperates with multiple implementations of RFC 5303. How’s that possible, and does it matter whether you configure the ancient Cisco routers (release 15.x) to use IETF 3-way handshake instead of the “proprietary” one?

I took a trip to the Wireshark land to figure out the details (you can download the capture file):

Start netlab Tools without Changing Topology File

Dan Partelly figured out that we have to configure the standard (IETF) 3-way IS-IS handshake on old IOSv images. On the other hand, all IS-IS integration tests pass for IOSv and IOSvL2. I wondered what was going on.

Fortunately, a few months ago, I spent some time installing the client-side Edgeshark components on my laptop. All I needed to do was enable the edgeshark tool in my lab topology and restart the lab.

SwiNOG 40: A Day of Awesomeness

A few days ago, I attended a SwiNOG meeting for the first time and realized what a mistake I was making — I should have been there years ago.

Not only was the event impeccably organized (what else would you expect in Switzerland) and at the best event location I have ever experienced (it’s hard to beat this view), it was also full of short, interesting, up-to-the-point presentations (you can already view the slide decks, YouTube videos should be available shortly). Plus, I met so many old friends I haven’t seen in years, and people I communicated with for years but never met before.

It’s not like the organizers would need any more publicity (the event was sold out), but if you happen to be near Switzerland in time for the next meeting, make sure to be there.

Thanks again to the wonderful SwiNOG core team for a fantastic experience! I hope we’ll meet again at the next SwiNOG meeting!

Testing OSPF Device Configurations

A year ago, I described how we use the netlab validate command to test device configuration templates for most platforms supported by netlab. That blog post included a simple “this is how you test interface address configuration” example; now, let’s move to something a bit more complex: baseline OSPF configuration.

Testing the correctness of OSPF configurations seems easy:

• Build a lab with a test device and a few other OSPF devices
• Configure the devices
• Log into the test device and inspect OSPF operational data

There’s just a tiny little fly in this ointment…

Quality of OSPFv2 NSSA Implementations

A few weeks ago, we added OSPF areas functionality to netlab. In the next release¹, you’ll be able to configure stub areas, NSSA areas, inter-area route summarization and filtering (OSPF ranges), and summarization of NSSA type-7 prefixes for OSPFv2 and OSPFv3.

OSPFv2 (defined in RFC 2328) is 27 years old, and NSSA functionality (RFC 3101) was last touched 22 years ago. One would hope the implementations in network devices are mature and feature-complete. Yeah, keep dreaming 🤦‍♂️.

Static Routes in netlab Lab Topologies

As much as we’d love everything in our networks to be dynamic, auto-configured, or software-defined, reality often intervenes and forces us to use static routes, so we needed a mechanism to specify them in netlab lab topologies.

A static route has two components: the destination prefix and the next hop – the device that we hope knows how to reach that destination. The next hop is usually specified as an IPv4 or IPv6 address, but may also include outgoing interface information¹.

Network Digital Twins: Between PowerPoint and Reality

A Network Artist left an interesting remark on one of my blog posts:

It’s kind of confusing sometimes to see the digital twin (being a really good idea) never really take off.

His remark prompted me to resurface a two-year-old draft listing a bunch of minor annoyances that make Networking Digital Twins more of a PowerPoint project than a reality.

Dear Vendors, EVPN Route Attributes Matter

Another scary tale from the Archives of Sloppy Code: we can’t decide whether some attributes are mandatory or optional.

When I was fixing the errors in netlab SR-OS configuration templates, I couldn’t get the EBGP-based EVPN with overlapping leaf AS numbers to work. I could see the EVPN routes in the SR-OS BGP table, but the device refused to use them. I concluded (incorrectly) that there must be a quirk in the SR-OS EVPN code and moved on.

Public Videos: Whole IPv6 Curriculum

Based on the feedback I received on LinkedIn and in private messages, I made all my IPv6 content public; you can watch those videos without an ipSpace.net account.

Want to spend more time watching free ipSpace.net videos? The complete list is here.

netlab 25.06: Fixing Nokia SR-OS Configuration Templates

TL&DR: netlab release 25.06 was published last week.

Before discussing the new features, let’s walk the elephant out of the room: I changed the release versions to YY.MM scheme, so I will never again have to waste my time on the existential question of which number in the release specification to increase.

Now for the new features:

Finding Source Routing Paths

In the previous blog post, we discussed the generic steps that network devices (or a centralized controller) must take to discover paths across a network. Today, we’ll see how these principles are applied in source routing, one of the three main ways to move packets across a network.

Brief recap: In source routing, the sender has to specify the (loose or strict) path a packet should take across the network. The sender thus needs a mechanism to determine that path, and as always, there are numerous solutions to this challenge. We’ll explore a few of them, using the sample topology shown in the following diagram.

ArubaCX Cannot Count When Dealing with VXLAN

This blog post describes yet another bizarre example of how reliable digital twins are, but don’t worry; they all work great in PowerPoint.

After “fixing” the integration tests to deal with ArubaCX’s notion of VXLAN VNI having 16 bits, the bridging test worked, but the IRB tests kept failing.

In the IRB test, the lab has two layer-3 switches. Each of them should be able to bridge within a VLAN/VXLAN segment and route across the segments.

netlab 2.0: Routers, Hosts, Gateways and Bridges

In a previous blog post, I explained how you can use bridges in a netlab topology to create custom LAN segments. Netlab supports two other node roles (host and router), and we’ll eventually add gateways.

netlab assumes that most network devices are routers (it considers a firewall to be a router in disguise), apart from Linux hosts, but you can always change what a node is with the role node attribute:

Interesting: Juniper MX and Jumbo Frames

Did you know that there’s an Ethernet link between the Packet Forwarding Engine (PFE – data plane) and Routing Engine (RE – control plane) in every Juniper MX? That’s why you have to run two VMs to emulate it (sometimes conveniently packed into one larger VM, proving RFC 1925 rule 6a).

That Ethernet link happens to have the MTU fixed at 1500 bytes. Guess what happens in the world where everyone uses jumbo frames? Did you say fragmentation? Bingo! And what do you think happens when one of those fragments gets dropped due to control-plane policing, and the rest of them are stuck in the reassembly queue? You’ll find the gory details in a lengthy blog post by Nitzan Tzelniker.

Publishing Content as an Introvert

I got an interesting question from a reader. He listened to my podcast with Eric Chou and decided to try to learn in public:

Currently, I’m studying for the CCNP ENARSI exam, and would like to start posting my labs to LinkedIn, and perhaps even upload my lab topologies and configs to Git.

That’s a great idea. I would minimize the LinkedIn part¹ and focus on Git:

Finding End-to-End Paths: Topology and Endpoints

We know there are three main ways to move packets across a network. However, before we can start forwarding packets, someone has to populate the forwarding tables in the intermediate devices or build the sequence of nodes to traverse in source routing.

Usually, whoever is responsible for the contents of the forwarding tables must first discover the network topology. Let’s start there, using the following network diagram to illustrate the discussion.

Weird: Ports on Linux Bridge Are Stuck

Just when you thought you got used to the weirdnesses in the networking implementations, you get a curveball like this one. Life is never dull if you test network devices.

Before releasing netlab release 2.0, I ran the full suite of integration tests for all devices for which I have the images. Interestingly, most VXLAN tests failed for Cumulus Linux 4.x even though we haven’t touched that code for ages.

Next step: trying to figure out what changed. The configuration changes were minimal. Even worse, the failure was non-deterministic. Somehow, we managed to transform a Cumulus Linux 4.x VM into a Heisenberg switch.

Where Are the NETCONF/YANG Tools?

Jo attempted to follow the vendor Kool-Aid recommendations and use NETCONF/YANG to configure network devices. Here’s what he found (slightly edited):

IMHO, the whole NETCONF ecosystem primarily suffers from a tooling problem. Or I haven’t found the right tools yet.

ncclient is (as you mentioned somewhere else) an underdocumented mess. And that undocumented part is not even up to date. The commit hash at the bottom of the docs page is from 2020… I am amazed how so many people got it working well enough to depend on it in their applications.