The Crappy Cisco IOS CLI Error Reporting

TL&DR: It’s really hard to figure out when a Cisco IOS box configured via its CLI reports a configuration error.

One of the drawbacks of using Netmiko to configure network devices is its inherent lack of error detection – while it can switch into configuration mode and exit it (including a commit operation if needed), it doesn’t detect configuration errors.

No big deal (I thought); the configuration error messages generated by Cisco IOS always start with a percent sign (%), so I’d just use that as the error_pattern parameter of the send_config_from_file command. A few integration test failures later, that turned out to be a bad idea; Cisco IOS starts errors and warnings with the percentage sign, and there’s absolutely no reliable way to differentiate between the two.

netlab 26.10: SRv6, More Netmiko

netlab release 26.10 adds new SRv6 implementations and features:

• Full SRv6 support (IS-IS, layer-3 services, L3VPN) on Junos
• SRv6 with IS-IS and SRv6 layer-3 services on Nokia SR Linux
• IPv4 layer-3 services with SRv6 locators exchanged over EBGP on FRRouting and Cisco IOS XR

Other new features include:

• Generic prefix sets for static routes
• containerlab nodes can start in batches or wait for other nodes to start.
• netlab install netmiko installs the Netmiko and Paramiko packages needed for Netmiko-based device configuration and helps you change the device configuration mode for supported devices.

Worth Reading: Don’t Fight the Users’ Desire Paths

Chris Siebenmann publishes articles written from an interesting perspective: he’s a Unix herder at a university (based on my ancient, similar experience, that’s usually worse than herding cats).

In one of his recent articles, he applied the desire paths ideas to IT: don’t fight how people are doing things; either kindly nudge them in the right direction, or help them get stuff done the way they like to get it done, but in the least harmful way. I could definitely use some of his wisdom decades ago 🤷🏻‍♂️.

Configuring SRv6 on Junos (Locator and IGP)

I wanted to make my SRv6 lab examples usable with more platforms. Adding netlab Junos SRv6 support (included in the 26.10 release) was relatively easy; here’s what I learned while working on the pull request:

MUST Bookmark: Watch the Net

Andre Toonk published yet another Internet-measuring tool: watchthenet.com tries to ping every IPv4 address every few hours (probing every /24 every 49 seconds), resulting in a rapidly updated state of the Internet. If an ISP has a bad hair day, Andre knows within minutes.

Make sure you bookmark his website and read his blog post for more details.

Prefix Sets: Simplifying netlab ACLs and Prefix Filters

When I implemented prefix filters in netlab (release 1.9.0), I wanted them to match the final device configurations as closely as possible, including the one prefix per entry rule to match the device configuration sequence numbers. That worked well, but resulted in YAML bloat, since each matching prefix requires several lines of YAML.

We tried to use the same approach with ACLs but quickly gave up when we realized the port not in range operation requires multiple ACL entries on most platforms. With the matching sequence numbers paradigm gone with the wind, it made no sense to limit ourselves to one prefix per ACL entry – the initial ACL entry implementation already accepted lists of prefixes and expanded them into ACL entries.

Public Videos: Reliability Theory

The 5-hour Reliability Theory: Networking through a Systems Analysis Lens webinar by Rachel Traylor is now available without a valid ipSpace.net account. Enjoy!

To watch more free ipSpace.net videos, browse them here, or click the Access Free Content button in any webinar roadmap.

Using Syslog in netlab Labs

netlab release 26.09 added support for Syslog clients and servers. We implemented the clients on numerous platforms; the only Syslog server implementation is the dnsmasq container.

Adding Syslog to a netlab lab is (almost) trivial:

• You have to (re)build the dnsmasq container with the netlab clab build dnsmasq command to add rsyslogd to the container image
• When you add a dnsmasq node to a lab, it automatically starts a Syslog server
• The Syslog client is enabled on other nodes (or globally) with the services.syslog.server setting.

The OSPF MTU Mismatch Saga

The OSPF is stuck in database exchange state symptom is probably familiar to every networking engineer who ever had to pass a certification exam. It’s caused by the MTU mismatch between adjacent OSPF routers.

The recent (unexpected) trip into the how crazy is it to declare your interface MTU to be zero wonderland uncovered way more than I ever wanted to know about this arcane topic. Here are a few highlights (thanks a million to Aleksey Popov for researching this topic and posting information here, here, and here).

EVPN with SR-MPLS Core

After a long journey, we finally we got to the final scenario in my ITNOG10 Segment Routing workshop: EVPN services over an SR-MPLS core.

I used the same lab topology as in the previous services-focused scenarios blog post, replacing two PE-to-host subnets with a stretched VLAN.

netlab 26.09: Syslog, More DNS, Netmiko

netlab release 26.09 brings more network services goodies:

• The services module supports Syslog clients and servers.
• DNS and Syslog are implemented on most supported platforms.

Other new features include:

• The new netmiko device configuration mode for virtual machines
• Generic prefix sets that simplify writing ACL rules and prefix lists

Sunsetting netlab Vagrant/libvirt provider

When I started the netlab project, Vagrant was the go-to tool if you wanted to build a virtual environment described in a text configuration file (an idea popularized as infrastructure-as-code). It wasn’t ideal for what we were doing, but a tool rarely does a great job when used far away from its intended use case.

netlab initially supported Vagrant with VirtualBox, quickly adding support for KVM/libvirt through the vagrant-libvirt plugin. Life was good… until it wasn’t.

How vagrant-libvirt Plugin Deals with Duplicate Subnets

TL&DR: Badly. The vagrant-libvirt plugin mysteriously crashes when an existing virtual network (with a different libvirt name) uses the same IP subnet as the desired management network.

Background: netlab is using the vagrant-libvirt plugin to manage libvirt/KVM virtual machines with Vagrant. As I already have that infrastructure, I use it to start standalone virtual machines (usually to test various Ubuntu releases) on my Linux server. Things work great… until they don’t.

Here’s how I managed to waste half a day chasing imaginary gremlins caused by a simple error.

MPLS/VPN with SR-MPLS Core

As we discussed in the BGP-Free Core with SR-MPLS blog post, SR-MPLS works as a drop-in replacement for the traditional MPLS control plane. No wonder it works well (when properly implemented) with MPLS/VPN services – the second “fun” scenario in my ITNOG10 Segment Routing workshop. It uses the same topology as the BGP-Free Core scenario:

Configuring Management IP Addresses to Virtual Network Devices

It goes without saying that if you want to configure (virtual) network devices with any semi-sane configuration mechanism¹, the device must have a working IP address. Here’s the time-honored method² to assign an IP address to a virtual network device:

• Start the virtual machine (using a GUI)³
• Open a new window: either a telnet session to the virtual console port or a full-blown virtual console (GUI) session.
• Manually configure the IP address, the SSH server, and the user credentials on the first interface.

Worth Reading: NatJack

Every time I wrote something along the lines of NAT is not a security feature, I got the expected pushback from people defending their bad suboptimal design choices.

Fortunately, rational¹ engineers no longer need to have that discussion: the NatJack website documents² a half-dozen attacks on typical NAT implementations.

Obvious next step: coping mechanisms like “this is all theoretical”, like the “but the remote host cannot reply” argument made 23 years after the Slammer worm 🤦‍♂️.

Worth Reading: An Interactive Introduction to the STP

I can’t even start imagining how much time Vincent Bernat spent creating his interactive introduction to the spanning tree protocol (starting with STP implemented in WebAssembly); the results are simply phenomenal (there’s also a video version for people who forgot how to read).

I don’t think we’ll ever need another STP intro after this one ;)

Please Help Me Follow Your Content

Finding good content in the swamp of AI slop is hard, and whenever I find a blog post worth reading, I’d like to know when the author publishes the next one.

For stupid reasons (like a day having only 24 hours, and me having a life), I can’t periodically scan all websites I find interesting, but fortunately, I don’t have to. Two well-known mechanisms work reliably: email updates and RSS feeds.

BGP-Free Core with SR-MPLS

The beauty of SR-MPLS is that it’s a drop-in replacement for the traditional (LDP- or RSVP-based) MPLS control plane. For example, you could replace LDP with SR-MPLS in a network using MPLS to implement a BGP-free transport core, and it just keeps working.

This scenario was the first “fun” scenario in the ITNOG10 Segment Routing workshop. The core network uses the same topology as in the previous examples; I added two hosts and BGP routing.

Networking Aspects of Running VMs in Containers

The vrnetlab project and its containerlab fork implement a wonderful idea: let’s package virtual machines (primarily network devices that cannot be containerized) as containers to use reliable orchestration tools like containerlab to provision network topologies.

That approach might have a few drawbacks (depending on how the container images are built), but the obvious elephant in the room is: how do you make the virtual network plumbing work?