Fat Fingers Strike Again…

Level3 had a pretty bad bad-hair-day just a day before Pete Lumbis talked about Continuous Integration on the Building Network Automation Solutions online course (yes, it was a great lead-in for Pete).

According to messages circulating on mailing lists it was all caused by a fumbled configuration attempt. My wild guess: someone deleting the wrong route map, causing routes that should have been tagged with no-export escape into the wider Internet.

BGP Route Selection: a Failure of Intent-Based Networking

It’s interesting how the same pundits who loudly complain about the complexities of BGP (and how it will be dead any time soon and replaced by an SDN miracle) also praise the beauties of intent-based networking… without realizing that the hated BGP route selection process represents one of the first failures of intent-based approach to networking.

Let’s start with some definitions. There are two ways to get a job done by someone else:

New Design on www.ipSpace.net

One of my readers sent me a polite email a while ago saying “your site is becoming like $majorVendor’s web site – every corner looks completely different based on when you made it”

Source: Wikimedia Commons

The worst part is that he was right, so I spent the last two weeks as a website janitor, mopping up broken markup, fixing CSS cracks, polishing old texts…

Wow, another Year Swooshed By…

Could you believe it? Another year swooshed by… and it’s high time to stop being snarky and cynical, disconnect from the Internet, and spend a few days with people who really matter – our families.

For me, there’s another large group of people that matter: my users.

Unique IPv6 Prefix Per Host – How Complex Do You Want IPv6 to Be?

In December 2017 IETF published RFC 8273 created by the v6ops working group (which means there must have been significant consensus within the working group that we need the solution and that it makes at least marginal sense).

The RFC specifies a mechanism by which the first-hop router allocates a unique /64 IPv6 prefix for every host attached to a subnet and uses unicast and multicast RA responses sent to unicast MAC addresses to give every host the impression that it’s the sole host on its own subnet.

The first thought of anyone even vaguely familiar with how complex IPv6 already is should be “WTF???” Unfortunately, there are good reasons we need this monstrosity.

Salt Deep Dive on Building Network Automation Solutions Online Course

In the first few sessions of the Building Network Automation Solutions online course we used Ansible as the tool-of-choice because it’s the easiest automation tool to get started with. Now that we’ve established the baseline, it’s time to explore the alternatives.

In a live session on February 27th 2018, Mircea Ulinic will describe Salt, an open source, general-purpose event-driven automation framework that we briefly discussed in Episode 77 of Software Gone Wild podcast.

BGP: the Tragedy of the Commons

Every now and then someone looks at a few recent BGP incidents (from fat fingers to more dubious ones) and says “we need a better BGP”.

It’s like being unable to cope with your kids or your team members because you don’t have the guts to tell them NO and trying to solve the problem by implementing new procedures and rules.

Like anything designed on a few napkins BGP has its limit. They’re well known, and most of them have to do with trusting your neighbors instead of checking what they tell you.

The solutions to the problem are pretty simple and have been known for decades (BCP38 was published in May 2000). In a nutshell you have to:

• Build a global repository of who owns what address space;
• Document who connects to whom and what their peering policies are;
• Filter the updates received from your customers and peers based on the information from those repositories;
• Filter the traffic from addresses that are obviously spoofed.

We have most of the tools we need to get the job done; you’ll find them described in Best Current Practice (BCP) 194. It’s also not impossible to get the job done Continue reading

Please Respond: Survey on Interconnection Agreements

Marco Canini is working on another IXP-related research project and would like to get your feedback on inter-AS interconnection agreements, or as he said in an email he sent me:

As academics, it would be extremely valuable for us to receive feedback from network operators in the industry.

It’s fantastic to see researchers who want to base their work on real-life experience (as opposed to ideas that result in great-looking YouTube videos but fail miserably when faced with reality), so if you’re working for an ISP please take a few minutes and fill out this survey.

What Exactly Should My MAC Address Be?

Looks like I’m becoming the gateway-of-last-resort for people encountering totally weird Nexus OS bugs. Here’s another beauty…

I'm involved in a Nexus 9500 (NX-OS) migration project, and one bug recently caused vPC-connected Catalyst switches to err-disable (STP channel-misconfig) their port-channel members (CSCvg05807), effectively shutting down the network for our campus during what was supposed to be a "non-disruptive" ISSU upgrade.

Weird, right? Wait, there’s more…

Video: Avaya [now Extreme] Data Center Solutions

I haven’t done an update on what Avaya was doing in the data center space for years, so I asked my good friend Roger Lapuh to do a short presentation on:

• Avaya’s data center switches and their Shortest Path Bridging (SPB) fabric;
• SPB fabric features;
• Interesting use cases enabled by SPB fabric.

The videos are now available to everyone with a valid ipSpace.net account – the easiest way to get it is a trial subscription.

Data Center BGP: Autonomous Systems and AS Numbers

Two weeks ago we discussed whether it makes sense to use BGP as the routing protocol in a data center fabric. Today we’ll tackle three additional design challenges:

• Should you use IBGP or EBGP?
• When should you run BGP on the spine switches?
• Should every leaf switch have a different AS number or should they share the same AS number?

Create IP Multicast Tree Graphs from Operational Data

A while ago I created an Ansible playbook that creates network diagrams from LLDP information. Ben Roberts, a student in my Building Network Automation Solutions online course used those ideas to create an awesome solution: he’s graphing multicast trees.

Here’s how he described his solution:

Moving Complexity to Application Layer?

One of my readers sent me this question:

One thing that I notice is you mentioned moving the complexity to the upper layer. I was wondering why browsers don't support multiple IP addresses for a single site – when a browser receives more than one IP address in a DNS response, it could try to perform TCP SYN to the first address, and if it fails it will move to the other address. This way we don't need an anycast solution for DR site.

Of course I pointed out an old blog post ;), and we all know that Happy Eyeballs work this way.

First Speakers in the Spring 2018 Automation Online Course

For the first two sessions of the Building Network Automation Solutions online course I got awesome guest speakers, and it seems we’ll have another fantastic lineup in the Spring 2018 course:

Most network automation solutions focus on device configuration based on user request – service creation or change of data model describing the network. Another very important but often ignored aspect is automatic response to external events, and that’s what David Gee will describe in his presentation.

New Content: Debugging Ansible Playbooks and Jinja2 Templates

Here’s a quote from one of my friends who spent years working with Ansible playbooks:

Debugging Ansible is one of the most terrible experiences one can endure…

It’s not THAT bad, particularly if you have a good debugging toolbox. I described mine in the Debugging Ansible Playbooks part of the Ansible for Networking Engineers online course.

Please note that the Building Network Automation Solutions online course includes all material from the Ansible online course.

Worth Reading: Postpone Inbox Procrastination

This blog post by Ethan Bank totally describes my (bad) Inbox habits. If you're anything like me, you might find Ethan's ideas useful (I do... following them is a different story though).

Video: Switch Buffer Architectures

A while ago (in the time of big-versus-small buffers brouhaha) I asked JR Rivers to do a short presentation focusing on buffering requirements of data center switches. He started by describing typical buffer architectures you might find in data center switches.

Response: Upgrading Network Device Software

I got numerous responses to the “Why Does It Take So Long to Upgrade Network Devices,” the best ones coming from Béla Várkonyi and Frederic Cuiller.

Béla is sick-and-tired of the stuff vendors are shipping:

Automate Remote Site Hardware Refresh Process

Every time we finish the Building Network Automation Solutions online course I ask the attendees to share their success stories with me. Stan Strijakov was quick to reply:

I have yet to complete the rest of the course and assignments, but the whole package was a tremendous help for me to get our Ansible running. We now deploy whole WAN sites within an hour.

Of course I wanted to know more and he sent me a detailed description of what they’re doing:

Stop Googling and Start Testing

Here’s a question I got on one of my ancient blog posts:

How many OSPF process ID can be used in a single VRF instance?

Seriously? You have to ask that? OK, maybe the question isn’t as simple as it looks. It could be understood as: