Must Read: Ironies of Automation

Stumbled upon a 35-year-old article describing the ironies of automation (HT: The Morning Paper). Here’s a teaser…

Unfortunately automatic control can ‘camouflage’ system failure by controlling against the variable changes, so that trends do not become apparent until they are beyond control.

In simpler words: when things fail, they fail really badly because the intermittent failures were kept hidden. Keep that in mind the next time someone tells you how wonderful software-defined AI-assisted networking is going to be.

Another perspective on "engineering" in IT

Found a nice article about Margaret Hamilton, the lady who coined the term "software engineering".

Engineering—back in 1969 as well as here in 2020—carries a whole set of associated values with it, and one of the most important is the necessity of proofing for disaster before human usage. You don’t “fail fast” when building a bridge: You ensure the bridge works first.

Now be a good "networking engineer" and go and stretch another VLAN around the globe... ;)

NetDev 0x13 on Software Gone Wild

The last Software Gone Wild podcast recorded in 2019 focused on advances in Linux networking - in particular on interesting stuff presented at NetDev 0x13 conference in Prague. The guests (in alphabetical first name order) Jamal Hadi Salim, Shrijeet Mukherjee, Sowmini Varadhan, and Tom Herbert shared their favorite topics, and commented on the future of Linux networking.

Worth Reading: Apple II Had the Lowest Input Latency Ever

It's amazing how heaping layers of complexity (see also: SDN or intent-based whatever) manages to destroy performance faster than Moore's law delivers it. The computer with lowest keyboard-to-screen latency was (supposedly) Apple II built in 1983, with modern Linux having keyboard-to-screen RTT matching the transatlantic links.

No surprise there: Linux has been getting slower with every kernel release and it takes an enormous effort to fine-tune it (assuming you know what to tune). Keep that in mind the next time someone with a hefty PPT slide deck will tell you to build a "provider cloud" with packet forwarding done on Linux VMs. You can make that work, and smart people made that work, but you might not have the resources to replicate their feat.

Do We Need Math in Networking?

I should have known better, but I couldn’t resist being pulled into a Twitter spat around the question “whether networking engineers need to know something about math” a long while ago.

Before going into the details, let’s start with Wikipedia definition: “Engineering is the use of scientific principles to design and build machines, structures, and other things” including “specific emphasis on particular areas of applied mathematics, applied science, and types of application”.

So feel free to believe that you don’t need any math or other science (because there’s very little science behind what we do in networking) in your job, in which case you might want to stop reading… but then at least please think twice about your job title.

There Is no Layer-2 in Public Cloud

The amount of layer-2 tricks we use to make enterprise networking work never ceases to amaze me - from shared IP addresses used by various clustering solutions (because it’s too hard to read the manuals and configure DNS) to shared MAC addresses used by first-hop router redundancy protocols (because it would be really hard to send a Gratuitous ARP message on failover) and all sorts of shenanigans we’re forced to engage in to enable running servers to be moved willy-nilly around the Earth.

Review: Webinars in 2019

A year ago I wrote about ipSpace.net plans for 2019. While we created over 50 hours of webinar content and over 20 hours of course content in 2019, let’s see how well we delivered on my promises.

Following AWS Networking webinar, we’ll do a similar one on Azure (in early autumn 2019) and GCP (late 2019 or early 2020)

Azure: done. GCP: postponed. Let’s see how serious Google is about that particular project.

Worth Reading: Understanding Scale Computing HC3 Edge Fabric

A long while ago someone told me about a "great" idea of using multi-port server NICs to build ad-hoc (or hypercube or whatever) server-only networks. It's pretty easy to prove that the approach doesn't make sense if you try to build generic any-to-any-connectivity infrastructure... but it makes perfect sense in a small environment.

One can only hope Scale Computing keeps their marketing closer to reality than some major vendors (that I will not name because I'm sick-and-tired of emails from their employees telling me how I'm unjustly picking on the stupidities their marketing is evangelizing) and will not start selling this approach as save-the-world panacea... but we can be pretty sure there will be people out there using it in way-too-large environments, and then blame everything else but their own ignorance or stubbornness when the whole thing explodes into their faces.

Worth Reading: Why Doctors Think They’re the Best

Another great article by Scott Alexander, this time explaining the consequences of selection bias. The applicability to network engineering (and continuous use of shaky solutions) is left as an exercise for the reader.

Learning, Grades and Certifications

Cleaning my Inbox I found links to two interesting blog post: The Lesson to Unlearn and Coders in the Hand of a Missing God.

You might also want to read my old certification ramblings, or watch the How Networks Really Work webinar in case you care more about knowledge than certifications.

That’s It for 2019 ;))

Yesterday we ran the last live webinar session for 2019, and all we have to do before locking the virtual office doors and returning to our families is to get the videos back from our editor and publish them. The “this is what we did in 2019” and “this is what we plan to do in 2020” blog posts will have to wait till we come back.

We’ll be (mostly) gone until early January… unless of course you have an urgent support problem. The desperate paperwork ideas like “I need you to sign, stamp, and notarize this legal document written in a language you’ve never seen in your life” will have to survive for a few weeks without our immediate attention (but then they usually don’t disappear on their own).

I hope you’ll be able to do something similar, disconnect from the crazy pace of networking world, forget all the unicorns you’ve been trying to tame during this year, and focus on your loved ones - they need you more than the router sitting in the basement of your building. We would also like to wish you all the best in 2020!

Oh, and Continue reading

Automation Solution: Data Center Fabric with Tenant Connectivity

I always tell attendees in the Building Network Automation Solutions to create minimalistic data models with (preferably) no redundant information. Not surprisingly, that’s a really hard task (see this article for an example) - using a simple automation tool like Ansible you end with either a messy and redundant data model or Jinja2 templates (or Ansible playbooks) full of hard-to-understand and impossible-to-maintain business logic.

Figure Out What Problem You’re Trying to Solve

A long while ago I got into an hilarious Tweetfest (note to self: don’t… not that I would ever listen) starting with:

Which feature and which Cisco router for layer2 extension over internet 100Mbps with 1500 Bytes MTU

The knee-jerk reaction was obvious: OMG, not again. The ugly ghost of BRouters (or is it RBridges or WAN Extenders?) has awoken. The best reply in this category was definitely:

I cannot fathom the conversation where this was a legitimate design option. May the odds forever be in your favor.

A dozen “this is a dumpster fire” tweets later the problem was rephrased as:

Video: FRRouting Overview

In October 2019, Donald Sharp did a short webinar describing FRRouting, the hottest open-source routing suite.

As always, he started with an overview of what FRRouting is, and where you could use it.

You Don’t Need IP Renumbering for Disaster Recovery

This is a common objection I get when trying to persuade network architects they don’t need stretched VLANs (and IP subnets) to implement data center disaster recovery:

Changing IP addresses when activating DR is hard. You’d have to weigh the manageability of stretching L2 and protecting it, with the added complexity of breaking the two sites into separate domains [and subnets]. We all have apps with hardcoded IP’s, outdated IPAM’s, Firewall rules that need updating, etc.

Let’s get one thing straight: when you’re doing disaster recovery there are no live subnets, IP addresses or anything else along those lines. The disaster has struck, and your data center infrastructure is gone.

Practice Your Public Cloud Networking with Hands-On Exercises

Design assignments and hands-on exercises were always a big part of ipSpace.net online courses, and our new Networking in Public Cloud Deployments course is no different.

You’ll start with a simple scenario: deploy a virtual machine running a web server. Don’t worry about your Linux skills, you’ll get the necessary (CCIE-level) instructions and the source code for the web server. Building on that, you’ll create another subnet and deploy another virtual machine acting as a back-end application server.

And then we’ll get to the fun part:

EVPN Route Targets, Route Distinguishers, and VXLAN Network IDs

Got this interesting question from one of my readers:

BGP EVPN message carries both VNI and RT. In importing the route, is it enough either to have VNI ID or RT to import to the respective VRF?. When importing routes in a VRF, which is considered first, RT or the VNI ID?

A bit of terminology first (which you’d be very familiar with if you ever had to study how MPLS/VPN works):

BGP- and Car Safety

The Facts and Fiction: BGP Is a Hot Mess blog post generated tons of responses, including a thoughtful tweet from Laura Alonso:

Is your argument that the technology works as designed and any issues with it are a people problem?

A polite question like that deserves more than 280-character reply, but I tried to do my best:

BGP definitely works even better than designed. Is that good enough? Probably, and we could politely argue about that… but the root cause of most of the problems we see today (and people love to yammer about) is not the protocol or how it was designed but how sloppily it’s used.

Laura somewhat disagreed with my way of handling the issue:

Update: Using pyATS in Network Automation

A month ago I described how Paddy Kelly used pyATS to get VRF data from a Cisco router to create per-VRF connectivity graphs.

Recently he also wrote a short article describing how to get started with pyATS and Ansible. Thank you, Paddy!

Video: Cloud Models, Layers and Responsibilities

In late spring 2019, Matthias Luft and Florian Barth presented a short webinar on cloud concepts, starting with the obvious topic: cloud models, layers, and responsibilities.

You need Free ipSpace.net Subscription to watch the video, and the Standard ipSpace.net Subscription to register for a deeper dive into cloud security with Matthias Luft (next live session on December 10th: Identity and Access Management).