Hard Truths Not Taught in Schools

J Metz published a great article describing six hard truths not taught in school. As all good things should come in 7-tuples, here’s another one I was told ages ago when I was a young hotshot full of myself:

Professions were created for a reason – they enable people to do the work they’re qualified to do.

Needless to say, it took me decades to fully understand its implications.

Webinar Plans for 2018

Last week I described what we did in 2017. Now let’s see what the ipSpace.net subscribers will get in 2018.

New webinars

I thought that 2017 would be a year of the cloud, but that was not to be – I was too busy creating network automation and data center content.

Synchronize Network Management Parameters across Network Devices

While I have stock homework assignments prepared for every module of the Building Network Automation Solutions online course I always encourage the students to pick a challenge from their production network and solve it during the course.

Pavel Rovnov decided to focus on consistency of network management parameters (NTP, SNMP, SSH and syslog configuration) across Extreme and Cumulus switches, Fortinet firewalls and several distributions of Linux.

Packet Forwarding on Linux on Software Gone Wild

Linux operating system is used as the foundation for numerous network operating systems including Arista EOS and Cumulus Linux. It provides most networking constructs we grew familiar with including interfaces, VLANs, routing tables, VRFs and contexts, but they behave slightly differently from what we’re used to.

In Software Gone Wild Episode 86 Roopa Prabhu and David Ahern explained the fundamentals of packet forwarding on Linux, and the differences between Linux and more traditional network operating systems.

Webinars in 2017

2017 was one of the busiest years since I started the ipSpace.net project.

It started with an Ansible for Networking Engineers session covering advanced Ansible topics and network device configurations. Further sessions of that same webinar throughout 2017 added roles, includes, extending Ansible with dynamic inventory, custom modules and filters, and using NAPALM with Ansible.

Ansible, Chef, Puppet or Salt? Which One Should I Use?

One of the first things I did when I started my deep-dive into network automation topics was to figure what tools people use to automate stuff and (on a pretty high level) what each one of these tools do.

You often hear about Ansible, Chef and Puppet when talking about network automation tools, with Salt becoming more popular, and CFEngine being occasionally mentioned. However, most network automation engineers prefer Ansible. Here are a few reasons.

Event-Driven Automation on Building Network Automation Solutions Online Course

Most engineers talking about network automation focus on configuration management: keeping track of configuration changes, generating device configurations from data models and templates, and deploying configuration changes.

There’s another extremely important aspect of network automation that’s oft forgotten: automatic response to internal or external events. You could wait for self-driving networks to see it implemented, or learn how to do it yourself.

On March 20th live session of Building Network Automation Solutions online course David Gee will dive deeper into event-driven network automation. As he explains the challenge:

When it comes to running infrastructure and infrastructure services, a lot of the decision making is human based. Someone reads a ticket, someone decides what to do. Someone gets alerted to an event and that someone does something about it. This involvement causes friction in the smooth-running nature of automated processes. Fear not! Something can be done about it.

We all know the stories of ITIL and rigid process management and David will show you how event-driven automation could be made reality even with strict and rigid controls, resulting in an environment that reacts automatically to stimuli from your services and infrastructure. We will discuss what events are, when they're important, how Continue reading

Meltdown and Its Networking Equivalents

One of my readers sent me this question:

Do you have any thoughts on this meltdown HPTI thing? How does a hardware issue/feature become a software vulnerability? Hasn't there always been an appropriate level of separation between kernel and user space?

There’s always been privilege-level separation between kernel and user space, but not the address space separation - kernel has been permanently mapped into the high-end addresses of user space (but not visible from the user-space code on systems that had decent virtual memory management hardware) since the days of OS/360, CP/M and VAX/VMS (RSX-11M was an exception since it ran on 16-bit CPU architecture and its designers wanted to support programs up to 64K byte in size).

Worth Reading: Robust IPAM

Elisa Jasinska covered several IPAMs in her overview of open-source network automation tools, and we had Jeremy Stretch talking about NetBox in the Building Network Automation Solutions online course, but if you’re looking for a really robust easy-to-implement solution, check out this document from 1998 (deployment experience, including a large-scale one).

When Did IT Practitioners Lose Their Curiosity?

One of my readers sent me an interestingly sad story as a response to my importance of fundamentals rant. Here it is… enjoy ;)

2017-01-14: Updated with a different viewpoint

Upcoming ipSpace.net Events

2018 has barely started and we’re already crazily busy:

• The update session of Network Automation Use Cases webinar on January 16^th will talk about intent-based networking and data models;
• A week later (January 23^rd) it’s back to basics: how do you select an optimal VPN service.

The last week of January is Cisco Live Europe week. I’ll be there as part of the Tech Field Day Extra event – drop by or send me an email if you’ll be in Barcelona during that week.

Fat Fingers Strike Again…

Level3 had a pretty bad bad-hair-day just a day before Pete Lumbis talked about Continuous Integration on the Building Network Automation Solutions online course (yes, it was a great lead-in for Pete).

According to messages circulating on mailing lists it was all caused by a fumbled configuration attempt. My wild guess: someone deleting the wrong route map, causing routes that should have been tagged with no-export escape into the wider Internet.

BGP Route Selection: a Failure of Intent-Based Networking

It’s interesting how the same pundits who loudly complain about the complexities of BGP (and how it will be dead any time soon and replaced by an SDN miracle) also praise the beauties of intent-based networking… without realizing that the hated BGP route selection process represents one of the first failures of intent-based approach to networking.

Let’s start with some definitions. There are two ways to get a job done by someone else:

New Design on www.ipSpace.net

One of my readers sent me a polite email a while ago saying “your site is becoming like $majorVendor’s web site – every corner looks completely different based on when you made it”

Source: Wikimedia Commons

The worst part is that he was right, so I spent the last two weeks as a website janitor, mopping up broken markup, fixing CSS cracks, polishing old texts…

Wow, another Year Swooshed By…

Could you believe it? Another year swooshed by… and it’s high time to stop being snarky and cynical, disconnect from the Internet, and spend a few days with people who really matter – our families.

For me, there’s another large group of people that matter: my users.

Unique IPv6 Prefix Per Host – How Complex Do You Want IPv6 to Be?

In December 2017 IETF published RFC 8273 created by the v6ops working group (which means there must have been significant consensus within the working group that we need the solution and that it makes at least marginal sense).

The RFC specifies a mechanism by which the first-hop router allocates a unique /64 IPv6 prefix for every host attached to a subnet and uses unicast and multicast RA responses sent to unicast MAC addresses to give every host the impression that it’s the sole host on its own subnet.

The first thought of anyone even vaguely familiar with how complex IPv6 already is should be “WTF???” Unfortunately, there are good reasons we need this monstrosity.

Salt Deep Dive on Building Network Automation Solutions Online Course

In the first few sessions of the Building Network Automation Solutions online course we used Ansible as the tool-of-choice because it’s the easiest automation tool to get started with. Now that we’ve established the baseline, it’s time to explore the alternatives.

In a live session on February 27th 2018, Mircea Ulinic will describe Salt, an open source, general-purpose event-driven automation framework that we briefly discussed in Episode 77 of Software Gone Wild podcast.

BGP: the Tragedy of the Commons

Every now and then someone looks at a few recent BGP incidents (from fat fingers to more dubious ones) and says “we need a better BGP”.

It’s like being unable to cope with your kids or your team members because you don’t have the guts to tell them NO and trying to solve the problem by implementing new procedures and rules.

Like anything designed on a few napkins BGP has its limit. They’re well known, and most of them have to do with trusting your neighbors instead of checking what they tell you.

The solutions to the problem are pretty simple and have been known for decades (BCP38 was published in May 2000). In a nutshell you have to:

• Build a global repository of who owns what address space;
• Document who connects to whom and what their peering policies are;
• Filter the updates received from your customers and peers based on the information from those repositories;
• Filter the traffic from addresses that are obviously spoofed.

We have most of the tools we need to get the job done; you’ll find them described in Best Current Practice (BCP) 194. It’s also not impossible to get the job done Continue reading

Please Respond: Survey on Interconnection Agreements

Marco Canini is working on another IXP-related research project and would like to get your feedback on inter-AS interconnection agreements, or as he said in an email he sent me:

As academics, it would be extremely valuable for us to receive feedback from network operators in the industry.

It’s fantastic to see researchers who want to base their work on real-life experience (as opposed to ideas that result in great-looking YouTube videos but fail miserably when faced with reality), so if you’re working for an ISP please take a few minutes and fill out this survey.

What Exactly Should My MAC Address Be?

Looks like I’m becoming the gateway-of-last-resort for people encountering totally weird Nexus OS bugs. Here’s another beauty…

I'm involved in a Nexus 9500 (NX-OS) migration project, and one bug recently caused vPC-connected Catalyst switches to err-disable (STP channel-misconfig) their port-channel members (CSCvg05807), effectively shutting down the network for our campus during what was supposed to be a "non-disruptive" ISSU upgrade.

Weird, right? Wait, there’s more…