Automating network troubleshooting with NetQ + Ansible

Network Automation is so hot right now! Joking aside, DevOps tools like Ansible, Puppet, Chef and Salt as well as proprietary tools like Apstra are becoming all the rage in computer networks everywhere. There are python courses, network automation classes and even automation focused events for the first time in the history of computer networks (or at least it feels like it).

For this blog post I want to focus on automating network troubleshooting, the forgotten stepchild of network automation tasks. I think most automation tools focus on provisioning (or first time configuring) because so many network engineers are new to network automation in general. While I think that is great (and I want to encourage everyone to automate!) I think there is so much more potential for network automation. I am introducing Sean’s third category of automation use-cases — OPS!

I want to combine Cumulus NetQ, a fabric validation system, with Ansible to:

• Figure out IF there is a problem (solved by NetQ)
• Figure out WHAT the problem is (solved by NetQ)
• FIX the problem (solved by Ansible)
• AUTOMATE the above 3 tasks (solved by Ansible)

Because I think looking at terminal windows is super boring (no Continue reading

Backing up configs with the Ansible NCLU module

With the release of Ansible 2.3 the Cumulus Linux NCLU module is now part of Ansible core. This means when you `apt-get install ansible`, you get the NCLU module pre-installed! This blog post will focus on using the NCLU module to backup and restore configs on Cumulus Linux. To read more about the NCLU module from its creator, Barry Peddycord, click here.

The consulting team uses Ansible very frequently when helping customers fully automate their data centers. A lot of our playbooks use the Ansible template module because it is very efficient and idempotent, and Cumulus Linux has been built with very robust reload capabilities for both networking and Quagga/FRR. This reload capability allows the box to perform a diff on either `etc/network/interfaces` or `etc/quagga/Quagga.conf` so when a flat-file is overridden with the template module, only the “diff” (or difference) is applied. This means if swp1-10 were already working and we added configuration for swp11-20, an ifreload will only add the additional config and be non-disruptive for swp1-10. This reload capability is imperative to data centers and our customers couldn’t live without it.

However, many customers also want to build configs with NCLU (or the net commands) when Continue reading

NetDevOps: important idempotence

As more and more network engineers dive into network automation, the word idempotence keeps coming up. What is it? Why is it important? Why should we care? Idempotence is often described as the ability to perform the same task repeatedly and produce the same result. I want to demonstrate a super simple example of what this means.

If I am logged into a Linux box and want to add an IP address to the loopback address, I could use something simple like a sed command.

root@leaf01:mgmt-vrf:~# sed -i '/loopback/ a address 1.1.1.1/32' /etc/network/interfaces

This produces exactly what I want!
auto lo
iface lo inet loopback
        address 1.1.1.1/32
        address 10.0.0.11/32

I have appended the address 1.1.1.1/32 to the loopback interface stanza of the /etc/network/interfaces file. Now what happens if I run that same exact command again?

Running the command again produces the following output:
auto lo
iface lo inet loopback
        address 1.1.1.1/32
        address 1.1.1.1/32
        address 10.0.0.11/32

That is not what I wanted. I performed the same task but instead of just leaving the file alone, since the 1.1.1. Continue reading

5 host network configurations for MLAG

Host network configurations for MultiChassis Link Aggregation (MLAG, also referred to as dual-attach or ‘high availability’) can vary from host OS to host OS, even amongst Linux distributions. The most recommended and robust method is to use Link Aggregation Control Protocol (LACP), which is supported on most host operating systems natively. Host bonds or bonding refers to a variety of bonding methods, but for the purpose of this article it will refer to LACP bonds. The terms etherchannel, link aggregation group (LAG), NIC teaming, port-channel and bond can be used interchangeably to refer to LACP depending on the vendor’s nomenclature. For the sake of simplicity, we will just call it bonds or bonding. This post will take your through the steps for host network configurations for MLAG across five different operating systems.

Why LACP? LACP is a IEEE standard that has been available since 2000 known as 802.3ad. This makes a highly interoperable standards approach to bonding that can work across many network vendors and host operating systems. LACP is superior to static configuration (also referred to bond-mode ON) because there is a control protocol keeping the bond active. This means failover is predictable and automatic. This is also Continue reading

Linux networking: It’s not just SDN

Oftentimes, Cumulus Linux gets confused for an SDN (software-defined networking) solution. In conversations with potential customers, I’ve noticed that some of them find it difficult to distinguish between SDN, open networking and Cumulus Linux. When I talk to network engineers, I start by clarifying the SDN buzzword head on. The term gets overused, and is often defined by other confusing acronyms or marketing jargon. To complicate things further, SDN is often thought of as equivalent to OpenFlow, which is flawed in my opinion.

What is SDN?

If I were to more accurately describe SDN based on my experiences in the networking industry, I would define it more broadly. Instead of defining SDN as a specific solution (such as OpenFlow), I define SDN as a highly automatable and programmable network infrastructure.

What SDN providers exist today?

• OpenFlow: Many companies and communities drive OpenFlow solutions, but today there is no guarantee any one solution can interoperate with any other.
• Proprietary or vendor-specific: Solutions such as Cisco’s ACI and Juniper Contrails are closed solutions that are positioned as SDN. Arguably, certain OpenFlow solutions can fall under here as well since they don’t all adhere to an OpenFlow standard.
• Network virtualization with technologies Continue reading

DevOps Tools for Modern Data Centers

Back in October, 2015, I spoke at All Things Open in Raleigh, North Carolina, an event focused on open technology and open source software. I was very excited by this event because many attendees work in or manage data centers, which means they are very familiar with Linux but have little experience with the networking stack. Cumulus Networks is the first major networking company to contribute a true Linux networking operating system for data center switches, which is highly disruptive to the industry and drives a lot of fun conversations with open-minded individuals.

The talk I did for All Things Open last October titled “Using DevOps Tools for Modern Data Centers” focuses on the new concept of NetDevOps or DevOps for Network devices. Since the network operating system is Cumulus Linux, why not use open source off-the-shelf automation tools that are already being leveraged in the data center to act as a controller.  These tools have an extremely large user base, are vendor neutral — that is, not proprietary — and can scale easily.

So what are the benefits of using open source tools? One of the most important benefits from a networking point of view is provisioning. Imagine you have 1000 Continue reading