Why Network Engineers Should Learn Programming

Because Microsoft Excel is not a text editor. Seriously.

This is a followup to the previous post, inspired by Ethan Banks of Packet Pushers. If you do operational networking at all, you deal with text files all the time: logs, debug output, configuration files, command line diagnostics, and more. I'm constantly amazed when I see people open Word or Excel to do their text editing, often one keystroke at a time.

The number one reason to learn basic programming is to automate that stuff. Personally, I use a combination of traditional Unix shell tools and Python to get the job done, but you could probably do it all with one or the other.

There are lots of other reasons to learn programming too, many of which will be discussed on an upcoming Packet Pushers episode. But if you don't believe any of those, this one alone makes it worth the effort.

Step away from the spreadsheet. Do it now.

“VIF down” Issues with UCSM 2.2(1b)

Sadly, this will be another post regarding issues I’ve had with UCSM firmware release 2.2(1b). During the upgrade process, I experienced a lot of issues with data plane connectivity - after I activated (and subsequently rebooted) a Fabric Interconnect, and it came up with the new NXOS version, a slew of blades would have persistent errors regarding virtual interfaces (VIFs) that wouldn’t come back online. Here is the error report for a single blade where I was seeing these errors:

“VIF down” Issues with UCSM 2.2(1b)

Sadly, this will be another post regarding issues I’ve had with UCSM firmware release 2.2(1b). During the upgrade process, I experienced a lot of issues with data plane connectivity - after I activated (and subsequently rebooted) a Fabric Interconnect, and it came up with the new NXOS version, a slew of blades would have persistent errors regarding virtual interfaces (VIFs) that wouldn’t come back online. Here is the error report for a single blade where I was seeing these errors:

Super Easy Twitter Bots

Super Easy Twitter Bots

I often get really quite mad ideas on writing twitter bots, But I often get pretty bored of doing all of the boiler plate that is required when wanting to achieve these things.

The typical process to making a twitter bot

Cisco UCS – “Unable to Communicate With UCSM Controller”

When upgrading UCS firmware, it’s important to periodically check the state of the HA clustering service running between the two Fabric Interconnects. The infrastructure portions of UCS are generally redundant due to these two FIs but only if the clustering service has converged - so it’s important to use the “show cluster state” command to verify this is the case. During a firmware upgrade to 2.2(1b), I checked this: 6296FAB-A# connect local-mgmt 6296FAB-A(local-mgmt)# show cluster state Cluster Id: 8048cd6e-5d54-11e3-b36c-002a6a499d04 Unable to communicate with UCSM controller The error message - “unable to communicate with UCSM controller” worried me, and it was given when I ran the “show cluster state” command as well as the “cluster lead” command - the latter of which is necessary to switch an FI’s role in the cluster from subordinate to primary.

Cisco UCS – “Unable to Communicate With UCSM Controller”

When upgrading UCS firmware, it’s important to periodically check the state of the HA clustering service running between the two Fabric Interconnects. The infrastructure portions of UCS are generally redundant due to these two FIs but only if the clustering service has converged - so it’s important to use the “show cluster state” command to verify this is the case. During a firmware upgrade to 2.2(1b), I checked this: 6296FAB-A# connect local-mgmt 6296FAB-A(local-mgmt)# show cluster state Cluster Id: 8048cd6e-5d54-11e3-b36c-002a6a499d04 Unable to communicate with UCSM controller The error message - “unable to communicate with UCSM controller” worried me, and it was given when I ran the “show cluster state” command as well as the “cluster lead” command - the latter of which is necessary to switch an FI’s role in the cluster from subordinate to primary.

Cisco UCS – “Unable to Communicate With UCSM Controller”

When upgrading UCS firmware, it’s important to periodically check the state of the HA clustering service running between the two Fabric Interconnects. The infrastructure portions of UCS are generally redundant due to these two FIs but only if the clustering service has converged - so it’s important to use the “show cluster state” command to verify this is the case. During a firmware upgrade to 2.2(1b), I checked this: 6296FAB-A# connect local-mgmt 6296FAB-A(local-mgmt)# show cluster state Cluster Id: 8048cd6e-5d54-11e3-b36c-002a6a499d04 Unable to communicate with UCSM controller The error message - “unable to communicate with UCSM controller” worried me, and it was given when I ran the “show cluster state” command as well as the “cluster lead” command - the latter of which is necessary to switch an FI’s role in the cluster from subordinate to primary.

Farewell to Networking

Almost twenty years ago, I began my career in networking.  HP hubs and routers, no VLANs, one router PHY port per subnet.  From there I installed an ATM backbone using LANE in the venerable Catalyst 5500 platform, then moved on to GigE in 3750 stacks and finally to 10G Nexuses (Nexa, Nexi?).  I’ve seen WiFi […]

Author information

Matthew Mengel

Matthew was a Senior Network Engineer for a regional educational institution in Australia for over 15 years, working with Cisco equipment across many different product areas. However, in April 2011 he resigned, took seven months of long service leave to de-stress and re-boot before becoming a network engineer for a medium sized non-profit organisation. At the end of 2013, he left full-time networking behind after winning a scholarship to study for a PhD in astrophysics. He is on twitter infrequently as @mengelm.

The post Farewell to Networking appeared first on Packet Pushers Podcast and was written by Matthew Mengel.

Cisco UCS Error – “Process Failed”

One of the (sadly numerous) issues I’ve run into while upgrading to Cisco UCSM version 2.2(1b) is this little error message indicating that a service failed to start: This gives us an error code of F0867 and it’s letting us know that the UCSM process httpd_cimc.sh failed on one of our Fabric Interconnects. For those that don’t know, you can get a list of processes within UCSM by connecting to local management and running “show pmon state”.

Cisco UCS Error – “Process Failed”

One of the (sadly numerous) issues I’ve run into while upgrading to Cisco UCSM version 2.2(1b) is this little error message indicating that a service failed to start: This gives us an error code of F0867 and it’s letting us know that the UCSM process httpd_cimc.sh failed on one of our Fabric Interconnects. For those that don’t know, you can get a list of processes within UCSM by connecting to local management and running “show pmon state”.

Cisco UCS Error – “Process Failed”

One of the (sadly numerous) issues I’ve run into while upgrading to Cisco UCSM version 2.2(1b) is this little error message indicating that a service failed to start: This gives us an error code of F0867 and it’s letting us know that the UCSM process httpd_cimc.sh failed on one of our Fabric Interconnects. For those that don’t know, you can get a list of processes within UCSM by connecting to local management and running “show pmon state”.

Quick Thoughts on Learning Python

I was scheduled to be a guest on an upcoming episode of the Packet Pushers podcast, on the topic of Python for network engineers. Unfortunately due to bad luck I'm not going to be able to make the recording. Here are some quick thoughts on learning Python. If you're already an expert programmer you already know how to learn languages, so this post isn't for you.

Scenario 1: You've coded in another language, but you're not an expert.
I would start with the basic Python class at Google Code. It's targeted specifically at people who know basic programming skills in some other language. It was perfect for me; I went through the exercises and was able to quickly start writing simple, useful Python scripts.

Scenario 2: You don't know how to write code at all.
Start with the Udacity CS101 class if you like guided learning, or Learn Python the Hard Way if you prefer books. Be prepared to spend a lot of time on either. It's not easy the first time around.

After you've gotten through one of those two scenarios, do the following:


  1. Spend time browsing the documentation for the Python Standard Library. Python is a large language, Continue reading

Quick Thoughts on Learning Python

I was scheduled to be a guest on an upcoming episode of the Packet Pushers podcast, on the topic of Python for network engineers. Unfortunately due to bad luck I'm not going to be able to make the recording. Here are some quick thoughts on learning Python. If you're already an expert programmer you already know how to learn languages, so this post isn't for you.

Scenario 1: You've coded in another language, but you're not an expert.
I would start with the basic Python class at Google Code. It's targeted specifically at people who know basic programming skills in some other language. It was perfect for me; I went through the exercises and was able to quickly start writing simple, useful Python scripts.

Scenario 2: You don't know how to write code at all.
Start with the Udacity CS101 class if you like guided learning, or Learn Python the Hard Way if you prefer books. Be prepared to spend a lot of time on either. It's not easy the first time around.

After you've gotten through one of those two scenarios, do the following:


  1. Spend time browsing the documentation for the Python Standard Library. Python is a large language, Continue reading

Healthy Paranoia Show 21: Windows Forensics with Andrew Case

That’s right, it’s time for another surveillance-free, EFF-approved episode of Healthy Paranoia! Where the passwords are salted and the packets are always encrypted. This episode is hosted by the infamous Mrs. Y, queen of metadata and official privacy advocate for Healthy Paranoia, and recorded in the NSA-proofed SCIF with Grecs, of Novainfosec.com and Shmoocon Firetalks. […]

Author information

Mrs. Y

Snarkitecht at Island of Misfit Toys

Mrs. Y is a recovering Unix engineer working in network security. Also the host of Healthy Paranoia and official nerd hunter. She likes long walks in hubsites, traveling to security conferences and spending time in the Bat Cave. Sincerely believes that every problem can be solved with a "for" loop. When not blogging or podcasting, can be found using up her 15 minutes in the Twittersphere or Google+ as @MrsYisWhy.

The post Healthy Paranoia Show 21: Windows Forensics with Andrew Case appeared first on Packet Pushers Podcast and was written by Mrs. Y.

[Storage Flow Control] Part 1- Introduction

When making the leap to adopting FCoE as a storage medium, there are a few things to consider in order to be successful. Many of these concepts are foreign to the storage administrator who has been operating a native Fibre Channel SAN for the better part of the last decade or more - this is because while Fibre Channel networks are costly, they are purpose-built. There is no concept of a loop in Fibre Channel - with Ethernet we deal with these all the time.

[Storage Flow Control] Part 1- Introduction

When making the leap to adopting FCoE as a storage medium, there are a few things to consider in order to be successful. Many of these concepts are foreign to the storage administrator who has been operating a native Fibre Channel SAN for the better part of the last decade or more - this is because while Fibre Channel networks are costly, they are purpose-built. There is no concept of a loop in Fibre Channel - with Ethernet we deal with these all the time.

[Storage Flow Control] Part 1- Introduction

When making the leap to adopting FCoE as a storage medium, there are a few things to consider in order to be successful. Many of these concepts are foreign to the storage administrator who has been operating a native Fibre Channel SAN for the better part of the last decade or more - this is because while Fibre Channel networks are costly, they are purpose-built. There is no concept of a loop in Fibre Channel - with Ethernet we deal with these all the time.

Using the latest Open vSwitch with Devstack and OpenDaylight

While setting up my OpenDaylight OVSDB and Devstack following the awesome instructions from Kyle Mestery, I thought it would be fun to run the latest OVS from source on my compute nodes...

To do this, execute the following commands on one of your compute nodes before running stack.sh

sudo apt-get -y --force-yes install build-essential devscripts

gcc dkms make automake autoconf debhelper libssl-dev
pkg-config python-all python-qt4 python-zopeinterface
python-twisted-conch gdebi-core dh-autoreconf hardening-wrapper
libtool graphviz ipsec-tools module-assistant python-twisted-web
racoon git

git clone git://git.openvswitch.org/openvswitch
cd openvswitch
./boot.sh
dpkg-buildpackage -b -us -uc -nc
cd ..

Once you’ve built the .deb’s you can copy these to your other compute nodes using scp:

scp *.deb ubuntu@devstack-compute2:

Replace ubuntu with your username and devstack-compute2 with the name or IP address of your other compute nodes.

Finally we can install the packages as follows:

sudo dpkg -i *.deb

At the time of writing this will build Open vSwitch 2.1.90. You can check the version as follows:

sudo ovs-vsctl --version

Which will give the following output

ovs-vsctl (Open vSwitch) 2.1.90
Compiled Jan 16 2014 15:18:45

Huge thanks to @FlorianOtel for his help with Devstack!

@dave_tucker

A brief departure from talking IPv6

There is a lot of news surrounding Net Neutrality, and potential repercussions of decisions made by courts, and some players out there that want to grab as much cash as they can, and claim it is in the best interest of their customers.

Netflix is just an example people love citing because it is bandwidth intensive, yet is not the entire story itself. Take a moment and understand how the Internet is pieced together. The Internet is a mass of interconnections between networks. These interconnections happen basically 1 of 3 ways:

transit: network A pays network B to reach every other network that isn’t A or B. Good networks usually get multiple transits for failover, and/or alternate paths to those other networks. You can buy multiple ports for bonding to increase capacity, etc. Average transit price without a Service Level Agreement (SLA, guaranteed connectivity or you can yell at us a lot and we credit you) is around $1-2/mbit, and with a SLA can hit upwards of $10/mbit. These are current avg. prices when buying 10G at a time of connectivity/capacity right now.

peering (settlement free, or “free”): Network A spends a bunch of money to get into popular Continue reading