Archive

Category Archives for "Networking"

“VIF down” Issues with UCSM 2.2(1b)

Sadly, this will be another post regarding issues I’ve had with UCSM firmware release 2.2(1b). During the upgrade process, I experienced a lot of issues with data plane connectivity - after I activated (and subsequently rebooted) a Fabric Interconnect, and it came up with the new NXOS version, a slew of blades would have persistent errors regarding virtual interfaces (VIFs) that wouldn’t come back online. Here is the error report for a single blade where I was seeing these errors:

“VIF down” Issues with UCSM 2.2(1b)

Sadly, this will be another post regarding issues I’ve had with UCSM firmware release 2.2(1b). During the upgrade process, I experienced a lot of issues with data plane connectivity - after I activated (and subsequently rebooted) a Fabric Interconnect, and it came up with the new NXOS version, a slew of blades would have persistent errors regarding virtual interfaces (VIFs) that wouldn’t come back online. Here is the error report for a single blade where I was seeing these errors:

Super Easy Twitter Bots

Super Easy Twitter Bots

I often get really quite mad ideas on writing twitter bots, But I often get pretty bored of doing all of the boiler plate that is required when wanting to achieve these things.

The typical process to making a twitter bot

Cisco UCS – “Unable to Communicate With UCSM Controller”

When upgrading UCS firmware, it’s important to periodically check the state of the HA clustering service running between the two Fabric Interconnects. The infrastructure portions of UCS are generally redundant due to these two FIs but only if the clustering service has converged - so it’s important to use the “show cluster state” command to verify this is the case. During a firmware upgrade to 2.2(1b), I checked this: 6296FAB-A# connect local-mgmt 6296FAB-A(local-mgmt)# show cluster state Cluster Id: 8048cd6e-5d54-11e3-b36c-002a6a499d04 Unable to communicate with UCSM controller The error message - “unable to communicate with UCSM controller” worried me, and it was given when I ran the “show cluster state” command as well as the “cluster lead” command - the latter of which is necessary to switch an FI’s role in the cluster from subordinate to primary.

Cisco UCS – “Unable to Communicate With UCSM Controller”

When upgrading UCS firmware, it’s important to periodically check the state of the HA clustering service running between the two Fabric Interconnects. The infrastructure portions of UCS are generally redundant due to these two FIs but only if the clustering service has converged - so it’s important to use the “show cluster state” command to verify this is the case. During a firmware upgrade to 2.2(1b), I checked this: 6296FAB-A# connect local-mgmt 6296FAB-A(local-mgmt)# show cluster state Cluster Id: 8048cd6e-5d54-11e3-b36c-002a6a499d04 Unable to communicate with UCSM controller The error message - “unable to communicate with UCSM controller” worried me, and it was given when I ran the “show cluster state” command as well as the “cluster lead” command - the latter of which is necessary to switch an FI’s role in the cluster from subordinate to primary.

Cisco UCS – “Unable to Communicate With UCSM Controller”

When upgrading UCS firmware, it’s important to periodically check the state of the HA clustering service running between the two Fabric Interconnects. The infrastructure portions of UCS are generally redundant due to these two FIs but only if the clustering service has converged - so it’s important to use the “show cluster state” command to verify this is the case. During a firmware upgrade to 2.2(1b), I checked this: 6296FAB-A# connect local-mgmt 6296FAB-A(local-mgmt)# show cluster state Cluster Id: 8048cd6e-5d54-11e3-b36c-002a6a499d04 Unable to communicate with UCSM controller The error message - “unable to communicate with UCSM controller” worried me, and it was given when I ran the “show cluster state” command as well as the “cluster lead” command - the latter of which is necessary to switch an FI’s role in the cluster from subordinate to primary.

Farewell to Networking

Almost twenty years ago, I began my career in networking.  HP hubs and routers, no VLANs, one router PHY port per subnet.  From there I installed an ATM backbone using LANE in the venerable Catalyst 5500 platform, then moved on to GigE in 3750 stacks and finally to 10G Nexuses (Nexa, Nexi?).  I’ve seen WiFi […]

Author information

Matthew Mengel

Matthew was a Senior Network Engineer for a regional educational institution in Australia for over 15 years, working with Cisco equipment across many different product areas. However, in April 2011 he resigned, took seven months of long service leave to de-stress and re-boot before becoming a network engineer for a medium sized non-profit organisation. At the end of 2013, he left full-time networking behind after winning a scholarship to study for a PhD in astrophysics. He is on twitter infrequently as @mengelm.

The post Farewell to Networking appeared first on Packet Pushers Podcast and was written by Matthew Mengel.

Cisco UCS Error – “Process Failed”

One of the (sadly numerous) issues I’ve run into while upgrading to Cisco UCSM version 2.2(1b) is this little error message indicating that a service failed to start: This gives us an error code of F0867 and it’s letting us know that the UCSM process httpd_cimc.sh failed on one of our Fabric Interconnects. For those that don’t know, you can get a list of processes within UCSM by connecting to local management and running “show pmon state”.

Cisco UCS Error – “Process Failed”

One of the (sadly numerous) issues I’ve run into while upgrading to Cisco UCSM version 2.2(1b) is this little error message indicating that a service failed to start: This gives us an error code of F0867 and it’s letting us know that the UCSM process httpd_cimc.sh failed on one of our Fabric Interconnects. For those that don’t know, you can get a list of processes within UCSM by connecting to local management and running “show pmon state”.

Cisco UCS Error – “Process Failed”

One of the (sadly numerous) issues I’ve run into while upgrading to Cisco UCSM version 2.2(1b) is this little error message indicating that a service failed to start: This gives us an error code of F0867 and it’s letting us know that the UCSM process httpd_cimc.sh failed on one of our Fabric Interconnects. For those that don’t know, you can get a list of processes within UCSM by connecting to local management and running “show pmon state”.

Quick Thoughts on Learning Python

I was scheduled to be a guest on an upcoming episode of the Packet Pushers podcast, on the topic of Python for network engineers. Unfortunately due to bad luck I'm not going to be able to make the recording. Here are some quick thoughts on learning Python. If you're already an expert programmer you already know how to learn languages, so this post isn't for you.

Scenario 1: You've coded in another language, but you're not an expert.
I would start with the basic Python class at Google Code. It's targeted specifically at people who know basic programming skills in some other language. It was perfect for me; I went through the exercises and was able to quickly start writing simple, useful Python scripts.

Scenario 2: You don't know how to write code at all.
Start with the Udacity CS101 class if you like guided learning, or Learn Python the Hard Way if you prefer books. Be prepared to spend a lot of time on either. It's not easy the first time around.

After you've gotten through one of those two scenarios, do the following:


  1. Spend time browsing the documentation for the Python Standard Library. Python is a large language, Continue reading

Quick Thoughts on Learning Python

I was scheduled to be a guest on an upcoming episode of the Packet Pushers podcast, on the topic of Python for network engineers. Unfortunately due to bad luck I'm not going to be able to make the recording. Here are some quick thoughts on learning Python. If you're already an expert programmer you already know how to learn languages, so this post isn't for you.

Scenario 1: You've coded in another language, but you're not an expert.
I would start with the basic Python class at Google Code. It's targeted specifically at people who know basic programming skills in some other language. It was perfect for me; I went through the exercises and was able to quickly start writing simple, useful Python scripts.

Scenario 2: You don't know how to write code at all.
Start with the Udacity CS101 class if you like guided learning, or Learn Python the Hard Way if you prefer books. Be prepared to spend a lot of time on either. It's not easy the first time around.

After you've gotten through one of those two scenarios, do the following:


  1. Spend time browsing the documentation for the Python Standard Library. Python is a large language, Continue reading

Healthy Paranoia Show 21: Windows Forensics with Andrew Case

That’s right, it’s time for another surveillance-free, EFF-approved episode of Healthy Paranoia! Where the passwords are salted and the packets are always encrypted. This episode is hosted by the infamous Mrs. Y, queen of metadata and official privacy advocate for Healthy Paranoia, and recorded in the NSA-proofed SCIF with Grecs, of Novainfosec.com and Shmoocon Firetalks. […]

Author information

Mrs. Y

Snarkitecht at Island of Misfit Toys

Mrs. Y is a recovering Unix engineer working in network security. Also the host of Healthy Paranoia and official nerd hunter. She likes long walks in hubsites, traveling to security conferences and spending time in the Bat Cave. Sincerely believes that every problem can be solved with a "for" loop. When not blogging or podcasting, can be found using up her 15 minutes in the Twittersphere or Google+ as @MrsYisWhy.

The post Healthy Paranoia Show 21: Windows Forensics with Andrew Case appeared first on Packet Pushers Podcast and was written by Mrs. Y.

[Storage Flow Control] Part 1- Introduction

When making the leap to adopting FCoE as a storage medium, there are a few things to consider in order to be successful. Many of these concepts are foreign to the storage administrator who has been operating a native Fibre Channel SAN for the better part of the last decade or more - this is because while Fibre Channel networks are costly, they are purpose-built. There is no concept of a loop in Fibre Channel - with Ethernet we deal with these all the time.

[Storage Flow Control] Part 1- Introduction

When making the leap to adopting FCoE as a storage medium, there are a few things to consider in order to be successful. Many of these concepts are foreign to the storage administrator who has been operating a native Fibre Channel SAN for the better part of the last decade or more - this is because while Fibre Channel networks are costly, they are purpose-built. There is no concept of a loop in Fibre Channel - with Ethernet we deal with these all the time.

[Storage Flow Control] Part 1- Introduction

When making the leap to adopting FCoE as a storage medium, there are a few things to consider in order to be successful. Many of these concepts are foreign to the storage administrator who has been operating a native Fibre Channel SAN for the better part of the last decade or more - this is because while Fibre Channel networks are costly, they are purpose-built. There is no concept of a loop in Fibre Channel - with Ethernet we deal with these all the time.

Using the latest Open vSwitch with Devstack and OpenDaylight

While setting up my OpenDaylight OVSDB and Devstack following the awesome instructions from Kyle Mestery, I thought it would be fun to run the latest OVS from source on my compute nodes...

To do this, execute the following commands on one of your compute nodes before running stack.sh

sudo apt-get -y --force-yes install build-essential devscripts

gcc dkms make automake autoconf debhelper libssl-dev
pkg-config python-all python-qt4 python-zopeinterface
python-twisted-conch gdebi-core dh-autoreconf hardening-wrapper
libtool graphviz ipsec-tools module-assistant python-twisted-web
racoon git

git clone git://git.openvswitch.org/openvswitch
cd openvswitch
./boot.sh
dpkg-buildpackage -b -us -uc -nc
cd ..

Once you’ve built the .deb’s you can copy these to your other compute nodes using scp:

scp *.deb ubuntu@devstack-compute2:

Replace ubuntu with your username and devstack-compute2 with the name or IP address of your other compute nodes.

Finally we can install the packages as follows:

sudo dpkg -i *.deb

At the time of writing this will build Open vSwitch 2.1.90. You can check the version as follows:

sudo ovs-vsctl --version

Which will give the following output

ovs-vsctl (Open vSwitch) 2.1.90
Compiled Jan 16 2014 15:18:45

Huge thanks to @FlorianOtel for his help with Devstack!

@dave_tucker

A brief departure from talking IPv6

There is a lot of news surrounding Net Neutrality, and potential repercussions of decisions made by courts, and some players out there that want to grab as much cash as they can, and claim it is in the best interest of their customers.

Netflix is just an example people love citing because it is bandwidth intensive, yet is not the entire story itself. Take a moment and understand how the Internet is pieced together. The Internet is a mass of interconnections between networks. These interconnections happen basically 1 of 3 ways:

transit: network A pays network B to reach every other network that isn’t A or B. Good networks usually get multiple transits for failover, and/or alternate paths to those other networks. You can buy multiple ports for bonding to increase capacity, etc. Average transit price without a Service Level Agreement (SLA, guaranteed connectivity or you can yell at us a lot and we credit you) is around $1-2/mbit, and with a SLA can hit upwards of $10/mbit. These are current avg. prices when buying 10G at a time of connectivity/capacity right now.

peering (settlement free, or “free”): Network A spends a bunch of money to get into popular Continue reading

Large flow marking using hybrid OpenFlow

Top of rack switches are in a unique position at the edge of the network to implement traffic engineering controls. Marking large flows describes a use case for dynamically detecting and marking large flows as they enter the network:
Figure 1: Marking large flows
Physical switch hybrid OpenFlow example described how real-time sFlow analytics can be used to trigger OpenFlow controls to block denial of service attacks. This article will describe how the sFlow-RT, Floodlight OpenFlow controller, and Alcatel-Lucent OmniSwitch hybrid OpenFlow SDN controller setup can be programmed to dynamically detect and mark large (Elephant) flows as they enter the network.
Figure 2: Large flow marking controller results
In the experimental setup, a flood ping is used to generate a large flow:
ping -f 10.0.0.238 -s 1400
Figure 2 shows the results, the left half of the chart shows traffic when the controller is disabled and the right half shows traffic when the controller is enabled. The blue line trends the largest unmarked flow seen in the network and the gold line shows the largest marked flow. When controller is disabled, none of the traffic is marked. When the controller is enabled, sFlow-RT detects the large flow Continue reading