Category Archives for "Herding Packets"

Coming to SD-WAN: The Build vs. Buy Decision

Earlier this month, I attended Networking Field Day 13, where we heard from VeloCloud on their SD-WAN solution. Their presentation and case study got me thinking about how most businesses will consume SD-WAN and where business customers may fall on the “Buy” vs. “Build” spectrum.

At the outset of the NFD13 presentation, VeloCloud CEO Sanjay Uppal recapped some stats: VeloCloud has been around for just about 4 years, and at this point has around 600 enterprise customers and is deployed to about 50,000 sites. If VeloCloud was a product line from an encumbent networking vendor with stats like that, they would be declaring it as a very successful mainstream product. I point this out as I think it demonstrates that SD-WAN solutions and vendors are moving out of “startup” mode and into a mainstream solution.

Screen Shot 2016-11-26 at 4.46.30 PM.png

One of the things that has set VeloCloud apart from many of their competitors since their inception has been their focus on building a true multi-tenant solution from the beginning, as well as their choice to partner with service providers to provide a solution for managed SD-WAN. Strong API capabilties and flexible zero-touch provisioning features support this as well. This is what really caught my attention Continue reading

NFD13 Forecast – Cloudy With a Chance of Software

Well, this week is a week of firsts — or at least, firsts in a while. It’s my first time actually posting something new on the blog in a while, and it will also be my first Tech Field Day event in a while.


I’m honored to have been invited back to NFD13, happening on Thursday, 11/17 and Friday, 11/18. I’m very excited to be joining the TFD team for another event! The delegate panel includes some of the most experienced and well respected names in the networking industry (myself excluded, obviously), and the sponsor list is really quite exciting. I’m looking forward to seeing some old friends, and making some new ones along the way.

I really like when NFD events have a mix of focuses (foci?). It keeps the conversation varied and interesting for both the delegates and the audience. For NFD 13, I see several big themes for the sponsor list:

Data Center Fabric/Automation/Orchestration

apstralogo-100x22             400px-nec_logo-svg_

Network Testing/Modeling/Audit/Management

forward-networks_above_logo-100x57                 ixia_logo-100x47           400px-solarwinds_logo

Software Defined WAN

velocloud_logo_m             viptela

Note: I could be mis-classifying a couple of Continue reading

Performing Ping Sweeps with IOS TclSh

It’s been a while since I’ve gotten a blog post up, but with my CCIE recertification out of the way I’m hoping to ramp some volume back up. We’re talking about some sexy stuff today… Ping sweeps! First off, let’s cover why you’d need to sweep up your pings. Some people use the ping sweep as a means to “find” hosts on the network. The problem with this is, devices with host-based firewalls active may not respond to an ICMP ping. If you’re pinging from off the local subnet, there are other reasons you might not get a response back as well, like a host having a mis-programmed default gateway or subnet mask, or an interface ACL on the routing device. That said, ping sweeps are still incredibly useful for helping to find vacant IP addresses on a LAN. Or, at least, IP addresses that are not currently active. Always consult your properly maintained IP documentation to find IPs you can safely use for new deployments (yes, I’m laughing at that one too…).

Anyway, how do ping sweeps help identify active IPs if we can’t trust the ping responses? Well, just because a device may not respond to the ICMP Continue reading

Creating Uplink Port-Channels in UCS Manager

Recently, a customer asked me for a quick how-to for plumbing and configuring northbound port-channels on their UCS B-series setup. The basic install including management access had been completed some time ago, but as projects sometimes go, this one had been back-burnered for some time so we were just getting around to making it work.

I spun up my copy of the UCSPE (obtainable here) and grabbed a few screenshots to provide a quick walkthrough. The customer was able to follow my instructions quickly and with no further follow-up questions, so I figured I’d toss this into a quick blog post for anyone else looking to do the same.


Each 6200-series UCS Fabric Interconnect will have a single 4-member port-channel that goes up to the pair of Nexus using a vPC.
UCS Design


1. Preconfigure the ports on the Nexus switches. Since this is a vPC arrangement, each Nexus will require identical configuration:
interface port-channel5
description To ucs6248-a
switchport mode trunk
switchport trunk allowed vlan 1-50
spanning-tree port type edge trunk
vpc 5
interface Ethernet1/5
description To ucs6248-a
switchport mode trunk
switchport trunk allowed vlan 1-50
channel-group 5 mode active
no shutdown
interface Ethernet1/6
description To ucs6248-a
switchport mode trunk
 Continue reading

ASA File Operation Tips

I’ve been working on Cisco’s ASA firewall platform for years, and I continue to work on a variety of environments with multiple generations of the ASA for clients at H.A. Storage. One of my favorite features of the ASA platform has been the quality of the high-availability failover mechanism, which is generally very reliable, fast, and seamless.
The ASA operates in an Active/Standby high-availability model (don’t believe that the ASA is *truly* Active/Active — that’s a marketing feature). However, one sore spot that has frustrated me as long as I’ve been working on the fact that the filesystem has no synchronization between failover mates and requires manual efforts to keep files in sync. Other configuration aspects of the ASAs including some XML customization files that are not stored in the running config all get automatically sync’d to the standby unit, but for actual files that show up on the flash filesystem, this does not happen.
This has certainly caused me some frustration and occasional embarrassment over the years, but one thing I’ve learned along the way is that when doing file operations either from the CLI or the ASDM, it’s important to follow one simple rule:
Delete from the active, upload to the Continue reading

Thoughts on Building Tools versus “Programming”

A couple weeks ago at Networking Field Day 9, Brocade presented with their usual A-list of networking gurus. One of the presenters was Jon Hudson, a very engaging, visionary speaker. His talk, shown below, was about the state of network programmability.

During the conversation (which is well worth watching), discussion turned to the question of “will network engineers become programmers?” posed by John Herbert of Jon Hudson’s response elicited applause from the room. He said:

“The trouble I have with that statement is, most network engineers I know, like myself, we know how how to code. We went to school for it, and we chose not to.” – Jon Hudson

The conversation went on to discuss the value of programmability for the sake of consistency in the management and configuration of large-scale network fabrics (which I don’t think anyone would really debate as a “Good Thing”), but Jon’s quote about being a programmer and some of the sidebar that flowed from it created a fair bit of activity in the Twitter stream. Following the presentation, my attention was called to a mailing list on which a question was asked about networking engineerings being “given a Continue reading

The Buzz About NetBeez

One of the great benefits of attending Tech Field Day events is the opportunity to learn about new startups that I might otherwise not have heard about. And one of the great things about startups is their ability to apply a fresh set of eyes to long-standing problems without being bogged down by existing products or past decisions.

The problem I’m thinking of in this case is monitoring the network right out to the user edge in a cost-effective manner. Network performance monitoring platforms tend to be network-centric, looking at port statistics and reachability from the monitoring platform to the access switch. Application performance monitoring tools often monitor from a single, central point of the network either generating transactions or trying to intercept user transactions and interpolate performance characteristics while deriving performance stats. Each of these solutions can have a place, but neither one really gives you a view from the ground. Existing options to get a user-like view that really comes from the edges of your network (where the users are, after all) are very limited. Enter NetBeez, with a very simple premise: Monitor from the edge. Every edge. Wired edges. Wireless edges. From the user’s perspective. And use a central, or cloud-based, dashboard Continue reading

NFD8 Recap: Nuage Networks – One to Watch

Last fall, I attended the Tech Field Day NFD8 event, and one of the presenting companies was Nuage Networks. This was actually the second time I’d seen Nuage present at an NFD event, the first one being NFD6 a year earlier. Upon my return from NFD8, I did a short write-up of each presenting sponsor for my coworkers at H.A. Storage Systems to keep them informed. The following is my recap of Nuage Networks after their presentation in which I explain why I think Nuage is really on-target with their SDN solution and is definitely a solution to keep an eye on.

Nuage Networks is definitely an SDN company to watch. They are a subsidiary of Alcatel-Lucent (sort of like Cisco’s Insieme but apparently there are no current plans to spin them back in), so they have good financial backing — better than many startups. They have a very mature vision of complete end-to-end SDN with automated deployment tools and fabric-wide management, but they’ve gone beyond what several of the competitors have to look at massive, massive scaling as a core requirement.
Rather than using VXLAN or some other new protocol for things like federation between fabrics, they simply use Continue reading

In the Data Center, No One Can Hear You Lose Your Hearing

Working for a data center-focused reseller/integrator like H.A. Storage Systems, I spend my fair share of time in various data center environments. I have, for years, elected to use some sort of hearing protection when in these facilities. I have constantly been amazed, though, at how few other workers in data centers do the same.

Honestly, I’m not sure why most people I see in data center facilities don’t use hearing protection. Perhaps they think they’ll only be on the data center floor for a few minutes. We all know it never goes like that! Perhaps they think it makes it too hard to hear co-workers. Personally, I find it easier to communicate with someone when I have ear protection on. Maybe workers think it’s not loud enough for ear protection to be necessary. Unfortunately, this isn’t true in most cases.

Can You Hear Me Now?

As an example, I was recently down at the Sungard Availability Services colo facility in Philadelphia. It’s one of the largest colos in the immediate area with 230,000 square feet of raised floor. I was in there all day long with one of my customers, for two days while we traced out Continue reading

Normalizing ACLs to Support Automated Changes

Although I look forward to network fabric management seeing broad deployment, the fact is that many networks (and especially enterprise LAN/WAN) will be managed with traditional methods for some time yet. Inconsistencies in device configurations can present a barrier to some types of automation. In this article, we’ll explore that very challenge and a resolution I came up with to handle it.

Not long ago, I was trying to automate an ACL line insertion task with a popular network configuration push tool that basically does CLI interaction with something like Expect. I needed to push a similar change to about 20 devices with minimal effort. Unfortunately, when looking at the ACL on several sample devices out of the target device pool, I saw things like this:
R1(config)#do sh access-list NAT
Extended IP access list NAT
14 deny ip
20 deny ip
25 permit ip
30 permit ip 0. Continue reading

Managing the Network as a Fabric — About Time!

Earlier this September, I attended the Tech Field Day Networking Field Day 8 event. Over the course of three days, we saw presentations from many very interesting vendors including a mix of startups and established market leaders. One trend that really stuck out to me more this time around than at any previous NFD event was a nearly ubiquitous emphasis on data center network fabric management. In other words, truly managing an entire data center network (or at least a sub-block of it) as a single unit.

Just of the NFD8 presenters who were providing this option, we had Cisco with their ACI model (but it stands to reason that even the now-well-established FEX model has very similar capabilities), Big Switch Networks with their Big Cloud Fabric, Pluribus Networks’ Netvisor Software Defined Fabric, and Nuage Networks Virtual Services Platform. Each of these products has unique value propositions, so I’m not suggesting they’re all the same but rather pointing out that this concept of fabric-level management is clearly at the forefront of most, if not all, leading-edge data center solutions at this point. The concept has been building for a couple years, and other vendors are also pursuing this model Continue reading

Cisco Live – The Minimalist Packing List

Cisco Live 2014 is right around the corner! It’s almost time to start packing. The other day, Keith Miller (@packetologist), a first-time Cisco Live attendee, asked me on Twitter:

I have a bit of a reputation among some of my consulting clients as being ready for just about anything. Normally, that means my laptop bag weighs about 50 lbs. But for Cisco Live, I choose to travel light. I’ve seen people in the airport on the way to, and from, the event with a LOT of stuff. Sure, some folks are presenters or carrying company stuff but for the rest, you probably just have too much stuff.

Why choose to travel as light as possible? Here are just a few reasons:

  1. Airline bag check fees
  2. Airlines are great at losing/abusing your stuff once its out of your hands
  3. Due to #1, everyone is always fighting for room in the overhead bins and you end up checking your “carry on” anyway
  4. If you land early, you have to check your 3 tons of bags, or else Continue reading

Using EEM to Remotely Change a WAN IP – Part 2

In my last EEM post I provided a simple means to change an IP address and default route of a Cisco router using a script that makes the change without requiring interactive user input. This is helpful if you are remotely changing a device’s WAN/Internet IP and waiting for some on-site hands to move a cable over to a new ISP or WAN SP connection. That first script, however, would make the change and then exit. What would happen if the new Internet connection had a problem, or the on-site help couldn’t move the cable for some reason? Proper testing and preparation should help you avoid most of those issues but you just never know.

One way to deal with this possibility is to issue a “reload in 10” before kicking off the EEM change script. If the change can’t be completed, the router will reboot back to its previous configuration. That’s fine, but I like to avoid a full reboot whenever possible, and “reload in” has always been a rather clunky rollback mechanism.

Another idea mentioned by Jody Lemoine in the comments of the last post, is taking advantage of the newer configuration archive and rollback features. While my Continue reading

Using EEM to Remotely Change a WAN IP – Part 1

I often work remotely on customers’ infrastructures with their remote hands on-site. When a small office or branch changes ISPs or IP blocks, I occasionally find myself in a position where I have to change the only public IP address of a device like a branch office router or firewall, with no out-of-band management. The trouble with this is fairly obvious (on a Cisco device): by changing the IP address via which I am accessing the device over SSH, I will lose my own management session to it. Once the management session is lost, I can’t update the default route, and now the device is broken and I get to walk the on-site hands (who are often not very Cisco-literate) through changing a default route.

There are, of course, several ways to avoid this situation all together:
  • Have out-of-band access using a 3G/4G/LTE-connected terminal server (I wrote about one of these before)
  • Use a remote app like GetConsole so the remote hands can get me console access out of band using their smart phone
  • Use something with a proper commit/rollback mechanism like a Juniper device
  • Dial-up modem to the AUX port!
Clearly, from the list above, there are means to Continue reading

Dell Aims for the Clouds with Z9500 Spine

While at Networking Field Day 7, we got a small preview of a new switch Dell Networking has just announced, the Z9500. At some point I’ll have another post coming discussing more of Dell’s presentation at NFD7, but I wanted to briefly talk about this new product and what it brings to the table for Dell.

To be frank, Dell’s acquisition of Force10 Networks originally felt to me like a “me too” play so that Dell could compete with Cisco UCS and HP in the “full data center stack” play combining compute, storage, and networking in a single SKU or a playbook of blessed configurations. I wasn’t really expecting Dell to innovate all that much in this space. But based on the information I have at this point, I think that position is unfounded.
Here’s a picture of the new beast from a demo rack Dell showed us at NFD:
Dell Z9500

Dell Z9500

To summarize the hardware: it’s a hell of a lot of density. The Z9500 platform presents 132 line-rate 40G ports in a 3RU chassis. So, if one were inclined, one could potentially cram 14 such chassis into a standard 42RU cabinet to concentrate 1,848 40G ports into Continue reading

Taking a New Approach to Cisco Live 2014

It’s getting to be that time of year again when geeks across North America (and beyond) start getting excited for Cisco Live! The buzz is starting a bit early this year, but that’s because Cisco Live is about a month earlier this year (May 18-22) than it has been in recent years.


Many of the exciting details of the event have surfaced at this point, including the announcement of Salman Khan, founder of Khan Academy as the closing guest keynote speaker, and that the Customer Appreciation Event will feature Lenny Kravitz and the Imagine Dragons as musical acts. We’ve even seen this year’s backpack design, courtesy of a couple Tweets from Cisco’s Kathleeen Mudge:

Last year, I put together a post of Cisco Live tips, most of which are just as applicable this year as last. My personal approach, however, will be a bit different this year.

Plans Continue reading

Weighing AWS VPN Options

Earlier this week, a client asked for some assistance in building a VPN from their corporate office to Amazon Web Services for a project they were doing. I’ve done this a few times before, a few different ways, so I proceeded to give my client some pros and cons of the two most common methods I’ve used. After putting that analysis together, I realized it could be helpful for others so here it is (with the addition of a few snazzy diagrams!).
This post is not meant to be a treatise on AWS connectivity, just a quick analysis with some (maybe) little-considered effects of a given design choice. Amazon documents several other recipes which are, of course, valid in various circumstances. Note that I don’t have any examples of configuration. The AWS documentation pages have very thorough configuration examples for each design.

 Option 1

Build the VPNs off the Internet routers themselves. Route AWS traffic in to the corporate network through the firewall. In an ideal world, you’d probably dedicate some routers for this purpose, but I’ve never had anyone do that. We’re talking about a LAN-to-LAN VPN, here; one doesn’t commonly deploy totally dedicated infrastructure for each new Continue reading

Server Brawn + Switch Brains = Infrastructure Fabric

Last week I attended Networking Field Day 7, and was introduced to Pluribus Networks. Pluribus is taking an interesting approach to building the data center fabric, by combining high-performance data center top-of-rack (ToR) switching with powerful server internals in a platform they’ve dubbed the Freedom Server-Switch.



The Freedom platform can be loaded to bare with RAM and storage along with some pretty powerful CPUs (this data sheet provides all the details), which enables embedding various network (and not-so-network) services right in the network at every edge. The platform runs the NetVisor operating system, based on BSD. This software can be had in various feature levels:


Various services that can be enabled beyond typical L2/L3 network services include DHCP, DNS, PXE, load balancing, CDN functions, NAT, NAS (yes, really), and traffic analytics. Since these switches are designed for deployment as leaf nodes in leaf-spine architecture datacenters, this embeds these services right at the network ingress point for each connected device.
You may be thinking about the potential administrative overhead included with performing advanced network services on each ToR switch, but that burden is eased with fabric-wide management features that allow an administrator to Continue reading

Faking an ASA as a DNS Forwarder

I came across a good tip the other day that was very helpful during a small site firewall migration. Here’s the back story:

I was migrating a small single-site customer that had, up to this point, been using a FIOS-provided consumer-type router/firewall/access point to some Cisco gear including an ASA firewall for better firewall/VPN capabilities. This is fairly common with small businesses that start out with essentially consumer-style connectivity and finally begin to grow to a point of needing business-grade capabilities. My preparation went fine, and when the time came I swapped the ASA firewall in place of the FIOS-provided one. Then everything broke.

I had meticulously prepared the ASA to take over immediately from the old FIOS router, even going so far as to spoof the FIOS router’s MAC address on the ASA’s inside interface for now so as not to disrupt the 60-or-so clients that were all on the single attached internal subnet while their ARP caches timed out since we were doing the install and cut-over during working hours. I had set up a DHCP scope on the ASA as well, which instructed clients to use some public DNS resolvers as this small business has, so far, Continue reading

Goodbye Snowpocalypse, Hello Networking Field Day 7!

Snowpoc Resized

It’s been a long winter here in Pennsylvania. Near record-breaking for snowfall. But yesterday I traveled to beautiful and temperate San Jose to attend Networking Field Day 7!
I’m honored to have been selected as a delegate for another Tech Field Day event, as these events are a fantastic opportunity to engage with vendors and industry peers. I use the term “peers” only because we work in the same industry. Everyone else is smarter than me.

I’m excited to rub elbows and network with the exceptional delegate list. I have met nearly all of this event’s delegates before and I respect the expertise and experience of every single participant. I feel I have learned so much and made so many valuable connections through TFD events and I’m grateful to Gestalt IT and the TFD community for another opportunity to participate.

Most of all, I’m excited for the opportunity to represent you, the networking/IT community at large. Asking the questions you would ask. I will be live Tweeting during the presentations, so direct your questions my way and I’ll do my best to ask your questions if I miss something you want to know about.


I was going to Continue reading