Orange Matter: Automation Paralysis

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. This post examines how it’s easy to get so focused on automating the small stuff we have difficulty turning that into the more cohesive automation solution that we’d like to have.

This post appeared on Orange Matter as “Automation Paralysis: Why We Get Stuck Automating The Small Stuff“, but I’m also linking to the version posted on Thwack, mainly because that format allowed me to use more images and be slightly more irreverent. Irreverent? Moi? Of course.

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Automation Paralysis and give me a share/like. Thank you!

Orange Matter: Where is Your Configuration Source of Truth?

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. The post linked here looks at where we define our source of truth for device configurations; is it the device itself? Should it be? This is a key question when looking at automation, and one we should all be asking ourselves.

This post appeared on Orange Matter as “Where Is Your Config Source of Truth?“, but I’m also linking to the version posted on Thwack, mainly because that format allowed me to use more images and be slightly more irreverent, which is perhaps a bit more in character.

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Where is Your Configuration Source of Truth? and give me a share/like. Thank you!

New Year, New Post, NFDx!

You may be thinking “Wait, he hasn’t posted in ages.. how lazy is he?” but thankfully I haven’t been entirely slothful for the last seven months. Most recently I authored a series of six posts related to SDN and automation on the Solarwinds Orange Matter blog. I can’t republish that content here, but I will be sharing links to the posts in the coming days and I hope you’ll find them interesting and thought-provoking.

Cisco SP – Networking Field Day Exclusive!

More immediately, I’m preparing to start the new year with a quick trip to see Cisco’s Service Provider group at a Networking Field Day Exclusive event. I’ve seen the proposed agenda, and it looks like it’s going to be an intense day filled with the kind of topics that I know my readers will appreciate. As always, I’ll be posting about some of the topics covered (maybe even all of them…who knows?), but it’s even better if you can take part too.

The event takes place on Tuesday, January 15th, 2019. If you can, I recommend hopping on the live stream on Tech Field Day and then using the #TFDx hashtag on Twitter to join in the Continue reading

Mellanox, Ixia and Cumulus: Part 3

Last–but not least–in the technology triumvirate presenting a joint session at Networking Field Day 17 was Cumulus Networks. This post looks at the benefits of Cumulus Linux as a NOS on the Mellanox Spectrum Ethernet switch platform.

Cumulus Networks

I’ve not yet managed to deploy Cumulus Linux in anger, but it’s on a fairly short list of Network Operating Systems (NOS) which I would like to evaluate in earnest, because every time I hear about it, I conclude that it’s a great solution. In fact, I’m having difficulty typing this post because I have to stop frequently to wipe the drool from my face.

Cumulus Linux supports around 70 switches from 8 manufacturers at this time, and perhaps obviously, that includes the Mellanox Spectrum switches that were presented during this session. This is the beauty of disaggregation of course; it’s possible to make a hardware selection, then select the software to run on it. Mellanox made a fairly strong case for why the Spectrum-based hardware is better than others, so now Cumulus has to argue for why they would be the best NOS to run on the Mellanox hardware.

Cumulus Linux, as the name suggests, is based on Debian linux. Continue reading

Mellanox, Ixia and Cumulus: Part 2

This post is part two of three in a series looking at the joint presentations made by Mellanox, Ixia and Cumulus at Networking Field Day 17, in February 2018. More specifically, this post looks at what part Ixia has to play in the deployment of an Ethernet switch fabric built using Mellanox switches and running Cumulus Linux as the Network Operating System (NOS).

Ixia

What confused me most about a presentation from Mellanox, Ixia and Cumulus about Ethernet fabrics was to figure out what role Ixia would be playing in the disaggregated model. Mellanox makes the switch hardware and Cumulus makes the switch software, so Ixia fits, well, where exactly?

IxNetwork

IxNetwork is billed as an end-to-end validation solution which in many ways undersells what it’s all about. Rather than being just more traffic-generating test equipment, IxNetwork can emulate multiple switch and server devices so that a single piece of test hardware can be connected to what it believes is a large existing infrastructure, and that hardware’s behavior and resiliency can be validated. In the demo topology, IxNetwork connects to a physical Mellanox Spectrum switch running Cumulus Linux, emulating connected servers as well as an entire leaf/switch EVPN/VXLAN fabric, attached Continue reading

Mellanox, Ixia and Cumulus: Part 1

When I saw that Mellanox was presenting at Networking Field Day 17, I was definitely curious. When I found out that I would in fact be watching a joint presentation by Mellanox, Cumulus Networks and Ixia, it is fair to consider my interest piqued. Why would these three companies present together?

It turns out that these three companies present quite a compelling story, both individually–as you would probably expect–but also when used in combination. This post looks at the role of Mellanox Ethernet switches in an Ethernet fabric.

Mellanox

To me, Mellanox has been one of those ‘behind the scenes’ companies whose hardware is all over the place but whose name, in Ethernet circles at least, is less well known. Storage and compute engineers on the other hand are likely more familiar with the Mellanox name, especially in the context of Infiniband switches and network interface cards (NICs). In 2016 Mellanox acquired EZchip, allowing the development of some very capable Ethernet switches and an expansion of the company’s portfolio; to paraphrase Amit Katz (VP, WW Ethernet Switch), Mellanox connects PCI-Express interfaces together by building NICs, cables and switches.

At the Networking Field Day event in February 2018, Continue reading

Security: Mitigating Spectre on Older Intel CPUs

I suspect all of my readers are well aware of the Spectre exploit affecting, among others, Intel CPUs going back many years. Intel for their part, after a few missteps, have issued microcode updates for more recent CPUs. But for those of us with computers running older CPUs, the solutions are less likely to be forthcoming. Thankfully there is a solution.

Branch Prediction and Speculative Execution

The Spectre exploit affects processors which perform branch prediction, a kind of optimistic lookahead where the processor prepares and executes a potential instruction before it is actually requested. For example, if the processor encounters conditional code (like and if..then..else construct), based on previous behavior it predicts what the most likely outcome is and thus which branch of code would be executed as a result, then loads and executes that code in advance (hence “speculative execution”). If the branch prediction is correct, then since the code was already executed the code will benefit from improved performance. Spectre abuses some predictable timing behavior of the speculative execution to be able to extract other processes’ data from the CPU caches. In other words, it’s bad news for security.

The only way to restore security Continue reading

Response: The Need For Stretched VLANs (@ioshints)

A recent post from Ivan Pepelnjak entitled Revisited: The Need For Stretched VLANs made me smile rather bitterly as Ivan dug into the apparent continued desire for stretched layer 2 networks and the reasons people give for the solution’s requirement and validity. I love a good bit of snark as much as the next nerd, so as you can imagine, I’m all over that post.

However, I confess I did wince slightly – in the way one might do when an old wound is poked with a sharp stick – as Ivan made a passing sarcastic reference to Microsoft’s amazing Network Load Balancing technology:

My mind was thrown back to the heady days of 2009 when I stumbled across another post from Mr Pepelnjak, this time entitled Turn a switch into a hub … the Microsoft Way which bemoaned the unadulterated stupidity of Microsoft’s attempt to use layer 2 network flooding to accomplish clustering. I had discovered the nature of this behavior at a previous client and had my mind blown by the very stupid and non-standards-compliant way in which this had been implemented.

The reason my mind went to that post, however, is because if I recall correctly it’s Continue reading

This History Of Networking, from The Network Collective

I mentioned The Network Collective previously when I responded to the very first episode of the videocast/podcast (what TWiT would call a netcast). Since then the three founders and co-hosts (Jordan, Eyvonne and Phil) have published an impressive 20 community roundtable episodes and have somehow also found time to launch a History of Networking series co-hosted by Russ White (yes, that Russ White).

History Of Networking

I’m a bit of a nerd when it comes to computer history, and I love reading books that give the inside story about the birth of the personal computer, the story of Silicon Valley, the rise and fall of technology companies and so on. However, the history of networking is nowhere near as well covered, which is a real shame. Thankfully, The Network Collective are filling that gap handsomely with a list of guests so far that blows my mind. For example:

Paul Vixie

If you’ve ever heard of Vixie cron, BIND DNS, DNSSEC, the Internet Software Consortium (ISC), you’ve found things Mr Vixie has had his hands all over. It’s fascinating to hear him talking about the history of DNS adoption, and his role in maintaining BIND in a nascent Continue reading

Networking Field Day 17 (NFD17) Redux

As I write this it’s Friday afternoon in Silicon Valley, Networking Field Day 17 (NFD17) is over, and I am truly exhausted after a week of presentations covering topics like Ethernet switch hardware, testing/simulation and network operating system (Mellanox, Ixia, Cumulus), Field Area Networks (Cisco), visibility and automation (Juniper AppFormix), distributed network monitoring (Thousand Eyes) and much more.

I will be posting in the coming weeks about some of the individual presentations, but while I dry off from the information firehose I thought I’d briefly summarize some of my personal highlights from the week.

Themes

One recurring theme throughout the week was automation. Cisco for example, spoke about the capabilities for Zero Touch Provisioning (ZTP) in IOS XR. Juniper presented the OpenContrail platform which, impressively, appears to have a new and very positive sense of direction after (in my opinion) floating around somewhat aimlessly for a while. Extreme Networks spoke about Workflow Composer, and its ability to integrate with tools like Splunk to auto-remediate detected issues. VMware demonstrated the impressive capabilities of NSX to configure distributed firewalling and microsegmentation down to the container level, while VeloCloud from VMware demonstrated another facet of automation, that of the software defined WAN.

Another Continue reading

Cisco, Mellanox, Ixia and Cumulus: Last Day of NFD17!

In case you have missed the noise on my Twitter feed (@mrtugs) in the last couple of days, I’m currently at Networking Field Day 17 in Silicon Valley, and today (Friday, January 26) is the last day of presentations. So far this week, along with eleven other lucky delegates, we’ve been treated to presentations from Juniper, Thousand Eyes, Extreme Networks and VMware, including Velocloud from VMware. As usual, it has been a firehose of information and thankfully all the videos will be posted soon so I can go back and figure out what I might have missed.

The last two days of presentations have seen a very strong focus on automation, network fabric (including cloud connectivity) and hybrid cloud services. It’s uncanny how everything aligns, sometimes!

Today is the last day of NFD17 and we’re going to be starting at Cisco at 8AM PST, then after lunch we’re hearing from Mellanox, Ixia and Cumulus, all beginning at 1:30PM PST. We live stream all the presentations, so if you want to tune in and join us, pop over to the NFD17 site and the stream will be live on that page. If you are watching in real time and have a Continue reading

Automatic Product Pitch Generator

Because we all like a little bit of fun, I created an automatic Product Pitch Generator for network vendors. More accurately, a conversation about buzzwords occurred in the context of Networking Field Day 17 where it was jokingly suggested that we needed to take some of the amazing words we were hearing and make a generator. Here’s the best part (if you look at it this way); we ended up with two generators!

Big props to Jordan Martin who hacked together some Python, and created his Network Product Buzzword Generator which I think is hilarious (go try it out!). Meanwhile, I was hacking together some Go and came up with this mess of a Product Pitch Generator:

Hit Refresh to get a new Pitch!

And finally, if you’re watching the NFD17 livestream, why not ML-wash yourself and play NFD17 Buzzword Bingo?

Have fun!

If you liked this post, please do click through to the source at Automatic Product Pitch Generator and give me a share/like. Thank you!

MP on Vertitech IT’s “Best IT Blogs 2018”

This is a quick post to say thanks to the folks at Vertitech IT for listing movingpackets.net among their Best IT Blogs for 2018 (“Must-Read Resources for CIOs, IT & Security Pros”). MP was on the Best IP Blogs of 2017 as well, and it’s an honor to be on the list for a second year.

Vertitech explain the creation of this list thus:

Information Technology.  Sometimes we get so focused on the bits and bytes side of the equation we forget about the information part.  When it comes right down to it, IT is all about using technology to inform, to communicate, to make the business of doing business easier and more understandable.

That’s why we compiled this list. Originally created last year with 50 top IT blogs, we’ve expanded this year’s update to include 70 leading resources for IT professionals, including blogs, discussion forums, niche industry publications, and the best resources for CIOs and CTOs.  VertitechIT’s top 70 IT blogs, forums, and resources were selected because they are among the most current, frequently updated, credible, and informative sources of information related to IT on the web today. From musings of industry leaders, to the Continue reading

Twinax – Cheap, Cheerful and Annoyingly Chubby

What’s not to love about twinax? Formerly the exclusive domain of IBM systems, twinax has seen itself reborn in the last few years in the form of the Direct Attach Cable (DAC) used to connect systems at speeds of 10Gbps and 40Gbps (by way of bundling four twinax pairs in a single cable).

Direct Attach Cables

Before diving into the pros and cons of DAC, it’s important to understand the different varieties that are available. A DAC is a cable which has SFP+ format connectors hard-wired on each end; plug each end into an SFP+ socket and, vendor support notwithstanding, the link should come up. A direct attach cable is frequently and erroneously referred to as a “DAC cable”, so if the words “PIN number” give you the jitters, working anywhere with DACs is likely to drive you to drink.

Passive Copper DAC (Twinax)

The most common kind of DAC is the passive DAC. The SFP+ connector on a passive DAC, give or take some electrical protection circuitry, is pretty much a direct connection from the copper in the twinax to the copper contacts which connect to the host device:

Sending a 10G signal over a single copper pair requires Continue reading

Microburst: PSIRT Notifications – Are They Good Or Bad?

If your hardware or software vendor issues a lot of PSIRT (Product Security Incident Response Team) notifications, is that a good thing or a bad thing? After all, a PSIRT bulletin means that there’s a security issue with the product, so lots of PSIRTs means that the product is insecure, right?

What about the alternative, then? If a vendor issues very few PSIRT notifications does it mean that their product is somehow more secure? This is an issue I’ve been thinking about a lot over the last year, and the conclusion I came to is that if a vendor is not issuing regular bulletins, it’s a bad thing. Either the vendor doesn’t think its customers should be aware of vulnerabilities in the product, or perhaps the bugs aren’t being fixed. A PSIRT bulletin involves the vendor admitting that it got something wrong and potentially exposed its customers to a security vulnerability, and I’m ok with that. Sure, I don’t like sloppy coding, but I do appreciate the transparency.

I believe that when a vendor is shy about publishing security notifications it’s probably a decision made by management based on the naive belief that limiting the number of times they admit Continue reading

One Little Thing Can Break All Your Automation

I’ve been doing some work automating A10 Networks load balancers recently, and while testing I discovered a bug which broke my scripts. As hard as I try to code my scripts to cope with both success and failure, I realized that the one thing I can’t do much with is a failure of a key protocol.

Breaking Badly

So what matters when I’m automating device configurations? Three things come to mind immediately:

The Network / Transport

I need reliable network connectivity, or my automation requests will constantly be timing out or failing. While a script should be able to cope with an occasional network failure, unreliable networks are not fun to work with.

Data Transfer Format

Pick a format (XML, JSON) and use it consistently. I’m happy so long as I can send a request and read a response in a given format. If I send a request using JSON, send a response using JSON. Funnily enough I was troubleshooting WordPress xmlrpc recently and noticed that when there was an error, the XML command being issued received a 404 error followed by, well, you’d hope an XML error response, right? No, because it was an HTTP 404 error, the site Continue reading

Unifi US-8 PoE Passthrough With The Cisco 3560CX

As part of my “everything should be on a UPS” strategy, I recently replaced a regular 8-port gigabit switch with a Ubiquiti Unifi US-8 Ethernet switch because the US-8 can be powered using POE (Power Over Ethernet) provided by a UPS-protected switch in my basement, so it should stay up in the event of a power outage. This also allowed me to indirectly provide UPS protection for the Ubiquiti wireless AP in that location because the US-8 has a PoE passthrough port with which I could power the AP. Clever, right?

POE Passthrough

To clarify (because a picture is worth many thousands of my words), here’s how things were:

And here’s how things are after installing the Ubiquiti Unifi US-8:

The new setup worked well, but I noticed after a few days that the uptime for the Unifi US-8 kept on resetting; that is, it appeared to be rebooting. The Cisco 3560CX switch which is providing the POE can supply 30W per port, which is plenty enough for the US-8 and the wireless AP to be daisy-chained like this, yet when I looked at the logs on the 3560CX, I found an error:

Oct 23 18:23:12.124 UTC: %ILPOWER-3-CONTROLLER_PORT_ERR:  Continue reading

How To Access Devices with Unsupported SSL Ciphers

With the HeartBleed bug effectively killing off SSLv3 and vulnerabilities in cipher block chaining ruling out another whole swathe of SSL ciphers, network engineers may have found themselves trying to connect to a device and either getting no response (Safari), or getting a response like this (Chrome):

Or this (Firefox):

Once upon a time, it was possible to go into settings and enable the old, insecure ciphers again, but in more recent updates, those ciphers no longer exist within the code and are thus inaccessible. So what to do? My answer was to try a proxy.

Charles Proxy

The first proxy I looked at seemed promising. Although not free, Charles Proxy offers a 30 day free trial, and that seemed like a good thing to try. It’s limited additionally by only running for 30 minutes at a time before it has to be reloaded, but for my testing purposes that was not a problem.

During installation I declined to give Charles Proxy permission to configure the system proxy settings. Instead, I manually updated just my Firefox browser to use the proxy which was now listening on port 127.0.0.1:8888. Since I was making an SSL connection, I also Continue reading

Hive Mind, Help Me Out with A10 AXAPI?

Dear Internet,

I am writing some automation code in Go to create client-ssl templates on an A10 load balancer running AXAPI version 2. It’s going as swimmingly as it can with the v2 API, but one area of non-complete API coverage has led to another issue and I’m wondering if anybody has seen the same thing.

Background – Disabling SSLv3

SSL access to VIPs on the A10 load balancer is controlled by means of client-ssl templates which define which certificates should be presented and the ciphers and protocols supported for the incoming connection. In this case therefore, disabling SSLv3 is accomplished in the client-ssl template (unfortunately there is no global switch to turn SSLv3 off by default). A typical template might look like this in the configuration:

slb template client-ssl mytemplate
   cert my_certificate
   key my_private_key
   chain-cert some_chain_cert
   disable-sslv3
!

As it turns out, all aspects of the client-ssl template are exposed via the API except for “disable-sslv3” which shows neither as a returned property of the template (highly annoying), nor as a property which can be set when creating a template (also annoying). Thus to replicate a template like the one above, I choose to set everything I can using the Continue reading

What Next Now the KRACK Smoke is Clearing?

It’s only four days since we were blessed with news of the KRACK vulnerability in WPA2, so what have we learned now that we’ve had some time to dig into the problem?

Patching Infrastructure (Access Points)

In terms of patching wireless access points the good news is that most of the enterprise vendors at least are on the ball and have either released patches, have them in testing, or have at least promised them in the near future. While one of the primary victims of KRACK in these devices is 802.11r (Fast Roaming) which is not likely to be used in most home environments, it’s more common to see repeater or mesh functionality in the home, and because the AP acts as a wireless client in these cases, it is susceptible to the vulnerability. So if you just have a single AP in the home, chances are that updating the firmware because of KRACK is not that urgent. That’s probably a good thing given the number of wireless access points embedded in routers managed by internet providers, running on old and unsupported hardware, or created by vendors who are no longer in business.

Patching Clients

The clients are where Continue reading