John Herbert

Author Archives: John Herbert

Response to “Certifications Are Not A Big Deal. Stop Being a Princess About It.”

In a post which now appears to have been deleted, Greg Ferro got right to the point in his article Response: Certifications Are Not A Big Deal. Stop Being a Princess About It.. The majority of this response was written while Greg’s post was still active, but I had to come back and inject more context after I spotted on June 30, 2019 that the post had become unavailable.

To save you digging in the WayBackMachine, the history to Greg’s post as I understand it is that Greg made some comments in Episode 238 of the Packet Pushers’ Network Break suggesting that vendor certifications were trivial. A listener evidently gave some strong feed back disagreeing with this, and so in Episode 239 of the Packet Pushers’ Network Break Greg responded to that feedback, and reiterated his position about certification study, specifically framed around Cisco’s CCNP. Greg made some reasonable points; that the certification programs from the vendors are not designed to teach fundamentals in the same way that, say, a computer science degree might do, and that the aim is really to make money for the vendor, and reduce their tech support costs, and as such the vendor certification education Continue reading

Cranky Old Network Engineer Complains About The Youth Of Today

If you’re very old (like me) you’ll likely remember the halcyon days when IP routing was not enabled by default on Cisco routers. Younger gamers may find this hard to believe, which makes it even stranger when I keep bumping into an apparently common misconception about how routers work. Let’s take a look at what I’m beefing about.

No IP Routing?

To put this in context for the younger gamers, it’s worth noting that at the time, a typical “enterprise” might be running IP, but was equally likely to run IPX, AppleTalk, DECnet or some other protocol which may – or may not – support routing. Yes, there was life before the Internet Protocol became ubiquitous. If you’re curious, the command to enable IP routing is, well:

ip routing

Guess how IPX routing was enabled:

ipx routing

Appletalk?

appletalk routing

DECnet Phase IV?

decnet [network-number] routing <decnet-address>

Ok, so the pattern isn’t entirely consistent, but it’s close enough. In one way things are much simpler now because routers tend to handle IP (and IPv6) and nothing else. On the other hand there are so many more IP-related features available, I think we should just be grateful that there’s only one Continue reading

The Achilles Heel of the API

I’ve been developing yet more automation recently, and I’ve been hitting two major stumbling blocks that have had a negative impact on my ability to complete the tooling.

API Documentation

When APIs were first made available, the documentation from many vendors was simply incomplete; it seemed that the documentation team was always a release or two behind the people implementing the API. To fix that, a number of vendors have moved to a self-documenting API system along the lines of Swagger. The theory is that if you build an API endpoint, you’re automatically building the documentation for it at the same time, which is a super idea. This has improved the API’s endpoint coverage but in some cases has resulted in thorough documentation explaining what the endpoints are, but little to no documentation explaining why one would choose to use a particular endpoint. 

As a result, with one API in particular I have been losing my mind trying to understand which endpoint I should use to accomplish a particular task, when no less than three of them appear to handle the same thing. I’m then left using trial and error to determine the correct path, and at the end Continue reading

A10 Networks ACOS Root Privilege Escalation

The following summarizes a root privilege escalation vulnerability that I identified in A10 ACOS ADC software. This was disclosed to A10 Networks in June 2016 and mitigations have been put in place to limit exposure to the vulnerability.

A10 Networks Cookie Vulnerability

SUMMARY OF VULNERABILITY

Any user assigned sufficient privilege to upload an external health monitor (i.e a script) and reference it from a health monitor can gain root shell access to ACOS.

At this point, I respectfully acknowledge Raymond Chen’s wise words about being on the other side of an airtight hatch; if the malicious user is already a system administrator or has broad permissions, then one could argue that they could already do huge damage to the ADC in other ways. However, root access could allow that user to install persistent backdoors or monitoring threats in the underlying OS where other users can neither see nor access them. It could also allow a partition-level administrator to escalate effectively to a global admin, by way of being able to see the files in every partition on the ADC.

SOFTWARE VERSIONS TESTED:

This vulnerability was originally discovered and validated initially in ACOS 2.7.2-P4-SP2 and is present in 4.x as Continue reading

Meraki In The Middle – Smart Security Cameras

I’ve been looking at security cameras recently, in part because my home owners association needs to upgrade the system which monitors some of the amenities. We want motion detection features and, obviously, remote access to view live cameras and recorded footage without having to go to the location. Unfortunately there’s a gap in the market which seems to be exactly where I’m looking. Cisco Meraki may have just stepped in and bridged that gap.

The Problem Space

Low-End Products

Over the last few years, a wide variety of small security cameras have become available, any of which which at first glance would appear suitable. These include products like Netgear’s Arlo, Amazon’s Blink, Google’s Nest Cam and more. After some brief testing, however, I’m a little less convinced that they are what we’re looking for. It sounds silly to say it, because it’s not like this is something they hide, but these products are all aimed at the home user market. Dashboard logins are single user, based on an email address and the web interfaces may not work well for much more than five or so cameras. The camera choices are fairly limited, and as they’ll be streaming their Continue reading

Orange Matter: Why Your Infrastructure Sucks For Automation

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. Let’s face it, unless we get to build an infrastructure from the ground up, our existing mass of one-off solutions and workarounds makes automating our infrastructure an absolute nightmare.

This post appeared on Orange Matter as “Why Your Infrastctructure Sucks For Automation“, but I’m also linking to the version posted on Thwack, because that version of the post includes pretty pictures. And who doesn’t like a pretty picture?

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Why Your Infrastructure Sucks For Automation and give me a share/like. Thank you!

Viavi Enterprise Provides Unexpected Network Insights

Many of us will have experienced the challenges of taking a performance alert (or user complaint) and drilling down to root cause. Performance issues can be intermittent, and it can be difficult to get visibility of what caused a problem at a particular time. Viavi Enterprise thinks it has the answer, combining analysis of packet feeds (e.g. from taps and mirror ports) and IPFix, xFlow and cloud service flow logs to monitor application performance as it would be experienced by a user. Sounds good? It looked pretty good, too.

Johnny Five Need Input!

Nothing can happen without data, and that comes from a number of potential sources.

Observer Gigastor

The Observer Gigastor product is available as a virtualized solution (to capture east-west traffic in virtualized environments), a portable appliance for tactical deployment, and two hardware appliance models (in a charming shade of purple) which can provide from 96TB to 1.2PB of storage. The idea of Gigastor is to capture packets at line rate and retain the raw packet data in case it’s needed later. The packets are analyzed, and that metadata is fed to the reporting and visualization system, Observer Apex.

Observer GigaFlow

It’s not always possible Continue reading

A10 Networks ACOS Critical Insecure Cookie Vulnerability 2 of 2

The following summarizes an HTTP persistence cookie vulnerability that I identified in A10 ACOS ADC software. This was disclosed to A10 Networks in June 2016 and has now been resolved.

A10 Networks Cookie Vulnerability

As noted in a previous post, ACOS uses insecure HTTP/HTTPS persistence cookies which can allow a malicious user to craft a cookie determining the server and port to which a persistent session should be sent. In addition, for vports using the default “port-based” HTTP cookie persistence, it was discovered that when using the default persistence cookie type, ACOS does not perform a check to ensure that the server/port defined in the cookie is within the configured service-group for that VIP.

The only sanity check appears to be to ensure that the server IP read from the cookie has been configured on the A10 within the same partition. If that constraint is met, packets will be forwarded by ACOS to the real server based solely on the value contained in the cookie. This is extremely serious as it allows a malicious user to connect, for example, through a public VIP and access back end servers used by other VIPs, including those only accessible via internal IPs.

SUMMARY OF VULNERABILITY

When using Continue reading

Orange Matter: All I Want For Christmas is RESTCONF

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. I need clear commands, structured response data and simple access to it all; so how about giving me REST-based APIs on all my infrastructure equipment?

This post appeared on Orange Matter as “All I Want For Christmas is RESTCONF“, but I’m also linking to the version posted on Thwack, in case you prefer to read and comment there.

I’d love it if you were to take a moment to visit and read, and maybe even comment!

(Featured image created by Kira auf der Heide on Unsplash)

If you liked this post, please do click through to the source at Orange Matter: All I Want For Christmas is RESTCONF and give me a share/like. Thank you!

A10 Networks ACOS Critical Insecure Cookie Vulnerability 1 of 2

The following summarizes an HTTP persistence cookie vulnerability that I identified in A10’s ACOS ADC software. This issue was disclosed to A10 Networks in June 2016 and has since been resolved.

A10 Networks Cookie Vulnerability

This vulnerability results in information disclosure about names of service-groups and IPs of real servers, as well as the ability to manipulate the content of the cookies.

SUMMARY OF VULNERABILITY

The ACOS documentation for HTTP persistence cookies notes that “For security, address information in the persistence cookies is encrypted.” However, the address information is not “encrypted”; rather, the real server IP and port information is weakly obfuscated and is easily decoded, exposing information about the internal network. The simplicity of the obfuscation also makes it trivial to manually create a cookie which ACOS would decode and honor.

Additionally, cookies configured using the service-group command option have the service-group’s full name included in the persistence cookie as plain text. This vulnerability applies to HTTP/HTTPS VIP types that have been configured to use a cookie-based persistence template.

SOFTWARE VERSIONS TESTED

This vulnerability was discovered and validated initially in ACOS 2.7.2-P4-SP2 and reconfirmed most recently in ACOS 4.1.1-P3.

VULNERABLE VERSIONS

This behavior has been core to Continue reading

Orange Matter: Automating the Automators

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. APIs are critical to operating infrastructure programmatically, but ultimately we need to add one or more layers of API-based middleware to make the solution usable and flexible.

This post appeared on Orange Matter as “Automating The Automators“, but I’m also linking to the version posted on Thwack, mainly because that format allows me to use more images and be slightly more irreverent; you don’t want to miss the great artwork on this one.

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Automating the Automators and give me a share/like. Thank you!

Cisco SP Nails It at NFDx

The Networking Field Day Exclusive one-day event with Cisco’s Service Provider business unit definitely exceeded my expectations, and I believe showcased a different approach to technology and their customers than we might have seen from the Cisco Systems of four or five years ago.

Segment Routing

The topic-du-jour was definitely Segment Routing, and Cisco delivered great presentations on both SR-TE (Segment Routing – Tunnel Engineering) with SR Flexible Algorithm, and SRv6 (Segment Routing for IPv6). 

SR FlexAlgo

SR FlexAlgo effectively allows a network to calculate metric- and constraint-based primary and backup paths on demand and in a distributed fashion. For example, a policy might be that traffic to a given prefix should follow the lowest latency path using only MACSEC encrypted links, or perhaps the lowest cost path while staying within a particular geographical region. Cool stuff, and while it won’t fix every problem, conceptually I can see this as a relatively accessible way into Segment Routing, and one which could deliver tunnel engineering in a way that would be highly complex or impossible using RSVP-TE and a constraint-based IGP calculation.

SRv6

I had not looked at SRv6 before, and it’s a fascinatingly different beast to regular IPv4-based Segment Continue reading

Orange Matter: Silo-Busting and Dream-Dashing

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. I love automation, but it seems that dreams of a smooth customer experience can be destroyed by the persistence of engineering silos in many organizations.

This post appeared on Orange Matter as “Silo-Busting and Dream-Dashing; More Fun With Automation“, but I’m also linking to the version posted on Thwack, mainly because that format allowed me to use more images and be slightly more irreverent. Actually, quite a lot more irreverent in this particular case…

Silo Busting Automation

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Silo-Busting and Dream-Dashing and give me a share/like. Thank you!

Orange Matter: Automation Paralysis

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. This post examines how it’s easy to get so focused on automating the small stuff we have difficulty turning that into the more cohesive automation solution that we’d like to have.

This post appeared on Orange Matter as “Automation Paralysis: Why We Get Stuck Automating The Small Stuff“, but I’m also linking to the version posted on Thwack, mainly because that format allowed me to use more images and be slightly more irreverent. Irreverent? Moi? Of course.

Automation Paralysis

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Automation Paralysis and give me a share/like. Thank you!

Orange Matter: Where is Your Configuration Source of Truth?

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. The post linked here looks at where we define our source of truth for device configurations; is it the device itself? Should it be? This is a key question when looking at automation, and one we should all be asking ourselves.

This post appeared on Orange Matter as “Where Is Your Config Source of Truth?“, but I’m also linking to the version posted on Thwack, mainly because that format allowed me to use more images and be slightly more irreverent, which is perhaps a bit more in character.

Where Is Your Config Source of Truth?

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Where is Your Configuration Source of Truth? and give me a share/like. Thank you!

New Year, New Post, NFDx!

You may be thinking “Wait, he hasn’t posted in ages.. how lazy is he?” but thankfully I haven’t been entirely slothful for the last seven months. Most recently I authored a series of six posts related to SDN and automation on the Solarwinds Orange Matter blog. I can’t republish that content here, but I will be sharing links to the posts in the coming days and I hope you’ll find them interesting and thought-provoking.

Cisco SP – Networking Field Day Exclusive!

More immediately, I’m preparing to start the new year with a quick trip to see Cisco’s Service Provider group at a Networking Field Day Exclusive event. I’ve seen the proposed agenda, and it looks like it’s going to be an intense day filled with the kind of topics that I know my readers will appreciate. As always, I’ll be posting about some of the topics covered (maybe even all of them…who knows?), but it’s even better if you can take part too.

The event takes place on Tuesday, January 15th, 2019. If you can, I recommend hopping on the live stream on Tech Field Day and then using the #TFDx hashtag on Twitter to join in the Continue reading

Mellanox, Ixia and Cumulus: Part 3

Last–but not least–in the technology triumvirate presenting a joint session at Networking Field Day 17 was Cumulus Networks. This post looks at the benefits of Cumulus Linux as a NOS on the Mellanox Spectrum Ethernet switch platform.

Cumulus/Mellanox/Ixia Logos

Cumulus Networks

I’ve not yet managed to deploy Cumulus Linux in anger, but it’s on a fairly short list of Network Operating Systems (NOS) which I would like to evaluate in earnest, because every time I hear about it, I conclude that it’s a great solution. In fact, I’m having difficulty typing this post because I have to stop frequently to wipe the drool from my face.

Cumulus Linux supports around 70 switches from 8 manufacturers at this time, and perhaps obviously, that includes the Mellanox Spectrum switches that were presented during this session. This is the beauty of disaggregation of course; it’s possible to make a hardware selection, then select the software to run on it. Mellanox made a fairly strong case for why the Spectrum-based hardware is better than others, so now Cumulus has to argue for why they would be the best NOS to run on the Mellanox hardware.

Cumulus Linux, as the name suggests, is based on Debian linux. Continue reading

Mellanox, Ixia and Cumulus: Part 2

This post is part two of three in a series looking at the joint presentations made by Mellanox, Ixia and Cumulus at Networking Field Day 17, in February 2018. More specifically, this post looks at what part Ixia has to play in the deployment of an Ethernet switch fabric built using Mellanox switches and running Cumulus Linux as the Network Operating System (NOS).

Cumulus/Mellanox/Ixia Logos

Ixia

What confused me most about a presentation from Mellanox, Ixia and Cumulus about Ethernet fabrics was to figure out what role Ixia would be playing in the disaggregated model. Mellanox makes the switch hardware and Cumulus makes the switch software, so Ixia fits, well, where exactly?

IxNetwork

IxNetwork is billed as an end-to-end validation solution which in many ways undersells what it’s all about. Rather than being just more traffic-generating test equipment, IxNetwork can emulate multiple switch and server devices so that a single piece of test hardware can be connected to what it believes is a large existing infrastructure, and that hardware’s behavior and resiliency can be validated. In the demo topology, IxNetwork connects to a physical Mellanox Spectrum switch running Cumulus Linux, emulating connected servers as well as an entire leaf/switch EVPN/VXLAN fabric, attached Continue reading

Mellanox, Ixia and Cumulus: Part 1

When I saw that Mellanox was presenting at Networking Field Day 17, I was definitely curious. When I found out that I would in fact be watching a joint presentation by Mellanox, Cumulus Networks and Ixia, it is fair to consider my interest piqued. Why would these three companies present together?

Cumulus/Mellanox/Ixia Logos

It turns out that these three companies present quite a compelling story, both individually–as you would probably expect–but also when used in combination. This post looks at the role of Mellanox Ethernet switches in an Ethernet fabric.

Mellanox

To me, Mellanox has been one of those ‘behind the scenes’ companies whose hardware is all over the place but whose name, in Ethernet circles at least, is less well known. Storage and compute engineers on the other hand are likely more familiar with the Mellanox name, especially in the context of Infiniband switches and network interface cards (NICs). In 2016 Mellanox acquired EZchip, allowing the development of some very capable Ethernet switches and an expansion of the company’s portfolio; to paraphrase Amit Katz (VP, WW Ethernet Switch), Mellanox connects PCI-Express interfaces together by building NICs, cables and switches.

At the Networking Field Day event in February 2018, Continue reading

Security: Mitigating Spectre on Older Intel CPUs

I suspect all of my readers are well aware of the Spectre exploit affecting, among others, Intel CPUs going back many years. Intel for their part, after a few missteps, have issued microcode updates for more recent CPUs. But for those of us with computers running older CPUs, the solutions are less likely to be forthcoming. Thankfully there is a solution.

Spectre Logo

 

Branch Prediction and Speculative Execution

The Spectre exploit affects processors which perform branch prediction, a kind of optimistic lookahead where the processor prepares and executes a potential instruction before it is actually requested. For example, if the processor encounters conditional code (like and if..then..else construct), based on previous behavior it predicts what the most likely outcome is and thus which branch of code would be executed as a result, then loads and executes that code in advance (hence “speculative execution”). If the branch prediction is correct, then since the code was already executed the code will benefit from improved performance. Spectre abuses some predictable timing behavior of the speculative execution to be able to extract other processes’ data from the CPU caches. In other words, it’s bad news for security.

The only way to restore security Continue reading

1 2 3 10