John Herbert

Author Archives: John Herbert

A10 Networks ACOS Root Privilege Escalation

The following summarizes a root privilege escalation vulnerability that I identified in A10 ACOS ADC software. This was disclosed to A10 Networks in June 2016 and mitigations have been put in place to limit exposure to the vulnerability.

A10 Networks Cookie Vulnerability

SUMMARY OF VULNERABILITY

Any user assigned sufficient privilege to upload an external health monitor (i.e a script) and reference it from a health monitor can gain root shell access to ACOS.

At this point, I respectfully acknowledge Raymond Chen’s wise words about being on the other side of an airtight hatch; if the malicious user is already a system administrator or has broad permissions, then one could argue that they could already do huge damage to the ADC in other ways. However, root access could allow that user to install persistent backdoors or monitoring threats in the underlying OS where other users can neither see nor access them. It could also allow a partition-level administrator to escalate effectively to a global admin, by way of being able to see the files in every partition on the ADC.

SOFTWARE VERSIONS TESTED:

This vulnerability was originally discovered and validated initially in ACOS 2.7.2-P4-SP2 and is present in 4.x as Continue reading

Meraki In The Middle – Smart Security Cameras

I’ve been looking at security cameras recently, in part because my home owners association needs to upgrade the system which monitors some of the amenities. We want motion detection features and, obviously, remote access to view live cameras and recorded footage without having to go to the location. Unfortunately there’s a gap in the market which seems to be exactly where I’m looking. Cisco Meraki may have just stepped in and bridged that gap.

The Problem Space

Low-End Products

Over the last few years, a wide variety of small security cameras have become available, any of which which at first glance would appear suitable. These include products like Netgear’s Arlo, Amazon’s Blink, Google’s Nest Cam and more. After some brief testing, however, I’m a little less convinced that they are what we’re looking for. It sounds silly to say it, because it’s not like this is something they hide, but these products are all aimed at the home user market. Dashboard logins are single user, based on an email address and the web interfaces may not work well for much more than five or so cameras. The camera choices are fairly limited, and as they’ll be streaming their Continue reading

Orange Matter: Why Your Infrastructure Sucks For Automation

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. Let’s face it, unless we get to build an infrastructure from the ground up, our existing mass of one-off solutions and workarounds makes automating our infrastructure an absolute nightmare.

This post appeared on Orange Matter as “Why Your Infrastctructure Sucks For Automation“, but I’m also linking to the version posted on Thwack, because that version of the post includes pretty pictures. And who doesn’t like a pretty picture?

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Why Your Infrastructure Sucks For Automation and give me a share/like. Thank you!

Viavi Enterprise Provides Unexpected Network Insights

Many of us will have experienced the challenges of taking a performance alert (or user complaint) and drilling down to root cause. Performance issues can be intermittent, and it can be difficult to get visibility of what caused a problem at a particular time. Viavi Enterprise thinks it has the answer, combining analysis of packet feeds (e.g. from taps and mirror ports) and IPFix, xFlow and cloud service flow logs to monitor application performance as it would be experienced by a user. Sounds good? It looked pretty good, too.

Johnny Five Need Input!

Nothing can happen without data, and that comes from a number of potential sources.

Observer Gigastor

The Observer Gigastor product is available as a virtualized solution (to capture east-west traffic in virtualized environments), a portable appliance for tactical deployment, and two hardware appliance models (in a charming shade of purple) which can provide from 96TB to 1.2PB of storage. The idea of Gigastor is to capture packets at line rate and retain the raw packet data in case it’s needed later. The packets are analyzed, and that metadata is fed to the reporting and visualization system, Observer Apex.

Observer GigaFlow

It’s not always possible Continue reading

A10 Networks ACOS Critical Insecure Cookie Vulnerability 2 of 2

The following summarizes an HTTP persistence cookie vulnerability that I identified in A10 ACOS ADC software. This was disclosed to A10 Networks in June 2016 and has now been resolved.

A10 Networks Cookie Vulnerability

As noted in a previous post, ACOS uses insecure HTTP/HTTPS persistence cookies which can allow a malicious user to craft a cookie determining the server and port to which a persistent session should be sent. In addition, for vports using the default “port-based” HTTP cookie persistence, it was discovered that when using the default persistence cookie type, ACOS does not perform a check to ensure that the server/port defined in the cookie is within the configured service-group for that VIP.

The only sanity check appears to be to ensure that the server IP read from the cookie has been configured on the A10 within the same partition. If that constraint is met, packets will be forwarded by ACOS to the real server based solely on the value contained in the cookie. This is extremely serious as it allows a malicious user to connect, for example, through a public VIP and access back end servers used by other VIPs, including those only accessible via internal IPs.

SUMMARY OF VULNERABILITY

When using Continue reading

Orange Matter: All I Want For Christmas is RESTCONF

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. I need clear commands, structured response data and simple access to it all; so how about giving me REST-based APIs on all my infrastructure equipment?

This post appeared on Orange Matter as “All I Want For Christmas is RESTCONF“, but I’m also linking to the version posted on Thwack, in case you prefer to read and comment there.

I’d love it if you were to take a moment to visit and read, and maybe even comment!

(Featured image created by Kira auf der Heide on Unsplash)

If you liked this post, please do click through to the source at Orange Matter: All I Want For Christmas is RESTCONF and give me a share/like. Thank you!

A10 Networks ACOS Critical Insecure Cookie Vulnerability 1 of 2

The following summarizes an HTTP persistence cookie vulnerability that I identified in A10’s ACOS ADC software. This issue was disclosed to A10 Networks in June 2016 and has since been resolved.

A10 Networks Cookie Vulnerability

This vulnerability results in information disclosure about names of service-groups and IPs of real servers, as well as the ability to manipulate the content of the cookies.

SUMMARY OF VULNERABILITY

The ACOS documentation for HTTP persistence cookies notes that “For security, address information in the persistence cookies is encrypted.” However, the address information is not “encrypted”; rather, the real server IP and port information is weakly obfuscated and is easily decoded, exposing information about the internal network. The simplicity of the obfuscation also makes it trivial to manually create a cookie which ACOS would decode and honor.

Additionally, cookies configured using the service-group command option have the service-group’s full name included in the persistence cookie as plain text. This vulnerability applies to HTTP/HTTPS VIP types that have been configured to use a cookie-based persistence template.

SOFTWARE VERSIONS TESTED

This vulnerability was discovered and validated initially in ACOS 2.7.2-P4-SP2 and reconfirmed most recently in ACOS 4.1.1-P3.

VULNERABLE VERSIONS

This behavior has been core to Continue reading

Orange Matter: Automating the Automators

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. APIs are critical to operating infrastructure programmatically, but ultimately we need to add one or more layers of API-based middleware to make the solution usable and flexible.

This post appeared on Orange Matter as “Automating The Automators“, but I’m also linking to the version posted on Thwack, mainly because that format allows me to use more images and be slightly more irreverent; you don’t want to miss the great artwork on this one.

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Automating the Automators and give me a share/like. Thank you!

Cisco SP Nails It at NFDx

The Networking Field Day Exclusive one-day event with Cisco’s Service Provider business unit definitely exceeded my expectations, and I believe showcased a different approach to technology and their customers than we might have seen from the Cisco Systems of four or five years ago.

Segment Routing

The topic-du-jour was definitely Segment Routing, and Cisco delivered great presentations on both SR-TE (Segment Routing – Tunnel Engineering) with SR Flexible Algorithm, and SRv6 (Segment Routing for IPv6). 

SR FlexAlgo

SR FlexAlgo effectively allows a network to calculate metric- and constraint-based primary and backup paths on demand and in a distributed fashion. For example, a policy might be that traffic to a given prefix should follow the lowest latency path using only MACSEC encrypted links, or perhaps the lowest cost path while staying within a particular geographical region. Cool stuff, and while it won’t fix every problem, conceptually I can see this as a relatively accessible way into Segment Routing, and one which could deliver tunnel engineering in a way that would be highly complex or impossible using RSVP-TE and a constraint-based IGP calculation.

SRv6

I had not looked at SRv6 before, and it’s a fascinatingly different beast to regular IPv4-based Segment Continue reading

Orange Matter: Silo-Busting and Dream-Dashing

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. I love automation, but it seems that dreams of a smooth customer experience can be destroyed by the persistence of engineering silos in many organizations.

This post appeared on Orange Matter as “Silo-Busting and Dream-Dashing; More Fun With Automation“, but I’m also linking to the version posted on Thwack, mainly because that format allowed me to use more images and be slightly more irreverent. Actually, quite a lot more irreverent in this particular case…

Silo Busting Automation

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Silo-Busting and Dream-Dashing and give me a share/like. Thank you!

Orange Matter: Automation Paralysis

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. This post examines how it’s easy to get so focused on automating the small stuff we have difficulty turning that into the more cohesive automation solution that we’d like to have.

This post appeared on Orange Matter as “Automation Paralysis: Why We Get Stuck Automating The Small Stuff“, but I’m also linking to the version posted on Thwack, mainly because that format allowed me to use more images and be slightly more irreverent. Irreverent? Moi? Of course.

Automation Paralysis

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Automation Paralysis and give me a share/like. Thank you!

Orange Matter: Where is Your Configuration Source of Truth?

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. The post linked here looks at where we define our source of truth for device configurations; is it the device itself? Should it be? This is a key question when looking at automation, and one we should all be asking ourselves.

This post appeared on Orange Matter as “Where Is Your Config Source of Truth?“, but I’m also linking to the version posted on Thwack, mainly because that format allowed me to use more images and be slightly more irreverent, which is perhaps a bit more in character.

Where Is Your Config Source of Truth?

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Where is Your Configuration Source of Truth? and give me a share/like. Thank you!

New Year, New Post, NFDx!

You may be thinking “Wait, he hasn’t posted in ages.. how lazy is he?” but thankfully I haven’t been entirely slothful for the last seven months. Most recently I authored a series of six posts related to SDN and automation on the Solarwinds Orange Matter blog. I can’t republish that content here, but I will be sharing links to the posts in the coming days and I hope you’ll find them interesting and thought-provoking.

Cisco SP – Networking Field Day Exclusive!

More immediately, I’m preparing to start the new year with a quick trip to see Cisco’s Service Provider group at a Networking Field Day Exclusive event. I’ve seen the proposed agenda, and it looks like it’s going to be an intense day filled with the kind of topics that I know my readers will appreciate. As always, I’ll be posting about some of the topics covered (maybe even all of them…who knows?), but it’s even better if you can take part too.

The event takes place on Tuesday, January 15th, 2019. If you can, I recommend hopping on the live stream on Tech Field Day and then using the #TFDx hashtag on Twitter to join in the Continue reading

Mellanox, Ixia and Cumulus: Part 3

Last–but not least–in the technology triumvirate presenting a joint session at Networking Field Day 17 was Cumulus Networks. This post looks at the benefits of Cumulus Linux as a NOS on the Mellanox Spectrum Ethernet switch platform.

Cumulus/Mellanox/Ixia Logos

Cumulus Networks

I’ve not yet managed to deploy Cumulus Linux in anger, but it’s on a fairly short list of Network Operating Systems (NOS) which I would like to evaluate in earnest, because every time I hear about it, I conclude that it’s a great solution. In fact, I’m having difficulty typing this post because I have to stop frequently to wipe the drool from my face.

Cumulus Linux supports around 70 switches from 8 manufacturers at this time, and perhaps obviously, that includes the Mellanox Spectrum switches that were presented during this session. This is the beauty of disaggregation of course; it’s possible to make a hardware selection, then select the software to run on it. Mellanox made a fairly strong case for why the Spectrum-based hardware is better than others, so now Cumulus has to argue for why they would be the best NOS to run on the Mellanox hardware.

Cumulus Linux, as the name suggests, is based on Debian linux. Continue reading

Mellanox, Ixia and Cumulus: Part 2

This post is part two of three in a series looking at the joint presentations made by Mellanox, Ixia and Cumulus at Networking Field Day 17, in February 2018. More specifically, this post looks at what part Ixia has to play in the deployment of an Ethernet switch fabric built using Mellanox switches and running Cumulus Linux as the Network Operating System (NOS).

Cumulus/Mellanox/Ixia Logos

Ixia

What confused me most about a presentation from Mellanox, Ixia and Cumulus about Ethernet fabrics was to figure out what role Ixia would be playing in the disaggregated model. Mellanox makes the switch hardware and Cumulus makes the switch software, so Ixia fits, well, where exactly?

IxNetwork

IxNetwork is billed as an end-to-end validation solution which in many ways undersells what it’s all about. Rather than being just more traffic-generating test equipment, IxNetwork can emulate multiple switch and server devices so that a single piece of test hardware can be connected to what it believes is a large existing infrastructure, and that hardware’s behavior and resiliency can be validated. In the demo topology, IxNetwork connects to a physical Mellanox Spectrum switch running Cumulus Linux, emulating connected servers as well as an entire leaf/switch EVPN/VXLAN fabric, attached Continue reading

Mellanox, Ixia and Cumulus: Part 1

When I saw that Mellanox was presenting at Networking Field Day 17, I was definitely curious. When I found out that I would in fact be watching a joint presentation by Mellanox, Cumulus Networks and Ixia, it is fair to consider my interest piqued. Why would these three companies present together?

Cumulus/Mellanox/Ixia Logos

It turns out that these three companies present quite a compelling story, both individually–as you would probably expect–but also when used in combination. This post looks at the role of Mellanox Ethernet switches in an Ethernet fabric.

Mellanox

To me, Mellanox has been one of those ‘behind the scenes’ companies whose hardware is all over the place but whose name, in Ethernet circles at least, is less well known. Storage and compute engineers on the other hand are likely more familiar with the Mellanox name, especially in the context of Infiniband switches and network interface cards (NICs). In 2016 Mellanox acquired EZchip, allowing the development of some very capable Ethernet switches and an expansion of the company’s portfolio; to paraphrase Amit Katz (VP, WW Ethernet Switch), Mellanox connects PCI-Express interfaces together by building NICs, cables and switches.

At the Networking Field Day event in February 2018, Continue reading

Security: Mitigating Spectre on Older Intel CPUs

I suspect all of my readers are well aware of the Spectre exploit affecting, among others, Intel CPUs going back many years. Intel for their part, after a few missteps, have issued microcode updates for more recent CPUs. But for those of us with computers running older CPUs, the solutions are less likely to be forthcoming. Thankfully there is a solution.

Spectre Logo

 

Branch Prediction and Speculative Execution

The Spectre exploit affects processors which perform branch prediction, a kind of optimistic lookahead where the processor prepares and executes a potential instruction before it is actually requested. For example, if the processor encounters conditional code (like and if..then..else construct), based on previous behavior it predicts what the most likely outcome is and thus which branch of code would be executed as a result, then loads and executes that code in advance (hence “speculative execution”). If the branch prediction is correct, then since the code was already executed the code will benefit from improved performance. Spectre abuses some predictable timing behavior of the speculative execution to be able to extract other processes’ data from the CPU caches. In other words, it’s bad news for security.

The only way to restore security Continue reading

Response: The Need For Stretched VLANs (@ioshints)

A recent post from Ivan Pepelnjak entitled Revisited: The Need For Stretched VLANs made me smile rather bitterly as Ivan dug into the apparent continued desire for stretched layer 2 networks and the reasons people give for the solution’s requirement and validity. I love a good bit of snark as much as the next nerd, so as you can imagine, I’m all over that post.

John Herbert, Expressing Extreme Disbelief At The Horror He Is Reading

However, I confess I did wince slightly – in the way one might do when an old wound is poked with a sharp stick – as Ivan made a passing sarcastic reference to Microsoft’s amazing Network Load Balancing technology:

ipspace.net: Revisited: The Need For Stretched VLANs

My mind was thrown back to the heady days of 2009 when I stumbled across another post from Mr Pepelnjak, this time entitled Turn a switch into a hub … the Microsoft Way which bemoaned the unadulterated stupidity of Microsoft’s attempt to use layer 2 network flooding to accomplish clustering. I had discovered the nature of this behavior at a previous client and had my mind blown by the very stupid and non-standards-compliant way in which this had been implemented.

The reason my mind went to that post, however, is because if I recall correctly it’s Continue reading

This History Of Networking, from The Network Collective

I mentioned The Network Collective previously when I responded to the very first episode of the videocast/podcast (what TWiT would call a netcast). Since then the three founders and co-hosts (Jordan, Eyvonne and Phil) have published an impressive 20 community roundtable episodes and have somehow also found time to launch a History of Networking series co-hosted by Russ White (yes, that Russ White).

The Network Collective

History Of Networking

I’m a bit of a nerd when it comes to computer history, and I love reading books that give the inside story about the birth of the personal computer, the story of Silicon Valley, the rise and fall of technology companies and so on. However, the history of networking is nowhere near as well covered, which is a real shame. Thankfully, The Network Collective are filling that gap handsomely with a list of guests so far that blows my mind. For example:

Paul Vixie

Paul Vixie on the History of Networking

If you’ve ever heard of Vixie cron, BIND DNS, DNSSEC, the Internet Software Consortium (ISC), you’ve found things Mr Vixie has had his hands all over. It’s fascinating to hear him talking about the history of DNS adoption, and his role in maintaining BIND in a nascent Continue reading

Networking Field Day 17 (NFD17) Redux

As I write this it’s Friday afternoon in Silicon Valley, Networking Field Day 17 (NFD17) is over, and I am truly exhausted after a week of presentations covering topics like Ethernet switch hardware, testing/simulation and network operating system (Mellanox, Ixia, Cumulus), Field Area Networks (Cisco), visibility and automation (Juniper AppFormix), distributed network monitoring (Thousand Eyes) and much more.

I will be posting in the coming weeks about some of the individual presentations, but while I dry off from the information firehose I thought I’d briefly summarize some of my personal highlights from the week.

Networking Field Day Logo

Themes

One recurring theme throughout the week was automation. Cisco for example, spoke about the capabilities for Zero Touch Provisioning (ZTP) in IOS XR. Juniper presented the OpenContrail platform which, impressively, appears to have a new and very positive sense of direction after (in my opinion) floating around somewhat aimlessly for a while. Extreme Networks spoke about Workflow Composer, and its ability to integrate with tools like Splunk to auto-remediate detected issues. VMware demonstrated the impressive capabilities of NSX to configure distributed firewalling and microsegmentation down to the container level, while VeloCloud from VMware demonstrated another facet of automation, that of the software defined WAN.

Another Continue reading

1 2 3 10