John Herbert

Author Archives: John Herbert

Why Haven’t I Tried ZeroTier Before?

I have a confession to make: I am not currently using ZeroTier. It turns out that in this respect I am in a minority among my peers at Networking Field Day 27 and after listening to a great introduction to ZeroTier by company founder and original software author Adam Ierymenko, I now know that I need to change this.

ZeroTier Logo

ZeroTier

ZeroTier facilitates the creation of an arbitrarily distributed virtual ethernet switch through which devices can communicate. Since it’s not immediately obvious what that means, here are a few scenarios where ZeroTier can provide a solution:

Home User

Imagine that you want to access your home network when you’re out on the road. ZeroTier can hook you up.

Multi-cloud Connectivity

What if you would like compute instances in multiple cloud providers to be able to communicate directly with one another as if they were on the same VLAN? What if you could also allow your developers to connect to that VLAN and seamlessly access the compute instances without any knowledge or care about which cloud provider is hosting the instance? Or maybe you’d like the cloud instances to appear as if they were on the data center VLAN? ZeroTier can do Continue reading

HDMI Dummy Plug Success with VNC!

Spoiler alert, but I am pleased to report back that my experiment with adding an HDMI dummy plug to my Dell laptop has fixed my issues with VNC.

As I theorized in my post “VNC Cannot Currently Show the Desktop” and have since confirmed, when the laptop lid is closed, the laptop disconnects the monitor and Windows runs truly “headless”. Unfortunately VNC uses DirectX Desktop Duplication to grab a copy of what would be on the screen, and if there’s no screen there’s nothing for VNC to grab an duplicate copy of, so VNC is left doing a lot of hard work grabbing screen images using CPU rather than using the far more efficient DirectX shortcuts.

My proposed solution to this was to order an HDMI Dummy Plug, a little HDMI connector which pretends to be an HDMI monitor so that the laptop believes it has an active monitor connected. My other hope was that by having a fake external monitor for VNC to mirror, I might also be able to set it up with a higher resolution than the laptop’s own internal 1920×1080 screen, which might allow me to have a higher resolution remote session using VNC. Continue reading

VNC Cannot Currently Show the Desktop

I have a Dell Latitude E5440 laptop which most of the time I run headless in a 3D-printed stand next to its slightly bigger brother, a Dell E6500 or similar.

The laptops don’t take up much space on my desk in this vertical configuration (which is helpful as I have four laptops on my desk) and I use VNC to remote into them when I need to work on a Windows system. My main system is an Apple MacBook Pro, and I have that in a similar vertical dock with two 27″ monitors, a bluetooth keyboard and touchpad, and a USB-C port expander/charger. By using VNC I can keep using the peripherals I like and quickly switch between systems while sharing copy/paste buffers as well, which is pretty much perfect.

There’s one nagging little problem though, that I can’t get around. When I access the E5440 using RealVNC, it is slow to show the screen when initially connecting and every time there is a Windows UAC prompt I have to wait about five seconds or so while staring at a black screen which says “Cannot currently show the desktop”.

This is somewhat annoying and after digging around a bit I Continue reading

Zodiac FX Gets a 3D Printed Case

Not content with having dug the Northbound Networks Zodiac FX out of a pile of overlooked technology in my office, I thought that the poor thing desperately needed to have a case to sit in. When I originally received the switch, I did not have a 3D printer and had no idea what it would take to make a case; now though, I do have a 3D printer … and no idea what it would take to make a case. Sounds like a plan to me!

Measuring the Zodiac FX

The most important tool I bought to go with my 3D printer (a Creality CR6-SE) was some digital calipers. I discovered early on how important it was to ensure that if I was going to screw up, I should be able to screw up accurately.

Rexbeti Calipers

These calipers are made by RexBeti, and if you’ve never heard of that company that’s ok, because before I purchased this I hadn’t either. The calipers claim to be accurate to 0.01mm, but I don’t have any way to validate that claim, so let’s just assume that they are. I do know that it beats using a ruler. A few minutes of careful Continue reading

Upgrading Firmware on Northbound Networks Zodiac FX

Recent versions of firmware (after v0.80) running on the Northbound Networks Zodiac FX can be updated directly from the web interface, or using XMODEM via the serial console. But what if, say, you had sat one your Zodiac FX for a while and are running firmware earlier than v0.81 and have a sudden, unexpected desire to upgrade the firmware? Say you are, for example, me?

The process turned out to be less straightforward than I had hoped, so I am documenting the successful steps I followed in case it’s of use to somebody else.

My (Brief) Zodiac FX Background

Back in 2015 I backed a Kickstarter project for this awesome-sounding four-port FastEthernet SDN switch with OpenFlow support. It sounded so cool that I even ordered a two-pack as I thought it would be more fun to have two OpenFlow switches to mess around with). The project was funded successfully, but embarrassingly when the beautifully-made boards arrived in early 2016, for some reason I never quite got around to playing with them. I think it was in part because it was just a printed circuit board without a case and without easy access to 3D printing I was turned Continue reading

Farewell to Northbound Networks

Digging through my office looking for some other technology which I had misplaced, I stumbled across a small box containing a Northbound Networks Zodiac-FX, a small 4-port FastEthernet OpenFlow SDN switch which I had picked up after backing a 2015 kickstarter campaign.

Northbound Networks Zodiac FX SDN Switch

These were a pretty cool idea, and at the time OpenFlow (OF) was the hottest thing around, everything was being SDN-washed, and the idea that a regular user like myself could afford actual hardware with OF capabilities to toy with in the home lab was beyond belief. Of course, it was possible to virtualize OF with Mininet, but there’s something about using a real switch that goes beyond that. Even though, as you’ll in a future post, I ended up wasting that opportunity, I am still honored to have backed it, and my hat is off to Northbound Networks’ founder Paul Zanna for what he has accomplished.

Paying My Respects

With that in mind, I’m sad to note that when I went to the Northbound Networks website, I discovered that some time around August 2020 the company stopped manufacturing SDN hardware.

Northbound Networks home page, January 2021

Since the original Zodiac FX campaign, Paul had expanded the available products to include an 802. Continue reading

Response to “Certifications Are Not A Big Deal. Stop Being a Princess About It.”

In a post which now appears to have been deleted, Greg Ferro got right to the point in his article Response: Certifications Are Not A Big Deal. Stop Being a Princess About It.. The majority of this response was written while Greg’s post was still active, but I had to come back and inject more context after I spotted on June 30, 2019 that the post had become unavailable.

To save you digging in the WayBackMachine, the history to Greg’s post as I understand it is that Greg made some comments in Episode 238 of the Packet Pushers’ Network Break suggesting that vendor certifications were trivial. A listener evidently gave some strong feed back disagreeing with this, and so in Episode 239 of the Packet Pushers’ Network Break Greg responded to that feedback, and reiterated his position about certification study, specifically framed around Cisco’s CCNP. Greg made some reasonable points; that the certification programs from the vendors are not designed to teach fundamentals in the same way that, say, a computer science degree might do, and that the aim is really to make money for the vendor, and reduce their tech support costs, and as such the vendor certification education Continue reading

Cranky Old Network Engineer Complains About The Youth Of Today

If you’re very old (like me) you’ll likely remember the halcyon days when IP routing was not enabled by default on Cisco routers. Younger gamers may find this hard to believe, which makes it even stranger when I keep bumping into an apparently common misconception about how routers work. Let’s take a look at what I’m beefing about.

No IP Routing?

To put this in context for the younger gamers, it’s worth noting that at the time, a typical “enterprise” might be running IP, but was equally likely to run IPX, AppleTalk, DECnet or some other protocol which may – or may not – support routing. Yes, there was life before the Internet Protocol became ubiquitous. If you’re curious, the command to enable IP routing is, well:

ip routing

Guess how IPX routing was enabled:

ipx routing

Appletalk?

appletalk routing

DECnet Phase IV?

decnet [network-number] routing <decnet-address>

Ok, so the pattern isn’t entirely consistent, but it’s close enough. In one way things are much simpler now because routers tend to handle IP (and IPv6) and nothing else. On the other hand there are so many more IP-related features available, I think we should just be grateful that there’s only one Continue reading

The Achilles Heel of the API

I’ve been developing yet more automation recently, and I’ve been hitting two major stumbling blocks that have had a negative impact on my ability to complete the tooling.

API Documentation

When APIs were first made available, the documentation from many vendors was simply incomplete; it seemed that the documentation team was always a release or two behind the people implementing the API. To fix that, a number of vendors have moved to a self-documenting API system along the lines of Swagger. The theory is that if you build an API endpoint, you’re automatically building the documentation for it at the same time, which is a super idea. This has improved the API’s endpoint coverage but in some cases has resulted in thorough documentation explaining what the endpoints are, but little to no documentation explaining why one would choose to use a particular endpoint. 

As a result, with one API in particular I have been losing my mind trying to understand which endpoint I should use to accomplish a particular task, when no less than three of them appear to handle the same thing. I’m then left using trial and error to determine the correct path, and at the end Continue reading

A10 Networks ACOS Root Privilege Escalation

The following summarizes a root privilege escalation vulnerability that I identified in A10 ACOS ADC software. This was disclosed to A10 Networks in June 2016 and mitigations have been put in place to limit exposure to the vulnerability.

A10 Networks Cookie Vulnerability

SUMMARY OF VULNERABILITY

Any user assigned sufficient privilege to upload an external health monitor (i.e a script) and reference it from a health monitor can gain root shell access to ACOS.

At this point, I respectfully acknowledge Raymond Chen’s wise words about being on the other side of an airtight hatch; if the malicious user is already a system administrator or has broad permissions, then one could argue that they could already do huge damage to the ADC in other ways. However, root access could allow that user to install persistent backdoors or monitoring threats in the underlying OS where other users can neither see nor access them. It could also allow a partition-level administrator to escalate effectively to a global admin, by way of being able to see the files in every partition on the ADC.

SOFTWARE VERSIONS TESTED:

This vulnerability was originally discovered and validated initially in ACOS 2.7.2-P4-SP2 and is present in 4.x as Continue reading

Meraki In The Middle – Smart Security Cameras

I’ve been looking at security cameras recently, in part because my home owners association needs to upgrade the system which monitors some of the amenities. We want motion detection features and, obviously, remote access to view live cameras and recorded footage without having to go to the location. Unfortunately there’s a gap in the market which seems to be exactly where I’m looking. Cisco Meraki may have just stepped in and bridged that gap.

The Problem Space

Low-End Products

Over the last few years, a wide variety of small security cameras have become available, any of which which at first glance would appear suitable. These include products like Netgear’s Arlo, Amazon’s Blink, Google’s Nest Cam and more. After some brief testing, however, I’m a little less convinced that they are what we’re looking for. It sounds silly to say it, because it’s not like this is something they hide, but these products are all aimed at the home user market. Dashboard logins are single user, based on an email address and the web interfaces may not work well for much more than five or so cameras. The camera choices are fairly limited, and as they’ll be streaming their Continue reading

Orange Matter: Why Your Infrastructure Sucks For Automation

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. Let’s face it, unless we get to build an infrastructure from the ground up, our existing mass of one-off solutions and workarounds makes automating our infrastructure an absolute nightmare.

This post appeared on Orange Matter as “Why Your Infrastctructure Sucks For Automation“, but I’m also linking to the version posted on Thwack, because that version of the post includes pretty pictures. And who doesn’t like a pretty picture?

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Why Your Infrastructure Sucks For Automation and give me a share/like. Thank you!

Viavi Enterprise Provides Unexpected Network Insights

Many of us will have experienced the challenges of taking a performance alert (or user complaint) and drilling down to root cause. Performance issues can be intermittent, and it can be difficult to get visibility of what caused a problem at a particular time. Viavi Enterprise thinks it has the answer, combining analysis of packet feeds (e.g. from taps and mirror ports) and IPFix, xFlow and cloud service flow logs to monitor application performance as it would be experienced by a user. Sounds good? It looked pretty good, too.

Johnny Five Need Input!

Nothing can happen without data, and that comes from a number of potential sources.

Observer Gigastor

The Observer Gigastor product is available as a virtualized solution (to capture east-west traffic in virtualized environments), a portable appliance for tactical deployment, and two hardware appliance models (in a charming shade of purple) which can provide from 96TB to 1.2PB of storage. The idea of Gigastor is to capture packets at line rate and retain the raw packet data in case it’s needed later. The packets are analyzed, and that metadata is fed to the reporting and visualization system, Observer Apex.

Observer GigaFlow

It’s not always possible Continue reading

A10 Networks ACOS Critical Insecure Cookie Vulnerability 2 of 2

The following summarizes an HTTP persistence cookie vulnerability that I identified in A10 ACOS ADC software. This was disclosed to A10 Networks in June 2016 and has now been resolved.

A10 Networks Cookie Vulnerability

As noted in a previous post, ACOS uses insecure HTTP/HTTPS persistence cookies which can allow a malicious user to craft a cookie determining the server and port to which a persistent session should be sent. In addition, for vports using the default “port-based” HTTP cookie persistence, it was discovered that when using the default persistence cookie type, ACOS does not perform a check to ensure that the server/port defined in the cookie is within the configured service-group for that VIP.

The only sanity check appears to be to ensure that the server IP read from the cookie has been configured on the A10 within the same partition. If that constraint is met, packets will be forwarded by ACOS to the real server based solely on the value contained in the cookie. This is extremely serious as it allows a malicious user to connect, for example, through a public VIP and access back end servers used by other VIPs, including those only accessible via internal IPs.

SUMMARY OF VULNERABILITY

When using Continue reading

Orange Matter: All I Want For Christmas is RESTCONF

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. I need clear commands, structured response data and simple access to it all; so how about giving me REST-based APIs on all my infrastructure equipment?

This post appeared on Orange Matter as “All I Want For Christmas is RESTCONF“, but I’m also linking to the version posted on Thwack, in case you prefer to read and comment there.

I’d love it if you were to take a moment to visit and read, and maybe even comment!

(Featured image created by Kira auf der Heide on Unsplash)

If you liked this post, please do click through to the source at Orange Matter: All I Want For Christmas is RESTCONF and give me a share/like. Thank you!

A10 Networks ACOS Critical Insecure Cookie Vulnerability 1 of 2

The following summarizes an HTTP persistence cookie vulnerability that I identified in A10’s ACOS ADC software. This issue was disclosed to A10 Networks in June 2016 and has since been resolved.

A10 Networks Cookie Vulnerability

This vulnerability results in information disclosure about names of service-groups and IPs of real servers, as well as the ability to manipulate the content of the cookies.

SUMMARY OF VULNERABILITY

The ACOS documentation for HTTP persistence cookies notes that “For security, address information in the persistence cookies is encrypted.” However, the address information is not “encrypted”; rather, the real server IP and port information is weakly obfuscated and is easily decoded, exposing information about the internal network. The simplicity of the obfuscation also makes it trivial to manually create a cookie which ACOS would decode and honor.

Additionally, cookies configured using the service-group command option have the service-group’s full name included in the persistence cookie as plain text. This vulnerability applies to HTTP/HTTPS VIP types that have been configured to use a cookie-based persistence template.

SOFTWARE VERSIONS TESTED

This vulnerability was discovered and validated initially in ACOS 2.7.2-P4-SP2 and reconfirmed most recently in ACOS 4.1.1-P3.

VULNERABLE VERSIONS

This behavior has been core to Continue reading

Orange Matter: Automating the Automators

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. APIs are critical to operating infrastructure programmatically, but ultimately we need to add one or more layers of API-based middleware to make the solution usable and flexible.

This post appeared on Orange Matter as “Automating The Automators“, but I’m also linking to the version posted on Thwack, mainly because that format allows me to use more images and be slightly more irreverent; you don’t want to miss the great artwork on this one.

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Automating the Automators and give me a share/like. Thank you!

Cisco SP Nails It at NFDx

The Networking Field Day Exclusive one-day event with Cisco’s Service Provider business unit definitely exceeded my expectations, and I believe showcased a different approach to technology and their customers than we might have seen from the Cisco Systems of four or five years ago.

Segment Routing

The topic-du-jour was definitely Segment Routing, and Cisco delivered great presentations on both SR-TE (Segment Routing – Tunnel Engineering) with SR Flexible Algorithm, and SRv6 (Segment Routing for IPv6). 

SR FlexAlgo

SR FlexAlgo effectively allows a network to calculate metric- and constraint-based primary and backup paths on demand and in a distributed fashion. For example, a policy might be that traffic to a given prefix should follow the lowest latency path using only MACSEC encrypted links, or perhaps the lowest cost path while staying within a particular geographical region. Cool stuff, and while it won’t fix every problem, conceptually I can see this as a relatively accessible way into Segment Routing, and one which could deliver tunnel engineering in a way that would be highly complex or impossible using RSVP-TE and a constraint-based IGP calculation.

SRv6

I had not looked at SRv6 before, and it’s a fascinatingly different beast to regular IPv4-based Segment Continue reading

Orange Matter: Silo-Busting and Dream-Dashing

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. I love automation, but it seems that dreams of a smooth customer experience can be destroyed by the persistence of engineering silos in many organizations.

This post appeared on Orange Matter as “Silo-Busting and Dream-Dashing; More Fun With Automation“, but I’m also linking to the version posted on Thwack, mainly because that format allowed me to use more images and be slightly more irreverent. Actually, quite a lot more irreverent in this particular case…

Silo Busting Automation

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Silo-Busting and Dream-Dashing and give me a share/like. Thank you!

Orange Matter: Automation Paralysis

Orange Matter Logo

I’ve been blogging for Solarwinds recently, posting on Orange Matter, with a cross-post to the Thwack Geek Speak forum. This post examines how it’s easy to get so focused on automating the small stuff we have difficulty turning that into the more cohesive automation solution that we’d like to have.

This post appeared on Orange Matter as “Automation Paralysis: Why We Get Stuck Automating The Small Stuff“, but I’m also linking to the version posted on Thwack, mainly because that format allowed me to use more images and be slightly more irreverent. Irreverent? Moi? Of course.

Automation Paralysis

I’d love it if you were to take a moment to visit and read, and maybe even comment!

If you liked this post, please do click through to the source at Orange Matter: Automation Paralysis and give me a share/like. Thank you!

1 2 3 11