Time to move away from HPE Software

If you are still using HPE Software, you should actively plan to migrate away. The recent divestiture does not look good to me – I think existing customers are going to get soaked. Plan your migration now.

I’ve said it before, that I retain a soft spot for Hewlett-Packard. They gave me my first professional job out of university. I served my sentence doing HP OpenView consulting, and HP-UX Administration, but still: it got me started. Once you have some professional experience, it’s much easier to move to the next role.

It saddens me to watch HP’s ongoing struggles. It’s sad to watch a big ship get broken up for parts. But things had to change. They need to do something to adapt to the realities of modern IT demands.

There was one line in the recent announcement about divesting HPE’s software assets that stood out to me:

Micro Focus expects to improve the margin on HPE’s software assets by approximately 20 percentage points by the end of the third full financial year following the closing of the transaction

(Emphasis added).

It has been clear for a while that HP Software was no longer a core asset for HPE. It Continue reading

Stop using mobiles for conference calls

Stop using legacy mobile audio, especially for conference calls. There are better alternatives. You’re doing your customers and colleagues a disservice by using mobile audio. It’s time we moved on. PSTN is not much better either – switch to VoIP, and give your ears a break from crappy audio connections.

Refresher: Audio Quality Standards

There are many different methods of encoding speech for transmission across networks. There are trade-offs with each, balancing bandwidth, voice quality, and endpoint requirements. The interesting point is that there is not a direct relationship between bandwidth and quality. Half the bandwidth does not have to mean half the quality.

The Mean opinion score test provides a way of ‘scoring’ the quality of a call. 1 is Bad, 5 is Excellent. G.711 encoding has a score of 4.1, which is very good quality, but uses 64kbps per call. GSM has a score of 3.5, which is the minimum acceptable level…but it only uses 12.2kbps. Pretty good tradeoff if you’re in a bandwidth-constrained environment.

But we’re no longer constrained by bandwidth. We don’t need to squeeze that audio call down to only a few kbps. We can use other options such as FaceTime, Continue reading

Relocated at last

Just a quick note to let you know that I am now based in the San Francisco Bay Area. After much preparation, and administrative hassle, everything is now sorted. My company has relocated me to the Bay Area, where I will work at the San Jose HQ.

Anna has of course joined me. We’re living in short-term accommodation in San Francisco right now, and over the next couple of months we’ll figure out where we want to stay long-term.

Lots to do, and lots to learn. But I think it will be a good move for me professionally, and I hope that Anna enjoys it too.

If you live in the Bay Area, or you’re passing through, I’d love to catch up with you, once we get settled. I’m looking forward to being able to unpack my bags in about a week or so!

netmiko support for Brocade ICX and MLXe

netmiko is a “Multi-vendor library to simplify Paramiko SSH connections to network devices,” written by Kirk Byers. It doesn’t solve all of your pain with dealing with CLI-only network devices, but it tries to at least take away the low-level hassle of setting up a connection, and handling variations with things like enable mode, line-breaks, etc.

I’ve submitted a couple of PRs over the last few days to support Brocade ICX and MLXe devices – #235, #236 and #237. These have now been merged into the master code.

This has not yet had extensive testing. Please try it out, and report any issues.

I’m currently looking at VDX support. Looks like a few oddities around detecting the prompt, and dealing with the banner. Feel free to pitch in!

VRRP Skew Time (and always be learning…)

It’s funny how you can work with something for years, but miss a small detail. This week I learnt about Skew Time for VRRP. The reason for it is completely obvious once you think about it, but for some reason the detail had escaped me for all these years.

VRRP Hellos

VRRP sends out a “hello” multicast every <hello> seconds. Usually this is something like every 1 or 3 seconds. Unlike HSRP, only the current master sends out hello messages. This contains the current master priority & status.

The backup devices listen out for this hello message. If they think they have a higher priority, or if they fail to hear the hello message, they will assume the role of master.

Down Interval

Changing from backup to master because of one missed hello could cause network instability. There’s a common rule used for all keepalive-type messages, where backup devices will wait for three missed polls/keepalives before declaring something ‘down.’

VRRP is similar. It waits three poll intervals before declaring the master ‘down,’ and attempting to Continue reading

Travel Badge of Shame

All frequent flyers strive to the top tier of their program. Qantas Platinum, BA Gold, KrisFlyer Elite Gold, United Premier 1K. They all want that extra level of benefits, those extra upgrades.

But a former manager said:

“You don’t really want to be on the top tier. You want to be on the tier just below, where you get most of the useful benefits like priority check-in, priority luggage, and lounge access. The top tier is actually a badge of shame, because it says you travel too much.”

Well…

Yeah. After spending the last few years at AirNZ Gold, I’ve now moved up a level to Elite. Too much travel in the last year, almost all of it in Economy. Four trips to the US, 2 trips to Europe, 2 trips to Australia, plus a few domestic trips. Too damn much.

I don’t think I’ll be able retain it beyond this year. Will have to make the most of it for my upcoming Asia + US trips. Elite Airpoints Dollar Upgrades look like the most useful thing, since the couple of free upgrades get used up pretty quickly. If only I could also use those upgrades on trips to Europe via Asia…

GCP, and Regaining Trust

Google is telling us they’re serious about the cloud. They’re hiring the right people, spending the big bucks, and even (gasp!) talking to customers! (Oh how that must stick in their craw). They have great technology, they’ve proved it out at scale, and the price is right.

There’s just one nagging doubt in the back of our minds. Is Google serious about this? Are they going to turn around one day and say “GCP is too hard to maintain, we’re dropping it. Besides, self-driving Segways are the future.”

Fool me once…

Because they have form in this. I present Exhibit A, Google Reader. Yes, that old saw. Yes, yes I am still bitter. No, I won’t let it go.

I used Google Reader daily. I loved it. It came from a pre-Twitter, pre-Facebook time. A time when we used to have to visit a list of sites to keep up with things. We’d have to remember to check our friend’s travel blog every few weeks, just in case there was a new post. Sure, we used Slashdot as an aggregator, but everyone knows that’s been dead/dying since Rob Malda sold out to the man. (Has Netcraft has Continue reading

Networking’s not so bad

Ivan’s post this week was a good reminder that other parts of IT aren’t perfect either. It’s not all roses on the other side of the fence. Networking has done many good things, and often showed the way.

Consider a conversation between a sysadmin & a network engineer:

Look at how I can virtualise these systems! Now I can isolate users and consolidate hardware resources. They have no idea they’re on the same hardware. It’s incredible!

Oh. Bit like these VLANs, VRFs, and VDCs we’ve been doing for 15+ years now?

Look at how I can use Puppet to define this server’s complete configuration using a single text file! This is amazing! I can use version control for my infrastructure!

Oh. You mean like this single text file that defines the configuration of my network device here? Yes, yes that does seem useful.

Why do you networking people have so many different ways of configuring systems? Why don’t you just have one common API?

Oh. You mean like the way that there’s a Universal install script Linux systems?

SNMP sucks. The data format is terrible, implementations are inconsistent. Why don’t you switch to gRPC?

Wait, weren’t you telling me last Continue reading

War Stories: Backup NICs, DNS and AD

A return to our sporadic series of networking war stories. This time it’s fun with dedicated backup networks, DNS auto-registration, and Active Directory. Thank God it’s a lot easier these days with virtualisation. But back then…

Backups suck, but you need to do them somehow

Back in the olden days we had a dedicated tape drive connected to each server. Daily/weekly backups were written to the local tape drive using a SCSI connection. Someone would walk around the servers each day and change the tapes. It was simple, and it worked, but it doesn’t scale.

Two things happened – server numbers started exploding, and Gigabit Ethernet became practical. That meant that it became practical to have centralised ‘backup’ servers connected to tape drives, and to stream backup data across the network. Much better scale – we only needed to install an agent on each server, and the centralised backup servers needed to have enough tapes + tape drives. This also gave us much better central control & visibility of our backups.

Of course, we were worried about the impact of streaming large backup files across the network. We didn’t want that to affect production traffic, so we installed dedicated backup Continue reading

Efficiency vs Effectiveness

I’ve been wondering about how we’re approaching networking change. We know we need to make things better. Are we changing the ‘right’ things? I’ve got a feeling that we’re not, but I suspect that we’re too constrained by higher-order systems.

Simon Wardley wrote a great post on Efficiency vs Effectiveness. He gave a slightly contrived example of an organisation that is optimising the wrong thing. They plan on using robotics to automate server modifications to fit their custom racks. The problem is that they miss the point altogether. Yes, they’re optimising their flow. But they should ask: Is this the right flow?

Cheques: Apparently people still use them?

Recently I came across the “Wells Fargo Mobile Deposit” application. It sounds good – a faster way to deposit cheques(checks):

Mobile Deposit is secure, easy to use, and convenient.

• Deposit checks directly into your eligible account using your Android or Apple® mobile device or your Windows Phone.
• Take photos of the front and back of your check and submit. It’s that easy.
• Get confirmation on your device and by email for each successful deposit.
• Save time with fewer trips to an ATM or store.

Except…did anyone tell them that cheques Continue reading

Configure the Brocade NOS REST API to use HTTPS

Brocade VDX switches have REST and NETCONF interfaces. The REST API uses the built-in HTTP server. By default, this uses plain-text HTTP. As of NOS 6.0, you can (and should!) use HTTPS. If NOS has a certificate configured, it will automatically use HTTPS. Here’s how to configure it.

Pre-Change Tests

Let’s just do a couple of quick checks before we begin. Check that the switch is only listening on port 80, and that it responds to simple API queries:

Lindsays-MacBook:~ lhill$ nmap -p80,443 10.254.4.125

Starting Nmap 7.00 ( https://nmap.org ) at 2016-02-05 18:56 NZDT
Nmap scan report for 10.254.4.125
Host is up (0.14s latency).
PORT STATE SERVICE
80/tcp open http
443/tcp closed https

Nmap done: 1 IP address (1 host up) scanned in 0.52 seconds

Lindsays-MacBook:~ lhill$ curl -u admin:password -d "<activate-status></activate-status>" http://10.254.4.125/rest/operational-state/activate-status
<output xmlns='urn:brocade.com:mgmt:brocade-firmware'>
<overall-status>0</overall-status>
<activate-entries>
<rbridge-id>1</rbridge-id>
<status>0</status>
</activate-entries>
</output>

Lindsays-MacBook:~ lhill$ ssh [email protected]
[email protected]'s password:
Welcome to the Brocade Network Operating System Software
admin connected from 10.252.131.4 using ssh on Leaf-203025
Leaf-203025# show http server status
rbridge-id 1: Status: HTTP Enabled and HTTPS  Continue reading

Help! My Boss is Scared of Automation!!!

A reader asked “What can I do if my boss won’t let me automate my tasks?” Sadly some people still have a fear of automating even common, well-understood tasks. They’re worried about automation run amok. They think it’s safer to have a human typing in commands. But you know better. Humans have a place. But that place is not executing the same sequence of steps, over and over.

You need to prepare for change. Continuing to do repetitive tasks manually does not have a future. Either your boss will have a change of heart, or you’re going to change jobs. You have to prepare yourself for either eventuality. Here’s some thoughts on what to do.

Just Do It

First option: Just do it. Don’t bother asking, just get on with automating things you do often. You should be doing this anyway.

Last year we heard the story of a Russian hacker that had taken automation a little further than usual, with gems such as:

• kumar-asshole.sh – scans the inbox for emails from “Kumar” (a DBA at our clients). Looks for keywords like “help”, “trouble”, “sorry” etc. If keywords are found – the script SSHes into the clients server and rolls back Continue reading

Learning to Love Codenames

One of the things I struggled with when starting at a vendor was dealing with project codenames. There is no secret decoder ring – you have to learn the names the hard way. I couldn’t understand why descriptive names weren’t used. It took a while, but I’ve come to understand the reasoning behind the obscure names now. It’s still a stretch to say I ‘love’ them, but I can at least understand them now.

Naming Standards & Bikeshedding

When I started my professional career, it was common to name servers using things like Greek & Roman Gods, or Star Wars characters. Billing might run on Apollo, while Medusa was used for third-party connections.

This is fine for 5-10 servers, but clearly doesn’t scale. I’ve wasted many long and pointless hours in server naming “bikeshedding” discussions. Grumpy old sysadmins would argue that it was far easier to remember names like Bert & Ernie than web01/web02. The Young Turks saw that as a way of hoarding knowledge. It seemed to deliberately make it more difficult for newcomers/outsiders. They preferred descriptive names that gave some indication of what the system was doing, where it was located, etc.

Arguments went back and forth, then virtualisation came Continue reading

Modifying Packet Captures with tcprewrite

Recently I wanted to look at the structure of sFlow packets. Of course I can read the specs, but it’s often easier to look at some real packets. So I set up a simple network, configured sFlow, created some traffic across the network, and used tcpdump to capture the sFlow packets.

Unfortunately I had a bit of a brain fade, and configured sFlow to use port 2055, not port 6343. So it looked like this:

vagrant@ubuntu:~$ tcpdump -r sflow.cap
reading from file sflow.cap, link-type EN10MB (Ethernet)
13:48:37.812602 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 148
13:48:57.813663 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 148
13:48:59.061629 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 232
13:49:17.806908 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 148
13:49:37.804433 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 148
13:49:57.806000 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 148
13:50:17.808959 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP,  Continue reading

NZ IPv6 & DNSSEC Update

A year ago I published a table of New Zealand ISP IPv6 support. At the time support was fairly poor. I’m pleased to report that things have gotten better over the last year. There has also been a very pleasing uptick in DNSSEC support.

IPv6 Changes

The big movers here are Trustpower & Orcon, who have both enabled IPv6 by default for their users. So now we have the two largest ISPs still only offering IPv4, but all of the next tier of ISPs are offering IPv6. New Zealand has a flexible ISP market, and almost all consumers can change provider quickly & easily. This means that IPv6 is effectively available for all who want it.

New Zealand IPv6 Availability – Click image to see APNIC data

The numbers are still small, but we can see a move upwards towards the end of the year when Orcon & Trustpower enabled IPv6. Many legacy home routers have IPv6 disabled, but as these get replaced/reconfigured, I expect to see a steady increase in IPv6 uptake across those ISPs.

The two market leaders – Spark & Vodafone still only offer broken promises. In 2014 Vodafone implied it was not far away: “I can Continue reading

Brocade VDX SNMP Changes

Brocade tightened up some SNMP settings with NOS 6.0.x. This improves security, but it also means that you will need to modify your configuration if you upgrade. If you don’t, SNMP won’t work, and you’ll get errors with BNA/Nagios/Cacti/etc. Here’s the changes, and how to get SNMP working with NOS 6.0.x. NB This applies to VDX Data Centre switches. Other product lines have different configuration.

5.x and earlier defaults

NOS 5.x and earlier had default SNMP settings that looked like this:

snmp-server contact "Field Support."
snmp-server location "End User Premise."
snmp-server sys-descr "Brocade VDX Switch."
snmp-server community ConvergedNetwork
snmp-server community OrigEquipMfr rw
snmp-server community "Secret C0de" rw
snmp-server community common
snmp-server community private rw
snmp-server community public
snmp-server user snmpadmin1 groupname snmpadmin
snmp-server user snmpadmin2 groupname snmpadmin
snmp-server user snmpadmin3 groupname snmpadmin
snmp-server user snmpuser1
snmp-server user snmpuser2
snmp-server user snmpuser3

Yeah. Pretty open. So if you’re lazy, and your NMS tried a default discovery string of Continue reading

Using InfluxDB + Grafana to Display Network Statistics

I loathe MRTG graphs. They were cool in 2000, but now they’re showing their age. We have much better visualisation tools available, and we don’t need to be so aggressive with aggregating old data. I’ve been working with InfluxDB + Grafana recently. Much cooler, much more flexible. Here’s a walk-through on setting up InfluxDB + Grafana, collecting network throughput data, and displaying it.

Background – InfluxDB + Grafana

There’s three parts to this:

• Grafana: This is our main UI. Grafana is a “…graph and dashboard builder for visualizing time series metrics.” It makes it easy to create dashboards for displaying time-series data. It works with several different data sources such as Graphite, Elasticsearch, InfluxDB, and OpenTSDB.
• InfluxDB: This is where we store the data that Grafana displays. InfluxDB is “…an open-source distributed time series database with no external dependencies.” It’s a relatively new project, and is not quite at 1.0 yet, but it shows a lot of promise. It can be used in place of Graphite. It is very flexible, and can store events as well as time series data.
• Influxsnmp: We need to get data from the network into InfluxDB. There are a few options for Continue reading

Sit Stand Desk Setup

I work from home these days. Therefore it’s important that I have a decent desk setup. My previous setup was pretty crappy, but I only worked from home part-time. I’ve been using a standing desk at home, and wanted to move to a sit/stand model for full-time use. Here’s what I did.

Desk & Monitor Arrangement

I bought the Cubit Highrise desk, with a 1200mm x 700mm surface. This is a New Zealand-made manual height-adjustable desk. The adjustable legs allow for the height to be set anywhere between 660 and 1060mm. I paid $660NZD including shipping, from Total Office. That was the best deal at the time.

I added a Fleximounts L02 monitor stand. This is a desk-mounted monitor stand, with two gas spring arms. One arm has a tray for my MBPr laptop, the other has an LG IPS236 23″ monitor. It cost me $134USD including shipping. It’s in USD because I picked it up on one of my recent trips to San Jose.

I also use a wireless Apple keyboard and an Apple Magic Trackpad.

How’s it working out?

I’ve been very happy. My previous setup was a crappy desk with a platform added to get it to standing height. That Continue reading

Brocade BNA API

Brocade Network Advisor (BNA) has a REST API for accessing Fibre Channel-related data. The documentation includes a sample Python script showing how to connect to the API to retrieve Fabric info. The script given only works with Python 3.x. It’s also a pain to copy out of the documentation as you end up with a few extra characters in there. Here’s a version that will work with Python 2.7. I’ve also made a few other modifications – in this one, you can set the BNA IP, Username & Password at the top of the script.  I’ve also made it PEP8-compliant.

#!/usr/bin/env python

import httplib
import json
import sys

BNAServer = "10.200.5.181"
BNAUsername = "Administrator"
BNAPassword = "password"

# Create HTTPConnection object and connect to the server.
connection = httplib.HTTPConnection(BNAServer)

###########################
# Log in to Network Advisor
###########################

# Send login request
connection.request(
    'POST',
    '/rest/login',
    headers={
        "WSUsername": BNAUsername,
        "WSPassword": BNAPassword,
        "Accept": "application/vnd.brocade.networkadvisor+json;version=v1"}
    )

print()
print("Sending login request to Network Advisor...")

# Get the response
response = connection.getresponse()
# Display the response status print()
print ("Status= ", response.status)
# If successful (status = 200), display the returned session token
if response.status  Continue reading

Closing out Projects

We put a lot of energy into new projects. We argue about the design, we plan the cutover, we execute it…and then we move on. But decommissioning the old system is critical part of any project. It’s not over until you’ve switched off the old system.

Years ago I was involved in the buildout of a new network. The new network was a thing of beauty. A clear design, the best equipment, redundant everything. It was replacing a legacy network, one that had grown organically.

The new network was built out. Late one night the key services were cut over, and things were looking good. Everyone was happy, and we had a big party to celebrate. The project group disbanded, and everyone moved on to other things. Since the project was closed out, funding & resources stopped. Success, right?

Except…the old equipment was still running. A handful of applications were left on the old network. Some annoying services used undocumented links between the networks. Even worse, disused WAN links were still in place, and still being billed for.

The problem was that the project was officially ‘over.’ Who’s responsible for finishing off that last bit of cleanup?

I’ve seen similar things in Continue reading