Configure the Brocade NOS REST API to use HTTPS

Brocade VDX switches have REST and NETCONF interfaces. The REST API uses the built-in HTTP server. By default, this uses plain-text HTTP. As of NOS 6.0, you can (and should!) use HTTPS. If NOS has a certificate configured, it will automatically use HTTPS. Here’s how to configure it.

Pre-Change Tests

Let’s just do a couple of quick checks before we begin. Check that the switch is only listening on port 80, and that it responds to simple API queries:

Lindsays-MacBook:~ lhill$ nmap -p80,443 10.254.4.125

Starting Nmap 7.00 ( https://nmap.org ) at 2016-02-05 18:56 NZDT
Nmap scan report for 10.254.4.125
Host is up (0.14s latency).
PORT STATE SERVICE
80/tcp open http
443/tcp closed https

Nmap done: 1 IP address (1 host up) scanned in 0.52 seconds

Lindsays-MacBook:~ lhill$ curl -u admin:password -d "<activate-status></activate-status>" http://10.254.4.125/rest/operational-state/activate-status
<output xmlns='urn:brocade.com:mgmt:brocade-firmware'>
<overall-status>0</overall-status>
<activate-entries>
<rbridge-id>1</rbridge-id>
<status>0</status>
</activate-entries>
</output>

Lindsays-MacBook:~ lhill$ ssh [email protected]
[email protected]'s password:
Welcome to the Brocade Network Operating System Software
admin connected from 10.252.131.4 using ssh on Leaf-203025
Leaf-203025# show http server status
rbridge-id 1: Status: HTTP Enabled and HTTPS  Continue reading

Help! My Boss is Scared of Automation!!!

A reader asked “What can I do if my boss won’t let me automate my tasks?” Sadly some people still have a fear of automating even common, well-understood tasks. They’re worried about automation run amok. They think it’s safer to have a human typing in commands. But you know better. Humans have a place. But that place is not executing the same sequence of steps, over and over.

You need to prepare for change. Continuing to do repetitive tasks manually does not have a future. Either your boss will have a change of heart, or you’re going to change jobs. You have to prepare yourself for either eventuality. Here’s some thoughts on what to do.

Just Do It

First option: Just do it. Don’t bother asking, just get on with automating things you do often. You should be doing this anyway.

Last year we heard the story of a Russian hacker that had taken automation a little further than usual, with gems such as:

• kumar-asshole.sh – scans the inbox for emails from “Kumar” (a DBA at our clients). Looks for keywords like “help”, “trouble”, “sorry” etc. If keywords are found – the script SSHes into the clients server and rolls back Continue reading

Learning to Love Codenames

One of the things I struggled with when starting at a vendor was dealing with project codenames. There is no secret decoder ring – you have to learn the names the hard way. I couldn’t understand why descriptive names weren’t used. It took a while, but I’ve come to understand the reasoning behind the obscure names now. It’s still a stretch to say I ‘love’ them, but I can at least understand them now.

Naming Standards & Bikeshedding

When I started my professional career, it was common to name servers using things like Greek & Roman Gods, or Star Wars characters. Billing might run on Apollo, while Medusa was used for third-party connections.

This is fine for 5-10 servers, but clearly doesn’t scale. I’ve wasted many long and pointless hours in server naming “bikeshedding” discussions. Grumpy old sysadmins would argue that it was far easier to remember names like Bert & Ernie than web01/web02. The Young Turks saw that as a way of hoarding knowledge. It seemed to deliberately make it more difficult for newcomers/outsiders. They preferred descriptive names that gave some indication of what the system was doing, where it was located, etc.

Arguments went back and forth, then virtualisation came Continue reading

Modifying Packet Captures with tcprewrite

Recently I wanted to look at the structure of sFlow packets. Of course I can read the specs, but it’s often easier to look at some real packets. So I set up a simple network, configured sFlow, created some traffic across the network, and used tcpdump to capture the sFlow packets.

Unfortunately I had a bit of a brain fade, and configured sFlow to use port 2055, not port 6343. So it looked like this:

vagrant@ubuntu:~$ tcpdump -r sflow.cap
reading from file sflow.cap, link-type EN10MB (Ethernet)
13:48:37.812602 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 148
13:48:57.813663 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 148
13:48:59.061629 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 232
13:49:17.806908 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 148
13:49:37.804433 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 148
13:49:57.806000 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP, length 148
13:50:17.808959 IP 10.254.4.125.44695 > 10.254.4.170.2055: UDP,  Continue reading

NZ IPv6 & DNSSEC Update

A year ago I published a table of New Zealand ISP IPv6 support. At the time support was fairly poor. I’m pleased to report that things have gotten better over the last year. There has also been a very pleasing uptick in DNSSEC support.

IPv6 Changes

The big movers here are Trustpower & Orcon, who have both enabled IPv6 by default for their users. So now we have the two largest ISPs still only offering IPv4, but all of the next tier of ISPs are offering IPv6. New Zealand has a flexible ISP market, and almost all consumers can change provider quickly & easily. This means that IPv6 is effectively available for all who want it.

New Zealand IPv6 Availability – Click image to see APNIC data

The numbers are still small, but we can see a move upwards towards the end of the year when Orcon & Trustpower enabled IPv6. Many legacy home routers have IPv6 disabled, but as these get replaced/reconfigured, I expect to see a steady increase in IPv6 uptake across those ISPs.

The two market leaders – Spark & Vodafone still only offer broken promises. In 2014 Vodafone implied it was not far away: “I can Continue reading

Brocade VDX SNMP Changes

Brocade tightened up some SNMP settings with NOS 6.0.x. This improves security, but it also means that you will need to modify your configuration if you upgrade. If you don’t, SNMP won’t work, and you’ll get errors with BNA/Nagios/Cacti/etc. Here’s the changes, and how to get SNMP working with NOS 6.0.x. NB This applies to VDX Data Centre switches. Other product lines have different configuration.

5.x and earlier defaults

NOS 5.x and earlier had default SNMP settings that looked like this:

snmp-server contact "Field Support."
snmp-server location "End User Premise."
snmp-server sys-descr "Brocade VDX Switch."
snmp-server community ConvergedNetwork
snmp-server community OrigEquipMfr rw
snmp-server community "Secret C0de" rw
snmp-server community common
snmp-server community private rw
snmp-server community public
snmp-server user snmpadmin1 groupname snmpadmin
snmp-server user snmpadmin2 groupname snmpadmin
snmp-server user snmpadmin3 groupname snmpadmin
snmp-server user snmpuser1
snmp-server user snmpuser2
snmp-server user snmpuser3

Yeah. Pretty open. So if you’re lazy, and your NMS tried a default discovery string of Continue reading

Using InfluxDB + Grafana to Display Network Statistics

I loathe MRTG graphs. They were cool in 2000, but now they’re showing their age. We have much better visualisation tools available, and we don’t need to be so aggressive with aggregating old data. I’ve been working with InfluxDB + Grafana recently. Much cooler, much more flexible. Here’s a walk-through on setting up InfluxDB + Grafana, collecting network throughput data, and displaying it.

Background – InfluxDB + Grafana

There’s three parts to this:

• Grafana: This is our main UI. Grafana is a “…graph and dashboard builder for visualizing time series metrics.” It makes it easy to create dashboards for displaying time-series data. It works with several different data sources such as Graphite, Elasticsearch, InfluxDB, and OpenTSDB.
• InfluxDB: This is where we store the data that Grafana displays. InfluxDB is “…an open-source distributed time series database with no external dependencies.” It’s a relatively new project, and is not quite at 1.0 yet, but it shows a lot of promise. It can be used in place of Graphite. It is very flexible, and can store events as well as time series data.
• Influxsnmp: We need to get data from the network into InfluxDB. There are a few options for Continue reading

Sit Stand Desk Setup

I work from home these days. Therefore it’s important that I have a decent desk setup. My previous setup was pretty crappy, but I only worked from home part-time. I’ve been using a standing desk at home, and wanted to move to a sit/stand model for full-time use. Here’s what I did.

Desk & Monitor Arrangement

I bought the Cubit Highrise desk, with a 1200mm x 700mm surface. This is a New Zealand-made manual height-adjustable desk. The adjustable legs allow for the height to be set anywhere between 660 and 1060mm. I paid $660NZD including shipping, from Total Office. That was the best deal at the time.

I added a Fleximounts L02 monitor stand. This is a desk-mounted monitor stand, with two gas spring arms. One arm has a tray for my MBPr laptop, the other has an LG IPS236 23″ monitor. It cost me $134USD including shipping. It’s in USD because I picked it up on one of my recent trips to San Jose.

I also use a wireless Apple keyboard and an Apple Magic Trackpad.

How’s it working out?

I’ve been very happy. My previous setup was a crappy desk with a platform added to get it to standing height. That Continue reading

Brocade BNA API

Brocade Network Advisor (BNA) has a REST API for accessing Fibre Channel-related data. The documentation includes a sample Python script showing how to connect to the API to retrieve Fabric info. The script given only works with Python 3.x. It’s also a pain to copy out of the documentation as you end up with a few extra characters in there. Here’s a version that will work with Python 2.7. I’ve also made a few other modifications – in this one, you can set the BNA IP, Username & Password at the top of the script.  I’ve also made it PEP8-compliant.

#!/usr/bin/env python

import httplib
import json
import sys

BNAServer = "10.200.5.181"
BNAUsername = "Administrator"
BNAPassword = "password"

# Create HTTPConnection object and connect to the server.
connection = httplib.HTTPConnection(BNAServer)

###########################
# Log in to Network Advisor
###########################

# Send login request
connection.request(
    'POST',
    '/rest/login',
    headers={
        "WSUsername": BNAUsername,
        "WSPassword": BNAPassword,
        "Accept": "application/vnd.brocade.networkadvisor+json;version=v1"}
    )

print()
print("Sending login request to Network Advisor...")

# Get the response
response = connection.getresponse()
# Display the response status print()
print ("Status= ", response.status)
# If successful (status = 200), display the returned session token
if response.status  Continue reading

Closing out Projects

We put a lot of energy into new projects. We argue about the design, we plan the cutover, we execute it…and then we move on. But decommissioning the old system is critical part of any project. It’s not over until you’ve switched off the old system.

Years ago I was involved in the buildout of a new network. The new network was a thing of beauty. A clear design, the best equipment, redundant everything. It was replacing a legacy network, one that had grown organically.

The new network was built out. Late one night the key services were cut over, and things were looking good. Everyone was happy, and we had a big party to celebrate. The project group disbanded, and everyone moved on to other things. Since the project was closed out, funding & resources stopped. Success, right?

Except…the old equipment was still running. A handful of applications were left on the old network. Some annoying services used undocumented links between the networks. Even worse, disused WAN links were still in place, and still being billed for.

The problem was that the project was officially ‘over.’ Who’s responsible for finishing off that last bit of cleanup?

I’ve seen similar things in Continue reading

IPv6-based Wi-Fi Hotspots

Apple’s 2015 WWDC event included a great session on IPv6 & TCP changes coming with iOS 9. There is a related post to the IETF v6ops mailing list here. The new IPv6 hotspot is very interesting to me. These are my notes on how hotspot functionality can work with IPv6, and no NAT.

IPv4 Hotspot – (aka the simplicity of NAT?)

The current IPv4 hotspots use simple NAT, similar to most home network setups. The mobile network assigns a public IPv4 /32 address to the handset, H. The handset picks a local RFC1918 address space for its connectivity to local clients, and hands that out via DHCP. Hide NAT is used to provide outbound internet connectivity for those clients.

What about IPv6? Isn’t NAT verboten?

NAT is evil, right? We can’t use NAT to hide the local clients behind the handset. So how do we provide IPv6 hotspot functionality? One way would be to use DHCPv6 PD. When the hotspot is enabled, the mobile device could request a prefix via DHCPv6 PD. That could then be used for local devices.

Unfortunately the Continue reading

Stretching the Container Metaphor

The Docker/shipping container metaphor is overdone. I don’t think people have fully thought through what it might mean if containers do the same thing to computing as they did to shipping. Are we prepared for hipsters taking over derelict data centers?

There is an unpublished rule that all Docker articles must be accompanied by a picture of shipping containers. Forbes is a particularly egregious offender. I don’t know if it’s the work of a serial offender sub-editor, or if it’s a company-wide policy. I suspect the latter.

Then there’s the DC2 Desktop Container Computer Kickstarter campaign:

(I must admit I do like this one)

But what happened when shipping converted to using containers? Consolidation of ports, dramatic reduction in required labour force, leading to waterfront dereliction. Years later cities re-discovered their waterfront spaces, leading to redevelopment & gentrification.

Wharfs went from this:

(Image from State Library of South Australia, CC license)

To this:

(Image by David Dixon, CC license)

To be re-born as this:

(Image from Wikimedia Commons, CC license)

So does that mean that our data centers will go from this:

(Image from Intel Free Press, CC license)

To this:

(Image from Wolfgang Stief, CC license)

To Continue reading

War Stories: ITIL Process vs Practice

Our irregular War Stories returns, with a story about a network I worked on with strict change control, but high technical debt. What should have been a simple fix became far more pain than it should have been. Lesson learned: next time just leave things alone. I’m sure the ITIL true believers loved their process, but did they realise it stopped people fixing problems?

A classic problem: Duplex mismatch

I spotted a duplex mismatch with one of the services I was responsible for. Throughput was low, and the NIC was showing late collisions. Classic mismatch. Should be an easy enough fix, right? Whoa there son. This is an ITIL shop. No changes without an approved change request!

Logging Changes: An Exercise in Frustration

Change policy at this company was for a lead time for one week for most systems, or two weeks for some ‘important’ systems. Changes had to be submitted and approved before the deadline. There was no reason for the delay. Nothing happened during those two weeks, there was no extra review, you just had to wait, because that was the process.

This company had a Change Management system built on top of a main-frame application. Seriously? Yes, seriously. But it was Continue reading

Brocade PyNOS Python Libraries

PyNOS v1.1 has been published. This is a python library that simplifies automating Brocade VDX systems. It is built on top of ncclient, and uses NETCONF to communicate with the VDX systems. Using the libraries is much simpler than writing your own NETCONF calls.

What can I do with it?

Use Python to script configuration or management tasks against VDX devices, e.g.:

• Configure interfaces & VLANs
• Find LLDP neighbors
• Find out which port a MAC is connected to
• Configure BGP
• Configure SNMP

You can also use Python as an interactive shell to run commands against multiple systems.

Examples:

Connect to device and check firmware version & uptime:

>>> import pynos.device
>>> conn = ('172.22.90.100', '22')
>>> auth = ('admin', 'password')
>>> dev=pynos.device.Device(conn=conn, auth=auth)
>>> dev.connection
True
>>> dev.firmware_version
'6.0.1'
>>> dev.system.uptime
{'seconds': '1', 'hours': '13', 'minutes': '0', 'days': '1'}
>>>

Change switchport description:

>>> with pynos.device.Device(conn=conn, auth=auth) as dev:
...     dev.interface.description(
...     int_type='tengigabitethernet', name='225/0/38',
...     desc=’RTR1 Ethernet1’)

Who should use it?

Any Brocade VDX customers that want to automate network configuration – e.g. to integrate with their provisioning systems.

It’s helpful to have Continue reading

TruView Live Application Monitoring

Fluke Networks recently released TruView Live, a subscription-based service for monitoring internal & external applications. Tests can run from Fluke-managed cloud locations, your own systems, or from dedicated hardware appliances. I’ve been testing it out, and I like it so far.

Provisioning

Overall setup is pretty straightforward. Choose what you want to monitor, and how you want to monitor it – from AWS locations, from your own server, or from a dedicated hardware device.

Global Pulse

Global Pulses run on Fluke-managed AWS instances. You just pick the Global Locations you want to run from, and assign tests as needed.

Go to Administration -> Pulses -> Deploy Global Pulse. Select the locations you want, and click Deploy.

Virtual Pulse

A Virtual Pulse is an application running on Windows (7/8/2008/2012) or Linux systems (RHEL 7.0, Ubuntu 14.04). This does not need to be a dedicated device – e.g. You might need Continue reading

Unsupported BNA Hacks

Here’s a couple of quick hacks for working with Brocade Network Advisor. It’s unsupported, but you can run BNA on Ubuntu. You can also suppress the client-side JRE version mismatch warning.

Ubuntu Install

If you try to install BNA on Ubuntu, it fails during the DB initialization & setup phase. There are two reasons for this:

• gawk is not where the installer thinks it should be
• Some scripts run as “/bin/sh”, but use bashisms.

Before running the installation, make these two changes:

• Run “sudo ln -s /usr/bin/gawk /bin/gawk”
• Run “sudo dpkg-reconfigure dash” and select “No”

After that the DB setup will complete. Leaving the gawk symlink in place won’t hurt anything else. You can probably change the system shell back to dash, but you may run into problems if you run any of the BNA utility scripts.

Client-side JRE check

When you launch the BNA Desktop client, it checks your local JRE version against a list of supported versions. It’s Continue reading

Security – Just Another Risk

I made a conscious decision to move away from full-time information security work. I retain an interest, and try to keep up with developments, but I don’t want to be “the security guy.” There are several reasons for it, but a large part is due to the hype, the bullshit, and general inability for the security industry to act like grown-ups.

The most frustrating part was the inability to properly classify risk. Robert Graham put this eloquently here:

Infosec isn’t a real profession. Among the things missing is proper “risk analysis”. Instead of quantifying risk, we treat it as an absolute. Risk is binary, either there is risk or there isn’t. We respond to risk emotionally rather than rationally, claiming all risk needs to be removed. This is why nobody listens to us. Business leaders quantify and prioritize risk, but we don’t, so our useless advice is ignored.

Security folk often forget that they are just another risk. Yes, it’s a risk shipping the product with that bug. But not shipping at all might be a larger risk to the business. Even complete data breach may or may not be catastrophic to the business – RSA is still Continue reading

/bin/sh – checking for bash vs dash incompatibilities

I have been investigating a problem where an application would install on RHEL/CentOS, but not on Ubuntu. I tracked it down to a problem with shell scripts that assumed that /bin/sh was bash. Ubuntu uses dash by default, so some ‘bashisms‘ don’t work. This will be old news to Ubuntu types that migrated to dash a while back, but I normally use CentOS/RHEL systems, and/or well-behaved cross-platform scripts. Luckily ‘checkbashisms‘ can help with figuring out what changes are needed.

I don’t want to go into the history of Unix shells, but there are probably more shell variants than there are *nix variants. Some are very different, and completely incompatible. But others are only different in subtle ways, and most things works without modification. If your script explicitly calls the required shell with “#!/bin/zsh” or “#!/bin/csh”, all will be fine. The problem comes when your script starts with “#!/bin/sh”. That will call the system shell, which can vary across different systems. If you’re using that, your script should be portable, and only implement a subset of possible functionality. People get in the habit of using “/bin/sh”,  but using shell-specific features. That’s when things get ugly when you run Continue reading

Add Brocade MLX & VDX Support to HP IMC

HP IMC 7.1 E0303P13 does not support configuration backups for Brocade MLX & VDX devices. But they do have an extensible model, so it’s easy to add support. Here’s how to do it, and how to fix the Brocade ICX support.

Here’s the steps to add support for MLX & VDX devices to HP IMC:

1. Download the current set of adapters from GitHub.
2. Unpack the zip file, and copy the adapters into place.
3. Add Device Series & Device Model definitions.
4. Restart IMC, re-synchronise, and check file transfer modes.

Going into a bit more detail:

NetOps Custom Adapters

This GitHub repository maintains a set of 3rd-party developed adapters for HP IMC. You can download individual files, create a local copy of the repo using Git, or just download a zip file containing all current scripts from here.

On the IMC server, adapters are stored at (IMC)/server/conf/adapters/ICC. You’ll see directories for all supported vendors there:

[root@imc ~]# cd /opt/iMC/server/conf/adapters/ICC
[root@imc ICC]# ls
3Com    Alcatel-Lucent  Aruba Networks  Avocent  Cabletron  Dell  Enterasys         F5       Fortigate  H3C              Hillstone  IBM                Continue reading

IPv6-test.com and SRX firewall policies

ipv6-test.com is a useful site for testing IPv4 & IPv6 connectivity. It checks that v4 & v6 are working as expected, and reports your browser v4/v6 preferences. It does have one oddity with ICMPv6 tests. Here’s what I did to work around it with my SRX setup.

The site runs a suite of tests and gives you a score out of 20. Most dual-stack home users will probably get 17/20. They deduct 1 point for no reverse DNS entry for v6, and 2 points for “ICMP Filtered”

How can you improve your score ?

1. Reconfigure your firewall
Your router or firewall is filtering ICMPv6 messages sent to your computer. An IPv6 host that cannot receive ICMP messages may encounter problems like some web pages loading partially or not at all.

2. Get a reverse DNS record

The first one is fine, but the second issue is a worry. ICMP is a critical part of IPv6. It’s needed for things like Neighbor Discovery, and Packet Too Big messages.

Most home user firewall setups will be fairly simple. Basically ‘Allow everything out, and allow related traffic back in. Drop everything else.’ Surely the default policy on the SRX should be allowing related Continue reading