Closing out Projects

We put a lot of energy into new projects. We argue about the design, we plan the cutover, we execute it…and then we move on. But decommissioning the old system is critical part of any project. It’s not over until you’ve switched off the old system.

Years ago I was involved in the buildout of a new network. The new network was a thing of beauty. A clear design, the best equipment, redundant everything. It was replacing a legacy network, one that had grown organically.

The new network was built out. Late one night the key services were cut over, and things were looking good. Everyone was happy, and we had a big party to celebrate. The project group disbanded, and everyone moved on to other things. Since the project was closed out, funding & resources stopped. Success, right?

Except…the old equipment was still running. A handful of applications were left on the old network. Some annoying services used undocumented links between the networks. Even worse, disused WAN links were still in place, and still being billed for.

The problem was that the project was officially ‘over.’ Who’s responsible for finishing off that last bit of cleanup?

I’ve seen similar things in Continue reading

IPv6-based Wi-Fi Hotspots

Apple’s 2015 WWDC event included a great session on IPv6 & TCP changes coming with iOS 9. There is a related post to the IETF v6ops mailing list here. The new IPv6 hotspot is very interesting to me. These are my notes on how hotspot functionality can work with IPv6, and no NAT.

IPv4 Hotspot – (aka the simplicity of NAT?)

The current IPv4 hotspots use simple NAT, similar to most home network setups. The mobile network assigns a public IPv4 /32 address to the handset, H. The handset picks a local RFC1918 address space for its connectivity to local clients, and hands that out via DHCP. Hide NAT is used to provide outbound internet connectivity for those clients.

What about IPv6? Isn’t NAT verboten?

NAT is evil, right? We can’t use NAT to hide the local clients behind the handset. So how do we provide IPv6 hotspot functionality? One way would be to use DHCPv6 PD. When the hotspot is enabled, the mobile device could request a prefix via DHCPv6 PD. That could then be used for local devices.

Unfortunately the Continue reading

Stretching the Container Metaphor

The Docker/shipping container metaphor is overdone. I don’t think people have fully thought through what it might mean if containers do the same thing to computing as they did to shipping. Are we prepared for hipsters taking over derelict data centers?

There is an unpublished rule that all Docker articles must be accompanied by a picture of shipping containers. Forbes is a particularly egregious offender. I don’t know if it’s the work of a serial offender sub-editor, or if it’s a company-wide policy. I suspect the latter.

Then there’s the DC2 Desktop Container Computer Kickstarter campaign:

(I must admit I do like this one)

But what happened when shipping converted to using containers? Consolidation of ports, dramatic reduction in required labour force, leading to waterfront dereliction. Years later cities re-discovered their waterfront spaces, leading to redevelopment & gentrification.

Wharfs went from this:

(Image from State Library of South Australia, CC license)

To this:

(Image by David Dixon, CC license)

To be re-born as this:

(Image from Wikimedia Commons, CC license)

So does that mean that our data centers will go from this:

(Image from Intel Free Press, CC license)

To this:

(Image from Wolfgang Stief, CC license)

To Continue reading

War Stories: ITIL Process vs Practice

Our irregular War Stories returns, with a story about a network I worked on with strict change control, but high technical debt. What should have been a simple fix became far more pain than it should have been. Lesson learned: next time just leave things alone. I’m sure the ITIL true believers loved their process, but did they realise it stopped people fixing problems?

A classic problem: Duplex mismatch

I spotted a duplex mismatch with one of the services I was responsible for. Throughput was low, and the NIC was showing late collisions. Classic mismatch. Should be an easy enough fix, right? Whoa there son. This is an ITIL shop. No changes without an approved change request!

Logging Changes: An Exercise in Frustration

Change policy at this company was for a lead time for one week for most systems, or two weeks for some ‘important’ systems. Changes had to be submitted and approved before the deadline. There was no reason for the delay. Nothing happened during those two weeks, there was no extra review, you just had to wait, because that was the process.

This company had a Change Management system built on top of a main-frame application. Seriously? Yes, seriously. But it was Continue reading

Brocade PyNOS Python Libraries

PyNOS v1.1 has been published. This is a python library that simplifies automating Brocade VDX systems. It is built on top of ncclient, and uses NETCONF to communicate with the VDX systems. Using the libraries is much simpler than writing your own NETCONF calls.

What can I do with it?

Use Python to script configuration or management tasks against VDX devices, e.g.:

• Configure interfaces & VLANs
• Find LLDP neighbors
• Find out which port a MAC is connected to
• Configure BGP
• Configure SNMP

You can also use Python as an interactive shell to run commands against multiple systems.

Examples:

Connect to device and check firmware version & uptime:

>>> import pynos.device
>>> conn = ('172.22.90.100', '22')
>>> auth = ('admin', 'password')
>>> dev=pynos.device.Device(conn=conn, auth=auth)
>>> dev.connection
True
>>> dev.firmware_version
'6.0.1'
>>> dev.system.uptime
{'seconds': '1', 'hours': '13', 'minutes': '0', 'days': '1'}
>>>

Change switchport description:

>>> with pynos.device.Device(conn=conn, auth=auth) as dev:
...     dev.interface.description(
...     int_type='tengigabitethernet', name='225/0/38',
...     desc=’RTR1 Ethernet1’)

Who should use it?

Any Brocade VDX customers that want to automate network configuration – e.g. to integrate with their provisioning systems.

It’s helpful to have Continue reading

TruView Live Application Monitoring

Fluke Networks recently released TruView Live, a subscription-based service for monitoring internal & external applications. Tests can run from Fluke-managed cloud locations, your own systems, or from dedicated hardware appliances. I’ve been testing it out, and I like it so far.

Provisioning

Overall setup is pretty straightforward. Choose what you want to monitor, and how you want to monitor it – from AWS locations, from your own server, or from a dedicated hardware device.

Global Pulse

Global Pulses run on Fluke-managed AWS instances. You just pick the Global Locations you want to run from, and assign tests as needed.

Go to Administration -> Pulses -> Deploy Global Pulse. Select the locations you want, and click Deploy.

Virtual Pulse

A Virtual Pulse is an application running on Windows (7/8/2008/2012) or Linux systems (RHEL 7.0, Ubuntu 14.04). This does not need to be a dedicated device – e.g. You might need Continue reading

Unsupported BNA Hacks

Here’s a couple of quick hacks for working with Brocade Network Advisor. It’s unsupported, but you can run BNA on Ubuntu. You can also suppress the client-side JRE version mismatch warning.

Ubuntu Install

If you try to install BNA on Ubuntu, it fails during the DB initialization & setup phase. There are two reasons for this:

• gawk is not where the installer thinks it should be
• Some scripts run as “/bin/sh”, but use bashisms.

Before running the installation, make these two changes:

• Run “sudo ln -s /usr/bin/gawk /bin/gawk”
• Run “sudo dpkg-reconfigure dash” and select “No”

After that the DB setup will complete. Leaving the gawk symlink in place won’t hurt anything else. You can probably change the system shell back to dash, but you may run into problems if you run any of the BNA utility scripts.

Client-side JRE check

When you launch the BNA Desktop client, it checks your local JRE version against a list of supported versions. It’s Continue reading

Security – Just Another Risk

I made a conscious decision to move away from full-time information security work. I retain an interest, and try to keep up with developments, but I don’t want to be “the security guy.” There are several reasons for it, but a large part is due to the hype, the bullshit, and general inability for the security industry to act like grown-ups.

The most frustrating part was the inability to properly classify risk. Robert Graham put this eloquently here:

Infosec isn’t a real profession. Among the things missing is proper “risk analysis”. Instead of quantifying risk, we treat it as an absolute. Risk is binary, either there is risk or there isn’t. We respond to risk emotionally rather than rationally, claiming all risk needs to be removed. This is why nobody listens to us. Business leaders quantify and prioritize risk, but we don’t, so our useless advice is ignored.

Security folk often forget that they are just another risk. Yes, it’s a risk shipping the product with that bug. But not shipping at all might be a larger risk to the business. Even complete data breach may or may not be catastrophic to the business – RSA is still Continue reading

/bin/sh – checking for bash vs dash incompatibilities

I have been investigating a problem where an application would install on RHEL/CentOS, but not on Ubuntu. I tracked it down to a problem with shell scripts that assumed that /bin/sh was bash. Ubuntu uses dash by default, so some ‘bashisms‘ don’t work. This will be old news to Ubuntu types that migrated to dash a while back, but I normally use CentOS/RHEL systems, and/or well-behaved cross-platform scripts. Luckily ‘checkbashisms‘ can help with figuring out what changes are needed.

I don’t want to go into the history of Unix shells, but there are probably more shell variants than there are *nix variants. Some are very different, and completely incompatible. But others are only different in subtle ways, and most things works without modification. If your script explicitly calls the required shell with “#!/bin/zsh” or “#!/bin/csh”, all will be fine. The problem comes when your script starts with “#!/bin/sh”. That will call the system shell, which can vary across different systems. If you’re using that, your script should be portable, and only implement a subset of possible functionality. People get in the habit of using “/bin/sh”,  but using shell-specific features. That’s when things get ugly when you run Continue reading

Add Brocade MLX & VDX Support to HP IMC

HP IMC 7.1 E0303P13 does not support configuration backups for Brocade MLX & VDX devices. But they do have an extensible model, so it’s easy to add support. Here’s how to do it, and how to fix the Brocade ICX support.

Here’s the steps to add support for MLX & VDX devices to HP IMC:

1. Download the current set of adapters from GitHub.
2. Unpack the zip file, and copy the adapters into place.
3. Add Device Series & Device Model definitions.
4. Restart IMC, re-synchronise, and check file transfer modes.

Going into a bit more detail:

NetOps Custom Adapters

This GitHub repository maintains a set of 3rd-party developed adapters for HP IMC. You can download individual files, create a local copy of the repo using Git, or just download a zip file containing all current scripts from here.

On the IMC server, adapters are stored at (IMC)/server/conf/adapters/ICC. You’ll see directories for all supported vendors there:

[root@imc ~]# cd /opt/iMC/server/conf/adapters/ICC
[root@imc ICC]# ls
3Com    Alcatel-Lucent  Aruba Networks  Avocent  Cabletron  Dell  Enterasys         F5       Fortigate  H3C              Hillstone  IBM                Continue reading

IPv6-test.com and SRX firewall policies

ipv6-test.com is a useful site for testing IPv4 & IPv6 connectivity. It checks that v4 & v6 are working as expected, and reports your browser v4/v6 preferences. It does have one oddity with ICMPv6 tests. Here’s what I did to work around it with my SRX setup.

The site runs a suite of tests and gives you a score out of 20. Most dual-stack home users will probably get 17/20. They deduct 1 point for no reverse DNS entry for v6, and 2 points for “ICMP Filtered”

How can you improve your score ?

1. Reconfigure your firewall
Your router or firewall is filtering ICMPv6 messages sent to your computer. An IPv6 host that cannot receive ICMP messages may encounter problems like some web pages loading partially or not at all.

2. Get a reverse DNS record

The first one is fine, but the second issue is a worry. ICMP is a critical part of IPv6. It’s needed for things like Neighbor Discovery, and Packet Too Big messages.

Most home user firewall setups will be fairly simple. Basically ‘Allow everything out, and allow related traffic back in. Drop everything else.’ Surely the default policy on the SRX should be allowing related Continue reading

It’s 2015: “Supports IPv6″ should mean full support

It’s 2015. ARIN is finally out of IPv4 addresses, more than 20% of Google users in the US are using IPv6…and vendors are still doing a half-assed job with IPv6 support. I purchased a new TP-Link Wi-Fi router/modem recently, and it doesn’t fully support IPv6. It’s not good enough, and I will be returning it.

I purchased the Archer D5 “AC1200 Wireless Dual Band Gigabit ADSL2+ Modem Router.” The website blurb includes this:

IPv6 Supported. The next generation of Internet protocol, helping you to future-proof your network.

And the specifications page says: “IPv6 and IPv4 dual stack.”

I checked the documentation for how to configure IPv6. This FAQ walks through configuring IPv6 on several TP-Link devices. Note that it includes this line “…choose Connection type (Here we just set up PPPoE as an example, if you are not sure, please contact your IPv6 provider)”

In New Zealand, most ADSL services are delivered as PPPoA. The specifications page says this device supports PPPoA. My ISP provides native IPv6 via DHCPv6 PD. So everything should be good to go, right?

Not so much. The Archer D5 does indeed support PPPoA. It also supports IPv6 with DHCPv6 PD. But it Continue reading

Networking Pioneers, Settlers and Town Planners

Can we broadly separate Networking into Pioneers, Settlers, and Town Planners? I’ve been thinking about how to apply Simon Wardley’s PST model to networking. This leads to thinking about how we can encourage networking evolution. The model needs a lot of fleshing out, but I’m interested in what others think.

Pioneers, Settlers and Town Planners (PST)

Simon Wardley has written about “Pioneers, Settlers and Town Planners (PST)” in many places – e.g. here and here. It derives from Cringely’s Commandos, Infantry, Police model (see Chapter 12 of Accidental Empires). It provides a organisational structure, grouping areas based upon their current state of evolution. It recognises that no one operational model works for all parts of a business. You can’t say “We’re using Agile, or Lean, or Six Sigma” – you need to use the appropriate model for each area. Even “Bi-Modal IT” is too limiting, as the divide is too great.

(Image from blog.gardeviance.org, used under Creative Commons License)

Applying PST to networking

The model is a helpful way of thinking about the role of different groups across a business. It also helps us understand why teams need to evolve over time, Continue reading

Considering On-Call Pay

Let’s say you’ve been offered a new job. $70k base salary, with up to $20k per year extra for on-call duties. Great! $90k! That’s $10k more than my salary now! Sign me up!

Wait a minute. Not so fast.

Years ago I received some good advice: Treat your on-call payment as a separate item. Don’t consider it as part of your base salary. This is because you need to be paid properly both for doing your job, and for the inconvenience of being on-call.

Evaluate the base salary for what it is: Your salary for doing your day-to-day job. Ignoring the on-call part, did they offer you enough money for the role? Is it a good match for your experience?

Separately decide if the on-call payment is enough to justify being on-call. Does it represent a fair payment for the extra work?

Why does this matter? A few reasons:

• Being on-call has a big impact on your life, and you need to be properly paid for it.
• You still need to get paid properly for the job you do Monday-Friday.
• On-call payments will vary. Your job role could change. Maybe the team gets bigger, and you go on-call less often. Now you’re earning Continue reading

Brocade Certified vRouter Engineer

If you’ve visited the Brocade website recently, you’ve probably seen the “Free NFV Certification” banner. I signed up for this several months ago, but had put off completing the course. I had a little downtime recently prior to starting work at Brocade, so I completed this course & exam. Here’s my impressions.

What’s the Course/Exam About?

From the official documentation:

As a Brocade Certified vRouter Engineer, you must be able to demonstrate the ability to install, configure and troubleshoot features of Brocade Vyatta Network OS.

i.e. it’s primarily about the basics of Vyatta.

What’s Included?

Here’s what you get when you sign up:

1. A download link to the Brocade Vyatta 5400 vRouter image.
2. Access to the “Brocade Certified vRouter Engineer Course.”
3. Links to certification materials and communities.
4. Promo code for the BCVRE exam.

Note that you can run the Vyatta image on a local hypervisor, or if you prefer you can select it from the AWS marketplace. Personally I ran it on VMware Fusion on my laptop. It’s light on resource, so you can easily spin up several Continue reading

Think Bigger

I get frustrated by those who take a narrow view of technology, and progress in general. They see things in terms of where they are now, and where they were. But they struggle to see a bit further out. The Internet of Things is a good example of this.

I made the mistake of reading the comments on a recent El Reg article (I know, I know: Never read the comments). I came across this comment about the IoT:

…The innocent child asked “but why would the toaster need to talk to the ‘fridge?” The marketing gurus had no answer and a few years later the outfit went bankrupt. In all the time since, no one has been able to answer that question.

From there the comments devolved into a rather pointless discussion about milk, bread, spam on toast and Twitter. This is a fairly common theme on El Reg articles (along with “cloud has little appeal for 90% of SM server/computing requirements”, but that’s another issue).

I find it frustrating when people take a narrow, short-sighted view when looking at technology trends. We all see things from our own perspective, but it’s good to lift your head Continue reading

The Next Step: Brocade

I am happy to announce that I am now a Product Manager at Brocade. This is a big move for me, and one I am very excited about. I will get to work on the future of networking with a lot of smart people .

Why Brocade?

It’s simple really:

He aha te mea nui o te ao?
He tangata! He tangata! He tangata!

What is the most important thing in the world?
It is people! It is people! It is people!

Brocade has hired a lot of very clever people. This includes several Tech Field Day delegates, such as @joshobrien77, @Cloudtoad and @DavidJohnGee. I want to learn & grow, and being surrounded by great people is one of the best ways to do that.

Of course there’s more to it than just people. The problem with attending events like Network Field Day is that it’s addictive. You spend a week talking with smart people about the future. Big discussions, about what’s wrong with the current state, and what the future could be.

Then you return to a job where you’re working with networks that haven’t changed operation in over a decade. Adding VLANs to a 3750 and editing Check Point firewall rules Continue reading

Using Check Point Identity Awareness with NAT

Check Point Identity Awareness is problematic in environments that have multiple customers, overlapping private address space, and NAT. It can be done, if you understand the traffic flows, the connections needed, and how to combine several features. Here’s how I did it.

Background: Typical Check Point Management Flows

A quick reminder of the traditional flows used for Check Point firewall management:

Check Point Management Clients (e.g. SmartDashboard, SmartLog) connect to the management server to configure policies, view logs, etc.

Policies are compiled and pushed from the management server to the firewall(s). Logs are sent from the firewall back to the management server. All good.

Identity Awareness: Additional Connections

Identity Awareness lets you define rules based upon user identities, rather than IP addresses. So you can say “This AD group is allowed to connect directly to the SQL Server.” Much nicer Continue reading

HP IMC Adapter Directory Naming

This week’s lesson: Be consistent with your vendor naming when working with HP IMC Custom Device Adapters. When you create the new adapter directory, use exactly the same vendor name as used within the UI. Otherwise IMC may not recognise your new adapter. Case matters too, even on Windows!

HP IMC ships with a set of “Device Adapters” that define functions such as backups, configuration deployment, firmware upgrades, etc. These adapters are sets of XML, TCL and Perl files. They define which devices are supported, for what functions, and how to execute those functions.

Obviously HP can’t support every device ever made. But they’re quite happy for you to write your own adapters, or extend the ones they have. So if you’ve got a few unsupported switches, and they have some sort of sensible interface, you can write your own adapters.

These are stored at /server/conf/adapters/ICC/. Under there, you have a set of folders for each vendor. Under each vendor folder is an adapter-index.xml file, which maps SNMP sysOIDs to adapters. You must have a mapping in the adapter-index.xml file for your sysOID. (nb you can use wildcards). If those XML files change, you need to restart IMC.

Continue reading

F5 APM, SRX and DTLS NAT Timeout

I have been having issues using the F5 APM client behind a Juniper SRX-110 using hide NAT. I believe I’ve tracked it down to the default timeout settings used for UDP services. Here’s what I did to resolve it.

Constant Connection Timeouts

The laptop client was behind the SRX-110, using hide NAT. The initial client connection would work, and things would look good for a while. The the client would stop receiving packets. Traffic graphs would show a little bit of outbound traffic, and nothing inbound. Eventually, the client might decide it needed to reconnect. But usually, it would sit there for a few minutes doing nothing. Then I would force a disconnect, which would take a while, and then reconnect. Exceedingly frustrating.

Connecting the client to a different network – e.g. using a phone hotspot – worked fine. No dropouts. Using a wired connection behind the SRX had the same issue. So clearly the problem was related to the SRX.

TLS & DTLS

I dug into the traffic flows to better understand what was going on. This SSL VPN solution makes an initial TLS connection using TCP 443. It then switches over to DTLS using UDP 4433 for ongoing encrypted Continue reading