It’s 2015: “Supports IPv6″ should mean full support

It’s 2015. ARIN is finally out of IPv4 addresses, more than 20% of Google users in the US are using IPv6…and vendors are still doing a half-assed job with IPv6 support. I purchased a new TP-Link Wi-Fi router/modem recently, and it doesn’t fully support IPv6. It’s not good enough, and I will be returning it.

I purchased the Archer D5 “AC1200 Wireless Dual Band Gigabit ADSL2+ Modem Router.” The website blurb includes this:

IPv6 Supported. The next generation of Internet protocol, helping you to future-proof your network.

And the specifications page says: “IPv6 and IPv4 dual stack.”

I checked the documentation for how to configure IPv6. This FAQ walks through configuring IPv6 on several TP-Link devices. Note that it includes this line “…choose Connection type (Here we just set up PPPoE as an example, if you are not sure, please contact your IPv6 provider)”

In New Zealand, most ADSL services are delivered as PPPoA. The specifications page says this device supports PPPoA. My ISP provides native IPv6 via DHCPv6 PD. So everything should be good to go, right?

Not so much. The Archer D5 does indeed support PPPoA. It also supports IPv6 with DHCPv6 PD. But it Continue reading

Networking Pioneers, Settlers and Town Planners

Can we broadly separate Networking into Pioneers, Settlers, and Town Planners? I’ve been thinking about how to apply Simon Wardley’s PST model to networking. This leads to thinking about how we can encourage networking evolution. The model needs a lot of fleshing out, but I’m interested in what others think.

Pioneers, Settlers and Town Planners (PST)

Simon Wardley has written about “Pioneers, Settlers and Town Planners (PST)” in many places – e.g. here and here. It derives from Cringely’s Commandos, Infantry, Police model (see Chapter 12 of Accidental Empires). It provides a organisational structure, grouping areas based upon their current state of evolution. It recognises that no one operational model works for all parts of a business. You can’t say “We’re using Agile, or Lean, or Six Sigma” – you need to use the appropriate model for each area. Even “Bi-Modal IT” is too limiting, as the divide is too great.

(Image from blog.gardeviance.org, used under Creative Commons License)

Applying PST to networking

The model is a helpful way of thinking about the role of different groups across a business. It also helps us understand why teams need to evolve over time, Continue reading

Considering On-Call Pay

Let’s say you’ve been offered a new job. $70k base salary, with up to $20k per year extra for on-call duties. Great! $90k! That’s $10k more than my salary now! Sign me up!

Wait a minute. Not so fast.

Years ago I received some good advice: Treat your on-call payment as a separate item. Don’t consider it as part of your base salary. This is because you need to be paid properly both for doing your job, and for the inconvenience of being on-call.

Evaluate the base salary for what it is: Your salary for doing your day-to-day job. Ignoring the on-call part, did they offer you enough money for the role? Is it a good match for your experience?

Separately decide if the on-call payment is enough to justify being on-call. Does it represent a fair payment for the extra work?

Why does this matter? A few reasons:

• Being on-call has a big impact on your life, and you need to be properly paid for it.
• You still need to get paid properly for the job you do Monday-Friday.
• On-call payments will vary. Your job role could change. Maybe the team gets bigger, and you go on-call less often. Now you’re earning Continue reading

Brocade Certified vRouter Engineer

If you’ve visited the Brocade website recently, you’ve probably seen the “Free NFV Certification” banner. I signed up for this several months ago, but had put off completing the course. I had a little downtime recently prior to starting work at Brocade, so I completed this course & exam. Here’s my impressions.

What’s the Course/Exam About?

From the official documentation:

As a Brocade Certified vRouter Engineer, you must be able to demonstrate the ability to install, configure and troubleshoot features of Brocade Vyatta Network OS.

i.e. it’s primarily about the basics of Vyatta.

What’s Included?

Here’s what you get when you sign up:

1. A download link to the Brocade Vyatta 5400 vRouter image.
2. Access to the “Brocade Certified vRouter Engineer Course.”
3. Links to certification materials and communities.
4. Promo code for the BCVRE exam.

Note that you can run the Vyatta image on a local hypervisor, or if you prefer you can select it from the AWS marketplace. Personally I ran it on VMware Fusion on my laptop. It’s light on resource, so you can easily spin up several Continue reading

Think Bigger

I get frustrated by those who take a narrow view of technology, and progress in general. They see things in terms of where they are now, and where they were. But they struggle to see a bit further out. The Internet of Things is a good example of this.

I made the mistake of reading the comments on a recent El Reg article (I know, I know: Never read the comments). I came across this comment about the IoT:

…The innocent child asked “but why would the toaster need to talk to the ‘fridge?” The marketing gurus had no answer and a few years later the outfit went bankrupt. In all the time since, no one has been able to answer that question.

From there the comments devolved into a rather pointless discussion about milk, bread, spam on toast and Twitter. This is a fairly common theme on El Reg articles (along with “cloud has little appeal for 90% of SM server/computing requirements”, but that’s another issue).

I find it frustrating when people take a narrow, short-sighted view when looking at technology trends. We all see things from our own perspective, but it’s good to lift your head Continue reading

The Next Step: Brocade

I am happy to announce that I am now a Product Manager at Brocade. This is a big move for me, and one I am very excited about. I will get to work on the future of networking with a lot of smart people .

Why Brocade?

It’s simple really:

He aha te mea nui o te ao?
He tangata! He tangata! He tangata!

What is the most important thing in the world?
It is people! It is people! It is people!

Brocade has hired a lot of very clever people. This includes several Tech Field Day delegates, such as @joshobrien77, @Cloudtoad and @DavidJohnGee. I want to learn & grow, and being surrounded by great people is one of the best ways to do that.

Of course there’s more to it than just people. The problem with attending events like Network Field Day is that it’s addictive. You spend a week talking with smart people about the future. Big discussions, about what’s wrong with the current state, and what the future could be.

Then you return to a job where you’re working with networks that haven’t changed operation in over a decade. Adding VLANs to a 3750 and editing Check Point firewall rules Continue reading

Using Check Point Identity Awareness with NAT

Check Point Identity Awareness is problematic in environments that have multiple customers, overlapping private address space, and NAT. It can be done, if you understand the traffic flows, the connections needed, and how to combine several features. Here’s how I did it.

Background: Typical Check Point Management Flows

A quick reminder of the traditional flows used for Check Point firewall management:

Check Point Management Clients (e.g. SmartDashboard, SmartLog) connect to the management server to configure policies, view logs, etc.

Policies are compiled and pushed from the management server to the firewall(s). Logs are sent from the firewall back to the management server. All good.

Identity Awareness: Additional Connections

Identity Awareness lets you define rules based upon user identities, rather than IP addresses. So you can say “This AD group is allowed to connect directly to the SQL Server.” Much nicer Continue reading

HP IMC Adapter Directory Naming

This week’s lesson: Be consistent with your vendor naming when working with HP IMC Custom Device Adapters. When you create the new adapter directory, use exactly the same vendor name as used within the UI. Otherwise IMC may not recognise your new adapter. Case matters too, even on Windows!

HP IMC ships with a set of “Device Adapters” that define functions such as backups, configuration deployment, firmware upgrades, etc. These adapters are sets of XML, TCL and Perl files. They define which devices are supported, for what functions, and how to execute those functions.

Obviously HP can’t support every device ever made. But they’re quite happy for you to write your own adapters, or extend the ones they have. So if you’ve got a few unsupported switches, and they have some sort of sensible interface, you can write your own adapters.

These are stored at /server/conf/adapters/ICC/. Under there, you have a set of folders for each vendor. Under each vendor folder is an adapter-index.xml file, which maps SNMP sysOIDs to adapters. You must have a mapping in the adapter-index.xml file for your sysOID. (nb you can use wildcards). If those XML files change, you need to restart IMC.

Continue reading

F5 APM, SRX and DTLS NAT Timeout

I have been having issues using the F5 APM client behind a Juniper SRX-110 using hide NAT. I believe I’ve tracked it down to the default timeout settings used for UDP services. Here’s what I did to resolve it.

Constant Connection Timeouts

The laptop client was behind the SRX-110, using hide NAT. The initial client connection would work, and things would look good for a while. The the client would stop receiving packets. Traffic graphs would show a little bit of outbound traffic, and nothing inbound. Eventually, the client might decide it needed to reconnect. But usually, it would sit there for a few minutes doing nothing. Then I would force a disconnect, which would take a while, and then reconnect. Exceedingly frustrating.

Connecting the client to a different network – e.g. using a phone hotspot – worked fine. No dropouts. Using a wired connection behind the SRX had the same issue. So clearly the problem was related to the SRX.

TLS & DTLS

I dug into the traffic flows to better understand what was going on. This SSL VPN solution makes an initial TLS connection using TCP 443. It then switches over to DTLS using UDP 4433 for ongoing encrypted Continue reading

Check Point SmartLog – Recommended

Trigger warning for Check Point haters: I’m about to say nice things about Check Point.

Continuing the recent theme of Check Point-related posts, I’d like to give Check Point credit for once. SmartLog is what I always wanted from Tracker/Log Viewer, and they’re not even charging me extra for it. Shocking, I know.

Traditional Log Analysis

15-20 years ago, Check Point was well ahead of the competition when it came to viewing firewall logs. “Log Viewer” or “SmartView Tracker,”^[1] let you filter logs by source, destination, service, etc., and quickly see what was happening. The GUI worked well enough, and junior admins could learn it quickly.

Most other firewalls only had syslog. That meant that your analysis tools were limited to grep and awk. Powerful yes, but a bit of a learning curve. There was also the problem of ‘saving’ a search – you’d end up hunting through your shell history, trying to recreate that 15-stage piped work of art. Splunk wasn’t around then.

Times Change

Tracker has several issues:

• Log files are ‘flat’ files. It is a proprietary binary format, but it’s still flat, with no indexing. The format is very structured, but searches are slow when the files get large.
• Searches Continue reading

Check Point – Don’t Use the ‘Install On’ Column

I got caught out by Check Point’s “Install On” column recently. Most people don’t need this setting any more, but it’s still there for legacy reasons. Time to re-evaluate.

When you create a firewall policy using Check Point, you define the set of possible installation targets. That is, the firewalls that this policy may be installed on. When you compile & install policy, you can choose from this list of targets, and only this list.

Most organisations will only have one installation target per policy. But sometimes you want to have the same policy on multiple firewalls. This is pretty easy to do, and might make sense if you have many common rules.

But then you say “What if I had 30 common rules, 50 that only applied to firewall A, and another 50 that only applied to firewall B?” That’s where people start using the “Install On” column. This lets you define at a Continue reading

IPv4 Address Transfer Prices Down?

Last year I wrote about the IPv4 Address Transfer Process. Recently I was involved in another IPv4 transfer. I was surprised to see that IPv4 prices have fallen in the last year. I have done some rudimentary analysis of the APNIC transfer statistics to try to figure out why.

APNIC publishes statistics on transfers at ftp.apnic.net/public/transfers/apnic. These text files list all resource transfers that have taken place – the to & from organisation, the resource type, the date, etc. I am very interested in looking at the trends. How many transactions take place each month, and how many addresses are being transferred?

I wrote a simple Python script to do this analysis for me. It retrieves the latest statistics, and converts them into a Google chart:

(If you’re reading this via RSS, and the chart doesn’t display, you may need to click here to see the web version).

Note this does not do live updates. It is a point in time, generated using the current data at the time the script is run. If you would like to update the code to do live updates, fork it from Github here. I’d also love to update the script to Continue reading

Musing: Generalist to Specialist and Back Again

Recently I’ve been musing on IT Generalists vs Specialists. We used to have more generalist roles, covering all parts of the stack. ITIL then pushed us towards greater specialisation. I believe that we’ve gone back to valuing the Generalist, as the person who can glue components together. Will the pendulum swing again?

Generalists: Soup-to-Nuts

When I started working in IT, our roles were more generalist in nature. We did everything. To set up a new app, we racked the servers and switches, installed the OS, configured the network, installed the DB & application, and made it all work.

We weren’t specialists in any one area, but we knew how everything fitted together. So if something broke, we could probably figure it out. If we had to investigate a problem, we could follow it through all layers of the stack. When we found the problem, we had license to fix it.

ITIL takes over: Specialisation

Sometime around the early-mid 2000s the “ITIL Consultants” moved in. Their talk of structure, processes and SLAs seduced senior management. We couldn’t just have people who Got Shit Done. No, everyone needed to be placed in a box, with formal definitions around what they could & Continue reading

NetBeez Review

NetBeez presented at Network Field Day 9, where they showed us their solution for distributed network performance monitoring. They gave the delegates a NetBeez agent to take home and test. I’ve run it for the last two months, and I’ve been happy with how it has performed.

Physical Install

The unit was supplied with a US power plug. I was contemplating using an adapter, but I didn’t have any spare power points near where I wanted to install it. Hmmm. Then I realised that the power connection is just a USB port anyway. The Ethernet cable needed to go into my SRX-110, so I wonder if…

Yup! Powers from the USB port on the SRX. Perfect.

Monitoring Setup

The device powered on, and it soon showed up on the NetBeez web dashboard. This is where you can configure your agents, define what tests you want them to run, and see the results.

I added a few simple monitors:

• HTTP monitoring to this website
• HTTP monitoring to www.netopscommunity.net
• PING monitoring to www.netopscommunity.net

All very straightforward to add the monitors, and pick which agents to run them from. Running the same tests from multiple agents gives you a distributed status Continue reading

Reminder: Solarized for Better Terminals

I have used the “Solarized” colour scheme on my Mac for several years. This is:

… a sixteen color palette…designed for use with terminal and gui applications

If you spend a lot of time using the Terminal, this makes a huge difference. It gives me the right combination of colours to make sure everything is readable, and reduces eye-strain.

I’ve used it for so long that I’ve forgotten about it. It’s become “normal” for me.

PuTTY Defaults == Unusable

Recently I’ve been forced to use PuTTY on Windows. I’d forgotten how terrible the default colour scheme is, particularly when you’re using VIM, or doing an “ls” on a RHEL system. Check this screenshot:

The default LS_COLORS on a RHEL system, using PuTTY defaults, will displays directories in dark blue on a black background. Hopeless. I can’t read those directory names.

Solarized to the rescue

I downloaded the “Solarized Dark” registry file from here. Double-click that to merge the registry settings. You’ll then see a new PuTTY Saved session “Solarized Dark”:

Load that session. Save it as the Default Settings if you like. Add any other settings you need – e.g. username, SSH key. Add the hostname/IP, and connect. Now see how Continue reading

Automate All The Things? Maybe Not

I’m fundamentally lazy. That’s why automation appeals: less work for me. Get the machine to do it instead. But automating everything isn’t always the right answer. Sometimes you need to ask yourself: Does this task need to be done at all? Or can I get someone else to do it for me?

Automating tasks carries some overhead. If you’re really unlucky, you’ll end up spending more time on the automation than doing it manually:

So if you can eliminate tasks, you’re in a much better position. Here’s a few contrived examples, based around a fictitious email provider:

Eliminating Tasks: Maximum Email Size for ‘Special’ Users

15-20 years ago we had limited bandwidth, and limited storage. It seemed reasonable to limit the maximum email message size. Otherwise people would send monstrous 2MB attachments. Of course, there were always ’special’ cases that needed to be able to send enormous 5MB AVI files. So we had special groups of users defined that could send large emails.

Users could put in a request to the Help Desk to get access to send large emails. That would go via some manager, who would of course approve it. Someone would then need to manually update that Continue reading

You can’t put the future on hold

Greg Ferro recently participated in an “Ask Me Anything” thread on Reddit. In that thread, user “1DumbQuestion” made this comment:

Last, never finished my CCIE because of what I perceive will happen with SDN in the next coming years.

I’ve seen similar comments from others over the last couple of years. This concerns me because it seems that people are saying “There’s too much change going on here, and I don’t know how it will all work out. So I’ll just do nothing.”

Don’t be one of those people.

You should take a hard look at your career, and try to understand where the industry is going. If you think that CCIE study is not the best use of your time, that’s fine. But you should make a conscious choice about that. Crucially, you must decide where else to invest your time and energy.

If you firmly believe that networking will change dramatically over the next few years, then take active steps to prepare yourself. Think about your current skills, and where you have gaps. Maybe you need to learn more about Linux. Maybe it’s configuration management, or Python scripting. Put your time into Continue reading

What’s In My Bag (Hint: not much)

Recently @BobMcCouch posted a photo of the contents of his bags. He’s got a lot of gear, including a hammer, and a dent-puller. He assures us that it’s for lifting tiles, but I’m not so sure. Sounds to me like he’s worried about a few dings in the supermarket carpark.

It all sounded a bit scary. I want to provide a different perspective, that of someone who tries to minimise what they carry. I don’t want young engineers to think that they have to build up a huge toolbox, and the physical strength to lug it around. You might choose to do that, but it’s not the only path.

The Bag Itself

My general rules for a laptop bag are that it should be as small as I can get away with, and it should not look too much like a laptop bag. So pretty much anything from Targus is inappropriate.

Today I use the “ Continue reading

F5 Data Groups, Wildcards and tmsh

Just a quick note about a problem I ran into with adding data groups to an F5 system using tmsh. I wanted to add a string data group containing a list of URIs mapping to other URIs. This was for use in an iRule that will redirect these URIs.

So I thought that this tmsh script would do the trick:

modify ltm data-group redir_uris records add {"/first-uri" { data "/new-uri"}}
modify ltm data-group redir_uris records add {"/second-uri" { data "/new-uri"}}

Every time I tried it, I got this result:

Syntax Error: the "create" command does not accept wildcard configuration identifiers

Hmm. But I don’t have any wildcards. So what’s the problem? I couldn’t figure it out at the time, and ended up having to resort to manually entering the data group via the web interface. A bit slow, but luckily it was only around 20 entries.

Today I found out what was going wrong: SOL12999: “Data group records beginning with a slash character cannot be added using tmsh.”

Description: You cannot add data group records that begin with a slash ( / ) character to data groups using tmsh.

This issue occurs when all of the following conditions are met:

• You Continue reading

Check Point – Upgrade Without Dropping Connections

Check Point firewall upgrades have always been painful. The loss of connection state is a big part of this. Existing connections stop working, and many applications need restart. It looks like there is a way of minimising this pain on upgrade.

Stateful firewalls record the current ‘state’ of traffic passing through, so they can recognise and allow reply or related traffic. If you have a firewall cluster, they need to synchronise state between the cluster members. This is so that if there is a failover, the new Active node will be aware of all connections currently in flight.

If you have a failover, and the standby member is NOT aware of current connection state, it will drop all currently open sessions. Any packet that isn’t a SYN packet will get dropped, and the applications need to establish new connections. Some applications handle this well – especially those that use many short-lived connections such as HTTP or DNS. But other applications that have long-running connections – e.g. DB connections – may struggle with this. They think the connection is still open, and take a long time to figure out it’s broken. They may eventually recover on their own, or they may Continue reading