NFD9 Prep: SD-WAN

Software Defined WAN, or SD-WAN, looks to be a theme of Network Field Day 9, with presenters such as CloudGenix and VeloCloud showing us their offerings. At first glance, SD-WAN sounds pretty compelling. Who wouldn’t want to slash their WAN OpEx? How do these solutions work, and do they have legs? I’m hoping to find out.

What’s SD-WAN all about?

SD-WAN is about applying concepts of SDN to WAN networks. The goals are to increase flexibility and reduce WAN costs. This can be achieved through transport independence, dynamic path management, and better config management.

Historically we used private WAN circuits – leased lines, MPLS, etc. These had great SLAs, but the monthly costs were huge. The bandwidth was low, but guaranteed. Now that many places have access to high-speed Internet tails, it’s a lot harder to justify that cost. It’s very tempting to run IPSec VPNs across Internet links instead.

Those consumer Continue reading

Updated Big Switch Labs

Just a quick note to say that Big Switch have updated their demo lab system. This is an entirely virtual lab environment that simulates a Big Switch network. You can try out both Big Cloud Fabric and Big Tap Monitoring Fabric.

The lab gives you full CLI & GUI access to a sandboxed environment, with controllers, leaf/spine switches, and endpoints. Big Switch have written a sample lab you can work through, to show off the features, but you’re not limited there. You’re free to try out whatever features you like.

If you’re interested in what they’re doing, I recommend signing up.

Don’t Underestimate Your Users

The “consumerisation of IT” has an interesting side-effect. Historically people mainly used computers for work. But now that many people have smartphones, tablets and laptops at home, their perception and understanding of technology has shifted. Old assumptions about training required when upgrading applications or client operating systems may no longer apply.

This comment at The Register aligns with what I’m seeing:

…We’re at the point now where users are using Windows 8 at home and wondering why the work computer is so dated. It’s the perception of IT people that users can’t handle change holding up that change, not the ability of the users. At home that same set of users has managed quite well with updated versions of Office, updated Windows, iPads, Android tablets, Facebook, video messaging and various other completely new things. Somehow they coped without extensive training and therapy. From what I’ve seen, it’s actually IT staff who don’t like Windows 8 and are trying to keep users away from it…

I can recall being involved in Office upgrades just a few years ago, and being nervous about how that would be perceived. We were concerned that there would be major push-back, because the exact locations of the buttons Continue reading

NFD9 Prep: NetBeez

I’m reviewing the presenters for Network Field Day 9, in particular looking at those I’m not familiar with. NetBeez is one of those making their first Tech Field Day appearance.

NetBeez

We all know that our users and the applications they access are incredibly distributed. We don’t control all the network elements, but the network team still gets the blame if things go wrong. You need greater visibility to prove it’s not the network, but getting that visibility is tough. Current options for probes aren’t always cost-effective to deploy across many sites. Many sites don’t have any local server infrastructure.

That’s where NetBeez comes in. They have developed Raspberry Pi-based agents that can easily be deployed to many locations. Plug in power, plug in a network cable, and it phones home. Go to the NetBeez dashboard, and from there you can configure the tests you want the agent to run.

Since the devices are so small, they can easily be deployed to a range of small sites, and can simulate a range of user traffic. Tests include Ping, HTTP, Traceroute, DNS. A particularly nice feature is the ability to run an ad-hoc iPerf test with custom parameters.

The dashboard shows you how the Continue reading

Headwinds, or Uphill?

As some of my readers know, I’ve done a fair bit of bike touring. Two of the challenges of bike touring are riding uphill, and riding into headwinds.

Riding uphill is tough. 2,300m passes in snow, or 3,200m passes in sunshine, it’s tough going. But you put your head down, and keep turning the pedals, because you know that eventually you will reach the top, and there will be a downhill reward.

Riding into headwinds is a different story. You can battle into headwinds for days, and never get any reward. It saps your energy, and you don’t know if or when it will ever end. The wind could just keep coming from that direction. I’ve gone to different countries just to avoid the wind in the past.

They’re both hard. But one of them has an end, and a reward. The other one can just keep on sucking away your will to live.

Applying it to Life

This applies to the rest of your life. The tricky bit is that sometimes you don’t know if you’re going uphill, or into the wind. Both of them feel hard, and you can’t always see the end in sight.

A Continue reading

Network Field Day 9

I had a fantastic time at Network Field Day 8, and now I’ve been lucky enough to be invited back to NFD9 this February.

As usual, the Tech Field Day crew have put together a great mix of vendors. I particularly like the look of the SDN WAN-focused vendors, such as VeloCloud and cloudgenix. Much of the early SDN focus has been on the DC use-case, but that has limited applicability in my local market. SDN WAN solutions definitely apply to the New Zealand market though. I can think of several organisations where I’d love to have better WAN options today.

I’m also very happy to see Cumulus Networks making a first appearance.  I’ve done a lot of Linux work during my career, and there’s many times I would have loved to have all the capabilities of a GNU/Linux environment on a switch. I think they will have a huge influence on how Network OSes are delivered in future.

Network Management has always been a large part of my career too, so I’m looking forward to hearing updates from SolarWinds, and to find out more about NetBeez.

There’s some old faces and new attending. I’m looking forward to meeting people who I’ve Continue reading

Resources for learning HP Comware

HP is making more resources available to help with learning Comware. They’ve added free labs and courses to the already published simulators and virtual routers. This is a good resource for those looking to get started with Comware.

HP Network Simulator (HNS, aka Simware)

HP’s Network Simulator (HNS) is a modelling tool for simulating HP Comware networks. It includes Layer-2 functionality, and lets you test things like LACP & IRF. I found it too slow when I first tried it, but this has improved significantly with current versions. It is free to download.

HP has now started publishing simple labs you can work through with HNS:

• Lab 1 – Basic management
• Lab 2 – Port and link management
• Lab 3 – Port-based and protocol-based VLANs

These are short labs that cover HNS setup, and device configuration. Quick and easy, they show how to use the tool, and give you a taste of Comware configuration. They’ve also released a free 1-hour online course that goes through how to use HNS.

VSR1000

I’ve covered the HP VSR1000 previously. This Continue reading

Rate my IOS?

Review schemes are useful for identifying good consumer products and applications. But that doesn’t mean that everything needs to prompt me to leave a review. Cisco has started prompting for reviews for IOS versions, but I’m not convinced it makes sense for network operating systems. Perhaps it will do one day when disaggregated hardware/software is the norm for network devices.

Reviews for Consumer Apps – no problem

I love the 1Password password manager. It’s a well-polished app, and has been great value. Part of making my life better means not annoying me with frequent prompts for review:

1Password never prompts you for a review. We value your workflow too much to interrupt it. If you feel generous and have a couple of minutes, please leave a review. It means the world to us.

I like the Pocket app too. It prompts me to leave a review every single time it gets updated, which annoys the hell out of me. But hey, it’s free, so maybe I shouldn’t complain too much.

Pocket and 1Password are examples of consumer applications in a competitive market. The barrier to switching is relatively low, and they live and die on reviews. In a crowded market, customers rely on reviews, and the Continue reading

Big Switch Chaos Monkey Network Testing

Whenever you build a complex system, you need to test that it works as expected, including properly handling failures. It’s easy enough to do simple component failure testing, but harder to do rapid automated failure tests. Big Switch is showing that it can be done though. Hopefully we can keep improving our testing to pick up some more of the software failures.

Testing is hard

Over the course of my career I’ve built many clustered systems – HP-UX Serviceguard, firewalls, routers, load balancers, RedHat Clusters, etc. Good clusters have redundant everything – servers, power supplies, disks, NICs, etc.

The commissioning process always included testing. We’d go through each of the components, trying to simulate failures. Unplug each of the power cables, the network cables, unseat a hard drive, remove a hot-swappable fan, etc. That would test out the redundant components within each server, and then of course you’d simulate a complete system failure, forcing full failover.

This is all important stuff, but it doesn’t pick up all the failures – e.g. What happens if you’ve got a faulty patch lead, and the link starts flapping? Sometimes a simple failure gets messy when it happens repeatedly over a short Continue reading

Operations Manager to OMi Migration Path

HP has finally announced a migration path for Operations Manager to OMi. It’s about time too. This looks like a good path. If you want to stick with HP Software for managing your services, you should be investigating it.

The writing’s been on the wall for a while. HP has stopped investment in Operations Manager. I asked last year if HP had abandoned Operations Manager. This year I noted that it was kicking, but only just. My advice was:

To customers using HP OM…start planning your migration away from it, if you haven’t already. To customers considering purchasing it: Don’t, unless you’re buying it as part of an overall BSM/OMi implementation, and the salesfolk have guaranteed you can change your licensing over at no cost in future.

Well, HP has finally announced the OM2OMi Evolution program. Key points:

• License entitlement – OM servers can get equivalent licenses for OpsBridge Premium
• Operations Agent 11 works with both OM and OMi, so you don’t have to do the Agent migration at the same time
• Migration tools to assist with switching over

They do include this quote:

Well no one at HP is going to try to force you into replacing a product you love. Rest Continue reading

Christmas Change Freeze – Good or Bad?

We’re approaching Christmas, and for many of us, that means we’re about to enter an extended change freeze. This means an extended period when we shouldn’t change anything, hoping to improve stability. ITIL Change Management tells us this is good. I’m not convinced.

The Christmas Change Freeze

Many businesses impose some form of change freeze across all production systems during the Christmas/New Years period. In theory, all network/compute/storage changes are deferred until January. In practice, high priority changes will still be made if you jump up and down enough. The rate of change should still be lower during this period though.

Some change freezes may only run from just before Christmas until early January. Other businesses will go into a change freeze for as long as five weeks. My experience is that Southern Hemisphere businesses have a longer change freeze than Northern Hemisphere ones. I assume this is because many staff take extended leave over the Austral summer.

Why Have One?

There are differing opinions about the usefulness Continue reading

War Stories: Unix Security

A different kind of war story this time: Unix security blunders. Old-school Unix-types will mutter about how much more secure Unix systems are than Windows, but that glosses over a lot. In a former life I worked as an HP-UX sysadmin, and I saw some shocking default configurations. I liked HP-UX – so much better laid out than Solaris – but it was very insecure by default. Here’s a few things I’ve come across:

Gaining Root

We’d lost the root password for a test HP-UX server. We had user access, but not root. The server was located in a different DC, and we didn’t really feel like going and plugging in a console cable to reset the root password. So we started looking around at how we might get access. After a while I found these two things:

1. Root’s home directory was ‘/‘ – this was the default on HP-UX
2. The Remote Login service was running

And now for the kicker:

hpux lhill$ ls -ld /
drwxrwxrwx 30 root wheel 1020 1 Nov 13:57 /

Put those together, and you can see it’s easy to gain root. All we needed to do was create /.rhosts, and add whatever Continue reading

Outsourcing Mistakes

Outsourcing is complex, and there are lots of ways it can go wrong, or simply fail to deliver. I’ve put together a list of things that I see going wrong with outsourcing arrangements. Of course it’s not exclusive!

There’s a few different types of outsourcing. It might mean procuring a commodity service – e.g. IaaS, or it might mean handing over your existing environment and staff to a third party. There’s also a whole range of levels in between, but the usual deal is: Some part of your environment gets managed or delivered by someone else, according to the terms of a fixed agreement.

Here’s a few things I’ve learnt to watch out for (nb not all these items apply to all types of agreements):

Not keeping up to date

If your outsourcer is managing your software, the contract usually covers applying security patches and bug fixes. But what gets missed is larger upgrades – e.g. ESXi 4.1 to 5.x. Everything goes OK for a while…and then your version goes End of Support.

It then becomes a major drama to get the upgrades sorted out. For financial purposes, you may not be able to do major Continue reading

Juniper SRX-110H EoL

Somehow I missed this when it was announced, but the Juniper SRX-110H-VA is End of Life, and is no longer supported for new software releases.

End of Life announcement is here, with extra detail in this PDF. Announcement was Dec 10 2013, with “Last software engineering support” date Dec 20 2013.

This is now starting to take effect, with 12.1X47 not supported on this platform:

Note: Upgrading to Junos OS Release 12.1X47-D10 or later is not supported on the J Series devices or on the low-memory versions of the SRX100 and SRX200 lines. If you attempt to upgrade one of these devices to Junos OS 12.1X47-D10, installation will be aborted with the following error message:

ERROR: Unsupported platform <platform-name >for 12.1X47 and higher

The replacement hardware is the SRX-110H2-VA, which has 2GB of RAM instead of 1GB. Otherwise it’s exactly the same, which seems a missed opportunity to at least update to local 1Gb switching.

Michael Dale has a little more info here, along with tips for tricking a 240H into installing 12.1X47.

So I decided to see if I could work around this and trick JunOS into installing on my 240H, I Continue reading

Wipebook – A Portable Whiteboard

It is a stereotype, but engineers really do like whiteboards. Problem is, you can’t carry one around with you. Plus there’s still a few unenlightened employers who don’t provide whiteboards. Enter the Wipebook, a spiral-bound notebook made of whiteboard-like pages:

I normally carry a notebook for scratching out notes while talking to customers, sketching diagrams, working through problems, etc. I don’t archive these notes – most are just short-term things, and I shred them. Important stuff gets turned into OmniFocus tasks/emails/etc.

So the Wipebook looks perfect for me. My wife bought one for me recently, and I’ve started using it at work. So far, it’s working as expected. I can quickly scribble notes, sketch a diagram, make corrections, etc. When I’m done with it, I wipe the page down.

It’s not perfect – the pages don’t always wipe down perfectly, and obviously it gets bumped around in my bag. So it won’t last forever. But it’s a nice touch that I can open & close the bindings, so I can easily get rid of any pages that are too beaten up.

The pens have a small eraser on the end, but it’s only suitable for very minor corrections. I have a Continue reading

iRules/Tcl – Watch the Comments

It’s pretty common practice to ‘comment out’ lines in scripts. The code stays in place, but doesn’t get executed. Perfect for testing, when you might need more debug output, or you want to run a slightly different set of actions. But you have to be careful when commenting out lines – it might catch you out, and the F5 iRules editor won’t save you.

Normally it’s pretty simple to comment out a line. Here’s a quick Bash example:

#!/bin/bash

FILECOUNT=`ls /tmp|wc -l`

if [ $FILECOUNT -lt 7 ]
 then
        echo "There are fewer than 7 files in /tmp"
        run_command
fi
...

When I’m testing the script, I might not want to actually run that command. So I’ll quickly comment it out like this:

#!/bin/bash

FILECOUNT=`ls /tmp|wc -l`

if [ $FILECOUNT -lt 7 ]
 then
        echo "There are fewer than 7 files in /tmp"
        #run_command
fi
...

The ‘#’ tells the shell to ignore anything else on that line. All pretty straightforward.

Today I was debugging an F5 synchronisation issue, where I got this message on synchronisation:

BIGpipe parsing error (/config/bigip.conf Line 333):
   012e0054:3: The braced list of attributes is not closed for 'rule'.

The offending section looked like this:

when  Continue reading

Complexity vs Security

Many of the ‘security’ measures in our networks add complexity. That may be an acceptable tradeoff, if we make a meaningful difference to security. But often it feels like we just add complexity for no real benefit.

Here’s some examples of what I’m talking about:

• Multiple Firewall Layers: Many networks use multiple layers of firewalls. If you have a strong policy that says all traffic must go via a server within a DMZ, this makes sense. But often we end up with the same connections going through multiple firewalls. We end up configuring the same rules in multiple places. No security benefit, but increased chance of making mistakes, and added troubleshooting complexity.
• Chained proxies: It’s pretty common to use a proxy server, to enforce HR and security controls on what users browse. But some organisations have chained proxies, where an internal proxy server connects to an upstream proxy server to get Internet access. The upstream proxy doesn’t add anything from a policy or control perspective. It’s just another point to configure and troubleshoot.
• NAT/Routing: First let me be clear: NAT is not complete security in itself, but it can form a valid part of your overall network security policy. That Continue reading

War Stories: Cursed VLANs

I’ve written before about switch ports being permanently disabled. This time it’s something new to me: VLANs that refuse to forward frames.

A Simple Network

The network was pretty straightforward. A pair of firewalls connecting through a pair of switches to a pair of routers:

Sub-interfaces were used on the routers and firewalls, with trunks to the switches. VLAN 100 was used for 100.100.100.0/24, and VLAN 200 was used for 200.200.200.0/24. The switches were configured to pass VLANs 100 & 200.

All was working as expected. All devices could see each other on all VLANs.

Until it stopped

We received reports that we’d lost reachability to Router A’s VLAN 200 sub-interface. After doing some investigation, we could see that Firewall-A could no longer see Router A’s MAC address on G0.200. But everything else was fine – the VLAN 100 interface worked perfectly. So we knew it couldn’t be a physical interface issue.

Hmmm. What’s going on? First instinct: check the switch port configuration. Has anything changed? Nope. VLAN 200 still there, configured as expected. The router & firewall were still tagging frames with VLAN 200. But they couldn’t see each other, and the Continue reading

Ops Work vs Project Work

There’s a constant tension between delivering new services, and running the existing services well. How do you figure out how to prioritise work between Operations tasks and Project work? Skewing too far either way leads to problems. Maybe the answer is in how we structure Operations tasks?

Definitions

• Operations work: Dealing with outages, trouble tickets, support requests, etc. System monitoring – reviewing data for capacity planning, and identifying new areas to monitor. Automated repetitive tasks. Patches, upgrades, minor changes to existing services. Accountants would call this work OpEx.
• Project work: Design, test and deployment of new services. Major upgrades or enhancements to existing services. This is usually classified as CapEx. For some businesses, this work is customer-billable.

What happens when you’re imbalanced?

• Too much Project work: If you’re flat out deploying new systems (and dealing with the fallout), it’s easy to let Operations work slip. Maybe you don’t get around to automating that log rotation script, or paying attention to the slope of that consumption graph. It’s OK for a while too…things seem to be trucking along. But then you start having outages due to simple things like logs filling directories, or you hit a capacity limit, and there’s a 6-week Continue reading

Meeting Rules

Years ago a wise engineer gave me these rules for meetings:

1. Never go into a meeting unless you know what the outcome will be.
2. Plan to leave the meeting with less work than when you went in.

Stick to those rules, and you’ll do well.

OK, so maybe the second rule’s not so serious, but the first one has a grain of truth. You don’t need to know exactly what the decision should be, but you should be clear about what you want to get decided. If it’s particularly important, you should have already discussed it with the key attendees, and you should know what they’re thinking. You don’t want any surprises.

Too many meetings have no clear purpose, or they can only agree that ‘a decision needs to be made…pending further research.’ Avoid those sorts of meetings. Otherwise it ends up like…well….Every Meeting Ever: