HP IMC Silent Installation

HP IMC installation is normally a manual process, with plenty of clickey clickey clickey. This is OK for production systems, as most sites will only have one or maybe two IMC servers. But for my lab, I wanted to automate the install, so I can quickly spin up a new lab system. I have now found an undocumented, unsupported way of doing this.

There’s two parts to this – preparing the underlying OS & DB, and installing IMC. I am writing Ansible playbooks to handle the OS + DB setup. That’s working, but it needs a bit of cleanup. Once that’s done, I’ll integrate it with Vagrant. Then I should be able to completely automate the install of a lab IMC system. I will write another post on that once it’s complete.

To install IMC silently, create an “install.cfg” file to define your settings. Then tweak the installation script to call the silent installer, not the interactive install.

DNSSEC – Moving the Needle

The New Zealand ISP market is dominated by Spark, Vodafone & CallPus/Orcon. A side effect of this is that if one player does the Right Thing™, it really moves the needle. Recently, Spark has done the Right Thing with DNSSEC.

DNSSEC takeup has been low with New Zealand ISPs. The APNIC stats indicated that around 5% of users were using DNS resolvers that had DNSSEC validation capabilities. But in December 2014, that number jumped to ~15%:

It turns out this is because Spark has enabled DNSSEC validation on some of their resolvers. NZRS have done some analysis, and found that Spark turned on 4 new resolvers that do DNSSEC validation:

They’re still running their old resolvers, so right now it’s hit & miss for their customers. But it’s a great start, and presumably they’ll upgrade the remaining systems soon.

So Vodafone, CallPlus, Snap, Trustpower…when are you going to take customer security seriously too? And Spark…how long until DNSSEC is enabled for all your resolvers?

And please, no arguments about “we’re not sure if it will work.” Google has been doing it since March 2013…who do you think processes more DNS requests per day? Google, or your ISP?

The Year of IPv6?

IPv6 adoption has been slow. But I think it’s reaching a tipping point. I’m very close to calling 2015 “The year of IPv6.” There’s plenty of people who won’t believe me, but the statistics are very interesting. You need to keep a close on eye on what the data is saying.

Recently I asked the question “What percentage of Internet traffic needs to be IPv6 for you to consider IPv6 to be mainstream/arrived/the year of IPv6?”

@bobbobob had the best answer for when IPv6 can be considered ‘mainstream’:

@northlandboy When I see criminal minds, or law and order using a poorly faked IPV6 address when they're 'hacking', I'll say it's arrived.

— Rabbit Sultan (@bobbobob) February 21, 2015

But @icemarkom was probably technically correct with this answer:

@northlandboy More than 50%.

— Marko Milivojevic (@icemarkom) February 20, 2015

So how far away is that? It’s tough trying to measure IPv6 adoption. Traffic patterns are region- & user-specific. The services that Chinese users access are different to those that a New Zealand business users. Traffic is often concentrated with a few ISPs and/or a few big services (Google, Facebook, Twitter, etc).

I like to use the Google IPv6 statistics Continue reading

Indeni, Smart Advisors and Crowd-Sourcing

Monitoring needs to move on from traditional fault and performance polling. It should include identifying common misconfigurations and known faults. We’re all using the same technologies, so we’ve all got the same problems. I like the look of Indeni, a new approach to this problem. It uses a form of crowd-sourcing to act as a smart advisor.

Precious Snowflakes?

We all think we’re precious snowflakes. But we’re not. We use the same technologies, glued together in the same ways. That means we all have the same problems, and make the same misconfigurations.

Vendors frequently publish new bug fixes, KB articles, EOS notices, etc. Some of these apply to products/versions/features we’re using. We struggle to keep up with the volume, and we miss these – so maybe our network is running with a known issue. Striking an unknown bug is bad. Getting caught out by a published issue is worse. Having an outage because we didn’t make sure the routing tables were in sync on our firewall cluster is unforgivable.

Vendors Need Help Too

Information flow is a two-way problem. The vendors can’t always see how customers deploy their products in the real world. They think they know. They write manuals, they write Continue reading

VeloCloud & Information Brokerage

VeloCloud was the first presenter at Network Field Day 9. They are one of the new breed of SD-WAN vendors. I’m impressed by what they’re doing, and and the potential it offers for re-thinking the way we do WAN connectivity. But I think the most interesting part is the increased visibility into how networks are performing.

I won’t go into the details of how it all works – Brandon covers some of it here, and you can look through VeloCloud’s site to understand it more. I want to focus on a few details around data analysis, and information brokerage.

Internet Quality Monitoring

In this video, Kangwarn Chinthammit talks about how VeloCloud is using their devices to monitor Internet quality. Because they’re installed in a wide range of locations, with many different WAN connection types, they’re building up some interesting data.

They’ve been able to do some deeper analysis of the data, and break down quality measurements by location, circuit type, hour, and day. Some of the interesting results include:

• A good ISP in one location may not be any good in another. So you can’t just pick one ISP.
• Quality varies during the day, and across the year. It might be Continue reading

NFD9: Cumulus Networks

Cumulus Networks gave a great presentation at Network Field Day 9. They presented their vision of how they’re working to improve networking. But they were also clear about what they don’t do, and where they will instead enable others.

Linux on a switch? Seems natural to me

Many network engineers started out running cables, and doing low-level networking. They build up to designing & running more complex networks. I came at it from a different direction. I first ran Linux systems in 1999. My first professional job was working with HP-UX in 2000, and I later moved into running Check Point firewalls on Nokia IPSO. I was well-used to working with Unix-like systems, and it was completely natural to me to run tcpdump on a network device.

To become an effective network security engineer, I had to learn more about routing & switching. But because I had that *nix background, I was always frustrated by the limited capabilities offered by IOS. “| include” is a poor substitute for grep. Yes, you can do some stuff with TCL, but would you want to? Packet capture was a poor joke until recently.

So when I first heard about Cumulus, it made a Continue reading

Rolling out Change

We all know that “Change is Hard.” But often we, as engineers, focus on the technical aspects of that change. How do I minimise customer impact while upgrading those routers? How can I migrate customer data safely to the new system? But we can forget about the wider implications of what we’re doing. If we do, we may struggle to get our changes implemented, or see poor take-up of new systems.

Why Can’t I Make That Change?

I was talking to an engineer who had planned a huge configuration management implementation. Everything had been manually configured in the past, but this was hitting scale issues. So he had worked for months on a fully automated process. It was going to be amazing. It would configure everything, across all systems and applications. Standards enforced, apps deployments done in a repeatable way, etc. It was going to be a thing of beauty. No-one would ever need to login to a server again. Total automation.

It was all tested, and was just waiting for approval to put it into production. But for some reason, no-one was willing to give the go-ahead to roll it out. Weeks were dragging by, and things were going Continue reading

Doing Community Programs Right

You know I’m not the biggest fan of vendor clubs (or influencer marketing programs, call them what you like). But if you’re going to do it, you might as well do it right. Don’t let it just become a ‘free T-shirt club':

If you're building the community program – ensure it scales so it doesn't end up being a free tshirt club.Connecting and Knowledge xfer=must

— Anthony Burke (@pandom_) February 5, 2015

@pandom is spot-on. The ideal community program should not just be a method to blast out press releases, or give out a few free shirts in the hope of currying favour. The program manager has taken care to select people who are positive about the company, share with the community and have opinions about where the vendor is going.

That is a valuable resource that should not be wasted. A good program should seek to engage in a two-way dialogue. Not just pushing out info, but seeking feedback on what’s working, and what’s not. Don’t just push out a few early release notices – have honest discussions about roadmaps, plans, etc. Help your members connect with each other – who knows what benefit that might lead to in future?

FWIW, I’m Continue reading

CCIE Renewed – Exam 400-101

The problem with obtaining certifications is that you need to renew them. CCIE is no different – I first passed the lab in September 2012, and I was overdue for renewing it. I’m pleased to report that I have now done that, and it is now current until September 2016. Here’s some of my impressions of the 400-101 exam.

I had planned on using the CCDE written exam to renew my R&S CCIE, and then decide if I would go on to attempt the CCDE practical exam. But it seems that the CCDE exam writers and I just don’t share the same mindset. I tried, but it wasn’t working for me, and I wasn’t making progress. So I went back to R&S for my re-cert.

New Blueprint

I originally passed version 4, exam number 350-101. This has been updated to version 5. The written exam is now 400-101. Of course, this doesn’t mean that everything changes. Core L2 & L3 protocols don’t change that much. BGP, OSPF and EIGRP and still BGP, OSPF and EIGRP.

There are some key changes though, such as:

• Frame relay gone, DMVPN in
• IS-IS back in – theory only
• New EIGRP features, such as named mode
• More MPLS Continue reading

IPv6 availability in New Zealand

IPv6 has been around a fair while, and we’re constantly encouraged to learn it and use it. I agree with the sentiment, but it’s been hard for most users, when few ISPs offer IPv6 for residential users. Hurricane Electric offers a great free IPv6 tunnel broker service, but that’s impractical for most people. What they need is for their ISP to offer native IPv6, by default.

The ISPs in New Zealand with the largest market share don’t offer IPv6, but some of the smaller ones do. The design of the ISP market here means that users can easily switch between a large range of suppliers, and choose the mix of price/service they want. When I last changed ISP a couple of years ago, I specifically chose an ISP that offers IPv6.

Last year that ISP disabled IPv6 for a few weeks due to some technical issues, and I was disappointed with the support they offered. I wanted to evaluate my other options, but couldn’t find any good source of data that showed which ISPs were offering IPv6. There’s plenty of talk out there about trials, and the like, but most of that hasn’t been updated in years.

So I pulled Continue reading

NFD9 Prep: SD-WAN

Software Defined WAN, or SD-WAN, looks to be a theme of Network Field Day 9, with presenters such as CloudGenix and VeloCloud showing us their offerings. At first glance, SD-WAN sounds pretty compelling. Who wouldn’t want to slash their WAN OpEx? How do these solutions work, and do they have legs? I’m hoping to find out.

What’s SD-WAN all about?

SD-WAN is about applying concepts of SDN to WAN networks. The goals are to increase flexibility and reduce WAN costs. This can be achieved through transport independence, dynamic path management, and better config management.

Historically we used private WAN circuits – leased lines, MPLS, etc. These had great SLAs, but the monthly costs were huge. The bandwidth was low, but guaranteed. Now that many places have access to high-speed Internet tails, it’s a lot harder to justify that cost. It’s very tempting to run IPSec VPNs across Internet links instead.

Those consumer Continue reading

Updated Big Switch Labs

Just a quick note to say that Big Switch have updated their demo lab system. This is an entirely virtual lab environment that simulates a Big Switch network. You can try out both Big Cloud Fabric and Big Tap Monitoring Fabric.

The lab gives you full CLI & GUI access to a sandboxed environment, with controllers, leaf/spine switches, and endpoints. Big Switch have written a sample lab you can work through, to show off the features, but you’re not limited there. You’re free to try out whatever features you like.

If you’re interested in what they’re doing, I recommend signing up.

Don’t Underestimate Your Users

The “consumerisation of IT” has an interesting side-effect. Historically people mainly used computers for work. But now that many people have smartphones, tablets and laptops at home, their perception and understanding of technology has shifted. Old assumptions about training required when upgrading applications or client operating systems may no longer apply.

This comment at The Register aligns with what I’m seeing:

…We’re at the point now where users are using Windows 8 at home and wondering why the work computer is so dated. It’s the perception of IT people that users can’t handle change holding up that change, not the ability of the users. At home that same set of users has managed quite well with updated versions of Office, updated Windows, iPads, Android tablets, Facebook, video messaging and various other completely new things. Somehow they coped without extensive training and therapy. From what I’ve seen, it’s actually IT staff who don’t like Windows 8 and are trying to keep users away from it…

I can recall being involved in Office upgrades just a few years ago, and being nervous about how that would be perceived. We were concerned that there would be major push-back, because the exact locations of the buttons Continue reading

NFD9 Prep: NetBeez

I’m reviewing the presenters for Network Field Day 9, in particular looking at those I’m not familiar with. NetBeez is one of those making their first Tech Field Day appearance.

NetBeez

We all know that our users and the applications they access are incredibly distributed. We don’t control all the network elements, but the network team still gets the blame if things go wrong. You need greater visibility to prove it’s not the network, but getting that visibility is tough. Current options for probes aren’t always cost-effective to deploy across many sites. Many sites don’t have any local server infrastructure.

That’s where NetBeez comes in. They have developed Raspberry Pi-based agents that can easily be deployed to many locations. Plug in power, plug in a network cable, and it phones home. Go to the NetBeez dashboard, and from there you can configure the tests you want the agent to run.

Since the devices are so small, they can easily be deployed to a range of small sites, and can simulate a range of user traffic. Tests include Ping, HTTP, Traceroute, DNS. A particularly nice feature is the ability to run an ad-hoc iPerf test with custom parameters.

The dashboard shows you how the Continue reading

Headwinds, or Uphill?

As some of my readers know, I’ve done a fair bit of bike touring. Two of the challenges of bike touring are riding uphill, and riding into headwinds.

Riding uphill is tough. 2,300m passes in snow, or 3,200m passes in sunshine, it’s tough going. But you put your head down, and keep turning the pedals, because you know that eventually you will reach the top, and there will be a downhill reward.

Riding into headwinds is a different story. You can battle into headwinds for days, and never get any reward. It saps your energy, and you don’t know if or when it will ever end. The wind could just keep coming from that direction. I’ve gone to different countries just to avoid the wind in the past.

They’re both hard. But one of them has an end, and a reward. The other one can just keep on sucking away your will to live.

Applying it to Life

This applies to the rest of your life. The tricky bit is that sometimes you don’t know if you’re going uphill, or into the wind. Both of them feel hard, and you can’t always see the end in sight.

A Continue reading

Network Field Day 9

I had a fantastic time at Network Field Day 8, and now I’ve been lucky enough to be invited back to NFD9 this February.

As usual, the Tech Field Day crew have put together a great mix of vendors. I particularly like the look of the SDN WAN-focused vendors, such as VeloCloud and cloudgenix. Much of the early SDN focus has been on the DC use-case, but that has limited applicability in my local market. SDN WAN solutions definitely apply to the New Zealand market though. I can think of several organisations where I’d love to have better WAN options today.

I’m also very happy to see Cumulus Networks making a first appearance.  I’ve done a lot of Linux work during my career, and there’s many times I would have loved to have all the capabilities of a GNU/Linux environment on a switch. I think they will have a huge influence on how Network OSes are delivered in future.

Network Management has always been a large part of my career too, so I’m looking forward to hearing updates from SolarWinds, and to find out more about NetBeez.

There’s some old faces and new attending. I’m looking forward to meeting people who I’ve Continue reading

Resources for learning HP Comware

HP is making more resources available to help with learning Comware. They’ve added free labs and courses to the already published simulators and virtual routers. This is a good resource for those looking to get started with Comware.

HP Network Simulator (HNS, aka Simware)

HP’s Network Simulator (HNS) is a modelling tool for simulating HP Comware networks. It includes Layer-2 functionality, and lets you test things like LACP & IRF. I found it too slow when I first tried it, but this has improved significantly with current versions. It is free to download.

HP has now started publishing simple labs you can work through with HNS:

• Lab 1 – Basic management
• Lab 2 – Port and link management
• Lab 3 – Port-based and protocol-based VLANs

These are short labs that cover HNS setup, and device configuration. Quick and easy, they show how to use the tool, and give you a taste of Comware configuration. They’ve also released a free 1-hour online course that goes through how to use HNS.

VSR1000

I’ve covered the HP VSR1000 previously. This Continue reading

Rate my IOS?

Review schemes are useful for identifying good consumer products and applications. But that doesn’t mean that everything needs to prompt me to leave a review. Cisco has started prompting for reviews for IOS versions, but I’m not convinced it makes sense for network operating systems. Perhaps it will do one day when disaggregated hardware/software is the norm for network devices.

Reviews for Consumer Apps – no problem

I love the 1Password password manager. It’s a well-polished app, and has been great value. Part of making my life better means not annoying me with frequent prompts for review:

1Password never prompts you for a review. We value your workflow too much to interrupt it. If you feel generous and have a couple of minutes, please leave a review. It means the world to us.

I like the Pocket app too. It prompts me to leave a review every single time it gets updated, which annoys the hell out of me. But hey, it’s free, so maybe I shouldn’t complain too much.

Pocket and 1Password are examples of consumer applications in a competitive market. The barrier to switching is relatively low, and they live and die on reviews. In a crowded market, customers rely on reviews, and the Continue reading

Big Switch Chaos Monkey Network Testing

Whenever you build a complex system, you need to test that it works as expected, including properly handling failures. It’s easy enough to do simple component failure testing, but harder to do rapid automated failure tests. Big Switch is showing that it can be done though. Hopefully we can keep improving our testing to pick up some more of the software failures.

Testing is hard

Over the course of my career I’ve built many clustered systems – HP-UX Serviceguard, firewalls, routers, load balancers, RedHat Clusters, etc. Good clusters have redundant everything – servers, power supplies, disks, NICs, etc.

The commissioning process always included testing. We’d go through each of the components, trying to simulate failures. Unplug each of the power cables, the network cables, unseat a hard drive, remove a hot-swappable fan, etc. That would test out the redundant components within each server, and then of course you’d simulate a complete system failure, forcing full failover.

This is all important stuff, but it doesn’t pick up all the failures – e.g. What happens if you’ve got a faulty patch lead, and the link starts flapping? Sometimes a simple failure gets messy when it happens repeatedly over a short Continue reading

Operations Manager to OMi Migration Path

HP has finally announced a migration path for Operations Manager to OMi. It’s about time too. This looks like a good path. If you want to stick with HP Software for managing your services, you should be investigating it.

The writing’s been on the wall for a while. HP has stopped investment in Operations Manager. I asked last year if HP had abandoned Operations Manager. This year I noted that it was kicking, but only just. My advice was:

To customers using HP OM…start planning your migration away from it, if you haven’t already. To customers considering purchasing it: Don’t, unless you’re buying it as part of an overall BSM/OMi implementation, and the salesfolk have guaranteed you can change your licensing over at no cost in future.

Well, HP has finally announced the OM2OMi Evolution program. Key points:

• License entitlement – OM servers can get equivalent licenses for OpsBridge Premium
• Operations Agent 11 works with both OM and OMi, so you don’t have to do the Agent migration at the same time
• Migration tools to assist with switching over

They do include this quote:

Well no one at HP is going to try to force you into replacing a product you love. Rest Continue reading