Author Archives: lindsay
Author Archives: lindsay
I’ve given in to the Sunk Cost Fallacy once more: I’ve renewed my CCIE. There was a lot of foot dragging this time around, and I only had four months to spare. But it’s done, for another year. Here’s some quick notes on my prep, and thoughts on the exam.
I decided to sit the CCIE R&S Written Exam to renew. This was the easiest route for me. I don’t use Cisco products on a day to day basis, so certifying with a different track would be very hard for me.
The version hasn’t changed since the last time I sat it. It’s still 400-100, v5.1. The only difference is that the “Evolving Technologies” section has been tweaked a little. Think Automation toolsets, Cloud concepts, etc.
I used the study guide I purchased last time from “CCIE in 8 Weeks”. I also re-subscribed to their online practice exams. I meant to only subscribe for 3 months, but…I couldn’t get motivated to do this exam. I ended up paying for another 3 months access, before I finally knuckled down and did the study while I was on vacation. I flicked through my old CCIE flashcards a few times too.
Using network CLI for automation has always been fragile. But it keeps surprising me with the way it breaks. This time, it was a combination of Ansible, Arista,
replace: config and
terminal length used as a config command.
I often hang out in the NTC Slack channel. A user reported they were having a problem with Ansible and EOS. Basic changes worked, but when they used eos_config with the
replace: config option, it just timed out. We knew basic authentication & connectivity was fine, it had to be something else.
But it made no sense, because these modules are widely used. What’s going on?
Some commands produce more than one screen’s worth of output - for example,
show run can be hundreds of lines long. Most screens don’t have hundreds of lines, so pagination is used. The network Continue reading
We are now Lawful Permanent Residents of the United States - aka Green Card Holders. It took a few years to get to this point. Here’s our timeline, why we did it, what it means for us, and what next.
I first moved to the US on an L-1B visa. This is an intra-company transfer visa, that let me move to the US to continue working for Brocade.
Total Continue reading
I use a Privacy Filter on my laptop screen when traveling. I’m doing a bit of time on planes these days, and it makes a big difference. Most of my code is Open Source, but other content is proprietary. High chance of competitors being on the same plane as me, so better to make it harder for others to see.
The only problem with these screens is that if you frequently take it off like I do, the adhesive strips collect dust, and stop sticking after a while. Recently someone asked me how to get them replaced.
3M does not sell replacement strips…but they do something even better: they give them away for free. Pretty cool ah?
Just go here, fill in the details, and they’ll send you some more. How good is that?
We have left the Bay Area, and headed North. We have moved to the Greater Seattle area - specifically the Eastside, between Bellevue and Redmond. We’ve given up the old apartment in San Francisco for a larger, nicer house…for a lot less in rent. A lot fewer bars & restaurants, a lot more trees, parks and lakes.
The typical Bay Areas response is: “But why??? It rains all the time in the Pacific Northwest!!!!”
A few things:
1. Yes, it rains more here than San Francisco, but not as much as people think. It’s not even in the top 10 cities in the US for annual rainfall. Boston, New York, Washington DC all receive more.
2. Rain is OK. In fact rain is good. You don’t get lush forests through irrigation. You also don’t get clean streets just from street sweepers.
The main attractions for us are:
Much better lifestyle for us. It’s easy to go mountain biking, running, hiking, skiing here.
Much lower rent. Yes, rents have gone up a lot here, but it’s still much better value than San Francisco. I pay much less rent here, but I get a nice place, and the Continue reading
I have been battling to get the combination of CircleCI, Docker and systemd to play together. After much frustration, I have a workable solution. Machine Executor,
privileged: true, cgroup passthrough, and disabling AppArmor.
In the StackStorm team we use CircleCI with most of our repositories. We check things like code style checks, and run unit tests. With every Pull Request we trigger these checks, and checks must pass before merging. Some repos also use CircleCI for post-merge deployment steps.
We use Ansible and Terraform to manage some of our internal infrastructure. All configurations are stored in Git. All changes to that configuration must be submitted as a Pull Request. All PRs need approval, and all commit checks must pass. We use CircleCI to run these commit checks.
We run multiple checks, but for Ansible playbooks, they include using
ansible-playbook --syntax-check. We then spin up a Docker container using CircleCI and run some of our playbooks twice, checking that it passes both times, and that the second run records no changes.
Here’s a snippet of some of our CircleCI configuration:
version: 2 jobs: build: working_directory: Continue reading
We published Ansible modules for Extreme SLX devices earlier this year. Now we have modules covering all the main Extreme Switching & Routing product families - SLX, VDX, MLX, EXOS, VSP.
All modules are available in the current GA version of Ansible (2.7), except for
voss_config. That one proved a bit trickier for me to write, and I didn’t get it done in time for the 2.7 cutoff. That one is an open Pull Request against the Ansible
devel branch. That should get reviewed and merged soon. It will then make its way into the next GA release. You can of course use the code direct from my branch in the meantime.
All modules use the
network_cli plugin. See Platform Options for general information about how to use this connection type.
Thanks to Continue reading
This year I’ve written several Ansible modules. It wasn’t that hard, yet some people claimed they had been waiting “years” for those modules. There was nothing stopping anyone else doing it, yet they hadn’t. There’s a weird reticence amongst network engineers to learn or write any code, even when it could make a large difference to their job. People either do nothing, or they create complex Ansible playbooks to work around their reluctance to write Python. It’s not that scary. Why don’t people put in a bit of effort?
Ansible playbooks use YAML, a somewhat human-readable markup language. These are instructions for “what” Ansible should do - e.g. “Use the Cisco ios_config module to ensure that this configuration line
The underlying modules use Python. These are the “how” - they take the instructions from the playbooks, and turn those into device connections to devices, making configuration changes, checking state, etc.
Some people look at these modules as a mystery black box that only the vendor can write. They think that the only way they can interact with Ansible is via playbooks.
This leads to two situations:
1/ Twiddling thumbs Continue reading
It is 6 years since I passed the CCIE Lab Exam. The dreaded email has arrived:
CCIE: Your CCIE status is ‘suspended’ and you need to recertify in twelve months.
Time to re-evaluate what the CCIE means to me. Should renew it? Should people start out on the CCIE track now? My opinions have shifted over the years.
My career has shifted over the last few years. I work for a Network Vendor, but networking is only part of what I do. I am a Product Manager, focused on automation. I spend very little time looking at network devices, or CLI. I spend my time talking to customers, updating roadmaps, writing Python, reviewing Pull Requests.
My future will be working with technologies like Serverless Computing, IoT, and Edge.
CCIE R&S doesn’t cover any of that.
It is unlikely that I will ever work as a traditional hands-on network engineer again. Not impossible, but unlikely. I doubt that any future employer will care about whether I have a current CCIE certification. At this point my experience Continue reading
Too many IXPs (and networkers in general) are using horrible outdated methods of graphing data. These are an ugly eyesore, and should be updated to something from this century. Big IXPs in particular have no excuse: they have the resources to do better.
This caused a bit of a stir. As El Reg put it:
[According to Dave Temkin] The internet exchange industry is ripping customers off, charging too much for features people don’t need, and spending millions on staff salaries, unnecessary marketing and social events.
You can argue amongst yourselves as to how much IXPs should invest, how closely their port prices should track transit costs, etc. Or maybe you just like all the free drinks, dammit.
I think that if they’re going to spend money rather than reduce prices, they should spend it on something I care about: Data visualization. Most IXPs traffic graphs are an eyesore, they’re outdated, and it’s time they were fixed.
Here’s some typical traffic graphs from some of the biggest IXPs in the world:
But , of course there’s the usual herp-derp comments, and a big spike in moving repos to Gitlab
We're seeing 10x the normal daily amount of repositories #movingtogitlab https://t.co/7AWH7BmMvM We're scaling our fleet to try to stay up. Follow the progress on https://t.co/hN0ce379SC and @movingtogitlab— GitLab (@gitlab) June 3, 2018
Most of those repositories will be inconsequential single-user repos, but it is still so much wasted effort. If your knee-jerk reaction is to immediately stop doing real work, and move your code somewhere else, you haven’t been paying attention. The world has moved on.
Back in 2014 I wrote Keep an Open Mind:
I get frustrated because these people aren’t paying attention to what Microsoft has been doing. They have come a very long way since the 2002 Bill Gates email setting security as the top priority. It’s a big ship to turn, and it took time. Their overall security model and practices are far better than they were, and stability is no longer an issue. Their business strategy is very different now too.
But poor Continue reading
I hate long support lifecycles for hardware and software. Yes, you should be able to buy a new iPhone or switch and use it for 3+ years. But some people want 10+yrs of support, and wail and moan when vendors end support. This is wrong. It drives up costs & complexity, and makes your systems less robust, not more. It’s a false sense of security. Plan to buy smaller & cheaper, and upgrade frequently.
Vendors don’t want to do long support lifecycles. They will do them, because people pay for it, but there comes a point where they put a line in the sand. “Sorry, that system is now EoL.”
Costs: Testing software and hardware combinations is hard work. Add many years of released hardware & software combinations, and it gets much harder. More racks of gear & more permutations == more costs.
Complexity: It’s hard enough to test against a small set. But now you have to deal with obscure systems acquired from a third party 7 years ago? Complexity == time and money.
Motivation: Hands up who wants to work on legacy systems? Exactly. It’s hard to motivate engineers to support Continue reading
I’ve seen a few Twitter threads recently about learning to live with the sudden plenty of working for tech companies. If you didn’t grow up that way, the adjustment takes time. It made me think about a few things I’ve learnt about corporate travel, and mistakes I’ve made along the way. People who grew in the corporate world instinctively know stuff I had to learn. Here’s some of the mistakes, and learnings:
There’s been a few threads recently on Twitter related to the concept of “growing up poor, and learning how to adapt to working in well-paid industries.”
Here’s an example thread:
Starting a conference for people that work in tech but grew up poor. Talks:— Mikeal Rogers (@mikeal) May 1, 2018
* Low fee ways to send money to family.
* Making yourself feel ok about spending $50 on brunch.
* Where people with money store their extra money instead of just leaving it in your checking account.
Read the thread - there’s some gems in there. Stuff like these hit home for me:
my first week at my job i spent $30 at the safeway to feed myself all week until they Continue reading
Cloud computing is a lot more than “someone else’s computer” and it annoys the hell out of me when people keep trotting out this tired old excuse. There is much more to service delivery than compute power. You do yourself and your customers a disservice if you don’t do your research.
A few years ago it was fashionable to dismiss cloud as “just someone else’s computer”, e.g.:
There is no cloud. It's just someone else's computer. pic.twitter.com/9d3S5chWQq— David Whittaker (@rundavidrun) February 13, 2016
You can even buy coffee mugs.
In a time when most cloud computing was Infrastructure as a Service, there was an element of truth to it. But…
The problem is that there’s still people thinking this. Check these recent tweets.
These people don’t realize that the world has moved on a long way. There is much more to cloud computing than just “someone else’s computer.”
Consider a simple example, like email. To provide email services from “my computer” I also need power, cooling, rack space, servers, storage, networking, operating system, software, application configuration and maintenance, etc…not to mention the operational expertise to keep it all going.
Here’s something I’ve been working on recently: Ansible modules for Extreme SLX switches & routers. Ansible is a popular automation framework, and with good reason: it has a low barrier to entry. Time to usefulness is short. But you need device-specific modules to work with networking devices. Finally we have some modules for SLX. Read on for how to use them.
This blog is not an intro to Ansible in general. There’s plenty of good intros out there. This is specifically about demonstrating Ansible with SLX switches.
Ansible is an agent-less configuration management system. It uses “playbooks”, written in YAML, to define desired configuration state. “modules” written in Python translate this into whatever is needed to configure the system, application, database or network device.
Ansible has been making great strides in adding network automation capabilities. But we haven’t had any modules for working with ~Brocade~ Extreme devices. That is now changing.
PaulQuack has contributed MLXe (Ironware) modules, which will go GA in Ansible 2.5 (due for release in March 2018). And I’ve been working on modules for the SLX, with my colleagues. These have not yet been merged upstream, but it’s Open Source, so you can grab Continue reading
It takes a village to raise a child. Or so the old saying goes. Creating a product is the same. It takes more than small group of developers (or parents) to raise a product. There’s a lot more to creating a product than writing an application. Don’t mistake a feature or application for a product.
People hear about new applications or protocols, or small companies selling for millions. They then leap to conclusions:
“Why is it that big vendors like Cisco need thousands of people to create a product? Facebook can put 6 engineers on a project and produce something like Open/R. It’s easy, right? We don’t need big vendors any more!”
“Look at Instagram - they only had a dozen people and they sold their company for $1 billion dollars! You don’t need any more people than that.”
Let’s look a bit closer at Instagram. How much revenue did they have? Zero.
How long were they in business for? A couple of years. So how many generations of product were they supporting? One. And did they have complete support structures for users? No. How many products had gone through end of Continue reading
Two years ago I wrote about how to use InfluxDB & Grafana for better visualization of network statistics. I still loathe MRTG graphs, but configuring InfluxSNMP was a bit of a pain. Luckily it’s now much easier to collect SNMP data using Telegraf. InfluxDB and Grafana have also improved a lot. Read on for details about to monitor network interface statistics using Telegraf, InfluxDB and Grafana.
There’s three parts to this:
Grafana: Grafana is “The open platform for beautiful analytics and monitoring.” It makes it easy to create dashboards for displaying data from many sources, particularly time-series data. It works with several different data sources such as Graphite, Elasticsearch, InfluxDB, and OpenTSDB. We’re going to use this as our main front end for visualising our network statistics.
InfluxDB: InfluxDB is “…a data store for any use case involving large amounts of timestamped data.” This is where we we’re going to store our network statistics. It is designed for exactly this use-case, where metrics are collected over time.
It is now almost 12 months since the first announcement that Broadcom was to acquire Brocade, and sell off the IP parts of the business. It took another 6 months to get confirmation that Extreme Networks would be buying my business unit (SRA).
For regulatory reasons, the Broadcom/Brocade transaction has still not closed. The original plan was to close that deal first, then close the Extreme transaction. But due to the delays, they re-arranged things, and now the Extreme deal has finally closed. Desks have been cleared, moving crews are working all weekend, and come Monday, I will have a new “Extreme Networks” badge.
What does this mean for me? My group is moving to become part of Extreme Networks. In the short term, I keep working with the same core group of people. But now we will be part of a new wider group, with a different strategic focus.
We will have new systems and applications to integrate StackStorm with, new use-cases, and maybe further opportunities beyond StackStorm. So far all signs are pointing to this being a positive move, and I am looking forward to getting this transition behind us.
There’s a lot of angst in the networking community about programming, SDN, automation, and what it means for networking careers. Plenty of people will tell you don’t worry about it, focus on the fundamentals, there’s plenty of work, you will be fine.
There is some truth in that. There are still lots of jobs in networking. People with solid skillsets should have no problem finding a good job.
Don’t fool yourself. Things are changing.
I’ve seen some research from Gartner that indicates that organisations have been steadily decreasing their Network Operations teams over the last five years. They have also been reducing their Data Networks spend. (Sadly I don’t have publication rights for this research, so you’ll just have to take my word for it).
This is going to put pressure on networking engineers. Your role will be forced to change, if for no other reason than that you are going to have less budget, and fewer people to do the work.
So you’d better think about what that means for how your role might change.
Everything comes at a cost: steak dinners & pre-sales engineering has to get paid for somehow. That should be obvious to most. Feature requests also come at a cost, both upfront, and ongoing. Those ongoing costs are not always understood.
It’s easy to look at vendor gross margins, and assume that there is plenty of fat. But remember that Gross margin is just Revenue minus cost of goods sold. It’s not profit. It doesn’t include sales & marketing costs, or R&D costs. Those costs affect net income, which is ‘real’ income. Companies need to recoup those costs somehow if they want to make money. Gross margin alone doesn’t pay the bills.
A “four-legged sales call” is when two people show up for sales calls. The usual pattern is an Account Manager for the ‘relationship’ stuff, with a Sales Engineer acting as truth police. These calls can be very useful. It’s a good way to talk about the current business challenges, discuss product roadmaps, provide feedback on what’s working, and what’s not. The Sales Engineer can offer implementation advice, maybe help with some configuration issues.
Often a sales call includes lunch or dinner. Breaking bread together Continue reading