iNOG-10 & RIPE-Hackathon
In June 2018, I was lucky enough to attend the iNOG 10 session in Dublin, co-present a talk and also take part in the RIPE hackathon.
This post is a share on the experience. This isn’t because I’m running out of non-technical material, but this is to uncover both events for those that may want to attend, take part and experience what these kinds of sessions offer.
iNOG
The iNOG Irish Network Operators community surfaced briefly with events in 2005 (originally as the IENOG) but fell silent and was reborn in 2015 as the organisation
as it is today. Started by five returnees to Ireland and some economic migrants, the group has been seeing a high number of attendees to the events and over 700 members on Meetup! Not bad for something that came in on a started on a boat!!! (See below).
The group aims to deliver valuable content to the audience free of charge. Whilst ‘valuable’ has a variety of meanings depending on the audience, the general idea is to share experience of network based activities. As you can imagine, this is very wide ranging and just in the iNOG 10 session, talks were given on automation, data Continue reading
Automation for Success
Businesses with high growth, complex tasks and repetition, tend to rely on or require automation to fulfill business challenges. Introducing automation is not without challenges and sometimes they can be quite significant. Identifying success is one of the early crucial activities that creates a business alignment. The identification exercise highlights justification for one more decisions and the removal of friction. Some of the decisions are not easy to make and friction is not easy to experience without applying pressure to various parts of an organization.
What follows is a number of scenarios with some reasoning around the kind of challenges that you’re already facing or likely to face.
If the absolute reasons are known, accepted and aligned against, you have just laid one of the foundational layers for success.
Challenge: High Growth
Great news and bad news. You’re in a business under stress from high growth! Lots of great challenges to solve and high pressure from not having them currently solved.
With high growth businesses, engineers or administrators are not under threat of being “automated out of their job”. If you identify as one of these people, you have an opportunity to learn new skills, be rewarded for finishing projects Continue reading
Automation for Reliability
Statistics says, the more often you do something, the higher the chances of a negative event occurring when you do it.
Applying this revelation, if you fly regularly, the chances increase of a delayed flight, or being involved in an incident or accident. A somewhat macabre reference perhaps.
Let’s take something real which happened to me this week (11th June 2018). Whilst working out of one of Juniper’s regional offices, I returned back to the hotel room to carry on working whilst putting my feet up. Something felt strange in the room but I couldn’t put my finger on the weirdness. After a couple of hours, I realised that all of my belongings were gone from the room. Everything! Thanks to a mix-up with the house keeping system, the maids threw my collection of travel items in to some bags ready for disposal. Thanks to a procedure that the hotel operates, for my items to be thrown to the garbage, a manager is required to sign off on the request. A process saved my belongings and I’m thankful that the managers knew this process and also knew where my stuff was likely to be. Before my items were returned, I Continue reading
Juniper vQFX10K on ESXi 6.5
A quick and dirty post on running the Juniper vQFX on VMWare ESXi.
You might be wondering why ESXi seeing as we’re all cloudy types. ESXi is purely a case of laziness. Each server in my control has ESXi 6.5 installed. This becomes tin management at the most basic level.
Part of my home network has a DMZ which has several public IP addresses and I expose systems and VNFs externally over the internet. More recently thanks to the IP fabric craze, part of what I’m exploring is easy integration and feature enhancement on Juniper vQFX instances. Two choices exist:
- Install vQFX on servers with KVM
- Install on ESXi
I went for the easy ground (because why make it harder than it has to be?) Turns out, it wasn’t as straight forward as it should be, although not difficult. Just a niggle.
Installation Process
Grab yourself the RE and PFE images from the Juniper download site:
https://www.juniper.net/support/downloads/?p=vqfxeval I Grabbed the 18.1 RE and the 17.4 PFE image.
Next, extract the two
.vmdkfiles from the
.boxfiles. You can use the trusty tar tool to extract the files required. Below are two files Continue reading
Configuring SSL for gRPC on Junos
This is a short article on creating a self-signed root certificate which can be used to self-sign certificates for the purposes of treating our telemetry and instrumentation exploration with the security love it deserves. I also cover configuration of mutual SSL for gRPC on Junos. An article of dual purposes!
One of the things I see far too often is clear-text transport being used in demonstrations, labs and even production. This isn’t acceptable. We live in a world where security has to be woven in from the ground up. How do you really know your system works if you leave out all of the security controls?
I hear your teeth grinding. Why do you want to do this? First of all, even though we can bypass security on gRPC with Junos by going for insecure connectivity with clear-text, we shouldn’t. The world we live in is all about the data and the smallest amount of it can give the ‘bad guys’ a lead.
Now we’re done with the why, we need to deal with the how. There are three approaches to PKI that are common:
- Run around with your hair on fire rambling nonsense
- Create your own Certificate Authority (CA) Continue reading
Describing Network Automation: Automate the Coffee
How to Describe Automation
Cisco Live, Milan, 2014, the place where everyone drinks a caffé! It was this year that Cisco’s DevNet began to grow and my passion for software, automation and networking was in for a roller-coaster ride. I watched various refreshment stands delivering coffee to the endless queues of guests and began to see something special in the thing that I originally called an espresso!
For so long we’ve used pipes and water to describe networking itself and for a long time I was hunting for a good way to talk about network automation. Turns out a caffé is a great way to describe automation and especially network automation. We also feel emotionally about it and understand the process used to have one placed in ones hand.
Annoyingly so, when automation is the topic up for conversation, we start with "Let’s automate the network" and not with what it is we want to automate. If you’ve raised your eyebrow, point in case. Even worse is when you’re asked for a use-case. The answer is nothing more than a reflection: "Tell me what your humans do". This isn’t a product, it’s the deep integration of human process and digitised Continue reading
Juniper JET & Golang
Network programmability and network automation go hand-in-hand (pun intended) and I’ve been waiting for an opportunity to play with the Juniper IDL (.proto) files to build a JET (Juniper Extension Toolkit) application. Thanks to Marcel Wiget’s efforts, the opening I’ve been waiting for came along!
So what is JET?
JET is a couple of things:
- Ability to run Python, C and C++ applications onboard both veriexec and non-veriexec enabled Junos
- Ability to create an off-box application using GRPC and MQTT
JET allows you to program Junos out of the normal NETCONF, CLI, SNMP and ephemeral DB methods that we’re all fairly used to. The other thing is, it’s quick. Like really quick. With GRPC and MQTT, we can program a network element using mechanisms the software world is used to. I’ve been saying for a long time our data is no longer our own and JET allows us to bridge organisational worlds in multiple ways. Pretty cool.
So what did you do?
Not having a huge amount of time for this, I opted for off-box and took Marcel’s code as the base for how to use the APIs exposed via GRPC.
The application uses the “bgp_route_service” JET API Continue reading
Automation: Easy Button vs Sentient Voodoo Magic Button
Automation has become this “all-encompassing thingy” much like SDN. It’s a software industry problem and it’s critical more now than ever that we do not slip backwards by trying to drag a broken idea forwards.
This post contains nothing new and should act as polish on common sense. If you’re on the look-out for removing pain and getting stuff done with the power of automation, read on.
If your processes and operating handbook for your team or organisation is in disarray, it will not come too much as a surprise when your automation team implements something inherently broken. Naturally the technology, shortly followed by the team will take a boat load of blame. Whilst artificial intelligence and machine learning is promising, unless you have a team of subject matter experts or have very deep pockets, automation today is simple and the patterns are hard wired. Even decision making logic has been should be pre-thought. Automation platforms do not today think creatively and do not possess sentient capabilities. If they did, I would be on a beach right now drinking mojitos, smoking cigars and wondering what to do with my time on this rock (the answer by the way would be Continue reading
Ubuntu 16.04 git lfs broken install
When something simple goes wrong, frustration is never proportionate to the anticipated ease.
On Ubuntu 16.04 on a Fresh install, I came across this cracker.
sudo apt-add-repository ppa:git-core/ppa Cannot add PPA: 'ppa:~git-core/ubuntu/ppa'. ERROR: '~git-core' user or team does not exist.
Ok. Must be a typo. After spending twenty seconds or so on this, which is still way too long to be looking for typos on something so simple, it isn’t a typo.
This information comes straight out of the
gitcommunity. Pun intended; what a git.
After cracking some knuckles and blowing warm air on my finger tips, I went on the hunt for a fix.
Most of the Google results for this issue gravitates around proxy devices interfering with the traffic. Being on corporate wifi, this is entirely possible.
After tethering to my cell phone, rebooting the OS and clearing out some caches, the issue persisted. Adding the sources to
aptmanually also failed.
There is always more than one way to skin a cat and in my case, installing via package cloud worked.
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash sudo apt-get install git-lfs
In this world of virtualisation and micro-services, things like this are Continue reading
A Christmas Support Story
Warning: Non-Technical Post
As it’s the festive period and this time of the year is for caring and sharing, here’s a short story from many years ago. This might make some chuckle, but some of these times were not pleasant and I can assure you, they were very real!
Like most IT related people, I started in support. The job paid peanuts, it was shift work and I had much to learn. Being quite eager to please, many mistakes were made and in these cases seniors were supposed to help the younglings (like me). For some companies, a functioning support network just isn’t there and low rank power struggles leave you fighting fires a la solo.
Within the first three months of the job, I experienced two major backhaul fibre outages, a group of people stealing our generator power cables and the air conditioning system failed to the point of meltdown. We also had a total power outage which took 40 hours or so of non-stop work to get everything back online and healthy.
These kinds of experiences make or break you. The phones do not stop ringing (at least when the power is on) and customers rightfully do not Continue reading
Network Automation Engineer Persona: Proof-of-concepts
If you’ve chosen to do a proof-of-concept (PoC), you should already know what the challenge or requirement is, what your satisfactory results look like and what the product or tool set is that will deliver on your commitment. A proof-of-concept is a recipe that should give you a well baked set of results.
Approaching a PoC
Approaching a PoC is a conscious decision to trial an idea that if successful, your business will put in to daily production to satisfy one or more business requirements.
So why aren’t the results more apparent in our day to day lives? Why do we not see these decision points more transparently?
Some people approach PoCs as a tyre kicking exercise, or a means to get a vendor to commit to them and then use as leverage against cost. If you’re just playing, you’re wasting time. If you’re tyre kicking then be prepared for lack of commitment or interest in the future from individuals or organisations. Genuine PoCs and evaluations are a normal and acceptable part of business, so the responses are different from those assisting with PoCs.
Approach a PoC with clear intent and understanding of the challenge that faces you. You must Continue reading
Network Automation Engineer Persona: Targeted Learning
The series of Network Automation Engineer (NAE) Persona blog posts have churned some comments in the community around learning. Part of the feedback appears to be the hero syndrome fighting back and also the odd misunderstanding, which might be a result of fear. Change often results in fear, so this is natural.
Network Heroes
Some people love networking because it makes them feel special. Some people have got used to that special feeling and hang on to the fact that they’re
important. These network engineers feel like a “Packet Lord”.
Automation is designed to remove from humans deterministic and testable tasks. One result is fewer self-titled “Prime minister of Packets” and fewer bottlenecks. Taking IaC (Infrastructure as Code), it becomes so much simpler to define tasks, implement them and test for success or failure. Dealing with sources of truth is part of the natural flow of the process instead of an afterthought on a Friday afternoon.
The hero very much becomes the norm at this point.
Articles
At a high level, the articles discuss the evolving NAE persona and not what you should or shouldn’t learn. The articles do not discourage learning, they recognise and promote learning. After all, you Continue reading
Cloud Native: Upgrading a Workflow Engine or Orchestrator
On a train this morning, I read Ivan Pepelnjak’s Twitter stream (because what else is there to do whilst relaxing with a coffee?), I came across this blog post on upgrading virtual-appliances.
Couldn’t agree more with the approach, but what about upgrading a workflow engine or orchestrator? I’ll call this entity a ‘wfeo’ just to make typing this article easier.
The perceived turmoil in undertaking this kind of an upgrade task is enough to make new born babies cry. Fear not. Any half decent wfeo contains it’s gubbins (workflows, drivers, logic, data) in a portable and logical data structure.
Taking StackStorm as an example, each integration (official parlance; ‘pack’), is arranged into a set of directories.
Within each directory are more directories with special names and a set of files like READMEs, configuration schemas and pack information. These top level directories that contain the pack, are portable between install bases of StackStorm giving us the power to easily clone installations, repair logic in case of a troubled upgrade and install logic freshly for new installations.
As with any platform, some syntax might change so always read the release notes for the platform and packs.
Ivan’s point is that you Continue reading
Network Automation: Leaky Abstractions
I hear people talk about leaky abstractions all the time. I’m not sure that some of the people that use it have researched the term.
As network-automation blurs the line between software and networking, terms like this are used more commonly than you might expect.
When you hear someone say ‘leaky abstraction’, what does it really mean? This question drove me to a little research effort.
The term ‘leaky abstraction‘ was popularised in 2002 by Joel Spolsky. I totally misunderstood this statement when I first heard it, so naturally the researcher in me went off trawling the web to get a more correct view.
My original and misinformed understanding is explained in the example below.
The Example
Taking the example of a car, the abstraction interface or vehicle controls allows a user to manoeuvre the vehicle between a start and end point whilst keeping the passenger as comfortable as possible.
A car has air modification capability, human body heaters and it can even project audio to your ears. Most vehicles have an on switch (engine start or power switch), they have directional and velocity controls that come in the form of a steering wheel, a set of pedals Continue reading
Network Engineer Persona: Part Four
Part three introduced the first three key skills. This part presents the introduction to the last three core skills and a call to action.
Key Skill Four
I’m trying very hard to refrain from using the term DevOps, but the fundamentals of the DevOps movement are super important. The DevOps fundamental pillars are improving the flow of work, improving the quality using a feedback loop and sharing. A huge array of books have been created on the topic of DevOps in addition to blog posts and podcasts. If we view the persona of the Network Automation Engineer through the lens of the DevOps persona, the two are very similar. If we are to increase the flow of tasks and improve the quality of them using automation, then we need to be able to fix the issues close to the source of the problems and share knowledge. We do that with logging and an attitude change. Logging is critical for successful automation projects as well as attitude.
Knowing how to transmit logs, how to capture logs, how to sort through them and how to realize events from them is an entire skill. There are software stacks dedicated to this mission like Continue reading
Network Engineer Persona: Part Three
Part three! Let’s get straight to business and carry on where we left off from part two.
Key Skill One
Thinking about automation in an agnostic way is your first footstep. Automation is about data flowing through building blocks that do things and decision points, allow you when to do things.
Removing CLI and replacing it with an abstraction layer isn’t much of a win. For instance, I regularly talk about the process of creating a VLAN and applying it to an Ethernet switch-port on a tagged interface. This somewhat simple ‘workflow’ creates more conversational friction than imagine-able. Let’s work through it.
Task: Create a VLAN
This task requires domain-specific parameters to a VLAN. These are: ‘VLAN_Number’ and ‘VLAN_Description’.
Task: Apply VLAN to Switchport
This task requires domain-specific parameters to a switchport. These are: ‘Port_Name’ and ‘VLAN_Number’.
Note how the inputs flow through the actions within the workflow?
The green arrows descending illustrate the ‘success transition path’ for each action component.
So, what about these questions?
1. Is the VLAN in use?
We can be more specific here, but it adds complications to the answer. Version two is: “Is the VLAN in use in the network zone that the device Continue reading
Network Automation Engineer Persona: Part Two
This article is number two in a series. The first part can be found here.
There has been a thought trend in the last few years leading network engineers to think they need to be developers. This is totally nonsense. When we want to learn a new skill, there is a precursor which says “I want to do X, so therefore I need to learn about what X”. If you’re thinking “I should be learning Python”, I ask to what goal? What is making you ask this question? Maybe the question should be, for a network automation engineer role, what skills do I need to learn? Stop guessing!
The Network Automation Engineer role combines deep network knowledge, with the ability to describe, collect and transmit domain specific data through one or more abstraction layer type components. It requires knowledge of how to collect data from databases and data-stores of various types. Where does a list of IP addresses get stored? How are they stored? How are they retrieved? The role requires an awareness of the cause for making a change and the implication of making them. Gaining the skills to become this persona isn’t a full career change, but a Continue reading
Network Automation Engineer Persona: Part One
This is the first post of a series on the persona between “network engineer” and “developer”. This role does not exist in myth, but it is a natural evolutionary step forwards. This position inherits skills from both ends of the scale, but in itself is an emerging role in organisations globally.
Why describe personas? They are important because:
1. They provide a roadmap for a career
2. They provide a set of skill requirements to master for a role matching the persona
3. They provide a set of tool consumption and usage hints
4. They provide a viewing glass to defining thought processes
Some of the challenges both vendors and network technology consumers are facing today are related to the set of evolving personas in our field, therefore it’s crucial to understand them properly. Remember when you wanted to be a network engineer? You became the persona and worked your way through a set of learnings. Your thoughts and habits changed, along with your recognition and self awareness.
Evolution of Roles
Every industry evolves and some industries disappear. The need to move packets about on the Internet hasn’t evolved out of existence just yet; our current highly generalized reality is: Continue reading
Declarative and Imperative Automation Thinking
Event driven or workflow driven automation for the uninitiated can appear daunting. With the rise of machine learning which adds more complexity to our field, simple rule driven automation feels more in reach than it ever has. This post aims to introduce you to a viewing lens for the world of great automation.
Converting processes in to workflows can be a tough task to accomplish and whilst this is relevant, it isn’t for this post. That’s one for another day but a great talking point and one that will be addressed.
The Layer Cake
Good automation tooling offers two views of the world; one high level that you pass arbitrary data around and one that handles interface implementation, which converts the arbitrary data to meaningful contextual data. In other words, a declarative and an imperative view, the ‘what’ and the ‘how’. The dimension that exists between the two can make or break the tool. If we have to care too much about how data is passed from one layer to the other, we’re not users, we’re more co-developers and it’s a kit not a tool.
Our Layer Cake model is simple at a high level. Two layers are enough to Continue reading
Automation and State
To State or not to State
State
State its purpose.
That is a state matter.
It was a messy night. Alice was filthy. She was in a state.
The router had some state.
What to do with state
Automation is a constantly changing state of affairs. It raises questions like:
a) If a service or API is idempotent, do I have to track state?
b) Should my workflows consider external state?
c) Should I normalise state?
Idempotency
Something is said to be idempotent if it gives you the same response if you call it repetitively.
I always view idempotency quite simply.
- Bob makes a terrible mistake building a NETCONF server for Alice.
- Alice punished Bob with the task of hoovering the office floor.
- Bob starts hoovering at 3pm when Alice is out of the office.
- Alice Tweets Bob mid-hoovering (because they’re millennials) with “Bob, it’s time to hoover.”.
- As Bob is idempotent, Bob carries on hoovering and ignores Alice’s Tweet.
If Bob wasn’t idempotent, he might have packed away the hoover, gotten it back out and started hoovering again (also assuming Bob was actually delivering on his punishment and hadn’t outsourced it to a cleaner).
Can you imagine the Continue reading