There was a recent blog by Mark Burgess, founder and creator of CFEngine. It is a must read (on his personal blog).  He really makes you think where we are as an industry, question if we are on the right path, and quite frankly calling out certain technologies as pity attempts compared to what is needed. Regardless of all that, we cannot forget one key point, the industry is in fact moving forward right now. 

As I read his post, I remembered a conversation that I saw on twitter not too long between John Willis, Lori MacVittie, and Joe Onisick. It was more or less on the intersection of SDN and DevOps. 

The article by Burgess and the Twitter conversation really got me thinking. 

When you combine these interactions, even at the highest levels, you have to wonder, what is the right approach from both a vendor (product) and user standpoint? Are we on the right path? Do we have it all wrong? Must we fail first before getting it right?

SDN Will Simplify

SDN controllers will totally simplify things, but even then, the APIs they expose are arguably too low level for the average consumer of Continue reading

Over the past few months, I’ve been posting on using Ansible for network automation.  Changing things up a bit, this post will cover using Ansible for server automation and I’ll share a few Ansible playbooks that I’ve built and have been using to bootstrap servers and prep them for various applications such as OpenStack and NSX deployments.  

Step 1 - Playbook 1
Creating password-less root account

Since Ansible uses SSH by default for connecting to the servers, you will realize the first thing that needs to be done is to copy the public key of where you will execute playbooks from onto the “new” server.  To do this, I use a playbook that is called server_one_time_run.yml.  You will notice that in this playbook, and only in this playbook, I have remote_user set to jedelman and sudo set to yes.

I’ve been testing against bare-metal and virtual machine installs using an Ubuntu ISO image.  During the OS install process, “jedelman” is the account that was created on all hosts and virtual machines. 

This playbook runs and copies over the public key in the root directory.  We are essentially creating a password-less login for Continue reading

Over the past few years, I’ve had the opportunity to work with best and the brightest in the industry.  The reach started with my co-workers, partners, and vendors, but gradually expanded due to the likes of maintaining a blog and occasionally being on Twitter.  In a recent exchange with someone who gave me a massive pivot and jump start in my career almost 10 years ago, it reminded me of a presentation this same person gave back then.  

One of the key themes of this presentation was “Intrinsic Motivation.”  This was the first time I had ever heard the phrase – the speaker talked about one’s inner desire and self-motivation as the reason for wanting, learning, and doing.  It’s a feeling that is hard to describe, but I remember thinking during the presentation, “Hmm, I think I may have that.”  It’s not something many of us talk about, but those that have it can often see it or sense it in others.  On the surface, it could be called a passion.  It could be a hobby you love or when work starts to overlap as your hobby.  Maybe it’s just OCD coupled Continue reading

In my previous post about Docker, I focused on an introduction to networking with Docker.  That post had a fair amount of traction mainly due to it being #dockercon the week it was published, and seemingly, people had an interest in learning more about it.  Following the post, there were a few folks (@hartley and others) that pointed me to some great links about more advanced concepts in Docker and a site that validated what I was speculating with leveraging overlay tunnels as means for connectivity between nodes running Docker.

After reading through a few posts and the links about libswarm and libchan, it was on my mental to-do list to learn more about them and test more advanced designs.  In the meantime, the friendly folks at Digital Ocean let me know about a local meetup where James Turnbull, VP of Services at Docker, was presenting.  The meetup was tonight.  It was a short presentation, but for me, it was eye opening (it may be because I’ve been *mostly* focused on networking).  So, you won’t find a deep dive here like the last post, but rather, some general thoughts on Docker motivated from Continue reading

There has been a ton of information out there on Docker over the last week.  Because the impact on networking is often overlooked for new technologies, I figured I’d get a head start to understand the basics of Docker Networking.  This post documents the steps I took to test docker analyzing the network constructs that are automatically configured during container creation.

First, I installed Docker using instructions for Ubuntu 12.04 (LTS) 64-bit.

Post install, but before a container was created, here is the output of my Ubuntu machine.  Two interfaces: eth3 (192.168.1.134) and lo (127.0.0.1).  This Ubuntu machine is running in virtual box and eth3 is bridged onto my home network of 192.168.1.0/24.

Creating my first Docker container. This took about a minute (maybe less) to download and start.  Pretty impressive.  Notice the last line in the screen shot below.  It takes you right into the container shown at ‘root@c7ad293f989:/#’ 

In a new bash prompt because the existing shell is now used for the container, check out an ‘ifconfig.’  Notice the two new additions: docker0 and veth068f.  docker0 is a Linux bridge and veth068f Continue reading

Automating the configuration, provisioning, and management of particular workflows for cloud gets a lot of attention these days.  While automation makes perfect sense for deploying workloads faster, there are also other areas where automation can be leveraged to improve the overall operational efficiency of the IT Ops team. 

One of these areas is automating the validation of configuration changes.   This could mean validating changes deployed via the CLI for existing networks or validating changes made by SDN controllers for those new shiny physical AND virtual networks.  It doesn’t matter.  Connectivity tests can also be automated.  Much more can be automated than configuration and policy for data centers.

I remember doing annual power shutdowns of the data center and IDFs where I worked years ago.  I remember doing OS upgrades on critical network devices.  I also remember the chaos and the amount of people that needed to be on a bridge validating “everything looked okay” when the devices came back online.  Was everything always okay?  Hardly, but it wasn’t until business started on the following Monday, those one off issues were uncovered and fixed.  If there were tools verifying routing tables, Continue reading

If you read this site often, you already know I’ve been doing quite a bit of work with Ansible specifically as it pertains to networking.  While I will be showing another video very soon in a follow up post, I wanted to take a step back and cover a few things before doing so.  The focus here is less about the technology and more my general mindset around automation PLATFORMS, code, open source, and why I do it.  Just something I’d like to share because I’m occasionally asked questions around these topics.

Culture 

It’s not about the tool, in my case Ansible, it’s about the process, methodology, and ideas that go into thinking differently.  For me personally, Ansible just had a lower barrier for entry, but I’ve grown to quite like it.  Actually, I liked it from the get-go.   In order to effectively embrace platforms like Ansible, there needs to be a change in culture first.  It was amazing to see the emphasis on culture during the recent DevOps days in Pittsburgh.  I tuned in and out during the live stream and it seemed every time I had a chance to watch, it Continue reading

When networks are deployed in a box by box model, network admins know exactly what, where, and how something is being configured.  In highly dynamic environments, this may not be the case.  This is why it’s crucial to understand what is really going on behind the scenes.  In OpenStack, there are several components that together are comprised to make OpenStack Networking (aka Neutron).  These include the Neutron server, dhcp agent, metadata agent, L3 agent, and then the agents that would reside in the infrastructure to be programmed (on either physical and/or virtual switches).  For example, in Open vSwitch deployments, there would be a Neutron OVS agent on each host/server.  And this could vary based on which particular vendor plugin is being used too!

In this post, I’m going to mainly focus on the Neutron Layer 3 agent because I had a hard time grasping this one at first.  It turns out that it’s not so bad after all.

When I first started reading about Neutron, I saw many references that there was only one (1) layer 3 agent supported in a given deployment.  That just didn’t seem to make sense because that Continue reading

[Special and huge thanks to Scott Lowe for answering an endless amount of questions I had while writing this post and testing with NSX/OVS over the last few days. Thanks to Deepesh as well who I bounced OVS questions off of when I needed to give Scott a break. ]

In Open vSwitch 101, I described the three main components that make up Open vSwitch (OVS) from an architectural standpoint, namely ovs-vswitchd, ovsdb-server, and the fast path kernel module.  If you start to work with OVS, the first thing you realize is that it takes quite a bit more knowledge to really understand it.  This post will focus on some design principles and options when running OVS on a hypervisor like KVM in conjunction with a network virtualization solution.

To make this a little more practical, we will use a scenario which consists of a KVM host with 3 physical NICs – 1 x 1G and 2 x 10G.  The 1G interface will be used for management and the 2 x 10G interfaces will be used for actual transport of VM traffic. 

This example will also assume the use of an overlay network virtualization solution Continue reading

There is so much discussion on if network engineers need to be programmers that I was almost getting pissed off last week.  It was an odd and funny feeling.  Anyway, I've written in the past here and here about the use of Ansible for networking.  In this post and video, the goal is to show why network engineers don’t need to be "hardcore programmers."

Below is a short demo of using Ansible to automate basic network configuration tasks on Cisco routers.  As more Ansible modules come out (or any other tool that does the job), we will quickly realize, the network engineer doesn’t need to be a “hardcore programmer,” but rather understand the tool and articulate the requirements such that new modules can easily be added by others.  If the tools we’ll have in the future are “platforms” that can be customized by the consumer/customer meaning the vendor isn’t required to add more functionality, then that’ll pretty awesome for both the network engineer and the business.  It’s a win win.

FYI- the modules being demo’d are leveraging Cisco’s onePK as the API to connect to and make changes to the routers.

Hope you enjoy. Continue reading

In the last post, I talked about how Ansible could be used for various forms of network automation.  In the comments, Michael asked if Ansible could also be used for network test automation and verification.  Since I’m just starting to explore Ansible, I figured why not try it out.  The short answer is, it’s possible.  Let’s take a look at an example proving this out.

I just built a playbook that verifies a router, or group of routers, has reachability to a defined list of destinations.  Maybe you are deploying a new site? Maybe a new host?  Is the route being sent correctly to all sites?  Now, a playbook like this can be used to ensure every site (from the router) has reachability to the new site, subnet, or host.  You define the destination.  Cool, right?

Update: Before reading anymore, it's recommended to read the first post on Ansible for Networking.

Let’s do a quick walk through.

Here is the playbook.

As you can see, there is not a lot to it.  This playbook that we are looking at, called ‘pinger’, is using a module called ‘auto_pinger’.  Continue reading

[This article is the outcome of some great conversations and exchanges I’ve had recently with Jeremy Schulman (@nwkautomaniac) around automation and Devops in the world of networking.  Thank you to Jeremy for those late tweaks before getting this posted!  Thanks to Kirk Byers (@kirkbyers) as well - he was also gracious enough to respond to clarify a few things and assisted with this post indirectly.]

There have been numerous articles written that describe the what and the why of Devops.  Reading through a few of these, you find references to CAMS --- you’ll read how “Devops is about CAMS.”  CAMS stands for Culture, Automation, Measurement, and Sharing.  Imagine working in an environment where automation is embraced?  We know most networks are not leveraging nearly any type of automation.  While we usually talk about engineers (of all types) not embracing automation, is the harsh reality most organizations are from having the right culture to embrace automation? 

What if there was a way to start testing network automation without taking a risk in a production environment?  And to top it off, the network automation was being done using some of the same tools that Continue reading

You can’t listen to an interview or podcast, an industry panel, or read a Q&A about the future of networking that doesn't involve skill sets.  The biggest question of them all – what skills should network engineers focus on so they don’t become irrelevant? If you really want to know what skills make sense, why ask, when you can do an easy search to see what skills companies are looking for these days in a variety of roles.  Combine SDN with DevOps into your search criteria and the results may surprise you.  They sure surprised me.  

I don’t usually peruse the job boards, but I’m on LinkedIn daily, so I figured I’d put their job site to the test and see what skills are in demand by searching for specific products and tools.  This is the best way to search, right?  I did numerous searches using the following keywords: SDN, Puppet, Chef, Ansible, Python, Cisco, network, and CCIE.

Here are the results:

• There were only five (5) jobs available when searching for SDN, CCIE, and Python.  Reminder, I only searched LinkedIn.
• The first was for a vendor as a “SDN Cloud Architect.”  Continue reading

It’s been two weeks since I attended my 3rd consecutive Open Networking Summit (ONS) and I’m glad to say, I finally found some time to get some notes and thoughts on paper about the conference.  Here are some on SDN at Google and Microsoft, and how they compare and contrast to industry incumbents’ solutions, but also how programmable NFV can be game changing in the Enterprise.  I also include thoughts on how Embrane and Big Switch play into this.

Enter Andromeda

Google talked about their [home grown] network virtualization solution.  It leverages a custom SDN controller called Andromeda that controls physical switches, virtual switches, programmable NFV devices, and also ties into the storage platforms deployed.  Google talked about how they have showed industry leadership with technologies such as GFS, MapReduce, B4 WAN, etc.  If I extrapolate, they expect Andromeda to do for data center networking what GFS is doing for distributed scale out storage.  Who will be the Nutanix of networking?

Google, like few others, still define SDN as the separation of the control plane and data plane.  Google states, “logically centralized/ hierarchical control plane with peer to peer data plane beats full Continue reading

Over the past few weeks, I’ve written about the idea behind a common programmable abstraction layer.  Previous articles are here and here.  It’s worth stating that something like a CPAL can be used with or without SDN controllers and with or without cloud management platforms.  As can be seen from the previous write ups and the video/demo below, today its primary focus is data extraction and data visibility.  It can use device APIs or controller APIs.  It’s about accessing the data you need quicker.  It’s that simple.  No more jumping from device to device and having to manage text and excel files.  

Edit 3/15/2014:
Github repo for CPAL

If there is a controller in the environment, you can still view data around particular physical and virtual switches in the environments by creating the right modules.  Same can be said if there was a CMP/CMS deployed.  While a CPAL can easily make changes to the network, it’s about taking small steps that can have a larger impact on how we use new APIs on network devices and controllers.  And if we don’t strive for a common framework now, we will end Continue reading

Two of the three companies promoting white box, now more commonly known as bare metal, switching are Cumulus and Big Switch Networks.  There has been coverage on each of these companies, but the question always arises, “does Cumulus support OpenFlow?”  I had the chance to talk to JR Rivers, Cumulus CEO, at the last Open Networking User Group (ONUG) during a Tech Field Day video and heard the answer from him then, but hadn’t seen anything documented publicly. 

There was a SDN Meetup at Stanford last week where JR gave his take on SDN and a great overview on Cumulus, which happens to be on Vimeo.  More importantly, he touches upon the question regarding OpenFlow support in the Cumulus Linux software stack during the video.

Coming directly from JR, his response (around the 58 minute mark):

In the previous post, I talked about a common programmable abstraction layer (CPAL).  To better understand the thought process behind having a common PAL, it makes sense to review some of the work Jeremy Schulman has been doing.  Jeremy often refers to the Python interactive shell as the new CLI for networking.  When you watch him give a demo using the Python shell as a CLI, it is second nature and looks exactly like a network CLI.  It makes perfect sense.

It gives real time access to the network devices in a programmatic fashion.  Being programmatic is key here – no screen scraping, etc.  The downside --- you need to know Python.  Well, yes and no.  Of course, you would be using the Python interpreter so you’ll be leveraging programming concepts -- things like variables, functions, classes, etc. 

But, before you think you need to become a professional software developer, the other good news is that Jeremy has created the Junos OS PyEZ library that drastically simplifies the interaction and baseline knowledge to start programming Juniper devices.  Juniper supports native XML APIs, but the PyEZ module abstracts that and eliminates Continue reading

In late January, there were some big names on stage at the latest Open Compute Summit.  I’d like to focus on one keynote panel that was called, “Opening Up Network Hardware.”  The panelists for this session included Martin Casado (VMware), Matthew Liste (Goldman Sachs), Dave Maltz (Microsoft), and JR Rivers (Cumulus) and was led by Najam Ahmad (Facebook).  If you haven’t watched the session already, it’s definitely worth it.  You can check it out here.

It’s always interesting when there are consumers of the technology on stage with those that sell and build their own technology.  Even in a session like this, it’s hard to put Facebook and Microsoft in the consumer/user group because they are increasingly rolling their own.  But Goldman Sachs (GS) on the other hand is unique --- and more relevant to the Enterprise.  They are a user of Enterprise technology, but what makes them unique, in my opinion, is that they seem to be crossing the chasm or straddling Enterprise technology and rolling their own (DIY) with technology from OCP and what they announced at last year’s ONS, which was soon after they joined the ONF.  And there is probably Continue reading

In a recent post, I wrote about some Python work I was testing on the Nexus 3000.  The end conclusion was that open Linux platforms will offer more flexibility --- for the consumer of the technology, ultimately the customer.  In this post, we’ll take a look at an example that integrates Python with the native Linux operating system.  

In the context of networking, the question often arises, what does having access to Linux really gain you?  For one, as you can see from my last post and you’ll see in this one, for native scripting within bash and Python is of extreme value in itself, not to mention you’d also have the ability to load any piece of software you want to that is compatible with Linux (think about tools, mgmt/monitoring platforms, etc.).

Okay, so you’d have the ability to use Python on a network switch.  So what?  What about running onboard analytics on the switch?  What about sending the exact data you need, the data you use to troubleshoot, the data part of your operational workflow, directly upstream to a head end server, or just simply to an existing syslog Continue reading