Using Docker Machine with AWS

As part of a broader effort (see the post on my 2016 projects) to leverage public cloud resources more than I have in the past, some Docker Engine-related testing I’ve been conducting recently has been done using AWS EC2 instances instead of VMs in my home lab. Along the way, I’ve found Docker Machine to be quite a handy tool, and in this post I’ll share how to use Docker Machine with AWS.

By and large, using Docker Machine with AWS is pretty straightforward. You can get an idea of what information Docker Machine needs by running docker-machine create -d amazonec2 --help. (You can also view the documentation for the AWS driver specifically.) The key pieces of information you need are:

• --amazonec2-access-key: This is your AWS access key. Docker Machine can read it from the $AWS_ACCESS_KEY_ID environment variable, or—if you have the AWS CLI installed—Docker Machine can read it from there.
• --amazonec2-secret-key: This is your AWS secret key. As with the AWS access key, Docker Machine can read this from an environment variable ($AWS_SECRET_ACCESS_KEY) or from the AWS CLI credentials file (by default, found in ~/.aws/credentials).
• --amazonec2-region: The AWS driver defaults to Continue reading

Vagrant, Ubuntu “Wily Werewolf,” and Networking

In what has been a fairly classic “yak shaving” exercise, I’ve been working on getting Ubuntu 15.10 “Wily Werewolf” running with Vagrant so that I can perform some testing with some other technologies that need a Linux kernel version of at least 4.2 (which comes with Ubuntu 15.10 by default). Along the way, I ran smack into a problem with Ubuntu 15.10’s networking configuration when used with Vagrant, and in this post I’m going to explain what’s happening here and provide a workaround.

The issue (described here on GitHub, among other places) involves a couple of changes in Ubuntu Linux (and upstream Debian GNU/Linux as well, although I haven’t personally tested it). One of the changes is in regards to how network interfaces are named; instead of the “old” eth0 or eth1 naming convention, Ubuntu 15.10 now uses persistent interface names like ens32 or ens33. Additionally, an update to the “ifupdown” package now returns an error where an error apparently wasn’t returned before.

The end result is that when you try to create a Vagrant VM with multiple network interfaces, it fails. Using a single network interface is fine; the issue only rears its Continue reading

Technology Short Take #63

Welcome to Technology Short Take #63. I’ve managed to (mostly) get back to my Friday publishing schedule, though I’m running much later in the day this time around than usual. I’ll try to correct that for the next one. In any case, here’s another collection of links and articles from around the Net on the major data center technology areas. Have fun reading!

Networking

• At DevOps Networking Forum 2016, I had the opportunity to share a presentation on some Linux networking options. If you’d like to see the presentation, it’s available on Slideshare and Speakerdeck. If you’d like to re-create the demo environment, check out the presentation’s GitHub repository. I’m also thinking of creating a video version of the presentation with some expanded content; I’d love to hear from readers if they would find that useful.
• Here’s another topic that came up at the recent DevOps Networking Forum: Spotify’s SDN Internet Router (SIR). Here’s a two-part series (Part 1 and Part 2) that discusses the SIR, the motivations for building it, the challenges they faced in building SIR, and the solutions to those challenges. It’s a pretty interesting read, in my opinion.
• I recently came across a couple Continue reading

On the Topic of Lock-In

While talking with customers over the past couple of weeks during a multi-country/multi-continent trip, one phrase that kept coming up is “lock-in”, as in “we’re trying to avoid lock-in” or “this approach doesn’t have any lock-in”. While I’m not a huge fan of memes, this phrase always brings to mind The Princess Bride, Vizzini’s use of the word “inconceivable,” and Inigo Montoya’s famous response. C’mon, you know you want to say it: “You keep using that word. I do not think it means what you think it means.” I feel the same way about lock-in, and here’s why.

Lock-in, as I understand how it’s viewed, is an inability to migrate from your current solution to some other solution. For example, you might feel “locked in” to Microsoft (via Office or Windows) or “locked in” to Oracle (via their database platform or applications), or even “locked in” to VMware through vCenter and vSphere. Although these solutions/platforms/products might be the right fit for your particular problem/need, the fact that you can’t migrate is a problem. Here you are, running a product or solution or platform that is the right fit for your needs, but because you may not be able Continue reading

Technology Short Take #60

Welcome to Technology Short Take #60. As usual, I’ve gathered what I hope to be a useful but varied collection of articles and links on key data center technologies. I hope something I’ve included here will be helpful—enjoy!

Networking

• The rise of Linux in importance in the networking industry continues, with Arista announcing enhancements to Arista EOS to support—among other things—Docker containers. (See this SDx Central article or this Network World article for more details.)
• As a partial counterpoint to the increased use/visibility of Linux in the networking industry, check out Ivan’s take on Dell’s OS10 announcement.
• And while we are on the topic of Linux and open source in the networking industry…read this entertaining and informative article on the importance of (truly) open source libraries in networking.
• NIC offloads can make a noticeable difference in performance with VXLAN-based overlay networks, so it’s worth some time making sure that VXLAN offloads are supported and enabled. Dale Coghlan has an article on the Broadcom bnx2x driver and VXLAN offload that may save you some time in this area.
• This post on automating NSX security rules using Splunk (and some custom code) is pretty cool. Granted, it’s a VMware post, so Continue reading

Using Docker with macvlan Interfaces

In this post, I’m going to show you how to use macvlan interfaces with Docker for networking. The use of macvlan interfaces presents an interesting networking configuration for Docker containers that may (depending on your use case) address issues with the standard Linux bridge configuration.

Macvlan interfaces, if you’re unfamiliar with them, are a (somewhat) recent addition to the Linux kernel that enables users to add multiple MAC address-based logical interfaces to a single physical interface. These logical interfaces must reside in the same broadcast domain as the associated physical interface, which means that Docker containers attached to macvlan interfaces also will be in the same broadcast domain as the associated physical interface. In other words, the Docker containers will be on the same network as the host—no IPTables rules, no Linux bridge, just attached directly to the host’s network. This introduces some interesting possibilities (and potential challenges), but I’ll save that discussion for a future post.

Right now, macvlan supported is implemented via an unsupported Docker Network plugin hosted on GitHub. However, I suspect that the macvlan functionality found in this plugin will find its way into the core of Docker Network, and probably sooner rather than later.

You’ll Continue reading

Looking Ahead: My 2016 Projects

Almost every year since 2012, I’ve been publishing a list of projects/goals for the upcoming year (here’s the original list for 2012, then 2013, I skipped 2014, and here’s the list for 2015). In this post, I’m going to share with you the list of projects/goals for 2016.

Here’s the list for 2016. For some of the items below, I’m also going to include a stretch goal, something I’ll aim toward but won’t count against myself if I don’t actually attain it.

1. Complete a new book (again). In addition to actually completing the new network automation book I’m writing with Jason Edelman and Matt Oswalt (it’s available now as an Early Access edition), I have another book project lined up that I intend to finish and get published in 2016.

2. Make more open source contributions. I failed this one miserably last year (see last year’s report card), but I am intent on making this one happen. Over time, I expect that this will just be part of who I am, but until then I’m going to explicitly call it out. Since I’m not a programmer (not yet, may never be), these contributions will have Continue reading

Multi-Machine Vagrant Environments with JSON

In this post I’d like to show you how to use a JSON-formatted data file to create and configure multi-machine Vagrant environments. This isn’t a new idea, and certainly not anything that I came up with or created. I’m simply presenting it here as an alternative option to the approach of using YAML with Vagrant for multi-machine environments (some people may prefer JSON over YAML).

If you’re unfamiliar with Vagrant, I’d start with my introduction to Vagrant. Then I’d recommend reviewing my original article on using YAML with Vagrant, followed by the updated/improved method that addresses a shortcoming with the original approach. These earlier posts will provide some basics that I’ll build on in this post.

To use a JSON-formatted data file as an external data source for Vagrant, the code in the Vagrantfile looks really similar to the code you’d use for YAML:

# -*- mode: ruby -*-
# # vi: set ft=ruby :
# Specify minimum Vagrant version and Vagrant API version
Vagrant.require_version '>= 1.6.0'
VAGRANTFILE_API_VERSION = '2'
# Require JSON module
require 'json'
# Read YAML file with box details
servers = JSON.parse(File.read(File.join(File.dirname Continue reading

Technology Short Take #59

Welcome to Technology Short Take #59, the first Technology Short Take of 2016. As we start a new year, here’s a collection of links and articles from around the web. Here’s hoping you find something useful to you!

Networking

• Nir Yechiel posted an article on using the Cumulus VX QCOW2 image with Fedora and KVM. Cumulus VX, if you aren’t aware, is a community-supported virtual appliance version of Cumulus Linux aimed at helping folks preview and test “full-blown” Cumulus Linux (which, of course, requires compatible hardware).
• NAPALM (Network Automation and Programmability Layer with Multivendor support) looks like a really cool tool. I haven’t yet had the opportunity to work with it, but it is definitely something I’d like to explore in more detail. Here’s an article on an effort to add Cisco IOS support to NAPALM. Gabriele (the author of that post) also has a nice article on some resources to get you started with network automation.
• Using Python and Netmiko for network automation is the topic of this post by Colin McAlister. This is a good introductory post, and one that I plan to leverage as I dive deeper into these tools.
• Kuryr (the OpenStack project to allow Docker Continue reading

An Improved Way to use YAML with Vagrant

In this post, I’d like to share with you an improved way to use YAML with Vagrant. I first discussed the use of YAML with Vagrant in a post on simplifying multi-machine Vagrant environments, where I simply factored out variable data into an external YAML file. The original approach I described had (at least) one significant drawback, though, which this new approach adddresses.

(By the way, this “improved” way is probably just a matter of better coding. I’m not an expert with Ruby, so Ruby experts may look at this and find it to be quite obvious.)

Here’s the original snippet of a Vagrantfile that I shared in that first Vagrant/YAML post:

# -*- mode: ruby -*-
# # vi: set ft=ruby :
# Specify minimum Vagrant version and Vagrant API version
Vagrant.require_version ">= 1.6.0"
VAGRANTFILE_API_VERSION = "2"
# Require YAML module
require 'yaml'
# Read YAML file with box details
servers = YAML.load_file('servers.yaml')
# Create boxes
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# Iterate through entries in YAML file
servers.each do |servers|
config.vm.define servers["name"] do |srv|
 Continue reading

Upcoming VMUG Events

I’m extremely honored to have the opportunity to help support VMware User Group (VMUG) meetings all over the world. I will be speaking at a few upcoming events; if you’re going to be at one of these events, I’d love to meet you, say hi, and chat for a bit. Here are the details.

Tuesday, February 23, 2016 I’m really excited to be back in Sydney again for an opportunity to speak at the Sydney VMUG UserCon (see the event page for full details).

Thursday, February 25, 2016 Two days after the Sydney event I’ll be in Melbourne to help support the Melbourne VMUG UserCon. (More details here.)

First week in March 2016 The dates for these events are still being finalized (I’ll update this post when I have more details), but I’ll be in South Africa for a series of VMUG events there as well (Johannesburg, Durban, and Cape Town). This will be my first time in South Africa, and I’m really looking forward to meeting and talking with customers there.

Aside from these VMUG events, if you’re in one of these regions, are a current (or potential) customer of VMware, and you’d like to meet to talk Continue reading

Full Stack Journey Episode #1: Bart Smith

In this first-ever episode of the Full Stack Journey podcast, I talk with Bart Smith (old GitHub account migrating to new GitHub account, YouTube channel). Bart shares some details about his journey from being a Microsoft-centric infrastructure engineer to what he calls a cloud-native full-stack engineer. Here are some notes from our conversation, along with some additional resources Bart wanted to share with readers/listeners. Enjoy!

The podcast audio recording is available on Soundcloud.

Show Notes

• His journey started in June 2014 as a result of the Microsoft announcement regarding support for Linux and Kubernetes on Azure—this really indicated a shift in the industry.
• Bart’s view is that a full-stack engineer knows about operations, the hardware stack (compute, storage, network), the software (network, operating system [OS], management, logging), and most importantly knows how to “code” an immutable infrastructure. An operations full-stack engineer can read code, work with developers, and be part of a DevOps team of support DevOps teams in deploying code into production both to on-premises solutions and off-premises solutions.
• IT folks don’t need to be strictly involved in software engineering to benefit from a journey toward a more full-stack role.
• His journey from Microsoft-centric engineer to Continue reading

Launching a New Podcast

After quite a bit of deliberation and planning, I’m excited (and nervous!) to announce the launch of a new podcast, titled “The Full Stack Journey Podcast”. Here are the details, structured in a Q&A format.

What topics does this new podcast cover?

The focus of The Full Stack Journey Podcast is to talk about the journey to becoming a full-stack engineer. That term is a bit of a loaded term—some people like it, some people don’t, and there’s some disagreement over exactly what it means. I use the term to describe someone who can work at multiple layers of the modern data center stack, crossing between different silos. This isn’t to say that a full-stack engineer is an expert in all these areas, but it does mean that a full-stack engineer has at least some knowledge and experience in all these areas, with expertise and experience in at least one of them. The podcast aims to provide real, relevant, practical information at helping people with their “full-stack journey.”

Why is the idea of becoming a full-stack engineer important enough to warrant launching a podcast?

I strongly believe the future of the infrastructure engineer does not lie in being Continue reading

My Getting Things Done Setup, Circa Early 2016

Almost six years ago I shared my (then) current Getting Things Done (GTD) setup, in which I described how I used various tools, techniques, and applications to try to maximize my productivity. I’d been toying with updating that post, but I wasn’t sure that anyone would find it useful. However, a recent e-mail from a reader indicated that there probably is some interest; with that in mind, then, here’s an update on my GTD-like setup, circa early 2016.

Before I dive into the details, a couple quick notes:

• First, I call this a “GTD-like” setup because it doens’t necessarily strongly adhere to all the tenets of Getting Things Done. I’ve adapted the system to fit my particular role and responsibilities, which is something I strongly encourage every reader to also do.
• Although I’ve previously discussed moving away from OS X (and this is something that I continue to evaluate and explore), this is—for now—a decidedly Mac-specific system. It’s probably possible to emulate a similar system on other platforms, but I leave that as an exercise for interested readers.

If you read the 2010 post, you may recall that I think of my workflow as having three “layers” of applications:

• Continue reading

Looking Back: 2015 Project Report Card

In early 2015, I posted a look ahead at my planned 2015 projects, where I took a quick look at some of the self-development projects I set out for myself over the course of 2015. In this post, I’m going to review my progress on those 2015 projects.

The 2015 projects were as follows:

1. Complete a new book
2. Make more open source contributions
3. Expand to a new configuration management solution
4. Complete a “wildcard project”

So, how well did I do? Let’s take a look.

1. Complete a new book: Technically, I haven’t (fully) completed a new book, but given that my new book project with Jason Edelman and Matt Oswalt on network automation is available now as an Early Access edition, I suppose this should count for something. Strangely enough, this wasn’t the book project I had in mind at the start of 2015, but sometimes things like this take unexpected turns. Grade: C

2. Make more open source contributions: I expected this one to be easy, but it turns out that this is the area where my performance is the worst. I submitted a pull request to Terraform (for a docs update), but I did not make the contributions Continue reading

Next-Generation Network Engineering Skills

There’s no question that the networking industry is undergoing significant changes. Sparked in part by software-defined networking (SDN), this sea change now includes an expanded focus on application programming interfaces (APIs), automation frameworks and toolkits, and improved manageability. As the industry undergoes this change, though, networking engineers must also undergo a change.

To help address this change, I’m very excited to announce a new book project targeting “next-generation network engineering skills.” I’ve joined forces with two folks that I really admire—Jason Edelman and Matt Oswalt—to write a new book focusing on the skills we believe are essential for the next-generation network engineer:

The Early Access edition of the book is available now. If you’re familiar with O’Reilly’s Early Access program, you know that this is an incomplete version right now, but you’ll get regular updates and the final version of the book once it is complete. Plus, you get to provide feedback to us (the authors) while we write, which in turn helps improve the book. (And we greatly desire your feedback!)

So what’s in this book? Here’s a quick look at some of the topics we’re tackling:

• An introduction to Linux, written from a Continue reading

Running Ansible Through an SSH Bastion Host

This post will expand on some previous posts—one showing you how to set up and use an SSH bastion host and a second describing one use case for an SSH bastion host—to show how the popular configuration management tool Ansible can be used through an SSH bastion host.

The configuration/setup required to run Ansible through an SSH bastion host is actually reasonably straightforward, but I saw a lot of incomplete articles out there as I was working through this myself. My hope is to supplement the existing articles, as well as the Ansible documenation, to make this sort of configuration easier for others to embrace and understand.

Prerequisites

There are two key concepts involved here that you’ll want to be sure you understand before you proceed:

1. You’ll want to make sure you’re comfortable with using an SSH bastion host. If you don’t understand how this works or how to set it up, I recommend you spend some time on this topic first, as it’s crucial to how Ansible will behave/function. This article by Grant Taylor has some good information.
2. Spend some time making sure you know how to use SSH multiplexing. This is useful for Ansible in general, but Continue reading

Technology Short Take #58

Welcome to Technology Short Take #58. This will be the last Technology Short Take of 2015, as next week is Christmas and the following week is the New Year’s holiday. Before I present this episode’s collection of links, articles, and thoughts on various data center technologies, allow me to first wish all of my readers a very merry and very festive holiday season. Now, on to the content!

Networking

• Looking for a step-by-step install guide for VMware NSX? This one is a bit older (refers to NSX 6.1 and the current release is 6.2), but you might find it helpful nevertheless.
• Ray Budavari—who is an absolutely fantastic NSX resource—has a blog post up on the integration between VMware NSX and vRealize Automation. This first post (judging by the “part 1” in the title this will be a series) is a bit light on details, but I’m hoping future posts will dive deeper into this topic.
• Mustafa Akin has an article on Docker’s new overlay networking functionality. He describes how to set it up (on both VirtualBox and Digital Ocean) and provides some high-level performance information as well.
• If you’d like to play around with Cumulus Linux but don’t Continue reading

Using Cloud-Init to Register an Instance into Consul

This post describes a method for using cloud-init to register a cloud instance into Consul on provisioning. I tested this on OpenStack, but it should work on any cloud platform that supports metadata services that can be leveraged by cloud-init.

I worked out the details for this method because I was interested in using Consul as a means to provide a form of “dynamic DNS” for OpenStack instances. (You can think of it as service registration and discovery for OpenStack instances.) As I’ll point out later in this post, there are a number of problems with this approach, but—if for no other reason—it was helpful as a learning exercise.

The idea was to automatically register OpenStack instances into Consul as they were provisioned. Since Consul offers a DNS interface, other instances and/or workloads could use DNS to look up these nodes’ registration. Consul offers an HTTP API (see here for details), so I started there. I used Paw (a tool I described here) to explore Consul’s HTTP API, building the necessary curl commands along the way. Once I had the right curl commands, the next step was to build a shell script that would pull the current Continue reading

Modifying OpenStack Security Groups with Terraform

In this post I’d like to discuss a potential (minor) issue with modifying OpenStack security groups with Terraform. I call this a “potential minor” issue because there is an easy workaround, which I’ll detail in this post. I wanted to bring it to my readers’ attention, though, because as of this blog post this matter had not yet been documented.

As you probably already know if you read my recent introduction to Terraform blog post, Terraform is a way to create configurations that automate the creation or configuration of infrastructure components, possibly across a number of different providers and/or platforms. In the introductory blog post, I showed you how to write a Terraform configuration that would create an OpenStack logical network and subnet, create a logical router and attach it to the logical network, and then create an OpenStack instance and associate a floating IP. In that example, I used a key part of Terraform, known as interpolation.

Broadly speaking, interpolation allows Terraform to reference variables or attributes of other objects created by Terraform. For example, how does one refer to a network that he or she has just created? Here’s an example taken from the introductory blog post:

 Continue reading