Scott's Weblog

Author Archives: Scott's Weblog

Technology Short Take #60

Welcome to Technology Short Take #60. As usual, I’ve gathered what I hope to be a useful but varied collection of articles and links on key data center technologies. I hope something I’ve included here will be helpful—enjoy!

Networking

Using Docker with macvlan Interfaces

In this post, I’m going to show you how to use macvlan interfaces with Docker for networking. The use of macvlan interfaces presents an interesting networking configuration for Docker containers that may (depending on your use case) address issues with the standard Linux bridge configuration.

Macvlan interfaces, if you’re unfamiliar with them, are a (somewhat) recent addition to the Linux kernel that enables users to add multiple MAC address-based logical interfaces to a single physical interface. These logical interfaces must reside in the same broadcast domain as the associated physical interface, which means that Docker containers attached to macvlan interfaces also will be in the same broadcast domain as the associated physical interface. In other words, the Docker containers will be on the same network as the host—no IPTables rules, no Linux bridge, just attached directly to the host’s network. This introduces some interesting possibilities (and potential challenges), but I’ll save that discussion for a future post.

Right now, macvlan supported is implemented via an unsupported Docker Network plugin hosted on GitHub. However, I suspect that the macvlan functionality found in this plugin will find its way into the core of Docker Network, and probably sooner rather than later.

You’ll Continue reading

Looking Ahead: My 2016 Projects

Almost every year since 2012, I’ve been publishing a list of projects/goals for the upcoming year (here’s the original list for 2012, then 2013, I skipped 2014, and here’s the list for 2015). In this post, I’m going to share with you the list of projects/goals for 2016.

Here’s the list for 2016. For some of the items below, I’m also going to include a stretch goal, something I’ll aim toward but won’t count against myself if I don’t actually attain it.

  1. Complete a new book (again). In addition to actually completing the new network automation book I’m writing with Jason Edelman and Matt Oswalt (it’s available now as an Early Access edition), I have another book project lined up that I intend to finish and get published in 2016.

  2. Make more open source contributions. I failed this one miserably last year (see last year’s report card), but I am intent on making this one happen. Over time, I expect that this will just be part of who I am, but until then I’m going to explicitly call it out. Since I’m not a programmer (not yet, may never be), these contributions will have Continue reading

Multi-Machine Vagrant Environments with JSON

In this post I’d like to show you how to use a JSON-formatted data file to create and configure multi-machine Vagrant environments. This isn’t a new idea, and certainly not anything that I came up with or created. I’m simply presenting it here as an alternative option to the approach of using YAML with Vagrant for multi-machine environments (some people may prefer JSON over YAML).

If you’re unfamiliar with Vagrant, I’d start with my introduction to Vagrant. Then I’d recommend reviewing my original article on using YAML with Vagrant, followed by the updated/improved method that addresses a shortcoming with the original approach. These earlier posts will provide some basics that I’ll build on in this post.

To use a JSON-formatted data file as an external data source for Vagrant, the code in the Vagrantfile looks really similar to the code you’d use for YAML:

# -*- mode: ruby -*-
# # vi: set ft=ruby :
# Specify minimum Vagrant version and Vagrant API version
Vagrant.require_version '>= 1.6.0'
VAGRANTFILE_API_VERSION = '2'
# Require JSON module
require 'json'
# Read YAML file with box details
servers = JSON.parse(File.read(File.join(File.dirname Continue reading

Technology Short Take #59

Welcome to Technology Short Take #59, the first Technology Short Take of 2016. As we start a new year, here’s a collection of links and articles from around the web. Here’s hoping you find something useful to you!

Networking

  • Nir Yechiel posted an article on using the Cumulus VX QCOW2 image with Fedora and KVM. Cumulus VX, if you aren’t aware, is a community-supported virtual appliance version of Cumulus Linux aimed at helping folks preview and test “full-blown” Cumulus Linux (which, of course, requires compatible hardware).
  • NAPALM (Network Automation and Programmability Layer with Multivendor support) looks like a really cool tool. I haven’t yet had the opportunity to work with it, but it is definitely something I’d like to explore in more detail. Here’s an article on an effort to add Cisco IOS support to NAPALM. Gabriele (the author of that post) also has a nice article on some resources to get you started with network automation.
  • Using Python and Netmiko for network automation is the topic of this post by Colin McAlister. This is a good introductory post, and one that I plan to leverage as I dive deeper into these tools.
  • Kuryr (the OpenStack project to allow Docker Continue reading

An Improved Way to use YAML with Vagrant

In this post, I’d like to share with you an improved way to use YAML with Vagrant. I first discussed the use of YAML with Vagrant in a post on simplifying multi-machine Vagrant environments, where I simply factored out variable data into an external YAML file. The original approach I described had (at least) one significant drawback, though, which this new approach adddresses.

(By the way, this “improved” way is probably just a matter of better coding. I’m not an expert with Ruby, so Ruby experts may look at this and find it to be quite obvious.)

Here’s the original snippet of a Vagrantfile that I shared in that first Vagrant/YAML post:

# -*- mode: ruby -*-
# # vi: set ft=ruby :
# Specify minimum Vagrant version and Vagrant API version
Vagrant.require_version ">= 1.6.0"
VAGRANTFILE_API_VERSION = "2"
# Require YAML module
require 'yaml'
# Read YAML file with box details
servers = YAML.load_file('servers.yaml')
# Create boxes
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# Iterate through entries in YAML file
servers.each do |servers|
config.vm.define servers["name"] do |srv|
 Continue reading

Upcoming VMUG Events

I’m extremely honored to have the opportunity to help support VMware User Group (VMUG) meetings all over the world. I will be speaking at a few upcoming events; if you’re going to be at one of these events, I’d love to meet you, say hi, and chat for a bit. Here are the details.

Tuesday, February 23, 2016 I’m really excited to be back in Sydney again for an opportunity to speak at the Sydney VMUG UserCon (see the event page for full details).

Thursday, February 25, 2016 Two days after the Sydney event I’ll be in Melbourne to help support the Melbourne VMUG UserCon. (More details here.)

First week in March 2016 The dates for these events are still being finalized (I’ll update this post when I have more details), but I’ll be in South Africa for a series of VMUG events there as well (Johannesburg, Durban, and Cape Town). This will be my first time in South Africa, and I’m really looking forward to meeting and talking with customers there.

Aside from these VMUG events, if you’re in one of these regions, are a current (or potential) customer of VMware, and you’d like to meet to talk Continue reading

Full Stack Journey Episode #1: Bart Smith

In this first-ever episode of the Full Stack Journey podcast, I talk with Bart Smith (old GitHub account migrating to new GitHub account, YouTube channel). Bart shares some details about his journey from being a Microsoft-centric infrastructure engineer to what he calls a cloud-native full-stack engineer. Here are some notes from our conversation, along with some additional resources Bart wanted to share with readers/listeners. Enjoy!

The podcast audio recording is available on Soundcloud.

Show Notes

  • His journey started in June 2014 as a result of the Microsoft announcement regarding support for Linux and Kubernetes on Azure—this really indicated a shift in the industry.
  • Bart’s view is that a full-stack engineer knows about operations, the hardware stack (compute, storage, network), the software (network, operating system [OS], management, logging), and most importantly knows how to “code” an immutable infrastructure. An operations full-stack engineer can read code, work with developers, and be part of a DevOps team of support DevOps teams in deploying code into production both to on-premises solutions and off-premises solutions.
  • IT folks don’t need to be strictly involved in software engineering to benefit from a journey toward a more full-stack role.
  • His journey from Microsoft-centric engineer to Continue reading

Launching a New Podcast

After quite a bit of deliberation and planning, I’m excited (and nervous!) to announce the launch of a new podcast, titled “The Full Stack Journey Podcast”. Here are the details, structured in a Q&A format.

What topics does this new podcast cover?

The focus of The Full Stack Journey Podcast is to talk about the journey to becoming a full-stack engineer. That term is a bit of a loaded term—some people like it, some people don’t, and there’s some disagreement over exactly what it means. I use the term to describe someone who can work at multiple layers of the modern data center stack, crossing between different silos. This isn’t to say that a full-stack engineer is an expert in all these areas, but it does mean that a full-stack engineer has at least some knowledge and experience in all these areas, with expertise and experience in at least one of them. The podcast aims to provide real, relevant, practical information at helping people with their “full-stack journey.”

Why is the idea of becoming a full-stack engineer important enough to warrant launching a podcast?

I strongly believe the future of the infrastructure engineer does not lie in being Continue reading

My Getting Things Done Setup, Circa Early 2016

Almost six years ago I shared my (then) current Getting Things Done (GTD) setup, in which I described how I used various tools, techniques, and applications to try to maximize my productivity. I’d been toying with updating that post, but I wasn’t sure that anyone would find it useful. However, a recent e-mail from a reader indicated that there probably is some interest; with that in mind, then, here’s an update on my GTD-like setup, circa early 2016.

Before I dive into the details, a couple quick notes:

  • First, I call this a “GTD-like” setup because it doens’t necessarily strongly adhere to all the tenets of Getting Things Done. I’ve adapted the system to fit my particular role and responsibilities, which is something I strongly encourage every reader to also do.
  • Although I’ve previously discussed moving away from OS X (and this is something that I continue to evaluate and explore), this is—for now—a decidedly Mac-specific system. It’s probably possible to emulate a similar system on other platforms, but I leave that as an exercise for interested readers.

If you read the 2010 post, you may recall that I think of my workflow as having three “layers” of applications:

Looking Back: 2015 Project Report Card

In early 2015, I posted a look ahead at my planned 2015 projects, where I took a quick look at some of the self-development projects I set out for myself over the course of 2015. In this post, I’m going to review my progress on those 2015 projects.

The 2015 projects were as follows:

  1. Complete a new book
  2. Make more open source contributions
  3. Expand to a new configuration management solution
  4. Complete a “wildcard project”

So, how well did I do? Let’s take a look.

  1. Complete a new book: Technically, I haven’t (fully) completed a new book, but given that my new book project with Jason Edelman and Matt Oswalt on network automation is available now as an Early Access edition, I suppose this should count for something. Strangely enough, this wasn’t the book project I had in mind at the start of 2015, but sometimes things like this take unexpected turns. Grade: C

  2. Make more open source contributions: I expected this one to be easy, but it turns out that this is the area where my performance is the worst. I submitted a pull request to Terraform (for a docs update), but I did not make the contributions Continue reading

Next-Generation Network Engineering Skills

There’s no question that the networking industry is undergoing significant changes. Sparked in part by software-defined networking (SDN), this sea change now includes an expanded focus on application programming interfaces (APIs), automation frameworks and toolkits, and improved manageability. As the industry undergoes this change, though, networking engineers must also undergo a change.

To help address this change, I’m very excited to announce a new book project targeting “next-generation network engineering skills.” I’ve joined forces with two folks that I really admire—Jason Edelman and Matt Oswalt—to write a new book focusing on the skills we believe are essential for the next-generation network engineer:

New book cover

The Early Access edition of the book is available now. If you’re familiar with O’Reilly’s Early Access program, you know that this is an incomplete version right now, but you’ll get regular updates and the final version of the book once it is complete. Plus, you get to provide feedback to us (the authors) while we write, which in turn helps improve the book. (And we greatly desire your feedback!)

So what’s in this book? Here’s a quick look at some of the topics we’re tackling:

Running Ansible Through an SSH Bastion Host

This post will expand on some previous posts—one showing you how to set up and use an SSH bastion host and a second describing one use case for an SSH bastion host—to show how the popular configuration management tool Ansible can be used through an SSH bastion host.

The configuration/setup required to run Ansible through an SSH bastion host is actually reasonably straightforward, but I saw a lot of incomplete articles out there as I was working through this myself. My hope is to supplement the existing articles, as well as the Ansible documenation, to make this sort of configuration easier for others to embrace and understand.

Prerequisites

There are two key concepts involved here that you’ll want to be sure you understand before you proceed:

  1. You’ll want to make sure you’re comfortable with using an SSH bastion host. If you don’t understand how this works or how to set it up, I recommend you spend some time on this topic first, as it’s crucial to how Ansible will behave/function. This article by Grant Taylor has some good information.
  2. Spend some time making sure you know how to use SSH multiplexing. This is useful for Ansible in general, but Continue reading

Technology Short Take #58

Welcome to Technology Short Take #58. This will be the last Technology Short Take of 2015, as next week is Christmas and the following week is the New Year’s holiday. Before I present this episode’s collection of links, articles, and thoughts on various data center technologies, allow me to first wish all of my readers a very merry and very festive holiday season. Now, on to the content!

Networking

Using Cloud-Init to Register an Instance into Consul

This post describes a method for using cloud-init to register a cloud instance into Consul on provisioning. I tested this on OpenStack, but it should work on any cloud platform that supports metadata services that can be leveraged by cloud-init.

I worked out the details for this method because I was interested in using Consul as a means to provide a form of “dynamic DNS” for OpenStack instances. (You can think of it as service registration and discovery for OpenStack instances.) As I’ll point out later in this post, there are a number of problems with this approach, but—if for no other reason—it was helpful as a learning exercise.

The idea was to automatically register OpenStack instances into Consul as they were provisioned. Since Consul offers a DNS interface, other instances and/or workloads could use DNS to look up these nodes’ registration. Consul offers an HTTP API (see here for details), so I started there. I used Paw (a tool I described here) to explore Consul’s HTTP API, building the necessary curl commands along the way. Once I had the right curl commands, the next step was to build a shell script that would pull the current Continue reading

Modifying OpenStack Security Groups with Terraform

In this post I’d like to discuss a potential (minor) issue with modifying OpenStack security groups with Terraform. I call this a “potential minor” issue because there is an easy workaround, which I’ll detail in this post. I wanted to bring it to my readers’ attention, though, because as of this blog post this matter had not yet been documented.

As you probably already know if you read my recent introduction to Terraform blog post, Terraform is a way to create configurations that automate the creation or configuration of infrastructure components, possibly across a number of different providers and/or platforms. In the introductory blog post, I showed you how to write a Terraform configuration that would create an OpenStack logical network and subnet, create a logical router and attach it to the logical network, and then create an OpenStack instance and associate a floating IP. In that example, I used a key part of Terraform, known as interpolation.

Broadly speaking, interpolation allows Terraform to reference variables or attributes of other objects created by Terraform. For example, how does one refer to a network that he or she has just created? Here’s an example taken from the introductory blog post:

 Continue reading

Technology Short Take #57

Welcome to Technology Short Take #57. I hope you find something useful here!

Networking

Using SSH Multiplexing

In this post, I’m going to discuss how to configure and use SSH multiplexing. This is yet another aspect of Secure Shell (SSH), the “Swiss Army knife” for administering and managing Linux (and other UNIX-like) workloads.

Generally speaking, multiplexing is the ability to carry multiple signals over a single connection (see this Wikipedia article for a more in-depth discussion). Similarly, SSH multiplexing is the ability to carry multiple SSH sessions over a single TCP connection. This Wikibook article goes into more detail on SSH multiplexing; in particular, I would call your attention to the table under the “Advantages of Multiplexing” to better understand the idea of multiple SSH sessions with a single TCP connection.

One of the primary advantages of using SSH multiplexing is that it speeds up certain operations that rely on or occur over SSH. For example, let’s say that you’re using SSH to regularly execute a command on a remote host. Without multiplexing, every time that command is executed your SSH client must establish a new TCP connection and a new SSH session with the remote host. With multiplexing, you can configure SSH to establish a single TCP connection that is kept alive for a specific period Continue reading

A Use Case for an SSH Bastion Host

In this post, I’m going to explore one specific use case for using an SSH bastion host. I described this configuration and how to set it up in a previous post; in this post, though, I’d like to focus on one practical use case.

This use case is actually one I depicted graphically in my earlier post:

SSH bastion host diagram

This diagram could represent a couple different examples. For example, perhaps this is an AWS VPC. Security best practices suggest that you should limit access from the Internet to your instances as much as possible; unless an instance needs to accept traffic from the Internet, don’t assign a public IP address (or an Elastic IP address). However, without a publicly-accessible IP address, how does one connect to and manage the instance? You can’t SSH to it without a publicly-accessible IP address—unless you use an SSH bastion host.

Or perhaps this diagram represents an OpenStack private cloud, where users can deploy instances in a private tenant network. In order for those instances to be accessible externally (where “externally” means external to the OpenStack cloud), the tenant must assign each instance a floating IP address. Security may not be as much of a concern Continue reading

An Introduction to Terraform

In this post, I’m going to provide a quick introduction to Terraform, a tool that is used to provision and configure infrastructure. Terraform allows you to define infrastructure configurations and then have those configurations implemented/created by Terraform automatically. In this respect, you could compare Terraform to similar solutions like OpenStack Heat, AWS CloudFormation, and others.

Before I continue, though, allow me to first address this question: why Terraform?

Why Terraform?

This is a fair question, and one that you should be asking. After all, if Terraform is considered similar to OpenStack Heat or AWS CloudFormation, then why use Terraform instead of one of the comparable solutions? I believe there are a couple (related) reasons why you might consider Terraform over a similar solution:

  1. Within a single Terraform definition, you can orchestrate across multiple cloud services. For example, you could create instances with a cloud provider (AWS, DigitalOcean, etc.), create DNS records with a DNS provider, and register key/value entries in Consul. Heat and CloudFormation are, quite naturally, designed to work almost exclusively with OpenStack and AWS, respectively. (Astute readers will know that Heat supports CloudFormation templates, but you get the idea.) Therefore, one reason to use Terraform Continue reading

1 2 3 … 7