In this post, I’m going to show you a quick tip I used today to combine the power of Vagrant with that of Docker Machine to quickly and easily create Docker-enabled virtual machines (VMs) on your laptop. This could be useful in a variety of scenarios; I leave it as an exercise for the reader to determine the best way to leverage this functionality in his or her own environment.
In my case, I needed to be able to easily create/destroy/recreate a couple of Docker-enabled VMs for a project on which I’m working. The problem I faced was that the tools I would normally use for such a task—Vagrant and Docker Machine—each had problems when used on their own:
As it turns Continue reading
Welcome to Technology Short Take #52, the latest collection of news, links, and articles from around the web on data center technologies.
In this post, I’m going to show you how to configure VLAN trunking with Mikrotik RouterOS, and along the way provide a brief introduction to this software and some of the functionality it offers. While it is Linux-based, RouterOS operates quite a bit differently than a lot of the other network operating systems with which I’ve worked, and so I hope that this post will help ease the learning curve a bit for others who decide to take the same path.
First, let me provide a quick bit of background. I found myself in need of a switch that was both Layer 2/3 capable with both 10/100/1000Mbps ports as well as 10Gbps SFP+ ports. Of course, this was for my home lab, so budget is a concern. I cast out a quick call on Twitter, asking for some recommendations, and a few folks recommended I have a look at RouterBoard/Mikrotik; specifically, the CRS-24G-2S+IN (see here for more details). The specs looked good, the price was reasonable, and several folks expressed their satisfaction with the product, so I bought one.
Upon receiving it, I found myself trying to unravel RouterOS (their Linux-based operating system). Their wiki is fairly helpful, but Continue reading
I’ve been wrestling with an Ubuntu network configuration issue over the last couple of weeks (off and on between working on other projects), and today I finally found a fix for the problem. The issue was that Ubuntu wouldn’t pick up changes to network interfaces. The fix is so simple I’m almost embarrassed to talk about it (it seems like something that I should have known), but I’m posting it here in case others run into the same issue.
Here’s a bit more context: I was switching some of the network interfaces in my Ubuntu 14.04.2 servers from a “standard” network configuration to using VLAN interfaces (after all, it seemed like such a shame to not more fully utilize the 10GbE and 40GbE interfaces in these servers). Before the reconfiguration, the servers had a network interface configuration file (located in /etc/network/interfaces.d and sourced in /etc/network/interfaces) that looked something like this:
auto p55p1
iface p55p1 inet static
address 172.16.3.201
netmask 255.255.255.0This interface was connected to a port on a Cumulus Linux-powered Dell S6000-ON that was configured as an access port on a particular VLAN. Everything seemed to work just Continue reading
Following up on my earlier post on Cumulus Linux networking concepts, I wanted to build on that information with a guide on configuring VLAN trunking. This would be useful in a number of different scenarios: supporting multiple (VLAN-backed) port groups on vSphere hosts, or connecting an Open vSwitch (OVS) bridge on a KVM or Xen hypervisor to multiple VLANs. You might also need to use a VLAN trunking configuration to connect a Cumulus Linux-powered switch to another switch.
For this configuration, I’m going to use the new VLAN-aware bridging functionality introduced in Cumulus Linux 2.5. There are two pieces involved in making this work:
Let’s look at each of these pieces individually.
In order to provide layer 2 (switched) connectivity between front-panel ports on a Cumulus Linux-powered switch, the ports have to be part of a bridge. In this case, we’ll create a VLAN-aware bridge, which simplifies the configuration (in my opinion). It’s a bit less “true” to the Linux way of doing things, but simpler.
Owing to its Debian roots, you’ll configure the bridge by either adding a stanza to /etc/network/interfaces or Continue reading
At DockerCon 2015 in San Francisco, I had the opportunity to meet with a few vendors in the Docker ecosystem. Here are some notes from my vendor briefings.
StackEngine describes themselves as enterprise-grade container application management. They tout features like being able to compose Docker applications using a drag-and-drop interface, deploy containers across multiple hosts, and provide automation—all with the sort of controls that enterprise IT groups are seeking. That’s all well and good, but the key problem in my mind is that these are features Docker is seeking for themselves. Docker Compose offers the ability to specify applications. True, there’s no GUI (yet). Alas, StackEngine can translate their GUI application design into YAML, but it doesn’t comply with Docker Compose. Thus, it ends up being more competitive than complimentary, in my opinion. Docker Swarm and the upcoming Docker Network address some of StackEngine’s deployment functionality, and if Project Orca takes off as an official effort—well, let’s just say I hope that StackEngine has more planned. This is not to say that StackEngine isn’t a well-engineered solution offering real value; rather, this is to say that StackEngine appears to be, unfortunately, in the crosshairs for functionality Docker is aiming Continue reading
This is a liveblog for the DockerCon 2015 session titled “Scaling New Services: From Container Creation to Automated Deployments”. This session is being led by the Disney Systems Engineering team and will feature a discussion/demo involving Docker, Mesos, Chef, Consul, and HAProxy.
The session starts with an introduction by Alex Williams, founder of The New Stack, who quickly turns it over to the Disney staff—Brian Scott and Patrick O’Connor. Brian starts with an overview of all the various companies within Disney, and the challenges that breadth creates. He then discusses the role of Disney’s Systems Engineering team, and the responsibilities of the team. That includes managing infrastructure, both on-premises as well as cloud-based infrastructure.
So, why Docker? To improve the guest experience, Disney needs to be able to move fast. They want to get away from managing VMs and cattle to managing containers and micro-bots. Brian talks about issues with onboarding developers, battling configuration drifts, and similar challenges. Disney started on their Docker journey 6-10 months ago, and lots of teams are still exploring the use cases for Docker. Some teams are already using it in the CI pipeline, and other teams are evaluating production use cases. CI is a Continue reading
This is a liveblog from the day 2 general session at DockerCon 2015. I was running late from some early morning meetings (sorry folks), so I wasn’t able to catch the first part of the general session (about the first 15 minutes or so). Here’s what I was able to capture.
Chris Buckley, Director of DevOps at Business Insider, took the stage to provide an overview of how Business Insider (BI) started using Docker. Buckley provides some “lessons learned”:
This led BI to Fig (now Docker Compose), which led to a decrease in the time it took to get a development environment up and running. With the combination of Vagrant and Docker, BI was able to reduce that to just a couple of hours. When BI revisited production apps, they turned to use Upstart/SysV scripts for containers, but this wasn’t quite the right fit. BI turned back to Puppet, building a parameterized Puppet class to create containers, links, set environment options, and define dependencies on other containers/services starting first.
Before Docker, the workflow was developers to GitHub to Jenkins, which then pushed to Continue reading
This is a liveblog of the Docker Networking breakout session. This session is led by Madhu Venugopal and Jana Radhakrishnan, both formerly of Socketplane (and now with Docker following the acquisition). They are introduced by John Willis, also formerly of Socketplane and well-known within the DevOps community.
Some display issues plague the session at the beginning, so it appears that Murphy’s Law is back with a vengeance.
Madhu starts out the session with an overview of why networking (in particular Docker networking) is so important. Networking is vast and complex, and networking is an inherent part of distributed applications. Therefore, it’s important to make networking developer-friendly and application-driven. He shares a vision: “We’ll do for networking what Docker did for compute”. So what are the goals from this vision?
Libnetwork is a key part of this effort. It was open-sourced in April, with over 200 pull requests and 200 GitHub stars. Windows and FreeBSD ports are in progress. Libnetwork is part of the Docker 1.7 release with limited functionality, allowing users to test it before it is fully enabled in Continue reading
This is the “Top Secret Docker Session led by Gordon the Turtle,” which is really a session on Docker Plugins. However, since Docker Plugins were only announced this morning during the general session, the title for this session had to be obscured. On stage are ClusterHQ (Luke Marsden), Glider Labs (Jeff Lindsay), and Weaveworks (Alexis Richardson).
Marsden starts the session with a brief history of the Docker Plugins project, and how it grew out of Powerstrip. Marsden reiterates that he said Powerstrip would be successful if they would “throw it away” in 6 months. Four months later, the Docker Plugins project is now officially announced, and Powerstrip is no longer necessary.
Marsden next turns the stage over to Jeff Lindsay. Lindsay talks about why the Docker Plugins project is so important—every customer is unique, and customers want/need the freedom to choose the right solution to use the tools that best solve their particular problem(s).
Jeff Lindsay turns it over to Alexis Richardson, who outlines the core requirements for Docker Plugins. Richardson outlines 3 requirements, but he doesn’t have a slide that lists those requirements, so I couldn’t capture them. Plugins today are limited to storage and networking, but that isn’t Continue reading
This is a liveblog of the DockerCon 2015 session on resilient routing and discovery, part of the “Advanced Tech” track. Simon Eskilden (@Sirupsen on Twitter) from Shopify is the speaker for this session.
Not surprisingly (you’d understand this if you walked Eskilden’s presentation from DockerCon EU 2015), he starts out with a mention of the walrus (his favorite animal). Eskilden starts with a brief overview of Shopify (his employer) and Shopify’s production deployment of Docker (they’ve had Docker in production for over a year). Eskilden freely acknowledges that moving to a microservices-based architecture increases complexity and is not “free”. In order to help address the complexity brought on by microservices-based architectures, Eskilden wants to talk about resiliency, service discovery, and routing.
Eskilden reinforces that companies shouldn’t be implementing Docker solely for the sake of implementing Docker; it should be for a reason, a purpose (for him, it’s making sure Shopify’s services stay up and available). Resiliency is about building a reliable system from a bunch of unreliable components. Total availability is the availability per service to the power of the number of services. This means that the more services there are, the lower the total availability is. (To help Continue reading
This is a liveblog for the day 1 general session at DockerCon 2015, taking place this week (today and tomorrow, anyway) at the Marriott Marquis in San Francisco, CA. This is my first DockerCon, and I’m looking forward to picking up lots of new knowledge.
The general session starts with a video (cartoon) about something working in development but not in production, and how Solomon Hykes came up with the idea for containers and Docker. It’s a humorous, tongue-in-cheek production. As the video wraps up, Docker CEO Ben Golub takes the stage.
Golub starts with a personal story about the various startups for which he’s worked, and the importance of his “two fold test” (that it has global significance and that it is easy to explain when you go home for Thanksgiving). Maybe the Thanksgiving test didn’t quite make it, but Golub does think (naturally) that Docker has global significance. Golub says that Docker has become a fundamental part of how companies build, ship, and run distributed applications, and that Docker is a key part of how industries and cultures are being transformed. He attributes this success to the Docker community and the Docker ecosystem. Rightfully so, Golub credits the Continue reading
Yesterday, I posted about using Vagrant to learn Ansible, in which I showed you one way to combine these two tools to make it easier to learn Ansible. This is a combination I’m currently using as I continue to explore Ansible. Today, I’m going to expand on yesterday’s post by showing you how to make Vagrant automatically build an Ansible inventory for a particular Vagrant environment.
As you may already know, the Vagrantfile that Vagrant uses to instantiate and configure the VMs in a particular Vagrant environment is just Ruby. As such, it can be extended in a lot of different ways to do a lot of different things. In my case, I’ve settled on a design pattern that involves a separate YAML file with all the VM-specific data, which is read by the Vagrantfile when the user runs vagrant up. The data in the YAML file determines how many VMs are instantiated, what box is used for each VM, and the resources that are allocated to each VM. This is a design pattern I’ve used repeatedly in my GitHub “learning-tools” repository, and it seems to work pretty well (for me, at least).
Using this arrangement, since I Continue reading
I’ve been spending some time with Ansible recently, and I have to say that it’s really growing on me. While Ansible doesn’t have a steep learning curve, there is still a learning curve—albeit a smaller/less steep curve—so I wanted to share here a “trick” that I found for using Vagrant to help with learning Ansible. (I say “trick” here because it isn’t that this is complicated or undocumented, but rather that it may not be immediately obvious how to combine these two.)
Note that this is not to be confused with using Ansible from within Vagrant as a provisioner; that’s something different (see the Vagrant docs for more information on that use case). What I’m talking about is having a setup where you can easily explore how Ansible works and iterate through your playbooks using a Vagrant-managed VM.
Here are the key components:
Vagrantfile and any associated support files).Vagrantfile, you’ll need an Continue readingAs I’ve recently had the opportunity to start working with Cumulus Linux (running on a Dell S6000-ON switch), in this post I wanted to share a few concepts I’ve learned about networking with Cumulus Linux.
I’m not a networking guru, but I’m also not new to configuring network equipment—I’ve configured GRE tunnels on a Cisco router, set up link-state tracking, and enabled jumbo frames on a Nexus 5000 (to name a few examples). I’ve worked with Cisco gear, HP equipment, Dell PowerConnect switches, and Arista EOS-powered switches. However, as a full distribution of Linux, networking with Cumulus Linux is definitely different from your typical network switch. To help make the transition easier, I’ll share here a few things I’ve learned so far.
It’s important to understand that Cumulus Linux isn’t just a “Linux-based network OS”—it’s actually a full Linux distribution (based on Debian). Lots of products are Linux-based these days, but often hide the full power of Linux behind some sort of custom command-line interface (CLI) or shell. Not so in this case! I think this fact is perhaps a bit easy to overlook, but it shapes everything that happens in Cumulus Linux:
Rubrik today announced a new Series B investment (of $41 million) and introduced their r300 Series Hybrid Cloud Appliance, powered by what they’re touting as a “Converged Data Management” platform. Wow—that’s a mouthful, isn’t it? It sounds a bit like buzzword bingo, but after having spent a bit of time talking to Rubrik last week, there are some interesting (in my opinion) things going on here.
So what exactly is Rubrik doing? Here’s the “TL;DR” for those of you that don’t have the patience (or the time) for anything more in-depth: Rubrik is targeting the secondary storage and backup/recovery market with a solution that combines a distributed file system, a distributed metadata service, clustering, and a distributed task scheduler to provide a scale-out backup/recovery solution that also seamlessly integrates cloud storage platforms for long-term retention. The catch-phrase they’re using is “Time Machine for cloud infrastructure” (I wonder how our good friends in Cupertino will react to the use of that phrase?).
Here’s a bit more detail on the various components of the solution:
As part of a lab rebuild I’ve been doing over the last few weeks (funny how hardware failures can lead to a lab rebuild), I’ve been expanding the use of Ansible for configuration automation. In this post, I’m going to share the process I’ve created for bootstrapping newly-built servers into Ansible.
I developed this Ansible bootstrapping process to work in conjunction with the fully automated Ubuntu installation method that I described in an earlier post. The idea is that I would be able to boot a new server (virtual or physical), choose a configuration from the PXE menu, and a few minutes later have a built Ubuntu system. Then, with a single command, I could “bootstrap” the server into an Ansible configuration automation system. This latter part—configuring systems to work with Ansible—is what I’ll be describing here.
First, a (very) brief overview of Ansible. Ansible is a configuration automation tool that leverages standard SSH connections to remote devices in order to perform its work. Ansible is agentless, so no software has to be pre-installed on the managed servers, but this means Ansible has to authenticate against remote systems in order to establish these SSH connections. This authentication should, in ideal Continue reading
Welcome to Technology Short Take #51, another collection of posts and links about key data center technologies like networking, virtualization, cloud management, and applications/operating systems. Here’s hoping you find something useful in this collection!
In this post I’ll show you how to use apt-cacher-ng as an Apt proxy for Ubuntu systems on your network. I’m sure there are a lot of other resources that also provide this information, but I’m including it here for the sake of completeness and making it as easy as possible for others. Using an Apt proxy will help reduce the traffic coming from your network to external repositories, but it simpler and easier than running your own internal repository or mirror.
This isn’t the first time I’ve discussed apt-cacher-ng; almost two years ago I showed you how to use Puppet to configure Ubuntu to use apt-cacher-ng. This post focuses on the manual configuration of an Apt proxy.
On the server side, setting up an Apt proxy is as simple as one command:
apt-get install apt-cacher-ng
I’m sure there are some optimizations or advanced configurations, but this is enough to get the Apt proxy up and running.
On the client side, there are a couple of ways to configure the system. You could use a tool like Puppet (as described here), or manually configure the system. If you choose manual configuration, you can place the configuration in either /etc/apt/apt. Continue reading
Recently on Twitter, I mentioned that I had managed to successfully create a fully automated process for installing Ubuntu Server 14.04.2, along with a method for bootstrapping Ansible. In this post, I’m going to describe the installation process I built and the components that went into making it work. I’ll discuss the Ansible bootstrap process in a separate post. I significantly doubt that there is anything new or unique here, but hopefully this information will prove helpful to others facing similar challenges.
Before I continue, allow me to briefly discuss why I didn’t use a system like Cobbler instead of putting together my own system. Cobbler is a great tool. For me, though, this was also about deepening my own knowledge. I wanted to better understand the various components involved and how they interacted, and I didn’t feel I would really be able to do that with a “prebuilt” system like Cobbler. If you are more interested in getting something up and running as opposed to learning more about how it works (and that’s OK), then I’d recommend you skip this post and go download Cobbler. If, on the other hand, you want to make this into more Continue reading