Archive

Category Archives for "blog.scottlowe.org"

Using Vagrant with Libvirt on Fedora 27

In this post, I’m going to show you how to use Vagrant with Libvirt via the vagrant-libvirt provider when running on Fedora 27. Both Vagrant and Libvirt are topics I’ve covered more than a few times here on this site, but this is the first time I’ve discussed combining the two projects.

If you’re unfamiliar with Vagrant, I recommend you start first with my quick introduction to Vagrant, after which you can browse all the “Vagrant”-tagged articles on my site for a bit more information. If you’re unfamiliar with Libvirt, you can browse all my “Libvirt”-tagged articles; I don’t have an introductory post for Libvirt.

Background

I first experimented with the Libvirt provider for Vagrant quite some time ago, but at that time I was using the Libvirt provider to communicate with a remote Libvirt daemon (the use case was using Vagrant to create and destroy KVM guest domains via Libvirt on a remote Linux host). I found this setup to be problematic and error-prone, and discarded it after only a short while.

Recently, I revisited using the Libvirt provider for Vagrant on my Fedora laptop (which I rebuilt with Fedora 27). As I mentioned in this post Continue reading

AWS re:Invent 2017 Keynote with Andy Jassy

This is a liveblog of the re:Invent 2017 keynote with Andy Jassy, taking place on Wednesday at the Venetian. As fully expected given the long queues and massive crowds, even arriving an hour early to the keynote isn’t soon enough; there’s already a huge crowd gathered to make it into the venue. Fortunately, I did make it in and scored a reasonable seat from which to write this liveblog.

The pre-keynote time is filled with catchy dance music arranged by a live DJ (same live DJ as last year, if I’m not mistaken). There’s already been quite a few announcements made this year even before today’s keynote: Amazon Sumerian (AR/VR service), new regions and availability zones (AZs), and new bare metal instances, just to name a few of the big ones. There’s been a great deal of speculation regarding what will be announced in today’s keynote, but there’s no doubt there will be a ton of announcements around service enhancements and new services. Rumors are flying about a managed Kubernetes offering; we shall see.

Promptly at 8am, the keynote starts with a brief video, and Andy Jassy, CEO of AWS, takes the stage. Jassy welcomes attendees to the sixth annual Continue reading

Liveblog: Deep Dive on Amazon Elastic File System

This is a liveblog of the AWS re:Invent 2017 session titled “Deep Dive on Amazon Elastic File System (EFS).” The presenters are Edward Naim and Darryl Osborne, both with AWS. This is my last session of day 2 of re:Invent; thus far, most of my time has been spent in hands-on workshops with only a few breakout sessions today. EFS is a topic I’ve watched, but haven’t had time to really dig into, so I’m looking forward to this session.

Naim kicks off the session with looking at the four phases users go through when they are choosing/adopting a storage solution:

  1. Choosing the right storage solution
  2. Testing and optimizing
  3. Ingest (loading data)
  4. Running it (operating it in production)

Starting with Phase 1, Naim outlines the three main things that people think about. The first item is storage type. The second is features and performance, and the third item is economics (how much does it cost). Diving into each of these items in a bit more detail, Naim talks about file storage, block storage, and object storage, and the characteristics of each of these approaches. Having covered these approaches, Naim returns to file storage (naturally) and talks about why file Continue reading

Liveblog: IPv6 in the Cloud – Protocol and Service Overview

This is a liveblog of an AWS re:Invent 2017 breakout session titled “IPv6 in the Cloud: Protocol and Service Overview.” The presenter’s name is Alan Halachmi, who is a Senior Manager of Solutions Architecture at AWS. As with so many of the other breakout sessions and workshops here at re:Invent this year, the queues to get into the session are long and it’s expected that the session will be completely full.

Halachmi starts the session promptly at 11:30am (the scheduled start time) by reviewing the current state of IP4 exhaustion, then quickly moves to a “state of the state” regarding IPv6 adoption on the Internet. Global IPv6 adoption is currently around 22%, and is expected to hit 25% by the end of the year. Mobile and Internet of Things (IoT) are driving most of the growth, according to Halachmi. T-Mobile, for example, now has 89% of their infrastructure running on IPv6.

Transitioning again rather quickly, Halachmi moves into an overview of the IPv6 protocol itself. IPv4 uses a 32-bit address space; IPv6 uses a 128-bit address space (29 orders of magnitude larger than IPv4). IPv4 uses dotted decimal with CIDR (Classless Interdomain Routing) notation; IPv6 uses colon-separated hextet notation Continue reading

A Sample Makefile for Creating Blog Articles

In October of this year, I published a blog post talking about a sample Makefile for publishing blog articles. That post focused on the use of make and a Makefile for automating the process of a publishing a blog post. This post is a companion to that post, and focuses on the use of a Makefile for automating the creation of blog posts.

Since early 2015, this site has been running as a static site with the content created using Markdown. In its first iteration as a static site, the HTML was generated using Jekyll and hosted on GitHub Pages. In the current iteration, the HTML is generated using Hugo, hosted on Amazon S3, and served via Amazon CloudFront. In both cases, the use of Markdown as the content format also required specific front-matter to instruct the static site generator how to create the HTML. For Hugo, the front-matter looks something like this (I use YAML, but other formats are supported):

---
author: slowe
categories: Explanation
date: 2017-11-27T12:00:00Z
tags:
- Writing
- Blogging
- Productivity
title: Sample Blog Post title
url: /2017/11/27/sample-blog-post-title/
---

There are obviously a lot of different ways to automate the creation of this front-matter Continue reading

Installing MultiMarkdown 6 on Fedora 27

Long-time readers are probably aware that I’m a big fan of Markdown. Specifically, I prefer the MultiMarkdown variant that adds some additional extensions beyond “standard” Markdown. As such, I’ve long used Fletcher Penny’s MultiMarkdown processor (the latest version, version 6, is available on GitHub). While Fletcher offers binary builds for Windows and macOS, the Linux binary has to be compiled from source. In this post, I’ll provide the steps I followed to compile a MultiMarkdown binary for Fedora 27.

The “How to Compile” page on the MMD-6 Wiki is quite sparse, so a fair amount of trial-and-error was needed. To keep my main Fedora installation as clean as possible, I used Vagrant with the Libvirt provider to create a “build VM” based on the “fedora/27-cloud-base” box.

Once the VM was running, I installed the necessary packages to compile the source code. It turns out only the following packages were necessary:

sudo dnf install gcc make cmake gcc-c++

Then I downloaded the source code for MMD-6:

curl -LO https://github.com/fletcher/MultiMarkdown-6/archive/6.2.3.tar.gz

Unpacking the archive with tar created a MultiMarkdown-6-6.2.3 directory. Changing into that directory, then the instructions from the Wiki page worked as expected:

make
 Continue reading

Using Docker Machine with KVM and Libvirt

Docker Machine is, in my opinion, a useful and underrated tool. I’ve written before about using Docker Machine with various services/providers; for example, see this article on using Docker Machine with AWS, or this article on using Docker Machine with OpenStack. Docker Machine also supports local hypervisors, such as VMware Fusion or VirtualBox. In this post, I’ll show you how to use Docker Machine with KVM and Libvirt on a Linux host (I’m using Fedora 27 as an example).

Docker Machine ships with a bunch of different providers, but the KVM/Libvirt provider must be obtained separately (you can find it here on GitHub). Download a binary release (make sure it is named docker-machine-driver-kvm), mark it as executable, and place it somewhere in your PATH. Fedora 27 comes with KVM and the Libvirt daemon installed by default (in order to support the Boxes GUI virtualization app), but I found it helpful to also install the client-side tools:

sudo dnf install libvirt-client

This will make the virsh tool available, which is useful for viewing Libvirt-related resources. Once you have both the KVM/Libvirt driver and the Libvirt client tools installed, you can launch a VM:

docker-machine create -d kvm --kvm-network  Continue reading

Happy Thanksgiving 2017

In the US, today (Thursday, November 23) is Thanksgiving. I’d like to take a moment to reflect on the meaning of Thanksgiving.

Thanksgiving means different things to different people:

  • To folks outside the US, it often just a day with drastically reduced email volume and no interruptions from US-based coworkers. (Enjoy!)
  • To folks in the US, it’s a holiday filled with food (turkey, anyone?). There may also be family gatherings, football (American football, of course), and possibly some shopping. (There will most certainly be shopping tomorrow.)
  • To many people, it’s also a time to be thankful or grateful for the good things in their lives.
  • To Christians, like myself, it’s often a time to reflect on the blessings that God placed in your life. I know that I am quite blessed—blessed with a great family, an amazing wife, and the opportunity to work in a fast-paced industry (among many many other blessings).

Whatever Thanksgiving means to you, I hope that you enjoy the holiday. Happy Thanksgiving!

Installing Older Docker Client Binaries on Fedora

Sometimes there’s a need to have different versions of the Docker client binary available. On Linux this can be a bit challenging because you don’t want to install a “full” Docker package (which would also include the Docker daemon); you only need the binary. In this article, I’ll outline a process I followed to get multiple (older) versions of the Docker client binary on my Fedora 27 laptop.

The process has two steps:

  1. Download the RPMs for the older releases from the Docker Yum repository (for Fedora, that repository is here).
  2. Extract the files from the RPM without actually installing the RPM.

For step 1, you can use the curl program to download specific RPMs. For example, to download version 1.12.6 of the Docker client binary, you’d download the appropriate RPM like this:

curl -LO https://yum.dockerproject.org/repo/main/fedora/24/Packages/docker-engine-1.12.6-1.fc24.x86_64.rpm

You’ll note that the URL above appears to be tied to a particular Fedora version (24, in this case). However, that’s only significant/applicable for the entire RPM package; once you extract the specific binaries, you should have no issues running the binaries on a different version (I was able to run older versions of Continue reading

Installing Postman on Fedora 27

I recently had a need to install the Postman native app on Fedora 27. The Postman site itself only provides a link to the download and a rather generic set of instructions for installing the Postman native app (a link to these instructions for Ubuntu 16.04 is also provided). There were not, however, any directions for Fedora. Hence, I’m posting the steps I took to set up the Postman native app on my Fedora 27 laptop.

(Note that these instructions will probably work with other versions of Fedora as well, but I’ve only used them on Fedora 27.)

Here are the steps I followed:

  1. Download the installation tarball, either via your browser of choice or via the command line. If you’d prefer to use the command line, this command should take care of you:

    curl -L https://www.getpostman.com/app/download/linux64 -O postman-linux-x64.tar.gz

  2. Unpack the tarball into the directory of your choice. I prefer to put third-party applications such as this into the /opt directory; you can (obviously) put it wherever you prefer. This command should do the trick:

    sudo tar xvzf postman-linux-x64.tar.gz -C /opt

    If you prefer a directory other than /opt, specify the Continue reading

Making AWS re:Invent More Family-Friendly

AWS re:Invent is just around the corner, and Spousetivities will be there to help bring a new level of family friendliness to the event. If you’re thinking of bringing a spouse, partner, or significant other with you to Las Vegas, I’d encourage you to strongly consider getting him or her involved in Spousetivities.

Want a sneak peek at what’s planned? Have a look:

  • Monday’s activity is a full-day trip to Death Valley, including a stop at Bad Water Basin (significant because it is 280 feet below sea level, making it the lowest place in the Western Hemisphere!). Lunch is included, of course.
  • On Tuesday, Spousetivities participants will get to visit a number of locations on the Las Vegas Strip, including Siegfried and Roy’s Secret Garden, the Wildlife Habitat at the Flamingo, and the Shark Reef at Mandalay Bay. Transportion is provided for longer connections, but there will be some walking involved—wear comfortable shoes!
  • Wednesday includes a visit to Red Rock Canyon and Hoover Dam. There will some opportunities for short sightseeing walks in Red Rock Canyon (plus the 13-mile scenic drive), and the Hoover Dam tour includes access to the generator room (a very cool sight).
  • Wrapping up the Continue reading

Technology Short Take 90

Welcome to Technology Short Take 90! This post is a bit shorter than most, as I’ve been on the road quite a bit recently. Nevertheless, there’s hopefully something here you’ll find useful.

Networking

Security

How to Tag Docker Images with Git Commit Information

I’ve recently been working on a very simple Flask application that can be used as a demo application in containerized environments (here’s the GitHub repo). It’s nothing special, but it’s been useful for me as a learning exercise—both from a Docker image creation perspective as well as getting some additional Python knowledge. Along the way, I wanted to be able to track versions of the Docker image (and the Dockerfile used to create those images), and link those versions back to specific Git commits in the source repository. In this article, I’ll share a way I’ve found to tag Docker images with Git commit information.

Before I proceed any further, I’ll provide the disclaimer that this information isn’t unique; I’m building on the work of others. Other articles sharing similar information include this one; no doubt there are countless more I haven’t yet seen. I’m presenting this information here simply to show one way (not the only way) of including Git commit information with a Docker image.

Getting the necessary information from Git is actually far easier than one might think. This variation of the git log command will print only the full hash of the last commit Continue reading

Deep Dive into Container Images in Kolla

This is a liveblog of my last session at the Sydney OpenStack Summit. The session title is “OpenStack images that fit your imagination: deep dive into container images in Kolla.” The presenters are Vikram Hosakote and Rich Wellum, from Cisco and Lenovo, respectively.

Hosakote starts with an overview of Kolla. Kolla is a project to deploy OpenStack services into Docker containers. There are two ways to use Kolla: using Ansible (referred to as Kolla-Ansible) or using Kubernetes (referred to as Kolla-Kubernetes). Hosakote mentions that Kolla-Kubernetes also uses Helm and Helm charts; this makes me question the relationship between Kolla-Kubernetes and OpenStack-Helm.

Why Kolla? Some of the benefits of Kolla, as outlined by Hosakote, include:

  • Fast(er) deployment (Hosakote has deployed in as few as 9 minutes)
  • Easy maintenance, reconfiguration, patching, and upgrades
  • Containerized services are found in container registry
  • One tool to do multiple things

Hosakote briefly mentions his preference for Kolla over other tools, including Juju, DevStack, PackStack, Fuel, OpenStack-Ansible, TripleO, OpenStack-Puppet, and OpenStack-Chef.

Other benefits of using containers for OpenStack:

  • Reproduce golden state easily
  • No more “Works in DevStack” but doesn’t work in production
  • Production-ready images (this seems specific to Kolla, not just containers for OpenStack control plane)
  • Continue reading

Carrier-Grade SDN-Based OpenStack Networking Solution

This session was titled “Carrier-Grade SDN Based OpenStack Networking Solution,” led by Daniel Park and Sangho Shin. Both Park and Shin are from SK Telecom (SKT), and (based on the description) this session is a follow-up to a session from the Boston summit where SK Telecom talked about an SDN-based networking solution they’d developed and released for use in their own 5G-based network.

Shin starts the session with some presenter introductions, and sets the stage for the presentation. Shin first provides some background on SKT, and discusses the steps that SKT has been taking to prepare their network for 5G infrastructure/services. This involves more extensive use of virtual network functions (VNFs) and software-defined infrastructure based on open software and open hardware. Shin reinforces that the SKT project (which is apparently called COSMOS?) exclusively leverages open source software.

Diving into a bit more detail, Shin talks about SONA Fabric (which is used to control the leaf/spine fabric used as the network underlay), SONA (which handles virtual network management), and TACO (which is an SKT-specific version of OpenStack). The network monitoring solution is called TINA, and this feeds into an integrated monitoring system known as 3DV.

TACO (stands for SKT All Continue reading

Can OpenStack Beat AWS in Price

This is a liveblog of the session titled “Can OpenStack Beat AWS in Price: The Trilogy”. The presenters are Rico Lin, Bruno Lago, and Jean-Daniel Bonnetot. The “trilogy” refers to the third iteration of this presentation; each time the comparison has been done in a different geographical region (first in Europe, then in North America, and finally here in Asia-Pacific).

Lago starts the presentation with an explanation of the session, and each of the presenters introduce themselves, their companies, and their backgrounds. In this particular case, the presenters are representing Catalyst (runs Catalyst Cloud in New Zealand), OVH, and EasyStack—all three are OpenStack-powered public cloud offerings.

Lago explains that they’ll cover three common OpenStack scenarios:

  • Private cloud
  • Managed private cloud
  • OpenStack-powered public cloud

Lin takes point to talk a bit about price differences in different geographical regions. Focusing on AWS, Lin points out that AWS services are about 8% higher in Europe than in North America. Moving to APAC, AWS services are about 29% higher than in North America. With this 29% price increase, I can see where OpenStack might be much more competitive in APAC than in North America (and this, in turn, may explain why OpenStack seems much Continue reading

Lessons Learnt from Running a Container-Native Cloud

This is a liveblog of the session titled “Lessons Learnt from Running a Container-Native Cloud,” led by Xu Wang. Wang is the CTO and co-founder of Hyper.sh, a company that has been working on leveraging hypervisor isolation for containers. This session claims to discuss some lessons learned from running a cloud leveraging this sort of technology.

Wang starts with a brief overview of Hyper.sh. The information for this session comes from running a Hypernetes (Hyper.sh plus Kubernetes)-based cloud for a year.

So, what is a “container-native” cloud? Wang provides some criteria:

  • A container is a first-class citizen in the cloud. This means container-level APIs and the ability to launch containers without a VM.
  • The cloud offers container-centric resources (floating IPs, security groups, etc.).
  • The cloud offers container-based services (load balancing, scheduled jobs, functions, etc.).
  • Billing is handled on a per-container level (not on a VM level).

To be honest, I don’t see how any cloud other than Hyper.sh’s own offering could meet these criteria; none of the major public cloud providers (Microsoft Azure, AWS, GCP) currently satisfy Wang’s requirements. A “standard” OpenStack installation doesn’t meet these requirements. This makes the session more like a Continue reading

Make Your Application Serverless

This is a liveblog from the last day of the OpenStack Summit in Sydney, Australia. The title of the session is “Make Your Application Serverless,” and discusses Qinling, a project for serverless (Functions-as-a-Service, or FaaS) architectures/applications on OpenStack. The presenters for the session are Lingxian Kong and Feilong Wang from Catalyst Cloud.

Kong provides a brief background on himself and his co-presenter (Wang), and explains that Catalyst Cloud is an OpenStack-based public cloud based in New Zealand. Both presenters are active technical contributors to OpenStack projects.

Kong quickly transitions into the core content of the presentation, which focuses on serverless computing and Qinling, a project for implementing serverless architectures on OpenStack. Kong points out that serverless computing doesn’t mean there are no servers, only that the servers (typically VMs) are hidden from view. Functions-as-a-Service, or FaaS, is a better term that Kong prefers. He next provides an example of how a FaaS architecture may benefit applications, and contrasts solutions like AutoScaling Groups (or the equivalent in OpenStack) with FaaS.

Some key characteristics of serverless, as summarized by Kong:

  • No need to think about servers
  • Run your code, not the whole application
  • Highly available and horizontally scalable
  • Stateless/ephemeral
  • Lightweight/single-purpose functions
  • Event-driven Continue reading

How to Deploy 800 Servers in 8 Hours

This is a liveblog of the session titled “How to deploy 800 nodes in 8 hours automatically”, presented by Tao Chen with T2Cloud (Tencent).

Chen takes a few minutes, as is customary, to provide an overview of his employer, T2cloud, before getting into the core of the session’s content. Chen explains that the drive to deploy such a large number of servers was driven in large part by a surge in travel due to the Spring Festival travel rush, an annual event that creates high traffic load for about 40 days.

The “800 servers” count included 3 controller nodes, 117 storage nodes, and 601 compute nodes, along with some additional bare metal nodes supporting Big Data workloads. All these nodes needed to be deployed in 8 hours or less in order to allow enough time for T2cloud’s customer, China Railway Corporation, to test and deploy applications to handle the Spring Festival travel rush.

To help with the deployment, T2cloud developed a “DevOps” platform consisting of six subsystems: CMDB, OS installation, OpenStack deployment, task management, automation testing, and health check/monitoring. Chen doesn’t go into great deal about any of these subsystems, but the slide he shows does give away some information:

How to Deploy 800 Servers in 8 Hours

This is a liveblog of the session titled “How to deploy 800 nodes in 8 hours automatically”, presented by Tao Chen with T2Cloud (Tencent).

Chen takes a few minutes, as is customary, to provide an overview of his employer, T2cloud, before getting into the core of the session’s content. Chen explains that the drive to deploy such a large number of servers was driven in large part by a surge in travel due to the Spring Festival travel rush, an annual event that creates high traffic load for about 40 days.

The “800 servers” count included 3 controller nodes, 117 storage nodes, and 601 compute nodes, along with some additional bare metal nodes supporting Big Data workloads. All these nodes needed to be deployed in 8 hours or less in order to allow enough time for T2cloud’s customer, China Railway Corporation, to test and deploy applications to handle the Spring Festival travel rush.

To help with the deployment, T2cloud developed a “DevOps” platform consisting of six subsystems: CMDB, OS installation, OpenStack deployment, task management, automation testing, and health check/monitoring. Chen doesn’t go into great deal about any of these subsystems, but the slide he shows does give away some information:

1 … 15 16 17 18 19 … 34