Scott Lowe

Author Archives: Scott Lowe

Liveblog: VMworld 2019 Day 1 General Session

This is the liveblog from the day 1 general session at VMworld 2019. This year the event is back at Moscone Center in San Francisco, and VMware has already released some juicy news (see here, here, and here) in advance of the keynote this morning, foreshadowing what Pat is expected to talk about.

The keynote kicks off with the usual inspirational video, this one incorporating themes and references from a number of high-tech movies, including “The Matrix” and “Inception,” among others. As the video concludes, Pat Gelsinger takes the stage promptly at 9am.

Gelsingers speaks briefly of his 7 years at VMware (this is his 8th VMworld), then jumps into the content of his presentation with the theme of this morning’s session: “Tech in the Age of Any”. Along those lines, Gelsinger talks about the diversity of the VMworld audience, welcomes the attendees in Klingon, and speaks very quickly to the Pivotal and Carbon Black acquisitions that were announced only a few days ago.

Shifting gears, Gelsinger talks about “digital life” and how that translates into millions of applications and billions of devices and billions of users. He talks about how 5G, Edge, and AI are going Continue reading

Technology Short Take 118

Welcome to Technology Short Take #118! Next week is VMworld US in San Francisco, CA, and I’ll be there live-blogging and meeting up with folks to discuss all things Kubernetes. If you’re going to be there, look me up! Otherwise, I leave you with this list of links and articles from around the Internet to keep you busy. Enjoy!

Networking

  • Networking guru Ivan Pepelnjak has migrated his online presence to AWS; read more here.

Servers/Hardware

  • Interesting (but otherwise not terribly useful) article on how to turn a MacBook into a touchscreen. Lack of a touch screen remains the MacBook line’s second most egregious shortcoming against competing products (the first being the awful keyboard).

Security

Cloud Computing/Cloud Management

Creating Tagged Subnets Across AWS AZs Using Pulumi

As I mentioned back in May in this post on creating a sandbox for learning Pulumi, I’ve started using Pulumi more and more of my infrastructure-as-code needs. I did switch from JavaScript to TypeScript (which I know compiles to JavaScript on the back-end, but the strong typing helps a new programmer like me). Recently I had a need to create some resources in AWS using Pulumi, and—for reasons I’ll explain shortly—many of the “canned” Pulumi examples didn’t cut it for my use case. In this post, I’ll share how I created tagged subnets across AWS availability zones (AZs) using Pulumi.

In this particular case, I was using Pulumi to create all the infrastructure necessary to spin up an AWS-integrated Kubernetes cluster. That included a new VPC, subnets in the different AZs for that region, an Internet gateway, route tables and route table associations, security groups, an ELB for the control plane, and EC2 instances. As I’ve outlined in my latest post on setting up an AWS-integrated Kubernetes 1.15 cluster using kubeadm, these resources on AWS require specific AWS tags to be assigned in order for the AWS cloud provider to work.

As I started working on this, Continue reading

Reconstructing the Join Command for Kubeadm

If you’ve used kubeadm to bootstrap a Kubernetes cluster, you probably know that at the end of the kubeadm init command to bootstrap the first node in the cluster, kubeadm prints out a bunch of information: how to copy over the admin Kubeconfig file, and how to join both control plane nodes and worker nodes to the cluster you just created. But what if you didn’t write these values down after the first kubeadm init command? How does one go about reconstructing the proper kubeadm join command?

Fortunately, the values needed for a kubeadm join command are relatively easy to find or recreate. First, let’s look at the values that are needed.

Here’s the skeleton of a kubeadm join command for a control plane node:

kubeadm join <endpoint-ip-or-dns>:<port> \
--token <valid-bootstrap-token> \
--discovery-token-ca-cert-hash <ca-cert-sha256-hash> \
--control-plane \
--certificate-key <certificate-key>

And here’s the skeleton of a kubeadm join command for a worker node:

kubeadm join <endpoint-ip-or-dns>:<port> \
--token <valid-bootstrap-token> \
--discovery-token-ca-cert-hash <ca-cert-sha256-hash> \

As you can see, the information needed for the worker node is a subset of the information needed for a control plane node.

Here’s how to find or recreate all the various pieces of information you need:

Setting up an AWS-Integrated Kubernetes 1.15 Cluster with Kubeadm

In this post, I’d like to walk through setting up an AWS-integrated Kubernetes 1.15 cluster using kubeadm. Over the last year or so, the power and utility of kubeadm has vastly improved (thank you to all the contributors who have spent countless hours!), and it is now—in my opinion, at least—at a point where setting up a well-configured, highly available Kubernetes cluster is pretty straightforward.

This post builds on the official documentation for setting up a highly available Kubernetes 1.15 cluster. This post also builds upon previous posts I’ve written about setting up Kubernetes clusters with the AWS cloud provider:

All of these posts are focused on Kubernetes releases prior to 1.15, and given the changes in kubeadm in the 1.14 and 1.15 releases, I felt it would be helpful to revisit the process again for 1.15. For now, I’m focusing on the in-tree AWS cloud provider; however, in the very near future I’ll look at using the new external AWS cloud provider.

As pointed out in the “original” Continue reading

Converting Kubernetes to an HA Control Plane

While hanging out in the Kubernetes Slack community, one question I’ve seen asked multiple times involves switching a Kubernetes cluster from a non-HA control plane (single control plane node) to an HA control plane (multiple control plane nodes). As far as I am aware, this isn’t documented upstream, so I thought I’d walk readers through what this process looks like.

I’m making the following assumptions:

  • The existing single control plane node was bootstrapped using kubeadm. (This means we’ll use kubeadm to add the additional control plane nodes.)
  • The existing single control plane node is using a “stacked configuration,” in which both etcd and the Kubernetes control plane components are running on the same nodes.

I’d also like to point out that there are a lot of different configurations and variables that come into play with a process like this. It’s (nearly) impossible to cover them all in a single blog post, so this post attempts to address what I believe to be the most common situations.

With those assumptions and that caveat in mind, the high-level overview of the process looks like this:

  1. Create a load balancer for the control plane.
  2. Update the API server’s certificate.
  3. Update the kubelet Continue reading

Technology Short Take 117

Welcome to Technology Short Take #117! Here’s my latest gathering of links and articles from the around the World Wide Web (an “old school” reference for you right there). I’ve got a little bit of something for most everyone, except for the storage nerds (I’m leaving that to my friend J Metz this time around). Here’s hoping you find something useful!

Networking

Servers/Hardware

Security

Accessing the Docker Daemon via an SSH Bastion Host

Today I came across this article, which informed me that (as of the 18.09 release) you can use SSH to connect to a Docker daemon remotely. That’s handy! The article uses docker-machine (a useful but underrated tool, I think) to demonstrate, but the first question in my mind was this: can I do this through an SSH bastion host? Read on for the answer.

If you’re not familiar with the concept of an SSH bastion host, it is a (typically hardened) host through which you, as a user, would proxy your SSH connections to other hosts. For example, you may have a bunch of EC2 instances in an AWS VPC that do not have public IP addresses. (That’s reasonable.) You could use an SSH bastion host—which would require a public IP address—to enable SSH access to otherwise inaccessible hosts. I wrote a post about using SSH bastion hosts back in 2015; give that post a read for more details.

The syntax for connecting to a Docker daemon via SSH looks something like this:

docker -H ssh://user@host <command>

So, if you wanted to run docker container ls to list the containers running on a remote system, you’d Continue reading

Decoding a Kubernetes Service Account Token

Recently, while troubleshooting a separate issue, I had a need to get more information about the token used by Kubernetes Service Accounts. In this post, I’ll share a quick command-line that can fully decode a Service Account token.

Service Account tokens are stored as Secrets in the “kube-system” namespace of a Kubernetes cluster. To retrieve just the token portion of the Secret, use -o jsonpath like this (replace “sa-token” with the appropriate name for your environment):

kubectl -n kube-system get secret sa-token \
-o jsonpath='{.data.token}'

The output is Base64-encoded, so just pipe the output into base64:

kubectl -n kube-system get secret sa-token \
-o jsonpath='{.data.token}' | base64 --decode

The result you’re seeing is a JSON Web Token (JWT). You could use the JWT web site to decode the token, but given that I’m a fan of the CLI I decided to use this JWT CLI utility instead:

kubectl -n kube-system get secret sa-token \
-o jsonpath='{.data.token}' | base64 --decode | \
jwt decode -

The final -, for those who may not be familiar, is the syntax to tell the jwt utility to look at STDIN for the JWT it needs to Continue reading

Adding a Name to the Kubernetes API Server Certificate

In this post, I’m going to walk you through how to add a name (specifically, a Subject Alternative Name) to the TLS certificate used by the Kubernetes API server. This process of updating the certificate to include a name that wasn’t included could find use for a few different scenarios. A couple of situations come to mind, such as adding a load balancer in front of the control plane, or using a new or different URL/hostname used to access the API server (both situations taking place after the cluster was bootstrapped).

This process does assume that the cluster was bootstrapped using kubeadm. This could’ve been a simple kubeadm init with no customization, or it could’ve been using a configuration file to modify the behavior of kubeadm when bootstrapping the cluster. This process also assumes your Kubernetes cluster is using the default certificate authority (CA) created by kubeadm when bootstrapping a cluster. Finally, this process assumes you are using a non-HA (single control plane node) configuration.

Before getting into the details of how to update the certificate, I’d like to first provide a bit of background on why this is important.

Background

The Kubernetes API server uses digital certificates to both Continue reading

VMworld 2019 Prayer Time

For the last several years, I’ve organized a brief morning prayer time at VMworld. I didn’t attend the conference last year, but organized a prayer time nevertheless (and was able to join one morning for prayer). This year, now that I’m back at VMware (via the Heptio acquisition) and speaking at the conference, I’d once again like to coordinate a time for believers to meet. So, if you’re a Christian interested in gathering together with other Christians for a brief time of prayer, here are the details.

What: A brief time of prayer

Where: Yerba Buena Gardens behind Moscone North (near the waterfall)

When: Monday 8/26 through Thursday 8/29 at 7:45am (this should give everyone enough time to grab breakfast before keynotes/sessions start at 9am)

Who: All courteous attendees are welcome, but please note this will be a distinctly Christian-focused and Christ-centric activity (note that I encourage believers of other faiths/religions to organize equivalent activities)

Why: To spend a few minutes in prayer over the day, the conference, the attendees, and each other

As in previous years, you don’t need to RSVP or anything like that, although you’re welcome to if you’d like (just hit me up on Twitter).

Continue reading

Spousevitivities at VMworld 2019

This year VMworld—VMware’s annual user conference—moves back to San Francisco from Las Vegas. Returning to the Bay Area with VMworld is Spousetivities, which is happening again this year for the 11th year at VMworld. Better get your tickets sooner rather than later, there’s quite a good chance these activities will sell out!

Registration is open right now.

This year’s activities are funded in part by the generous and community-minded support of Veeam and VMUG, who are “putting their money where their mouth is” when it comes to promoting strong work/life balance at events like VMworld.

Here’s a quick look at what’s planned for VMworld 2019 in San Francisco:

Monday, August 26: Spousetivities kicks off the week with a walking food tour. This tour, like all the others, will depart from the Marriott Marquis.

Tuesday, August 27: This full-day event will take participants up to Wine Country for several wine tastings. Transportion is provided, of course, and participants will enjoy lunch on the tour as well.

Wednesday, August 28: Nature, shopping, tranquility, and quaint towns—this tour has it all! Participants will visit the Golden Gate Bridge, the Marin headlands, Muir Woods, and Sausalito. Transportion and Continue reading

Calculating the CA Certificate Hash for Kubeadm

When using kubeadm to set up a new Kubernetes cluster, the output of the kubeadm init command that sets up the control plane for the first time contains some important information on joining additional nodes to the cluster. One piece of information in there that (until now) I hadn’t figured out how to replicate was the CA certificate hash. (Primarily I hadn’t figured it out because I hadn’t tried.) In this post, I’ll share how to calculate the CA certificate hash for kubeadm to use when joining additional nodes to an existing cluster.

When looking to figure this out, I first started with the kubeadm documentation. My searches led me here, which states:

The hash is calculated over the bytes of the Subject Public Key Info (SPKI) object (as in RFC7469). This value is available in the output of “kubeadm init” or can be calculated using standard tools.

That’s useful information, but what are the “standard tools” being referenced? I knew that a lot of work had been put into kubeadm init phase (for breaking down the kubeadm init workflow), but a quick review of that documentation didn’t reveal anything. Reviewing the referenced RFC also didn’t provide any Continue reading

Building Jsonnet from Source

I recently decided to start working with jsonnet, a data templating language and associated command-line interface (CLI) tool for manipulating and/or generating various data formats (like JSON, YAML, or other formats; see the Jsonnet web site for more information). However, I found that there are no prebuilt binaries for jsonnet (at least, not that I could find), and so I thought I’d share here the process for building jsonnet from source. It’s not hard or complicated, but hopefully sharing this information will streamline the process for others.

As some readers may already know, my primary OS is Fedora. Thus, the process I share here will be specific to Fedora (and/or CentOS and possibly RHEL).

To keep my Fedora installation clean of any unnecessary packages, I decided to use a CentOS 7 VM—instantiated and managed by Vagrant—for the build process. If you don’t want to use a build VM, you can omit the steps involving Vagrant. You’ll also need to modify the commands used to install the necessary packages (on Fedora, you’d use dnf instead of yum, for example). Different distributions may also use different package names for some of the dependencies, so keep that in mind.

  1. Run Continue reading

Technology Short Take 116

Welcome to Technology Short Take #116! This one is a bit shorter than usual, due to holidays in the US and my life being busy. Nevertheless, I hope that I managed to capture something you find useful or helpful. As always, your feedback is welcome, so if you have suggestions, corrections, or comments, you’re welcome to contact me via Twitter.

Networking

  • David Gee discusses jSNAPy and how it can be used to enable unit tests in infrastructure-as-code scenarios.
  • Jon Langemak tackles understanding RTs (Route Targets) and RDs (Route Distinguishers) are in MPLS VPNs. I also appreciate that Jon included a “Lab time” section in his article that encourages readers to try out the concepts he’s explaining.

Servers/Hardware

  • Although I’ve by and large moved away from Apple hardware (I still have a MacBook Pro running macOS that sees very little use, and a Mac Pro running Fedora), I did see this article regarding a new keyboard for the MacBook Air and MacBook Pro. That’s good—the butterfly keyboards are awful (in my opinion).

Security

  • If you’re unfamiliar with public key infrastructure (PKI), digital certificates, or encryption, you may find this Linux Journal article helpful. It provides the basics behind X.509v3 digital Continue reading

Technology Short Take 115

Welcome to Technology Short Take #115! I’m back from my much-needed vacation in Bali, and getting settled back into work and my daily routine (which, for the last few weeks, was mostly swimming in the pool and sitting on the beach). Here’s a fresh new collection of links and articles from the around the web to propel myself back into blogging. I hope you find something useful here!

Networking

Blogging Break

I wanted to let readers know that there will be a break in my blogging over the next few weeks. Crystal and I are celebrating our 20th wedding anniversary and have decided to take a very long trip to someplace very far away from civilization so that we can relax, unplug, and simply enjoy each other’s company.

I’ll be back in civilization on June 7, and you can expect a quick post summarizing our trip (maybe with some photos). I’ll also have some feedback on how the Peak Designs 20L photo backpack worked out for me. Until then, have a great one!

Technology Short Take 114

Welcome to Technology Short Take #114! There will be a longer gap than usual before the next Tech Short Take (more details to come on Monday), but in the meantime here’s some articles and links to feed your technical appetite. Enjoy!

Networking

Servers/Hardware

Security

Cloud Computing/Cloud Management

The Linux Migration: Preparing for the Migration

As far back as 2012, I was already thinking about migrating away from Mac OS X (now known as macOS). While the migration didn’t start in earnest until late 2016, a fair amount of work happened in advance of the migration. Since I’ve had a number of folks ask me about migrating to Linux, I thought I’d supplement my Linux migration series with a “prequel” about some of the work that happened to prepare for the migration.

In the end—and I imagine some folks may get upset or offended at this—an operating system (OS) is really just a vehicle to deliver applications to the user. While users like myself have strong preferences about their OS and how their OS works, ultimately it is the ability to “get things done” that really matters. This is why I ended up suspending my Linux migration in August 2017; I didn’t have access to the applications I needed in order to do what I needed to do. (Though, to be fair, part of that was a lack of growth on my part, though that’s a different blog post for a different day.)

To that end, most of the work I did in Continue reading

A Sandbox for Learning Pulumi

I recently started using Pulumi, a way of using a general purpose programming language for infrastructure-as-code projects. I’ve been using Pulumi with JavaScript (I know, some folks would say I should question my life decisions), and while installing Pulumi itself is pretty low-impact (a small group of binaries) there are a number of dependencies that need to be installed when using Pulumi with JavaScript. As I’m a stickler for keeping my primary system very “clean” with regard to installed packages and software, I thought I’d create a means whereby I can easily spin up a “sandbox environment” for learning Pulumi.

When creating this sandbox environment, I turned to some tools that are very familiar:

  • I used virtualization (a virtual machine) as the isolation mechanism. The next step is to use a Linux container, like a Docker container, as the isolation mechanism, but I thought I’d start with something a bit simpler at first.
  • Vagrant provides a way of automating the creation/destruction of said VM. Again, Vagrant is well-understood and widely used.
  • Ansible provides the automation to configure the VM with the necessary software (Pulumi and associated dependencies).
  • I also thought that some folks might find it interesting or useful Continue reading
1 … 9 10 11 12 13 … 20