Scott Lowe

Author Archives: Scott Lowe

Bootstrapping a Kubernetes Cluster on AWS with Cluster API

Yesterday I published a high-level overview of Cluster API (CAPI) that provides an introduction to some of the concepts and terminology in CAPI. In this post, I’d like to walk readers through actually using CAPI to bootstrap a Kubernetes cluster on AWS.

It’s important to note that all of the information shared here is also found in the “Getting Started” guide in the AWS provider’s GitHub repository. My purpose here is provide an additional walkthrough that supplements that official documentation, not to supplant the official documentation, and to spread the word about how the process works.

Four basic steps are involved in bootstrapping a Kubernetes cluster on AWS using CAPI:

  1. Installing the necessary tools (a one-time task)
  2. Preparing the AWS account with the correct IAM roles and policies (this is a one-time task)
  3. Creating a management cluster (not required every single time)
  4. Creating a workload cluster

The following sections take a look at each of these steps in a bit more detail. First, though, I think it’s important to mention that CAPI is still in its early days (it’s currently at v1alpha1). As such, it’s possible that commands may (will) change, and API specifications may (will) change as further development Continue reading

An Introduction to Kubernetes Cluster API

In this post, I’d like to provide a high-level introduction to the Kubernetes Cluster API. The aim of Cluster API (CAPI, for short) is, as outlined in the project’s GitHub repository, “a Kubernetes project to bring declarative, Kubernetes-style APIs to cluster creation, configuration, and management”. This high-level introduction serves to establish some core terminology and concepts upon which I’ll build in future posts about CAPI.

First, let’s start with some terminology:

Bootstrap cluster: The bootstrap cluster is a temporary cluster used by CAPI. It’s used to create a more permanent cluster that is CAPI-enabled (the management cluster). Typically, the bootstrap cluster is created locally using kind (other options are possible), and is destroyed once the management cluster is up and running.

Management cluster: The CAPI-enabled cluster created by the temporary bootstrap cluster is the management cluster. The management cluster is long-lived, is running the CAPI provider components, and understands the CAPI Custom Resource Definitions (CRDs). Typically, users would use the management cluster to create and manage the lifecycle of one or more workload clusters.

Workload cluster: This is a cluster whose lifecycle is managed by CAPI via the management cluster, but isn’t actually CAPI-enabled itself and it doesn’t manage Continue reading

Liveblog: VMworld 2019 Day 1 General Session

This is the liveblog from the day 1 general session at VMworld 2019. This year the event is back at Moscone Center in San Francisco, and VMware has already released some juicy news (see here, here, and here) in advance of the keynote this morning, foreshadowing what Pat is expected to talk about.

The keynote kicks off with the usual inspirational video, this one incorporating themes and references from a number of high-tech movies, including “The Matrix” and “Inception,” among others. As the video concludes, Pat Gelsinger takes the stage promptly at 9am.

Gelsingers speaks briefly of his 7 years at VMware (this is his 8th VMworld), then jumps into the content of his presentation with the theme of this morning’s session: “Tech in the Age of Any”. Along those lines, Gelsinger talks about the diversity of the VMworld audience, welcomes the attendees in Klingon, and speaks very quickly to the Pivotal and Carbon Black acquisitions that were announced only a few days ago.

Shifting gears, Gelsinger talks about “digital life” and how that translates into millions of applications and billions of devices and billions of users. He talks about how 5G, Edge, and AI are going Continue reading

Technology Short Take 118

Welcome to Technology Short Take #118! Next week is VMworld US in San Francisco, CA, and I’ll be there live-blogging and meeting up with folks to discuss all things Kubernetes. If you’re going to be there, look me up! Otherwise, I leave you with this list of links and articles from around the Internet to keep you busy. Enjoy!

Networking

  • Networking guru Ivan Pepelnjak has migrated his online presence to AWS; read more here.

Servers/Hardware

  • Interesting (but otherwise not terribly useful) article on how to turn a MacBook into a touchscreen. Lack of a touch screen remains the MacBook line’s second most egregious shortcoming against competing products (the first being the awful keyboard).

Security

Cloud Computing/Cloud Management

Creating Tagged Subnets Across AWS AZs Using Pulumi

As I mentioned back in May in this post on creating a sandbox for learning Pulumi, I’ve started using Pulumi more and more of my infrastructure-as-code needs. I did switch from JavaScript to TypeScript (which I know compiles to JavaScript on the back-end, but the strong typing helps a new programmer like me). Recently I had a need to create some resources in AWS using Pulumi, and—for reasons I’ll explain shortly—many of the “canned” Pulumi examples didn’t cut it for my use case. In this post, I’ll share how I created tagged subnets across AWS availability zones (AZs) using Pulumi.

In this particular case, I was using Pulumi to create all the infrastructure necessary to spin up an AWS-integrated Kubernetes cluster. That included a new VPC, subnets in the different AZs for that region, an Internet gateway, route tables and route table associations, security groups, an ELB for the control plane, and EC2 instances. As I’ve outlined in my latest post on setting up an AWS-integrated Kubernetes 1.15 cluster using kubeadm, these resources on AWS require specific AWS tags to be assigned in order for the AWS cloud provider to work.

As I started working on this, Continue reading

Reconstructing the Join Command for Kubeadm

If you’ve used kubeadm to bootstrap a Kubernetes cluster, you probably know that at the end of the kubeadm init command to bootstrap the first node in the cluster, kubeadm prints out a bunch of information: how to copy over the admin Kubeconfig file, and how to join both control plane nodes and worker nodes to the cluster you just created. But what if you didn’t write these values down after the first kubeadm init command? How does one go about reconstructing the proper kubeadm join command?

Fortunately, the values needed for a kubeadm join command are relatively easy to find or recreate. First, let’s look at the values that are needed.

Here’s the skeleton of a kubeadm join command for a control plane node:

kubeadm join <endpoint-ip-or-dns>:<port> \
--token <valid-bootstrap-token> \
--discovery-token-ca-cert-hash <ca-cert-sha256-hash> \
--control-plane \
--certificate-key <certificate-key>

And here’s the skeleton of a kubeadm join command for a worker node:

kubeadm join <endpoint-ip-or-dns>:<port> \
--token <valid-bootstrap-token> \
--discovery-token-ca-cert-hash <ca-cert-sha256-hash> \

As you can see, the information needed for the worker node is a subset of the information needed for a control plane node.

Here’s how to find or recreate all the various pieces of information you need:

Setting up an AWS-Integrated Kubernetes 1.15 Cluster with Kubeadm

In this post, I’d like to walk through setting up an AWS-integrated Kubernetes 1.15 cluster using kubeadm. Over the last year or so, the power and utility of kubeadm has vastly improved (thank you to all the contributors who have spent countless hours!), and it is now—in my opinion, at least—at a point where setting up a well-configured, highly available Kubernetes cluster is pretty straightforward.

This post builds on the official documentation for setting up a highly available Kubernetes 1.15 cluster. This post also builds upon previous posts I’ve written about setting up Kubernetes clusters with the AWS cloud provider:

All of these posts are focused on Kubernetes releases prior to 1.15, and given the changes in kubeadm in the 1.14 and 1.15 releases, I felt it would be helpful to revisit the process again for 1.15. For now, I’m focusing on the in-tree AWS cloud provider; however, in the very near future I’ll look at using the new external AWS cloud provider.

As pointed out in the “original” Continue reading

Converting Kubernetes to an HA Control Plane

While hanging out in the Kubernetes Slack community, one question I’ve seen asked multiple times involves switching a Kubernetes cluster from a non-HA control plane (single control plane node) to an HA control plane (multiple control plane nodes). As far as I am aware, this isn’t documented upstream, so I thought I’d walk readers through what this process looks like.

I’m making the following assumptions:

  • The existing single control plane node was bootstrapped using kubeadm. (This means we’ll use kubeadm to add the additional control plane nodes.)
  • The existing single control plane node is using a “stacked configuration,” in which both etcd and the Kubernetes control plane components are running on the same nodes.

I’d also like to point out that there are a lot of different configurations and variables that come into play with a process like this. It’s (nearly) impossible to cover them all in a single blog post, so this post attempts to address what I believe to be the most common situations.

With those assumptions and that caveat in mind, the high-level overview of the process looks like this:

  1. Create a load balancer for the control plane.
  2. Update the API server’s certificate.
  3. Update the kubelet Continue reading

Technology Short Take 117

Welcome to Technology Short Take #117! Here’s my latest gathering of links and articles from the around the World Wide Web (an “old school” reference for you right there). I’ve got a little bit of something for most everyone, except for the storage nerds (I’m leaving that to my friend J Metz this time around). Here’s hoping you find something useful!

Networking

Servers/Hardware

Security

Accessing the Docker Daemon via an SSH Bastion Host

Today I came across this article, which informed me that (as of the 18.09 release) you can use SSH to connect to a Docker daemon remotely. That’s handy! The article uses docker-machine (a useful but underrated tool, I think) to demonstrate, but the first question in my mind was this: can I do this through an SSH bastion host? Read on for the answer.

If you’re not familiar with the concept of an SSH bastion host, it is a (typically hardened) host through which you, as a user, would proxy your SSH connections to other hosts. For example, you may have a bunch of EC2 instances in an AWS VPC that do not have public IP addresses. (That’s reasonable.) You could use an SSH bastion host—which would require a public IP address—to enable SSH access to otherwise inaccessible hosts. I wrote a post about using SSH bastion hosts back in 2015; give that post a read for more details.

The syntax for connecting to a Docker daemon via SSH looks something like this:

docker -H ssh://user@host <command>

So, if you wanted to run docker container ls to list the containers running on a remote system, you’d Continue reading

Decoding a Kubernetes Service Account Token

Recently, while troubleshooting a separate issue, I had a need to get more information about the token used by Kubernetes Service Accounts. In this post, I’ll share a quick command-line that can fully decode a Service Account token.

Service Account tokens are stored as Secrets in the “kube-system” namespace of a Kubernetes cluster. To retrieve just the token portion of the Secret, use -o jsonpath like this (replace “sa-token” with the appropriate name for your environment):

kubectl -n kube-system get secret sa-token \
-o jsonpath='{.data.token}'

The output is Base64-encoded, so just pipe the output into base64:

kubectl -n kube-system get secret sa-token \
-o jsonpath='{.data.token}' | base64 --decode

The result you’re seeing is a JSON Web Token (JWT). You could use the JWT web site to decode the token, but given that I’m a fan of the CLI I decided to use this JWT CLI utility instead:

kubectl -n kube-system get secret sa-token \
-o jsonpath='{.data.token}' | base64 --decode | \
jwt decode -

The final -, for those who may not be familiar, is the syntax to tell the jwt utility to look at STDIN for the JWT it needs to Continue reading

Adding a Name to the Kubernetes API Server Certificate

In this post, I’m going to walk you through how to add a name (specifically, a Subject Alternative Name) to the TLS certificate used by the Kubernetes API server. This process of updating the certificate to include a name that wasn’t included could find use for a few different scenarios. A couple of situations come to mind, such as adding a load balancer in front of the control plane, or using a new or different URL/hostname used to access the API server (both situations taking place after the cluster was bootstrapped).

This process does assume that the cluster was bootstrapped using kubeadm. This could’ve been a simple kubeadm init with no customization, or it could’ve been using a configuration file to modify the behavior of kubeadm when bootstrapping the cluster. This process also assumes your Kubernetes cluster is using the default certificate authority (CA) created by kubeadm when bootstrapping a cluster. Finally, this process assumes you are using a non-HA (single control plane node) configuration.

Before getting into the details of how to update the certificate, I’d like to first provide a bit of background on why this is important.

Background

The Kubernetes API server uses digital certificates to both Continue reading

VMworld 2019 Prayer Time

For the last several years, I’ve organized a brief morning prayer time at VMworld. I didn’t attend the conference last year, but organized a prayer time nevertheless (and was able to join one morning for prayer). This year, now that I’m back at VMware (via the Heptio acquisition) and speaking at the conference, I’d once again like to coordinate a time for believers to meet. So, if you’re a Christian interested in gathering together with other Christians for a brief time of prayer, here are the details.

What: A brief time of prayer

Where: Yerba Buena Gardens behind Moscone North (near the waterfall)

When: Monday 8/26 through Thursday 8/29 at 7:45am (this should give everyone enough time to grab breakfast before keynotes/sessions start at 9am)

Who: All courteous attendees are welcome, but please note this will be a distinctly Christian-focused and Christ-centric activity (note that I encourage believers of other faiths/religions to organize equivalent activities)

Why: To spend a few minutes in prayer over the day, the conference, the attendees, and each other

As in previous years, you don’t need to RSVP or anything like that, although you’re welcome to if you’d like (just hit me up on Twitter).

Continue reading

Spousevitivities at VMworld 2019

This year VMworld—VMware’s annual user conference—moves back to San Francisco from Las Vegas. Returning to the Bay Area with VMworld is Spousetivities, which is happening again this year for the 11th year at VMworld. Better get your tickets sooner rather than later, there’s quite a good chance these activities will sell out!

Registration is open right now.

This year’s activities are funded in part by the generous and community-minded support of Veeam and VMUG, who are “putting their money where their mouth is” when it comes to promoting strong work/life balance at events like VMworld.

Here’s a quick look at what’s planned for VMworld 2019 in San Francisco:

Monday, August 26: Spousetivities kicks off the week with a walking food tour. This tour, like all the others, will depart from the Marriott Marquis.

Tuesday, August 27: This full-day event will take participants up to Wine Country for several wine tastings. Transportion is provided, of course, and participants will enjoy lunch on the tour as well.

Wednesday, August 28: Nature, shopping, tranquility, and quaint towns—this tour has it all! Participants will visit the Golden Gate Bridge, the Marin headlands, Muir Woods, and Sausalito. Transportion and Continue reading

Calculating the CA Certificate Hash for Kubeadm

When using kubeadm to set up a new Kubernetes cluster, the output of the kubeadm init command that sets up the control plane for the first time contains some important information on joining additional nodes to the cluster. One piece of information in there that (until now) I hadn’t figured out how to replicate was the CA certificate hash. (Primarily I hadn’t figured it out because I hadn’t tried.) In this post, I’ll share how to calculate the CA certificate hash for kubeadm to use when joining additional nodes to an existing cluster.

When looking to figure this out, I first started with the kubeadm documentation. My searches led me here, which states:

The hash is calculated over the bytes of the Subject Public Key Info (SPKI) object (as in RFC7469). This value is available in the output of “kubeadm init” or can be calculated using standard tools.

That’s useful information, but what are the “standard tools” being referenced? I knew that a lot of work had been put into kubeadm init phase (for breaking down the kubeadm init workflow), but a quick review of that documentation didn’t reveal anything. Reviewing the referenced RFC also didn’t provide any Continue reading

Building Jsonnet from Source

I recently decided to start working with jsonnet, a data templating language and associated command-line interface (CLI) tool for manipulating and/or generating various data formats (like JSON, YAML, or other formats; see the Jsonnet web site for more information). However, I found that there are no prebuilt binaries for jsonnet (at least, not that I could find), and so I thought I’d share here the process for building jsonnet from source. It’s not hard or complicated, but hopefully sharing this information will streamline the process for others.

As some readers may already know, my primary OS is Fedora. Thus, the process I share here will be specific to Fedora (and/or CentOS and possibly RHEL).

To keep my Fedora installation clean of any unnecessary packages, I decided to use a CentOS 7 VM—instantiated and managed by Vagrant—for the build process. If you don’t want to use a build VM, you can omit the steps involving Vagrant. You’ll also need to modify the commands used to install the necessary packages (on Fedora, you’d use dnf instead of yum, for example). Different distributions may also use different package names for some of the dependencies, so keep that in mind.

  1. Run Continue reading

Technology Short Take 116

Welcome to Technology Short Take #116! This one is a bit shorter than usual, due to holidays in the US and my life being busy. Nevertheless, I hope that I managed to capture something you find useful or helpful. As always, your feedback is welcome, so if you have suggestions, corrections, or comments, you’re welcome to contact me via Twitter.

Networking

  • David Gee discusses jSNAPy and how it can be used to enable unit tests in infrastructure-as-code scenarios.
  • Jon Langemak tackles understanding RTs (Route Targets) and RDs (Route Distinguishers) are in MPLS VPNs. I also appreciate that Jon included a “Lab time” section in his article that encourages readers to try out the concepts he’s explaining.

Servers/Hardware

  • Although I’ve by and large moved away from Apple hardware (I still have a MacBook Pro running macOS that sees very little use, and a Mac Pro running Fedora), I did see this article regarding a new keyboard for the MacBook Air and MacBook Pro. That’s good—the butterfly keyboards are awful (in my opinion).

Security

  • If you’re unfamiliar with public key infrastructure (PKI), digital certificates, or encryption, you may find this Linux Journal article helpful. It provides the basics behind X.509v3 digital Continue reading

Technology Short Take 115

Welcome to Technology Short Take #115! I’m back from my much-needed vacation in Bali, and getting settled back into work and my daily routine (which, for the last few weeks, was mostly swimming in the pool and sitting on the beach). Here’s a fresh new collection of links and articles from the around the web to propel myself back into blogging. I hope you find something useful here!

Networking

Blogging Break

I wanted to let readers know that there will be a break in my blogging over the next few weeks. Crystal and I are celebrating our 20th wedding anniversary and have decided to take a very long trip to someplace very far away from civilization so that we can relax, unplug, and simply enjoy each other’s company.

I’ll be back in civilization on June 7, and you can expect a quick post summarizing our trip (maybe with some photos). I’ll also have some feedback on how the Peak Designs 20L photo backpack worked out for me. Until then, have a great one!

Technology Short Take 114

Welcome to Technology Short Take #114! There will be a longer gap than usual before the next Tech Short Take (more details to come on Monday), but in the meantime here’s some articles and links to feed your technical appetite. Enjoy!

Networking

Servers/Hardware

Security

Cloud Computing/Cloud Management

1 … 9 10 11 12 13 … 20