Scott Lowe

Author Archives: Scott Lowe

An Alternate Approach to etcd Certificate Generation with Kubeadm

I’ve written a fair amount about kubeadm, which was my preferred way of bootstrapping Kubernetes clusters until Cluster API arrived. Along the way, I’ve also discussed using kubeadm to assist with setting up etcd, the distributed key-value store leveraged by the Kubernetes control plane (see here, here, and here). In this post, I’d like to revisit the topic of using kubeadm to set up an etcd cluster once again, this time taking a look at an alternate approach to generating the necessary TLS certificates than what the official documentation describes.

There is absolutely nothing wrong with the process the official documentation describes (I’m referring to this page, by the way); this process just creates slightly “cleaner” certificates. What do I mean by “cleaner” certificates? The official documentation uses a series of kubeadm configuration files, one for each etcd cluster member, to control how the utility creates the necessary certificates and configuration files. The user is instructed to use these configuration files on a single system to generate the certificates for all the cluster members. This works fine, with one caveat: each of the certificates will have an extra hostname—the hostname of the system being used Continue reading

Technology Short Take 142

Welcome to Technology Short Take #142! This time around, the Networking section is a bit light, but I’ve got plenty of cloud computing links and articles for you to enjoy, along with some stuff on OSes and applications, programming, and soft skills. Hopefully there’s something useful here for you!

Networking

  • Bill Doerrfeld discusses the use of WebAssembly to extend the Envoy proxy here after a discussion with Idit Levine, CEO and founder of Solo.io. Envoy is increasingly becoming an important component in many cloud-native networking solutions, so the ability to extend or customize Envoy’s behavior is quite important in my opinion.

Servers/Hardware

Security

Cloud Computing/Cloud Management

Adding Multiple Items Using Kustomize JSON 6902 Patches

Recently, I needed to deploy a Kubernetes cluster via Cluster API (CAPI) into a pre-existing AWS VPC. As I outlined in this post from September 2019, this entails modifying the CAPI manifest to include the VPC ID and any associated subnet IDs, as well as referencing existing security groups where needed. I knew that I could use the kustomize tool to make these changes in a declarative way, as I’d explored using kustomize with Cluster API manifests some time ago. This time, though, I needed to add a list of items, not just modify an existing value. In this post, I’ll show you how I used a JSON 6902 patch with kustomize to add a list of items to a CAPI manifest.

By the way, if you’re not familiar with kustomize, you may find my introduction to kustomize post to be helpful. Also, for those readers who are unfamiliar with JSON 6902 patches, the associated RFC is useful, as is this site.

In this particular case, the addition of the VPC ID and the subnet IDs were easily handled with a strategic merge patch that referenced the AWSCluster object. More challenging, though, was the reference to the existing security Continue reading

Using WireGuard on macOS via the CLI

I’ve written a few different posts on WireGuard, the “simple yet fast and modern VPN” (as described by the WireGuard web site) that aims to supplant tools like IPSec and OpenVPN. My first post on WireGuard showed how to configure WireGuard on Linux, both on the client side as well as on the server side. After that, I followed it up with posts on using the GUI WireGuard app to configure WireGuard on macOS and—most recently—making WireGuard from Homebrew work on an M1-based Mac. In this post, I’m going to take a look at using WireGuard on macOS again, but this time via the CLI.

Some of this information is also found in this WireGuard quick start. Here I’ll focus only on using macOS as a WireGuard client, not as a server; refer to the WireGuard docs (or to my earlier post) for information on setting up a WireGuard server. I’ll also assume that you’ve installed WireGuard via Homebrew.

Generating Keys

The first step is to generate the public/private keys you’ll need. If the /usr/local/etc/wireguard (or the /opt/homebrew/etc/wireguard for users on an M1-based Mac) directory doesn’t exist, you’ll need to first create that directory. (It didn’t exist Continue reading

Installing Older Versions of Kumactl on an M1 Mac

The Kuma community recently released version 1.2.0 of the open source Kuma service mesh, and along with it a corresponding version of kumactl, the command-line utility for interacting with Kuma. To make it easy for macOS users to get kumactl, the Kuma community maintains a Homebrew formula for the CLI utility. That includes providing M1-native (ARM64) macOS binaries for kumactl. Unfortunately, installing an earlier version of kumactl on an M1-based Mac using Homebrew is somewhat less than ideal. Here’s one way—probably not the only way—to work around some of the challenges.

Note that this post really only applies to users of M1-based Macs; users of Intel-based Macs can extract the kumactl binary from the release archive available linked from the Kuma install docs. (The same goes for users of Linux distributions running on Intel-based hardware.) On the Kuma website, simply select the desired version of Kuma from the drop-down in the upper left, check the page for the direct download link, and off you go. This doesn’t work for M1-based Macs because at the time this post was written, the Kuma community was not providing ARM64 binaries. This leaves Homebrew as the only way (aside Continue reading

Making WireGuard from Homebrew Work on an M1 Mac

After writing the post on using WireGuard on macOS (using the official WireGuard GUI app from the Mac App Store), I found the GUI app’s behavior to be less than ideal. For example, tunnels marked as on-demand would later show up as no longer configured as an on-demand tunnel. When I decided to set up WireGuard on my M1-based MacBook Pro (see my review of the M1 MacBook Pro), I didn’t want to use the GUI app. Fortunately, Homebrew has formulas for WireGuard. Unfortunately, the WireGuard tools as installed by Homebrew on an M1-based Mac won’t work. Here’s how to fix that.

The key issues with WireGuard as installed by Homebrew on an M1-based Mac are:

  • On an M1-based Mac, Homebrew installs (by default) to the /opt/homebrew prefix. By comparison, Homebrew uses /usr/local on Intel-based Macs. Some of the WireGuard-related scripts are hard-coded to use /usr/local as the Homebrew prefix. Because the prefix has changed, though, these scripts now don’t work on an M1-based Mac.
  • WireGuard has a dependency on Bash. Unfortunately, the version of Bash supplied by macOS isn’t supported by WireGuard (it’s too old). Without a very specific PATH configuration, even installing the Homebrew version of Bash—which is Continue reading

Kubernetes Port Names and Terminating HTTPS Traffic on AWS

I recently came across something that wasn’t immediately intuitive with regard to terminating HTTPS traffic on an AWS Elastic Load Balancer (ELB) when using Kubernetes on AWS. At least, it wasn’t intuitive to me, and I’m guessing that it may not be intuitive to some other readers as well. Kudos to my teammates Hart Hoover and Brent Yarger for identifying the resolution, which I’m going to call out in this post.

This AWS Premium Support post outlines the basic scenario:

  • You’re running Kubernetes on AWS. The post references EKS, but as far as I know the issue is not limited to EKS, and should apply to self-managed Kubernetes clusters on AWS (assuming these clusters are configured with the AWS cloud provider).
  • You’ve published a Service of type LoadBalancer (which, in turn, creates a classic ELB). For self-managed clusters, this requires the AWS cloud provider to be installed and configured.
  • You want to terminate HTTPS traffic on the ELB. The post references the use of an ACM certificate, but I suspect it’s not limited to ACM certificates.

Consider the following YAML, taken directly from the previously-referenced AWS Premium Support article:

apiVersion: v1
kind: Service
metadata:
  name:  Continue reading

Technology Short Take 141

Welcome to Technology Short Take #141! This is the first Technology Short Take compiled, written, and published entirely on my M1-based MacBook Pro (see my review here). The collection of links shared below covers a fairly wide range of topics, from old Sun hardware to working with serverless frameworks in the public cloud. I hope that you find something useful here. Enjoy!

Networking

Review: Logitech Ergo K860 Ergonomic Keyboard

As part of an ongoing effort to refine my work environment, several months ago I switched to a Logitech Ergo K860 ergonomic keyboard. While I’m not a “keyboard snob,” I am somewhat particular about the feel of my keyboard, so I wasn’t sure how I would like the K860. In this post, I’ll provide my feedback, and provide some information on how well the keyboard works with both Linux and macOS.

Setup

Setting up the K860 is remarkably easy. The first system I tried to pair it with was an older Mac Pro workstation, and apparently the Bluetooth hardware on that particular workstation wasn’t new enough to support the K860 (Logitech indicates that Bluetooth 5.0 is needed; more on that in a moment). Instead, I popped in the USB-A wireless receiver, and was up and running with the K860 less than a minute later. This was using macOS, but the Mac Pro also dual-booted Linux, so I rebooted into Linux and found that the K860 with the Logitech-supplied USB receiver continued to work without any issues.

Linux, macOS, and Dual Boot Support

The key takeaway regarding Linux is this: if you’re interested in getting the K860 for use with Continue reading

Review: 2020 M1-Based MacBook Pro

I hadn’t done a personal hardware refresh in a while; my laptop was a 2017-era MacBook Pro (with the much-disliked butterfly keyboard) and my tablet was a 2014-era iPad Air 2. Both were serviceable but starting to show their age, especially with regard to battery life. So, a little under a month ago, I placed an order for some new Apple equipment. Included in that order was a new 2020 13" MacBook Pro with the Apple-designed M1 CPU. In this post, I’d like to provide a brief review of the 2020 M1-based MacBook Pro based on the past month of usage.

The “TL;DR” of my review is this: the new M1-based MacBook Pro offers impressive performance and even more impressive battery life. While the raw performance may not “blow away” its 2020 Intel-based counterpart—at least, it didn’t in my real-world usage—the M1-based MacBook Pro offered consistently responsive performance with a battery life that easily blew past any other laptop I’ve ever used, bar none.

Read on for more details.

Hardware

The build quality is really good, with a significant improvement in keyboard quality relative to the earlier butterfly keyboard models (such as my 2017-era MacBook Pro). However, the overall design Continue reading

The Next Step

The Greek philosopher Heraclitus is typically attributed as the creator of the well-known phrase “Change is the only constant.” Since I left VMware in 2018 to join Heptio, change has been my companion. First, there was the change of focus, moving to a focus on Kubernetes and related technologies. Then there was the acquisition of Heptio by VMware, and all the change that comes with an acquisition. Just when things were starting to settle down, along came the acquisition of Pivotal by VMware and several more rounds of changes as a result. Today, I mark the start of another change, as I begin a new role and take the next step in my career journey.

Last week, I announced via Twitter that I was leaving VMware to explore a new opportunity. Today, I start at Kong, Inc., as a Principal Field Engineer. Kong, if you aren’t already familiar, is a company focused on service connectivity for modern architectures, with products like their eponymous API gateway and the Envoy-powered Kuma service mesh. I’m really looking forward to getting much more familiar with Envoy, the Kong API gateway, Kuma, and related projects and technologies. I still get to be Continue reading

Technology Short Take 140

Welcome to Technology Short Take #140! It’s hard to believe it’s already the start of May 2021—my how time flies! In this Technology Short Take, I’ve gathered some links for you covering topics like Azure and AWS networking, moving from macOS to Linux (and back again), and more. Let’s jump right into the content!

Networking

Servers/Hardware

Security

Making Firefox on Linux use Private Browsing by Default

While there are a couple different methods to make Firefox use private browsing by default (see this page for a couple methods), these methods essentially force private browsing and disable the ability to use “regular” (non-private) browsing. In this post, I’ll describe what I consider to be a better way of achieving this, at least on Linux.

It’s possible this method will also work on Windows, but I haven’t tested it. If anyone gets a chance to test it and let me know, I’ll update this post and credit you accordingly. Just hit me on Twitter and let me know what you’ve found in your testing. I’ve also only tested this on Fedora, but it should be the same or very similar for any distribution that uses GNOME.

GNOME uses the idea of “desktop files” (typically found in /usr/share/applications or ~/.local/share/applications) to enable the launching of applications via the Activities screen or other mechanisms. (For more information on desktop files, see here.) These desktop files specify where the executable is found, what command-line parameters to use, what icon to use, what name the application should go by, etc. Desktop files also allow application developers or users Continue reading

Technology Short Take 139

Welcome to Technology Short Take #139! This Technology Short Take is a bit heavy on cloud, OS, and programming topics, but there should be enough other interesting links to be useful to plenty of folks. (At least, I hope that’s the case!) Now, let’s get on to the content!

Networking

  • Tony Mackay has a tutorial showing how to use Traefik to rate-limit requests to a WordPress instance.
  • Ali Al Idrees has a post on using NSX ALB (formerly Avi Networks) with Kubernetes clusters in a vSphere with Tanzu environment.
  • This post provides some examples of shared control planes (and thus shared failure domains) within networking.
  • In this post, Jakub Sitnicki digs way deep into the Linux kernel to uncover the answer to the question, “Why are there no entries in the conntrack table for SYN packets dropped by the firewall?” Get ready to get nerdy!
  • This article on eBPF and Isovalent (the company behind the Cilium CNI plugin for Kubernetes) has some statements with which I agree, and some that don’t make sense to me. For example, I agree with the statement that the “impact eBPF will have on networking, security and observability will be widespread”. However, Continue reading

Using WireGuard on macOS

A short while ago I published a post on setting up WireGuard for AWS VPC access. In that post, I focused on the use of Linux on both the server side (on an EC2 instance in your AWS VPC) as well as on the client side (using the GNOME Network Manager interface). However, WireGuard is not limited to Linux, and I recently configured one of my macOS systems to take advantage of this WireGuard infrastructure for access to the private subnets in my AWS VPC. In this post, I’ll walk readers through configuring macOS to use WireGuard.

The first step is installing WireGuard on macOS. This is remarkably easy; just go to the App Store and install the WireGuard app for macOS. (Hopefully this link will take you directly there if you’re on a macOS-based system.)

Once the app is installed, the next step is to configure the WireGuard tunnels. I found this to be a bit confusing at first, but only because I wasn’t clear on the relationship between the WireGuard app and the Network pane in System Preferences. In this case, you need to use the WireGuard app to create the tunnels, which will show up as Continue reading

Adding a MachineHealthCheck using Kustomize

MachineHealthChecks are a powerful feature in the Kubernetes Cluster API (CAPI), and something I played around with not too long ago on TGIK 143. Recently, I was helping to document the use of kustomize with Cluster API for inclusion in the upstream CAPI documentation, and I learned a simple trick with kustomize that I’d apparently overlooked in the past. If you’ve used kustomize for any great length of time you probably already know and have used the functionality I’ll describe in this post, but if you’re new to kustomize or, like me, a user of kustomize that hasn’t had time to dig into all of its functionality, then read on and see how you can use kustomize to add a MachineHealthCheck to a CAPI workload cluster.

If you’re not familiar with kustomize, then reading my introduction to kustomize may be useful before continuing on with the rest of this article.

In this use case—adding a MachineHealthCheck to an workload cluster in CAPI—I’ll work from the assumption that you have a “base” CAPI workload cluster definition (perhaps one you’ve generated using clusterctl config cluster). In the directory where this workload cluster manifest exists, you’d need to add a kustomization. Continue reading

Technology Short Take 138

Welcome to Technology Short Take #138. I have what I hope is an interesting and useful set of links to share with everyone this time around. I didn’t do so well on storage links; apologies to my storage-focused friends! However, there should be something for most everyone else. Enjoy!

Networking

  • I’ve been interested in learning more about gRPC, so this guide on analyzing gRPC messages using Wireshark may be useful.
  • Isovalent, the folks behind Cilium, recently unveiled the Network Policy Editor, a graphical way of editing Kubernetes Network Policies.
  • Ivan Pepelnjak, the font of all networking knowledge, has been discussing cloud networking in some detail for a good while now. The latest series of posts (found here and here) are, in my opinion, just outstanding. I want to be like Ivan when I grow up. #BeLikeIvan
  • If you work with TextFSM templates (see here for more information), then you might also like this post on writing a vim syntax plugin for TextFSM templates.
  • Want/need to better understand IPv6? Denise Fishburne has you covered. Denise also has you covered if you need BGP knowledge.

Security

Deploying a CNI Automatically with a ClusterResourceSet

Not too long ago I hosted an episode of TGIK8s, where I explored some features of Cluster API. One of the features I explored on the show was ClusterResourceSet, an experimental feature that allows users to automatically install additional components onto workload clusters when the workload clusters are provisioned. In this post, I’ll show how to deploy a CNI plugin automatically using a ClusterResourceSet.

A lot of this post is inspired by a similar post on installing Calico using a ClusterResourceSet. Although that post is for vSphere and this one focuses on AWS, much of the infrastructure differences are abstracted away by Kubernetes and Cluster API.

At a high level, using ClusterResourceSet to install a CNI plugin automatically looks like this:

  1. Make sure experimental features are enabled on your CAPI management cluster.
  2. Create a ConfigMap that contains the information to deploy the CNI plugin.
  3. Create a ClusterResourceSet that references the ConfigMap.
  4. Deploy one or more workload clusters that match the cluster selector specified in the ClusterResourceSet.

The sections below describe each of these steps in more detail.

Enabling Experimental Features

The preferred way to enable experimental features on your management cluster is to use a setting in the Continue reading

Setting up Wireguard for AWS VPC Access

Seeking more streamlined access to AWS EC2 instances on private subnets, I recently implemented Wireguard for VPN access. Wireguard, if you’re not familiar, is a relatively new solution that is baked into recent Linux kernels. (There is also support for other OSes.) In this post, I’ll share what I learned in setting up Wireguard for VPN access to my AWS environments.

Since the configuration of the clients and the servers is largely the same (especially since both client and server are Linux), I haven’t separated out the two configurations. At a high level, the process looks like this:

  1. Installing any necessary packages/software
  2. Generating Wireguard private and public keys
  3. Modifying the AWS environment to allow Wireguard traffic
  4. Setting up the Wireguard interface(s)
  5. Activating the VPN

The first thing to do, naturally, is install the necessary software.

Installing Packages/Software

On recent versions of Linux—I’m using Fedora (32 and 33) and Ubuntu 20.04—kernel support for Wireguard ships with the distribution. All that’s needed is to install the necessary userspace tools.

On Fedora, that’s done with dnf install wireguard-tools. On Ubuntu, the command is apt install wireguard-tools. (You can also install the wireguard meta-package, if you’d prefer.) Apple’s macOS, for example, Continue reading

Closing out the Tokyo Assignment

In late 2019, I announced that I would be temporarily relocating to Tokyo for a six-month assignment to build out a team focused on cloud-native services and offerings. A few months later, I was still in Colorado, and I explained what was happening in a status update on the Tokyo assignment. I’ve had a few folks ask me about it, so I thought I’d go ahead and shared that the Tokyo assignment did not happen and will not happen.

So why didn’t it happen? In my March 2020 update, I mentioned that paperwork, approvals, and proper budget allocations had slowed down the assignment, but then the pandemic hit. Many folks, myself included, expected that the pandemic would work itself out, but—as we now clearly know—it did not. And as the pandemic dragged on (and continues to drag on), restrictions on travel and concerns over public health and safety continued to mean that the assignment was not going to happen. As many of you know all too well, travel restrictions still exist even today.

OK, but why won’t it happen in the future, when the pandemic is under control? At the time when the Tokyo assignment was offered to me, there Continue reading

1 … 3 4 5 6 7 … 19