Category Archives for ""

Technology Short Take 148

Welcome to Technology Short Take #148, aka the Thanksgiving Edition (at least, for US readers). I’ve been scouring RSS feeds and various social media sites, collecting as many useful links and articles as I can find: from networking hardware and networking CI/CD pipelines to Kernel TLS and tricks for improving your working memory. That’s quite the range! I hope that you find something useful here.


Using Kustomize Components with Cluster API

I’ve been using Kustomize with Cluster API (CAPI) to manage my AWS-based Kubernetes clusters for quite a while (along with Pulumi for managing the underlying AWS infrastructure). For all the time I’ve been using this approach, I’ve also been unhappy with the overlay-based approach that had evolved as a way of managing multiple workload clusters. With the recent release of CAPI 1.0 and the v1beta1 API, I took this opportunity to see if there was a better way. I found a different way—time will tell if it is a better way. In this post, I’ll share how I’m using Kustomize components to help streamline managing multiple CAPI workload clusters.

Before continuing, I feel it’s important to point out that while the bulk of the Kustomize API is reasonably stable at v1beta1, the components portion of the API is still in early days (v1alpha1). So, if you adopt this functionality, be aware that it may change (or even get dropped).

More information on Kustomize components can be found in the Kustomize components KEP or in this demo document. The documentation on Kustomize components is somewhat helpful as well. I won’t try to rehash information found in those sources here, but Continue reading

Technology Short Take 147

Welcome to Technology Short Take #147! The list of articles is a bit shorter than usual this time around, but I’ve still got a good collection of articles and posts covering topics in networking, hardware (mostly focused on Apple’s processors), cloud computing, and virtualization. There’s bound to be something in here for most everyone! (At least, I hope so.) Enjoy your weekend reading!


  • Chris Parker shares the reason why 65535 is not part of the private autonomous system range. It’s an interesting history lesson and explanation, even if you aren’t a networking nerd.
  • Dmytro Shypovalov discusses ARP problems in EVPN. I laughed at his comment regarding people stepping on rakes (read the post).
  • Evgeny Khabarov’s part 4 in a series on using Envoy as an API gateway talks about authentication and authorization (aka AuthN/AuthZ). In particular, Khabarov focuses on Envoy’s ext_authz filter, which is what allows Envoy to check with an authorization service to see if a request is permitted or denied.
  • I was having a bit of difficulty fully grokking the Original Destination feature in Envoy, and I found this article to be helpful in understanding how it works and is configured. Another very helpful resource on Continue reading

Influencing Cluster API AMI Selection

The Kubernetes Cluster API (CAPI) project—which recently released v1.0—can, if you wish, help manage the underlying infrastructure associated with a cluster. (You’re also fully able to have CAPI use existing infrastructure as well.) Speaking specifically of AWS, this means that the Cluster API Provider for AWS is able to manage VPCs, subnets, routes and route tables, gateways, and—of course—EC2 instances. These EC2 instances are booted from a set of AMIs (Amazon Machine Images, definitely pronounced “ay-em-eye” with three syllables) that are prepared and maintained by the CAPI project. In this short and simple post, I’ll show you how to influence the AMI selection process that CAPI’s AWS provider uses.

There are a couple different ways to influence AMI selection, and all of them have to do with settings within the AWSMachineSpec, which controls the configuration of an AWSMachine object. (An AWSMachine object is an infrastructure-specific implementation of a logical Machine object.) Specifically, there are these options for influencing AMI selection:

  1. You can instruct CAPI to use a specific AMI with the ami field. (If this field is set, the other options do not apply.)
  2. You can modify the lookup format used to Continue reading

Creating Reusable Kuma Installation YAML

Using CLI tools—instead of a “wall of YAML”—to install things onto Kubernetes is a growing trend, it seems. Istio and Cilium, for example, each have a CLI tool for installing their respective project. I get the reasons why; you can build logic into a CLI tool that you can’t build into a YAML file. Kuma, the open source service mesh maintained largely by Kong and a CNCF Sandbox project, takes a similar approach with its kumactl tool. In this post, however, I’d like to take a look at creating reusable YAML to install Kuma, instead of using the CLI tool every time you install.

You might be wondering, “Why?” That’s a fair question. Currently, the kumactl tool, unless configured otherwise, will generate a set of TLS assets to be used by Kuma (and embeds some of those assets in the YAML regardless of the configuration). Every time you run kumactl, it will generate a new set of TLS assets. This means that the command is not declarative, even if the output is. Unfortunately, you can’t reuse the output, as that would result in duplicate TLS assets across installations. That brings me to the point of this Continue reading

Using the External AWS Cloud Provider for Kubernetes

In 2018, after finding a dearth of information on setting up Kubernetes with AWS integration/support, I set out to try to establish some level of documentation on this topic. That effort resulted in a few different blog posts, but ultimately culminated in this post on setting up an AWS-integrated Kubernetes cluster using kubeadm. Although originally written for Kubernetes 1.15, the process described in that post is still accurate for newer versions of Kubernetes. With the release of Kubernetes 1.22, though, the in-tree AWS cloud provider—which is what is used/described in the post linked above—has been deprecated in favor of the external cloud provider. In this post, I’ll show how to set up an AWS-integrated Kubernetes cluster using the external AWS cloud provider.

In addition to the post I linked above, there were a number of other articles I published on this topic:

Most of the information in these posts, if not all of it, is found in the latest iteration, but I wanted to include these links here for some additional context. Also, Continue reading

Kustomize Transformer Configurations for Cluster API v1beta1

The topic of combining kustomize with Cluster API (CAPI) is a topic I’ve touched on several times over the last 18-24 months. I first touched on this topic in November 2019 with a post on using kustomize with CAPI manifests. A short while later, I discovered a way to change the configurations for the kustomize transformers to make it easier to use it with CAPI. That resulted in two posts on changing the kustomize transformers: one for v1alpha2 and one for v1alpha3 (since there were changes to the API between versions). In this post, I’ll revisit kustomize transformer configurations again, this time for CAPI v1beta1 (the API version corresponding to the CAPI 1.0 release).

In the v1alpha2 post (the first post on modifying kustomize transformer configurations), I mentioned that changes were needed to the NameReference and CommonLabel transformers. In the v1alpha3 post, I mentioned that the changes to the CommonLabel transformer became largely optional; if you are planning on adding additional labels to MachineDeployments, then the change to the CommonLabels transformer is required, but otherwise you could probably get by without it.

For v1beta1, the necessary changes are very similar to v1alpha3, and (for the most part) are Continue reading

Technology Short Take 146

Welcome to Technology Short Take #146! Over the last couple of weeks, I’ve gathered a few technology-related links for you all. There’s some networking stuff, a few security links, and even a hardware-related article. But enough with the introduction—let’s get into the content!



  • Chris Mellor speculates that Cisco UCS may be on the way out; Kevin Houston responds with a “I don’t think so.” Who will be correct? I guess we will just have to wait and see.


Cloud Computing/Cloud Management

Installing Cilium via a ClusterResourceSet

In this post, I’m going to walk you through how to install Cilium onto a Cluster API-managed workload cluster using a ClusterResourceSet. It’s reasonable to consider this post a follow-up to my earlier post that walked you through using a ClusterResourceSet to install Calico. There’s no need to read the earlier post, though, as this post includes all the information (or links to the information) you need. Ready? Let’s jump in!


If you aren’t already familiar with Cluster API—hereafter just referred to as CAPI—then I would encourage you to read my introduction to Cluster API post. Although it is a bit dated (it was written in the very early days of the project, which recently released version 1.0). Some of the commands referenced in that post have changed, but the underlying concepts remain valid. If you’re not familiar with Cilium, check out their introduction to the project for more information. Finally, if you’re not familiar at all with the idea of ClusterResourceSets, you can read my earlier post or check out the ClusterResourceSet CAEP document.

Installing Cilium via a ClusterResourceSet

If you want to install Cilium via a ClusterResourceSet, the process looks something like this:

  1. Create Continue reading

Technology Short Take 145

Welcome to Technology Short Take #145! What will you find in this Tech Short Take? Well, let’s see…stuff on Envoy, network automation, network designs, M1 chips (and potential open source variants!), a bevy of security articles (including a couple on very severe vulnerabilities), Kubernetes, AWS IAM, and so much more! I hope that you find something useful here. Enjoy!



  • Howard Oakley of The Eclectic Light Company discusses some details on Apple’s M1 chip and what it does differently than other chips. Also included in this post are links to other articles with even more details—very helpful.
  • Are open source M1-style chips a possibility? This article Continue reading

Technology Short Take 144

Welcome to Technology Short Take #144! I have a fairly diverse set of links for readers this time around, covering topics from microchips to improving your writing, with stops along the way in topics like Kubernetes, virtualization, Linux, and the popular JSON-parsing tool jq along the way. I hope you find something useful!


  • A short while ago I was helping someone (an acquaintance of a friend) with some odd DNS issues. I never found the root cause, but we did find a workaround; however, along the way, someone shared this article with me. I thought it was useful, so now I’m sharing it with you.
  • Michael Kashin shares the journey of containerizing NVIDIA Cumulus Linux.



Establishing VPC Peering with Pulumi and Go

I use Pulumi to manage my lab infrastructure on AWS (I shared some of the details in this April 2020 blog post published on the Pulumi site). Originally I started with TypeScript, but later switched to Go. Recently I had a need to add some VPC peering relationships to my lab configuration. I was concerned that this may pose some problems—due entirely to the way I structure my Pulumi projects and stacks—but as it turned out it was more straightforward than I expected. In this post, I’ll share some example code and explain what I learned in the process of writing it.

Some Background

First, let me share some background on how I structure my Pulumi projects and stacks.

It all starts with a Pulumi project that manages my base AWS infrastructure—VPC, subnets, route tables and routes, Internet gateways, NAT gateways, etc. I use a separate stack in this project for each region where I need base infrastructure.

All other projects build on “top” of this base project, referencing the resources created by the base project in order to create their own resources. Referencing the resources created by the base project is accomplished via a Pulumi StackReference.

In my Continue reading

Using the AWS CLI to Tag Groups of AWS Resources

To conduct some testing, I recently needed to spin up a group of Kubernetes clusters on AWS. Generally speaking, my “weapon of choice” for something like this is Cluster API (CAPI) with the AWS provider. Normally this would be enormously simple. In this particular case—for reasons that I won’t bother going into here—I needed to spin up all these clusters in a single VPC. This presents a problem for the Cluster API Provider for AWS (CAPA), as it currently doesn’t add some required tags to existing AWS infrastructure (see this issue). The fix is to add the tags manually, so in this post I’ll share how I used the AWS CLI to add the necessary tags.

Without the necessary tags, the AWS cloud provider—which is responsible for the integration that creates Elastic Load Balancers (ELBs) in response to the creation of a Service of type LoadBalancer, for example— won’t work properly. Specifically, the following tags are needed:<cluster-name>

The latter two tags are mutually exclusive: the former should be assigned to public subnets to tell the AWS cloud provider where to place public-facing ELBs, while the latter is assigned to private subnets Continue reading

Technology Short Take 143

Welcome to Technology Short Take #143! I have what I think is an interesting list of links to share with you this time around. Since taking my new job at Kong, I’ve been spending more time with Envoy, so you’ll see some Envoy-related content showing up in this Technology Short Take. I hope this collection of links has something useful for you!




  • I saw this blog post about Curiefense, an open source Envoy extension to add WAF (web application firewall) functionality to Envoy.
  • This post on using SPIFFE/SPIRE, Kubernetes, and Envoy together shows how to implement mutual TLS (mTLS) for a simple application. As a learning resource, I thought this post was helpful. However, Continue reading

Starting WireGuard Interfaces Automatically with Launchd on macOS

In late June of this year, I wrote a piece on using WireGuard on macOS via the CLI, where I walked readers using macOS through how to configure and use the WireGuard VPN from the terminal (as opposed to using the GUI client, which I discussed here). In that post, I briefly mentioned that I was planning to explore how to have macOS' launchd automatically start WireGuard interfaces. In this post, I’ll show you how to do exactly that.

These instructions borrow heavily from this post showing how to use macOS as a WireGuard VPN server. These instructions also assume that you’ve already walked through installing the necessary WireGuard components, and that you’ve already created the configuration file(s) for your WireGuard interface(s). Finally, I wrote this using my M1-based MacBook Pro, so my example files and instructions will be referencing the default Homebrew prefix of /opt/homebrew. If you’re on an Intel-based Mac, change this to /usr/local instead.

The first step is to create a launchd job definition. This file should be named <label>.plist, and it will need to be placed in a specific location. The <label> value is taken from the name given to the job Continue reading

An Alternate Approach to etcd Certificate Generation with Kubeadm

I’ve written a fair amount about kubeadm, which was my preferred way of bootstrapping Kubernetes clusters until Cluster API arrived. Along the way, I’ve also discussed using kubeadm to assist with setting up etcd, the distributed key-value store leveraged by the Kubernetes control plane (see here, here, and here). In this post, I’d like to revisit the topic of using kubeadm to set up an etcd cluster once again, this time taking a look at an alternate approach to generating the necessary TLS certificates than what the official documentation describes.

There is absolutely nothing wrong with the process the official documentation describes (I’m referring to this page, by the way); this process just creates slightly “cleaner” certificates. What do I mean by “cleaner” certificates? The official documentation uses a series of kubeadm configuration files, one for each etcd cluster member, to control how the utility creates the necessary certificates and configuration files. The user is instructed to use these configuration files on a single system to generate the certificates for all the cluster members. This works fine, with one caveat: each of the certificates will have an extra hostname—the hostname of the system being used Continue reading

Technology Short Take 142

Welcome to Technology Short Take #142! This time around, the Networking section is a bit light, but I’ve got plenty of cloud computing links and articles for you to enjoy, along with some stuff on OSes and applications, programming, and soft skills. Hopefully there’s something useful here for you!


  • Bill Doerrfeld discusses the use of WebAssembly to extend the Envoy proxy here after a discussion with Idit Levine, CEO and founder of Envoy is increasingly becoming an important component in many cloud-native networking solutions, so the ability to extend or customize Envoy’s behavior is quite important in my opinion.



Cloud Computing/Cloud Management

Adding Multiple Items Using Kustomize JSON 6902 Patches

Recently, I needed to deploy a Kubernetes cluster via Cluster API (CAPI) into a pre-existing AWS VPC. As I outlined in this post from September 2019, this entails modifying the CAPI manifest to include the VPC ID and any associated subnet IDs, as well as referencing existing security groups where needed. I knew that I could use the kustomize tool to make these changes in a declarative way, as I’d explored using kustomize with Cluster API manifests some time ago. This time, though, I needed to add a list of items, not just modify an existing value. In this post, I’ll show you how I used a JSON 6902 patch with kustomize to add a list of items to a CAPI manifest.

By the way, if you’re not familiar with kustomize, you may find my introduction to kustomize post to be helpful. Also, for those readers who are unfamiliar with JSON 6902 patches, the associated RFC is useful, as is this site.

In this particular case, the addition of the VPC ID and the subnet IDs were easily handled with a strategic merge patch that referenced the AWSCluster object. More challenging, though, was the reference to the existing security Continue reading

Using WireGuard on macOS via the CLI

I’ve written a few different posts on WireGuard, the “simple yet fast and modern VPN” (as described by the WireGuard web site) that aims to supplant tools like IPSec and OpenVPN. My first post on WireGuard showed how to configure WireGuard on Linux, both on the client side as well as on the server side. After that, I followed it up with posts on using the GUI WireGuard app to configure WireGuard on macOS and—most recently—making WireGuard from Homebrew work on an M1-based Mac. In this post, I’m going to take a look at using WireGuard on macOS again, but this time via the CLI.

Some of this information is also found in this WireGuard quick start. Here I’ll focus only on using macOS as a WireGuard client, not as a server; refer to the WireGuard docs (or to my earlier post) for information on setting up a WireGuard server. I’ll also assume that you’ve installed WireGuard via Homebrew.

Generating Keys

The first step is to generate the public/private keys you’ll need. If the /usr/local/etc/wireguard (or the /opt/homebrew/etc/wireguard for users on an M1-based Mac) directory doesn’t exist, you’ll need to first create that directory. (It didn’t exist Continue reading

Installing Older Versions of Kumactl on an M1 Mac

The Kuma community recently released version 1.2.0 of the open source Kuma service mesh, and along with it a corresponding version of kumactl, the command-line utility for interacting with Kuma. To make it easy for macOS users to get kumactl, the Kuma community maintains a Homebrew formula for the CLI utility. That includes providing M1-native (ARM64) macOS binaries for kumactl. Unfortunately, installing an earlier version of kumactl on an M1-based Mac using Homebrew is somewhat less than ideal. Here’s one way—probably not the only way—to work around some of the challenges.

Note that this post really only applies to users of M1-based Macs; users of Intel-based Macs can extract the kumactl binary from the release archive available linked from the Kuma install docs. (The same goes for users of Linux distributions running on Intel-based hardware.) On the Kuma website, simply select the desired version of Kuma from the drop-down in the upper left, check the page for the direct download link, and off you go. This doesn’t work for M1-based Macs because at the time this post was written, the Kuma community was not providing ARM64 binaries. This leaves Homebrew as the only way (aside Continue reading

1 2 3 30