Scott Lowe

Author Archives: Scott Lowe

Tracking EC2 Instances used by EKS with AWS CLI

As a sort of follow-up to my previous post on using the AWS CLI to track the specific Elastic Network Interfaces (ENIs) used by Amazon Elastic Kubernetes Service (EKS) cluster nodes, this post focuses on the EC2 instances themselves. I feel this is less of a “problem” than tracking ENIs, but I wanted to share this information nevertheless. In this post, I’ll show you which AWS CLI command to use to list all the EC2 instances associated with a particular EKS cluster.

If you read the previous post on tracking ENIs used by EKS, you might think that you could use a very similar AWS CLI command (aws ec2 describe-instances instead of aws ec2 describe-network-interfaces) to track the EC2 instances in a cluster—and you’d be mostly correct. Like the ENIs, EKS does add a cluster-specific tag to all EC2 instances in the cluster. However, just to make life interesting, the tag used for EC2 instances is not the same as the tag used for ENIs. (If someone at AWS knows of a technical reason why these tags are different, I’d love to hear it.)

Instead of using the cluster.k8s.amazonaws.com/name tag that is used Continue reading

Tracking ENIs used by EKS with AWS CLI

I’ve recently been spinning up lots of Amazon Elastic Kubernetes Service (EKS) clusters (using Pulumi, of course) in order to test various Cilium configurations. Along the way, I’ve wanted to verify the association and configuration of Elastic Network Interfaces (ENIs) being used by the EKS cluster. In this post, I’ll share a couple of AWS CLI commands that will help you track the ENIs used by an EKS cluster.

When I first set out to find the easiest way to track the ENIs used by the nodes in an EKS cluster, I thought that AWS resource tags might be the key. I was right—but not in the way I expected. In the Pulumi program (written in Go) that I use to create EKS clusters, I made sure to tag all the resources.

For example, when defining the EKS cluster itself I assigned tags:

eksCluster, err := eks.NewCluster(ctx, "eks-cluster", &eks.ClusterArgs{
    Name:    pulumi.Sprintf("%s-test", regionNames[awsRegion]),
    // Some code omitted here for brevity
    Tags: pulumi.StringMap{
        "Name":   pulumi.Sprintf("%s-test", regionNames[awsRegion]),
        "owner":  pulumi.String(ownerTag),
         Continue reading

Technology Short Take 176

Welcome to Technology Short Take #176! This Tech Short Take is a bit heavy on security-related links, but there’s still some additional content in a number of other areas, so you should be able to find something useful—or at least interesting—in here. Thanks for reading!

Networking

Servers/Hardware

Security

  • In early February a vulnerability was uncovered in a key component of the Linux boot process. The vulnerability affects virtually all Linux distributions and allows attackers to bypass the secure boot protections and insert a low-level bootkit. While the requirements for exploiting the vulnerability Continue reading

Linting your Markdown Files

It’s no secret I’m a fan of Markdown. The earliest mention of Markdown on this site is all the way back in 2011, and it was only a couple years after that when I migrated this site from WordPress to Markdown. Back then, the site was generated from Markdown using Jekyll (via GitHub Pages); today it is generated from Markdown sources using Hugo. One thing I’ve not done, though, is perform linting (checking for errors or potential errors) of the Markdown source files. That’s all about to change! In this post, I’ll share with you how I started linting my Markdown files.

To handle the linting, there are (at least) a couple different options:

  1. markdownlint-cli (GitHub repository)
  2. markdownlint-cli2 (GitHub repository)

Both of these use the same markdownlint library under the hood. They’re both available as both a CLI tool or as a Docker container; markdownlint-cli2 is also available as a GitHub Action. In both cases, the CLI tool is installed via npm install (typically globally with --global or -g). The key difference between the two is that markdownlint-cli2 is configuration-driven, whereas markdownlint-cli offers the ability to use either a configuration file or command-line flags. I Continue reading

Technology Short Take 175

Welcome to Technology Short Take #175! Here’s your weekend reading—a collection of links and articles from around the internet on a variety of data center- and cloud-related topics. I hope you find something useful here!

Networking

Security

  • I attended a local meetup here in the Denver metro area a short while ago and was introduced to sops.
  • AMD processors have been discovered to have multiple security flaws; more details available here.
  • The Linux kernel project has become a CVE Numbering Authority (CNA); Greg Kroah-Hartman wrote a blog post that discusses this in more depth.

Cloud Computing/Cloud Management

Technology Short Take 174

Welcome to Technology Short Take #174! For your reading pleasure, I’ve collected links on topics ranging from Kubernetes Gateway API to recent AWS attack techniques to some geeky Linux and Git topics. There’s something here for most everyone, I’d say! But enough of my rambling, let’s get on to the good stuff. Enjoy!

Networking

  • I want to be Ivan Pepelnjak when I grow up. Why? Read this article on his response to someone wanting to use NSX to create availability zones.
  • Nico Vibert has a tutorial that takes readers through using Cilium’s Gateway API functionality to do L7 traffic management (HTTP redirects, HTTP rewrites, and HTTP mirroring).

Security

Using NAT Instances on AWS with Pulumi

For folks using AWS in their day-to-day jobs, it comes as no secret that AWS’ Managed NAT Gateway—responsible for providing outbound Internet connectivity to otherwise private subnets—is an expensive proposition. While the primary concern for large organizations is the data processing fee, the concern for smaller organizations or folks like me who run a cloud-based lab instead of a hardware-based home lab is the per-hour cost. In this post, I’ll show you how to use Pulumi to use a NAT instance for outbound Internet connectivity instead of a Managed NAT Gateway.

For a bit more about why Managed NAT Gateways aren’t ideal for larger organizations, I’d recommend this article by Corey Quinn. For smaller organizations or cloud-based labs, data processing fees probably aren’t the main concern (although I could be wrong); it would be the ~$32/mo per Managed NAT Gateway. Since many tools configure a Managed NAT Gateway per availability zone, now you’re talking more like $96/mo—and you haven’t even spun up any real workloads yet! Running your own NAT instance can dramatically reduce but not eliminate this expense.

Now that I’ve established why running a NAT instance can be beneficial, let’s review what you’ll need to have installed in Continue reading

Using SSH with the Pulumi Docker Provider

In August 2023, Pulumi released a version of the Docker provider that supported SSH-based connections to a Docker daemon. I’ve written about using SSH with Docker before (see here), and I sometimes use AWS-based “Docker build hosts” with my M-series Macs to make it easier/simpler (and sometimes faster) to build x86_64-based Docker images. Naturally, I’m using an SSH connection in those cases. Until this past weekend, however, I hadn’t really made the time to look deeper into how to use SSH with the Pulumi Docker provider. In this post, I’ll share some details that (unfortunately) haven’t yet made it into the documentation about using SSH with the Pulumi Docker provider.

First, let’s talk about some prerequisites to making this work.

  1. You’ll need Docker installed locally. I fairly certain this is only the docker CLI (much in the same way the Pulumi Kubernetes provider requires kubectl to be installed locally), but I haven’t verified this for certain yet. I tested this from a Linux system running Docker 24.0.7; I think the earliest version that is supported is 18.09.
  2. You’ll need Docker installed on the remote SSH host (obviously). I used Flatcar Container Linux (stable channel) on AWS.
  3. Continue reading

Technology Short Take 173

Welcome to Technology Short Take #173! After a lull in links to share last time around, it looks like things have rebounded and folks are in full swing writing new content for me to share with you. I think I have a decent round-up of links for you; hopefully you can find something useful here. Enjoy!

Networking

Servers/Hardware

Security

Cloud Computing/Cloud Management

Technology Short Take 172

Welcome to Technology Short Take #172, the first Technology Short Take of 2024! This one is really short, which I’m assuming reflects a lack of blogging activity over the 2023 holiday season. Nevertheless, I have managed to scrape together a few links to share with readers. As usual, I hope you find something useful. Enjoy!

Networking

  • Via this blog post, I learned that Ivan Pepelnjak has a GitHub repository of hands-on examples for learning public cloud networking (including both AWS and Azure). Ivan’s materials are always excellent, so if you’re looking for resources to help with expanding your networking skills into the public cloud, this should be on the short list. (I plan to submit a PR soon to add Pulumi examples, which the repository is currently missing.)

Cloud Computing/Cloud Management

  • Jonathan Major shares his experience using Pulumi with Google Cloud APIs. I think there are some code snippets in Jonathan’s article, but my instance of Firefox wouldn’t render the code snippets (instead only showing empty black boxes).
  • Nisar Ahmad explores whether Terraform or Pulumi is better for your use case.

Operating Systems/Applications

Selectively Replacing Resources with Pulumi

Because Pulumi operates declaratively, you can write a Pulumi program that you can safely run (via pulumi up) multiple times. If no changes are needed—meaning that the current state of the infrastructure matches what you’ve defined in your Pulumi program—then nothing happens. If only one resource needs to be updated, then it will update only that one resource (and any dependencies, if there are any). There may be times, however, when you want to force the replacement of specific resources. In this post, I’ll show you how to target specific resources for replacement when using Pulumi.

Here’s an example: I use Pulumi to manage my AWS-based lab resources, including SSH bastion hosts. However, because my code uses a dynamic AMI lookup, I’ve instructed Pulumi to ignore changes in the AMI ID for the bastion hosts (by appending pulumi.IgnoreChanges([]string{"ami"}) as a resource option). This gives me the control over when the bastion hosts get replaced, instead of Pulumi wanting to replace them every time the AMI ID changes.

With this in place, then, how do I tell Pulumi that I’m ready to replace the bastion hosts? Tearing down the entire stack isn’t an option. Fortunately, the pulumi CLI Continue reading

Dynamically Enabling the Azure CLI with Direnv

I’m a big fan of direnv, the tool that lets you load and unload environment variables depending on the current directory. It’s so very useful! Not too terribly long ago, I wanted to find a way to “dynamically activate” the Azure CLI using direnv. Basically, I wanted to be able to have the Azure CLI disabled (no configuration information) unless I was in a directory where I needed or wanted it to be active, and be able to make it active using direnv. I finally found a way to make it work, and in this blog post I’ll share how you can do this, too.

First, you’ll need both direnv and the Azure CLI installed (obviously). I’ll leave this as an exercise for the readers, but I’ll mention that if you want to use Azure CLI in a Python virtual environment you might find this article really helpful.

Next, you’ll want to create a couple of directories. I chose to “hide” these directories in a .config directory in my home directory. This directory is very commonly found (and used) on many Linux systems, but doesn’t typically exist on a macOS system. You can use this command to create the Continue reading

Conditional Git Configuration

Building on the earlier article on automatically transforming Git URLs, I’m back with another article on a (potentially powerful) feature of Git—the ability to conditionally include Git configuration files. This means you can configure Git to be configured (and behave) differently based on certain conditions, simply by including or not including Git configuration files. Let’s look at a pretty straightforward example taken from my own workflow.

Here’s a configuration stanza from my own system-wide Git configuration:

[includeIf "gitdir:~/Work/Code/Repos/"]
    path = ~/Work/Code/Repos/.gitconfig

The key here is the includeIf keyword. In this case, Git will include the referenced configuration file specified by path, if the location of the Git repository matches the path specification after gitdir. Basically, what this means is that all repositories under ~/Work/Code/Repos will trigger the inclusion of the additional configuration file.

Here’s the additional configuration file:

[user]
    email = name@work-domain.com
    name = Scott Lowe
[commit]
    gpgsign = false

As long as I group all work-relatd repositories in the specified directory path, these values override the system-wide values. This means I can specify my work e-mail address as the e-mail Continue reading

Automatically Transforming Git URLs

Git is one of those tools that lots of people use, but few people truly master. I’m still on my own journey of Git mastery, and still have so very far to go. However, I did take one small step forward recently with the discovery of the ability for Git to automatically rewrite remote URLs. In this post, I’ll show you how to configure Git to automatically transform the URLs of Git remotes.

The key here is the url configuration stanza and the associated insteadOf keyword. Added to your Git configuration—either globally or on a per-repository basis—these configuration options will tell Git to use a different URL every time it encounters the specified original URL.

Here’s an example:

[url "[email protected]:org/"]
    insteadOf = "https://github.com/org/"

The [email protected]:org/ is the replacement URL; that is, the URL that you want Git to use. The URL specified by the insteadOf keyword is the original URL; that is, the URL you want Git to replace. As you can see in the example, it’s possible not only to transform HTTPS-based URLs to SSH URLs (or vice versa), but it’s possible to constrain this transformation to repositories belonging to a specific organization or Continue reading

Technology Short Take 171

Welcome to Technology Short Take #171! This is the next installation in my semi-regular series that shares links and articles from around the interwebs on various technology areas of interest. Let the linking begin!

Networking

The networking section this time around is focused on application level protocols…but hey, they’re still networking protocols, right?

Security

Cloud Computing/Cloud Management

Saying Goodbye to the Full Stack Journey

In January 2016, I published the first-ever episode of the Full Stack Journey podcast. In October 2023, the last-ever episode of the Full Stack Journey podcast was published. After almost seven years and 83 episodes, it was time to end my quirky, eclectic, and unusual podcast that explored career journeys alongside various technologies, products, and open source projects. In this post, I wanted to share a few thoughts about saying goodbye to the Full Stack Journey.

First and foremost, let me say that I really enjoyed being the host of the Full Stack Journey podcast—far more than I expected I would, if I’m honest. While I didn’t love the logistics of producing a podcast, I did love getting to talk with folks, hear their stories, and learn about new things. So, while part of me is thankful to have a little less work to do, another part—a larger part—is sad to see it end.

That being said, some of you are probably wondering why it ended. I mentioned that I didn’t enjoy the logistics of producing a podcast; specifically, I didn’t enjoy audio editing. Some folks like it, but I didn’t. It was truly a chore for me. That was Continue reading

Guest Post: Moving Secrets Where They Belong

by Simen A.W. Olsen

Pulumi recently shipped Pulumi ESC, which adds the “Environment” tab to Pulumi Cloud. For us at Bjerk, this means we can move secrets into a secrets manager like Google Secrets Manager. Let me show you how we did it!

We are already rotating secrets with our own CLI tool, which works fine, meaning we are getting notifications in our Slack channel—which everyone tends to ignore until something real breaks. If you are curious how we are handling it today, we are using our own NPM package that throws an exception if a secret has expired. To ensure everything works smoothly, we utilize a GitHub Actions workflow that is scheduled to run daily for drift checking.

The secrets are shared between stacks using StackReferences, which has served us well.

Improving security

One issue with our current setup is that we publicly store encrypted secrets in our repository. Previously, we’ve thought of using Google Secrets Manager with the GetSecret function. That comes with its own territory, such as permissions to the secret and managing those permissions—not to mention that we already use multiple secret managers/vaults.

Now, with Pulumi ESC, it’s time to pick this Continue reading

Assigning Tags by Default on AWS with Pulumi

Appropriately tagging resources on AWS is an important part of effectively managing infrastructure resources for many organizations. As such, an infrastructure as code (IaC) solution for AWS must have the ability to ensure that resources are always created with the appropriate tags. (Note that this is subtly different from a policy mechanism that prevents resources from being created without the appropriate tags.) In this post, I’ll show you a couple of ways to assign tags by default when creating AWS resources with Pulumi. Code examples are provided in Golang.

There are at least two ways (perhaps more) of handling this with Pulumi:

  1. Adding the default tags to the stack configuration
  2. Adding the default tags to an explicit provider

Each approach has its advantages and disadvantages, so there isn’t—in my opinion, at least—a definitive “best way” to doing this. The best way for you will depend on your specific circumstances.

In both cases, the solution involves modifying the configuration of the resource provider Pulumi uses to provision AWS resources. Pulumi supports the notion of both default providers and explicit providers. The former are created automatically and are configured via the stack configuration. (In fact, using stack configuration is currently the Continue reading

Technology Short Take 170

Welcome to Technology Short Take #170! I had originally intended to get this published before the long Labor Day weekend, but didn’t quite have it ready. So, here you go—here’s your latest collection of links from around the internet focused on data center and cloud-related technologies. I hope that you find something useful here.

Networking

Servers/Hardware

  • I must admit that I always wanted to have a Sun workstation, and I’ve had an interest in SunOS/Solaris for years (check out this link if you don’t believe me). So, it’s natural that this post on reliving Continue reading

Mac, iPad, or Both?

Both Jason Snell and John Gruber, both stalwarts in the Apple journalism world, have recently weighed in on this topic. Jason says he’s given up on the iPad-only travel dream; John says he keeps throwing his iPad in his bag when he travels, even if he never uses it. I have thoughts on this topic—as you might think, considering I decided to write about it! (Ah, but what device did I use to write?)

Jason kicks off the discussion with a review of his iPad travel usage, which until the arrival of Apple Silicon, was going along swimmingly. Now, with Apple Silicon-powered Macs, things are different:

In the battle between iPad and Mac, I’m a longtime member of Team Both—I use my Mac most of the day at my desk, but when I write elsewhere in the house or backyard, I switch to an iPad Pro in the Magic Keyboard case. And that iPad (in a regular case) is my primary computing device when I’m not in work mode…But here I sit at my mother’s dining room table, typing on a MacBook Air. Something has changed in my approach to travel, and I’m trying to understand just Continue reading

1 2 3 … 19