Creating Tagged Subnets Across AWS AZs Using Pulumi
As I mentioned back in May in this post on creating a sandbox for learning Pulumi, I’ve started using Pulumi more and more of my infrastructure-as-code needs. I did switch from JavaScript to TypeScript (which I know compiles to JavaScript on the back-end, but the strong typing helps a new programmer like me). Recently I had a need to create some resources in AWS using Pulumi, and—for reasons I’ll explain shortly—many of the “canned” Pulumi examples didn’t cut it for my use case. In this post, I’ll share how I created tagged subnets across AWS availability zones (AZs) using Pulumi.
In this particular case, I was using Pulumi to create all the infrastructure necessary to spin up an AWS-integrated Kubernetes cluster. That included a new VPC, subnets in the different AZs for that region, an Internet gateway, route tables and route table associations, security groups, an ELB for the control plane, and EC2 instances. As I’ve outlined in my latest post on setting up an AWS-integrated Kubernetes 1.15 cluster using kubeadm, these resources on AWS require specific AWS tags to be assigned in order for the AWS cloud provider to work.
As I started working on this, Continue reading
How to properly remove LuCi from OpenWrt
Overview
LuCi is a very popular OpenWrt web interface. For an average user, LuCi is probably one of the main deciding factors between giving OpenWrt a try in the first place, or moving on to another user friendlier firmware like DD-WRT.
If you’re an advanced user however, most of the times you may find yourself adjusting settings either through UCI or by editing the config files manually. In fact at one point you may realize you’re not using LuCi at all and it’s just sitting there idle. Basically a component that’s not only using resources, but also providing an extra attack surface.
Now, one could just disable uHTTPd to address some of these concerns, but LuCi installs too many dependencies, and cluttering a router with things that you’ll hardly ever use, is not the best use of the very limited storage space available in most routers.
Another method that some use to “remove” LuCi, is by issuing:
opkg --autoremove remove luci
This may seem to work, but in reality LuCi packages are not really removed this way and the related files will only be masked by OverlayFS. This is because the packages are built into the firmware itself.
While OpenWrt Continue reading
Learn About Modern Apps with Docker at VMworld 2019
The Docker team will be on the show floor at VMworld the week of August 25. We’ll be talking about the state of modern application development, how to accelerate innovation efforts, and the role containerization and Docker play in powering these initiatives.
At VMworld, we’ll be talking about:
What’s New in Docker Enterprise 3.0
Docker Enterprise 3.0 shipped recently, making it the first and only desktop-to-cloud container platform in the market that lets you build and share any application and securely run them anywhere – from hybrid cloud to the edge. At VMworld, we’ll have demos that shows how Docker Enterprise 3.0 simplifies Kubernetes with the Docker Kubernetes Service (DKS) and enables companies to more easily build modern applications with Docker Desktop Enterprise and Docker Application.Accelerating Your Journey to the Cloud
Everyone is talking about moving workloads to the cloud to drive efficiencies and simplify ops, but many existing applications that power Continue readingReconstructing the Join Command for Kubeadm
If you’ve used kubeadm to bootstrap a Kubernetes cluster, you probably know that at the end of the kubeadm init command to bootstrap the first node in the cluster, kubeadm prints out a bunch of information: how to copy over the admin Kubeconfig file, and how to join both control plane nodes and worker nodes to the cluster you just created. But what if you didn’t write these values down after the first kubeadm init command? How does one go about reconstructing the proper kubeadm join command?
Fortunately, the values needed for a kubeadm join command are relatively easy to find or recreate. First, let’s look at the values that are needed.
Here’s the skeleton of a kubeadm join command for a control plane node:
kubeadm join <endpoint-ip-or-dns>:<port> \
--token <valid-bootstrap-token> \
--discovery-token-ca-cert-hash <ca-cert-sha256-hash> \
--control-plane \
--certificate-key <certificate-key>
And here’s the skeleton of a kubeadm join command for a worker node:
kubeadm join <endpoint-ip-or-dns>:<port> \
--token <valid-bootstrap-token> \
--discovery-token-ca-cert-hash <ca-cert-sha256-hash> \
As you can see, the information needed for the worker node is a subset of the information needed for a control plane node.
Here’s how to find or recreate all the various pieces of information you need:
- The Continue reading
AnsibleFest Atlanta – Here We Come!
AnsibleFest Atlanta is September 24th - 26th at the Hilton Atlanta, a few short blocks from Centennial Olympic Park. This year is going to be bigger and better than ever. As AnsibleFest continues to grow, so does its offerings. We are excited to offer more breakout sessions, more hands-on workshops, and more Ask an Expert sessions. This year we have expanded our AnsibleFest programming to offer 10 different tracks. We are also introducing the Open Lounge this year, which is a place to network, relax and recharge. It provides a great opportunity to meet and connect with passionate Ansible users, developers, and industry partners.
The AnsibleFest Agenda is live. Thank you to everyone who answered the call for submission. It was a challenge to narrow down the sessions from the record-setting submissions we received. We love our community, customers, partners, and appreciate everyone who contributed.
For those who are not familiar with AnsibleFest, or have not attended the event before, below are a few highlights of AnsibleFest that you won’t want to miss.
General Sessions
We have some amazing general sessions planned this year. The opening keynote at AnsibleFest will feature talks from Red Hat Ansible Automation Continue reading
Developing Docker-Powered Applications on Windows with WSL 2
This is a guest post from Docker Captain Antonis Kalipetis, a Senior Software Engineer at e-food — the leading online food delivery service in Greece. He is a Python lover and developer and helps teams embrace containers and improve their development workflow. He loves automating stuff and sharing knowledge around all things containers, DevOps and developer workflows. You can follow him on Twitter @akalipetis. WSL 2 (or Windows Subsystem for Linux version 2) is Microsoft’s second take on shipping a Linux Kernel with Windows. The first version was awesome as it translated Linux system calls to the equivalent Windows NT call in real time. The second version includes a full fledged virtual machine. It was only natural that Docker would embrace this change and ship a Docker Desktop for Windows version that runs on WSL 2 (WSL 1 had issues running the Docker daemon). This is still a Technical Preview, but after using it for a couple of days, I’ve completely switched my local development to take advantage of it and I’m pretty happy with it. In this blog, I’ll show you an example of how to develop Docker-powered applications using the Docker Desktop WSL 2 Tech Preview.Continue reading
Setting up an AWS-Integrated Kubernetes 1.15 Cluster with Kubeadm
In this post, I’d like to walk through setting up an AWS-integrated Kubernetes 1.15 cluster using kubeadm. Over the last year or so, the power and utility of kubeadm has vastly improved (thank you to all the contributors who have spent countless hours!), and it is now—in my opinion, at least—at a point where setting up a well-configured, highly available Kubernetes cluster is pretty straightforward.
This post builds on the official documentation for setting up a highly available Kubernetes 1.15 cluster. This post also builds upon previous posts I’ve written about setting up Kubernetes clusters with the AWS cloud provider:
- Setting up the AWS Cloud Provider
- Kubernetes, Kubeadm, and the AWS Cloud Provider
- Using Kubeadm to Add New Control Plane Nodes with AWS Integration
All of these posts are focused on Kubernetes releases prior to 1.15, and given the changes in kubeadm in the 1.14 and 1.15 releases, I felt it would be helpful to revisit the process again for 1.15. For now, I’m focusing on the in-tree AWS cloud provider; however, in the very near future I’ll look at using the new external AWS cloud provider.
As pointed out in the “original” Continue reading
Deploying Dockerized .NET Apps Without Being a DevOps Guru
This is a guest post by Julie Lerman. She is a Docker Captain, published author, Microsoft Regional Director and a long-time Microsoft MVP who now counts her years as a coder in decades. She makes her living as a coach and consultant to software teams around the world. You can follow Julie on her blog at thedatafarm.com/blog, or on Twitter at @julielerman.
Converting Kubernetes to an HA Control Plane
While hanging out in the Kubernetes Slack community, one question I’ve seen asked multiple times involves switching a Kubernetes cluster from a non-HA control plane (single control plane node) to an HA control plane (multiple control plane nodes). As far as I am aware, this isn’t documented upstream, so I thought I’d walk readers through what this process looks like.
I’m making the following assumptions:
- The existing single control plane node was bootstrapped using
kubeadm. (This means we’ll usekubeadmto add the additional control plane nodes.) - The existing single control plane node is using a “stacked configuration,” in which both etcd and the Kubernetes control plane components are running on the same nodes.
I’d also like to point out that there are a lot of different configurations and variables that come into play with a process like this. It’s (nearly) impossible to cover them all in a single blog post, so this post attempts to address what I believe to be the most common situations.
With those assumptions and that caveat in mind, the high-level overview of the process looks like this:
- Create a load balancer for the control plane.
- Update the API server’s certificate.
- Update the kubelet Continue reading
See Docker Enterprise 3.0 in Action in Our Upcoming Webinar Series
To showcase all of the capabilities of the platform and highlight what is new in this release, we invite you to join our 5-part webinar series to explore the technologies that make up Docker Enterprise 3.0. You’ll see several demos of the platform and gain a better understanding of how Docker can you help your organization deliver high-velocity innovation while providing you the choice and security you need. We designed the webinar both for those new to containers and Kubernetes, as well as those who are just here to learn more about what’s new. We’re excited to share what we’ve been working on.
Here’s an overview of what we’ll be covering in each session.
Part 1: Content Management
Tuesday, August 13, 2019 @ 11am PDT / 2pm EDTThis webinar will cover the Continue reading
Docker Release Party Recap
We Celebrated the Launch of Docker Enterprise 3.0 and Docker 19.03 Last Week
Last week, Docker Captain Bret Fisher hosted a 3-day Release Party for Docker 19.03 and Docker Enterprise 3.0. Captains and the Docker team demonstrated some of their favorite new features and answered live audience questions. Here are the highlights (You can check out the full release party here).Docker Desktop Enterprise
To kick things off, Docker Product Manager Ben De St Paer-Gotch shared Docker Desktop Enterprise. Docker Desktop Enterprise ships with the Enterprise Engine and includes a number of features that makes enterprise development easier and more productive. For example, version packs allow developers to switch between Docker Engine versions and Kubernetes versions, all from the desktop.For admins, Docker Desktop Enterprise includes the ability to lock down the settings of Docker Desktop, so developers’ machines stay aligned with corporate requirements. Ben also demonstrated Docker Application Designer, a feature that allows users to create new Docker applications by using a library of templates, making it easier for developers in the enterprise to get updated app templates – or “gold standard” versions like the right environment variable settings, custom code, custom editor settings, Continue reading
Technology Short Take 117
Welcome to Technology Short Take #117! Here’s my latest gathering of links and articles from the around the World Wide Web (an “old school” reference for you right there). I’ve got a little bit of something for most everyone, except for the storage nerds (I’m leaving that to my friend J Metz this time around). Here’s hoping you find something useful!
Networking
- Ronald de Jong sheds some light on a potential “gotcha” with distributed multi-tier routing configuration in NSX. The key takeaway here is that anytime there is firewalling or other stateful services in place, the traffic will route to the (non-distributed) service router instead of just the distributed routers.
- Northbound Networks provides some information on the P4 programming language.
- Via the Packet Pushers blog, Daniel Himes provides a guide to a starting point for enterprise (network) automation.
Servers/Hardware
- AnandTech has a nice article on the future PCIe 6.0 spec, expected in to land in 2021.
Security
- Ronnie Flathers shares how he uses Docker for penetration testing.
- Interested in running Hashcat on AWS EC2? See this article by Ben Mason.
- Another data breach, this time from Capital One, hit the news this past week. Corey Quinn has his Continue reading
Accessing the Docker Daemon via an SSH Bastion Host
Today I came across this article, which informed me that (as of the 18.09 release) you can use SSH to connect to a Docker daemon remotely. That’s handy! The article uses docker-machine (a useful but underrated tool, I think) to demonstrate, but the first question in my mind was this: can I do this through an SSH bastion host? Read on for the answer.
If you’re not familiar with the concept of an SSH bastion host, it is a (typically hardened) host through which you, as a user, would proxy your SSH connections to other hosts. For example, you may have a bunch of EC2 instances in an AWS VPC that do not have public IP addresses. (That’s reasonable.) You could use an SSH bastion host—which would require a public IP address—to enable SSH access to otherwise inaccessible hosts. I wrote a post about using SSH bastion hosts back in 2015; give that post a read for more details.
The syntax for connecting to a Docker daemon via SSH looks something like this:
docker -H ssh://user@host <command>
So, if you wanted to run docker container ls to list the containers running on a remote system, you’d Continue reading
Kubernetes Operators with Ansible Deep Dive: Part 2
In part 1 of this series, we looked at operators overall, and what they do in OpenShift/Kubernetes. We peeked at the Operator SDK, and why you'd want to use an Ansible Operator rather than other kinds of operators provided by the SDK. We also explored how Ansible Operators are structured and the relevant files created by the Operator SDK when building Kubernetes Operators with Ansible.
In this the second part of this deep dive series, we'll:
- Take a look at creating an OpenShift Project and deploying a Galera Operator
- Next we’ll check the MySQL cluster, then setup and test a Galera cluster
- Then we’ll test scaling down, disaster recovery, and demonstrate cleaning up
Creating the project and deploying the operator
We start by creating a new project in OpenShift, which we'll simply call test:
$ oc new-project test --display-name="Testing Ansible Operator" Now using project "test" on server "https://ec2-xx-yy-zz-1.us-east-2.compute.amazonaws.com:8443".
We won't delve too much into this role, however the basic operation is:
- Use set_fact to generate variables using the k8s lookup plugin or other variables defined in defaults/main.yml.
- Determine if any corrective action needs to be taken based on the above variables. For example, one Continue reading
5 Things to Try with Docker Desktop WSL 2 Tech Preview
As a refresher, this preview makes use of the new Windows Subsystem for Linux (WSL) version that Microsoft recently made available on Windows insider fast ring. It has allowed us to provide improvements to file system sharing, boot time and access to some new features for Docker Desktop users.
To do this we have changed quite a bit about how we interact with the operating system compared to Docker Desktop on Windows today:
Want to give it a go?
- Get setup on a Windows machine on the latest Windows Insider build. The first step for this is heading over to the Microsoft and getting set up as a Windows Insider: https://insider.windows.com/en-gb/getting-started/
- You’ll need to install the latest release branch (at least build version 18932) and you will then want to enable the WSL 2 feature in Windows: https://docs.microsoft. Continue reading
Write Maintainable Integration Tests with Docker
Testcontainer is an open source community focused on making integration tests easier across many languages. Gianluca Arbezzano is a Docker Captain, SRE at Influx Data and the maintainer of the Golang implementation of Testcontainer that uses the Docker API to expose a test-friendly library that you can use in your test cases.
All of these integration points require different layers of testing. Unit tests increase how fast you write code because you can mock all of your dependencies, set the expectation for your function and iterate until you get the desired transformation. But, we need more. We need to make sure that the integration with Redis, MongoDB or a microservice works as expected, not just that the mock works as we wrote it. Both are Continue reading
Decoding a Kubernetes Service Account Token
Recently, while troubleshooting a separate issue, I had a need to get more information about the token used by Kubernetes Service Accounts. In this post, I’ll share a quick command-line that can fully decode a Service Account token.
Service Account tokens are stored as Secrets in the “kube-system” namespace of a Kubernetes cluster. To retrieve just the token portion of the Secret, use -o jsonpath like this (replace “sa-token” with the appropriate name for your environment):
kubectl -n kube-system get secret sa-token \
-o jsonpath='{.data.token}'
The output is Base64-encoded, so just pipe the output into base64:
kubectl -n kube-system get secret sa-token \
-o jsonpath='{.data.token}' | base64 --decode
The result you’re seeing is a JSON Web Token (JWT). You could use the JWT web site to decode the token, but given that I’m a fan of the CLI I decided to use this JWT CLI utility instead:
kubectl -n kube-system get secret sa-token \
-o jsonpath='{.data.token}' | base64 --decode | \
jwt decode -
The final -, for those who may not be familiar, is the syntax to tell the jwt utility to look at STDIN for the JWT it needs to Continue reading
How to compile OpenWrt and still use the official repository
Overview
We all know what OpenWrt is. The amazing Linux distro built specifically for embedded devices.
What you can achieve with a rather cheap router running OpenWrt, is mind-boggling.
OpenWrt also gives you a great control over its build system. For normal cases, you probably don’t need to build OpenWrt from source yourself. That has been done for you already and all you need to do, is to just download the appropriate compiled firmware image and then upload it to your router1.
But for more advanced usages, you may find yourself needing to build OpenWrt images yourself. This could be due wanting to make some changes to the code, add some device specific options, etc.
Building OpenWrt from source is easy, well-documented, and works great. That is, until you start using opkg to install some new packages.
opkg will by default fetch new packages from the official repository (as one might expect), but depending on the package, the installation may or may not fail.
If you only want to add/remove some packages from a firmware, building OpenWrt from scratch is an overkill. You want to use OpenWrt Image Builder instead. OpenWrt Image Builder also does not suffer from Continue reading
Top 4 Tactics To Keep Node.js Rockin’ in Docker
This is a guest post from Docker Captain Bret Fisher, a long time DevOps sysadmin and speaker who teaches container skills with his popular Docker Mastery courses including Docker Mastery for Node.js, weekly YouTube Live shows, and consults to companies adopting Docker. Join Bret for an online meetup on August 28th, where he’ll give demos and Q&A on Node.js and Docker topics.
There’s a ton of info out there on using Node.js with Docker, but so much of it is years out of date, and I’m here to help you optimize your setups for Node.js 10+ and Docker 18.09+. If you’d rather watch my DockerCon 2019 talk that covers these topics and more, check it out on YouTube.
Let’s go through 4 steps Continue reading
Adding a Name to the Kubernetes API Server Certificate
In this post, I’m going to walk you through how to add a name (specifically, a Subject Alternative Name) to the TLS certificate used by the Kubernetes API server. This process of updating the certificate to include a name that wasn’t included could find use for a few different scenarios. A couple of situations come to mind, such as adding a load balancer in front of the control plane, or using a new or different URL/hostname used to access the API server (both situations taking place after the cluster was bootstrapped).
This process does assume that the cluster was bootstrapped using kubeadm. This could’ve been a simple kubeadm init with no customization, or it could’ve been using a configuration file to modify the behavior of kubeadm when bootstrapping the cluster. This process also assumes your Kubernetes cluster is using the default certificate authority (CA) created by kubeadm when bootstrapping a cluster. Finally, this process assumes you are using a non-HA (single control plane node) configuration.
Before getting into the details of how to update the certificate, I’d like to first provide a bit of background on why this is important.
Background
The Kubernetes API server uses digital certificates to both Continue reading