0

Creating an AWS VPC Endpoint with Pulumi

In this post, I’d like to show readers how to use Pulumi to create a VPC endpoint on AWS. Until recently, I’d heard of VPC endpoints but hadn’t really taken the time to fully understand what they were or how they might be used. That changed when I was presented with a requirement for the AWS EC2 APIs to be available within a VPC that did not have Internet access. As it turns out—and as many readers are probably already aware—this is one of the key use cases for a VPC endpoint (see the VPC endpoint docs). The sample code I’ll share below shows how to programmatically create a VPC endpoint for use in infrastructure-as-code use cases.

For those that aren’t familiar, Pulumi allows users to use one of a number of different general-purpose programming languages and apply them to infrastructure-as-code scenarios. In this example, I’ll be using TypeScript, but Pulumi also supports JavaScript and Python (and Go is in the works). (Side note: I intend to start working with the Go support in Pulumi when it becomes generally available as a means of helping accelerate my own Go learning.)

Here’s a snippet of TypeScript code that Continue reading

0

Manually Loading Container Images with containerD

I recently had a need to manually load some container images into a Linux system running containerd (instead of Docker) as the container runtime. I say “manually load some images” because this system was isolated from the Internet, and so simply running a container and having containerd automatically pull the image from an image registry wasn’t going to work. The process for working around the lack of Internet access isn’t difficult, but didn’t seem to be documented anywhere that I could readily find using a general web search. I thought publishing it here may help individuals seeking this information in the future.

For an administrator/operations-minded user, the primary means of interacting with containerd is via the ctr command-line tool. This tool uses a command syntax very similar to Docker, so users familiar with Docker should be able to be productive with ctr pretty easily.

In my specific example, I had a bastion host with Internet access, and a couple of hosts behind the bastion that did not have Internet access. It was the hosts behind the bastion that needed the container images preloaded. So, I used the ctr tool to fetch and prepare the images on the bastion, then Continue reading

0

Thinking and Learning About API Design

In July of 2018 I talked about Polyglot, a very simple project I’d launched whose only purpose was simply to bolster my software development skills. Work on Polyglot has been sporadic at best, coming in fits and spurts, and thus far focused on building a model for the APIs that would be found in the project. Since I am not a software engineer by training (I have no formal training in software development), all of this is new to me, and I’ve found myself encountering lots of questions about API design along the way. In the interest of helping others who may be in a similar situation, I thought I’d share a bit here.

I initially approached the API in terms of how I would encode (serialize?) data on the wire using JSON (I’d decided on using a RESTful API with JSON over HTTP). Starting with how I anticipated storing the data in the back-end database, I created a representation of how a customer’s information would be encoded (serialized) in JSON:

{
    "customers": [
        {
            "customerID": "5678",
            "streetAddress": "123 Main Street",
            "unitNumber": "Suite 123",
            "city": "Anywhere",
            "state": "CO",
            "postalCode": "80108",
            "telephone": "3035551212",
            "primaryContactFirstName": "Scott",
            "primaryContactLastName": "Lowe"
        }
    ]
 Continue reading

0

Faster builds in Docker Compose 1.25.1 thanks to BuildKit Support

One of the most requested features for the docker-compose tool is definitely support for building using Buildkit which is an alternative builder with great capabilities, like caching, concurrency and ability to use custom BuildKit front-ends just to mention a few… Ahhh with a nice blue output! And the good news is that Docker Compose 1.25.1 – that was just released early January – includes BuildKit support!

BuildKit support for Docker Compose is actually achieved by redirecting the docker-compose build to the Docker CLI with a limited feature set.

Enabling Buildkit build

To enable this, we have to align some stars.

First, it requires that the Docker CLI binary present in your PATH:

$ which
docker/usr/local/bin/docker

Second, docker-compose has to be run with the environment variable COMPOSE_DOCKER_CLI_BUILD set to 1 like in:

$ COMPOSE_DOCKER_CLI_BUILD=1 docker-compose build

This instruction tells docker-compose to use the Docker CLI when executing a build. You should see the same build output, but starting with the experimental warning.

As docker-compose passes its environment variables to the Docker CLI, we can also tell the CLI to use BuildKit instead of the default builder. To accomplish that, we can execute this:

$ COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build

A Continue reading

0

Docker Desktop release 2.2 is here!

We are excited to announce that we released a new Docker Desktop version today! Thanks to the user feedback on the new features initially released in the Edge channel, we are now ready to publish them into Stable. 

Before getting to each feature into detail, let’s see what’s new in Docker Desktop 2.2:

• WSL 2 as a technical preview, allowing access to the full system resources, improved boot time, access to Linux workspaces and improved file system performance
• A new file sharing implementation for Windows, improving the developer inner loop user experience
• A New Integrated Desktop Dashboard, to see at once glance your local running containers and Compose applications, and easily manage them.

WSL 2 – New architecture 

Back in July we released on Edge the technical preview of Docker Desktop for WSL 2, where we included an experimental integration of Docker running on an existing user Linux distribution. We learnt from our experience and re-architected our solution (covered in Simon’s blog) . 

This new architecture for WSL 2 allows users to: 

• Use Kubernetes on the WSL 2 backend
• Work with just WSL 2/turn off the traditional HyperV VM while working with WSL 2
• Continue Continue reading

0

Capturing Logs in Docker Desktop

Docker Desktop runs a Virtual Machine to host Docker containers. Each component within the VM (including the Docker engine itself) runs as a separate isolated container. This extra layer of isolation introduces an interesting new problem: how do we capture all the logs so we can include them in Docker Desktop diagnostic reports? If we do nothing then the logs will be written separately into each individual container which obviously isn’t very useful!

The Docker Desktop VM boots from an ISO which is built using LinuxKit from a list of Docker images together with a list of capabilities and bind mounts. For a minimal example of a LinuxKit VM definition, see https://github.com/linuxkit/linuxkit/blob/master/examples/minimal.yml — more examples and documentation are available in the LinuxKit repository. The LinuxKit VM in Docker Desktop boots in two phases: in the first phase, the init process executes a series of one-shot “on-boot” actions sequentially using runc to isolate them in containers. These actions typically format disks, enable swap, configure sysctl settings and network interfaces. The second phase contains “services” which are started concurrently and run forever as containerd tasks.

The following diagram shows a simplified high-level view of the boot process:

By default Continue reading

0

Technology Short Take 123

Welcome to Technology Short Take #123, the first of 2020! I hope that everyone had a wonderful holiday season, but now it’s time to jump back into the fray with a collection of technical articles from around the Internet. Here’s hoping that I found something useful for you!

Networking

• Eric Sloof mentions the NSX-T load balancing encyclopedia (found here), which intends to be an authoritative resource to NSX-T load balancing configuration and management.
• David Gee has an interesting set of articles exploring service function chaining in service mesh environments (part 1, part 2, part 3, and part 4).

Servers/Hardware

• Frank Denneman has a nice article aimed at helping readers understand PCIe device to NUMA Node Locality.
• Guiliano Bertello introduces PowerVCF, a PowerCLI module aimed at interacting with the SDDC Manager and VMware Cloud Foundation (VCF) stack. (If you’re not familiar with VCF, see here.)
• This is pricey, but I can’t say I don’t want one.
• James Hamilton is back with a post on the AWS Graviton2 ARM processors.

Security

• On January 13, Brian Krebs discussed the critical flaw (a vulnerability in crypt32.dll, a core Windows cryptographic component) that was rumored Continue reading

0

Removing Unnecessary Complexity

Recently, I’ve been working to remove unnecessary complexity from my work environment. I wouldn’t say that I’m going full-on minimalist (not that there’s anything wrong with that), but I was beginning to feel like maintaining this complexity was taking focus, willpower, and mental capacity away from other, more valuable, efforts. Additionally, given the challenges I know lie ahead of me this year (see here for more information), I suspect I’ll need all the focus, willpower, and mental capacity I can get!

When I say “unnecessary complexity,” by the way, I’m referring to added complexity that doesn’t bring any real or significant benefit. Sometimes there’s no getting around the complexity, but when that complexity doesn’t create any value, it’s unnecessary in my definition.

Primarily, this “reduction in complexity” shows up in three areas:

1. My computing environment
2. My home office setup
3. My lab resources

My Computing Environment

Readers who have followed me for more than a couple years know that I migrated away from macOS for about 9 months in 2017 (see here for a wrap-up of that effort), then again in 2018 when I joined Heptio (some details are available in this update). Since switching to Fedora on a Lenovo Continue reading

0

How Useful Is Ansible in a Cloud-Native Kubernetes Environment?

A question I've been hearing a lot lately is "why are you still using Ansible in your Kubernetes projects?" Followed often by "what's the point of writing your book Ansible for Kubernetes when Ansible isn't really necessary once you start using Kubernetes?"

I spent a little time thinking about these questions, and the motivation behind them, and wanted to write a blog post addressing them, because it seems a lot of people may be confused about what Kubernetes does, what Ansible does, and why both are necessary technologies in a modern business migrating to a cloud-native technology stack (or even a fully cloud-native business).

One important caveat to mention upfront, and I quote directly from my book:

While Ansible can do almost everything for you, it may not be the right tool for every aspect of your infrastructure automation. Sometimes there are other tools which may more cleanly integrate with your application developers' workflows, or have better support from app vendors.

We should always guard against the golden hammer fallacy. No single infrastructure tool—not even the best Kubernetes-as-a-service platform—can fill the needs of an entire business's IT operation. If anything, we have seen an explosion of specialist tools Continue reading

0

How to Add Approval Steps to Ansible Tower Workflows

Suppose you have a workflow set up in Red Hat Ansible Tower with several steps and needed another user to view and approve some or all of the nodes in the workflow.  Or maybe a job is running inside of a workflow but it should be viewed and approved within a specific time limit, or else get canceled automatically?  Perhaps it would be useful to be able to see how a job failed before something like a cleanup task gets set off?  It is now possible to insert a step in between any job template or workflow within that workflow in order to achieve these objectives.

 

Table of Contents

A New Feature for Better Oversight and More User Input

How to Add Approval Nodes to Workflows

What Happens When Something Needs Approval?

Approval Notifications

Timeouts

Approval-Specific Role-Based Access Controls

Summary

Where to Go Next

 

A New Feature for Better Oversight and More User Input

The Workflow Approval Node feature has been available in Ansible Tower since the release of version 3.6.0 on November 13, 2019.  In order to visually compare the additional functionality, examine the before and after examples of a workflow Continue reading

0

5 Software Development Predictions for 2020

• 

To kick off the new year, we sat down with Docker CEO Scott Johnston and asked him what the future holds for software development. Here are his 2020 predictions and trends to keep an eye on.

Existing Code and Apps Become New Again

Developers will find new ways to reuse existing code instead of reinventing the wheel to start from scratch. Additionally, we’ll see companies extend the value to existing apps by adding more functionality via microservices.

The Changing Definition of a Modern Application

Today’s applications are more complex than those of yesterday. In 2020, modern apps will power tomorrow’s innovation and this requires a diverse set of tools, languages and frameworks for developers. Developers need even more flexibility to address this new wave of modern apps and evolve with the rest of the industry.

Containers Pave the Way to New Application Trends

Now that containers are typically considered a common deployment mechanism, the conversation will evolve from the packaging of individual containers to the packaging of the entire application (which are becoming increasingly diverse and distributed). Organizations will increasingly look for guidance and solutions that help them unify how they build and manage Continue reading

0

Looking Back: 2019 Project Report Card

As has been my custom over the last five years or so, in the early part of the year I like to share with my readers a list of personal projects for the upcoming year (here’s the 2019 list). Then, near the end of that same year or very early in the following year, I evaluate how I performed against that list of personal projects (for example, here’s my project report card for 2018). In this post, I’ll continue that pattern with an evaluation of my progress against my 2019 project list.

For reference, here’s the list of projects I set out for myself for 2019 (you can read the associated blog post, if you like, for additional context):

1. Make at least one code contribution to an open source project. (Stretch goal: Make three code contributions to open source projects.)
2. Add at least three new technology areas to my “learning-tools” repository. (Stretch goal: Add five new technology areas to the “learning-tools” repository.)
3. Become more familiar with CI/CD solutions and patterns.
4. Create at least three non-written content pieces. (Stretch goal: Create five pieces of non-written content.)
5. Complete a “wildcard project” (if applicable).

Here’s how I Continue reading

0

2019 Docker Community Awards

0

New Year, New Adventure

I’ll skip the build-up and jump straight to the whole point of this post: a once-in-a-lifetime opportunity has come up and I’m embarking on a new adventure starting in early 2020. No, I’m not changing jobs…but I am changing time zones.

Sometime in the next month or two (dates are still being finalized), I’ll be temporarily relocating to Tokyo, Japan, to help build out VMware’s Cloud Native Field Engineering team to provide consulting and professional services around cloud-native technologies and modern application platforms for customers in Japan. Basically, my charter is to replicate the former Heptio Field Engineering team (now the Cloud Native Field Engineering Practice within VMware) in Japan.

Accomplishing this feat will involve a variety of responsibilities: a pretty fair amount of training/enablement, engaging customers on the pre-sales side, helping lead projects on the post-sales (delivery) side, mentoring team members, performing some project management, probably some people management, and the infamous “other duties as required.” All in about six months (the inital duration of my assignment), and all while learning Japanese! No big deal, right?

I’m both simultaneously excited and scared. I’m excited by the idea of living in Tokyo, but let’s be honest—the language barrier is Continue reading

0

Technology Short Take 122

Welcome to Technology Short Take #122! Luckily I did manage to get another Tech Short Take squeezed in for 2019, just so all my readers could have some reading materials for the holidays. I’m kidding! No, I mean I really am kidding—don’t read stuff over the holidays. Spend time with your family instead. The investment in your family will pay off in later years, trust me.

Networking

• Teammate Alex Brand has a look at VMware’s new OVS-based CNI plugin, Antrea.

Servers/Hardware

• Adam Leventhal takes a closer look at the financial side of AWS Outposts, which was announced as generally available at this year’s re:Invent.

Security

• Bruce Schneier writes about how some Chinese hackers are bypassing RSA software token authentication (the title is a bit more broad, implying other forms of two-factor authentication are affected, but the article focuses on attacks against the use of RSA software tokens).
• Robin Moffatt discusses detecting and analysing SSH attacks with ksqlDB.

Cloud Computing/Cloud Management

• Ahmet Alp Balkan talks about how the Knative Serving component addresses some key networking challenges within Kubernetes.
• Teri Radichel has a post with some structural tips to better understand and utilize AWS CloudFormation.
• Geert Baeke walks readers through deploying Continue reading

0

Year in Review: The Most Loved Docker Articles, Blogs and Tweets of 2019

As this decade comes to a close, we are rounding up some of your favorite content from 2019. Catch up on anything you missed and get ready for a lot more to come in 2020!

Docker Captain Content

Brian Christner did an analysis of VMware, Docker, and Kubernetes Google Trends and the results just might surprise you. . . or maybe not.

John Lees Miller updated his 2016 Lessons from Building a Node App in Docker. Run through the updated tutorial to learn how to Dockerize your node.js apps by setting up the socket.io chat example with Docker, from scratch to production-ready. 

Ajeet Singh Raina wrote nearly 30 blogs in 2019, and the most popular was 5 Minutes to Kubernetes Dashboard running on Docker Desktop for Windows 2.0.0.3. Find yourself five minutes before the end of the year to try this out yourself.

Łukasz Lach and Thomas Shaw spread holiday cheer with some seasonal docker run commands:

$ docker run -it lukaszlach/merry-christmas 

docker run --rm -t tomwillfixit/hohoho

Bret Fisher hosts a weekly Docker and DevOps YouTube live show – a fun and educational way to spend an hour on Thursdays. Continue reading

0

Containers Today Recap: The Future of the Developer Journey

There was amazing attendance at Containers Today in Stockholm a couple of weeks ago. For those who were unable to make it, here is a quick overview of what I talked about at the event in my session around the future of the developer journey. 

Before we talk about what we think will change the journey, we need to think about why it changes. The fundamental goal of any change to the way of working for developers should be to reduce the number of boring, mundane and repetitive tasks that developers have to do or to allow them to reach new customers/solve new problems. Developers create amazing value for companies and provide solutions to customers’ real world problems. But if they are having to spend half of their time working out how to get things into the hands of their customers, then you are getting half the value.

Developer Evolution

The role of developers has changed a lot over the last ~40 years. Developers no longer deploy to mainframes or in house hardware, they don’t do waterfall deployments and not many of them write in machine code. Developers have to now think about web languages and ML, work in Continue reading

0

Rebooting Network Devices with Ansible

With the Red Hat Ansible Automation Platform release in November, we released over 50 network resource modules to help make automating network devices easier and more turn-key for network engineers.  In addition to the new resource modules, Andrius also discussed fact gathering enhancements in his blog post, which means with every new resource module, users gain increased fact coverage for network devices.  For this blog post I want to cover another cool enhancement that may have gone unnoticed. This is the ability for network devices to make use of the wait_for_connection module.  If you are a network engineer that has operational Ansible Playbooks that need to reboot devices or take them offline, this module will help you make more programmatic playbooks to handle disconnects.  By leveraging wait_for_connection network automation playbooks can look and behave more like playbooks for Linux or Windows hosts.

 

Table of Contents

Comparing wait_for and wait_for_connection

Dealing with prompts

Using reset_connection in combination

Where to go next?

 

Comparing wait_for and wait_for_connection 

There are two great modules that can wait for a condition to be met, wait_for and the wait_for_connection.  I highly recommend against using the pause module if you Continue reading

0

Ansible Content Collections: The Sensu Go Use Case

In Getting Started With Ansible Content Collections, which presented the general idea behind what is becoming a new standard in the distribution of Ansible content, we learned about the what, the why and the how of Ansible Collections (and hopefully it got you excited about Ansible Collections!). In this post, we'll take things a bit further, continuing the journey into the world of Ansible Collections accompanied by the certified Sensu Go Ansible Collection that our team at XLAB Steampunk developed and supports for Sensu.

 

What to expect?

This article will guide you through the process of creating a fully functioning automated deployment of the Sensu Go monitoring agent and backend with the help of roles and modules included in the Sensu Go Ansible Collection. 

If you are not familiar with Sensu Go, this quick introduction to Sensu Go will help you get up to speed.

Before we begin, let's first talk about the collection we're taking along for the ride.

 

The Sensu Go Ansible Collection: the basics

What exactly do we need for a complete and fully functioning deployment of Sensu Go? First, the Sensu Go monitoring backend. Then, to allow the backend to Continue reading

0

Deep Dive Into the New Docker Desktop filesharing Implementation Using FUSE

The latest Edge release of Docker Desktop for Windows 2.1.7.0 has a completely new filesharing system using FUSE instead of Samba. The initial blog post we released presents the performance improvements of this new implementation and explains how to give feedback. Please try it out and let us know what you think. Now, we are going to go into details to give you more insight about the new architecture.

New Architecture

Instead of Samba running over a Hyper-V virtual network, the new system uses a Filesystem in Userspace (FUSE) server running over gRPC over Hypervisor sockets.

The following diagram shows the path taken by a single request from a container, for example to read a PHP file:

In step (1) the web-server in the container calls “read” which is a Linux system call handled by the kernel’s Virtual File System (VFS) layer. The VFS is modular and supports many different filesystem implementations. In our case we use Filesystem in Userspace (FUSE) which sends the request to a helper process running inside the VM labelled “FUSE client.” This process runs within the same namespace as the Docker engine. The FUSE client can handle some requests locally, but Continue reading