Advanced AMI Filtering with JMESPath

I recently had a need to do some “advanced” filtering of AMIs returned by the AWS CLI. I’d already mastered the use of the --filters parameter, which let me greatly reduce the number of AMIs returned by aws ec2 describe-images. In many cases, using filters alone got me what I needed. In one case, however, I needed to be even more selective in returning results, and this lead me to some (slightly more) complex JMESPath queries than I’d used before. I wanted to share them here for the benefit of my readers.

What I’d been using before was a command that looked something like this:

ec2 describe-images --owners 099720109477 \
--filters Name=name,Values="*ubuntu-xenial-16.04*" \
Name=virtualization-type,Values=hvm \
Name=root-device-type,Values=ebs \
Name=architecture,Values=x86_64 \
--query 'sort_by(Images,&CreationDate)[-1].ImageId'

The part after --query is a JMESPath query that sorts the results, returning only the ImageId attribute of the most recent result (sorted by creation date). In this particular case, this works just fine—it returns the most recent Ubuntu Xenial 16.04 LTS AMI.

Turning to Ubuntu Bionic 18.04, though, I found that the same query didn’t return the result I needed. In addition to the regular builds of 18.04, Canonical apparently also builds EKS Continue reading

Technology Short Take 111

Welcome to Technology Short Take #111! I’m a couple weeks late on this one; wanted to publish it earlier but work has been keeping me busy (lots and lots of interest in Kubernetes and cloud-native technologies out there!). In any event, here you are—I hope you find something useful for you!

Networking

• Daniel Dib has a great article on how network engineers need to evolve. The network isn’t going away, it’s just changing.
• I referenced part 1 of Ajay Chenampara’s series on the Ansible network-engine command parser back in Technology Short Take 102 (July of last year). I’m not sure how I missed that part 2 was published only 2 days later, so I’m rectifying that now. Go check out part 2.
• Paul Fitzgerald shows how to use Hyper-V and Docker to build a VyOS 1.2.0 ISO image. (VyOS is an open source Linux-based network operating system.)
• Matt Cowger shares how to use MetalLB with the Unifi USG to take advantage of Kubernetes LoadBalancer functionality in your home lab. Nifty.

Servers/Hardware

• James Hamilton is back with a more in-depth look at the components of the AWS Nitro System.
• I must say that my conclusions regarding Continue reading

containerd Graduates Within the CNCF

We are happy to announce that as of today, containerd, an industry-standard runtime for building container solutions, graduates within the CNCF. The successful graduation demonstrates containerd has achieved the maturity, stability and community acceptance required for broad ecosystem adoption. containerd has already been deployed in tens of millions of production systems today, making it the most widely adopted runtime and an essential upstream component of the Docker platform. containerd was donated to the CNCF as a top-level project because of its strong alignment with Kubernetes, gRPC and Prometheus and is the fifth project to make it to this tier. Built to address the needs of modern container platforms like Docker Enterprise and orchestration systems like Kubernetes, containerd ensures users have a consistent dev to ops experience.

From Docker’s initial announcement that it was spinning out its core runtime to its donation to the CNCF in March 2017, the containerd project has experienced significant growth and progress over the last two years. The primary goal of Docker’s donation was to foster further innovation in the container ecosystem by providing a core container runtime that could be leveraged by container system vendors and orchestration projects such as Kubernetes, Swarm, Continue reading

Thoughts on VPNs for Road Warriors

A few days ago I was talking with a few folks on Twitter and the topic of using VPNs while traveling came up. For those that travel regularly, using a VPN to bypass traffic restrictions is not uncommon. Prompted by my former manager Martin Casado, I thought I might share a few thoughts on VPN options for road warriors. This is by no means a comprehensive list, but hopefully something I share here will be helpful.

There were a few things I wanted to share with readers:

• I found commercial VPN services too unreliable (it’s not uncommon for commercial VPN services to get blocked and thus defeat the purpose of using a VPN).
• Instead, I’ve found more success using something like AutoVPN. AutoVPN helps you stand up an on-demand OpenVPN endpoint on AWS. I used this successfully in Beijing, setting up endpoints in Seoul, Singapore, Tokyo, and sometimes Sydney. Because these IP addresses are “ephemeral,” they’re far less likely to be blocked. (Here’s another example of using AWS as a personal VPN service.)
• I also had success using AutoVPN not to get around traffic restrictions, but to change my source IP. My wife needed to access some real Continue reading

Docker’s 6th Birthday: How do you #Docker?

Docker is turning 6 years old! Over the years, Docker Community members have found some amazing and innovative ways of using Docker technology and we’ve been blown away by all the use-cases we’ve seen from the community at DockerCon. From Docker for Space where NASA used Docker technology to build software to deflect asteroids to using “gloo” to glue together traditional apps, microservices and serverless, you all continue to amaze us year over year.

So this year, we want to celebrate you! From March 18th to the 31st, Docker User Groups all over the world will be hosting local birthday show-and-tell celebrations. Participants will each have 10-15 minutes of stage time to present how they’ve been using Docker. Think of these as lightning talks – your show-and-tell doesn’t need to be polished and it can absolutely be a fun hack and/or personal project. Everyone who presents their work will get a Docker Birthday #6 t-shirt and have the opportunity to submit their Docker Birthday Show-and-tell to present at DockerCon.

Are you new to Docker? Not sure you’d like to present? No worries! Join in the fun and come along to listen, learn, add to your sticker collection and Continue reading

Docker Desktop Enterprise Preview: Version Packs

This is the first in a series of articles we are publishing to provide more details on Docker Desktop Enterprise, which we announced at DockerCon Barcelona. Keep up with the latest Docker Desktop Enterprise news and release updates by signing up for the Docker Desktop Enterprise announcement list.

Docker’s engineers have been hard at work completing features and getting everything in ship-shape (pun intended) following our announcement of Docker Desktop Enterprise, a new desktop product that is the easiest, fastest and most secure way to develop production-ready containerized applications and the easiest way for developers to get Kubernetes running on their own machine.

In the first post of this series I want to highlight how we are working to bridge the gap between development and production with Docker Desktop Enterprise using our new Version Packs feature. Version Packs let you easily swap your Docker Engine and Kubernetes orchestrator versions to match the versions running in production on your Docker Enterprise clusters. For example, imagine you have a production environment running Docker Enterprise 2.0. As a developer, in order to make sure you don’t use any APIs or incompatible features that will break when you push an application to production Continue reading

Kubernetes, Kubeadm, and the AWS Cloud Provider

Over the last few weeks, I’ve noticed quite a few questions appearing in the Kubernetes Slack channels about how to use kubeadm to configure Kubernetes with the AWS cloud provider. You may recall that I wrote a post about setting up Kubernetes with the AWS cloud provider last September, and that post included a few snippets of YAML for kubeadm config files. Since I wrote that post, the kubeadm API has gone from v1alpha2 (Kubernetes 1.11) to v1alpha3 (Kubernetes 1.12) and now v1beta1 (Kubernetes 1.13). The changes in the kubeadm API result in changes in the configuration files, and so I wanted to write this post to explain how to use kubeadm 1.13 to set up a Kubernetes cluster with the AWS cloud provider.

I’d recommend reading the previous post from last September first. In that post, I listed four key configuration items that are necessary to make the AWS cloud provider work:

1. Correct hostname (must match the EC2 Private DNS entry for the instance)
2. Proper IAM role and policy for Kubernetes control plane nodes and worker nodes
3. Kubernetes-specific tags on resources needed by the cluster
4. Correct command-line flags added to the Kubernetes API server, controller Continue reading

Ansible Community Update — February 2019

Ansible is a popular project by many metrics, including over 42,000 commits on GitHub. Our community contributes a lot of pull requests (PRs) every month. Unfortunately, the volume of incoming PRs means contributors often have to wait days, weeks, or months for PRs to be merged. Sometimes it takes that long for a cursory review. We want to change that, but we need your help!

The Core team and community at large are kicking off new initiatives under the contributor experience umbrella. The idea is to help address causes that slow down quality PRs from being merged into Ansible's codebase.

To help with this, we are dedicating one day a month to doing a community review. The goals we are setting for these meetings are:

• Give potential new community members a place to learn and experiment with Ansible's review process and exchange feedback

• Identify process and documentation improvements via feedback provided from the Ansible community

• Give PRs needed attention; remove blockers where necessary

• Identify PRs that could be merged or closed

We’re particularly interested in feedback from people starting their journey with open source as it helps us to improve our processes and documentation. It’s helpful to have new contributors Continue reading

We’ve Got ❤️ For Our First Batch of DockerCon Speakers

As the world celebrates Valentine’s Day, at Docker, we are celebrating what makes our heart all aflutter – gearing up for an amazing DockerCon with the individuals and organizations that make up the Docker community. With that, we are thrilled to announce our first speakers for DockerCon San Francisco, April 29 – May 2.

DockerCon fan favorites like Liz Rice, Bret Fisher and Don Bauer are returning to the conference to share new insights and experiences to help you better learn how to containerize.

And we are excited to welcome new speakers to the DockerCon family including Ana Medina, Tommy Hamilton and Ian Coldwater to talk chaos engineering, building your production container platform stack and orchestration with Docker Swarm and Kubernetes. 

And we’re just getting started! This year DockerCon is going to bring more technical deep dives, practical how-to’s, customer case studies and inspirational stories. Stay tuned as we announce the full speaker line up this month.

Register Now

This #ValentinesDay #Docker announces its first speakers for #DockerCon San Francisco on April 29 to May 2
Click To Tweet

The post We’ve Got ❤️ For Our First Batch of DockerCon Speakers appeared first on Docker Blog.

Docker Security Update: CVE-2018-5736 and Container Security Best Practices

On Monday, February 11, Docker released an update to fix a privilege escalation vulnerability (CVE-2019-5736) in runC, the Open Container Initiative (OCI) runtime specification used in Docker Engine and containerd. This vulnerability makes it possible for a malicious actor that has created a specially-crafted container image to gain administrative privileges on the host. Docker engineering worked with runC maintainers on the OCI to issue a patch for this vulnerability.

Docker recommends immediately applying the update to avoid any potential security threats. For Docker Engine-Community, this means updating to 18.09.2 or 18.06.2. For Docker Engine- Enterprise, this means updating to 18.09.2, 18.03.1-ee-6, or 17.06.2-ee-19. Read the release notes before applying the update due to specific instructions for Ubuntu and RHEL operating systems.

Summary of the Docker Engine versions that address the vulnerability:

To better protect the container images run by Docker Engine, here are some additional recommendations and best practices:

Use Docker Official Images

Official Images are a curated set of Docker repositories hosted on Docker Hub that are designed to:

• Continue reading

How to run stunnel on your Android device

Overview

In this post we’re going to talk about how to run the amazing stunnel program on your Android device, and do so properly!

Later, this would allow us to setup a lot of cool things like:

• Wrapping OpenVPN traffic with it
• Using it as a SOCKS VPN
• Adding proper IMAPS/SMTPS support to our old email apps
• …

For this, we’re not going to use the old and very limited SSLDroid. It’s a bad idea, I don’t know why different sites still keep pushing it. It almost certainly has unpatched vulnerabilities. Please don’t use it.

Instead, we are going to use the official stunnel program, with the help of a proper wrapper.

stunnel Android binary

stunnel already supports Android devices and even the compiled version of it is available in it’s download page.

Since we’ll be using the compiled binary, you may need to compile stunnel yourself for your specific Android architecture before continuing¹. Chances are though, that your device is running on ARM and you are ready Continue reading

Ansible Operator: What is it? Why it Matters? What can you do with it?

The Red Hat Ansible Automation and Red Hat OpenShift teams have been collaborating to build a new way to package, deploy, and maintain Kubernetes native applications: Ansible Operator. Given the interest in moving workloads to Kubernetes, we are happy to introduce a new tool that can help ease the move toward cloud native infrastructure.

What is Kubernetes? The simplest definition of Kubernetes I’ve ever used is, “Kubernetes is a container orchestrator.” But that is a simplified definition.

What is OpenShift? Red Hat OpenShift Container Platform is an enterprise-grade Kubernetes distribution. It enables management of container applications across hybrid cloud and multicloud infrastructure.

First, let’s identify the problem operators can help us solve. Operators help simplify deployment, management, and operations of stateful applications in Kubernetes. But, writing an operator today can be difficult because of the knowledge of Kubernetes components required to do so. The Operator SDK is a framework that uses the controller-runtime library to help make writing operators more simple. The SDK enables Operator development in Go, Helm, or Ansible.

Why It Matters

What can an Ansible Operator give us that a generic operator doesn’t? The same things Ansible can give its users: a lower barrier to entry, faster iterations, Continue reading

Deep Dive on cli_command for Network Automation

In October Ansible 2.7 was released and brought us two powerful agnostic network modules, cli_command and cli_config. Do you have two or more network vendors within your environment? The goal of agnostic modules is to simplify Ansible Playbooks for network engineers that deal with a variety of network platforms. Rather than having to deal with platform specific modules (e.g. eos_config, ios_config, junos_config), you can now use cli_command or cli_config to reduce the amount of tasks and conditionals within a playbook, and make the playbook easier to use. This post will demonstrate how to use these modules and contrast them to platform specific modules. I’ll show some playbook examples and common use cases to help illustrate how you can use these new platform agnostic modules.

Both the cli_command and cli_config only work with the network_cli connection plugin. For those unfamiliar with the network_cli connection plugin check out this blog post I did last April. The goal of network_cli is to make playbooks look, feel and operate on network devices, the same way Ansible works on Linux hosts.

What can you do with the cli_command?

The cli_command allows you to run arbitrary commands on network devices. Let’s show a simple Continue reading

Scraping Envoy Metrics Using the Prometheus Operator

On a recent customer project, I recommended the use of Heptio Contour for ingress on their Kubernetes cluster. For this particular customer, Contour’s support of the IngressRoute CRD and the ability to delegate paths via IngressRoutes made a lot of sense. Of course, the customer wanted to be able to scrape metrics using Prometheus, which meant I not only needed to scrape metrics from Contour but also from Envoy (which provides the data plane for Contour). In this post, I’ll show you how to scrape metrics from Envoy using the Prometheus Operator.

First, I’ll assume that you’ve already installed and configured Prometheus using the Prometheus Operator, a task which is already fairly well-documented and well-understood. If this is something you think would be helpful for me to write a blog post on, please contact me on Twitter and let me know.

The overall process looks something like this:

1. Modify the Envoy DaemonSet (or Deployment, depending on your preference) to add a sidecar container and expose additional ports.
2. Modify the Service for the Envoy DaemonSet/Deployment to expose the ports you added in step 1.
3. Add a ServiceMonitor object (a CRD added by the Prometheus Operator) to tell Prometheus to scrape Continue reading

Technology Short Take 110

Welcome to Technology Short Take #110! Here’s a look at a few of the articles and posts that have caught my attention over the last few weeks. I hope something I’ve included here is useful for you also!

Networking

• Via Kirk Byers (who is himself a fantastic resource), I read a couple of articles on network automation that I think readers may find helpful. First up is a treatise from Mircea Ulinic on whether network automation is needed. Next is an older article from Patrick Ogenstad that provides an introduction to ZTP (Zero Touch Provisioning).
• The folks over at Cilium took a look at a recent CNI benchmark comparison and unpacked it a bit. There’s some good information in their article.
• I first ran into Forward Networks a few years ago at Fall ONUG in New York. At the time, I was recommending that they explore integration with NSX. Fast-forward to this year, and the company announces support for NSX and (more recently) support for Cisco ACI. The recent announcement of their GraphQL-based Network Query Engine (NQE)—more information is available in this blog post—is also pretty interesting to me.

Servers/Hardware

• Patrick Kennedy over at Serve The Home shares the Continue reading

How I lost my data trying to back it up

My data, my precious data… is gone.

Overview

This is a story about how I lost my data when trying to prevent it by backing it up.

Even though there were numerous other factors in play, I take full responsibility for what happened.

I have written this article hoping that it could save someone else from the same situation. No one should ever experience the loss of his/her data.

7 days and 10 hours ago:

Alrighty, It’s time for the offline backup routine again.

When it comes to backing up the full OS, I don’t believe in online backup solutions. This is specially true when I’m dealing with Windows servers. These kinda backups should serve as a disaster recovery solution and I’m not taking any chances.

So I always take them offline, outside of the OS, using a third party program to boot up the servers.

For years I have been using Paragon Software. Their products are great, reliable and I have nothing bad to say about them.

This time however, I thought maybe it’s time to ditch commercial softwares and use the solid ntfs-3g suite instead. Back when I started using Paragon, I didn’t even know what Linux was, Continue reading

Announcing Support for Windows Server 2019 within Docker Enterprise

Docker is pleased to announce support within the Docker Enterprise container platform for the Windows Server 2019 Long Term Servicing Channel (LTSC) release and the Server 1809 Semi-Annual Channel (SAC) release. Windows Server 2019 brings the range of improvements that debuted in the Windows Server 1709 and 1803 SAC releases into a LTSC release preferred by most customers for production use. The addition of Windows Server 1809 brings support for the latest release for customers who prefer to work with the Semi-Annual Channel. As with all supported Windows Server versions, Docker Enterprise enables Windows Server 2019 and Server 1809 to be used in a mixed cluster alongside Linux nodes.

Windows Server 2019 includes the following improvements:

• Ingress routing
• VIP service discovery
• Named pipe mounting
• Relaxed image compatibility requirements
• Smaller base image sizes

Docker and Microsoft: A Rich History of Advancing Containers

Docker and Microsoft have been working together since 2014 to bring containers to Windows Server applications, along with the benefits of isolation, portability and security. Docker and Microsoft first brought container technology to Windows Server 2016 which ships with a Docker Enterprise Engine, ensuring consistency for the same Docker Compose file and CLI commands across both Linux and Continue reading

5 Reasons to Attend DockerCon SF 2019

If you can only attend one conference this year – make it matter. DockerCon is the one-stop event for practitioners, contributors, maintainers, developers, and the container ecosystem to learn, network and innovate. And this year, we will continue to bring you all the things you love about DockerCon like Docker Pals, the Hallway Track and roundtables, and the sessions and content you wanted more of – including open source, transformational, and practical how-to talks. Take advantage of our lowest ticket price when you register by January 31, 2019. No codes required.

<Register Now>

And in case you are still not convinced, here are a few more reasons you shouldn’t miss this year’s DockerCon

1. Belong. The Docker Community is one of a kind and the best way to feel a part of it is at DockerCon. Take advantage the Docker Pals Program, Hallway Track, roundables and social events to meet new people and make lasting connections.

2.  Think big. Docker containers and our container platform are being used everywhere for everything – from sending rockets to space to literally saving the earth from asteroids to keeping e-commerce running smoothly for black friday shoppers. Come to DockerCon and Continue reading

Webinar: “How to Make Your Mark: Ansible Community Contributions”

On Wednesday, January 30, 2019, 9:30 AM EST, we will be hosting a webinar, How to Make Your Mark: Ansible Community Contributions. This webinar is tailored for everyone in the Ansible community. Whether you're a brand new member of the Ansible community or a full-fledged Red Hat Ansible Tower customer, contributing to the Ansible projects is a way to put features and fixes into the tools you use daily.

Join John "gundalow" Barker (Principal Software Engineer, Ansible Community) and Alicia Cozine (Technical Writer, Ansible Documentation) as they discuss and demonstrate contribution how-to’s and best practices. They will also quell some common myths about contributing to Ansible and dive into the Ansible development workflow.

There are no prerequisites for attending this webinar. You don’t need to know git, understand GitHub, nor write Python. There will also be a Q&A session during the webinar. If you have questions about how or where to get started after the webinar, please ask them! One of the best parts about being a community is continually improving the way we work. If you have suggestions for documentation, process, etc. but don’t know the best place to ask, this webinar will help.

Ansible Contributor Experience Working Group Continue reading

Playing The Long Game In Systems

The secret to the longevity of any big corporation is a nearly constant process of reinvention. …

Playing The Long Game In Systems was written by Timothy Prickett Morgan at .