Archive

Category Archives for "Systems"

Developing and Testing Ansible Roles with Molecule and Podman – Part 1

One of the beauties of the Red Hat Ansible Automation Platform is that the language to describe automation is readable not only by a few dedicated experts, but by almost anyone across the IT ecosystem. That means all IT professionals can take part in the automation, enabling cross team collaboration and really drive automation as a culture inside an organization. With so many people contributing to the automation, it is crucial to test the automation content in-depth. So when you’re developing new Ansible Content like playbooks, roles and collections, it’s a good idea to test the content in a test environment before using it to automate production infrastructure. Testing ensures the automation works as designed and avoids unpleasant surprises down the road. 

Testing automation content is often a challenge, since it requires the deployment of specific testing infrastructure as well as setting up the testing conditions to ensure the tests are relevant. Molecule is a complete testing framework that helps you develop and test Ansible roles, which allows you to focus on the content instead of focusing on managing testing infrastructure.

According to its official documentation, Molecule is a project:

 “designed to aid in the development and testing Continue reading

Scaling Docker to Serve Millions More Developers: Network Egress

In Part 1 of this blog we went into a deep dive that analyzed all of the images stored in Docker Hub, the world’s largest container registry. We did this to give you a better understanding of how our new Terms of Service updates will impact development teams who use Docker Hub to manage their container images and CI/CD pipelines.

Part 2 of this blog post takes a deep dive into rate limits for container image pulls. This was also announced as part of our updated Docker Terms of Service (ToS) communications. We detailed the following pull rate limits to Docker subscription plans that will take effect November 1, 2020:

  • Free plan – anonymous users: 100 pulls per 6 hours 
  • Free plan – authenticated users: 200 pulls per 6 hours
  • Pro plan – unlimited
  • Team plan – unlimited

Docker defines pull rate limits as the number of manifest requests to Docker Hub. Rate limits for Docker image pulls are based on the account type of the user requesting the image – not the account type of the image’s owner. For anonymous (unauthenticated) users, pull rates are limited based on the individual IP address. 

We’ve been getting questions from customers Continue reading

Scaling Docker’s Business to Serve Millions More Developers: Storage

At Docker, our mission is to enable developers worldwide to quickly bring their ideas to life by reducing the complexity of application development. While over 6.5 million registered Docker developers are enjoying the benefits of Docker today, we want to scale Docker’s business to the tens of millions of developers who are just discovering Docker. Offering free tools and services is a cornerstone of our mission, and these are funded by our paid subscription services.

In this blog series, we will deep dive into why and how the recently announced Terms of Service changes were introduced. This blog, Part 1, will explore the inactive image retention policy and how it will impact development teams who use Docker Hub for managing their container images. Part 2 will focus on the new rate limits policies that were introduced for image pulls.

A deeper look at Docker Hub images

Delivering containerized applications in a portable, secure, and resource efficient manner also requires tools and services for securely storing and sharing applications with your development team. Today, Docker is proud to offer the world’s largest container registry, Docker Hub, which is used by over 6.5 million developers around the world. Over 15 Continue reading

Red Hat Ansible Tower Performance Improvements between 3.6 and 3.7

As one of our customers pointed out, "job events are not showing in Tower UI", causing significant performance issues for users trying to view job status updates. To make Red Hat Ansible Tower more approachable in viewing Real-Time job status updates, we’ve applied the following performance improvements. 

 

Performance Improvements

Between the 3.6 and 3.7 releases, there have been significant performance advancements to improve event processing, job running performance and the user interface. This work was done in conjunction with our customers and the Red Hat Scale and Performance team. These include:

  • Added notable performance improvements to event processing to drastically speed up stdout ingestion speed.
  • Updated Ansible Tower to no longer rely on RabbitMQ for clustering and event distribution. Redis is added as a new dependency for event handling.
  • Improved performance in the User Interface for various job views when many simultaneous users are logged into Ansible Tower.
  • Improved job run performance and the write speed of stdout for running playbooks and parallel jobs through optimization of the job dependency/scheduling algorithm.
  • Fixed event processing for inventories with very large numbers of hosts to prevent Ansible Tower slow down.
  • Improved running jobs to no longer block associated Continue reading

Technology Short Take 130

Welcome to Technology Short Take #130! I’ve had this blog post sitting in my Drafts folder waiting to be published for almost a month, and I kept forgetting to actually make it live. Sorry! So, here it is—better late than never, right?

Networking

Security

Cloud Computing/Cloud Management

Docker Desktop & WSL 2 – Backport Update

While we have continued to make improvements to our Windows experience on Docker Desktop for users of HyperV, we are excited to see that Microsoft has announced the backport of WSL 2 to Windows version 1903 and 1909. This means that as of today, Docker Desktop Edge users will be able to use Docker Desktop with WSL 2 rather than our legacy HyperV based backend. This is available not only for Windows Pro and Windows Enterprise, but also for Windows Home users. This is the first time that Docker has been available on Windows Home versions 1903 and 1909! ?

This means that these developers will be able to take advantage of WSL 2 and Docker’s integration, allowing developers to store their code within their WSL 2 distro and run the Docker CLI from within this distro. This removes the need to access files stored on the Windows host and provides significant performance improvements for users.

To find out more about using Docker Desktop with WSL 2, check out Simon’s full tips and tricks article. If you want to learn more about how Docker developed the WSL 2 backend you can have a look through our history of the integration Continue reading

Deploying a Minecraft Docker Server to the cloud

One of the simplest examples that people have used over the years of demoing Docker is quickly standing up and running a Minecraft server. This shows the power of using Docker and has a pretty practical application!

Recently I wanted to set up a server but I wanted to persist one and as I have given away my last raspberry pi I needed to find a new way to do this. I decided that I would have a go at running this in Azure using the $200 free credits you get in your first month.

The first thing I decided to do was to check out the existing Docker Images for Minecraft servers to see if there were any that looked good to use, to do this I went to Docker Hub and searched for minecraft:

I liked the look of minecraft-server repo, so I clicked through to have a look at the image and link through to the Github repo.

To start I decide to just test out running this locally on my machine with the ‘simple get started’ Docker Run command:

$ docker run -d -p 25565:25565 --name mc -e EULA=TRUE
 itzg/minecraft-server

In the Docker Desktop Dashboard, I Continue reading

How To Use the Official NGINX Docker Image

NGINX is one of the most popular web servers in the world. Not only is NGINX a fast and reliable static web server, it is also used by a ton of developers as a reverse-proxy that sits in front of their APIs. 

In this tutorial we will take a look at the NGINX Official Docker Image and how to use it. We’ll start by running a static web server locally then we’ll build a custom image to house our web server and the files it needs to serve. We’ll finish up by taking a look at creating a reverse-proxy server for a simple REST API and then how to share this image with your team.

Prerequisites

To complete this tutorial, you will need the following:

NGINX Official Image

The Docker Official Images are a curated set of Docker repositories hosted on Docker Hub that have been scanned for vulnerabilities and are maintained by Docker employees and upstream maintainers.

Official Continue reading

Automating Mitigation of the Microsoft (CVE-2020-1350) Security Vulnerability in Windows Domain Name System Using Ansible Tower

On July 14, 2020, a Critical Remote Code Execution (RCE) vulnerability in Windows DNS Server was released that is classified as a ‘wormable’ vulnerability, and has a CVSS base score of 10.0. This issue results from a flaw in Microsoft’s DNS server role implementation and affects all Windows Server versions. Non-Microsoft DNS Servers are not affected.

Updates to this vulnerability are available. However, in some use cases, applying the update quickly might not be practical: in many enterprises, even hotfixes need to run through a series of tests that require time. For such cases, a registry-based workaround is available that also requires restarting the DNS service.  However, doing so manually is time consuming and prone to error, especially if many servers are involved. For customers with the Red Hat Ansible Automation Platform, a playbook has been written to automate the workaround.

 

Background of the vulnerability

The vulnerability is described in CVE-2020-1350

Wormable vulnerabilities have the potential to spread via malware between vulnerable computers without user interaction. Windows DNS Server is a core networking component. While this vulnerability is not currently known to be used in active attacks, it is essential that customers apply Windows updates to address Continue reading

Docker Hub Incident Review – 5 July 2020

Background

This is Docker’s first time publishing an incident report publicly. While we have always done detailed post mortems on incidents internally, as part of the changing culture at Docker, we want to be more open externally as well. For example, this year we have started publishing our roadmap publicly and asking our users for their input. You should expect to see us continue publishing reports for most significant incidents.

In publishing these reports, we hope others can learn from the issues we have faced and how we have dealt with them. We hope it builds trust in our services and our teams. We also think this one is pretty interesting due to the complex interaction between multiple services and stakeholders.

Incident Summary

Amazon Linux users in several regions encountered intermittent hanging downloads of Docker images from the Docker Hub registry between roughly July 5 19:00 UTC and July 6 06:30 UTC. The issue stemmed from an anti-botnet protection mechanism our CDN provider Cloudflare had deployed. Teams from Docker, Cloudflare, and AWS worked together to pinpoint the issue and the mechanism in question was disabled, leading to full service restoration.

What Happened

At about 01:45 UTC on Monday July 6th Continue reading

AnsibleFest 2020 – The Biggest AnsibleFest EVER

It is almost that time of year again for everyone’s favorite automation event! 2020 has given us our fair share of change (and then some). But we’re not just facing new challenges. We’re adapting to them and innovating to overcome them together. We’re distributed yet we’re connected -- connected to new technologies, to new ways of working, and most importantly, to each other.

This year’s AnsibleFest is now a virtual experience, and we are using this opportunity to engage and collaborate with Ansible users across the globe. It will be a free virtual experience where our communities can connect to a wider audience to collaborate and solve problems. The venue may be different this year, but it is still the same AnsibleFest you know and love.

 

Keynotes

This year we have a great lineup of keynote speakers. We have brought together a group of people rich with Ansible knowledge, tapped to share meaningful insights with you right at home:

  • Richard Henshall, Senior Manager for Product Management - Ansible Product Updates
  • Matt Jones, Ansible Senior Principal Software Engineer - The Future of Automation
  • Chris Wright, Red Hat CTO - Automation at the Edge
  • Robyn Bergeron, Senior Principal Community Architect - Continue reading

Deploying WordPress to the Cloud

I was curious the other day how hard it would be to actually set up my own blog or rather I was more interested in how easy it is now to do this with containers. There are plenty of platforms that host blogs for you but is it really now as easy to just run one yourself?

In order to get started, you can sign up for a Docker ID, or use your existing Docker ID to download the latest version of Docker Desktop Edge which includes the new Compose on ECS experience. 

Start with the local experience

To start I setup a local WordPress instance on my machine, grabbing a Compose file example from the awesome-compose repo.

Initially I had a go at running this locally on with Docker Compose:

$ docker-compose up -d

Then I can get the list of running containers:

$ docker-compose ps
           Name                          Command               State          Ports
--------------------------------------------------------------------------------------
deploywptocloud_db_1          docker-entrypoint.sh --def ...   Up      3306/tcp, 33060/tcp
deploywptocloud_wordpress_1   docker-entrypoint.sh apach ...   Up      0.0.0.0:80->80/tcp

And then lastly I had a look to see that this was running correctly:

Deploy to the Cloud

Great! Now I needed to look at the contents of the Compose file Continue reading

Docker’s sessions at KubeCon 2020

In a few weeks, August 17-20, lots of us at Docker in Europe were looking forward to hopping on the train down to Amsterdam for KubeCon CloudNativeCon Europe. But like every other event since March, this one is virtual so we will all be at home joining remotely. Most of the sessions are pre recorded with live Q&A, the format that we used at DockerCon 2020. As a speaker I really enjoyed this format at DockerCon, we got an opportunity to clarify and answer extra questions during the talk. It will be rather different from the normal KubeCon experience with thousands of people at the venue though!

Our talks

Chris Crone has been closely involved with the CNAB (Cloud Native Application Bundle) project since the launch in late 2018. He will be talking about how to Simplify Your Cloud Native Application Packaging and Deployments, and will explain why CNAB is a great tool for developers. Packaging up entire applications into self contained artifacts is a really useful tool, an extension of packaging up a single container. The tooling, especially Porter has been making a lot of progress recently so if you heard about CNAB before and are wondering what Continue reading

Docker Talks Live Stream Monthly Recap

Here at Docker, we have a deep love for developers and with more and more of the community working remotely, we thought it would be a great time to start live streaming and connecting with the community virtually. 

To that end, Chad Metcalf (@metcalfc) and I (@pmckee) have started to live stream every Wednesday at 10am Pacific Time on YouTube. You can find all of the past streams and subscribe to get notifications when we go live on our YouTube channel.

Every week we will cover a new topic focusing on developers and developer productivity using the Docker platform. We will have guest speakers, demo a bunch of code and answer any questions that you might have. 

Below I’ve compiled a list of past live streams that you can watch at your leisure and we look forward to seeing you on the next live stream.

Docker ♥ AWS – A match made in heaven

Cloud container runtimes are complex and the learning curve can be steep for some developers. Not all development teams have DevOps teams to partner with which shifts the burden of understanding runtime environments, CLIs, and configuration for the cloud to the Continue reading

How To Setup Your Local Node.js Development Environment Using Docker – Part 2

In part I of this series, we took a look at creating Docker images and running Containers for Node.js applications. We also took a look at setting up a database in a container and how volumes and network play a part in setting up your local development environment.

In this article we’ll take a look at creating and running a development image where we can compile, add modules and debug our application all inside of a container. This helps speed up the developer setup time when moving to a new application or project. 

We’ll also take a quick look at using Docker Compose to help streamline the processes of setting up and running a full microservices application locally on your development machine.

Fork the Code Repository

The first thing we want to do is download the code to our local development machine. Let’s do this using the following git command:

git clone [email protected]:pmckeetx/memphis.git

Now that we have the code local, let’s take a look at the project structure. Open the code in your favorite IDE and expand the root level directories. You’ll see the following file structure.

├── docker-compose.yml
├── notes-service
│   ├── config
│    Continue reading

Using an Inventory Plugin from a Collection in Ansible Tower

Many IT environments grow more and more complex. It is more important than ever that an automation solution always has the most up to date information about what nodes are present and need to be automated. To answer this challenge, the Red Hat Ansible Automation Platform uses inventories: lists of managed nodes.

In its simplest form, inventories can be static files. This is ideal when getting started with Ansible, but as the automation is scaled, a static inventory file is not enough anymore:

  1. How do we update and maintain a list of all of our managed nodes if something changes, if workloads are spun up or teared down?
  2. How do we classify our infrastructure so that we can be more selective in what managed nodes we automate against?

The answer to both of these questions is to use a dynamic inventory: a script or a plugin that will go to a source of truth and discover the nodes that need to be managed. It will also automatically classify the nodes by putting them into groups, which can be used to more selectively target devices when automating with Ansible.

Inventory plugins allow Ansible users to use external platforms to Continue reading

Automating Security with CyberArk and Red Hat Ansible Automation Platform

Proper privilege management is crucial with automation. Automation has the power to perform multiple functions across many different systems. When automation is deployed enterprise-wide, across sometimes siloed teams and functions, enterprise credential management can simplify adoption of automation — even complex authentication processes can be integrated into the setup seamlessly, while adding additional security in managing and handling those credentials.

Depending on how users have defined them, users can craft Ansible Playbooks that require access to credentials and secrets that have wide access to organizational systems. These are necessary to systems and IT resources to accomplish their automation tasks, but they’re also a very attractive target for bad actors. In particular, they are tempting targets for advanced persistent threat (APT) intruders. Gaining access to these credentials could give the attacker the keys to the entire organization.

Most breaches involve stolen credentials, and APT intruders prefer to leverage privileged accounts like administrators, service accounts with domain privileges, and even local admin or privileged user accounts.

You’re probably familiar with the traditional attack flow: compromise an environment, escalate privilege, move laterally, continue to escalate, then own and exfiltrate. It works, but it also requires a lot of work and a lot of Continue reading

Docker Index: Dramatic Growth in Docker Usage Affirms the Continued Rising Power of Developers

Developers have always been an integral part of business innovation and transformation. With the massive increase in Docker usage, we can see the continued rising importance of developers as they create the next generation of cloud native applications. 

You may recall in February we introduced the Docker Index, which gives a snapshot and analysis of developer and dev team preferences and trends based on anonymized data from 5 million Docker Hub users, 2 million Docker Desktop users and countless other developers engaging with content on Docker Hub.

According to a newly updated Docker Index, the eight months between November 2019 and July 2020 have seen a dramatic swell in consumption across the Docker community and ecosystem. How exactly is usage expanding? Let us count the ways.

Last November, there were 130 billion pulls on Docker Hub. That seemed worth talking about, so we shared this data in a blog in February. But since then consumption of the world’s most popular repository for application components (Docker Hub lest there be any doubt) has skyrocketed; in July, total pulls on Docker Hub reached 242 billion. That’s almost a doubling of pulls in a little over six months. (To be Continue reading

Creating an AWS ELB using Pulumi and Go

In case you hadn’t noticed, I’ve been on a bit of a kick with Pulumi and Go recently. There are two reasons for this. First, I have a number of “learning projects” (things that I decide I’d like to try or test) that would benefit greatly from the use of infrastructure as code. Second, I’ve been working on getting more familiar with Go. The idea of combining both those reasons by using Pulumi with Go seemed natural. Unfortunately, examples of using Pulumi with Go seem to be more limited than examples of using Pulumi with other languages, so in this post I’d like to share how to create an AWS ELB using Pulumi and Go.

Here’s the example code:

elb, err := elb.NewLoadBalancer(ctx, "elb", &elb.LoadBalancerArgs{
	NamePrefix:             pulumi.String(baseName),
	CrossZoneLoadBalancing: pulumi.Bool(true),
	AvailabilityZones:      pulumi.StringArray(azNames),
	Instances:              pulumi.StringArray(cpNodeIds),
	HealthCheck: &elb.LoadBalancerHealthCheckArgs{
		HealthyThreshold:   pulumi.Int(3),
		Interval:           pulumi.Int(30),
		Target:             pulumi.String("SSL:6443"),
		UnhealthyThreshold: pulumi.Int(3),
		Timeout:            pulumi.Int(30),
	},
	Listeners: &elb.LoadBalancerListenerArray{
		&elb.LoadBalancerListenerArgs{
			InstancePort:     pulumi.Int(6443),
			InstanceProtocol: pulumi.String("TCP"),
			LbPort:           pulumi.Int(6443),
			LbProtocol:       pulumi.String("TCP"),
		},
	},
	Tags: pulumi.StringMap{
		"Name": pulumi.String(fmt.Sprintf("cp-elb-%s", baseName)),
		k8sTag: pulumi.String("shared"),
	},
})

You can probably infer from the code above that this Continue reading

Containerized Python Development – Part 3

This is the last part in the series of blog posts showing how to set up and optimize a containerized Python development environment. The first part covered how to containerize a Python service and the best development practices for it. The second part showed how to easily set up different components that our Python application needs and how to easily manage the lifecycle of the overall project with Docker Compose.

In this final part, we review the development cycle of the project and discuss in more details how to apply code updates and debug failures of the containerized Python services. The goal is to analyze how to speed up these recurrent phases of the development process such that we get a similar experience to the local development one.

Applying Code Updates

In general, our containerized development cycle consists of writing/updating code, building, running and debugging it.

For the building and running phase, as most of the time we actually have to wait, we want these phases to go pretty quick such that we focus on coding and debugging.

We now analyze how to optimize the build phase during development. The build phase corresponds to image build time when we change Continue reading

1 … 25 26 27 28 29 … 126