How to Deploy GPU-Accelerated Applications on Amazon ECS with Docker Compose
Many applications can take advantage of GPU acceleration, in particular resource-intensive Machine Learning (ML) applications. The development time of such applications may vary based on the hardware of the machine we use for development. Containerization will facilitate development due to reproducibility and will make the setup easily transferable to other machines. Most importantly, a containerized application is easily deployable to platforms such as Amazon ECS, where it can take advantage of different hardware configurations.
In this tutorial, we discuss how to develop GPU-accelerated applications in containers locally and how to use Docker Compose to easily deploy them to the cloud (the Amazon ECS platform). We make the transition from the local environment to a cloud effortless, the GPU-accelerated application being packaged with all its dependencies in a Docker image, and deployed in the same way regardless of the target environment.
Requirements
In order to follow this tutorial, we need the following tools installed locally:
- Windows and MacOS: install Docker Desktop
- Linux: install Docker Engine and Compose CLI
- To deploy to Amazon ECS: an AWS account
For deploying to a cloud platform, we rely on the new Docker Compose implementation embedded into the Docker CLI binary. Therefore, when targeting a cloud Continue reading
How Developers Can Get Started with Python and Docker
Python started in 1991 with humble beginnings focusing on helping “automate the boring stuff.” But over the past few years, we’ve seen Python grow in popularity and become extremely useful not only for scripting but for building modern web applications, machine learning and data science.
The TIOBE Index for February has Python ranked at number 3 on the list. Python has also been in the top 8 rank programming languages for the past 7 years. With such a popular and powerful programming language comes a vibrate and large community.
To that end, we are excited to announce that we are releasing a series of programming language-specific guides to help developers go from discovering the basics of Docker to delivering your images into a production environment and more.
The first in our series is a focus on the Python development ecosystem. We have created a series of tutorials, how-tos, and guides focused on the Python community with much more coming in the future.
We are extremely excited to help Python developers become experts at developing and delivering the next generation of applications using the Docker platform. Below you will find a list of resources and our Python language-specific Continue reading
Save the Date! Next Docker Community All Hands
We’re excited to announce that our next Community All Hands will be on March 11th, 2021. This quarterly event is a unique opportunity for Docker staff and the broader Docker community to come together for live company updates, product updates, demos, community shout-outs and Q&A. We had more than 1,500 attendees for our last all-hands and we hope to double that this time.
This all-hands will be particularly special because it will coincide with none other than….you guessed it…Docker’s 8th birthday! For this “birthday edition,” we’re going to make the event extra special.
We’ll start by extending the format from 1 hour to 3 hours to pack in more Docker goodness. The main piece of feedback we got from our last all hands was that it was way too short. We had too much content that we tried to squeeze into 60 minutes. This longer format will give us plenty of time to cover everything we need to cover and let presenters catch their breath 
Another new feature of this all-hands will be integrated chat and multi-casting made possible by a new innovative video conferencing platform we’ll be using. This will give us the opportunity to present content Continue reading
Docker Index Shows Continued Massive Developer Adoption and Activity to Build and Share Apps with Docker
It’s been one year since we started publishing the Docker Index (stats, trends and analysis from developers and dev teams based on anonymized data from millions of Docker users). At that time we saw how Docker was being used at an incredible scale to power application building globally. Today we are excited to share the latest edition of the Docker Index, this time with some yearly and quarterly comparisons.
Every time we pull these user stats, we are blown away by the sheer volume and continued growth in activity happening across the Docker developer community. It’s clear to see that collaborative application development platforms are the foundation for developers who want to build, share, and run modern apps. We are also thrilled to see this type of growth more than one year after refocusing Docker on making developers’ lives easier. The Docker community has stayed with us and continues to grow at a tremendous pace, giving us very encouraging signals about the path that Docker is taking.
To begin, there has now been a total of 318 billion all time pulls on Docker Hub, an increase of 145% year-over-year. That’s right, the total number of pulls has increased Continue reading
CFP for DockerCon LIVE 2021 Now Open!
Ahoy! You can now submit your talk proposal for DockerCon LIVE 2021!
Taking place May 27, 2021, DockerCon brings together the entire community of Docker developers, contributors and partners to share, teach, and collaborate to grow the understanding and capabilities of modern application developers. The Docker community is growing fast and is incredibly diverse, and our aim is to have a conference that reflects this growth and diversity. To that end, we’re announcing the CFP a bit earlier this year to substantially increase the number of submissions to review.
Share your know-how
Like last year’s edition, DockerCon LIVE will be 100% virtual. To allow for conversation and ensure a stress-free delivery for the speaker, session talks will be pre-recorded and played at a specific time during the conference. Speakers will be able to chat live with their audience while their recorded session is broadcast and be available to answer questions in real-time. We’re really excited about this format and we look forward to introducing a host of new interactive features that’ll ensure that speakers (new and seasoned) and attendees alike have an exceptional experience.
The theme of this year’s DockerCon is developer team collaboration in the new remote-first world.
Before Continue reading
Donating Docker Distribution to the CNCF
We are happy to announce that Docker has contributed Docker Distribution to the Cloud Native Computing Foundation (CNCF). Docker is committed to the Open Source community and open standards for many of our projects, and this move will ensure Docker Distribution has a broad group maintaining what is the foundation for many registries.
What is Docker Distribution?
Distribution is the open source code that is the basis of the container registry that is part of Docker Hub, and also many other container registries. It is the reference implementation of a container registry and is extremely widely used, so it is a foundational part of the container ecosystem. This makes its new home in the CNCF highly appropriate.
Docker Distribution was a major rewrite of the original Registry code which was written in Python and was a much earlier design not using content addressed storage. This new version, written in Go, was designed to be an extensible library, so that different backends and subsystems could be designed. Docker formed the Open Container Initiative (OCI) in 2015, in the Linux Foundation, in order to standardise the specifications for the container ecosystem, including the registry and image formats.
Why are we donating Docker Continue reading
? Open Sourcing the Docker Hub CLI Tool
At Docker, we are committed to making developer’s lives easier, and maintaining and extending our commitment to the Open Source community and open standards for many of our projects. We believe building new capabilities into the Docker Platform in partnership with our developer community and in full transparency leads to much better software.
Last December, we announced the release of a new experimental Docker Hub CLI tool, also known as hub-tool. This new CLI lets you explore, inspect and manage your content on Docker Hub as well as work with your teams and manage your account. We demonstrated it during the last Docker Community All Hands in December 2020.
This tool is already available with Docker Desktop, so if you are a Windows or Mac user you can try it now. For Linux users, we are pleased to announce that we open sourced the hub-tool code, and it can be found at https://github.com/docker/hub-tool. You can download the binary directly on the release page.
With the open sourcing of hub-tool we have also cut a new v0.3.0 release which includes the following new features:
- Added an optional argument to the
account infocommand to check the status of Continue reading
New Docker and JFrog Partnership Designed to Improve the Speed and Quality of App Development Processes
Today, Docker and JFrog announced a new partnership to ensure developers can benefit from integrated innovation across both companies’ offerings. This partnership sets the foundation for ongoing integration and support to help organizations increase both the velocity and quality of modern app development.
The objective of this partnership is simple: how can we ensure developers can get the images they want and trust, and make sure they can access them in whatever development process they are using from a centralized platform? To this end, the new agreement between Docker and JFrog ensures that developers can take advantage of their Docker Subscription and Docker Hub Official Images in their Artifactory SaaS and on-premise environments so they can build, share and run apps with confidence.
At a high level, a solution based on the Docker and JFrog partnership looks like this:
In this sample architecture, developers can build apps with images, including Docker Official Images and images from popular OSS projects and software companies, from Docker Hub. As images are requested, they are cached into JFrog Artifactory, where images can be managed by corporate policies, cached for high performance, and mirrored across an organization’s infrastructure. Also, the images in Artifactory can take Continue reading
New Docker Reporting Provides Teams with Tools for Higher Efficiency and Better Collaboration
Today, we are very excited to announce the release of Audit Log, a new capability that provides the administrators of Docker Team subscription accounts with a chronological report of their team activities. The Audit Log is an unbiased system of record, displaying all the status changes for Docker organizations, teams, repos and tags. As a tracking tool for all the team activities, it creates a central historical repository of actionable insights to diagnose incidents, provide a record of app lifecycle milestones and changes, and provides a view into events creating audit trails for regulatory compliance reviews. The Audit Log is available for Team subscription accounts, and at this point, is not included with Free or Pro subscriptions.
Some typical scenarios where Audit Log will play a key role include:
- When several team members are collaborating on delivering a project, Audit Log creates a list of activities that becomes a ‘source of truth’ to validate which tags got deleted and which tags got pushed into repos, when these activities happened and which team members triggered them.
- Audit Log provides knowledge base continuity, delivering information on projects completed earlier when new team members need to familiarize themselves with work done Continue reading
Docker Captain Take 5 – Elton Stoneman
Docker Captains are select members of the community that are both experts in their field and are passionate about sharing their Docker knowledge with others. “Docker Captains Take 5” is a regular blog series where we get a closer look at our Captains and ask them the same broad set of questions ranging from what their best Docker tip is to whether they prefer cats or dogs (personally, we like whales and turtles over here). Today, we’re interviewing Elton Stoneman who has been a Docker Captain since 2016. He is a Container Consultant and Trainer and is based in Gloucestershire, United Kingdom.
How/when did you first discover Docker?
I was consulting as an API Architect, building out the backend services for a new Android device. My role was all about .NET services running in Azure, but we worked as a single team – and the people working on the operating system were using Docker to simplify their build tools.
I started looking into their setup and I was just stunned at how you could run complex software with a single Docker command – and have it run the same way on any machine. That was way back in 2014, Continue reading
Top Developer Trends for 2021
The year 2020 will go down in the history books for so many reasons. For Docker, despite the challenges of our November 2019 restructuring, we were fortunate to see 70% growth in activity from our 11.3 million monthly active users sharing 7.9 million apps pulled 13.6 billion times per month. Thank you, Docker team, community, customers, and partners!
But with 2020 behind us it’s natural to ask, “What’s next?” Here in the second week of January, we couldn’t be more excited about 2021. Why? Because the step-function shift from offline to online of every dimension of human activity brought about by the global pandemic is accelerating opportunities and challenges for development teams. What are the key trends relevant to development teams in 2021? Here are our top picks:
The New Normal: Open, Distributed Collaboration
While already a familiar teamwork model for many open source projects and Internet companies, the global pandemic seemingly overnight drove all software development teams to adopt new ways of working together. In fact, our 2020 survey of thousands of Docker developers about their ways of working found that 51% prefer to work mostly remote and only sometimes in an office if/when given Continue reading
Save the Date for DockerCon Live 2021!
As the calendar leaves 2020 in the rear view mirror, we’re looking forward to the year ahead. And as part of this, today we’re announcing the dates for DockerCon Live 2021. DockerCon Live will take place on May 27th, 2021. Sign up here to pre-register for the event!
Once again, DockerCon Live will be a free, online experience full of demos of products and innovation from Docker and our partners. You’ll get deep technical sessions from Docker experts, Docker Captains and luminaries from across the industry, along with a chance for the community to gather and connect with colleagues around the world.
Last year DockerCon Live 2020 was one of the largest events in the app dev industry– over 80,000 developers from 193 countries registered to hear over 50 sessions focused on best practices, real world techniques and how-to instruction for building containerized cloud-native solutions with Docker. Speakers joined from companies such as AWS, Google, Microsoft, Salesforce, Nginx, Snyk, Datadog, LaunchDarkly, among others.
For 2021, we are building on this format with a couple of new features including full day pre-conference technical workshops, additional content and more community activities. And there will be some surprises for everyone involved. If you Continue reading
Year in Review: The Most Viewed Docker Blog Posts of 2020 Part 2
It’s end of year round up time! The first post in this series covered the number 10-6 most viewed Docker blog posts. If you were wondering what the #1 most viewed blog post of the year was, then keep reading. The suspense will soon be over…
5) How to Develop Inside a Container Using Visual Studio Code Remote Containers
VS Code is another beloved tool. This guest post from Docker Community Leader Jochen Zehnder included some handy tricks for the Visual Studio Code Remote Containers extension that allows you to develop inside a container.
4) How to deploy on remote Docker hosts with docker-compose
There was some solid Compose momentum this year. This how-to post showed an example of how to access remote docker hosts via SSH and tcp protocols in hopes to cover a large number of use-cases.
3) How To Use the Official NGINX Docker Image
NGINX is super popular, so naturally so was this tutorial that took a look at the NGINX Official Docker Image and how to use it.
2) Containerized Python Development – Part 1
This post contained tips for how to containerize a Python service/tool and the best practices for it. Fun Continue reading
Docker Captain Take 5 – Gianluca Arbezzano
Docker Captains are select members of the community that are both experts in their field and are passionate about sharing their Docker knowledge with others. “Docker Captains Take 5” is a regular blog series where we get a closer look at our Captains and ask them the same broad set of questions ranging from what their best Docker tip is to whether they prefer cats or dogs (personally, we like whales and turtles over here). Today, we’re interviewing Gianluca Arbezzano who has been a Docker Captain since 2016. He is a Senior Software Staff Engineer at Equinix Metal and is based in Italy.
How/when did you first discover Docker?
At this point, it is not easy to pick a date. Four years ago I was in Dublin away from my home town near Turin. The Docker Meetup along with many other meetups were a great opportunity for nerds like me looking for new friends and to grab free pizza while having a good time. Back then I was working for a company that helps businesses move to the cloud. I saw that Docker was a powerful tool to master. Not only was Docker a useful tool and led me Continue reading
Year in Review: The Most Viewed Docker Blog Posts of 2020 Part 1
2020 was some type of year…as we wrap up a year that undoubtedly will never be forgotten, we rounded up the most viewed Docker blog posts. The following posts are some of what you, the Docker community, found to be most interesting and useful. Which was your favorite?
10) Announcing the Compose Specification
Starting the list with a *bang* is a post highlighting that we created a new open community to develop the Compose Specification. This new community is run with open governance and with input from all interested parties, allowing us together to create a new standard for defining multi-container apps that can be run from the desktop to the cloud.
9) Advanced Dockerfiles: Faster Builds and Smaller Images Using BuildKit and Multistage Builds
This post showed some more advanced patterns that go beyond copying files between a build and a runtime stage, allowing one to get the most out of the multistage build feature. Who doesn’t want more efficient multistage Dockerfiles?
8) Containerized Python Development – Part 2
The second in a series, this post discussed how to set up and wire other components to a containerized Python service. It showed a good way to Continue reading
Compiling Qt with Docker multi-stage and multi-platform
This is a guest post from Viktor Petersson, CEO of Screenly.io. Screenly is the most popular digital signage product for the Raspberry Pi. Find Viktor on Twitter @vpetersson.
For those not familiar with Qt, it is a cross-platform development framework that is used in a wide range of products, including cars (Tesla), digital signs (Screenly), and airplanes (Lufthansa). Needless to say, Qt is very powerful. One thing you cannot say about the Qt framework, however, is that it is easy to compile — at least for embedded devices. The countless blog posts, forum threads, and Stack Overflow posts on the topic reveal that compiling Qt is a common headache.
As long-term Qt users, we have had our fair share of battles with it at Screenly. We migrated to Qt for our commercial digital signage software a number of years ago, and since then, we have been very happy with both its performance and flexibility. Recently, we decided to migrate our open source digital signage software (Screenly OSE) to Qt as well. Since these projects share no code base, this was a greenfield opportunity that allowed us to start afresh and explore Continue reading
Closing Out 2020 with More Innovation for Developers
Recently our CEO Scott Johnston took a look back on all that Docker had achieved one year after selling the Enterprise business to Mirantis and refocusing solely on developers. We made significant investments to deliver value-enhancing features for developers, completed strategic collaborations with key ecosystem partners and doubled down on engaging its user community, resulting in a 70% year-over-year increase in Docker usage.
Even though we are winding down the calendar year, you wouldn’t know it based on the pace at which our engineering and product teams have been cranking out new features and tools for cloud-native development. In this post, I’ll add some context around all the goodness that we’ve released recently.
Recall that our strategy is to deliver simplicity, velocity and choice for dev teams going from code to cloud with Docker’s collaborative application development platform. Our latest releases, including Docker Desktop 3.0 and Docker Engine 20.10, accelerate the build, share, and run process for developers and teams.
Higher Velocity Docker Desktop Releases
With the release of Docker Desktop 3.0.0, we are totally changing the way we distribute Docker Desktop to developers. These changes allow for smaller, faster Docker Desktop Continue reading
WSL 2 GPU Support is Here
At Microsoft Build in the first half of the year, Microsoft demonstrated some awesome new capabilities and improvements that were coming to Windows Subsystem for Linux 2 including the ability to share the host machine’s GPU with WSL 2 processes. Then in June Craig Loewen from Microsoft announced that developers working on the Windows insider ring machines could now make use of GPU for the Linux workloads. This support for NVIDIA CUDA enabled developers and data scientists to use their local Windows machines for inner-loop development and experimentation.
Last week, during the Docker Community All Hands, we announced the availability of a developer preview build of Docker Desktop for WSL 2 supporting GPU for our Developer Preview Program. We already have more than 1,000 who have joined us to help test preview builds of Docker Desktop for Windows (and Mac!). If you’re interested in joining the program for future releases you should do it now!
Today we are excited to announce the general preview of Docker Desktop support for GPU with Docker in WSL2. There are over one and a half million users of Docker Desktop for Windows today and we saw in our roadmap how excited you Continue reading
? Docker Hub Experimental CLI tool
We are excited to let you know that we have released a new experimental tool. We would love to get your feedback on it. Today we have released an experimental Docker Hub CLI tool, the hub-tool. The new Hub CLI tool lets you explore, inspect and manage your content on Docker Hub as well as work with your teams and manage your account.
The new tool is available as of today for Docker Desktop for Mac and Windows users and we will be releasing this for Linux in early 2021.
The hub-tool is designed to map as closely to the top level features we know people are using in Docker Hub and provide a new way for people to start interacting with and managing their content. Let’s start by taking a look at the top level options we have.
What you can do
We can see that we have the ability to jump into your account, your content, your orgs and your personal access tokens.
From here I can dive into one of my repos
And from here I can then decide to list the tags in one of those repos. This also now lets me see when Continue reading
Connecting with the Docker Community– Recap of Our First Community All Hands
Last week, we held our first Community All Hands and the response was phenomenal. A huge thank you to all 1,100+ people who joined. If you missed it, you can watch the recording here. You can also find answers to those questions that came in towards the end that we didn’t have time to answer here.
This all-hands was an effort to further deepen our engagement with the community and bring users, contributors and staff together on a quarterly basis to share updates on what we’re working on and what our priorities are for 2021 and beyond. The event was also an opportunity to give the community direct access to Docker’s leadership and provide a platform to submit questions and upvote those that are most relevant and important to people.
The overwhelming piece of feedback we got from attendees was that the event was too short and people would have loved to see more demos. We certainly had a packed agenda and we did our best to squeeze in as much into an hour. For our next one (in February 2021!), we’ll aim to extend the event by 30 minutes and include more live demos. We’ll also try Continue reading