Rate Limiting by the Numbers
As a critical part of Docker’s transition into sustainability, we’ve been gradually rolling out limits on docker pulls to the heaviest users of Docker Hub. As we near the end of the implementation of the rate limits, we thought we’d share some of the facts and figures behind our effort. Our goal is to ensure that Docker becomes sustainable for the long term, while continuing to offer developers 100% free tools to build, share, and run their applications.
We announced this plan in August with an effective date of November 1. We also shared that “roughly 30% of all downloads on Hub come from only 1% of our anonymous users,” illustrated in this chart:
This shows the dramatic impact that a very small percentage of anonymous, free users have on all of Docker Hub. That excessive usage by just 1%–2% of our users results not only in an unsustainable model for Docker but also slows performance for the other 98%–99% of the 11.3 million developers, CI services, and other platforms using Docker Hub every month. Those developers rely upon us to save and share their own container images, as well as to pull images from Docker Verified Publishers Continue reading
Apple Silicon M1 Chips and Docker
Revealed at Apple’s ‘One More Thing’ event on Nov 10th, Docker was excited to see new Macs feature Apple silicon and their M1 chip. At Docker we have been looking at the new hypervisor features and support that are required for Mac to continue to delight our millions of customers. We saw the first spotlight of these efforts at Apple WWDC in June, when Apple highlighted Docker Desktop on stage. Our goal at Docker is to provide the same great experience on the new Macs as we do today for our millions of users on Docker Desktop for Mac, and to make this transition as seamless as possible.
Building the right experience for our customers means getting quite a few things right before we push a release. Although Apple has released Rosetta 2 to help move applications over to the new M1 chips, this does not get us all the way with Docker Desktop. Under the hood of Docker Desktop, we run a virtual machine, to achieve this on Apple’s new hardware we need to move onto Apple’s new hypervisor framework. We also need to do all the plumbing that provides the core experience of Docker Continue reading
Taking Your App Live with Docker and the Uffizzi App Platform
Tune in December 10th 1pm EST for our
Live DockTalk: Simplify Hosting Your App in the Cloud with Uffizzi and Docker
We’re excited to be working with Uffizzi on this joint blog. Docker and Uffizzi have very similar missions that naturally complement one another. Docker helps you bring your ideas to life by reducing the complexity of application development and Uffizzi helps you bring your ideas to life by reducing the complexity of cloud application hosting.
This blog is a step-by-step guide to setting up automated builds from your Github repo via Docker Hub and enabling Continuous Deployment to your Uffizzi app hosting environment.
Prerequisites
To complete this tutorial, you will need the following:
- Free Docker Account
- You can sign-up for a free Docker account and receive free unlimited public repositories
- An IDE or text editor to use for editing files. I would recommend VSCode
- Free Uffizzi App Platform Account
- Free Github Account
Docker Overview
Docker is an open platform for developing, shipping, and running applications. Docker containers separate your applications from your infrastructure so you can deliver software quickly.
With Docker, you can manage your infrastructure in the same ways you manage your applications. By Continue reading
Rate Limiting Questions? We have answers
As we have been implementing rate limiting on Docker Hub for free anonymous and authenticated image pulls, we’ve heard a lot of questions from our users about how this will affect them. And we’ve also heard a number of statements that are inaccurate or misleading about the potential impacts of the change. We want to provide some answers here to help Docker users clearly understand the changes, quantify what is involved, and help developers choose the right Docker subscription for their needs.
First let’s look at the realities of what rate limiting looks like, and quantify what is still available for free to authenticated Docker users. Anyone can use a meaningful number of Docker Hub images for free. Anonymous, unauthenticated Docker users get 100 container pull requests per six hours. And when a user signs up for a free Docker ID, they get 2X the quantity of pulls. At 200 pulls per six hours, that is approximately 24,000 container image pulls per month per free Docker ID. This egress level is adequate for the bulk of the most common Docker Hub usage by developers. (Docker users can check their usage levels at any time through the command line. Docker developer Continue reading
What’s New and What’s Changed in the Ansible Content Collection for Kubernetes
Increasing business demands are driving the need for increased automation to support rapid, yet stable, and reliable deployments of applications and supporting infrastructure. Kubernetes and cloud-native technologies are no different. That is why we recently released kubernetes.core 1.1, our first Certified Content Collection for deploying and managing Kubernetes applications and services.
Prior to the release of kubernetes.core 1.1, its contents were released as community.kubernetes. With this content becoming Red Hat supported and certified, a name change was in order. We are in the process of making that transition, starting with this release.
In this blog post, we will go over what else has changed and what’s new in this Content Collection as it transitions and enhances it from its community roots.
Focus on The Future
In looking to create a stable and supported release from the upstream sources that Red Hat is known for, the first thing we did was look at what was in community.kubernetes and elsewhere to organize it for the future. This not only led to the aforementioned name change: the content and underlying code was reorganized to be more maintainable and ready to serve as the Continue reading
Combining Snyk Scans in Docker Desktop and Docker Hub to Deploy Secure Containers
Last week, we announced that the Docker Desktop Stable release includes vulnerability scanning, the latest milestone in our container security solution that we are building with our partner Snyk. You can now run Snyk vulnerability scans directly from the Docker Desktop CLI. Combining this functionality with Docker Hub scanning functionality that we launched in October provides you with the flexibility of including vulnerability scanning along multiple points of your development inner loop, and provides better tooling for deploying secure applications.
You can decide if you want to run your first scans from the Desktop CLI side, or from the Hub. Customers that have used Docker for a while tend to prefer starting from the Hub. The easiest way to jump in is to configure the Docker Hub repos to automatically trigger scanning every time that you push an image into that repo. This option is configurable for each repository, so that you can decide how to onboard these scans into your security program. (Docker Hub image is available only for Docker Pro and Team subscribers, for more information about subscriptions visit the Docker Pricing Page.)
Once you enable scanning, you can view the scanning results either Continue reading
Review: CPLAY2air Wireless CarPlay Adapter
In late September, I was given a CPLAY2air wireless CarPlay adapter as a gift. Neither of my vehicles support wireless CarPlay, and so I was looking forward to using the CPLAY2air device to enable the use of CarPlay without having to have my phone plugged into a cable. Here’s my feedback on the CPLAY2air device after about six weeks of use.
In general, the device works reasonably well. Setup with the factory radio in my GMC Sierra truck was pretty straightforward, and only took a few minutes. Since then, the device connects to my phone every time I start the vehicle, and all CarPlay functions—music, maps, Siri, sending/reading messages, etc.—all work as expected. I can leave my phone in my pocket and still gain all the benefits of CarPlay, which is incredibly convenient.
The CPLAY2air is not without a few caveats, however. The wireless connection between the CPLAY2air and my iPhone does introduce some noticeable latency. When switching music tracks, for example, it will take between one and three seconds to stop playing the old track and start playing the new track. (By comparison, the same operation using traditional wired CarPlay is nearly instantaneous.) Similarly, when talking on Continue reading
Resizing Windows to a Specific Size on macOS
I recently had a need (OK, maybe more a desire than a need) to set my browser window(s) on macOS to a specific size, like 1920x1080. I initially started looking at one of the many macOS window managers, but after reading lots of reviews and descriptions and still being unclear if any of these products did what I wanted, I decided to step back to using AppleScript to accomplish what I was seeking. In this post, I’ll share the solution (and the articles that helped me arrive at the solution).
My first stop was this blog post by Ethan Banks. I tried replicating the AppleScript he used, but couldn’t get it to work. I’m still running macOS 10.14 “Mojave,” so perhaps his code was specific to macOS 10.15 “Catalina.” I moved on, never realizing there was another section to his post that had the information I needed (and would eventually find). Let that be a lesson to be sure to read the entire post next time.
Moving on, I arrived at this post. OK, this used a different mechanism than Ethan’s post. I tried it, and it sort of worked, but it didn’t create the window geometry Continue reading
Technology Short Take 133
Welcome to Technology Short Take #133! This time around, I have a collection of links featuring the new Raspberry Pi 400, some macOS security-related articles, information on AWS Nitro Enclaves and gVisor, and a few other topics. Enjoy!
Networking
- Pratik Mankad shows how to use DNS hostnames as targets for an AWS NLB. If we’re honest, it’s a bit of a hack; it uses AWS Lambda to periodically resolve the hostname and update the NLB target IP address(es) accordingly. Native DNS hostname support in NLBs would be a far better solution.
- Netflix has a good post on how they use prioritized load shedding to provide a good user experience during system outages.
Servers/Hardware
- The Raspberry Pi 400 is a neat offering. See this post for more details.
Security
- In the last Technology Short Take, I mentioned that some testing with macOS Big Sur indicated that Apple had left itself a backdoor for network traffic from its own applications. Here’s another article on the topic, with some additional technical detail.
- Here’s a fairly detailed article on macOS authorization.
- Wojciech Regula takes a look at accessing (“stealing”) confidential data stored in the macOS Keychain.
- How did I not know about Continue reading
Compose CLI ACI Integration Now Available
Today we are pleased to announce that we have reached a major milestone, reaching GA and our V1 of both the Compose CLI and the ACI integration. 
In May we announced the partnership between Docker and Microsoft to make it easier to deploy containerized applications from the Desktop to the cloud with Azure Container Instances (ACI). We are happy to let you know that all users of Docker Desktop now have the ACI experience available to them by default, allowing them to easily use existing Docker commands to deploy and manage containers running in ACI.
As part of this I want to also call out a thank you to the MSFT team who have worked with us to make this all happen! That is a big thank you to Mike Morton, Karol Zadora-Przylecki, Brandon Waterloo, MacKenzie Olson, and Paul Yuknewicz.
Getting started with Docker and ACI
As a new starter, to get going all you will need to do is upgrade your existing Docker Desktop to the latest stable version (2.5.0.0 or later), store your image on Docker Hub so you can deploy it (you can get started with Hub here) and then lastly you Continue reading
Automating Helm using Ansible
Increasing business demands are driving the need for increased automation to support rapid, yet stable, and reliable deployments of applications and supporting infrastructure. Kubernetes and cloud-native technologies are no different. For the Kubernetes platform, Helm is the standard means of packaging, configuring and deploying applications and services onto any cluster.
We recently released the kubernetes.core 1.1, our first Red Hat Certified Content Collection release, for general use. A big part of the new content that has been introduced is support for automating Helm operations. In this blog post, I will show you some common scenarios for its use in your automation.
Please note that prior to the release of kubernetes.core 1.1, its contents were released as community.kubernetes. With this content becoming Red Hat support and certified content, a name change was in order. We are in the process of making that transition.
A Quick Introduction to Helm
Helm is an open source tool used for packaging and deploying applications on Kubernetes. It is often called Kubernetes Package Manager. It is widely adopted by the Kubernetes community and the Cloud Native Computing Foundation (CNCF) graduate project.
Helm simplifies deployment of the applications by abstracting Continue reading
Updates on Hub Rate Limits, Partners and Customer Exemptions
The gradual enforcement of the Docker Hub progressive rate limiting enforcement on container image pulls for anonymous and free users began Monday, November 2nd. The next three hour enforcement window on Wednesday, November 4th from 9am to 12 noon Pacific time. During this window, the eventual final limit of 100 container pull requests per six hours for unauthenticated users and 200 for free users with Docker IDs will be enforced. After that window, the limit will rise to 2,500 container pull requests per six hours.
As we implement this policy, we are looking at the core technologies, platforms and tools used in app pipelines to ensure a transition that supports developers across their entire development lifecycle. We have been working with leading cloud platforms, CI/CD providers and other ISVs to ensure their customers and end users who use Docker have uninterrupted access to Docker Hub images. Among these partners are the major cloud hosting providers, CI/CD vendors such as CircleCI, and OSS entities such as Apache Software Foundation (ASF). You can find more information about programs on our Pricing Page as well as links to contact us for information about programs for ISVs and companies with more than 500 Continue reading
Expanded Support for Open Source Software Projects
Docker remains committed to providing a platform where the non-commercial open source developers can continue collaborating, innovating and pushing this industry into new directions.
In August, we announced to our dedicated community and ecosystem that we are creating new policies for image retention and data pull rates. We made these changes to make Docker a sustainable business for the long term, so that we can continue supporting the developer community and ecosystem that depends on the Docker platform. We got great feedback from our extensive user base, and adjusted our policies to delay the policies on image retention until mid-2021. The plan for data pull rates is moving forward, and starting today limits will be gradually enforced, with the plan to be fully applied in the coming weeks. The final limits will be:
- Unauthenticated users will be restricted to 100 pulls every 6 hours
- Authenticated free users will be restricted to 200 pulls every 6 hours
To support the open source community, Docker has created a special program for Open Source projects to get continued free access and freedom from restrictions for their communities and their users. For the approved, non-commercial, open source projects, we are thrilled to announce Continue reading
Pro and Team Subscriptions Embrace Docker Desktop
About a month ago we talked about how we planned to make Docker Desktop more first class as part of our Pro and Team subscriptions. Today we are pleased to announce that with the latest release of Docker Desktop we are launching support for Docker Desktop for Pro and Team users. This means that customers on Pro plans or team members on Team plans will be able to get support outside of the community support in our Github repository, this will include installation support, issues in running Desktop and of course the existing support for Docker Hub.
Along with this, we have our first Pro feature available in Docker Desktop! For Pro and Team users who have scanning enabled in Docker Hub, you will be able to see your scan results directly in the Docker Dashboard.
This is the first step in releasing unique features for Pro and Team users on Docker Desktop.
Along with this we are pleased to announce that in Docker Desktop 2.5 we have the GA release of the docker scan CLI powered by Snyk! To find out more about scanning images locally have a read of Marina’s blog post.
For customers Continue reading
Control your content with private Automation Hub
Private Automation Hub is now available as part of Red Hat Ansible Automation Platform release 1.2, providing an easier way for our customers to manage their Ansible content. Whether they produce private content, access trusted and supported content from Red Hat or obtain content from third party or other community sources, an internally controlled capability is essential to support the continued growth of automation. As automation becomes critical to managing IT activities, so too becomes the need to have a focal point where collaboration can be encouraged, content shared and trust reinforced.
Private Automation Hub is a self-hosted Ansible content management system. Organizations can host private hubs on their own infrastructure and manage it themselves. Similar to how Red Hat Satellite enables Red Hat Enterprise Linux customers to manage operating system content, private Automation Hub enables automation teams to manage Ansible automation content. Private Automation Hub allows curation and distribution of Ansible content as close as possible to Ansible Automation Platform clusters. Private Automation Hub is included in the Red Hat Ansible Automation Platform subscription.
Ansible content can be broken up into three main categories:
- Community content found in Ansible Galaxy
- Red Hat certified and supported content Continue reading
What you need to know about upcoming Docker Hub rate limiting
On August 13th, we announced the implementation of rate limiting for Docker container pulls for some users. Beginning November 2, Docker will begin phasing in limits of Docker container pull requests for anonymous and free authenticated users. The limits will be gradually reduced over a number of weeks until the final levels (where anonymous users are limited to 100 container pulls per six hours and free users limited to 200 container pulls per six hours) are reached. All paid Docker accounts (Pro, Team or Legacy subscribers) are exempt from rate limiting.
The rationale behind the phased implementation periods is to allow our anonymous and free tier users and integrators to see the places where anonymous CI/CD processes are pulling container images. This will allow Docker users to address the limitations in one of two ways: upgrade to an unlimited Docker Pro or Docker Team subscription, or adjust application pipelines to accommodate the container image request limits. After a lot of thought and discussion, we’ve decided on this gradual, phased increase over the upcoming weeks instead of an abrupt implementation of the policy. An up-do-date status update on rate limitations is available at https://www.docker.com/increase-rate-limits.
Docker users Continue reading
Checking Your Current Docker Pull Rate Limits and Status
Continuing with our move towards consumption-based limits, customers will see the new rate limits for Docker pulls of container images at each tier of Docker subscriptions starting from November 2, 2020.
Anonymous free users will be limited to 100 pulls per six hours, and authenticated free users will be limited to 200 pulls per six hours. Docker Pro and Team subscribers can pull container images from Docker Hub without restriction as long as the quantities are not excessive or abusive.
In this article, we’ll take a look at determining where you currently fall within the rate limiting policy using some command line tools.
Determining your current rate limit
Requests to Docker Hub now include rate limit information in the response headers for requests that count towards the limit. These are named as follows:
- RateLimit-Limit
- RateLimit-Remaining
The RateLimit-Limit header contains the total number of pulls that can be performed within a six hour window. The RateLimit-Remaining header contains the number of pulls remaining for the six hour rolling window.
Let’s take a look at these headers using the terminal. But before we can make a request to Docker Hub, we need to obtain a bearer token. We will then Continue reading
Setting Up Cloud Deployments Using Docker, Azure and Github Actions
A few weeks ago I shared a blog about how to use GitHub Actions with Docker, prior to that Guillaume has also shared his blog post on using Docker and ACI. I thought I would bring these two together to look at a single flow to go from your code in GitHub all the way through to deploying on ACI using our new Docker to ACI experience!
To start, let’s remember where we were with our last Github action. Last time we got to a point where our builds to master would be re-built and pushed to Docker Hub (and we used some caching to speed these up).
name: CI to Docker Hub
on:
push:
tags:
- "v*.*.*"
jobs:
build:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v2
-
name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v1
-
name: Cache Docker layers
uses: actions/cache@v2
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-
-
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
-
name: Build and push
id: docker_build
uses: docker/build-push-action@v2
with:
context: ./
file: ./Dockerfile
builder: ${{ steps.buildx.outputs.name Continue readingDocker’s Next Chapter: Our First Year
2020 has been quite the year. Pandemic, lockdowns, virtual conferences and back-to-back Zoom meetings. Global economic pressures, confinement and webcams aside, we at Docker have been focused on delivering what we set out to do when we announced Docker’s Next Chapter: Advancing Developer Workflows for Modern Apps last November 2019. I wish to thank the Docker team for their “can do!” spirit and efforts throughout this unprecedented year, as well as our community, our Docker Captains, our ecosystem partners, and our customers for their non-stop enthusiasm and support. We could not have had the year we had without you.
This next chapter is being jointly written with you, the developer, as so much of our motivation and inspiration comes from your sharing with us how you’re using Docker. Consider the Washington University School of Medicine (WUSM): WUSM’s team of bioinformatics developers uses Docker to build pipelines – consisting of up to 25 Docker images in some cases – for analyzing the genome sequence data of cancer patients to inform diagnosis and treatments. Furthermore, they collaborate with each other internally and with other cancer research institutions by sharing their Docker images through Docker Hub. In the words of WUSM’s Dr. Continue reading
Docker V2 Github Action is Now GA
Docker is happy to announce the GA of our V2 Github Action. We’ve been working with @crazy-max over the last few months along with getting feedback from the wider community on how we can improve our existing Github Action. We have now moved from our single action to a clearer division and advanced set of options that not only allow you to just build & push but also support features like multiple architectures and build cache.
The big change with the advent of our V2 action is also the expansion of the number of actions that Docker is providing on Github. This more modular approach and the power of Github Actions has allowed us to make the minimal UX changes to the original action and add a lot more functionality.
We still have our more meta build/push action which does not actually require all of these preconfiguration steps and can still be used to deliver the same workflow we had with the previous workflow! To Upgrade the only changes are that we have split out the login to a new step and also now have a step to setup our builder.
-
name: Setup Docker Buildx
uses: docker/setup-buildx-action@v1This Continue reading