CPU Management in Docker 1.13

Resource management for containers is a huge requirement for production users. Being able to run multiple containers on a single host and ensure that one container does not starve the others in terms of cpu, memory, io, or networking in an efficient way is why I like working with containers. However, cpu management for containers is still not as straightforward as what I would like. There are many different options when it comes to dealing with restricting the cpu usage for a container. With things like memory, its is very easy for people to think that , --memory 512m gives the container up to 512mb. With CPU, it’s hard for people to understand a container’s limit with the current options.

In Docker 1.13 we added a --cpus flag, which is the best tech for limiting cpu usage of a container with a sane UX that the majority of users can understand. Let’s take a look at a couple of the options in 1.12 to show why this is necessary.

There are various ways to set a cpu limit for a container. Cpu shares, cpuset, cfs quota and period are the three most common ways. We can just go Continue reading

Introducing Docker 1.13

Today we’re releasing Docker 1.13 with lots of new features, improvements and fixes to help Docker users with New Year’s resolutions to build more and better container apps. Docker 1.13 builds on and improves Docker swarm mode introduced in Docker 1.12 and has lots of other fixes. Read on for Docker 1.13 highlights.

Use compose-files to deploy swarm mode services

Docker 1.13 adds Compose-file support to the `docker stack deploy` command so that services can be deployed using a `docker-compose.yml` file directly. Powering this is a major effort to extend the swarm service API to make it more flexible and useful.

Benefits include:

• Specifying the number of desired instances for each service
• Rolling update policies
• Service constraints

Deploying a multi-host, multi-service stack is now as simple as:

docker stack deploy --compose-file=docker-compose.yml my_stack

Improved CLI backwards compatibility

Ever been bitten by the dreaded Error response from daemon: client is newer than server problem because your Docker CLI was updated, but you still need to use it with older Docker engines?

Starting with 1.13, newer CLIs can talk to older daemons. We’re also adding feature negotiation so that proper errors are returned if a new Continue reading

Docker for Windows Server and Image2Docker

In December we had a live webinar focused on Windows Server Docker containers. We covered a lot of ground and we had some great feedback – thanks to all the folks who joined us. This is a brief recap of the session, which also gives answers to the questions we didn’t get round to.

Webinar Recording

You can view the webinar on YouTube:

The recording clocks in at just under an hour. Here’s what we covered:

• 00:00 Introduction
• 02:00 Docker on Windows Server 2016
• 05:30 Windows Server 2016 technical details
• 10:30 Hyper-V and Windows Server Containers
• 13:00 Docker for Windows Demo – ASP.NET Core app with SQL Server
• 25:30 Additional Partnerships between Docker, Inc. and Microsoft
• 27:30 Introduction to Image2Docker
• 30:00 Demo – Extracting ASP.NET Apps from a VM using Image2Docker
• 52:00 Next steps and resources for learning Docker on Windows

Q&A

Can these [Windows] containers be hosted on a Linux host?

No. Docker containers use the underlying operating system kernel to run processes, so you can’t mix and match kernels. You can only run Windows Docker images on Windows, and Linux Docker images on Linux.

However, with an upcoming release to the Windows network stack, you will be Continue reading

InfraKit Under the Hood: High Availability

Back in October, Docker released Infrakit, an open source toolkit for creating and managing declarative, self-healing infrastructure. This is the first in a two part series that dives more deeply into the internals of InfraKit.

Introduction

At Docker,  our mission to build tools of mass innovation constantly challenges to look at ways to improve the way developers and operators work. Docker Engine with integrated orchestration via Swarm mode have greatly simplified and improved efficiency in application deployment and management of microservices. Going a level deeper, we asked ourselves if we could improve the lives of operators by making tools to simplify and automate orchestration of infrastructure resources. This led us to open source InfraKit, as set of building blocks for creating self-healing and self-managing systems.

There are articles and tutorials (such as this, and this) to help you get acquainted with InfraKit. InfraKit is made up of a set of components which actively manage infrastructure resources based on a declarative specification. These active agents continuously monitor and reconcile differences between your specification and actual infrastructure state. So far, we have implemented the functionality of scaling groups to support the creation of a compute cluster or Continue reading

DockerCon 2017 first speakers announced

To the rest of the world, DockerCon 2017 may seem a ways away, but here at Docker we are heads down reading your Call for Papers submissions and curating content to make this the biggest and best DockerCon to date. With that, we are thrilled to share with you the DockerCon 2017 Website with helpful information including ten of the first confirmed speakers and sessions.

If you want to join this amazing lineup and haven’t submitted your cool hack, use case or deep dive session, don’t hesitate! The Call for Papers closes this Saturday, January 14th.

Submit a talk

First DockerCon speakers

Laura Frank

Sr. Software Engineer, Codeship

Everything You Thought You Already Knew About Orchestration

Julius Volz

Co-founder, Prometheus

Monitoring, the Prometheus Way

Liz Rice

Co-founder & CEO, Microscaling Systems

What have namespaces done for you lately?

Thomas Graf

Principal Engineer at Noiro, Cisco

Cilium – BPF & XDP for containers

Brendan Gregg 

Sr. Performance Architect, Netflix

Container Tracing Deep Dive

Thomas Shaw

Build Engineer, Activision

Activision’s Skypilot: Delivering amazing game experiences through containerized pipelines

Fabiane Nardon

Chief Scientist Continue reading

Now Open: 2017 Docker Scholarship & Meet the 2016 Recipients!

Last year, Docker announced our inaugural Docker Scholarship Program in partnership with Hack Reactor. The 2017 scholarship to Hack Reactor’s March cohort is now open and accepting applications.

The scholarship includes full tuition to Hack Reactor, pending program acceptance, and recipients will be paired with a Docker mentor.

Applications will be reviewed and candidates who are accepted into the Hack Reactor program and meet Docker’s criteria will be invited to Docker HQ for a panel interview with Docker team members. Scholarships will be awarded based on acceptance to the Hack Reactor program, demonstration of personal financial need and quality of application responses. The Docker scholarship is open to anyone who demonstrates a commitment to advancing equality in their community. All gender and gender identities are encouraged to apply. Click here for more information.

Apply Now

We are excited to introduce our 2016 Docker scholarship recipients, Maurice Okumu and Savaughn Jones!

In their own words, learn more about Maurice and Savaughn below:

Maurice Okumu 

My name is Maurice Okumu and I was born and raised in Kenya. I came to the USA about three years ago after having lived in Dubai for more than five Continue reading

Docker Storage and Infinit FAQ

Last December, Docker acquired a company called Infinit. Using their technology, we will provide secure distributed storage out of the box, making it much easier to deploy stateful services and legacy enterprise applications on Docker.

During the last Docker Online Meetup, Julien Quintard, member of Docker’s technical staff and former CEO at Infinit, went through the design principles behind their product and demonstrated how the platform can be used to deploy a storage infrastructure through Docker containers in a few command lines.

Providing state to applications in Docker requires a backend storage component that is both scalable and resilient in order to cope with a variety of use cases and failure scenarios. The Infinit Storage Platform has been designed to provide Docker applications with a set of interfaces (block, file and object) allowing for different tradeoffs.

Check out the following slidedeck to learn more about the internals of their platform:

Unfortunately, the video recording from the meetup is not available this time around but you can watch the following presentation and demo of Infinit from its CTO Quentin Hocquet at the Docker Distributed Systems Summit:

Docker and Infinit FAQ

1. Do you consider NFS/GPFS and other HPC cluster Continue reading

Attending DockerCon? Choose what workshop to attend

Following in last year’s major success, we are excited to be bringing back and expand the paid workshops at DockerCon 2017. The pre-conference workshops will focus on a range of subjects from Docker 101 to deep dives in networking, Docker for JAVA and  advanced orchestration. Each workshop is designed to give you hands-on instruction and insight on key Docker topics, taught by Docker Engineers and Docker Captains. The workshops are a great opportunity to get better acquainted and excited about Docker technology to start off DockerCon week.

Take advantage of the lowest DockerCon pricing and get your Early Bird Ticket + Workshop now! Early Bird Tickets are limited and will sell out in the next two weeks!

Here are the basics of the DockerCon workshops:

Date: Monday, April 17, 2017

Time: 2:00pm – 5:00pm

Where: Austin Convention Center – 500 E. Cesar Chavez Street, Austin, TX

Cost: $150

Class size: Classes will remain small and are limited to 50 attendees per class.

Registration: The workshops are only open to DockerCon attendees. You can register for the workshops as an add-on package through the registration site here.

Below are overviews of each workshop. To learn more about each topic head over Continue reading

containerd livestream recap

In case you missed it last month, we announced that Docker is extracting a key component of its container platform, a part of the engine plumbing called containerd – a core container runtime – and committed to donating it to an open foundation.

You can find up-to-date roadmap, architecture and API definitions in the Github repository, and more details about the project in our engineering team’s blog post.

You can also watch the following video recording of the containerd online meetup, for a summary and Q&A with Arnaud Porterie, Michael Crosby, Stephen Day, Patrick Chanezon and Solomon Hykes from the Docker team:

Here is the list of top questions we got following this announcement:

Q. Are you planning to run docker without runC ?

A. Although runC is the default runtime, as of  Docker 1.12, it can be replaced by any other OCI-compliant implementation. Docker will be compliant with the OCI Runtime Specification

Q. What major changes are on the roadmap for swarmkit to run on containerd if any? 

A. SwarmKit is using Docker Engine to orchestrate tasks, and Docker Engine is already using containerd for container execution. So technically, you are already using Continue reading

DockerCon 2017: Call For Papers FAQ

It’s a new year, and we are looking for new stories of how you are using Docker technology to do big things. Submit your cool hack, use case or deep dive sessions before the DockerCon 2017 CFP closes on January 14th.

To help with your submissions, we’ve answered the most frequent questions below and put together a list of tips to help get your proposal selected.

Q. How do I submit a proposal?

A. Submit your proposal here.

Q. What kind of talks are you looking for?

A. This year, we are looking for cool hacks, user stories and deep dive submissions:

• Cool Hacks: Show us your cool hack and wow us with the interesting ways you can push the boundaries of the Docker stack. You do not have to have your hack ready by the submission deadline, just clearly explain your hack, what makes it cool and the technologies you will use.

• Using Docker: Tell us first-hand about your Docker usage, challenges and what you learned along the way and inspire us on how to use Docker to accomplish real tasks.

• Deep Dives: Propose code and demo heavy deep-dive sessions on what you have been able to transform with Continue reading

New Year’s Resolution: Learn Docker

Remember last year when I said the market for Docker jobs was blowing up? Well, it’s more than doubled in the last year. And Swarm is also rising quickly, growing 12829%, almost all of that in the last year. We expect that with our partnership with Microsoft and Windows Docker containers, that this will grow even faster in the next year as .NET developers start to containerize their applications and Windows IT Professionals start porting their infrastructure to Docker. Take a look at this trendline from indeed.com.

So what are you doing to increase your Docker skills? Want a few suggestions?

Whether you’re a developer or more an ops person, a great place to start is the Docker Labs repository, which has currently 28 labs for you to choose from. They range from beginner tutorials, to orchestration workshops, security and networking tutorials, and guides for using different programming languages and developer tools.

Of course there’s also the Docker Documentation, which has a rich set of resources.

At Dockercon 2017 in April, there will be rich set of material for beginners and experts alike, and you will get to meet people from all over the world Continue reading

Top Docker content of 2016

2016 has been an amazing year for Docker and the container industry. We had 3 major releases of Docker engine this year , and tremendous increase in usage. The community has been following along and contributing amazing Docker resources to help you learn and get hands-on experience. Here’s some of the top read and viewed content for the year:

Releases

Of course releases are always really popular, particularly when they fit requests we had from the community. In particular, we had:

• Docker for Mac & Docker for Windows Beta and GA release blog posts, and the video

• Docker 1.12 Built-in Orchestration release, and the DockerCon keynote where we announced it

• And the release of the Docker for AWS and Azure beta

Windows Containers

When Microsoft made Windows 2016 generally available, people rushed to

• Our release blog to read the news
• Tutorials to find out how to use Windows containers powered by Docker
• The commercial relationship blog post to understand how it all fits together

About Docker

We also provide a lot of information about how to use Docker. In particular, these posts and articles that we shared on social media were the most read:

• Containers are Not VMs by Mike Continue reading

Understanding Docker Networking Drivers and their use cases

Applications requirements and networking environments are diverse and sometimes opposing forces. In between applications and the network sits Docker networking, affectionately called the Container Network Model or CNM. It’s CNM that brokers connectivity for your Docker containers and also what abstracts away the diversity and complexity so common in networking. The result is portability and it comes from CNM’s powerful network drivers. These are pluggable interfaces for the Docker Engine, Swarm, and UCP that provide special capabilities like multi-host networking, network layer encryption, and service discovery.

Naturally, the next question is which network driver should I use? Each driver offers tradeoffs and has different advantages depending on the use case. There are built-in network drivers that come included with Docker Engine and there are also plug-in network drivers offered by networking vendors and the community. The most commonly used built-in network drivers are bridge, overlay and macvlan. Together they cover a very broad list of networking use cases and environments. For a more in depth comparison and discussion of even more network drivers, check out the Docker Network Reference Architecture.

Bridge Network Driver

The bridge networking driver is the first driver on our list. It’s simple to understand, Continue reading

Docker & Prometheus Joint Holiday Meetup Recap

Last Wednesday we had our 52nd meetup at Docker HQ, but this time we joined forces with the Prometheus user group to host a mega-meetup! There was a great turnout and members were excited to see the talks on using Docker with Prometheus, OpenTracing and the new Docker playground; play-with-docker.

First up was Stephen Day, a Senior Software Engineer at Docker, who presented a talk entitled, ‘The History of Metrics According to Me’. Stephen believes that metrics and monitoring should be built into every piece of software we create, from the ground up. By solving the hard parts of application metrics in Docker, he thinks it becomes more likely that metrics are a part of your services from the start. See the video of his intriguing talk and slides below.

‘The History of Metrics According to Me’ @stevvooe talking #metrics and #monitoring at the #Docker SF #meetup! @prometheusIO @CloudNativeFdn pic.twitter.com/6hk0yAtats

— Docker (@docker) December 15, 2016

Next up was Ben Sigelman, an expert in distributed tracing, whose talk ‘OpenTracing Isn’t Just Tracing: Measure Twice, Instrument Once’ was both informative and humorous. He Continue reading

Announcing Federal Security and Compliance Controls for Docker Datacenter

Continue reading

More details about containerd, Docker’s core container runtime component

Today we announced that Docker is extracting a key component of its container platform, a part of the engine plumbing–containerd a core container runtime–and commits to donating it to an open foundation. containerd is designed to be less coupled, and easier to integrate with other tools sets. And it is being written and designed to address the requirements of the major cloud providers and container orchestration systems.

Because we know a lot of Docker fans want to know how the internals work, we thought we would share the current state of containerd and what we plan for version 1.0. Before that, it’s a good idea to look at what Docker has become over the last three and a half years.

The Docker platform isn’t a container runtime. It is in fact a set of integrated tools that allow you to build ship and run distributed applications. That means Docker handles networking, infrastructure, build, orchestration, authorization, security, and a variety of other services that cover the complete distributed application lifecycle.

The core container runtime, which is containerd, is a small but vital part of the platform. We started breaking out containerd from the rest of the engine in Docker 1.11, Continue reading

containerd – a core container runtime project for the industry

Today Docker is spinning out its core container runtime functionality into a standalone component, incorporating it into a separate project called containerd, and will be donating it to a neutral foundation early next year. This is the latest chapter in a multi-year effort to break up the Docker platform into a more modular architecture of loosely coupled components.

Over the past 3 years, as Docker adoption skyrocketed, it grew into a complete platform to build, ship and run distributed applications, covering many functional areas from infrastructure to orchestration, the core container runtime being just a piece of it. For millions of developers and IT pros, a complete platform is exactly what they need. But many platform builders and operators are looking for “boring infrastructure”: a basic component that provides the robust primitives for running containers on their system, bundled in a stable interface, and nothing else. A component that they can customize, extend and swap out as needed, without unnecessary abstraction getting in their way. containerd is built to provide exactly that.

What Docker does best is provide developers and operators with great tools which make them more productive. Those tools come from integrating many different components into a Continue reading

Convert ASP.NET Web Servers to Docker with Image2Docker

A major update to Image2Docker was released last week, which adds ASP.NET support to the tool. Now you can take a virtualized web server in Hyper-V and extract a Docker image for each website in the VM – including ASP.NET WebForms, MVC and WebApi apps. 

Image2Docker is a PowerShell module which extracts applications from a Windows Virtual Machine image into a Dockerfile. You can use it as a first pass to take workloads from existing servers and move them to Docker containers on Windows.

The tool was first released in September 2016, and we’ve had some great work on it from PowerShell gurus like Docker Captain Trevor Sullivan and Microsoft MVP Ryan Yates. The latest version has enhanced functionality for inspecting IIS – you can now extract ASP.NET websites straight into Dockerfiles.

In Brief

If you have a Virtual Machine disk image (VHD, VHDX or WIM), you can extract all the IIS websites from it by installing Image2Docker and running ConvertTo-Dockerfile like this:

Install-Module Image2Docker
Import-Module Image2Docker
ConvertTo-Dockerfile -ImagePath C:\win-2016-iis.vhd -Artifact IIS -OutputPath c:\i2d2\iis

That will produce a Dockerfile which you can build into a Windows container image, using docker build.

How It Works

The Image2Docker Continue reading

DockerCon 2017: Registration And CFP Now Open!

DockerCon 2017 tickets are now available! Take advantage of our lowest pricing today – tickets are limited and Early Bird will sell out fast! We have extended DockerCon to a three-day conference with repeat sessions, hands-on labs and summits taking place on Thursday.

Register for DockerCon

The DockerCon 2017 Call for Proposals is open! Before you submit your cool hack or session proposals, take a look at our tips for getting selected below. We have narrowed the scope of sessions we’re looking for this year down to Cool Hacks and Use Cases. The deadline for submissions is January 14, 2017 at 11:59 PST.

Submit a talk

Proposal Dos:

Submitting a Cool Hack:

Be novel

Show us your cool hacks and wow us with the interesting ways you can push the boundaries of the Docker stack. Check out past audience favorites like Serverless Docker, In-the-air update of a drone with Docker and Resin.io and building a UI for container management with Minecraft for inspiration.

Be clear

You do not have to have your hack ready by the submission deadline, rather, plan to clearly explain your hack, what makes it cool and the technologies you will use.

All Continue reading

Docker for Azure Public Beta

Last week Docker for AWS went public beta, and today Docker for Azure reached the same milestone and is ready for public testing. Docker for Azure is a great way for ops to setup and maintain secure and scalable Docker deployments on Azure.

With Docker for Azure, IT ops teams can:

• Deploy a standard Docker platform to ensure teams can seamlessly move apps from developer laptops to Dockerized staging and production environments, without risk of incompatibilities or lock-in.
• Integrate deeply with underlying infrastructure to ensure Docker takes advantage of the host environment’s native capabilities and exposes a familiar interface to administrators.
• Deploy the platform to all the places where you want to run Dockerized apps, simply and efficiently
• Make sure the latest and greatest Docker versions are available for the hardware, OSs, and infrastructure you love, and provide solid upgrade paths from one Docker version to the next.

To try the latest Docker for Azure beta based on the latest Docker Engine betas, click the button below or get more details on the beta site:

Installation takes a few minutes, and will give you a fully functioning swarm, ready to deploy and scale Dockerized apps.

We first unveiled the Docker for Continue reading