Introducing Docker 1.13
Today we’re releasing Docker 1.13 with lots of new features, improvements and fixes to help Docker users with New Year’s resolutions to build more and better container apps. Docker 1.13 builds on and improves Docker swarm mode introduced in Docker 1.12 and has lots of other fixes. Read on for Docker 1.13 highlights.
Use compose-files to deploy swarm mode services
Docker 1.13 adds Compose-file support to the `docker stack deploy` command so that services can be deployed using a `docker-compose.yml` file directly. Powering this is a major effort to extend the swarm service API to make it more flexible and useful.
Benefits include:
- Specifying the number of desired instances for each service
- Rolling update policies
- Service constraints
Deploying a multi-host, multi-service stack is now as simple as:
docker stack deploy --compose-file=docker-compose.yml my_stackImproved CLI backwards compatibility
Ever been bitten by the dreaded Error response from daemon: client is newer than server problem because your Docker CLI was updated, but you still need to use it with older Docker engines?
Starting with 1.13, newer CLIs can talk to older daemons. We’re also adding feature negotiation so that proper errors are returned if a new Continue reading
Technology Short Take #76
Welcome to Technology Short Take #76, the first Technology Short Take of 2017. Normally, I’d publish this on a Friday, but due to extenuating circumstances (my mother-in-law’s funeral is tomorrow) I’m posting it today. Here’s hoping you find something useful!
Networking
- As of Vagrant 1.9.1, the Cumulus plugin for Vagrant is no longer needed. See this post for more details.
- Ivan Pepelnjak (one of my absolutely favorite folks in the IT industry) has a great post on using regex filters in Ansible to parse text printouts from network devices. This is a super-practical post that anyone working with network automation can put to use right away.
- Craig Matsumoto has an informative article on why machine learning is hard to apply to networking. I’ve heard David Meyer—who is quoted extensively in the article—speak several times, and most of the time his stuff is way beyond my understanding. At least now, though, I get the general direction he’s heading.
- Interested in using OVS with VXLAN on Hyper-V, but without OpenStack? This article by Alin Serdean from Cloudbase Solutions describes how it’s done.
- Here’s an article describing how to establish a site-to-site VPN between a Cisco ASA and Microsoft Azure. Anecdotally, Continue reading
Plain Text Productivity Redux
Almost 2 years ago, I set out on an experiment in plain text productivity. I won’t say the experiment was a failure; I did learn from the experiment, and gaining knowledge is usually a positive outcome. In the end, I switched back to OmniFocus, the OS X- and iOS-specific app I’d been using previously. In the last few weeks, though, I’ve revisited the idea of a plain text productivity system as part of my migration to Ubuntu Linux as my primary desktop OS, and I think I’ve resolved some of the issues that were present in my last attempt.
To recap, in my previous attempt I settled on the TaskPaper format (named after the OS X app of the same name). The format is extraordinarily flexible, and the OS X app is more powerful than you might expect. However, I uncovered some issues that made the solution untenable; namely:
- Handling future tasks is a problem.
- Handling repeating tasks is a problem.
- Handling tasks with due dates is a problem.
- Handling lots of tasks is a problem.
- Keeping task lists synced across computers can be a problem.
At the time, the app had no way to dynamically respond to Continue reading
Enabling an Apple MBP Wireless Adapter with Fedora 25
In this article, I want to share with you the steps I took to enable wireless networking on an older (mid-2011) 13” MacBook Pro running Fedora 25. This is driven by a continued need to evaluate Fedora 25, as I’ve run into a few potential roadblocks with Ubuntu 16.04 as my primary laptop OS. Using Fedora 25 instead may help resolve some of these issues, which primarily center around corporate collaboration.
First, you’ll want to enable the RPM Fusion repositories. This is pretty well documented on the RPM Fusion web site. This link will take you to the configuration page, which will provide links for graphical setup via your browser as well as CLI commands.
Once the RPM Fusion repositories (both Free and Nonfree) repositories are enabled, then it’s just a matter of installing a few packages:
-
First, install the “kernel-devel” package appropriate for your current kernel. The command to use is:
sudo dnf install "kernel-devel-uname-r == $(uname -r)"This could be user error on my part, but I’ve found that it’s necessary to use the full package (including version) instead of just “kernel-devel”. Otherwise, Fedora seems to have a tendency to install the latest package, which may not Continue reading
Docker for Windows Server and Image2Docker
In December we had a live webinar focused on Windows Server Docker containers. We covered a lot of ground and we had some great feedback – thanks to all the folks who joined us. This is a brief recap of the session, which also gives answers to the questions we didn’t get round to.
Webinar Recording
You can view the webinar on YouTube:
The recording clocks in at just under an hour. Here’s what we covered:
- 00:00 Introduction
- 02:00 Docker on Windows Server 2016
- 05:30 Windows Server 2016 technical details
- 10:30 Hyper-V and Windows Server Containers
- 13:00 Docker for Windows Demo – ASP.NET Core app with SQL Server
- 25:30 Additional Partnerships between Docker, Inc. and Microsoft
- 27:30 Introduction to Image2Docker
- 30:00 Demo – Extracting ASP.NET Apps from a VM using Image2Docker
- 52:00 Next steps and resources for learning Docker on Windows
Q&A
Can these [Windows] containers be hosted on a Linux host?
No. Docker containers use the underlying operating system kernel to run processes, so you can’t mix and match kernels. You can only run Windows Docker images on Windows, and Linux Docker images on Linux.
However, with an upcoming release to the Windows network stack, you will be Continue reading
OPML-to-Markdown Conversion Script
In this post, I’d like to share a script I wrote to help with converting Outline Processor Markup Language (OPML) documents to Markdown. If you read the recent update on my Linux migration plans, you may recall that I identified OPML files (created in OmniOutliner) as an area where some work was going to be required. This script is the result of my efforts in this area.
<aside>Before I continue, I want to very briefly point out that this script was written to help in my specific use case. It’s quite likely that you’ll want or need to adjust the behaviors of this script in order to meet the needs of your particular use case.</aside>
This script takes advantage of two tools: pandoc and sed. pandoc is a third-party tool that is easily installed on Ubuntu using apt or apt-get. (I haven’t checked other Linux distributions, but I suspect packages are available there as well.) pandoc is also available for OS X, making it a very handy cross-platform tool to have in my toolchest. (See this post for more information on how you can use pandoc in a Markdown-heavy environment.) sed, of course, is a Continue reading
InfraKit Under the Hood: High Availability
Back in October, Docker released Infrakit, an open source toolkit for creating and managing declarative, self-healing infrastructure. This is the first in a two part series that dives more deeply into the internals of InfraKit.
Introduction
At Docker, our mission to build tools of mass innovation constantly challenges to look at ways to improve the way developers and operators work. Docker Engine with integrated orchestration via Swarm mode have greatly simplified and improved efficiency in application deployment and management of microservices. Going a level deeper, we asked ourselves if we could improve the lives of operators by making tools to simplify and automate orchestration of infrastructure resources. This led us to open source InfraKit, as set of building blocks for creating self-healing and self-managing systems.
There are articles and tutorials (such as this, and this) to help you get acquainted with InfraKit. InfraKit is made up of a set of components which actively manage infrastructure resources based on a declarative specification. These active agents continuously monitor and reconcile differences between your specification and actual infrastructure state. So far, we have implemented the functionality of scaling groups to support the creation of a compute cluster or Continue reading
DockerCon 2017 first speakers announced
To the rest of the world, DockerCon 2017 may seem a ways away, but here at Docker we are heads down reading your Call for Papers submissions and curating content to make this the biggest and best DockerCon to date. With that, we are thrilled to share with you the DockerCon 2017 Website with helpful information including ten of the first confirmed speakers and sessions.
If you want to join this amazing lineup and haven’t submitted your cool hack, use case or deep dive session, don’t hesitate! The Call for Papers closes this Saturday, January 14th.
Submit a talk
First DockerCon speakers
Laura Frank
Sr. Software Engineer, Codeship
Everything You Thought You Already Knew About Orchestration
Julius Volz
Co-founder, Prometheus
Monitoring, the Prometheus Way
Liz Rice
Co-founder & CEO, Microscaling Systems
What have namespaces done for you lately?
Thomas Graf
Principal Engineer at Noiro, Cisco
Cilium – BPF & XDP for containers
Brendan Gregg
Sr. Performance Architect, Netflix
Container Tracing Deep Dive
Thomas Shaw
Build Engineer, Activision
Activision’s Skypilot: Delivering amazing game experiences through containerized pipelines
Fabiane Nardon
Chief Scientist Continue reading
Now Open: 2017 Docker Scholarship & Meet the 2016 Recipients!
Last year, Docker announced our inaugural Docker Scholarship Program in partnership with Hack Reactor. The 2017 scholarship to Hack Reactor’s March cohort is now open and accepting applications.
The scholarship includes full tuition to Hack Reactor, pending program acceptance, and recipients will be paired with a Docker mentor.
Applications will be reviewed and candidates who are accepted into the Hack Reactor program and meet Docker’s criteria will be invited to Docker HQ for a panel interview with Docker team members. Scholarships will be awarded based on acceptance to the Hack Reactor program, demonstration of personal financial need and quality of application responses. The Docker scholarship is open to anyone who demonstrates a commitment to advancing equality in their community. All gender and gender identities are encouraged to apply. Click here for more information.
Apply Now
We are excited to introduce our 2016 Docker scholarship recipients, Maurice Okumu and Savaughn Jones!
In their own words, learn more about Maurice and Savaughn below:
Maurice Okumu
My name is Maurice Okumu and I was born and raised in Kenya. I came to the USA about three years ago after having lived in Dubai for more than five Continue reading
Some Reading on Application Containers
One aspect of my pending migration to Ubuntu Linux on my primary laptop has been the opportunity to explore “non-traditional” uses for Linux containers. In particular, the idea of using Docker (or systemd-nspawn or rkt) to serve as a sandbox (of sorts) for GUI applications really intrigues me. This isn’t a use case that many of the container mechanisms are aiming to solve, but it’s an interesting use case nevertheless (to me, anyway).
So, in no particular order, here are a few articles I found about using Linux containers as application containers/sandboxes (mostly focused around GUI applications):
A Docker-Like Container Management using systemd
Running containers without Docker
Containerizing Graphical Applications on Linux with systemd-nspawn
Debian Containers with systemd-nspawn
Using your own containers with systemd-nspawn and overlayfs
I was successful in using Docker to containerize Firefox (see my “dockerfiles” repository on GitHub)), and was also successful in using systemd-nspawn in the same way, including the use of overlayfs. My experiments have been quite helpful and informative; I have some ideas that may percolate into future blog posts.
Docker Storage and Infinit FAQ
Last December, Docker acquired a company called Infinit. Using their technology, we will provide secure distributed storage out of the box, making it much easier to deploy stateful services and legacy enterprise applications on Docker.
During the last Docker Online Meetup, Julien Quintard, member of Docker’s technical staff and former CEO at Infinit, went through the design principles behind their product and demonstrated how the platform can be used to deploy a storage infrastructure through Docker containers in a few command lines.
Providing state to applications in Docker requires a backend storage component that is both scalable and resilient in order to cope with a variety of use cases and failure scenarios. The Infinit Storage Platform has been designed to provide Docker applications with a set of interfaces (block, file and object) allowing for different tradeoffs.
Check out the following slidedeck to learn more about the internals of their platform:
Unfortunately, the video recording from the meetup is not available this time around but you can watch the following presentation and demo of Infinit from its CTO Quentin Hocquet at the Docker Distributed Systems Summit:
Docker and Infinit FAQ
1. Do you consider NFS/GPFS and other HPC cluster Continue reading
Attending DockerCon? Choose what workshop to attend
Following in last year’s major success, we are excited to be bringing back and expand the paid workshops at DockerCon 2017. The pre-conference workshops will focus on a range of subjects from Docker 101 to deep dives in networking, Docker for JAVA and advanced orchestration. Each workshop is designed to give you hands-on instruction and insight on key Docker topics, taught by Docker Engineers and Docker Captains. The workshops are a great opportunity to get better acquainted and excited about Docker technology to start off DockerCon week.
Take advantage of the lowest DockerCon pricing and get your Early Bird Ticket + Workshop now! Early Bird Tickets are limited and will sell out in the next two weeks!
Here are the basics of the DockerCon workshops:
Date: Monday, April 17, 2017
Time: 2:00pm – 5:00pm
Where: Austin Convention Center – 500 E. Cesar Chavez Street, Austin, TX
Cost: $150
Class size: Classes will remain small and are limited to 50 attendees per class.
Registration: The workshops are only open to DockerCon attendees. You can register for the workshops as an add-on package through the registration site here.
Below are overviews of each workshop. To learn more about each topic head over Continue reading
How to Extend Ansible Through Plugins
Did you know a large portion of Ansible’s functionality comes from the Ansible plugin system? These important pieces of code augment Ansible’s core functionality such as parsing and loading inventory and Playbooks, running Playbooks and reading the results. Essentially, Ansible uses plugins to extend what the system is doing under the hood.
In this blog, I’ll review each of these plugins and offer a high-level overview on how to write your own plugin to extend Ansible functionality.
Action Plugins
One of the core critical plugins used by Ansible are action plugins. Anytime you run a module, Ansible first runs an action plugin.
Action plugins are a layer between the executor engine and the module and allow for controller-side actions to be taken before the module is executed. A good example of this is the template module. If you look at template.py in the modules directory, it’s basically a Python stub with documentation strings, everything is done by the action plugin. The template action plugin itself creates the template file locally as a temporary file, and then uses the copy or file modules to push it out to the target system.
If Ansible finds an action plugin with the Continue reading
containerd livestream recap
In case you missed it last month, we announced that Docker is extracting a key component of its container platform, a part of the engine plumbing called containerd – a core container runtime – and committed to donating it to an open foundation.
You can find up-to-date roadmap, architecture and API definitions in the Github repository, and more details about the project in our engineering team’s blog post.
You can also watch the following video recording of the containerd online meetup, for a summary and Q&A with Arnaud Porterie, Michael Crosby, Stephen Day, Patrick Chanezon and Solomon Hykes from the Docker team:
Here is the list of top questions we got following this announcement:
Q. Are you planning to run docker without runC ?
A. Although runC is the default runtime, as of Docker 1.12, it can be replaced by any other OCI-compliant implementation. Docker will be compliant with the OCI Runtime Specification
Q. What major changes are on the roadmap for swarmkit to run on containerd if any?
A. SwarmKit is using Docker Engine to orchestrate tasks, and Docker Engine is already using containerd for container execution. So technically, you are already using Continue reading
DockerCon 2017: Call For Papers FAQ
It’s a new year, and we are looking for new stories of how you are using Docker technology to do big things. Submit your cool hack, use case or deep dive sessions before the DockerCon 2017 CFP closes on January 14th.
To help with your submissions, we’ve answered the most frequent questions below and put together a list of tips to help get your proposal selected.
Q. How do I submit a proposal?
A. Submit your proposal here.
Q. What kind of talks are you looking for?
A. This year, we are looking for cool hacks, user stories and deep dive submissions:
- Cool Hacks: Show us your cool hack and wow us with the interesting ways you can push the boundaries of the Docker stack. You do not have to have your hack ready by the submission deadline, just clearly explain your hack, what makes it cool and the technologies you will use.
- Using Docker: Tell us first-hand about your Docker usage, challenges and what you learned along the way and inspire us on how to use Docker to accomplish real tasks.
- Deep Dives: Propose code and demo heavy deep-dive sessions on what you have been able to transform with Continue reading
My (Evolving) Multi-Platform Toolbelt
A few days ago I posted a tweet about a new tool I’d (re-)discovered called jrnl. Someone replied to that tweet, asking me to list my “multi-platform toolbelt.” While it’s still evolving (every day!), I thought it might make for a good blog post. So, here’s a list of my still-evolving multi-platform toolbelt.
-
Sublime Text: Over the last few years, I’ve moved to creating the vast majority of my content in Markdown (MultiMarkdown, to be more specific). At first I was using OS X-specific text editors (first TextMate 1.x, then BBEdit), but last year I switched to Sublime Text. Sublime Text supports OS X, Linux, and Windows. I don’t have any Windows-based systems, so I only use it on OS X and Linux.
-
Wire: My use of Wire is still a bit limited, but only because the reach of the platform is also still a bit limited (this is a classical example of network effect). I’m currently using Wire on Linux and OS X, with plans to extend to iOS and Android. (If you’re using Wire, feel free to look me up! My username is “scottslowe”).
-
IMAP/SMTP: I’ve standardized on using IMAP/SMTP for all my e-mail Continue reading
New Year’s Resolution: Learn Docker
Remember last year when I said the market for Docker jobs was blowing up? Well, it’s more than doubled in the last year. And Swarm is also rising quickly, growing 12829%, almost all of that in the last year. We expect that with our partnership with Microsoft and Windows Docker containers, that this will grow even faster in the next year as .NET developers start to containerize their applications and Windows IT Professionals start porting their infrastructure to Docker. Take a look at this trendline from indeed.com.
So what are you doing to increase your Docker skills? Want a few suggestions?
Whether you’re a developer or more an ops person, a great place to start is the Docker Labs repository, which has currently 28 labs for you to choose from. They range from beginner tutorials, to orchestration workshops, security and networking tutorials, and guides for using different programming languages and developer tools.
Of course there’s also the Docker Documentation, which has a rich set of resources.
At Dockercon 2017 in April, there will be rich set of material for beginners and experts alike, and you will get to meet people from all over the world Continue reading
Technology Short Take #75
Welcome to Technology Short Take #75, the final Technology Short Take for 2016. Fortunately, it’s not the final Technology Short Take ever, as I’ll be back in 2017 with more content. Until then, here’s some data center-related articles and links for your enjoyment.
Networking
- Ajay Chenampara has some observations about running Ansible at scale against network devices.
- Andrey Khomyakov shares some information on automating the setup of whitebox switches running Cumulus Linux in part 2 of this series on learning network automation.
- Russell Bryant has shared the results of some testing comparing ML2+OVS and OVN as backends for OpenStack networking. As Russell indicates in his post, some additional analysis is needed to truly understand what’s happening, but early looks at the results of his tests show performance improvements in OVN versus ML2+OVS when it comes to total time required to boot a VM.
- Ivan Pepelnjak shares a Python script that creates Ansible inventory from Vagrant’s SSH configuration. Handy.
Servers/Hardware
Nothing this time around!
Security
- James Green shares some information on Docker Bench, a tool designed to help secure Docker environments.
- Via Mike Foley, the community now has a new resource to help educate on VM escapes.
- Gordon Turner Continue reading
Top Docker content of 2016
2016 has been an amazing year for Docker and the container industry. We had 3 major releases of Docker engine this year , and tremendous increase in usage. The community has been following along and contributing amazing Docker resources to help you learn and get hands-on experience. Here’s some of the top read and viewed content for the year:
Releases
Of course releases are always really popular, particularly when they fit requests we had from the community. In particular, we had:
- Docker for Mac & Docker for Windows Beta and GA release blog posts, and the video
- Docker 1.12 Built-in Orchestration release, and the DockerCon keynote where we announced it
- And the release of the Docker for AWS and Azure beta
Windows Containers
When Microsoft made Windows 2016 generally available, people rushed to
- Our release blog to read the news
- Tutorials to find out how to use Windows containers powered by Docker
- The commercial relationship blog post to understand how it all fits together
About Docker
We also provide a lot of information about how to use Docker. In particular, these posts and articles that we shared on social media were the most read:
Looking Back: 2016 Project Report Card
As I’ve done for the last few years, in early 2016 I published a list of my planned personal projects for the year. In this post, I’d like to look back on that list of projects and grade myself on my progress (or lack of progress, if that is the case). Even though 2016 isn’t over yet, it’s close enough to the end of the year that things won’t change that much before 2017 is upon us.
For reference, here’s the list of planned 2016 projects:
- Complete a new book (again)
- Make more open source contributions
- Expand my knowledge and use of Python
- Expand my knowledge, use, and focus on public cloud services
- Complete a “wildcard project”
Let’s look at each of these planned projects and see how I fared.
-
Complete a new book (again): Well—in the spirit of total honesty and transparency—this was a major failure. Not only did I fail to complete the network automation book I’ve been working on with Matt Oswalt and Jason Edelman, but the other book project I had planned also did not go anywhere. Granted, the circumstances preventing the second book project were outside my control, but the fact remains I still did Continue reading