New capability of Docker - Moby & LinuxKit

, KAZUHIRO TACHIBANA, CONTAINER

Overview

  This blog introduces what and why Moby & LinuxKit and it’s fundamental concept and major featres. By the end, you will learn what you can benefit from Moby & LinuxKit.

What and Why is Moby?

  Container technologies have evolved over 30 years, and they have been adopted in many industries, such as cloud native application based ecosystem since Docker was released in 2013.
  Solomon Hykes, who is founder and CTO of Docker, said in his blog here, that while containers was going to mainstream, there was one trend we’d found that,

  Every industry and vertical market, finance, healthcare, government, travel, manufacturing. And every use case, modern web applications, traditional server applications, machine learning, industrial control systems, robotics. What many new entrants in the container ecosystem have in common is that they build specialized systems, targeted at a particular infrastructure, industry or use case. (Solomon Hykes, https://blog.docker.com/2017/04/introducing-the-moby-project)

  In 2017, Moby comes up with the idea that it would be needed to build such a specialized container system in every industry or use case with less effort in order to spread containers more widely to be mainstream.

  Moby is an open framework to assemble specialized container system (e.g. Docker) without reinventing the wheel.
  Moby provides the following components,

  • A library of containerized components for all vital aspects of a container system like containerd, runC, InfraKit, SwarmKit, HyperKit, VPNKit, DataKit etc.
  • Tools to assemble the components into a specialized container system, for a variety of backend.
  • A set of reference assemblies which can be used as-is, modified, or used as inspiration to create your own

What and Why is LinuxKit?

  LinuxKit is custom immutable, secure, leaner, and portable Linux subsystem which is embedded to container system as an underlying component to provide Linux minimal functionality for running container runtime.
  Even though Docker has extended its supported platforms to Windows(using hyper-V), Mac OS(using hyperkit), AWS, Azure and GCP, the most highest expectation is still not fulfilled that Docker runs on Linux Container(LXC), a platform which is not Linux native.
  Because LinuxKit achieves OS level immutable infrastructure as mentioned in above, it is possible to run Linux native Docker on any platform including Windows, Mac OS and it should prevent you from making unnecessary effort to set up non-unified Docker environment as per platform which may have potential risk to get vendor locked-in.
  And LinuxKit’s roots are in Alpine.

Concept

  What you can achieve in using Moby and LinuxKit combination is that Moby takes LinuxKit as an underlying component to assemble Linux distribution as a container system, that is very much similar to CoreOS, Rancher OS, Atomic etc, but more leaner.
  It provides Linux minimal functionality to run Linux native container in anywhere, which means that shifting to “Immutable Infrastructure”, which is one of the most important container attributes, from “Runtime Level” to “OS Level”, Moby and LinuxKit have capability of covering this level.

Component

  The component in here represents Docker image that is used to compose Linux distribution built by Moby.
  The component is supposed to be classified into a category as shown in below as per its role, which Moby references in the build process for Linux distribution.

Assembly

  Assembly would represent the rule where to unpack Docker image based on Filesystem Hierarchy Standard (a.k.a FHS).
  Docker images classified into “kernel” or “init” are unpacked into the root filesystem(rootfs), which should usually contain a kernel or init module directory, while the other classified into “onboot” or “services” are extracted under “containers/” directory where init runs them.
  The filesystem directory structure after build, looks like the below overall.

Runtime

  Docker images unpacked directory into the root filesystem(rootfs), which are used for kernel and init system, should bring up containerd and runC to start system or service containers unpacked to “containers” directory.
  Those running containers should be compliant with Open Container Initiative(a.k.a OCI) specification.

Backend

  Backend, most likely, would represent environment or platform providing virtualization software, like Hypervisor to construct virtualization environment in where you want to run built image.
  Because It depends on image format you build, which backend used for running the image will be decided. Available backend and supported image format in each backend are as shown in below table.

Features

  Moby provides two major features to handle Moby image: Building and Running.

Build

  Build starts up and manages Moby image build pipeline, downloading, unpacking, archiving, compressing, based on build configuration in yaml file, which specifies Docker images classified into one of category as shown in here, “kernel”, “init”, “onboot”, “services” and Docker images classified into “onboot” or “services” need to get specified with the relevant image options to OCI specification in order to make “config.json” file which is required for runC to run them in runtime.
  After compressing, Build generates Moby image with specified image format as shown in here, depending on which backend where you want to run Moby image.

Run

  Run runs Moby image on a backend as shown in here.
  Because LinuxKit which provides Linux minimal functionality is in Moby image as an underlying component, Linux native container run can be achieved on any hypervisor which is running on any platform.

Summary

  As for the summary, what we will benefit from Moby & LinuxKit in terms of their concept looks as listed in below.

  • Easy to assemble dozens of standard components into specialized Linux distribution with container runtime.
  • No need container specific leaner OS like CoreOS, Rancher OS, Atomic etc.
  • Linux native container can be run in anywhere.
  • OS level immutable infrastructure is achieved
  • Portability is more powerful.
  • Major backends, like GCP, Azure, AWS, OpenStack, KVM/QEMU,Hyper-V, HyperKit are supported.

  In addition to the benefits, there is one thing that gets added, even though there seems to be many advantages in using Moby & LinuxKit, it doesn’t mean all of the current Docker user have to start using Moby & LinuxKit.
  Usually, Moby & LinuxKit are strongly recommended to system engineers building a container system, but not recommended to application engineers looking for an easy solution to run their applications in containers or enterprise development teams looking for a stable container system with commerical support.
It would be advised to remind where you are for adopting Moby & LinuxKit.

Lastly, because Moby & LinuxKit should accelerate making container mainstream to build ecosystem, Rakuten will keep an eyes on their feasibility to bring the value to us.

Also, we are hiring, If you are interested in working as a container engineer, please contact us here