Overview

On a traditional Linux-based host, docker runs on the native OS and provisions and isolates containers with things like containerd and runc. Windows and MacOS don’t run Linux kernels and so you can’t run dockerd directly on the host OS. Instead, a Linux VM is used as an interstitial to run dockerd on. There’s some dark integration magic to allow things like bind volume mounts to the host OS from a container. In short, it looks like this:

You can get a proper explanation of it all from this docker blog post.

If you don’t believe me and want to connect directly to your docker host VM on MacOS, you can run this:

screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

You can also see the .iso it boots from at:

/Applications/Docker.app/Contents/Resources/linuxkit/docker-for-mac.iso

In line with Docker’s principle of “swappable batteries” the Moby framework includes LinuxKit for building small/secure Linux host OS’ that can run on all sorts of things like MacOS, Windows, Mainframes and more. This article has a good overview. Several months ago, Docker-CE for Mac moved away from using the boot2docker VirtualBox VM to a LinuxKit based HyperKit VM.

Making Changes — Trying to just build a kernel module

For our upcoming Defcon talk, I wanted to port the work we’d done on AWS to build WiFi CTF environments in the cloud, to Docker, so people can learn/practise WiFi hacking without needing hardware. This necessitates loading a kernel module mac80211_hwsim to create fake wifi devices. The LinuxKit VM doesn’t have these modules compiled, so my first failed attempt was to try and build these. You can build kernel modules for the existing LinuxKit VM by reading the documentation, and looking at these examples. One critical piece of information not included, is that you can grab the config for the running kernel from /proc/config.gz like so:

> docker run -it --rm -v /:/host -v $(pwd):/macos alpine:latest
/ # uname -a
Linux 23b3e591c4eb 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 Linux
/ # cp /host/proc/config.gz /macos/
/ # exit

You’ll also need the kernel source for that version, e.g.

wget https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.9.93.tar.xz

But, while this will work for simple modules. This won’t work for wifi modules, as the existing LinuxKit kernel doesn’t have base support for wifi capabilities. So building the modules will get you errors like this:

can't insert 'mac80211.ko': unknown symbol in module, or unknown parameter

Ok, it looks like we’re going to need to build a whole new kernel.

Rebuilding A New LinuxKit Kernel

Building a new kernel is relatively easy, especially if you’re familiar with building Linux kernels in general. The documentation is now clear and accurate. If you’re here for wifi, you don’t need to do this, as I’ve done it for you already.

First you need a copy of LinuxKit, then you need to work out what new kernel options you need to add, then you need to build your custom kernel.

Build your custom config.

git clone https://github.com/linuxkit/linuxkit
cd linuxkit/kerneldocker run --rm -ti -v $(pwd):/src linuxkit/kconfig
# In container
cd linux-4.9.96/
make menuconfig
####Configure the kernel the way you want, I enabled base wifi
cp .config /src/config-4.9.x-x86_64-custom
exit #Exit Container

For my purposes, I only added kernel options, so I stick those in a config-wifi file, so I can reuse them for other kernels, as these mostly haven’t changed across versions. You can now build your kernel, I’ve got the steps for that documented in the readme here. It should be a simple case of:

make EXTRA=-custom build_4.9.x-custom

This will build an image and store it in your docker image store with the tag linuxkit/kernel. You can now use this in your own LinuxKit builds. (Amusingly, this build is kernel panic’ing my vanilla docker-ce LinuxKit host right now, so there are still some bugs).

Build a New LinuxKit Image

Great, we’ve got a shiny new kernel image, time to build a new LinuxKit iso. There are a bunch of examples of these in the linuxkit/examples directory. It makes sense to start with the docker-for-mac.yml. Open it up in your favourite editor, and replace the line

image: linuxkit/kernel:4.14.52

with your kernel e.g.

image: linuxkit/kernel:4.14.52-wifi-ba03a8d668eb6be981e1ff71883b5e9e26274971-amd64

Or just use my prebuilt kernel:

image: singelet/kernel:4.14.52-wifi-ba03a8d668eb6be981e1ff71883b5e9e26274971-amd64

If you want to have the kernel modules loaded automatically with modprobe, you’ll also need to add this to the .yml file:

- name: modprobe
    image: linuxkit/modprobe:v0.4
    command: ["modprobe", "-a", "mac80211_hwsim"]

Next up, build yourself an iso with:

linuxkit build --format iso-efi docker-for-mac.yml

If you haven’t build linuxkit, just run make in its top-level directory, and you’ll get the binary in linuxkit/bin/linuxkit.

This will create a file named docker-for-mac-wifi-efi.iso. You can use this, by stopping Docker, backing up your existing docker-ce iso and replacing it with this one, then restarting it.

mv /Applications/Docker.app/Contents/Resources/linuxkit/docker-for-mac.iso /Applications/Docker.app/Contents/Resources/linuxkit/docker-for-mac.iso.orig
cp docker-for-mac-wifi-efi.iso /Applications/Docker.app/Contents/Resources/linuxkit/docker-for-mac.iso

You can check that it boots alright by watching the console with the screen command a the top of this post.

If all went well, you should now have a docker running with your shiny new LinuxKit host, and immediately notice several problems. This is where it gets messy.

Why Some Batteries Are Nicer Than Others

Despite Docker’s principle of swappable batteries, the LinuxKit image they build for Docker-CE for MacOS, has a image docker-ce with proprietary and non-redistributable code. This means that the docker-for-mac.yml that comes with LinuxKit, creates a LinuxKit image that:

• Can’t do any bind mounts, whether to the LinuxKit host or MacOS host
• Will never signal the taskbar icon that it’s started, so the GUI thinks it’s booting constantly.
• Probably several other things I haven’t noticed yet.

This is primarily because of a tool called transfused that communicates with the osxfs process running on the mac. It used to be open source, but dissapeared and after some significant commit archeology, I eventually just asked:

To which the reply was:

And his suggestion was to just copy it out of the existing image, but:

Shucks. Then there’s also sendtohost, which send simple state info to the Docker taskbar agent. It appears to come from a private docker repo called pinata (according to strings inside it). There’s an older version available here. There also possibly a lot more I haven’t figured out yet.

This is why I built get-dockerce, to extract the things you need from the existing LinuxKit image while it’s running. Because I can’t redistribute them. Plus, with the regular release cycle of docker-ce this stuff is likely to change over time. All it really does is copy two files, transfused and sendtohost. Those two files are then used in the docker-fakece image you’ll need to build yourself. After which, they can be used in the modified LinuxKit docker-for-mac.yml file. I had to painstakingly figure out how to make that work with limited documentation (I think LinuxKit .yml files are Docker Cloud Stack files) and by recreating the .iso and restarting Docker each time. There was all sorts of weirdness (like scripts with background’ing, &, directives refusing to execute those), and figuring out how to pass the fuse device through. But it works.

Unfortunately, while transfused will now let you bind mount locations on the LinuxKit host, I haven’t got it working to allow bind mounts to those macOS host. If you have any ideas …

Was It Worth It & What Would Be Ideal

While this was an interesting dive into the innards of LinuxKit and Docker for Mac, the latter part feels like a lot of work and ugly hacks, that are mostly not redistributable, and fragile to change. Ideally, Docker will release the docker-ce image they use, containing transfused and similar publicly on Docker hub (it’s already on everyone’s machines just in a hard to access way) as well as an updated docker-for-mac.yml. Then we could just change the kernel and build a first-tier LinuxKit image. A request I made here.

Alternativley, they could add wireless options to the kernel they ship with LinuxKit, to allow wifi modules to be built. This isn’t great because it’s super specific to a wifi edge case, and doesn’t help people wanting to build custom kernels.