Creating containers – Part 1

This is part one of a series of blog posts detailing how docker creates containers. We will dig deep into the various pieces that are stitched together to see what it takes to make docker run ... awesome.

First, what is a container?

I think the various pieces of technology that goes into creating a container is fairly commonplace. You should have seen a few cool looking charts in various presentations about Docker where you get a quick “Docker uses namespaces, cgroups, chroot, etc.” to create containers. But why does it take all these pieces to create a contaienr?
Why is it not a simple syscall and it’s all done for me? The fact is that container’s don’t exist, they are made up. There is no such thing as a “linux container” in the kernel. A container is a userland concept.

Namespaces

In part one I’ll talk about how to create Linux namespaces in the context of how they are used within docker. In later posts we will look into how namespaces are combined with other features like cgroups and an isolated filesystem to create something useful.

First off we need a high level explanation of what a namespace does and why it’s useful. Basically, a namespace is a scoped view of your underlying Linux system. There are a few different types of namespaces implemented inside the kernel. As we dig into each of the different namespaces below you can follow along by running docker run -it --privileged --net host crosbymichael/make-containers. This has a few preloaded files and configuration to get your started. Even though we will be creating namespaces inside an container that docker runs for us, don’t let that trip you up. I opted for this approach as providing a container preloaded with all the dependencies that you need to run the examples is why we are doing this in the first place. To make things a little easier, I’m using the --net host flag so that we are able to see your host’s network interfaces within our demo container. This will be useful in the network examples. We also need to provide the --privilged flag so that we have the correct permissions to create new namespaces within our container.

If you are interested in what the Dockerfile looks like then here it is:

FROM debian:jessie
RUN apt-get update && apt-get install -y \
    gcc \
    vim \
    emacs
COPY containers/ /containers/
WORKDIR /containers
CMD ["bash"]

I’ll be doing the examples in C, as it’s sometimes easier to explain the lower level details better than the abstractions that Go provides. So lets start…

NET Namespace

The network namespaces provides your own view of the network stack of your system. This can include your very own localhost. Make sure you are in the crosbymichael/make-containers and run the command ip a to view all the network interfaces of your host machine.

> ip a
root@development:/containers# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:19:ca:f2 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe19:caf2/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:20:84:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.103/24 brd 192.168.56.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe20:8447/64 scope link
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 56:84:7a:fe:97:99 brd ff:ff:ff:ff:ff:ff
    inet 172.17.42.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::5484:7aff:fefe:9799/64 scope link
       valid_lft forever preferred_lft forever

Ok cool, so this is all the network interfaces currently on my host system. Yours may look a little different but you get the idea. Now let’s write some code to create a new network namespace. For this we will write a skeleton of a small C binary that uses the clone syscall. We will start by using clone to run binaries that are already installed inside our demo container. The file skeleton.c should be in the working directory of the demo container. We will use this file as the basis of all our examples. Here is the code incase you don’t feel like running the container right now.

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sched.h>
#include <sys/wait.h>
#include <errno.h>
#define STACKSIZE (1024*1024)
static char child_stack[STACKSIZE];
struct clone_args {
        char **argv;
};
// child_exec is the func that will be executed as the result of clone
static int child_exec(void *stuff)
{
        struct clone_args *args = (struct clone_args *)stuff;
        if (execvp(args->argv[0], args->argv) != 0) {
                fprintf(stderr, "failed to execvp argments %s\n",
                        strerror(errno));
                exit(-1);
        }
        // we should never reach here!
        exit(EXIT_FAILURE);
}
int main(int argc, char **argv)
{
        struct clone_args args;
        args.argv = &argv[1];
        int clone_flags = SIGCHLD;
        // the result of this call is that our child_exec will be run in another
        // process returning it's pid
        pid_t pid =
            clone(child_exec, child_stack + STACKSIZE, clone_flags, &args);
        if (pid < 0) {
                fprintf(stderr, "clone failed WTF!!!! %s\n", strerror(errno));
                exit(EXIT_FAILURE);
        }
        // lets wait on our child process here before we, the parent, exits
        if (waitpid(pid, NULL, 0) == -1) {
                fprintf(stderr, "failed to wait pid %d\n", pid);
                exit(EXIT_FAILURE);
        }
        exit(EXIT_SUCCESS);
}

This is a small C binary that will allow you to run processes like ./a.out ip a. It uses the arguments that you pass on the cli as the arguments to whatever process you want to use. Don’t worry about the specific implementation too much as it’s the changes we will be making that are the interesting aspects. Remember, this will execute the binary and arguments of whatever program you want, this means if you want to run one of these demos below and have it spawn a shell session so that you can poke around in your new namespace then go ahead. It is a great way to explore and inspect these different namespaces at your own pace. So to get started let’s make a copy of this file to start working with the network namespace.

> cp skeleton.c network.c

Ok, within this file there is a very special var called clone_flags. This is where most of our changes will happen throughout this post. Namespaces are primarily controlled via the clone flags. The clone flag for the network namespace is CLONE_NEWNET. We need to change the line in the file int clone_flags = SIGCHLD; to int clone_flags = CLONE_NEWNET | SIGCHLD; so that the call to clone creates a new network namespace for our process. Make this change in network.c then compile and run.

> gcc -o net network.c
> ./net ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

The result of this run now looks very different from the first time we ran the ip a command. We only see a loopback interface in the output. This is because our process that was created only has a view of its network namespace and not of the host. And that’s it. That is how you create a new network namespace.

Right now this is pretty useless as you don’t have any usable interfaces. Docker uses this new network namespace to setup a veth interface so that your container has it’s own ip address allocated on a bridge, usually docker0. We won’t go down the path of how to setup interfaces in namespaces at this time. We can save that for another post.

So now that we know how a network namespace is created lets look at the mount namespace.

MNT Namespace

The mount namespace gives you a scoped view of the mounts on your system. It’s often confused with jailing a process inside a chroot or similar. This is not true! The mount namespaces does not equal a filesystem jail. So the next time you hear someone say that a container uses the mount namespace to “jail” the process inside it’s own root filesystem you can call bullshit because they don’t know what they are talking about. Do it, it’s fun 🙂

Let’s start by making a copy of skeleton.c again for our mount related changes. We can do a quick build and run to see what our current mount points looks like with the mount command.

> cp skeleton.c mount.c
> gcc -o mount mount.c
> ./mount mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,mode=755)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/dev/disk/by-uuid/d3aa2880-c290-4586-9da6-2f526e381f41 on /etc/resolv.conf type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/disk/by-uuid/d3aa2880-c290-4586-9da6-2f526e381f41 on /etc/hostname type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/disk/by-uuid/d3aa2880-c290-4586-9da6-2f526e381f41 on /etc/hosts type ext4 (rw,relatime,errors=remount-ro,data=ordered)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

This is what the mount points look like from within my demo container, yours may look different. In order to create a new mount namespace we use the flag CLONE_NEWNS. You may notice something may look weird here. Why is this flag not CLONE_NEWMOUNT or CLONE_NEWMNT? This is because the mount namespace was the first Linux namespace introduced and the name was just an undersight. If you write code you will understand that as you start building a feature or an application, you don’t often have a full picture of the end result. Anyway, let’s add CLONE_NEWNS to our clone_flags variable. It should look something like int clone_flags = CLONE_NEWNS | SIGCHLD;.

Lets go ahead and build mount.c again and run the same command.

> cp skeleton.c mount.c
> gcc -o mount mount.c
> ./mount mount

Nothing changed. Whaaat??? This is because the process that we run inside the new mount namespace still has a view on /proc of the underlying system. The result is that the new process sort of inherits a view on the underlying mounts. There are a few ways that we can prevent this, like using pivot_root, but we will leave that for an additional post on filesystem jails and how chroot / pivot_root interact with the container’s mount namespace.

However, one way that we can try out our new mount namespace is to, well, mount something. Let’s create a new tmpfs mount in /mytmp for this demo. We will do this mount in C and continue to run our same mount command as the args to our mount binary. In order to do a mount inside of our mount namespace we need to add the code to the child_exec function, before the call to execvp. The code within the child_exec function is run inside the newly created process, i.e., inside our new namespace. The code in child_exec should look like this:

// child_exec is the func that will be executed as the result of clone
static int child_exec(void *stuff)
{
        struct clone_args *args = (struct clone_args *)stuff;
        if (mount("none", "/mytmp", "tmpfs", 0, "") != 0) {
                fprintf(stderr, "failed to mount tmpfs %s\n",
                        strerror(errno));
                exit(-1);
        }
        if (execvp(args->argv[0], args->argv) != 0) {
                fprintf(stderr, "failed to execvp argments %s\n",
                        strerror(errno));
                exit(-1);
        }
        // we should never reach here!
        exit(EXIT_FAILURE);
}

We need to first create a directory /mytmp before we compile and run with the changes above.

> mkdir /mytmp
> gcc -o mount mount.c
> ./mount mount
# cutting out the common output...
none on /mytmp type tmpfs (rw,relatime)

I cut out the common output above from the first time I ran mount.
The result is that you should see a new mount point for our tmpfs mount. Nice! Go ahead and run mount in the current shell just for comparison.
Notice how the tmpfs mount is not displayed? That is because we created the mount inside our own mount namespace, not in the parent’s namespace.

Remember how I said that the mount namepaces does not equal a filesystem jail? Go ahead and run our ./mount binary with the ls command. Everything is there. Now you have proof!

UTS Namespace

The next namespace is the UTS namespace that is responsible for system identification. This includes the hostname and domainname. It allows a container to have it’s own hostname independently from the host system along with other containers. Let’s start by making a copy of skeleton.c and running the hostname command with it.

> cp skeleton.c uts.c
> gcc -o uts uts.c
> ./uts hostname
development

This should display your system’s hostname (development in my case). Like earlier, let’s add the clone flag for the UTS namespace to the clone_flags variable. The flag should be CLONE_NEWUTS. If you compile and run then you should see the exact same output. This is totally fine. The values in the UTS namespace are inherited from the parent. However, within this new namespace, we can change the hostname without it affecting the parent or other container’s that have a separate UTS namespace.

Let’s modify the hostname in the child_exec function. To do that, you will need to add the #include <unistd.h> header to gain access to the sethostname function, as well as the #include <string.h> header to use strlen needed by sethostname. The new body of the child_exec function should look like the following:

// child_exec is the func that will be executed as the result of clone
static int child_exec(void *stuff)
{
        struct clone_args *args = (struct clone_args *)stuff;
        const char * new_hostname = "myhostname";
        if (sethostname(new_hostname, strlen(new_hostname)) != 0) {
                fprintf(stderr, "failed to execvp argments %s\n",
                        strerror(errno));
                exit(-1);
        }
        if (execvp(args->argv[0], args->argv) != 0) {
                fprintf(stderr, "failed to execvp argments %s\n",
                        strerror(errno));
                exit(-1);
        }
        // we should never reach here!
        exit(EXIT_FAILURE);
}

Ensure that clone_flags within your main look like int clone_flags = CLONE_NEWUTS | SIGCHLD; then compile and run the binary with the same args. You should now see the value that we set returned from the hostname command. To verify that this change did not affect our current shell go ahead and run hostname and make sure that you have your original value back.

> gcc -o uts uts.c
> ./uts hostname
myhostname
> hostname
development

Awesome! We are doing good.

IPC Namespace

The IPC namespace is used for isolating interprocess communication, things like SysV message queues. Let’s make a copy of skeleton.c for this namespace.

> cp skeleton.c ipc.c

The way we are going to test the IPC namespace is by creating a message queue on the host, and ensuring that we cannot see it when we spawn a new process inside it’s own IPC namespace. Let’s first create a message queue in our current shell then compile and run our copy of the skeleton code to view the queue.

> ipcmk -Q
Message queue id: 65536
> gcc -o ipc ipc.c
> ./ipc ipcs -q
------ Message Queues --------
key        msqid      owner      perms      used-bytes   message
0xfe7f09d1 65536      root       644        0            0

Without a new IPC namespace you can see the same message queue that was created. Now let’s add the CLONE_NEWIPC flag to our clone_flags var to create a new IPC namespace for our process. The clone_flags var should look like int clone_flags = CLONE_NEWIPC | SIGCHLD;. Recompile and run the same command again:

> gcc -o ipc ipc.c
> ./ipc ipcs -q
------ Message Queues --------
key        msqid      owner      perms      used-bytes   message

Done! The child process is now in a new IPC namespace and has completely separate view and access to message queues.

PID Namespace

This one is fun. The PID namespace is a way to carve up the PIDs that one process can view and interact with. When we create a new PID namespace the first process will get to be the loved PID 1. If this process exits the kernel kills everyone else within the namespace. Let’s start by making a copy of skeleton.c for our changes.

> cp skeleton.c pid.c

To create a new PID namespace, we will have to set the clone_flags with CLONE_NEWPID. The variable should look like int clone_flags = CLONE_NEWPID | SIGCHLD;. Let’s test by running ps aux in our shell and then compile and run our pid.c binary with the same arguments.

> ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  20332  3388 ?        Ss   21:50   0:00 bash
root       147  0.0  0.1  17492  2088 ?        R+   22:49   0:00 ps aux
> gcc -o pid pid.c
> ./pid ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  20332  3388 ?        Ss   21:50   0:00 bash
root       153  0.0  0.0   5092   728 ?        S+   22:50   0:00 ./pid ps aux
root       154  0.0  0.1  17492  2064 ?        R+   22:50   0:00 ps aux

WTF??? We expected ps aux to be PID 1 or atleast not see any other pids from the parent. Why is that? proc. The process that we spawned still has a view of /proc from the parent, i.e. /proc mounted on the host system. So how do we fix this? How to we ensure that our new process can only view pids within it’s namespace? We can start by remounting /proc.
Because we will be dealing with mounts, we can take the opportunity to take what we learned from the MNT namespace and combine it with our PID namespace so that we don’t mess with the /proc of our host system.

We can start by including the clone flag for the mount namespace along side the clone flag for pid. It should look something like int clone_flags = CLONE_NEWPID | CLONE_NEWNS | SIGCHLD;. We need to edit the child_exec function and remount proc. This will be a simple unmount and mount syscall for the proc filesystem. Because we are creating a new mount namespace we know that this will not mess up our host system. The result should look like this:

// child_exec is the func that will be executed as the result of clone
static int child_exec(void *stuff)
{
        struct clone_args *args = (struct clone_args *)stuff;
        if (umount("/proc", 0) != 0) {
                fprintf(stderr, "failed unmount /proc %s\n",
                        strerror(errno));
                exit(-1);
        }
        if (mount("proc", "/proc", "proc", 0, "") != 0) {
                fprintf(stderr, "failed mount /proc %s\n",
                        strerror(errno));
                exit(-1);
        }
        if (execvp(args->argv[0], args->argv) != 0) {
                fprintf(stderr, "failed to execvp argments %s\n",
                        strerror(errno));
                exit(-1);
        }
        // we should never reach here!
        exit(EXIT_FAILURE);
}

Build and run this again to see what happens.

> gcc -o pid pid.c
> ./pid ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   9076   784 ?        R+   23:05   0:00 ps aux

Perfect! Our new PID namespace is now fully operational with the help of the mount namespace!

USER Namespace

The last namespace is the user namespace. This namespace is the new kid on the block and allows you to have users within this namespace that are not equal users outside of the namespace. This is accomplished via GID and UID mappings.

This one has a simple demo application without specifying a mapping, even though it’s a totally useless demo. If we add the flag CLONE_NEWUSER to our clone_flags then run something like id or ls -la you will notice that we get nobody within the user namespace. This is because the current user is undefined right now.

> cp skeleton.c user.c
# add the clone flag
> gcc -o user user.c
> ./user ls -la
total 84
drwxr-xr-x 1 nobody nogroup 4096 Nov 16 23:10 .
drwxr-xr-x 1 nobody nogroup 4096 Nov 16 22:17 ..
-rwxr-xr-x 1 nobody nogroup 8336 Nov 16 22:15 mount
-rw-r--r-- 1 nobody nogroup 1577 Nov 16 22:15 mount.c
-rwxr-xr-x 1 nobody nogroup 8064 Nov 16 21:52 net
-rw-r--r-- 1 nobody nogroup 1441 Nov 16 21:52 network.c
-rwxr-xr-x 1 nobody nogroup 8544 Nov 16 23:05 pid
-rw-r--r-- 1 nobody nogroup 1772 Nov 16 23:02 pid.c
-rw-r--r-- 1 nobody nogroup 1426 Nov 16 21:59 skeleton.c
-rwxr-xr-x 1 nobody nogroup 8056 Nov 16 23:10 user
-rw-r--r-- 1 nobody nogroup 1442 Nov 16 23:10 user.c
-rwxr-xr-x 1 nobody nogroup 8408 Nov 16 22:40 uts
-rw-r--r-- 1 nobody nogroup 1694 Nov 16 22:36 uts.c

This is a very simple example of the user namespace but you can go much deeper with it. We will save this for another post but the idea and hopes for the user namespace is that this will allow us to run as “root” within the container but not as “root” on the host system. Don’t forget you can always change ls -la to bash and have a shell inside the namespace to poke around and learn more.

In the end…

So to recap we went over the mount, network, user, PID, UTS, and IPC Linux namespaces. The majority of the code that we changed was not much, just adding a flag most of the time.
The “hard work” is mostly managing the interactions between the various kernel subsystems in order to meet our requirements. Like most of the descriptions before this, namespaces are just one of the tools that we use to make a container. I hope the PID example is a glimpse of how we use multiple namespaces together in order isolate and begin the creation of a container.

In future posts we will go into detail on how we jail the container’s processes inside a root filesystem, aka a docker image, as well as using cgroups and Linux capabilities. By the end we should be able to pull all these things together to create a container.

Also a thanks to tibor and everyone that helps review my brain dump of a first draft 😉

Other articles

1: Don’t boot init

List all installed packages

Build entire pkg directory

Install entire pkg directory

Test entire pkg

Format and simplify code

Detect races …

Tools

1: Use the cache

Servers

Factory Pattern

Update

The Problem

This post is still not formatting. I switched from worpress and this is a long post and have not had time to update it.