In the previous post, we examined some of the issues of using the host Docker daemon socket, Docker-outside-of-Docker (DooD), inside containers running on a CI/CD system. In this post, we will talk about how running the Docker daemon inside a Docker container, Docker-inside-Docker (DinD), works better with Kubernetes Pods as compared to DooD (If these terms are unfamiliar please check out the previous post). Our infrastructure runs on top of Kubernetes, and we also allow our users to create workflows that are composed of multiple steps. Each step in the workflow is a Kubernetes Pod.
Kubernetes Pods
From the Kubernetes website, a Pod is described in the following words:
A pod (as in a pod of whales or pea pod) is a group of one or more containers (such as Docker containers), the shared storage for those containers, and options about how to run the containers. Pods are always co-located and co-scheduled, and run in a shared context. A pod models an application-specific “logical host” - it contains one or more application containers which are relatively tightly coupled — in a pre-container world, they would have executed on the same physical or virtual machine.
Kubernetes Pods have some very useful properties:
- Each Kubernetes Pod gets a Pod IP that is reachable from all the nodes that form the Kubernetes cluster.
- Containers running in Kubernetes Pods share the same network namespace, i.e. Two containers say apache and mysql running on the same Pod, can talk to each other using "localhost". Also, both apache and mysql can be reached using the pod IP.
- Kubernetes garbage collects pods after they are terminated, preventing nodes from running out of space.
In the context of a Pod, if the user wants to have access to Docker, we have a choice between DooD and DinD. Let's see how DooD and DinD approaches differ when used in the context of a Pod.
Case 1: Pods and DooD
When Docker commands are sent to the host Docker daemon, Kubernetes does not know anything about this newly created container and rightfully does nothing to manage it. If a named container is created using Docker commands, container creation might fail if the named container already exists. Since the new container does not know anything about Pod networking, any open ports will not be reachable using the Pod's IP. As described in the previous post, the port mapping specification on the Docker command (e.g. docker run -v 8080:80 httpd:latest) will open ports on the host, a limited and contended resource. When the container terminates, the layers of graph storage will not be deleted by Kubernetes and logs will not be cleaned up by Kubernetes. In the diagram below, Pod A is a multi-container pod running Apache (httpd) and MySQL containers. Apache is listening on port 80 and MySQL is listening on port 3306. Both these ports can be accessed on the Pod's IP (192.168.2.100:80 and 192.168.2.100:3306). Pod B, on the other hand, uses the docker run command to start Apache. With DooD, the Apache container will run outside the Pod network and will not be available on Pod IP.
Figure 1: Docker outside of Docker on Kubernetes Pods
Below is a kubernetes specification for a Pod that uses DooD to create a container. You can run this on your Kubernetes cluster and see how this works. Copy the following yaml to a file, for example, say dood.yaml and type kubectl create -f dood.yaml on your shell.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
apiVersion: v1
kind: Pod
metadata:
name: dood
spec:
containers:
- name: docker-cmds
image: docker:1.12.6
command: ['docker', 'run', '-p', '80:80', 'httpd:latest']
resources:
requests:
cpu: 10m
memory: 256Mi
volumeMounts:
- mountPath: /var/run
name: docker-sock
volumes:
- name: docker-sock
hostPath:
path: /var/run
|
The Pod will create a container that will run outside of the Pod. By running the container using DooD, you lose out on the following for the spawned container:
- Pod Networking - Cannot access the container using Pod IP.
- Pod Lifecycle - On Pod termination, this container will keep running especially if the container was started with
-dflag. - Pod Cleanup - Graph storage will not be cleanup after pod terminates.
- Scheduling and Resource Utilization - Cpu and Memory requested by Pod, will only be for the Pod and not the container spawned from the Pod. Also, limits on CPU and memory set for the Pod will not be inherited by the spawned container.
Case 2: Pods and DinD
Docker-in-Docker works by running a Docker daemon inside a Docker container. The main requirement for DinD daemon is that it must not share the graph storage of the host's Docker daemon. Containers created with the DinD daemon are not visible to the host Docker daemon. We use the concept of Sidecar containers and create a Kubernetes Pod that contains a dind container. This container starts Docker daemon on /var/lib/docker. We can use EmptyDir and mount it as /var/lib/docker inside the dind container. The yaml file below describes such a Pod. The docker-cmds container issues Docker commands to start the Apache container. The sidecar container, dind-daemon, starts the Docker REST service on port 2375. Setting the DOCKER_HOST to tcp://localhost:2375 ensures that the Docker binary in the main container points to this Docker daemon using DOCKER_HOST environment variable.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
apiVersion: v1
kind: Pod
metadata:
name: dind
spec:
containers:
- name: docker-cmds
image: docker:1.12.6
command: ['docker', 'run', '-p', '80:80', 'httpd:latest']
resources:
requests:
cpu: 10m
memory: 256Mi
env:
- name: DOCKER_HOST
value: tcp://localhost:2375
- name: dind-daemon
image: docker:1.12.6-dind
resources:
requests:
cpu: 20m
memory: 512Mi
securityContext:
privileged: true
volumeMounts:
- name: docker-graph-storage
mountPath: /var/lib/docker
volumes:
- name: docker-graph-storage
emptyDir: {}
|
To check if Pod networking is running you can create a new Pod that has curl and specify the Pod IP on the curl command.
$> kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE dind 2/2 Running 1 6m 192.168.142.19 ip-10-128-6-95 $> kubectl run curl-pod --restart=Never \ --image=hiromasaono/curl -- curl 192.168.142.19 pod "curl-pod" created $> kubectl logs curl-pod <html><body><h1>It works!</h1></body></html> |
Now this container is running as a child process of the dind daemon process and inherits the CPU and memory constraints. When you delete the Pod, this container is killed and never shows up on the host. The EmptyDir used for DinD graph storage is also reclaimed by Kubernetes when Pod is deleted. Figure 2, shows this above behavior pictorially. When the ubuntu container runs the command "docker run httpd:latest", a new container is created inside the dind container. Name conflicts for containers created using docker run --name "something" are only possible within the pod, so we can safely execute this workflow step in parallel.
Figure 2: Docker inside Docker on Kubernetes Pods
Summary
At Applatix, we manage the Graph Storage for the DinD container so that there is reuse between workflows. This allows workflows to run faster by not having to always fetch layers in the Graph Storage. We also make it easy for users run Docker commands from inside containers, so that the user does not have to worry about storage provisioning, reuse and configuration. A unique feature of our system is that it tracks the dollar cost of running containers and allows our users to understand the cost of running their applications in the public cloud. Docker in Docker keeps the container from "escaping" the Pod and allows us to manage its resource utilization and cost. We find that using DinD allows us to use Docker consistently and reliably in our CI/CD system. If you have a question about this blog, feel free to drop us an email at info@applatix.com or @applatix on twitter.
