I containerize… Why I containerize you might ask? Because of reasons:
As it is quite common in modern world, containerizing comes with a price to pay. And the price is called: abstraction. I remember someone said that
Every abstraction layer solves one problem, by introducing ten different ones.
Or something quite close to that. Of course, that is an oversimplication (still being true), but I wouldn’t change the docker abstraction for now due to the benefits it gives us. This post, however, is not about the benefits but about a specific issue, which I was not aware of for quite a long time. It’s related to running apps, commands inside containers - and more closely - about stopping them.
I have created a sample project for this post, which can be found at https://github.com/gmaslowski/docker-shell-vs-exec. This project has a simple Spring based app and some Docker descriptor files, for building images and setting up container with docker-compose (please note, that described issues should correspond to any form of starting a docker container).
The simple snippet project focuses on two ways of executing commands inside a docker container:
ENTRYPOINT java XX:+ExitOnOutOfMemoryError Djava.security.egd=file:/dev/./urandom -jar /app.jar
ENTRYPOINT ["java", "-XX:+ExitOnOutOfMemoryError", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/app.jar"]
Both will have the same effect, at least when it comes to running containers on top of those images. If we build the application and the docker images, as specified in the
README, like this:
We would create two images:
dsve:shell- with command executing the Spring app in shell form
dsve:exec- with command executing the Spring app in exec form
By running the following script, we would deploy and run our containers with docker-compose:
The actual docker process runtime should look similar to that:
COMMAND section one can already see both ways (shell and exec) of executing the java app inside the container. Let us have a quick look into the containers to list the processes.
The difference is easy to spot, the same java command, one started with
/bin/sh and the other without it.
Well, it says many things, and it also describes the difference between shell and exec form. In my opinion, such “details” are often in places which are easy to overlook, and if you’re as impatient and careless :) as I am - you probably will overlook them as well. Careful reading of the docker documentation is strongly advised - https://docs.docker.com/engine/reference/builder/#entrypoint.
In a shell form, all environment variables will be evaluated as the actual provided command will be run within a shell by prepending
/bin/sh -c before it, which can also be observed in the snippet from previous section. In the exec form, however, there is no shell processing involved and the executable is being called directly. So please make sure that your env vars are being substituted before or that the executable you invoke does it.
I don’t want to focus on explaining the differences in much detail.
RUNis being used when building the image,
CMDserve the purpose of starting the actuall container and parameterize it when needed.
In this article http://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/ you can find a really great explanation of the difference and it really wouldn’t make sense to duplicate the content. Additionally, the difference has also been explained quite well in the Docker documentation in the section understand-how-cmd-and-entrypoint-interact.
But here we can get into troubles. If we try to stop a container with the shell form
there’s a significant time, which we might notice before the container stops. That’s because we extended the
stop_grace_period from the default 10s to 30s - mainly for the presentation purposes. But if you look closely into the logs, you won’t find any information from the Spring application notifying that the system sent a
SIGTERM signal. That’s due to the fact that this signal was send actually to the shell, which doesn’t pass any signals to the process it started. It is described in Docker documentation, however it is quite easy to miss that - I know I was myself not aware of those implications for a long time. And hence, after the
stop_grace_period passes, docker daemon sends a
SIGKILL signal causing the container to stop, forcefully.
On the other hand, the exec form stops almost immediately
and in the logs we cas spot that Sring based application handled the
SIGTERM command allowing to close all obtained resources:
And that’s the crucial part. In best case scenario, the problems will only cause longer waits for the container to stop. But in worst case scenario, if the application doesn’t free any used resources (like database connections, locks etc.)… yeah, you can imagine the consequences.
I spotted similar issues while working with k8s as the container orchestrator. And this should be fully understandable. The container in the pod tries to handle the
SIGTERM signal, and if it doesn’t, the orchestrator will
I’m curious about, What other things are commonly overlooked while using docker? If you have an example, just comment or send an email.