Exploring Docker Compose Environment variable behaviour

Docker and all the tools that comes with it can be wonderful to use when deploying and hosting applications. There is a plethora of different and fun things you can do with docker that would make your life much, much easier.

One of the tools which I personally grew quite fond of and use regularly is docker compose. The convenience of being able to use a declarative approach and configuring all of your build options for your application or even your application suit is an absolute bliss. With docker compose you can declaratively define how docker should deploy your application into containers, this is done though easy to read yaml configuration. Not only do you get a better oversight of the configuration for your application you also get the possibility to reuse and source control your application deployment configurations.

Alternatively you would have to use imperative commands in which you would have to define your deployment configuration with a docker run command. This can become particularly arduous and possibly confusing when you want to include many different options. For instance, imagine having to append your docker run command with 5 or 10 different environment variables using --env key=value repetitively, now imagine having to redo this every time you have to deploy your application. Needles to say this can become old very quickly.

Fortunately anything you can configure with imperative docker commands you can configure in a declarative docker compose file saving you time, effort and giving you oversight. Environment variables are a means to provide run-time information to containers. With docker compose you will also be able to define environment variables. Docker compose allows for a number of different ways to define environment variables for containers, each with their own priority of use. The next few paragraphs are focused around the different ways environment variables can be defined in docker compose and their behaviour depending on their priority.

Dockerfile

Starting at the lowest priority for defining environment variables for a container. Environment variables can be defined in the Dockerfile. This can simply be done through the use of the ENV instruction. The environment variables defined by the ENV instruction in the Dockerfile can be used in the resulting image and container. This is demonstrated in the following example.

FROM alpine:3.12

ENV InDockerfileEnv "I come from the dockerfile"

ENTRYPOINT ["sh","-c","env"]

The image, called envdemo, is deployed with the use of docker compose. To do so a yaml file is needed with docker compose syntax.

The following docker compose file is simple in nature and will be fleshed out as we progress. Currently its only purpose is to run the envdemo image.

version: '3.8'
services:
  brt-3d-downloadpage:
    image: envdemo

Now let’s see what happens when we deploy this docker compose configuration of the envdemo image. This can be done through the use of the docker-compose run command.

From the output after deploying the envdemo image, as seen below, we can determine that the InDockerfileEnv variable is available in the docker container and consequently will also be available as an environment variable for any application which may exist in the container.

Output of docker compose run for dockerfile example

Output of docker compose run

Environment File

Docker compose is built to scan the directory from which you run docker compose for a .evn file. “.env” is the default name for the environment variable file. It is possible to configure docker compose to look for a different file to use as an environment variable file, this can be done with the --env-file flag in combination with the docker-compose run command.

Once an environment file has been found docker compose will use this file to seed variables to the docker compose yaml. This is a quick and easy way to substitute multiple environment variables into the docker compose environment.

Environment variables from a .env file take priority over variables defined in the Dockerfile, let’s explore this in the following example.

The same variable used in the Dockerfile has been defined in the .env file.

InDockerfileEnv="I have been editted by the env file"
.env file

Using the same image we can adjust the docker compose file to configure an environment with a substituted variable as its value. Environment variables can be substituted into docker compose by using the following syntax’s:

  • <Key>: ${<Variable Name>} for example: InDockerfileEnv: ${InDockerfileEnv}
  • <Key>=${<Variable Name>} for instance: InDockerfileEnv=${InDockerfileEnv}
  • Or by only defining a key. Environment variables with only a key and no value will be resolved to the variable with the same key that is available to the host machine. For example: InDockerfileEnv

The first option will be used in this article, all three options however are possible.


version: '3.8'
services:
  brt-3d-downloadpage:
    image: envdemo
    environment: 
      InDockerfileEnv: ${InDockerfileEnv}

Deploying the image through docker compose allows it to handle finding and using the environment in the .env file. When inspecting the output below it is possible to see the environment priority at work. The .env file has overwritten the variable InDockerfileEnv previously defined in the Dockerfile.

docker-compose run output for dot env file example

docker-compose run output

Shell Environment Variables

Next on our list is shell or host environment variables. The environment variables defined on the machine from which docker compose is run can also be used to seed variables into the docker container environment.

Simply define a environment variable in your favourite command line interface and substitute that variable into the docker compose yaml file.

Environment variables defined on the host will take priority over variables defined in the .env file and the Dockerfile. To keep to the intentions of this article let’s use the variable we defined in the .env file and Dockerfile subsequently to see the prioritization in action.

Firstly an environment variable will need to be define on the host. In this case the environment variable InDockerfileEnv with a unique value has been defined on the host using the bash command line.

export InDockerfileEnv=”I have been defined in the CLI”
Creating system environment variable

No further changes are needed between the docker compose yaml from the previous chapter and this one.


version: '3.8'
services:
  brt-3d-downloadpage:
    image: envdemo
    environment: 
      InDockerfileEnv: ${InDockerfileEnv}

As we can see from the output and as promised the environment variable on the machine takes precedence over both the Dockerfile and the .env file.

docker-compose run output for shell environment example

docker-compose run output

Compose File

So far we have defined environment variables in Dockerfiles, .env files and by the use of host environment variables. We have also seen how the mechanics of docker compose is designed in such a way that some methods of defining environment variables will take priority over others. With docker compose the variables which is define in the docker compose yaml itself takes the cherry on top, getting the highest priority among the different ways you can present variables to your container environment.

Let’s see this in action with the examples used in this article. First we will have to edit the docker compose yaml to define a value for the InDockerfileEnv variable. The InDockerfileEnv variable has also been defined in the Dockerfile, .env file and as a host environment variable.

version: '3.8'
services:
  brt-3d-downloadpage:
    image: envdemo
    environment: 
      InDockerfileEnv: "I have been defined in the docker compose file"

Docker compose wil now assign the variable that is defined in the compose file instead of using the variables from the Dockerfile, .env file or host environment respectifly. This behaviour can be observed in the output after deploying the container via docker compose.

docker-compose run output for docker compose example

docker-compose run output

To no surprise we can see in the output that the environment variable defined in the docker compose yaml is used in the container environment.

Conclusion

Docker compose allows the user to define environment variables for containers in many different ways. In doing so docker compose also has a set of priorities for which method to use or listen to first. To avoid confusion (and getting unexpected variables piped to your container) it is beneficial to know in which order this priority scheme is setup and how it acts.

The lowest order of precedence are environment variables defined in the Dockerfile. This means that a environment variable with the same key defined in a .env file, shell environment or in a docker compose file will overwrite the value from the Dockerfile.

Dockerfile order of precedence diagram

Dockerfile order of precedence

The second lowest order of precedence are environment variables defined in a .env file. Docker compose allows for the use of .env files to source environment variables to the container environment. The environment variables defined in a .env file will overwrite any variable with the same key in a Dockerfile and will be overwritten by environment variables with the same key from the shell environment and docker compose files.

.env file order of precedence diagram

.env file order of precedence

The second highest order of precedence goes to environment variables defined in the shell environment of the host machine. The environment variables defined on the host and seeded to the container will replace environment variables of the same key in Dockerfiles and .env files, on the other hand environment variables from the host machine shell environment will be replaced by environment variables with the same key in a docker compose file.

Shell environment of precedence diagram

Shell environment of precedence

Lastly the highest order of precedence is reserved for environment variables defined in a docker compose yaml. Environment variables defined in a docker compose yaml will replace all environment variables of the same key in Dockerfiles, .env files and from the shell environment on the host machine.

Docker compose file order of precedence diagram

Docker compose file order of precedence