How to Secure Docker Containers with a Read-Only Filesystem

January 9, 2021

Contents of this article reposted from https://benlobaugh.medium.com/how-to-secure-docker-containers-with-a-read-only-filesystem-b27230f6efb

A compromised website sucks. A compromised website that an attacker can insert code into to manipulate your visitors is even worse! Out-of-the-box, Docker containers provide some security advantages over running directly on the host, however Docker provides additional features to increase security. There is a little known flag in Docker that will convert a container’s files system to read-only.

My father runs an e-commerce site for his dahlia flower hobby business at https://lobaughsdahlias.com. Many years ago, a hacker was able to gain access to the site files through FTP, and then injected a bit of code into the head of the theme. That code was capturing data about customers and sending it to a server in Russia. Luckily the issue was caught before any sensitive data was sent out, but it served to highlight the many layers it takes to secure a website.

Security is like an onion, peel back one layer and another reveals itself beneath. If the filesystem had been read-only, the attacker still would have been able to get in, but they would have been unable to alter anything on the filesystem, thus rendering their end goal moot.

To simplify securing a container, Docker provides a read-only runtime flag that will enforce the filesystem into a read-only state.

In the remainder of this article, I am going to show you how to utilize the read-only flag in Docker. This is not a theory lesson, but rather a walkthrough as I show you exactly the steps I took to deploy https://lobaugh.fish, a live production site, with a read-only filesystem.

Prerequisites

  • Working knowledge of Docker
  • Docker installed
  • Docker-compose installed
  • An existing website to play with

The https://lobaugh.fish site is a constantly evolving site where I can share my passion for fishkeeping. The site is built on top of WordPress and hosted in a VPS at Digital Ocean. The webserver in use is Apache, which will be important to know for later.

There are two ways to add the read-only flag: via the docker cli too, and via docker-compose.

When using the docker cli tool, simply add the `— read-only` flag, and presto, you have a read-only filesystem in the container.

E.G:

docker run — read-only [image-name]

Docker-compose is a wrapper for the cli tool that automatically fills in the flags for you. I prefer to use docker-compose because all the parameters that need to be passed to the container are stored in the docker-compose.yml file and I do not need to remember which I had used last time a container was started. This file can also be shared with other developers to ensure a consistent setup amongst the team, and even deployed directly to a server.

The flag for docker-compose is just as simple as the command line. All that needs to be added to the service is:

read_only: true

Yep, that simple. Restart the service and the container’s filesystem will be read-only.

Uh-oh! Danger, Danger, Danger! We have an issue!

When running `docker-compose up` we encountered the following error and the Apache server died:

lobaugh.fish | Sun Jan 10 04:29:45 2021 (1): Fatal Error Unable to create lock file: Bad file descriptor (9)

As it turns out, some server applications do need to write to the filesystem, for any variety of reasons. In particular, Apache writes a pid file to `/run/apache2` and a lock file in `/run/lock`. Without the ability to write those files, the apache2 service will not start.

To combat the issue of required writes by some applications, Docker has a `tmpfs` flag that can be supplied. The tmpfs flag points to a filesystem location and will allow writes to that single location on the filesystem. It must be kept in mind that these locations will be erased when the container ends. If you need the data to persist between container runs, use a volume.

When using the Docker cli, the flag is `tmpfs`:

docker run — read-only — tmpfs /run/apache2 — tmpfs /run/lock [image]

Similarly, docker-compose contains a `tmpfs` entry under the service:

tmpfs:
- /tmp
- /run/apache2
- /run/lock

Note: I added `/tmp` to enable uploads and other temporary filesystem actions from WordPress.

Re-run the `docker-compose up` command and the site comes alive.

Now for a couple of quick tests…

I am going to get a shell inside the container with:

docker exec -it -u 0 lobaugh.fish /bin/bash

That gets me in with root privileges, which provides the ability to write to any area of the system that a regular user would not have access to, such as `/etc`.

Let’s see if we can write a file to `/etc`. It should not be possible.

root@0c13e6454934:/var/www/html# touch /etc/fstest
touch: cannot touch '/etc/fstest': Read-only file system

Perfect! That is exactly what we wanted to see! The root user cannot write to the filesystem, which means it is truly loaded read-only.

Now lets try in `/tmp`, which we should be able to write to.

root@0c13e6454934:/var/www/html# touch /tmp/fstest
root@0c13e6454934:/var/www/html# ls /tmp
fstest

Perfect again! The file was successfully written.

At this point, we could perform more tests, and I did initially, but I will leave that to you to test more if you would like.

One other thing to consider- mounted volumes will retain the abilities they were mounted with. I.E. if you used the default volume command, that location will be writable. I do recommend that whenever possible the files running your app are built into the container image, and volumes not used in production. It will make your app more secure and scalable. Volumes can also incur a performance hit, and may not work well on all container platforms, such as Kubernetes.

Here is the final docker-compose.yml entry for https://lobaugh.fish:

version: "3.5"

services:
lobaugh.fish:
image: blobaugh/php:7.4-apache
restart: always
read_only: true
tmpfs:
- /tmp
- /run/apache2
- /run/lock

That is all it takes! One simple little trick that may dramatically increase the security of your site.

Leave a Reply

Your email address will not be published. Required fields are marked *