container fails to start when using readonly filesystem (20-envsubst-on-templates.sh)

[muellerst-hg]

Current Behavior

When mounting the root filesystem with read_only, the entrypoint script 20-envsubst-on-templates.sh fails to start with the following error:

dtrack-frontend-snapshot-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
dtrack-frontend-snapshot-1  | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
dtrack-frontend-snapshot-1  | /docker-entrypoint.d/20-envsubst-on-templates.sh: line 53: can't create /etc/nginx/conf.d/default.conf: Read-only file system

Tested with image: dependencytrack/frontend:snapshot (image id 1b096cd8f5da)

Readonly root filesystem is working fine using image dependencytrack/frontend:4.11.4 (image id c3d304f8c999)

Seems like this was introduced with 4ba0876

Steps to Reproduce

1. Create a file compose-snapshot.yml with the following content:

version: "3"

services:
  dtrack-frontend:
    image: dependencytrack/frontend:snapshot
    environment:
      - "API_BASE_URL=http://localhost:8081"
    ports:
      - "8080:8080"
    read_only: true
    volumes:
      - type: tmpfs
        target: /tmp

2. Start the container:

docker compose -f compose-snapshot.yml up

dtrack-frontend-snapshot-1  | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
dtrack-frontend-snapshot-1  | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
dtrack-frontend-snapshot-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
dtrack-frontend-snapshot-1  | 10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
dtrack-frontend-snapshot-1  | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
dtrack-frontend-snapshot-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
dtrack-frontend-snapshot-1  | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
dtrack-frontend-snapshot-1  | /docker-entrypoint.d/20-envsubst-on-templates.sh: line 53: can't create /etc/nginx/conf.d/default.conf: Read-only file system
dtrack-frontend-snapshot-1 exited with code 1

Expected Behavior

I expect root filesystem with readonly to work as before (in 4.11.4):

1. Create a file compose-4_11_4.yml with the following content:

version: "3"

services:
  dtrack-frontend-4-11-4:
    image: dependencytrack/frontend:4.11.4
    environment:
      - "API_BASE_URL=http://localhost:8181"
    ports:
      - "8180:8180"
    read_only: true
    volumes:
      - type: tmpfs
        target: /tmp

2. Start the container:

docker compose -f compose-4_11_4.yml up

dtrack-frontend-4-11-4-1    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
dtrack-frontend-4-11-4-1    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
dtrack-frontend-4-11-4-1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
dtrack-frontend-4-11-4-1    | 10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
dtrack-frontend-4-11-4-1    | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
dtrack-frontend-4-11-4-1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
dtrack-frontend-4-11-4-1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-oidc-configuration.sh
dtrack-frontend-4-11-4-1    | 30-oidc-configuration.sh: info: can not modify config.json - ENV configuration will be ignored
dtrack-frontend-4-11-4-1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
dtrack-frontend-4-11-4-1    | /docker-entrypoint.sh: Configuration complete; ready for start up
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: using the "epoll" event method
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: nginx/1.25.5
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: built by gcc 13.2.1 20231014 (Alpine 13.2.1_git20231014)
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: OS: Linux 5.15.133.1-microsoft-standard-WSL2
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1024:1048576
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker processes
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 24
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 25
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 26
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 27
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 28
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 29
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 30
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 31
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 32
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 33
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 34
dtrack-frontend-4-11-4-1    | 2024/07/08 09:04:32 [notice] 1#1: start worker process 35

Dependency-Track Frontend Version

4.12.0-SNAPSHOT

Browser

Mozilla Firefox

Browser Version

127.0.2

Operating System

Linux

Checklist

• I have read and understand the contributing guidelines
• I have checked the existing issues for whether this defect was already reported

[nscuro]

@Squixx Any idea how we can make your changes work with this requirement?

[Squixx]

I'm on vacation right now, so can't check. But seems like it has to with the way nginx converts the template to actual config on boot. I'll take a look when I get back (~17th July)

nginx/docker-nginx-unprivileged#88 could point that it's an issue with the inner workings of nginx, you could ensure the entire conf file is outside of the container like https://gist.github.com/monosoul/e462b03cba5542bfc52f135417a8209d in the case of read_only: true to ensure it works both ways.

Which makes me think we could solve this by only adding specific docs about read only.

But again I'm on mobile so have no way to test as of right now.

[Squixx]

muellerst-hg does it work for you with the above gist?

[muellerst-hg]

@Squixx even though 20-envsubst-on-templates.sh entrypoint does its job when applying this gist. The nginx config which is used is a bit different. I am not sure if this an issue. curl http://127.0.0.1:8080 returns the expected index.html file.

# config fetched from container process using nginx -T
diff -wu /tmp/rw.conf /tmp/ro.conf

--- /tmp/rw.conf        2024-07-23 12:55:26.722236147 +0200
+++ /tmp/ro.conf        2024-07-23 12:55:24.402242094 +0200
@@ -140,7 +140,10 @@
     video/x-msvideo                                  avi;
 }

-# configuration file /etc/nginx/conf.d/default.conf:
+# configuration file /etc/nginx/conf.d/10-include-tmp.conf:
+include /var/nginx.conf.d/*.conf;
+
+# configuration file /var/nginx.conf.d/default.conf:
 server {
   listen      8080;
   server_name _;
@@ -160,3 +163,49 @@
     root /usr/share/nginx/html;
   }
 }
+# configuration file /etc/nginx/conf.d/default.conf:
+server {
+    listen       8080;
+    server_name  localhost;
+
+    #access_log  /var/log/nginx/host.access.log  main;
+
+    location / {
+        root   /usr/share/nginx/html;
+        index  index.html index.htm;
+    }
+
+    #error_page  404              /404.html;
+
+    # redirect server error pages to the static page /50x.html
+    #
+    error_page   500 502 503 504  /50x.html;
+    location = /50x.html {
+        root   /usr/share/nginx/html;
+    }
+
+    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
+    #
+    #location ~ \.php$ {
+    #    proxy_pass   http://127.0.0.1;
+    #}
+
+    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
+    #
+    #location ~ \.php$ {
+    #    root           html;
+    #    fastcgi_pass   127.0.0.1:9000;
+    #    fastcgi_index  index.php;
+    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
+    #    include        fastcgi_params;
+    #}
+
+    # deny access to .htaccess files, if Apache's document root
+    # concurs with nginx's one
+    #
+    #location ~ /\.ht {
+    #    deny  all;
+    #}
+}
+
+

Here's the setup I have tested:

$ echo "include /var/nginx.conf.d/*.conf;" > include-tmp.conf
$ cat > compose.yml << EOF
version: "3.9"

services:
  dtrack-frontend-snapshot:
    image: dependencytrack/frontend:snapshot
    environment:
      - BASE_PATH=http://127.0.0.1:8080
      - NGINX_ENVSUBST_OUTPUT_DIR=/var/nginx.conf.d
    ports:
      - "8080:8080"
    read_only: true
    volumes:
      - type: tmpfs
        target: /tmp
      - ./include-tmp.conf:/etc/nginx/conf.d/10-include-tmp.conf:ro
    tmpfs:
      - /var/nginx.conf.d:rw,noexec,nosuid,nodev
 EOF

IMHO this is a non-intuitive workaround which requires knowledge about implementation details in order to understand what one is doing.
Nevertheless I agree, this is an upstream issue of docker-nginx-unprivileged because they do not cleanly separate default configs/templates and rendered config. An nginx container image which serves static files should by default run unpriv. with a readonly filesystem. The upstream entrypoint scripts are failing here.

[Squixx]

Yeah i agree, this requires some docker / nginx knowledge to get to run properly... but currently has no alternative unfortunately. @nscuro how do you want to approach this? could we write some docs for this usecase?

[troy256]

We are also getting this error on Helm chart deployment for the frontend when upgrading to 4.12.1. Deployment fails and if we look at the pod logs we see:

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: ipv6 not available
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.d/20-envsubst-on-templates.sh: line 53: can't create /etc/nginx/conf.d/default.conf: Read-only file system

[nscuro]

@troy256 The Helm chart also says that readOnlyRootFilesystem is not supported for the frontend, hence defaulting it to false: https://github.com/DependencyTrack/helm-charts/blob/24dacda40749b564f6a8bb54a2976ac5c8f3bac5/charts/dependency-track/values.yaml#L157-L158

[troy256]

Thanks @nscuro, unfortunately it is still failing with same error. Looks like it is mounting a config map to /etc/nginx/conf.d/default.conf. The config map is not read-only, so I don't think that's the issue. Is there anything else you can suggest?

  name: dependency-track-frontend
  ports:
    - containerPort: 8080
      name: http
      protocol: TCP
  readinessProbe:
    failureThreshold: 3
    httpGet:
      path: /
      port: http
      scheme: HTTP
    initialDelaySeconds: 60
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 2
  resources:
    limits:
      cpu: '1'
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: false
    runAsUser: 101
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
    - mountPath: /tmp
      name: tmp
    - mountPath: /etc/nginx/conf.d/
      name: default-conf
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-j9lvt
      readOnly: true

[Squixx]

The way this works is that after boot the config in conf.d is overwritten with the template populated by env vars, are you sure the user can actually write to that dir?

[troy256]

I found the issue. We were mounting or own custom Nginx conf config map to the conf.d directory, which interfered with the run of 20-envsubst-on-templates.sh script. I moved the change elsewhere and it now deploys. Thanks for your helping pointing me in the right direction.