Docker container start up very slowly with 180G nfs volume

[xiaohui-hiwintech]

We added a 180G nfs (aws efs) volume to my docker-compose recently, and it took about 7 minutes to start up, if we remove the nfs volume, it can start up in 1 minute, and we did some testing, the larger the nfs, the slower the startup.

Does anyone know why? Or is there some way to see what was the docker doing? I out put the docker logs, but only successful messages.

2022-04-13T06:28:46.662408516Z Hosting environment: Production
2022-04-13T06:28:46.662443259Z Content root path: /app
2022-04-13T06:28:46.662536439Z Now listening on: http://[::]:8002
2022-04-13T06:28:46.662550521Z Application started. Press Ctrl+C to shut down.

My docker version: 20.10.14. My docker compose version: 1.29.2. Below is my docker-compose.yml

version: '2'

services:      
    app:
        image: xxxx
        restart: always
        environment:
            - ASPNETCORE_ENVIRONMENT=Production
        ports:
            - "8002:8002"
        volumes:
            - "./BinaryObjects/Static:/app/App_Data/BinaryObjects/Static"
            - "./BinaryObjects/Temp:/app/App_Data/BinaryObjects/Temp"

/BinaryObjects is the root path of nfs. My docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 18
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.13.0-1021-aws
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 30.57GiB
 Name: ip-172-31-60-97
 ID: UJQ7:MKNL:OHW6:B222:D2YU:GOD7:2J6A:7JZB:PDAR:Z57Z:Y7MQ:VW4F
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: doyoukare
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[xiaohui-hiwintech]

Update: it is not related to the docker, it is related to dotnet application, if I move the NFS folder out of application folder, the docker container can start up normally. May be the dotnet runtime scan the application folder for some purposes, I don't know.

[psavva]

I have found the same problem.

I can also confirm that a 400GB app_data folder caused a huge startup delay of over 30 minutes