docker run with small '-m' memory hangs up and times out

Open grahamwhaley opened this issue 8 years ago • 0 comments

Description of problem

Running a container with too little memory seems to hang up the QEMU start, and eventually docker (I believe) times out and errors. I suspect we can detect the hang up earlier maybe in our runtime and error out more nicely - something to look into sometime. The error seems non-fatal (does not hang up docker permanently), so not a priority item.

Example: docker run --rm -ti --runtime=cc-runtime -m 32M ubuntu date hangs. Works with -m 64M

Expected result

Ideally I'd like us to error out earlier (or, well, work :-) ). When hung up I can see <defunct> qemu processes in a ps - which makes me think maybe we can detect this in the runtime and exit early.

Actual result

It looks like docker times out the operation after 2 minutes:

somebody@somemachine:~$ time docker run --rm -ti --runtime=cc-runtime -m 32M ubuntu date
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
docker: Error response from daemon: containerd: container did not start before the specified timeout.

real    2m0.289s
user    0m0.028s
sys     0m0.016s

Meta details

Running cc-collect-data.sh version 3.0.1 (commit dd85a0a217aaad11caef2b3f06dda20c779aa349) at 2017-11-08.16:02:30.344059910.

Runtime is /usr/local/bin/cc-runtime.

`cc-env`

Output of "/usr/local/bin/cc-runtime cc-env":

[Meta]
  Version = "1.0.5"

[Runtime]
  [Runtime.Version]
    Semver = "3.0.6"
    Commit = "077bd255175e1cc39d9a49b8aff3f9bb496acbbf"
    OCI = "1.0.0-dev"
  [Runtime.Config]
    Path = "/usr/share/defaults/clear-containers/configuration.toml"
    GlobalLogPath = ""

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.7.1(2.7.1+git.d4a337fe91-9.cc), Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"

[Image]
  Path = "/usr/share/clear-containers/clear-18400-containers.img"

[Kernel]
  Path = "/usr/share/clear-containers/vmlinuz-4.9.54-78.container"
  Parameters = ""

[Proxy]
  Type = "ccProxy"
  Version = "Version: 3.0.6-2-g32cad21"
  URL = "unix:///var/run/clear-containers/proxy.sock"

[Shim]
  Type = "ccShim"
  Version = "shim version: 3.0.6 (commit: 11b5eef07b4313010fe07f45bede2e81932990af)"
  Path = "/usr/libexec/clear-containers/cc-shim"

[Agent]
  Type = "hyperstart"
  Version = "<<unknown>>"

[Host]
  Kernel = "4.4.0-92-generic"
  CCCapable = true
  [Host.Distro]
    Name = "Ubuntu"
    Version = "16.04"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Core(TM) i5-6260U CPU @ 1.80GHz"

Runtime config files

Runtime default config files

/etc/clear-containers/configuration.toml
/usr/share/defaults/clear-containers/configuration.toml

Runtime config file contents

Config file /etc/clear-containers/configuration.toml not found Output of "cat "/usr/share/defaults/clear-containers/configuration.toml"":

# XXX: Warning: this file is auto-generated from file "config/configuration.toml.in".

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/clear-containers/vmlinuz.container"
image = "/usr/share/clear-containers/clear-containers.img"
machine_type = "pc"
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc
kernel_params = ""

# Default number of vCPUs per POD/VM:
# unspecified or 0 --> will be set to 1
# < 0              --> will be set to the actual number of physical cores
# > 0 <= 255       --> will be set to the specified number
# > 255            --> will be set to 255
default_vcpus = -1

# Default memory size in MiB for POD/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048
disable_block_device_use = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# Debug changes the default hypervisor and kernel parameters to
# enable debug output where available.
# Default false
# these logs can be obtained in the cc-proxy logs  when the
# proxy is set to run in debug mode
# /usr/libexec/clear-containers/cc-proxy -log debug
# or by stopping the cc-proxy service and running the cc-proxy
# explicitly using the same command line
#
#enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true

[proxy.cc]
url = "unix:///var/run/clear-containers/proxy.sock"

[shim.cc]
path = "/usr/libexec/clear-containers/cc-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[runtime]
## Uncomment to enable the global logging to the default path.
#global_log_path = "/var/lib/clear-containers/runtime/runtime.log"

# If enabled, the runtime will log additional debug messages to the global
# log, assuming that is also enabled.
# (default: disabled)
#enable_debug = true

Runtime logs

Global log not enabled.

Container manager details

Have docker

Docker

Output of "docker info":

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 32
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: cc-runtime runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-92-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.31GiB
Name: bignuc
ID: ROOX:AUAH:KW7J:BADM:E3PW:ORDN:KL42:FKXQ:77LD:VGKO:ZFY4:OG64
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 18
 Goroutines: 26
 System Time: 2017-11-08T16:02:30.380038569Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

No kubectl

Packages

ii  cc-proxy                           3.0.3+git.161a82a-8                        amd64       
ii  cc-runtime                         3.0.3+git.92c25e5-8                        amd64       
ii  cc-runtime-bin                     3.0.3+git.92c25e5-8                        amd64       
ii  cc-runtime-config                  3.0.3+git.92c25e5-8                        amd64       
ii  cc-shim                            3.0.3+git.4549a4d-8                        amd64       
ii  clear-containers-image             18400-36                                   amd64        Clear containers image
ii  linux-container                    4.9.54-78                                  amd64        linux kernel optimised for container-like workloads.
ii  qemu-lite                          2.7.1+git.d4a337fe91-9                     amd64        linux kernel optimised for container-like workloads.
ii  qemu-system-x86                    1:2.5+dfsg-5ubuntu10.16                    amd64        QEMU full system emulation binaries (x86)

Nov 08 '17 16:11 grahamwhaley