Error 503, PHP

[Kemel91]

For some reason, error 503 appears periodically, and after that it does not go away until the next restart of the container, what could be the problem? Logs:

2024/06/13 04:30:52 [info] 434#434 "laravel" application started
free(): invalid pointer
2024/06/13 04:30:52 [alert] 84#84 app process 412 exited on signal 6
munmap_chunk(): invalid pointer
2024/06/13 04:30:52 [alert] 84#84 app process 427 exited on signal 6
free(): invalid pointer
2024/06/13 04:30:52 [alert] 84#84 app process 370 exited on signal 6
2024/06/13 04:30:54 [info] 438#438 "laravel" application started
free(): invalid pointer
2024/06/13 04:30:55 [alert] 84#84 app process 414 exited on signal 6
2024/06/13 04:30:57 [info] 440#440 "laravel" application started
2024/06/13 04:30:57 [info] 441#441 "laravel" application started
2024/06/13 04:30:57 [alert] 84#84 app process 432 exited on signal 11
2024/06/13 04:30:57 [info] 442#442 "laravel" application started
2024/06/13 04:30:57 [alert] 84#84 app process 406 exited on signal 11
2024/06/13 04:30:57 [alert] 84#84 app process 438 exited on signal 11
2024/06/13 04:31:27 [info] 443#443 "laravel" application started
2024/06/13 04:31:27 [alert] 84#84 app process 426 exited on signal 11

There is more than 100 GB of free RAM on the server, there are no restrictions in the container. Unit config:

{
  "listeners": {
    "*:8082": {
      "pass": "routes"
    }
  },

  "routes": [
    {
      "match": {
        "uri": "!/index.php"
      },
      "action": {
        "share": "/var/www/html/public$uri",
        "fallback": {
          "pass": "applications/laravel"
        }
      }
    }
  ],

  "applications": {
    "laravel": {
      "type": "php",
      "root": "/var/www/html/public/",
      "script": "index.php",
      "processes": {
        "max": 500,
        "spare": 20,
        "idle_timeout": 20
      }
    }
  }
}

PHP 8.3 unit version: 1.32.1 PHP ini memory limit increased to 10 GB.

[ac000]

Are you getting coredumps? (might not be enabled)

I would really like to see a backtrace from one

$ gdb /path/to/unitd /path/to/coredump
(gdb) bt full

Or if your running systemd then it may be intercepting the core dumps,

$ coredumpctl

If that shows a list, good stuff

$ coredumpctl gdb

Will launch gdb on the most recent coredump...

[Kemel91]

Are you getting coredumps? (might not be enabled)

I would really like to see a backtrace from one

$ gdb /path/to/unitd /path/to/coredump
(gdb) bt full

Or if your running systemd then it may be intercepting the core dumps,

$ coredumpctl

If that shows a list, good stuff

$ coredumpctl gdb

Will launch gdb on the most recent coredump...

How I can enable coredumps in unitd processes? I can't find anything in doc this info, about enabling coredumps in unitd. How i can do this in docker container?

[ac000]

Not really a Unit specific thing...

In the container what does

$ cat /proc/sys/kernel/core_pattern

return?

When unit is running what does

$ cat /proc/`pidof -s unitd`/limits

show?

[Kemel91]

Not really a Unit specific thing...

In the container what does

$ cat /proc/sys/kernel/core_pattern

return?

When unit is running what does

$ cat /proc/`pidof -s unitd`/limits

show?

cat /proc/sys/kernel/core_pattern /tmp/core

cat /proc/pidof -s unitd/limits:|

Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 1048576 1048576 processes Max open files 1048576 1048576 files Max locked memory 8388608 8388608 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 1030780 1030780 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us

[ac000]

OK, so according to that you should be betting core dumps in /tmp/ named core or possibly core.pid (depending on the setting of /proc/sys/kernel/core_uses_pid). See anything there?

[ac000]

If you find a core file. You can do

$ gdb /path/to/uniid /path/to/corefile
(gdb) bt full
"Might need to press 'c' to get the full thing"

Alternatively if you are able to create a minimal reproducer...

[Kemel91]

If you find a core file. You can do

$ gdb /path/to/uniid /path/to/corefile
(gdb) bt full
"Might need to press 'c' to get the full thing"

Alternatively if you are able to create a minimal reproducer...

Thanks for the answer. Another question, are dumps uploaded only when a unitd crashes or can they be uploaded when some process crashes?

[ac000]

Coredumps are a standard UNIX (and other OS's) feature and will generally be generated for any processes that crash, or when receiving a signal whose default action is to coredump, (SIGABRT, SIGBUS, SIGFPE, SIGILL, SIGQUIT, SIGSEGV, SIGSYS, SIGTRAP, SIGXCPU, SIGXFSZ).

You can also force a running process to coredump via gcore(1)

[Kemel91]

The problem was in the JIT PHP setup