Daemon Memory leak in dll mode accumulating when switching maps

Reported by mail by bmorel.

Reproduction:

system: debian stable amd64
compile with dll mode, RelWithDebInfo
disable overcommit (to get crashes earlier and avoid thrashing on linux)
add "nextmap" in config/map/default.cfg
run daemonded in dll mode and watch it's memory usage, up to "out of memory" signals
the engine never shuts down, and thus a coredump is never created

What should happen:

memory usage should not increase for every map loaded or game started
daemonded should close instead of going inactive for hours, so that a supervision may do it's job of mitigating crash consequences
a coredump should be generated

Example of logs (including system ones) when the situation happens:

dmesg:

2025-02-06T11:37:42,661193+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:42,661224+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162076+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162087+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162089+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162090+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162092+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162093+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162095+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162096+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,791785+00:00 __vm_enough_memory: 792 callbacks suppressed
2025-02-10T10:08:07,791795+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,791809+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,802162+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,802179+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,802198+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,802201+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,802203+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T11:49:45,415094+00:00 __vm_enough_memory: pid: 25671, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-10T11:49:45,552329+00:00 __vm_enough_memory: pid: 25673, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-10T11:50:29,757170+00:00 __vm_enough_memory: pid: 25783, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-10T11:50:29,878847+00:00 __vm_enough_memory: pid: 25785, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-10T11:57:35,237775+00:00 __vm_enough_memory: pid: 26711, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-10T11:57:35,376574+00:00 __vm_enough_memory: pid: 26713, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-11T22:20:16,283992+00:00 __vm_enough_memory: pid: 26546, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-13T10:28:12,981957+00:00 __vm_enough_memory: pid: 13137, comm: daemonded, no enough memory for the allocation
2025-02-13T10:28:12,981980+00:00 __vm_enough_memory: pid: 13137, comm: daemonded, no enough memory for the allocation
2025-02-13T10:28:12,981996+00:00 __vm_enough_memory: pid: 13137, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,931662+00:00 __vm_enough_memory: pid: 27524, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,931675+00:00 __vm_enough_memory: pid: 27524, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,931677+00:00 __vm_enough_memory: pid: 27524, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,931683+00:00 __vm_enough_memory: pid: 27524, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,968643+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,968654+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,978680+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,978696+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,978708+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,978711+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation

stdout under log-rotate:

2025-02-14_07:50:42.39735 ClientDisconnect: 12 [] (XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX) "^9[bot]^4 Edwin M. Lillibridge^*"
2025-02-14_07:50:42.39939 Waiting for the VM thread...
2025-02-14_07:50:42.40470 ----- Server Initialization -----
2025-02-14_07:50:42.40475 Map: metro-b1-2
2025-02-14_07:50:42.42101 [FS] Loading pak '/home/user/www/unvanquished/mods/deadbeef/mod-deadbeef-light_0.55.0-17.dpk'...
2025-02-14_07:50:42.42141 [FS] Loading pak '/home/user/unvanquished/pkg-release/unvanquished_0.55.1.dpk'...
2025-02-14_07:50:42.42243 [FS] Loading pak '/home/user/unvanquished/pkg-release/tex-common_2.4.dpk'...
2025-02-14_07:50:42.42270 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-players_0.55.dpk'...
2025-02-14_07:50:42.42318 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-legacy_0.55.dpk'...
2025-02-14_07:50:42.42385 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-weapons_0.55.dpk'...
2025-02-14_07:50:42.42480 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-buildables_0.55.dpk'...
2025-02-14_07:50:42.42543 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-voices_0.54.dpk'...
2025-02-14_07:50:42.42581 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-soundtrack_0.54.dpk'...
2025-02-14_07:50:42.42599 [FS] Loading pak '/home/user/www/unvanquished/map-metro-b1-2_0+0.dpk'...
2025-02-14_07:50:42.42655 ^3Warn: [FS] Invalid filename 'sound/metro/metro messaj pickpocket.wav' in pak '/home/user/www/unvanquished/map-metro-b1-2_0+0.dpk'
2025-02-14_07:50:42.42724 [FS] Loading pak '/home/user/www/unvanquished/res-tremulous_1.2.3.dpk'...
2025-02-14_07:50:42.42950 [FS] Loading pak '/home/user/www/unvanquished/res-ambient_0.55.dpk'...
2025-02-14_07:50:42.51454 ^3Warn: CM_SetBorderInward: mixed plane sides
2025-02-14_07:50:42.51460 ^3Warn: CM_SetBorderInward: mixed plane sides
2025-02-14_07:50:42.52013 Loading VM module /home/user/unvanquished/sgame-native-dll.so...
2025-02-14_07:50:42.52293 Loaded sgame VM module in 3 msec
2025-02-14_07:50:42.68055 ^3Warn: SGame VM: Out of memory
2025-02-14_07:50:42.69493 [FS] Loading pak '/home/user/www/unvanquished/mods/deadbeef/mod-deadbeef-light_0.55.0-17.dpk'...
2025-02-14_07:50:42.69525 [FS] Loading pak '/home/user/unvanquished/pkg-release/unvanquished_0.55.1.dpk'...
2025-02-14_07:50:42.69619 [FS] Loading pak '/home/user/unvanquished/pkg-release/tex-common_2.4.dpk'...
2025-02-14_07:50:42.69644 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-players_0.55.dpk'...
2025-02-14_07:50:42.69688 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-legacy_0.55.dpk'...
2025-02-14_07:50:42.69752 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-weapons_0.55.dpk'...
2025-02-14_07:50:42.69839 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-buildables_0.55.dpk'...
2025-02-14_07:50:42.69895 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-voices_0.54.dpk'...
2025-02-14_07:50:42.69924 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-soundtrack_0.54.dpk'...
2025-02-14_07:50:42.69939 ----- Server Shutdown -----

Feb 14 '25 16:02 illwieckz

daemonded should close instead of going inactive for hours, so that a supervision may do it's job of mitigating crash consequences

Is the claim that Daemon keeps loading more maps even after the sgame has crashed? If so, that wouldn't happen in normal operation; it's only because of the perverse setup with the nextmap command being sent in the first frame every time.

Feb 14 '25 16:02 slipher

I guess he means there may be a feature to automatically stop the server when there hasn't been any player after a given time, so the service managing the server would just restart it and that would mitigate the issue.

Feb 14 '25 17:02 illwieckz

I confirm the process takes up more and more memory with each map loaded in dll mode. Not using the nextmap setup either, just normal loading of maps through server console.

Aug 01 '25 22:08 VReaperV