[BUG] Segmentation Fault in ipc_comp_free during IPC3 Fuzz Testing
Describe the bug
A segmentation fault occurs in the ipc_comp_free function during IPC3 fuzz testing. The issue is caused by a read memory access to a null pointer. This was detected using AddressSanitizer.
To Reproduce
- Run fuzz testing with the provided corpus.
- Observe the segmentation fault in the logs.
Reproduction Rate The issue occurs consistently during fuzz testing.
Expected behavior The fuzz testing should complete without causing a segmentation fault.
Impact This issue is a showstopper as it prevents the completion of fuzz testing and affects the stability of the IPC3 configuration.
Environment
- Branch name and commit hash.
- SOF: 7650d0ac21f2a51e7dfb63b785142e7bb5a0921a
- Name of the platform(s) on which the bug is observed.
- Platform: native_sim
Screenshots or console output
================== Job 1 exited with exit code 1 ============ INFO: Running with entropic power schedule (0xFF, 100). INFO: 11411 files found in ./ipc3_fuzz_corpus
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 11411 min: 1b max: 510b total: 1108557b rss: 49Mb
AddressSanitizer:DEADLYSIGNAL
=================================================================
==305587==ERROR: AddressSanitizer: SEGV on unknown address 0x00000000 (pc 0x08186fb0 bp 0xe63fae88 sp 0xe63fae50 T7)
==305587==The signal is caused by a READ memory access.
==305587==Hint: address points to the zero page.
#0 0x8186fb0 in ipc_comp_free /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c
#1 0x81783d2 in ipc_glb_tplg_free /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc3/handler.c:1394:8
#2 0x8177637 in ipc_cmd /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc3/handler.c:1657:9
#3 0x818b879 in ipc_platform_do_cmd /home/tmleman/work/repos/thesofproject/sof/src/platform/posix/ipc.c:160:2
#4 0x8185b8c in ipc_do_cmd /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-common.c:361:9
#5 0x81a934b in task_run /home/tmleman/work/repos/thesofproject/sof/zephyr/include/rtos/task.h:94:9
#6 0x81a8ebe in edf_work_handler /home/tmleman/work/repos/thesofproject/sof/zephyr/edf_schedule.c:32:16
#7 0x8260cbf in work_queue_main /home/tmleman/work/repos/thesofproject/zephyr/kernel/work.c:684:3
#8 0x815aa92 in z_thread_entry /home/tmleman/work/repos/thesofproject/zephyr/lib/os/thread_entry.c:48:2
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c in ipc_comp_free
Thread T7 created by T3 here:
#0 0x810602c in __interceptor_pthread_create (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x810602c) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
#1 0x82716e2 in nct_new_thread /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/nct.c:476:2
#2 0x816e397 in posix_new_thread /home/tmleman/work/repos/thesofproject/zephyr/arch/posix/core/posix_core_nsi.c:48:9
#3 0x816ddaf in arch_new_thread /home/tmleman/work/repos/thesofproject/zephyr/arch/posix/core/thread.c:55:30
#4 0x8262e7c in z_setup_new_thread /home/tmleman/work/repos/thesofproject/zephyr/kernel/thread.c:563:2
#5 0x8263293 in z_impl_k_thread_create /home/tmleman/work/repos/thesofproject/zephyr/kernel/thread.c:658:2
#6 0x8260777 in k_thread_create /home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/include/generated/zephyr/syscalls/kernel.h:85:9
#7 0x8260777 in k_work_queue_start /home/tmleman/work/repos/thesofproject/zephyr/kernel/work.c:752:8
#8 0x81a8c67 in scheduler_init_edf /home/tmleman/work/repos/thesofproject/sof/zephyr/edf_schedule.c:107:2
#9 0x818d748 in platform_init /home/tmleman/work/repos/thesofproject/sof/src/platform/posix/posix.c:77:2
#10 0x8177103 in primary_core_init /home/tmleman/work/repos/thesofproject/sof/src/init/init.c:314:6
#11 0x825cea9 in z_sys_init_run_level /home/tmleman/work/repos/thesofproject/zephyr/kernel/init.c:371:13
#12 0x825cea9 in bg_thread_main /home/tmleman/work/repos/thesofproject/zephyr/kernel/init.c:519:2
#13 0x815aa92 in z_thread_entry /home/tmleman/work/repos/thesofproject/zephyr/lib/os/thread_entry.c:48:2
Thread T3 created by T2 here:
#0 0x810602c in __interceptor_pthread_create (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x810602c) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
#1 0x82716e2 in nct_new_thread /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/nct.c:476:2
#2 0x816e397 in posix_new_thread /home/tmleman/work/repos/thesofproject/zephyr/arch/posix/core/posix_core_nsi.c:48:9
#3 0x816ddaf in arch_new_thread /home/tmleman/work/repos/thesofproject/zephyr/arch/posix/core/thread.c:55:30
#4 0x8262e7c in z_setup_new_thread /home/tmleman/work/repos/thesofproject/zephyr/kernel/thread.c:563:2
#5 0x825cd01 in prepare_multithreading /home/tmleman/work/repos/thesofproject/zephyr/kernel/init.c:672:14
#6 0x825c76d in z_cstart /home/tmleman/work/repos/thesofproject/zephyr/kernel/init.c:795:24
#7 0x8128e79 in __asan::AsanThread::ThreadStart(unsigned long long) (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x8128e79) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
Thread T2 created by T0 here:
#0 0x810602c in __interceptor_pthread_create (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x810602c) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
#1 0x82703ce in nce_boot_cpu /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/nce.c:209:2
#2 0x817526a in posix_boot_cpu /home/tmleman/work/repos/thesofproject/zephyr/soc/native/inf_clock/soc.c:126:2
#3 0x827e336 in nsif_cpu0_boot /home/tmleman/work/repos/thesofproject/zephyr/boards/native/native_sim/nsi_if.c:24:2
#4 0x8273942 in nsif_cpun_boot /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/nsi_cpun_if.c:16:1
#5 0x827319b in nsi_cpu_auto_boot /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/nsi_cpu_ctrl.c:38:4
#6 0x826fb46 in nsi_init /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/main.c:87:2
#7 0x827e4cc in LLVMFuzzerTestOneInput /home/tmleman/work/repos/thesofproject/sof/src/platform/posix/fuzz.c:30:3
#8 0x8083da3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned int) (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x8083da3) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
#9 0x8084d3d in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile> >&) (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x8084d3d) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
#10 0x8085261 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile> >&) (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x8085261) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
#11 0x807325c in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned int)) (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x807325c) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
#12 0x809cc67 in main (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x809cc67) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
#13 0xf0c21518 (/lib/i386-linux-gnu/libc.so.6+0x21518) (BuildId: 559bcd138c06a98975421c680491f5666816aef2)
==305587==ABORTING
MS: 1 ChangeBit-; base unit: 3242021c62b0183ca5022cf0394a277dd444e5ee
0x4,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x17,0x0,0x0,0x0,0xff,0xd9,0x0,0x3,0x30,0x6,0xf6,0x0,0x2c,0x8,0x60,0x8,0x0,0x0,0x0,0x2d,0x2d,0x3c,0x0,0x10,0x30,0x0,0x0,0x29,0x87,0x0,0x0,0x1,0x30,0xff,0xff,0x3,0x0,0x6,0x0,0x0,0x8,0x0,0xfc,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x17,0x0,0x0,0xff,0xfe,0x0,0x0,0x3,0x30,0x3,0xf6,0x8,0x60,0x0,0x0,0x0,0x23,0x1,0x0,0x0,0x0,0x60,0x0,0xf6,0x0,0x0,0x1,0x30,0xff,0xff,0x3,0x0,0x41,0x0,0x4,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x17,0x0,0x0,0x0,0xff,0xd9,0x0,0x3,0x30,0x6,0x0,0x0,0x1,0x30,0x60,0x8,0x0,0x0,0x0,0x2d,0x2d,0x3a,0x0,0x10,0x30,0x0,0x0,0x29,0x87,0x0,0x0,0x1,0x30,0xff,0xff,0x3,0x0,0x1,0xf6,0x8,0x0,0xfc,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x17,0x0,0x0,0xff,0xfe,0x0,0x0,0x2,0x30,0x3,0x2f,0x8,0x60,0x0,0x0,0x0,0x23,0x1,0x0,0x0,0x0,0x60,0x0,0xf6,0x2,0x0,0x0,0x0,0x87,0x0,0x0,0x1,0x30,0xff,0xff,0x3,0x0,0x41,0x0,0x0,0x3,0x0,0x0,0x11,0x0, \004\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\027\000\000\000\377\331\000\0030\006\366\000,\010`\010\000\000\000--<\000\0200\000\000)\207\000\000\0010\377\377\003\000\006\000\000\010\000\374\000\000\000\000\000\000\000\027\000\000\377\376\000\000\0030\003\366\010`\000\000\000#\001\000\000\000`\000\366\000\000\0010\377\377\003\000A\000\004\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\027\000\000\000\377\331\000\0030\006\000\000\0010`\010\000\000\000--:\000\0200\000\000)\207\000\000\0010\377\377\003\000\001\366\010\000\374\000\000\000\000\000\000\000\027\000\000\377\376\000\000\0020\003/\010`\000\000\000#\001\000\000\000`\000\366\002\000\000\000\207\000\000\0010\377\377\003\000A\000\000\003\000\000\021\000
artifact_prefix='./'; Test unit written to ./crash-1a8c95ba333361f3d7fd7465442d2d8eeab398df
Base64: BAAAAAAAAAAAAAAAAAAAABcAAAD/2QADMAb2ACwIYAgAAAAtLTwAEDAAACmHAAABMP//AwAGAAAIAPwAAAAAAAAAFwAA//4AAAMwA/YIYAAAACMBAAAAYAD2AAABMP//AwBBAAQAAAAAAAAAAAAAAAAAAAAXAAAA/9kAAzAGAAABMGAIAAAALS06ABAwAAAphwAAATD//wMAAfYIAPwAAAAAAAAAFwAA//4AAAIwAy8IYAAAACMBAAAAYAD2AgAAAIcAAAEw//8DAEEAAAMAABEA
$ hexdump crash-1a8c95ba333361f3d7fd7465442d2d8eeab398df
0000000 0004 0000 0000 0000 0000 0000 0000 0000
0000010 0017 0000 d9ff 0300 0630 00f6 082c 0860
0000020 0000 2d00 3c2d 1000 0030 2900 0087 0100
0000030 ff30 03ff 0600 0000 0008 00fc 0000 0000
0000040 0000 0017 ff00 00fe 0300 0330 08f6 0060
0000050 0000 0123 0000 6000 f600 0000 3001 ffff
0000060 0003 0041 0004 0000 0000 0000 0000 0000
0000070 0000 0000 0017 0000 d9ff 0300 0630 0000
0000080 3001 0860 0000 2d00 3a2d 1000 0030 2900
0000090 0087 0100 ff30 03ff 0100 08f6 fc00 0000
00000a0 0000 0000 1700 0000 feff 0000 3002 2f03
00000b0 6008 0000 2300 0001 0000 0060 02f6 0000
00000c0 8700 0000 3001 ffff 0003 0041 0300 0000
00000d0 0011
00000d2
@dbaluta @bhiregoudar @andyross fyi as impact with your configs
Is this a regression? Odd that it's suddenly popping up.
Also: can you attach that corpus file as a binary (or at least something like a base64 string that I can programmatically convert instead of writing software to do the endianness flipping on all the shorts)? And how was the fuzzer built, and with what config?
Is this a regression?
The issue might have been hidden due to the fuzzer used in CI having limited code coverage. I am not certain about the coverage achieved by SOF under OSS-Fuzz, but it is possible that the current fuzzer configuration in CI is not exercising all code paths effectively.
can you attach that corpus file
Here is the corpus file encoded as a base64 string:
BAAAAAAAAAAAAAAAAAAAABcAAAD/2QADMAb2ACwIYAgAAAAtLTwAEDAAACmHAAABMP//AwAGAAAIAPwAAAAAAAAAFwAA//4AAAMwA/YIYAAAACMBAAAAYAD2AAABMP//AwBBAAQAAAAAAAAAAAAAAAAAAAAXAAAA/9kAAzAGAAABMGAIAAAALS06ABAwAAAphwAAATD//wMAAfYIAPwAAAAAAAAAFwAA//4AAAIwAy8IYAAAACMBAAAAYAD2AgAAAIcAAAEw//8DAEEAAAMAABEA
And how was the fuzzer built, and with what config?
The fuzzer was built using the script provided in the SOF repository (sof/scripts/fuzz.sh).
System and Tool Versions:
- Operating System: Ubuntu 22.04
- Kernel Version: 6.5.0-25-generic
- CMake Version: 3.22.1
- Python Version: 3.10.12 (used within a virtual environment)
- West Version: 1.2.0
- Board: native_sim
- Toolchain: LLVM (Clang/ld)
- DTC Version: 1.6.1
- Clang Version: 14.0.0
- Clang++ Version: 14.0.0
- GNU LD Version: 2.38
- SOF Commit: 7650d0ac21f2a51e7dfb63b785142e7bb5a0921a
@tmleman @andyross @thesofproject/maintainers any input on correct priority for this bug? Should this block v2.12 release?
any input on correct priority for this bug? Should this block v2.12 release?
I am not sure how to prioritize this bug. However, I believe it is not a blocker for the v2.12 release.
any input on correct priority for this bug? Should this block v2.12 release?
I am not sure how to prioritize this bug. However, I believe it is not a blocker for the v2.12 release.
Will need to fix for v2.12 as this has not root cause and could also impact other use cases and IPC4.
@kv2019i @andyross I have identified the same problem using a fuzzer with UndefinedBehaviorSanitizer:
INFO: Running with entropic power schedule (0xFF, 100).
INFO: 29455 files found in ./ipc3_corpus
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 29455 min: 1b max: 510b total: 2367995b rss: 32Mb
/home/tmleman/work/repos/thesofproject/sof/zephyr/include/rtos/string.h:43:50: runtime error: addition of unsigned offset to 0x088fbb24 overflowed to 0x088fbb23
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/tmleman/work/repos/thesofproject/sof/zephyr/include/rtos/string.h:43:50 in
/home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2: runtime error: member access within null pointer of type 'struct list_item'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2 in
/home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2: runtime error: load of null pointer of type 'struct list_item *'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2 in
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==432370==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x00000000 (pc 0x080e38a0 bp 0xeb9fcfd8 sp 0xeb9fcfb0 T432404)
==432370==The signal is caused by a READ memory access.
==432370==Hint: address points to the zero page.
#0 0x80e38a0 in ipc_comp_free /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c
UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c in ipc_comp_free
==432370==ABORTING
MS: 2 InsertRepeatedBytes-InsertByte-; base unit: b4f43ebcf03f40281d674b9e826e0d9be4f3ed4f
0x0,0x0,0x0,0x26,0x0,0xff,0x60,0x87,0x0,0x0,0x2,0x30,0x0,0x53,0x53,0x0,0xff,0x0,0x53,0x0,0x0,0x20,0x30,0x53,0x53,0x5b,0x53,0x53,0x51,0x0,0x0,0x1,0x50,0x53,0x53,0x53,0x0,0x0,0x1,0x30,0x0,0x0,0x0,0x2,0xff,0xff,0xff,0xff,0x2,0xff,0x19,0xff,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x24,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,
\000\000\000&\000\377`\207\000\000\0020\000SS\000\377\000S\000\000 0SS[SSQ\000\000\001PSSS\000\000\0010\000\000\000\002\377\377\377\377\002\377\031\377\000\000\000\000\000\000\000\000\000$\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000
artifact_prefix='./findings/ipc3_2025-01-08_undefined/'; Test unit written to ./findings/ipc3_2025-01-08_undefined/crash-6eb29353a782085538e848200ca661280437a13b
Base64: AAAAJgD/YIcAAAIwAFNTAP8AUwAAIDBTU1tTU1EAAAFQU1NTAAABMAAAAAL/////Av8Z/wAAAAAAAAAAACQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
The root cause is likely an uninitialized list of sources and sinks inside the IPC component device. In case of an issue within string.h, I have submitted a separate issue: https://github.com/thesofproject/sof/issues/9768
Hmm, not sure how we end up here. I've been trying to repro this with the provided corpus but no luck so far. It would seem commit 403c33f8296564f6fc78d9d41fd4e50961652199 should catch an uninitialized bsink/source list, but this was merged already last year, so can't be this. ipc_get_comp_by_id() should be safe as well as the comp_list is initialized in ipc_init() which is part of posix platform init.
The newer log seems to point to &icd->cd->bsource_list /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2: runtime error: member access within null pointer of type 'struct list_item' SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2 in
... but yeah, there's a NULL check just before this.
My first thought is that Linux bug from a few years back where a previous UB in the same context allowed the compiler to assume that a pointer must be valid and elide a check. :) Would be hilarious if we got bitten by the same kind of thing, but maybe that's too much too hope for. @tmleman if you have a failing build, can you build with CONFIG_OUTPUT_DISASSEMBLY and post the resulting zephyr.lst file?
We are hitting timelimits for 2.12. As a) this is not tagged as release blocking, b) I can't reproduce with the provide corpus (albeit on a newer 24.04 Ubuntu based dev machine but still), and c) a fresh manual review of the code does not reveal gaps in ipc_comp_free, so a+b+c together I'll moved this to v2.13 and unassign myself for now.
We are hitting timelimits for 2.12. As a) this is not tagged as release blocking, b) I can't reproduce with the provide corpus (albeit on a newer 24.04 Ubuntu based dev machine but still), and c) a fresh manual review of the code does not reveal gaps in ipc_comp_free, so a+b+c together I'll moved this to v2.13 and unassign myself for now.
Ack - if it cant be reproduced from corpus then we can move, I suspect FW updates have fixed. We can close if no further fuzzer reports for v2.13
but yeah, there's a NULL check just before this.
@kv2019i The checks you are talking about only check the beginning of the list. The null pointer appears later when the list is iterated.
@andyross I think this explains why the ifs don't detect the NULL pointer, but I made sure they are not removed during the optimization process.
I managed to partially determine in debugging where this pointer comes from. Initially, the pointer has the correct value, but several runs later, I encountered a situation where it is overwritten with zeros during the memory clearing that follows the allocation of a new component. Another less common error is the wrong alignment of the address that this pointer points to. And similarly to before, the pointer value is overwritten during allocation, only this time by chunk_set.
In my opinion, this is a use after free error. The component's memory was freed but it was not removed from all lists. I tried to use Valgrind for debugging but whatever I do, I get an error that it failed to parse debug symbols. Address sanitizer does not detect this because we use a custom allocator. Manually poisoning the freed memory in the allocator may be helpful (which I am currently trying to implement).
I know @tmleman you are busy with multiple items, so let me push this to v2.14.
This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.
This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.
This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.