A memory allocation race in lx is causing nondeterministic crashes

[silentbicycle]

Running the following command:

$ while true; do sleep 0.1; ./build/bin/lx -l test < src/lx/lexer.lx && echo ok; done

will produce a steady stream of memory-related errors in lx.

There appears to be a race on malloc/realloc on a shared pointer. I am going to post a PR in a moment that disables multithreading, this issue exists as a reminder to fix it later. It's currently a distraction from other more important work.

[katef]

Hm I can't reproduce this, I tried with and without asan, ubsan and efence. Any ideas?

[silentbicycle]

You need to set the concurrency flag (-C) to use multiple threads. I just checked again on main and

$ while true; do sleep 0.1; ./build/bin/lx -C8 -l test < src/lx/lexer.lx && echo ok; done

still produces errors

ok
malloc_consolidate(): invalid chunk size
Aborted (core dumped)
ok
...
ok
double free or corruption (out)
Aborted (core dumped)
ok
corrupted size vs. prev_size
Aborted (core dumped)
ok
...
ok
corrupted size vs. prev_size
Aborted (core dumped)
ok
...
ok
malloc(): unsorted double linked list corrupted
Aborted (core dumped)
ok
double free or corruption (out)
Aborted (core dumped)
ok

It may depend somewhat on CPU load -- if I ran multiple processes of a busywait program to bog the CPU down the failures reduced in frequency, and they returned when I stopped the busywaiters.

It used to fail without the -C flag, because before ae73900f4cb4e3640621cc94511ed2fcb8448517 the default was 4 threads rather than 1.