Error occurred while processing a large number of files

[iwancat]

used version :dust-v1.1.1-i686-unknown-linux-musl ./dust -d 1 /mnt/countdir Indexing: /mnt/countdir 18349815 files, 36T ... |memory allocation of 560000 bytes failed Aborted (core dumped)

Then I added parameters -S ,Error message: Printing becomes longer ./dust -d 1 -S88388608 /mnt/countdir Indexing: /mnt/countdir 18018277 files, 36T ... /memory allocation of memory allocation of 150memory allocation of bytes failed memory allocation of 150150 bytes failed memory allocation of bytes failed memory allocation of memory allocation of 150150150 bytes failed bytes failed 150 bytes failed bytes failed Aborted (core dumped)

gdb dust -c core file : [New LWP 46342] Core was generated by ./dust -d 1 -S88388608 /mnt/countdir'. Program terminated with signal 6, Aborted. #0 0xf77f5430 in __kernel_vsyscall () (gdb) n`

From the top command , %Cpu(s): 1.0 us, 7.6 sy, and the dust process's RES was at 3.8 GB when the program experienced a core dump.

[tatref]

Can you try the 64bits version?

[bootandy]

What if you keep increasing the '-S' parameter ?

[iwancat]

What if you keep increasing the '-S' parameter ?

Continue with this error message ”Indexing: “ number close to before and then abort

[iwancat]

Can you try the 64bits version?

I have tried that version [dust-v1.1.1-x86_64-unknown-linux-musl.tar.gz]，but the error is still the same. May I ask the maximum number of files that this dust supports for retrieval.

[bootandy]

It's clearly run out of memory. I don't think I can do anything about that. You can keep increasing the -S parameter, you could also try a non musl build but I doubt that would help.