Control-Flag crash at Ubuntu

[raniervf]

Control-Flag crash at Ubuntu 64 (bits) Compiler: clang (14)

output: killed [pid] output directory: empty, none file generated.

[AlexisTM]

Same result on Ubuntu 20.04, with gcc 9.3.0.

[jgottschlich]

Thanks @raniervf and @AlexisTM - we're looking into this.

In addition, if you have the bandwidth and can locate the cause for the crash for it, we'd love to have you send us a pull request.

Best, The ControlFlag Team

[nhasabni]

Hi @raniervf and @AlexisTM,

Thanks for trying out ControlFlag. Would you mind filling out the template below for the bug report? That will help us triage and fix faster.

--

Describe the bug A clear and concise description of what the bug is.

Exact command to reproduce Please specify ControlFlag command that you used

Callstack (if it is a crash bug) or error info Please provide any info about the error - error message, code, etc.

Expected behavior A clear and concise description of what you expected to happen.

Environment (please complete the following information):

• OS:
• Compiler:
• 32-bit or 64-bit?
• Build command:

ControlFlag commit

Additional context Add any other context about the problem here.

[wsnipex]

looks like out of memory, this is what happens for me, when I try to scan a large C++ repo:

$ journalctl -xe |grep Killed

Out of memory: Killed process 197753 (cf_file_scanner) total-vm:10291624kB, anon-rss:10266540kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:20152kB oom_score_adj:0

for reference: the box has 16GB Ram and 2GB swap.

[nhasabni]

Thanks @wsnipex for finding the issue. Yes, our current version of the training dataset requires around 11GB of memory. We can certainly generate a smaller version of the training dataset for memory-constrained devices. We will add this to our current list of features. The detection results, however, would vary with smaller dataset.

[wsnipex]

I got it to run by closing all memory intensive applications and adjusting oom_killer: echo -15 > /proc/$(pidof cf_file_scanner)/oom_adj

it's been running for 5h now ;)

slightly off topic: I noticed it only uses [num physical cores] parallel threads by default, so hyperthreading/logical cores are unused.

[jgottschlich]

Whooohooo! Let us know how your results look -- how big is your code base? Apologies in advance for any false positives you see. We're still working on bringing that number down.

Regarding hyperthreading, we've found that using the hardware threads tends to work more efficiently than using all the logical / software threads that can be enabled using hyperthreading. However, perhaps that was just environment to our execution environment. Perhaps we can look at adding a flag to enable hyperthreading if a user would like to use them?

Thoughts, @nhasabni ?

[AlexisTM]

That is probably the reason. I have 16GB RAM on that machine.

[nhasabni]

@AlexisTM Feel free to use smaller training datasets that we have released a couple of days ago.

[AlexisTM]

I will try to use it with the least amount of apps first, as I do not have much time to retrain it.