codechecker icon indicating copy to clipboard operation
codechecker copied to clipboard

[store] Unique reports _before_ storing

Open Szelethus opened this issue 2 years ago • 0 comments

We used to zip every result file in the result directory and send it to the server. This is tremendously wasteful, as many report files contained the very same reports (for instance, if the report originated from a header file).

In this patch, I unique all reports, dump them in a tmpfile, and store that to the server. For a clang-tidy --enable-all analysis on xerces, this reduced the size of the zipfile drastically:

BEFORE: Compressing report zip file done (1.6GiB / 62.2MiB). AFTER: Compressing report zip file done (372.6MiB / 7.9MiB).

While this doesn't speed up CodeChecker parse or CodeChecker diff, it still speeds up the server quite a bit, both for storage and query times.

I also did a little cleanup on the debug logs -- the one in report converter spit out a line for literally every report, that feels a little unnecessary. I also added some logs to show that the parsing of result files is in progress, though that could be a little more verbose.

Szelethus avatar Jan 15 '24 10:01 Szelethus