client_python icon indicating copy to clipboard operation
client_python copied to clipboard

Use rust to improve reading .db file performance

Open csmarchbanks opened this issue 11 months ago • 0 comments

This is an idea to try to improve multiprocess collect performance by using rust to collect the metrics from various .db files. Current state of the benchmarks shows a 2x improvement on my machine (x86 linux).

---------------------------------------------------------------------------------------- benchmark: 2 tests ----------------------------------------------------------------------------------------
Name (time in ms)                        Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_speedup_collect_performance     33.9249 (1.0)      35.3835 (1.0)      34.6150 (1.0)      0.3620 (1.0)      34.6984 (1.0)      0.5581 (1.0)           6;0  28.8892 (1.0)          20           1
test_native_collect_performance      70.0792 (2.07)     81.2032 (2.29)     72.6370 (2.10)     3.4046 (9.41)     70.7422 (2.04)     3.4498 (6.18)          2;1  13.7671 (0.48)         13           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

csmarchbanks avatar Jan 24 '25 20:01 csmarchbanks