Negative values in Mops bench

[tomijaga]

Hey @ZenVoich,

I've been using the mops bench command to evaluate the performance of some libraries I'm developing. I've noticed an unusual behaviour in the heap benchmarks—they sometimes return negative values. I suspect this might be due to specific functions like replace(), get(), and delete() not making new heap allocations but instead releasing some of the allocated heap memory. As a result, the incremental garbage collector might be deallocating more memory than was initially allocated, resulting in negative values.

Another concern is the inconsistency of these results. When rerunning the benchmarks on the same codebase, the negative values are sometimes displayed on different columns. While this could be an issue with my implementation, I'm reaching out to see if you might know the root cause.

Here are the heap benchmark results:

Heap (1) - GitHub Link

	insert()	replace()	get()	entries()	delete()
RBTree	8,988,396	-22,144,144	13,168	1,889,084	18,369,008
BTree	1,228,812	1,157,820	485,272	602,388	5,609,008
B+Tree	574,452	413,092	-31,446,424	9,020	5,009,008

Heap (2) - GitHub Link

	insert()	replace()	get()	entries()	delete()
RBTree	9,061,892	-22,162,980	16,088	1,889,084	16,729,008
BTree	1,234,776	1,161,124	488,704	602,300	5,209,008
B+Tree	580,928	417,812	217,808	9,020	-27,046,888

Your insights on this would be greatly appreciated.

[ZenVoich]

Hey, @tomijaga

It seems like the incremental GC produces unstable results. I would recommend to use copying GC, I think I will change default GC in mops too. I have found that the copying GC produces more consistent results.

they sometimes return negative values

Yes, this is possible because mops counts heap in this way: <heap after benchmark cell run> minus <heap before benchmark cell run>