substrate-runtime-fuzzer icon indicating copy to clipboard operation
substrate-runtime-fuzzer copied to clipboard

Histogram of `DispatchResults`

Open kianenigma opened this issue 1 year ago • 0 comments

The main question i would have about this fuzzer before making it general to be used with any runtime is:

  1. what percentage of the blocks/extrinsics generated actually end up executing and doing something useful? If you have access to it, what is the histogram of Ok(_) vs Err(_)
  2. Similarly, we need a metric of the size of the state. Is it growing? It should, and if not, I would argue that the above histogram would also be all Err(_) and the runtime is not doing anything.

For example, I would expect a good fuzzer, with the genesis state of a single account having the total issuance to end up spreading the total issuance over many accounts after many iterations. But, is that actually happening?

This is the type of question I am trying to answer here.

I am aware that this fuzzer has already been very useful catching bugs, so maybe my skepticism is not needed, idk. What type of bugs have been detected? is it mostly in the API/Encode/Decode level, or has it also detected a bug deep in the logic of a pallet?

kianenigma avatar Jun 03 '24 07:06 kianenigma