paritytech
Fuzz testing for nomination pools
This PR is itself pretty insubstantial, there is not real code change other than adding a fuzz-style test for nomination pools.
Some notes about the approach: For now, I mainly did the fuzz-testing without using a proper fuzzer because it would allow for the mock to be reused. In the past we gated mock.rs behind some feature flag to make it available, and this time around I wanted to try the alternative approach.
Having done this, I can say that the main advantage of this type of testing is seamless setup, and the downside is reproducibility of failing tests.
For now, in order to have reproducible errors, you have to use a seeded randomness, and then re-use it. It requires a little bit of manual work, but it did allow me to solve every single test failure.
It is probably still better to move the fuzz-test use a proper fuzzer as follow-up. Nonetheless, it must be done such that the mock file is reused (a good first issue for a new team member or external contributor).
Some notes about the fuzzing outcome: As you see in the code, I randomly generate calls that may or may not be valid and bombard the pallet with these calls, and every so often I execute the entire set of sanity checks that are defined in the pallet.
Moreover, I inject a single agent into a random pool that is manually tracking the expected reward amount (see Agent in the code for details).
The tests generally get quite slow, particularly if you want to execute the sanity checks often. In the default setup, the state grows until 1000 pools and 10000 members, and with 1 million random submissions, this is the final outcome:
Aug 10 16:16:23.564 INFO runtime::nomination-pools: [1] 🏊♂️ running sanity checks at 1010000
Aug 10 16:16:23.640 INFO runtime::nomination-pools: [1] 🏊♂️ iteration 1010000, 1000 pools, 10000 members, 325223 ok 684777 err, events = [("Event::Created", 1000), ("Event::Bonded", 72300), ("Event::Unbonded", 41458), ("Event::PaidOut", 164750), ("Event::Withdrawn", 21622), ("Event::MemberRemoved", 1)]
All in all, this was super useful and I feel way more confident about nomination pools now.
