ProbabilitySamplingScheduler might be inverted
According to the comment for TestcaseScore::compute below, returning a higher value should make it more likely that a queue member is selected: https://github.com/AFLplusplus/LibAFL/blob/1dcfe8ef56f38cc15c9d2205756550fda7cdf85a/libafl/src/schedulers/testcase_score.rs#L23
It looks to me like the reciprocal of this number is used in the ProbabilitySamplingScheduler, resulting in bigger compute scores ending up with lower probability of selection:
https://github.com/AFLplusplus/LibAFL/blob/1dcfe8ef56f38cc15c9d2205756550fda7cdf85a/libafl/src/schedulers/probabilistic_sampling.rs#L87
In my local version I have changed the above line to just let prob = factor, and it seems to behave more like I would expect. Perhaps I'm missing something though.
I'll also add that I am updating testcase scores, so I also added the following to the end of the function (if this is a new key then the subtraction never happens, so works for new insert and replace):
if let Some(old_prob) = meta.map.insert(idx, prob) {
// subtract the old probability from total first
meta.total_probability -= old_prob;
}
meta.total_probability += prob;
@tokatoka can you take a look?
yes i think you are right can you send a PR?
I'll also add that I am updating testcase scores, so I also added the following to the end of the function (if this is a new key then the subtraction never happens, so works for new insert and replace):
To implement this, can you impl RemovableScheduler for Probabilistic scheduler