LibAFL icon indicating copy to clipboard operation
LibAFL copied to clipboard

ProbabilitySamplingScheduler might be inverted

Open DanBlackwell opened this issue 1 year ago • 3 comments

According to the comment for TestcaseScore::compute below, returning a higher value should make it more likely that a queue member is selected: https://github.com/AFLplusplus/LibAFL/blob/1dcfe8ef56f38cc15c9d2205756550fda7cdf85a/libafl/src/schedulers/testcase_score.rs#L23

It looks to me like the reciprocal of this number is used in the ProbabilitySamplingScheduler, resulting in bigger compute scores ending up with lower probability of selection: https://github.com/AFLplusplus/LibAFL/blob/1dcfe8ef56f38cc15c9d2205756550fda7cdf85a/libafl/src/schedulers/probabilistic_sampling.rs#L87

In my local version I have changed the above line to just let prob = factor, and it seems to behave more like I would expect. Perhaps I'm missing something though.

I'll also add that I am updating testcase scores, so I also added the following to the end of the function (if this is a new key then the subtraction never happens, so works for new insert and replace):

        if let Some(old_prob) = meta.map.insert(idx, prob) {
            // subtract the old probability from total first
            meta.total_probability -= old_prob;
        }
        meta.total_probability += prob;

DanBlackwell avatar Feb 24 '24 17:02 DanBlackwell

@tokatoka can you take a look?

domenukk avatar Feb 26 '24 11:02 domenukk

yes i think you are right can you send a PR?

tokatoka avatar Feb 26 '24 13:02 tokatoka

I'll also add that I am updating testcase scores, so I also added the following to the end of the function (if this is a new key then the subtraction never happens, so works for new insert and replace):

To implement this, can you impl RemovableScheduler for Probabilistic scheduler

tokatoka avatar Feb 26 '24 13:02 tokatoka