DistriNet
Evaluation is not always exhaustive
On some occasions, I've noticed that BugHog core finishes while there (seemingly) are still binaries left to test. After such a finish, starting the evaluation for a second time will make BugHog evaluate the missed binaries.
Not sure what causes this at the moment, but we should solve this in the future.
This issue might be caused in part by worker containers exiting unexpectedly. In this case, the associated experiment is not marked as "dirty", and thus, is not reported as having failed. The associated browser version is simply not part of the final result set.
The choice of not marking these issues as "dirty" was made because they are more related to infrastructural errors, as opposed to browser-related errors. We might want to reconsider this in the future, though.
This issue is partly fixed by eff772b6348af91428558c0a858b32bd9a75a907 where worker error reporting is improved worker containers are allowed more RAM, and a6657d5bdaaf755a57a561de7aec2aaabb05ad3d where and browser caching is made more efficient.