openwpm
Option to spoof window.navigator.webdriver?
window.navigator.webdriver exposes if a client is "controlled by automation". Potentially sites could treat a crawler differently based on the if the window.navigator.webdriver is true, leading to a crawl less representative of a real user visiting a site. The browser parameter of bot_mitigation already exists, perhaps enabling bot_mitigation should also spoof window.navigator.webdriver to be false?
In short, should this be spoofed to be false (or have there be an option to do so?)
There is this pull request on the matter: #526
Unfortunately, this solution is not ideal. At least, it is known to leave traces in the DOM.
If you are really concerned about detection, you could use changes made in this repository. It should easily work with OpenWPM v.0.20.0. Also, you can add a recent unbranded Firefox browser from Mozilla, if you do not want to use an older Firefox version.
Details on OpenWPM and bot detection can be found here (preprint).
