Fabian Gebert

SafeEyes 2.1.2 fails with 'NoneType' object is not subscriptable

3

On Ubuntu 20.10 64-bit with SafeEyes 2.1.2 installed through PPA I get following error on startup: ``` Traceback (most recent call last): File "/usr/bin/safeeyes", line 9, in load_entry_point('safeeyes==2.1.2', 'console_scripts', 'safeeyes')()...

Respect robots.txt / robots meta tag when crawling websites

1

I'd suggest to implement some functionality to make the web crawler respect index / disallow settings as defined in the robots.txt or robots meta tags of the website that is...