http-crawler icon indicating copy to clipboard operation
http-crawler copied to clipboard
verified publisher icon inglesp

Allow user to choose which links to follow

Open inglesp opened this issue 9 years ago • 1 comments

We currently follow all links, but in some cases this might not be appropriate

We should find a way to allow the user to configure which links to follow.

inglesp avatar Jun 09 '16 11:06 inglesp

I've just added the follow_external_links option to crawl, which defaults to True. I think this addresses the likely main use case for this.

inglesp avatar Jul 07 '16 18:07 inglesp