ScalaWebscraper

ScalaWebscraper copied to clipboard

Allow setting referrer in download request

1

Thanks for the tool, it's pretty useful. A nice addition would be the ability to set the referrer (and perhaps other variables, like user-agent) in the http request that's sent...

matt-gardner

Some domains block spiders when a lot of requests are being send to the server, to prevent this implement throttling on domain-level.

Rovak

Create a simple framework with a easy to use DSL

Rovak