Spoof/Change User-Agent

Open jonzlin95 opened this issue 8 years ago • 2 comments

Some sites don't play well without User-Agents (e.g. ESPN).

HTTPoison by default sends hackney which is blocked, so it could be useful to set it to

["User-Agent": "Elixir"] or some similar string to bypass these problems.

Feb 08 '17 21:02 jonzlin95

Why not faking a typical iPhone signature? I think some sites whitelist on UA strings

Feb 09 '17 07:02 Anonyfox

That could work too. Maybe it would be useful to add a way to pass in a User Agent to the Scrape.X function calls. I notice that you already have fetch(url, html_headers \ []), and that could just be passed onto the next set of functions.

I can make a simple PR for this.

Feb 10 '17 22:02 jonzlin95