Amit Upreti
Amit Upreti
please pull again and try, it should now pull the hashtag. Earlier the crawler was searching for the keyword when it should have been searching for hashtag.
thank you Much appreciated. please tag if you find other bugs. I will update both tomorrow. For now. I will revert the latest commit.
Could You please share the hashtag you were trying to scrape? I will try to reproduce this on my computer. Thanks
I just cloned the repo and ran the crawler for `festabbb`. It seems to be pulling data without any issues. It pulled 699 items.(i had concurrent request set to 5)...
yes, I am on Linux. But that shouldn't be an issue. I will test on my windows when I am home and get back to you. Could you try again...
download delay is in seconds. Please set it to 0 or 1(probably 0). Yes. Settings ROBOTSTXT_OBEY to false is a great idea. The crawler might be obeying some new settings...
From what I have experienced. The most data I could pull from twitter for a single hashtag was around 5k. It turns out if you browse twitter manually and keep...
This solves the issue. Now we can pass the proxy in our first request via meta. and our script would still handle retries with new proxies ``` if 'proxy' in...
@Yomguithereal The BSD 2-Clause License provides freedom to do whatever you want with the `user-agent` package as long as you include their original license with your python version. I am...
okay. I will start a repo in few days and ping you.