neokd
Bug fix and list replaced with deque
Description
Fixed a bug and switched over to a deque instead of a list
Related Issues
https://github.com/neokd/DataStorehouse/issues/112
Changes Made
load_proxy_list() doesn't rely on an absolute path anymore, this wasn't working as intended.
rotate_proxy() Switched over to a deque for better efficiency when rotating proxies
Screenshots (if applicable)
N/A
Checklist
Please review and check the following before submitting your pull request:
- [x] I have followed the project's coding conventions and guidelines
- [x] I have tested my changes thoroughly
- [ ] I have added/updated relevant documentation
- [x] My code follows best practices and is easy to understand
- [ ] I have added necessary test cases (if applicable)
- [ ] All existing tests are passing
- [x] My changes do not introduce any new warnings or errors
Additional Notes
I would like to add test cases for this
Someone is attempting to deploy a commit to a Personal Account owned by @neokd on Vercel.
@neokd first needs to authorize it.
![vercel[bot] avatar](https://avatars.githubusercontent.com/in/8329?v=4 "Published by a pub.dev verified publisher"){kind=small}
Can the proxy be made like an parameter? Like if user set as True then proxy works or else it tries to scrape without proxy.?
Can the proxy be made like an parameter? Like if user set as True then proxy works or else it tries to scrape without proxy.?
Are you referring to a user-agent a browser would use?
No, for now we were thinking that proxy is getting complicated so if we allow user to pass an argument that they want to use scrapper or not.
No, for now we were thinking that proxy is getting complicated so if we allow user to pass an argument that they want to use scrapper or not.
I see. I'll look into adding that option for the scraper.
@Bchass looks good I'll review it fully and merge it. Thinking to publish the scrapper as package on Pypi?
@Bchass yeah the project is maintained. There was merge conflict so i didn't merge this PR.