spidy
spidy copied to clipboard
rivermont
The simple, easy to use command line web crawler.
## Expected Behavior Docker should simplify things not make them harder ## Actual Behavior Docker is a strugle, you have to build image several times before it works: It ignores...
## Feature Description Some cookie handling functionality would be pretty valuable. Setting cookies in the config file should be trivial to implement. An option to send a GET or POST...
unusable
hi, i tried to use spidy b.c. it looked promising. Is it dead? first: sudo pip install -r requirements.txt doest work, reppy is not installable (python 3.9) snd: Docker is...
Calling any passers-by to take a moment to submit feature requests or bugs, no matter how small! Please see the README for a general overview of this project, `docs.md` for...
Most likely using something like [py2exe](http://www.py2exe.org/).
## Checklist - [x] Same issue has not been opened before. ## Expected Behavior All threads to stop as crawler prints info and saves files. ## Actual Behavior Once one...
## Feature Description It would be great to have a check in `tests.py` for the multithreading and queue. ## Checklist - [X] This feature does not already exist. - [X]...
Arguments for: - [ ] Overwrite existing save files - [ ] Raise Errors (possibly only for different severity levels?) - [ ] Save pages - [ ] Save words...
After multiple tries I have yet to get PyPI to format the README correctly. Current state can be viewed [here](https://pypi.org/project/spidy-web-crawler/). At the moment, I followed [this SO answer](https://stackoverflow.com/a/26737672) and convert...
Having a clicky interface has been a goal for a long time now. There are many users who abhor the command line but are still interested in the tools that...