crab
crab copied to clipboard
atomicptr
🦀 A versatile tool to crawl dozens of URLs from a given source, like a sitemap or an URL list.
upd: The url list of all request is saved in the `./output/output.txt` file. feat: Print query results as array with `output-json` upg: Dependencies upgraded. upd: Readme new flags added. I...
A few months ago the snap part of the pipeline broke and I haven't been able to get it working again since then so I decided to remove snap support...
Preferably as some sort of integration with the [vegeta load testing tool](https://www.scaleway.com/en/docs/vegeta-load-testing/).
```bash $ crab crawl:sitemap https://domain.com/sitemap.xml --output output.log # NO prints here... ``` It might probably make sense to have a different option for persisting the logs as a valid json...