scrapi
scrapi copied to clipboard
[WIP] Add "page_limit" argument to harvesters
Add arg that will grab only a certain number of pages of results. This will help for both local development and test generation, as we can limit the number of results that the test harvesters harvest, and local harvesters take in.
I wonder if it might be better to specify the number of times to follow links? That would allow us to choose the minimum number of jumps for our other harvesters to pass
I like that idea. I'll change it so it does that!