scrapi icon indicating copy to clipboard operation
scrapi copied to clipboard

[WIP] Add "page_limit" argument to harvesters

Open erinspace opened this issue 10 years ago • 2 comments

Add arg that will grab only a certain number of pages of results. This will help for both local development and test generation, as we can limit the number of results that the test harvesters harvest, and local harvesters take in.

erinspace avatar Sep 15 '15 14:09 erinspace

I wonder if it might be better to specify the number of times to follow links? That would allow us to choose the minimum number of jumps for our other harvesters to pass

fabianvf avatar Sep 16 '15 20:09 fabianvf

I like that idea. I'll change it so it does that!

erinspace avatar Sep 16 '15 20:09 erinspace