crawlee
crawlee copied to clipboard
Adding to queue with options parameter
i would suggest refactor little bit enqueueLinks to something like addLinksToQueue, and add to it more flexibility adding a new argument of type Object with extra parameters for example:
forefront - adding to front of queue. skip - skipping elements. limit - limit elements. shuffle - shuffle array.
await addLinksToQueue({
$,
requestQueue,
selector: '.row a',
baseUrl: request.loadedUrl,
transformRequestFunction: req => {
req.userData = {
page: 'somePage'
}
return req;
}
},{forefront:true,skip:5,limit:10,shuffle:true});