crawlee icon indicating copy to clipboard operation
crawlee copied to clipboard

Adding to queue with options parameter

Open maoryadin opened this issue 4 years ago • 0 comments

i would suggest refactor little bit enqueueLinks to something like addLinksToQueue, and add to it more flexibility adding a new argument of type Object with extra parameters for example:

forefront - adding to front of queue. skip - skipping elements. limit - limit elements. shuffle - shuffle array.

    await addLinksToQueue({
                $,
                requestQueue,
                selector: '.row a',
                baseUrl: request.loadedUrl,
                transformRequestFunction: req => {
                    req.userData = {
                        page: 'somePage'
                    }
                    return req;
                }
            },{forefront:true,skip:5,limit:10,shuffle:true});

maoryadin avatar Feb 17 '21 12:02 maoryadin