Makdi comments

Results 18 comments of


                                            Makdi

Gracefully closing the crawler with keepalive flag true

@B4nan any update?

Get Monitor output as Object

any update on this? @thomasdondorf

[Robots.txt] Incorrect robots txt result for uppercase user agents

Oh, now I get it. Yes that could be done since internally UAs mentioned are lower-cased and then compared with the incoming UA from function arguments. Upto you. @jnioche

[Robots.txt] Incorrect robots txt result for uppercase user agents

Let me know if you guys are planning to have this, I can create a PR for the same. Thanks.

[Robots.txt] Incorrect robots txt result for uppercase user agents

Hey @jnioche @sebastian-nagel Shall I create a PR for the same?

Support for crawling from secondary IP address

For http crawler, this was relatively easy. ```javascript preNavigationHooks: [ async (crawlingContext, gotOptions) => { gotOptions.localAddress = secondaryIpAddress } ] ``` Setting `gotOptions.localAddress` works. Still looking out for `PuppeteerCrawler` I...

Invalid host defined options error in http crawler when used with pkg post ESM edition

Hey @vladfrangu Yes, it only fails via docker. Works fine if run directly on host machine via node. Also in docker, it fails when used with https://github.com/vercel/pkg You can checkout...

Invalid host defined options error in http crawler when used with pkg post ESM edition

Hey @vladfrangu , it is reproducible only with `pkg` tool. We have a requirement to convert our app to binary hence using the same. It however works with [email protected], not...

Warning : Queue head returned a request that is already in progress

@B4nan

Warning : Queue head returned a request that is already in progress

Hey @B4nan any update on this?