Makdi

Warning : Queue head returned a request that is already in progress

6

### Which package is this bug report for? If unsure which one to select, leave blank @crawlee/core ### Issue description We have noticed a warning message in our crawlee logs,...

Invalid host defined options error in http crawler when used with pkg post ESM edition

5

### Which package is this bug report for? If unsure which one to select, leave blank @crawlee/http (HttpCrawler) ### Issue description Error ``` makdi-test | DEBUG HttpCrawler:SessionPool: No 'persistStateKeyValueStoreId' options...

Gracefully closing the crawler with keepalive flag true

1

### Which package is the feature request for? If unsure which one to select, leave blank @crawlee/core ### Feature I'm using puppeteer crawler with `keepAlive` as `true` and `crawler.run()` (without...

[Robots.txt] Incorrect robots txt result for uppercase user agents

7

Example for forbes.com robots txt https://www.forbes.com/robots.txt They have blocked all paths for `GPTBot` ``` User-agent: GPTBot Disallow: / ``` However for url `https://www.forbes.com/test` ```java public boolean canCrawl(String url, String userAgent,...

Support for crawling from secondary IP address

1

### Which package is the feature request for? If unsure which one to select, leave blank @crawlee/http (HttpCrawler) ### Feature Hi, I see with both HttpCrawler and PuppeteerCrawler, ProxyConfiguration is...

Adding multiple puppeteer blockers on single page object

2

Does adding multiple puppeteer blockers on single puppeteer [page](https://pptr.dev/api/puppeteer.page/) object work Example both ```javascript await PuppeteerBlocker.fromLists(fetch, [ 'https://secure.fanboy.co.nz/fanboy-annoyance.txt' ]).then(async (blocker) => { await blocker.enableBlockingInPage(page); }); await PuppeteerBlocker.fromPrebuiltAdsOnly(fetch).then(async (blocker) => {...