Add option to deactivate auto-retire when proxy is blocked

[rubydev]

Describe the feature In Playwright/Puppeteer crawler, when response is for example 403, crawler automatically throw Error: Request blocked - received 403 status code.. Please add option to disable this functionality (throwOnBlockedRequest).

Motivation I would like to handle blocked requests myself. For example I would like to count number of blocked requests, for calculating statistics and blocking ratio.

Constraints No Constraints.

Thanks!

[petrpatek]

I think you can do that in the postNavigationHooks. So, for example, It is possible to solve a blocking challenge, wait for the redirect and reassign the response in the crawlingContext to the new response from the redirect.

[rubydev]

That makes sense, thank you!

[B4nan]

Closing as the blocked status codes are nowadays configurable

https://crawlee.dev/api/core/interface/SessionPoolOptions#blockedStatusCodes