crawlee-python
crawlee-python copied to clipboard
Include whitelisted HTTP headers in the `extended_unique_key` computation
- Modify the
extended_unique_keycomputation to include a set of predefined HTTP headers, alongside the existing normalized URL and payload. - Only include headers from the whitelist.
- Identify which headers should be included, such as
Accept,Accept-Language,Authorization, and others that may affect request outcomes. - See the issue #178 for further context.
Hey will you assign this issue to me
Hi @ravi-hash, sure, thanks for your interest in Crawlee.