crawlee-python icon indicating copy to clipboard operation
crawlee-python copied to clipboard

Include whitelisted HTTP headers in the `extended_unique_key` computation

Open vdusek opened this issue 1 year ago • 2 comments

  • Modify the extended_unique_key computation to include a set of predefined HTTP headers, alongside the existing normalized URL and payload.
  • Only include headers from the whitelist.
  • Identify which headers should be included, such as Accept, Accept-Language, Authorization, and others that may affect request outcomes.
  • See the issue #178 for further context.

vdusek avatar Sep 27 '24 17:09 vdusek

Hey will you assign this issue to me

ravi-hash avatar Oct 02 '24 15:10 ravi-hash

Hi @ravi-hash, sure, thanks for your interest in Crawlee.

vdusek avatar Oct 03 '24 08:10 vdusek