crawlee-python icon indicating copy to clipboard operation
crawlee-python copied to clipboard

Curl Cffi Client

Open siddiqkaithodu opened this issue 1 year ago • 1 comments

Since we use HTTPX , Isn't better to use curl_cffi instead or use it an optional client that switches when the HTTPX gets defeated in the AntiBot war.

I am thinking about intelligently switching between multiple http clients. (The same as of proxy rotation)

siddiqkaithodu avatar Jul 12 '24 12:07 siddiqkaithodu

Hi, thanks for the issue. I think we can split this one into two parts. First, we can (probably, needs further investigation) switch from httpx to curl_cffi, which seems better meet our use-case. The second part would be the switching, which should be part of a separate issue/discussion.

vdusek avatar Jul 15 '24 10:07 vdusek