Unable to get content of page from special characters (utf-8) URL

Open Creatium opened this issue 2 years ago • 1 comments

Hey,

I am currently in progress of switching from main Selenium to Selenium Wire to start using proxies for scraping activities.

Now everything works fine and as expected, except websites that have special characters in their addresses, like https://männimetsa.ee/. Here I have actually two issues.

Website page_source results in empty content <html><head></head><body></body></html>. No matter what I do, if I implement waiting or not, this is what I get. No actual Exception in my try catch block is thrown. Just empty body.
If using request_interceptor I get this error: UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 1: ordinal not in range(128) . Even if inside interceptor I just print out the request.path, its the error that gets thrown in my face. And printing request.path results in this text: https://männimetsa.ee/ where the https://m part is as active URL.

Not sure if issue 1 is related to issue 2, but both issues only happen with special character website URLs. So I have a feeling that both issues are caused by the same main reason.

If I switch back to Selenium, I can get the content of website with no problems.

Dec 18 '23 15:12 Creatium

For now moved to using SeleniumBase for proxy functionality - no such issues.

Dec 19 '23 17:12 Creatium