Lam Nguyen

[MRG+1] Import scurl if found installed

9

Hey @lopuhin, I made this PR to run scurl test on w3lib itself. Let me know if there's something that can be improved here 😄

I got this when I was trying to install scrapy-bench on Python 3.7. Can you guys confirm this for me? Thank you! ``` Failed building wheel for vmprof Running setup.py...

I realized that when I cancelled the command with Ctrl-C, the csv files generated by the command itself are not deleted. Therefore, does it have any bad effect at all?...

WIP: use original tests from urllib

2

This PR restores all the tests from urllib, since the current tests in Scurl are modified to pass the tests. However, we need Scurl to act similar to urllib 😄

Additional components to implement

1

Here are some components that can be implemented to further enhance the performance of canonicalize_url func: - [ ] parse_qsl_to_bytes, which includes unquote_to_bytes from stdlib - [ ] urlencode -...

Right now Scurl fails to run on Windows. We will need to come up with a way to support windows :) Details on this are coming soom

tox has been found failed on Mac OS on py34 and py35 env. Tox creates an environment that is Mac OS 10.6 for some reason and this is the traceback...

icu is commented out in Chromium source

2

Right now, [this](https://github.com/nctl144/scurl/blob/master/vendor/gurl/url/url_canon_host.cc#L175) line is commented out. We should figure out a way to enable icu for this project :) The old chromium source of icu has this commented out...

Right now GURL could not handle such idna urls. All of the idna urls are marked as invalid. Although Google Chrome does parse these urls correctly! ``` >>> URL('банки.рф'.encode('idna')).is_valid() False...

Integrate scurl into w3lib and scrapy

1

We will need to work on integrating this library into Scrapy and W3lib. Make it an option for users to install it. Right now, we can prompt a message if...