Fix a bug for header field getting malformed when it extends over multiple lines

[okomestudio]

The message headers section of RFC 2616 (Hypertext Transfer Protocol -- HTTP/1.1) states that

Header fields can be extended over multiple lines by preceding each extra line with at least one SP or HT.

The current implementation of s3lib.http.headers_raw_to_dict function unfortunately tends to read only the first line of such multi-line header field (and discard the rest in the following loop making tuples).

This PR refactors the function to be able to properly read multi-line header field. To do this reliably, it also more strictly interprets CRLF (i.e., \r\n) as the end-of-line marker, as per RFC 2616.

[okomestudio]

I think the CI failed because the version of tox used no longer support Python 3.3. Making the tests pass requires downgrade to tox<=2.9.1, but I do not have control over it.

[codecov[bot]]

Codecov Report

Merging #111 into master will decrease coverage by 0.32%. The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #111      +/-   ##
==========================================
- Coverage   95.43%   95.11%   -0.33%     
==========================================
  Files           7        7              
  Lines         482      491       +9     
  Branches       98      101       +3     
==========================================
+ Hits          460      467       +7     
- Misses         15       17       +2     
  Partials        7        7              

Impacted Files	Coverage Δ
w3lib/util.py	69.69% <0.00%> (ø)	:arrow_up:
w3lib/form.py	96.42% <0.00%> (ø)	:arrow_up:
w3lib/url.py	98.04% <0.00%> (ø)	:arrow_up:
w3lib/http.py	95.12% <0.00%> (-4.88%)	:arrow_down:
w3lib/html.py	93.80% <0.00%> (ø)	:arrow_up:
w3lib/encoding.py	100.00% <0.00%> (ø)	:arrow_up:

[Gallaecio]

Will you consider keeping splitlines?

[okomestudio]

Before fixing, let me do a couple tests to see if reverting to splitlines do not introduce any other issues.

One thing though is that by not sticking to the standard, the code would be lenient on using \n as the line delimiter but it will not parse standard-compliant messages correctly.

It's okay by me if that's the design compromise for the library (and I myself am not sure how often servers do break the rule, using \n instead of \r\n). But I remember forking the repo to generate this patch due to an issue I had with real response headers being truncated. So there definitely is a trade-off, albeit it might be rare.

(One possible workaround might be to inspect each header to see which of \n and \r\n is used as the line delimiter. I'm not sure that's desired or reliable.)

[okomestudio]

Sorry for taking a while but I finally managed to update the PR as requested. Please note my comment about the optional switch. I hope that the PR makes sense now.

[yozachar]

Bumping to close outdated PR.