HTTP.jl icon indicating copy to clipboard operation
HTTP.jl copied to clipboard

HTTP.download() does not automatically decode Content-Encoding

Open staticfloat opened this issue 3 years ago • 1 comments

Verisons

Julia v1.7.3 HTTP.jl v1.2.0 MbedTLS.jl v1.1.1

Issue

When calling HTTP.get() on a resource with Content-Encoding: gzip, the response is automatically decoded for me:

julia> using HTTP
       io = IOBuffer()
       r = HTTP.get("https://httpbin.org/gzip", response_stream=io);
       println(String(take!(io)))
{
  "gzipped": true, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "HTTP.jl/1.7.3", 
    "X-Amzn-Trace-Id": "Root=1-62d6f292-06e09add2b03b541490576d4"
  }, 
  "method": "GET", 
  "origin": "75.172.111.107"
}

Using HTTP.download(), it does not:

julia> using HTTP
       HTTP.download("https://httpbin.org/gzip", "/dev/stdout"; update_period=Inf);
���b�=��j�0�w?��*Œ]�:��B�R\�*[gY���|]���x����ݵ(Kb.vY@���<��͠4�5�k#���3��wd
                                                                         eJ�n��:�tj���;����9�*��_��q�c>���3q5%N}��~~��IV��7��G��F�o�|��=>s�
�QZL��u�&C����p$6�g�-��{7������K�8�W����Sj2

staticfloat avatar Jul 19 '22 18:07 staticfloat

As far as I found, this is a regression introduced since v1.0.0, for files that could be downloaded correctly without compression

the thing is:

  1. #838 implemented gzip decompression for HTTP.get but not for HTTP.download
  2. #851 set "Accept-Encoding" => "gzip" as default header (R44-R46)

it works for me immediately by either:

  1. simply run pkg> pin HTTP@0 to use v0.9.17
  2. counterintuitively though, use HTTP.download(decompress = false, args...)

Heptazhou avatar Jul 24 '22 20:07 Heptazhou