Responses larger than 1MB have incorrect content length header

[patrickbcullen]

I just found a problem with vulcand that uses the stream package in this library. When the responses is more than 1MB, it attempts to chunk the response into 1MB chunks and send them back to the client. However the content-length header does not match the actual content size.

Curl returns the following error:

curl: (18) transfer closed with 16384 bytes remaining to read

When I look at the headers, the Content-Length is exactly 1MB + 16384 bytes and there is only 1MB of content in the response. I have not been able to pinpoint exactly how the chunking logic works, but I will try to create a unit test showing the problem as soon as I can. If you have any ideas where the problem may be let me know.

[MaikuMori]

I'm having similar problem where response is being truncated at 1048576 bytes when in fact it's 1237189 bytes long.

I'm using default stream settings (1MB in memory).

[iamruinous]

We ran into a similar problem with this when running vulcand in Docker scratch image. The fix came down to making an empty /tmp in our Dockerfile, otherwise ioutil.TempFile fails.

We'll be submitting a PR for oxy/fwd to at least log the error, it is being ignored as of right now.

[MaikuMori]

Thanks. I think this can be marked as resolved. I'm sure people will run into this problem in the future but the fix is clearly out of scope of this project (besides actually reporting the error).