Gunzip request and Gzipped again does not behave correctly

[Justin99b]

When trying to manipulate a beforeResponse, gunzip the request and gzip it again and set the request.body.buffer = zlib.gzipSync(request.body.buffer) and catching this with HttpToolkit this happends,

I dont know if this is a HttpToolkit issue or a Mockttp issue

Its possible i also misunderstanding something or mockttp already gunzip or gzips, that i dont know of

Or this happens

[Justin99b]

Maybe the fault is the Chunking? Because if i replace the buffer in beforeRequest the buffer on .getText will be the changed one but whats being send is actually the original buffer if i saw that correctly

[pimterry]

Ok, so you're using Mockttp in front of HTTP Toolkit, is that right? I.e. the client requests goes through Mockttp, and then gets forwarded to HTTP Toolkit? That's unusual (normally it makes more sense to put Mockttp on the upstream side as a proxy, between HTTP Toolkit & the target server) but yes it should work if that's what you want.

In general, if the content-encoding header is correct, Mockttp will automatically decode & re-encode request .body content for you, so you don't need to do that. You can use .rawBody to manually override content bodies without this behaviour, if you like.

In either case though, the main problem here must be your content-length header. The request body that you're sending appears to be longer than the content length in your headers. In effect, you're telling the server "I'm going to send 10 bytes of body" and then you're sending 20 bytes. That means:

• The body of the request is effectively truncated, so when trying to decode the body HTTP Toolkit reaches the 'end' (the length defined by your headers) without receiving a full gzip-format response. This causes the unexpected end of file error.
• The rest of the body (after the length) is considered as a separate request, but it's garbage (it's the remaining chunks of gzipped data) and so it fails to parse as an HTTP request. This causes the unsupported HTTP method error.

I think you probably want to:

• Ignore gunzip/gzip entirely, as long as your content-encoding headers are correct
• Drop the 'content-length' from the headers you return (with automatic encoding you'll always get this wrong). If you don't need to modify the headers, you don't need to return them at all, and they'll be regenerated automatically. Alternatively you can use transfer-encoding: chunked to avoid needing an explicit length, or return headers without either header in which case all data is part of the request, and then the connection closes at the end - less efficient for keep-alives but very simple.

In each case here, from what I can see both Mockttp & HTTP Toolkit are working correctly, the issue is that your code is explicitly trying to send an HTTP request that basically doesn't make sense. Mockttp will mostly handle things automatically to make sure requests are valid, but if you directly provide invalid headers etc then it won't stop you.

If you can provide a demo with your full code, that would help a lot to be more specific about what's going on here.

[Justin99b]

Ok, so you're using Mockttp in front of HTTP Toolkit, is that right? I.e. the client requests goes through Mockttp, and then gets forwarded to HTTP Toolkit? That's unusual

Well i am actually using HttpToolkit to see if my modified requests are formatted and changed exactly how i want them to be before sending them to the Upstream. Its extremly usefull for debugging since just using the console is not that great. Would be awesome if it would be possible to connect them together without having to use HttpToolkit as a nother proxy but rather than connect to the admin api and using HttpToolkit as sort of a frontend for everything to debug

Im currently unable to provide an example but i see the problem now.

i mentioned in #160 How im handling requests with calling beforeRequest and calling different beforeReqeusts.

Apperently this does not work for Chunked requests since you only get a portion of it.

And then the gzip is malformed.

The rest of the body (after the length) is considered as a separate request, but it's garbage (it's the remaining chunks of gzipped data) and so it fails to parse as an HTTP request. This causes the unsupported HTTP method error. I see so bascially everything that goes over the specified content-length is a seperet request?

If thats true then it would make sense why i saw many weird malformed requests before.

[pimterry]

Apperently this does not work for Chunked requests since you only get a portion of it.

Can you explain exactly what you mean by 'chunked' requests? It would be very helpful to be able to reproduce what you're seeing because there's something quite unusual happening here. Can you share an example request I can test with curl that doesn't work as expected, for example?

If you're talking about transfer-encoding: chunked - that will be handled automatically, so beforeRequest won't be called until all chunks have been completed, and it will include the whole body. I'm not aware of any scenario where you only get a portion of a valid request - this should only happen if your HTTP request has serious issues.

I think your main problem here almost certainly comes down to how you're updating the headers to match the changes you're making to the body, but it's difficult to be more specific without being able to see the code.