Gor can't replay PUT requests

[ccontino84]

I'm using the latest version of Goreplay (1.3.3) and I'm trying to replicate HTTP PUT requests to another host. This is the command that I'm using, where I intentionally set big buffers and timeouts to rule out issues with that:

./gor --input-raw :8080 --output-http-queue-len 10000 --copy-buffer-size 209715200 --input-raw-buffer-size 209715200 --output-http-workers 150 --output-http-worker-timeout 120s --input-raw-override-snaplen --output-http-timeout 120s --input-raw-expire 10s --output-http-response-buffer 209715200 --input-raw-allow-incomplete --output-http http://hostname --output-http-stats --stats --output-http-stats-ms 10000 --prettify-http --output-stdout --verbose 2

To generate the PUT request I'm using CURL:

head -c 1000 /dev/urandom > /tmp/testfile curl -vsk -T /tmp/testfile "http://localhost:8080/upload" > /dev/null

Which produces the following output:

* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> PUT /upload HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 1000
> Expect: 100-continue
> 
< HTTP/1.1 100 
} [1000 bytes data]
* We are completely uploaded and fine
< HTTP/1.1 201 
< Location: http://localhost:8080/upload
< Content-Length: 0
< Date: Thu, 25 Nov 2021 11:39:20 GMT
< 
* Connection #0 to host localhost left intact

Now the result is that the request won't be forwarded to the target hostname by Goreplay and exactly after 10s (see --input-raw-expire) I see the following log in the output:

[DEBUG][elapsed 100.149516ms]: [INPUT-RAW] message timeout reached, increase input-raw-expire 1 c7b21f907f0000010bda5966 1637840360410126198 0 SOME BINARY DATA [DEBUG][elapsed 151.744µs]: [HTTP-OUTPUT] error when sending: "malformed HTTP request "\xe2\xb5aCB\xb2\x8c\xe1\\і\xafá\xc0\x1f\x06\xa2\xa7\xdb\xe51\xa7ߣ\xabd\b\u007f\x1e"" [DEBUG][elapsed 69.160343ms]: [HTTP-OUTPUT] error when sending: "Put "http://hostname/upload": readfrom tcp XXX.XXX.XXX.XXX:39974->YYY.YYY.YYY.YYY:80: unexpected EOF"

If I don't use the "--input-raw-allow-incomplete" parameter I don't even see the log above. Am I doing something wrong? How can I get Goreplay working in this scenario?

[bruce34]

I'm tripping over this as well. It looks like it is because of a refactor sometime between 1.0.0rc2 and 1.3.3 that now only accepts: "HTTP/1.1 100 Continue\r\n\r\n" instead of (for example): "HTTP/1.1 100 \r\n\r\n" (or indeed any "Reason Phrase")

https://github.com/buger/goreplay/blob/master/internal/tcp/tcp_message.go#L384 is where it is assumed to be exactly 25 bytes long. The code would presumably need to parse the server response and count the number of bytes for that line.

I'm not quite sure where the correct fix should go, as the code will need to know what the server sends back before increasing p.Ack appropriately. A quick debug showed this function only had the client request in m.Data() when this code was run for me.