Throughput stats is misleading

[vitaly-castLabs]

What version of bombardier are you using?

v1.2.6 darwin/arm64

What did you do?

I run https://github.com/qicosmos/cinatra/blob/master/example/benchmark.cpp against ./bombardier -c 100 -n 500000 http://localhost:8090 with and without dummy headers:

1. No cheating, original code: throughput is ~40 MBps;
2. Option A. Add a dummy header to the response: throughput is now ~160 MBps:

resp.add_header("x-perf-dummy", "x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy");

2. Option B. Pad the body (curl http://localhost:8090/ will still return Hello, world! with this) and get throughput jump to ~53 MBps:

resp.set_status_and_content(status_type::ok, "Hello, world!\rHello, world!\rHello, world!\rHello, world!\rHello, world!");

My point is

Throughput alone is very unreliable - even without explicit cheating a server setting a bunch of optional headers on responses will have a major advantage. What I'd do is:

• Report performance in requests per seconds foremost;
• Calculate throughput only using the body size, headers excluded. Maybe also add a warning if Content-Length is not 13

[sebastienros]

I personally don't take throughput as an indicator of performance, but a way to check that the network is not a bottleneck and that the benchmark works as expected. And also think that headers should definitely be part of this measure otherwise it would have no value.

We agree that RPS is the metric to take into account, not throughput.

[codesenberg]

@vitaly-castLabs

• Report performance in requests per seconds foremost;
• Calculate throughput only using the body size, headers excluded. Maybe also add a warning if Content-Length is not 13

That's some very specific requirements. I suggest doing this in your own fork of the project.

Throughput stats is misleading

It's probably not immediately obvious what throughput is, but after all this time, I kind of expect people to assume that it's the total amount of data sent + received divided by test time as defined here: https://github.com/codesenberg/bombardier/blob/1c21a01b3dd29a0d5fea9d9556b2c8ddbede2cba/internal/test_info.go#L110-L115 Suddenly changing the behaviour/meaning might be unexpected by most users to say the least.