Performance: 1000 large messages per second gives a throughput of cca. 250 msg/s

[hariso]

Bug description

In the workload with 1k large messages (4 MB each) per second, we see that the pipeline throughput is 250 msg/s. We should investigate the reasons for the large discrepancy.

Steps to reproduce

1. Run https://github.com/ConduitIO/streaming-benchmarks/blob/haris/ec2-helpers/workloads/large-messages-1k-msg-per-sec.sh

Version

{ "version": "v0.3.0-nightly.20220811", "os": "linux", "arch": "amd64" }

[neovintage]

This might be related to the other issue where we're seeing drift: #571. The message size got bigger and we started to see the drift happen almost immediately with a smaller set of messages but message size got bigger.