Records larger than 4 MB cannot be processed with standalone plugins

[hariso]

Bug description

This was discovered while working on the performance benchmarks. Records larger than 4 MB in size cannot be streamed through a pipeline, if they've been read with a standalone plugin or if they need to be written using a standalone plugin.

Steps to reproduce

1. Create a pipeline
2. Add a built-in S3 source connector
3. Add a standalone file connector (but can be any other too)
4. Add a 5 MB file to S3
5. Wait a little bit

Expected behavior File is written to the destination file.

Actual behavior Following error appears:

2022-07-28T14:00:01+00:00 ERR destination acker node stopped before processing all messages error="nacked 1 messages when stopping destination acker node, some nacks failed: no DLQ handler configured" component=DestinationAckerNode node_id=fc3b6547-a28e-494f-81ba-8176cb83953b-acker pipeline_id=f14535da-3222-4856-9652-28c48684c1b6 stack=[{"file":"/home/haris/projects/conduitio/conduit/pkg/pipeline/stream/destination_acker.go","func":"github.com/conduitio/conduit/pkg/pipeline/stream.(*DestinationAckerNode).teardown","line":182},{"file":"/home/haris/projects/conduitio/conduit/pkg/pipeline/stream/source_acker.go","func":"github.com/conduitio/conduit/pkg/pipeline/stream.(*SourceAckerNode).registerNackHandler.func1","line":124}]
2022-07-28T14:00:01+00:00 ERR node stopped error="node fc3b6547-a28e-494f-81ba-8176cb83953b-acker stopped with error: failed to receive ack: error receiving ack: grpc: received message larger than max (5000117 vs. 4194304)" component=pipeline.Service node_id=fc3b6547-a28e-494f-81ba-8176cb83953b-acker stack=[{"file":"/home/haris/projects/conduitio/conduit/pkg/pipeline/lifecycle.go","func":"github.com/conduitio/conduit/pkg/pipeline.(*Service).runPipeline.func1","line":403},{"file":"/home/haris/projects/conduitio/conduit/pkg/pipeline/stream/destination_acker.go","func":"github.com/conduitio/conduit/pkg/pipeline/stream.(*DestinationAckerNode).worker","line":131},{"file":"/home/haris/projects/conduitio/conduit/pkg/connector/destination.go","func":"github.com/conduitio/conduit/pkg/connector.(*destination).Ack","line":242},{"file":"/home/haris/projects/conduitio/conduit/pkg/plugin/standalone/v1/client.go","func":"github.com/conduitio/conduit/pkg/plugin/standalone/v1.unwrapGRPCError","line":160}]

Notice stopped with error: failed to receive ack: error receiving ack: grpc: received message larger than max (5000117 vs. 4194304). By default, gRPC messages are limited to 4 MB in size. This default value is used in Conduit too.

Version

{"version":"v0.3.0-nightly.20220727","os":"linux","arch":"amd64"}

[uchennakevinm1]

This affects all of the connectors and is a GRPC limitation. Call this as a limitation in the docs and postpone it.