ClickHouse.Client icon indicating copy to clipboard operation
ClickHouse.Client copied to clipboard

Add overload for ClickHouseConnection.PostStreamAsync to support 'push stream'

Open i-sinister opened this issue 3 years ago • 0 comments

I'm having performance issues when sending large amount of data 'computed in c#' to ClickHouse. Similar problem was described in #60 and #61.

Current version of ClickHouseConnection.PostStreamAsync expects Stream which means that objects has to be first serialized into temporary MemoryStream or FileStream. This causes performance problems when produced stream is large (excessive memory usage or IO delays in case of FileStream).

Having an overload like the one below will allow to avoid unnecessary copying and, for some formats like tsv or json, to send data without using intermediate temporary stream, similar to how nowadays dotnet core formatters work:

        public async Task PostStreamAsync(
                string sql,
                string contentType,
                Action<Stream, HttpContent, TransportContext> writer)
        {
                ...
                postMessage.Content = new PushStreamContent(writer);
                postMessage.Content.Headers.ContentType = new MediaTypeHeaderValue(contentType);
                ...
        }

i-sinister avatar Jul 06 '22 06:07 i-sinister